2023-03-03T05:25:17.580 INFO:root:teuthology version: 0.0.1.dev49+gc7c3cc8 2023-03-03T05:25:17.581 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192093 --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --owner scheduled_vshankar@teuthology --verbose -- /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192093/orig.config.yaml 2023-03-03T05:25:17.649 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-03T05:25:17.736 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192093 branch: wip-vshankar-testing-20230303.023823 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: vshankar@redhat.com first_in_suite: false job_id: '7192093' kernel: client: branch: testing flavor: default kdb: true sha1: 75534f74ac3dd06f86697ff862e8be2945b3c8e6 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-vshankar-testing-20230303.023823 ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20230303.023823 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20230303.023823 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOZkGy1FXGwI7zar+JgG/pI7NbBPl0NOU0x42EcareR46Y0bdu2QAQdWacApFIsIq4S32kMe+audUeP0nlFn4Y= smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1SyCVoQPMW4XOWT/PxlpBz+odYm8m6q9lBogugRDxqVJ5FiaELoLATbj70I+3K/pcqEXXmsrIxYFalyDvRhIqJ5tUsqy/rinKy/S6kyX9ESiUmKWQHEOHe2Yyoy5YrjH18+LG9bgYyR8nI+0ifqdEJ0+t+nzPkN50DDJMPYoSdB7tvCJDeLNz8vOJ9nX7jqtTyWg2dtSFW/Ah0YoOTDieL3p67NoJ0Gf6uSHR8p0J0to4hC90yY/UAG4EQBtOMM4ZN/0DdPnKyQvuN9sGQ+XpTqQ+ok0HUulGbn0oEaovOL7glS1eipn+aqV0BNdlyldxTjQju5kdw7KVL6WFrH1cydV7KAnT04x6uv7dCGFj5199v2HZg47aAj9AYV7K08JUUsbPPiL7WolOXjcJbUAvLVlyrNDPoxGOtPYv0Fnq2oNCj6bK6IoN0vH0lXipVf4BjdIYxXxaPS9GXZGV2iQ2anuGvi2HBJ+KlYFSJgrqYFLLH6dqmyRApgmGknw1XcU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6 timestamp: 2023-03-03_04:39:14 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.31660 2023-03-03T05:25:17.737 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa; will attempt to use it 2023-03-03T05:25:17.737 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa/tasks 2023-03-03T05:25:17.738 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-03T05:25:17.739 INFO:teuthology.task.internal:Checking packages... 2023-03-03T05:25:17.768 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '472aaef4ec3b3a2c29f73ca67854785a910f1f9b' 2023-03-03T05:25:17.768 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-03T05:25:17.769 INFO:teuthology.packaging:ref: None 2023-03-03T05:25:17.769 INFO:teuthology.packaging:tag: None 2023-03-03T05:25:17.769 INFO:teuthology.packaging:branch: wip-vshankar-testing-20230303.023823 2023-03-03T05:25:17.770 INFO:teuthology.packaging:sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T05:25:17.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20230303.023823 2023-03-03T05:25:17.910 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-2739.g472aaef4 2023-03-03T05:25:17.910 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-03T05:25:17.950 INFO:teuthology.task.internal:no buildpackages task found 2023-03-03T05:25:17.951 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-03T05:25:17.958 INFO:teuthology.task.internal:Saving configuration 2023-03-03T05:25:17.981 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-03T05:25:17.986 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-03T05:25:18.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192093', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 05:20:06.320231', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOZkGy1FXGwI7zar+JgG/pI7NbBPl0NOU0x42EcareR46Y0bdu2QAQdWacApFIsIq4S32kMe+audUeP0nlFn4Y='} 2023-03-03T05:25:18.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192093', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 05:20:06.318684', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPgCK5wlzHtHtNsp2KQRa6bSNuai9zaVF2YFhVFG9eOvCDO9+MVl0CgwzAPJG7dPYrWmDq1hjeZMsY+/zjmNNNw='} 2023-03-03T05:25:18.039 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-03T05:25:18.044 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-03-03T05:25:18.045 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-03-03T05:25:18.045 INFO:teuthology.run_tasks:Running task console_log... 2023-03-03T05:25:18.126 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1e37f1f5e0>, signals=[15]) 2023-03-03T05:25:18.127 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-03T05:25:18.139 INFO:teuthology.task.internal:Opening connections... 2023-03-03T05:25:18.140 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2023-03-03T05:25:18.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T05:25:18.210 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-03-03T05:25:18.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T05:25:18.282 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-03T05:25:18.309 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2023-03-03T05:25:18.329 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2023-03-03T05:25:18.330 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2023-03-03T05:25:18.389 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2023-03-03T05:25:18.389 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="8" 2023-03-03T05:25:18.389 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2023-03-03T05:25:18.389 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2023-03-03T05:25:18.390 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="8" 2023-03-03T05:25:18.390 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el8" 2023-03-03T05:25:18.390 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-03T05:25:18.390 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2023-03-03T05:25:18.390 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-03T05:25:18.390 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2023-03-03T05:25:18.390 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T05:25:18.391 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T05:25:18.391 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-03T05:25:18.391 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2023-03-03T05:25:18.417 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-03-03T05:25:18.438 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-03-03T05:25:18.438 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-03-03T05:25:18.497 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2023-03-03T05:25:18.498 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8" 2023-03-03T05:25:18.498 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2023-03-03T05:25:18.498 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2023-03-03T05:25:18.498 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8" 2023-03-03T05:25:18.498 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2023-03-03T05:25:18.499 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-03T05:25:18.499 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2023-03-03T05:25:18.499 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-03T05:25:18.499 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2023-03-03T05:25:18.499 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T05:25:18.499 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T05:25:18.499 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-03T05:25:18.500 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-03-03T05:25:18.525 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-03T05:25:18.550 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-03T05:25:18.556 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-03T05:25:18.557 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2023-03-03T05:25:18.559 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-03-03T05:25:18.578 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-03T05:25:18.588 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-03T05:25:18.589 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2023-03-03T05:25:18.619 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-03-03T05:25:18.645 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-03T05:25:18.716 INFO:teuthology.run_tasks:Running task kernel... 2023-03-03T05:25:18.744 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'} 2023-03-03T05:25:18.745 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-03-03T05:25:18.745 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-03-03T05:25:18.746 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T05:25:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T05:25:18.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-03-03T05:25:18.987 DEBUG:teuthology.task.kernel:sha1 for client.0 is 75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T05:25:18.987 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "75534f74ac3dd06f86697ff862e8be2945b3c8e6"... 2023-03-03T05:25:18.987 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2023-03-03T05:25:19.007 INFO:teuthology.orchestra.run.smithi102.stdout:4.18.0-408.el8.x86_64 2023-03-03T05:25:19.007 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-408.el8.x86_64 vs 75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T05:25:19.008 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-03-03T05:25:19.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum upgrade -y linux-firmware 2023-03-03T05:25:22.881 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - AppStream 30 MB/s | 28 MB 00:00 2023-03-03T05:25:28.748 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - BaseOS 35 MB/s | 27 MB 00:00 2023-03-03T05:25:33.047 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Extras 186 kB/s | 18 kB 00:00 2023-03-03T05:25:33.491 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - PowerTools 18 MB/s | 5.7 MB 00:00 2023-03-03T05:25:37.124 INFO:teuthology.orchestra.run.smithi102.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-03-03T05:25:39.082 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 14 MB 00:01 2023-03-03T05:25:42.440 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 430 kB/s | 24 kB 00:00 2023-03-03T05:25:45.854 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2023-03-03T05:25:45.855 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2023-03-03T05:25:45.855 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2023-03-03T05:25:45.984 INFO:teuthology.task.kernel:Downloading kernel 75534f74ac3dd06f86697ff862e8be2945b3c8e6 on client.0... 2023-03-03T05:25:45.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=75534f74ac3dd06f86697ff862e8be2945b3c8e6 2023-03-03T05:25:46.132 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/ 2023-03-03T05:25:46.133 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.2.0_ceph_g75534f74ac3d-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/ --input-file=- 2023-03-03T05:25:46.135 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi102.front.sepia.ceph.com to complete... 2023-03-03T05:26:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03 05:26:19 URL:https://3.chacra.ceph.com/r/kernel/testing/75534f74ac3dd06f86697ff862e8be2945b3c8e6/centos/8/flavors/default/x86_64/kernel-6.2.0_ceph_g75534f74ac3d-1.x86_64.rpm [383026056/383026056] -> "/tmp/kernel.x86_64.rpm" [1] 2023-03-03T05:26:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:FINISHED --2023-03-03 05:26:19-- 2023-03-03T05:26:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:Total wall clock time: 33s 2023-03-03T05:26:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:Downloaded: 1 files, 365M in 33s (11.1 MB/s) 2023-03-03T05:26:19.532 INFO:teuthology.task.kernel:Installing kernel 75534f74ac3dd06f86697ff862e8be2945b3c8e6 on client.0... 2023-03-03T05:26:19.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-03-03T05:26:21.252 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying... ######################################## 2023-03-03T05:26:23.958 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing... ######################################## 2023-03-03T05:26:24.096 INFO:teuthology.orchestra.run.smithi102.stdout:Updating / installing... 2023-03-03T05:27:02.765 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-6.2.0_ceph_g75534f74ac3d-1 ######################################## 2023-03-03T05:27:23.749 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-03-03T05:27:23.749 DEBUG:teuthology.orchestra.run.smithi102:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-03-03T05:27:23.797 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/System.map-6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:23.798 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/config-6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:23.798 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/vmlinuz-6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:23.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:23.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel 2023-03-03T05:27:23.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch 2023-03-03T05:27:23.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86 2023-03-03T05:27:23.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto 2023-03-03T05:27:23.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-03-03T05:27:23.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-03-03T05:27:23.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-03-03T05:27:23.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-03-03T05:27:23.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-03-03T05:27:23.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crc32c-intel.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-avx2.ko 2023-03-03T05:27:23.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-03-03T05:27:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-03-03T05:27:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-03-03T05:27:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-03-03T05:27:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-03-03T05:27:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-03-03T05:27:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-03-03T05:27:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel 2023-03-03T05:27:23.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu 2023-03-03T05:27:23.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu/mce 2023-03-03T05:27:23.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-03-03T05:27:23.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm 2023-03-03T05:27:23.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm-amd.ko 2023-03-03T05:27:23.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm-intel.ko 2023-03-03T05:27:23.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/arch/x86/kvm/kvm.ko 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/ansi_cprng.ko 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/anubis.ko 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/arc4.ko 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_memcpy.ko 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_pq.ko 2023-03-03T05:27:23.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_raid6_recov.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_tx.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/async_xor.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/async_tx/raid6test.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/authenc.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/authencesn.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blake2b_generic.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blowfish_common.ko 2023-03-03T05:27:23.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/blowfish_generic.ko 2023-03-03T05:27:23.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/camellia_generic.ko 2023-03-03T05:27:23.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast5_generic.ko 2023-03-03T05:27:23.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast6_generic.ko 2023-03-03T05:27:23.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cast_common.ko 2023-03-03T05:27:23.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/ccm.ko 2023-03-03T05:27:23.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cmac.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crc32_generic.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crypto_engine.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/crypto_user.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/cts.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/des_generic.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/dh_generic.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/echainiv.ko 2023-03-03T05:27:23.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/essiv.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/fcrypt.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/khazad.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/lz4.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/lz4hc.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/md4.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/michael_mic.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/pcbc.ko 2023-03-03T05:27:23.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/pcrypt.ko 2023-03-03T05:27:23.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/rmd160.ko 2023-03-03T05:27:23.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/seed.ko 2023-03-03T05:27:23.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/serpent_generic.ko 2023-03-03T05:27:23.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/tcrypt.ko 2023-03-03T05:27:23.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/tea.ko 2023-03-03T05:27:23.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/twofish_common.ko 2023-03-03T05:27:23.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/twofish_generic.ko 2023-03-03T05:27:23.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/vmac.ko 2023-03-03T05:27:23.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/wp512.ko 2023-03-03T05:27:23.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xcbc.ko 2023-03-03T05:27:23.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xor.ko 2023-03-03T05:27:23.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/crypto/xxhash_generic.ko 2023-03-03T05:27:23.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers 2023-03-03T05:27:23.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi 2023-03-03T05:27:23.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/acpi_ipmi.ko 2023-03-03T05:27:23.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/acpi_pad.ko 2023-03-03T05:27:23.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/custom_method.ko 2023-03-03T05:27:23.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/ec_sys.ko 2023-03-03T05:27:23.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/platform_profile.ko 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/sbs.ko 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/sbshc.ko 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/acpi/video.ko 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/acard-ahci.ko 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/ahci_platform.ko 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/ata_generic.ko 2023-03-03T05:27:23.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/libahci_platform.ko 2023-03-03T05:27:23.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_acpi.ko 2023-03-03T05:27:23.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ali.ko 2023-03-03T05:27:23.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_amd.ko 2023-03-03T05:27:23.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_artop.ko 2023-03-03T05:27:23.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_atiixp.ko 2023-03-03T05:27:23.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_atp867x.ko 2023-03-03T05:27:23.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cmd640.ko 2023-03-03T05:27:23.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cmd64x.ko 2023-03-03T05:27:23.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_cypress.ko 2023-03-03T05:27:23.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_efar.ko 2023-03-03T05:27:23.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt366.ko 2023-03-03T05:27:23.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt37x.ko 2023-03-03T05:27:23.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt3x2n.ko 2023-03-03T05:27:23.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_hpt3x3.ko 2023-03-03T05:27:23.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_it8213.ko 2023-03-03T05:27:23.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_it821x.ko 2023-03-03T05:27:23.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_jmicron.ko 2023-03-03T05:27:23.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_marvell.ko 2023-03-03T05:27:23.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_mpiix.ko 2023-03-03T05:27:23.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_netcell.ko 2023-03-03T05:27:23.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ninja32.ko 2023-03-03T05:27:23.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ns87410.ko 2023-03-03T05:27:23.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_ns87415.ko 2023-03-03T05:27:23.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_oldpiix.ko 2023-03-03T05:27:23.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_opti.ko 2023-03-03T05:27:23.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_optidma.ko 2023-03-03T05:27:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pcmcia.ko 2023-03-03T05:27:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pdc2027x.ko 2023-03-03T05:27:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-03-03T05:27:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_piccolo.ko 2023-03-03T05:27:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_rdc.ko 2023-03-03T05:27:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sch.ko 2023-03-03T05:27:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_serverworks.ko 2023-03-03T05:27:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sil680.ko 2023-03-03T05:27:23.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sis.ko 2023-03-03T05:27:23.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_sl82c105.ko 2023-03-03T05:27:23.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_triflex.ko 2023-03-03T05:27:23.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pata_via.ko 2023-03-03T05:27:23.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/pdc_adma.ko 2023-03-03T05:27:23.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_inic162x.ko 2023-03-03T05:27:23.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_mv.ko 2023-03-03T05:27:23.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_nv.ko 2023-03-03T05:27:23.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_promise.ko 2023-03-03T05:27:23.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_qstor.ko 2023-03-03T05:27:23.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sil.ko 2023-03-03T05:27:23.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sil24.ko 2023-03-03T05:27:23.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sis.ko 2023-03-03T05:27:23.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_svw.ko 2023-03-03T05:27:23.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_sx4.ko 2023-03-03T05:27:23.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_uli.ko 2023-03-03T05:27:23.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_via.ko 2023-03-03T05:27:23.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ata/sata_vsc.ko 2023-03-03T05:27:23.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/auxdisplay 2023-03-03T05:27:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/auxdisplay/ks0108.ko 2023-03-03T05:27:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base 2023-03-03T05:27:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base/regmap 2023-03-03T05:27:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/base/regmap/regmap-i2c.ko 2023-03-03T05:27:23.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/bcma 2023-03-03T05:27:23.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/bcma/bcma.ko 2023-03-03T05:27:23.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block 2023-03-03T05:27:23.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/aoe 2023-03-03T05:27:23.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/aoe/aoe.ko 2023-03-03T05:27:23.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/brd.ko 2023-03-03T05:27:23.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/drbd 2023-03-03T05:27:23.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/drbd/drbd.ko 2023-03-03T05:27:23.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/floppy.ko 2023-03-03T05:27:23.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/loop.ko 2023-03-03T05:27:23.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/mtip32xx 2023-03-03T05:27:23.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-03-03T05:27:23.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/nbd.ko 2023-03-03T05:27:23.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/null_blk 2023-03-03T05:27:23.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/null_blk/null_blk.ko 2023-03-03T05:27:23.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/pktcdvd.ko 2023-03-03T05:27:23.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/rbd.ko 2023-03-03T05:27:23.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/virtio_blk.ko 2023-03-03T05:27:23.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkback 2023-03-03T05:27:23.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-03-03T05:27:23.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/xen-blkfront.ko 2023-03-03T05:27:23.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/zram 2023-03-03T05:27:23.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/block/zram/zram.ko 2023-03-03T05:27:23.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char 2023-03-03T05:27:23.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hangcheck-timer.ko 2023-03-03T05:27:23.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random 2023-03-03T05:27:23.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/amd-rng.ko 2023-03-03T05:27:23.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/intel-rng.ko 2023-03-03T05:27:23.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-03-03T05:27:23.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/via-rng.ko 2023-03-03T05:27:23.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/hw_random/virtio-rng.ko 2023-03-03T05:27:23.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi 2023-03-03T05:27:23.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-03-03T05:27:23.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-03-03T05:27:23.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-03-03T05:27:23.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_si.ko 2023-03-03T05:27:23.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-03-03T05:27:23.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-03-03T05:27:23.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/lp.ko 2023-03-03T05:27:23.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/mwave 2023-03-03T05:27:23.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/mwave/mwave.ko 2023-03-03T05:27:23.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia 2023-03-03T05:27:23.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia/cm4000_cs.ko 2023-03-03T05:27:23.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/pcmcia/cm4040_cs.ko 2023-03-03T05:27:23.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/ppdev.ko 2023-03-03T05:27:23.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tlclk.ko 2023-03-03T05:27:23.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm 2023-03-03T05:27:23.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm.ko 2023-03-03T05:27:23.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_atmel.ko 2023-03-03T05:27:23.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_infineon.ko 2023-03-03T05:27:23.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_nsc.ko 2023-03-03T05:27:23.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_tis.ko 2023-03-03T05:27:23.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-03-03T05:27:23.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/uv_mmtimer.ko 2023-03-03T05:27:23.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/char/virtio_console.ko 2023-03-03T05:27:23.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq 2023-03-03T05:27:23.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-03-03T05:27:23.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-03-03T05:27:23.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/p4-clockmod.ko 2023-03-03T05:27:23.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-03-03T05:27:23.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/powernow-k8.ko 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/cpufreq/speedstep-lib.ko 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/ccp/ccp.ko 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/padlock-aes.ko 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/padlock-sha.ko 2023-03-03T05:27:23.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat 2023-03-03T05:27:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_common 2023-03-03T05:27:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2023-03-03T05:27:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_dh895xcc 2023-03-03T05:27:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-03-03T05:27:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/virtio 2023-03-03T05:27:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-03-03T05:27:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dca 2023-03-03T05:27:23.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dca/dca.ko 2023-03-03T05:27:23.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/devfreq 2023-03-03T05:27:23.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-03-03T05:27:23.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma 2023-03-03T05:27:23.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/dw 2023-03-03T05:27:23.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/dw/dw_dmac.ko 2023-03-03T05:27:23.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/ioat 2023-03-03T05:27:23.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/dma/ioat/ioatdma.ko 2023-03-03T05:27:23.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac 2023-03-03T05:27:23.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/amd64_edac.ko 2023-03-03T05:27:23.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/e752x_edac.ko 2023-03-03T05:27:23.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/edac_mce_amd.ko 2023-03-03T05:27:23.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i3000_edac.ko 2023-03-03T05:27:23.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i3200_edac.ko 2023-03-03T05:27:23.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i5100_edac.ko 2023-03-03T05:27:23.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i5400_edac.ko 2023-03-03T05:27:23.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i7300_edac.ko 2023-03-03T05:27:23.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i7core_edac.ko 2023-03-03T05:27:23.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/i82975x_edac.ko 2023-03-03T05:27:23.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/ie31200_edac.ko 2023-03-03T05:27:23.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/sb_edac.ko 2023-03-03T05:27:23.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/edac/x38_edac.ko 2023-03-03T05:27:23.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware 2023-03-03T05:27:23.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware/edd.ko 2023-03-03T05:27:23.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/firmware/iscsi_ibft.ko 2023-03-03T05:27:23.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio/gpio-ich.ko 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/gpio/gpio-viperboard.ko 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-appleir.ko 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-aureal.ko 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-axff.ko 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-dr.ko 2023-03-03T05:27:23.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-elecom.ko 2023-03-03T05:27:23.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-elo.ko 2023-03-03T05:27:23.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-emsff.ko 2023-03-03T05:27:23.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gaff.ko 2023-03-03T05:27:23.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gt683r.ko 2023-03-03T05:27:23.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-gyration.ko 2023-03-03T05:27:23.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtek-kbd.ko 2023-03-03T05:27:23.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtek-mouse.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-holtekff.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-hyperv.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-icade.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-keytouch.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-kye.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-lcpower.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-led.ko 2023-03-03T05:27:23.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-lenovo.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-logitech-dj.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-multitouch.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-ortek.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-penmount.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-petalynx.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-picolcd.ko 2023-03-03T05:27:23.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-pl.ko 2023-03-03T05:27:23.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-plantronics.ko 2023-03-03T05:27:23.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-primax.ko 2023-03-03T05:27:23.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-rmi.ko 2023-03-03T05:27:23.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-arvo.ko 2023-03-03T05:27:23.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-common.ko 2023-03-03T05:27:23.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-isku.ko 2023-03-03T05:27:23.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-kone.ko 2023-03-03T05:27:23.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-03-03T05:27:23.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-konepure.ko 2023-03-03T05:27:23.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-03-03T05:27:23.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-lua.ko 2023-03-03T05:27:23.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-pyra.ko 2023-03-03T05:27:23.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-ryos.ko 2023-03-03T05:27:23.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat-savu.ko 2023-03-03T05:27:23.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-roccat.ko 2023-03-03T05:27:23.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-saitek.ko 2023-03-03T05:27:23.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-samsung.ko 2023-03-03T05:27:23.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sensor-hub.ko 2023-03-03T05:27:23.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sjoy.ko 2023-03-03T05:27:23.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sony.ko 2023-03-03T05:27:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-speedlink.ko 2023-03-03T05:27:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-steelseries.ko 2023-03-03T05:27:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-sunplus.ko 2023-03-03T05:27:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-thrustmaster.ko 2023-03-03T05:27:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-tivo.ko 2023-03-03T05:27:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-tmff.ko 2023-03-03T05:27:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-topseed.ko 2023-03-03T05:27:23.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-twinhan.ko 2023-03-03T05:27:23.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-uclogic.ko 2023-03-03T05:27:23.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-waltop.ko 2023-03-03T05:27:23.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-wiimote.ko 2023-03-03T05:27:23.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-xinmo.ko 2023-03-03T05:27:23.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-zpff.ko 2023-03-03T05:27:23.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/hid-zydacron.ko 2023-03-03T05:27:23.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/uhid.ko 2023-03-03T05:27:23.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hid/wacom.ko 2023-03-03T05:27:23.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv 2023-03-03T05:27:23.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_balloon.ko 2023-03-03T05:27:23.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_utils.ko 2023-03-03T05:27:23.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hv/hv_vmbus.ko 2023-03-03T05:27:23.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon 2023-03-03T05:27:23.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/abituguru.ko 2023-03-03T05:27:23.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/abituguru3.ko 2023-03-03T05:27:23.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/acpi_power_meter.ko 2023-03-03T05:27:23.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ad7414.ko 2023-03-03T05:27:23.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ad7418.ko 2023-03-03T05:27:23.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adc128d818.ko 2023-03-03T05:27:23.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1025.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1026.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1029.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm1031.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adm9240.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ads7828.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7410.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7411.ko 2023-03-03T05:27:23.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7462.ko 2023-03-03T05:27:23.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7470.ko 2023-03-03T05:27:23.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7475.ko 2023-03-03T05:27:23.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/adt7x10.ko 2023-03-03T05:27:23.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/amc6821.ko 2023-03-03T05:27:23.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/applesmc.ko 2023-03-03T05:27:23.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asb100.ko 2023-03-03T05:27:23.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asc7621.ko 2023-03-03T05:27:23.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/asus_atk0110.ko 2023-03-03T05:27:23.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/atxp1.ko 2023-03-03T05:27:23.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/coretemp.ko 2023-03-03T05:27:23.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-03-03T05:27:23.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/dme1737.ko 2023-03-03T05:27:23.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ds1621.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ds620.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/emc1403.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/emc6w201.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f71805f.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f71882fg.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/f75375s.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/fam15h_power.ko 2023-03-03T05:27:23.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/fschmd.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/g760a.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/g762.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/gl518sm.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/gl520sm.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/hwmon-vid.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/i5500_temp.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/i5k_amb.ko 2023-03-03T05:27:23.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ibmaem.ko 2023-03-03T05:27:23.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ibmpex.ko 2023-03-03T05:27:23.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ina209.ko 2023-03-03T05:27:23.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ina2xx.ko 2023-03-03T05:27:23.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/it87.ko 2023-03-03T05:27:23.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/k10temp.ko 2023-03-03T05:27:23.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/k8temp.ko 2023-03-03T05:27:23.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lineage-pem.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm63.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm73.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm75.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm77.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm78.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm80.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm83.ko 2023-03-03T05:27:23.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm85.ko 2023-03-03T05:27:23.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm87.ko 2023-03-03T05:27:23.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm90.ko 2023-03-03T05:27:23.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm92.ko 2023-03-03T05:27:23.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm93.ko 2023-03-03T05:27:23.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95234.ko 2023-03-03T05:27:23.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95241.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/lm95245.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc2945.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4151.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4215.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4222.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4245.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4260.ko 2023-03-03T05:27:23.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/ltc4261.ko 2023-03-03T05:27:23.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max16065.ko 2023-03-03T05:27:23.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max1619.ko 2023-03-03T05:27:23.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max1668.ko 2023-03-03T05:27:23.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max197.ko 2023-03-03T05:27:23.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6639.ko 2023-03-03T05:27:23.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6650.ko 2023-03-03T05:27:23.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/max6697.ko 2023-03-03T05:27:23.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/mcp3021.ko 2023-03-03T05:27:23.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6683.ko 2023-03-03T05:27:23.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6775-core.ko 2023-03-03T05:27:23.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct6775.ko 2023-03-03T05:27:23.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/nct7802.ko 2023-03-03T05:27:23.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pc87360.ko 2023-03-03T05:27:23.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pc87427.ko 2023-03-03T05:27:23.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pcf8591.ko 2023-03-03T05:27:23.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus 2023-03-03T05:27:23.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-03-03T05:27:23.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-03-03T05:27:23.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-03-03T05:27:23.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max16064.ko 2023-03-03T05:27:23.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max34440.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/max8688.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-03-03T05:27:23.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/powr1220.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch5627.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch5636.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sch56xx-common.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sht15.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sht21.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/shtc1.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/sis5595.ko 2023-03-03T05:27:23.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47b397.ko 2023-03-03T05:27:23.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47m1.ko 2023-03-03T05:27:23.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/smsc47m192.ko 2023-03-03T05:27:23.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/thmc50.ko 2023-03-03T05:27:23.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp102.ko 2023-03-03T05:27:23.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp103.ko 2023-03-03T05:27:23.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp401.ko 2023-03-03T05:27:23.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/tmp421.ko 2023-03-03T05:27:23.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/via-cputemp.ko 2023-03-03T05:27:23.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/via686a.ko 2023-03-03T05:27:23.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/vt1211.ko 2023-03-03T05:27:23.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/vt8231.ko 2023-03-03T05:27:23.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83627ehf.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83627hf.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83781d.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83791d.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83792d.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83793.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83795.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83l785ts.ko 2023-03-03T05:27:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/hwmon/w83l786ng.ko 2023-03-03T05:27:23.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c 2023-03-03T05:27:23.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos 2023-03-03T05:27:23.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-03-03T05:27:23.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-03-03T05:27:23.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses 2023-03-03T05:27:23.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-03-03T05:27:23.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-03-03T05:27:23.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-03-03T05:27:23.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-03-03T05:27:23.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-03-03T05:27:23.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-03-03T05:27:23.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-03-03T05:27:23.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-03-03T05:27:23.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-i801.ko 2023-03-03T05:27:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-isch.ko 2023-03-03T05:27:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-03-03T05:27:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-03-03T05:27:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-03-03T05:27:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-parport.ko 2023-03-03T05:27:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-03-03T05:27:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-03-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-03-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-03-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-03-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-03-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-via.ko 2023-03-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-03-03T05:27:23.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-03-03T05:27:23.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-dev.ko 2023-03-03T05:27:23.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-mux.ko 2023-03-03T05:27:23.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-03-03T05:27:23.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-smbus.ko 2023-03-03T05:27:23.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/i2c/i2c-stub.ko 2023-03-03T05:27:23.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband 2023-03-03T05:27:23.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core 2023-03-03T05:27:23.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_cm.ko 2023-03-03T05:27:23.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_core.ko 2023-03-03T05:27:23.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_umad.ko 2023-03-03T05:27:23.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-03-03T05:27:23.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/iw_cm.ko 2023-03-03T05:27:23.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/rdma_cm.ko 2023-03-03T05:27:23.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-03-03T05:27:23.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw 2023-03-03T05:27:23.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/cxgb4 2023-03-03T05:27:23.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-03-03T05:27:23.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx4 2023-03-03T05:27:23.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-03-03T05:27:23.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx5 2023-03-03T05:27:23.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-03-03T05:27:23.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mthca 2023-03-03T05:27:23.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-03-03T05:27:23.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp 2023-03-03T05:27:23.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/ipoib 2023-03-03T05:27:23.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-03-03T05:27:23.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/iser 2023-03-03T05:27:23.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-03-03T05:27:23.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/isert 2023-03-03T05:27:23.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-03-03T05:27:23.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srp 2023-03-03T05:27:23.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-03-03T05:27:23.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srpt 2023-03-03T05:27:23.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-03-03T05:27:23.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input 2023-03-03T05:27:23.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport 2023-03-03T05:27:23.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-03-03T05:27:23.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/fm801-gp.ko 2023-03-03T05:27:23.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/gameport.ko 2023-03-03T05:27:23.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/lightning.ko 2023-03-03T05:27:23.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/gameport/ns558.ko 2023-03-03T05:27:23.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/joydev.ko 2023-03-03T05:27:23.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/keyboard 2023-03-03T05:27:23.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/keyboard/gpio_keys.ko 2023-03-03T05:27:23.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/appletouch.ko 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/bcm5974.ko 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/cyapatp.ko 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/elan_i2c.ko 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/sermouse.ko 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/synaptics_usb.ko 2023-03-03T05:27:23.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/mouse/vsxxxaa.ko 2023-03-03T05:27:23.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/rmi4 2023-03-03T05:27:23.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/rmi4/rmi_core.ko 2023-03-03T05:27:23.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio 2023-03-03T05:27:23.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/altera_ps2.ko 2023-03-03T05:27:23.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/arc_ps2.ko 2023-03-03T05:27:23.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-03-03T05:27:23.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/serio/serio_raw.ko 2023-03-03T05:27:23.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/input/sparse-keymap.ko 2023-03-03T05:27:23.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu 2023-03-03T05:27:23.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu/amd 2023-03-03T05:27:23.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/iommu/amd/iommu_v2.ko 2023-03-03T05:27:23.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds 2023-03-03T05:27:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-blinkm.ko 2023-03-03T05:27:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-lm3530.ko 2023-03-03T05:27:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-lp3944.ko 2023-03-03T05:27:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/leds-ss4200.ko 2023-03-03T05:27:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger 2023-03-03T05:27:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-03-03T05:27:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-03-03T05:27:23.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-03-03T05:27:23.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-03-03T05:27:23.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2023-03-03T05:27:23.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-03-03T05:27:23.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-03-03T05:27:23.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-03-03T05:27:23.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-03-03T05:27:23.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md 2023-03-03T05:27:23.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/bcache 2023-03-03T05:27:23.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/bcache/bcache.ko 2023-03-03T05:27:23.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-bio-prison.ko 2023-03-03T05:27:23.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-cache-smq.ko 2023-03-03T05:27:23.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-cache.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-crypt.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-delay.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-flakey.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-log-userspace.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-multipath.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-queue-length.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-raid.ko 2023-03-03T05:27:23.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-round-robin.ko 2023-03-03T05:27:23.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-service-time.ko 2023-03-03T05:27:23.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-switch.ko 2023-03-03T05:27:23.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-thin-pool.ko 2023-03-03T05:27:23.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/dm-verity.ko 2023-03-03T05:27:23.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/faulty.ko 2023-03-03T05:27:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/linear.ko 2023-03-03T05:27:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/multipath.ko 2023-03-03T05:27:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/persistent-data 2023-03-03T05:27:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-03-03T05:27:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid0.ko 2023-03-03T05:27:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid1.ko 2023-03-03T05:27:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid10.ko 2023-03-03T05:27:23.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/md/raid456.ko 2023-03-03T05:27:23.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message 2023-03-03T05:27:23.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion 2023-03-03T05:27:23.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptbase.ko 2023-03-03T05:27:23.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptctl.ko 2023-03-03T05:27:23.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptfc.ko 2023-03-03T05:27:23.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptlan.ko 2023-03-03T05:27:23.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptsas.ko 2023-03-03T05:27:23.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptscsih.ko 2023-03-03T05:27:23.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/message/fusion/mptspi.ko 2023-03-03T05:27:23.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd 2023-03-03T05:27:23.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/lpc_ich.ko 2023-03-03T05:27:23.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/lpc_sch.ko 2023-03-03T05:27:23.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/mfd-core.ko 2023-03-03T05:27:23.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/sm501.ko 2023-03-03T05:27:23.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/viperboard.ko 2023-03-03T05:27:23.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/vx855.ko 2023-03-03T05:27:23.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/mfd/wl1273-core.ko 2023-03-03T05:27:23.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc 2023-03-03T05:27:23.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/altera-stapl 2023-03-03T05:27:23.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-03-03T05:27:23.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/apds9802als.ko 2023-03-03T05:27:23.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/apds990x.ko 2023-03-03T05:27:23.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/bh1770glc.ko 2023-03-03T05:27:23.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/cb710 2023-03-03T05:27:23.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/cb710/cb710.ko 2023-03-03T05:27:23.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom 2023-03-03T05:27:23.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/at24.ko 2023-03-03T05:27:23.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/eeprom.ko 2023-03-03T05:27:23.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-03-03T05:27:23.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/eeprom/max6875.ko 2023-03-03T05:27:23.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/enclosure.ko 2023-03-03T05:27:23.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/hpilo.ko 2023-03-03T05:27:23.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/isl29003.ko 2023-03-03T05:27:23.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/isl29020.ko 2023-03-03T05:27:23.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d 2023-03-03T05:27:23.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-03-03T05:27:23.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-03-03T05:27:23.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei 2023-03-03T05:27:23.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei-me.ko 2023-03-03T05:27:23.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei-txe.ko 2023-03-03T05:27:23.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/mei/mei.ko 2023-03-03T05:27:23.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-gru 2023-03-03T05:27:23.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-gru/gru.ko 2023-03-03T05:27:23.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp 2023-03-03T05:27:23.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xp.ko 2023-03-03T05:27:23.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xpc.ko 2023-03-03T05:27:23.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-03-03T05:27:23.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tifm_7xx1.ko 2023-03-03T05:27:23.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tifm_core.ko 2023-03-03T05:27:23.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/tsl2550.ko 2023-03-03T05:27:23.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_balloon.ko 2023-03-03T05:27:23.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_vmci 2023-03-03T05:27:23.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-03-03T05:27:23.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net 2023-03-03T05:27:23.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/bonding 2023-03-03T05:27:23.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/bonding/bonding.ko 2023-03-03T05:27:23.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa 2023-03-03T05:27:23.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/b53 2023-03-03T05:27:23.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/b53/b53_common.ko 2023-03-03T05:27:23.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/bcm-sf2.ko 2023-03-03T05:27:23.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6060.ko 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6xxx 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/dummy.ko 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/eql.ko 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-03-03T05:27:23.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-03-03T05:27:23.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-03-03T05:27:23.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-03-03T05:27:23.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390 2023-03-03T05:27:23.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/8390.ko 2023-03-03T05:27:23.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-03-03T05:27:23.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-03-03T05:27:23.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-03-03T05:27:23.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/adaptec 2023-03-03T05:27:23.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-03-03T05:27:23.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/agere 2023-03-03T05:27:23.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/agere/et131x.ko 2023-03-03T05:27:23.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/alteon 2023-03-03T05:27:23.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-03-03T05:27:23.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/altera 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/alx 2023-03-03T05:27:23.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-03-03T05:27:23.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1c 2023-03-03T05:27:23.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-03-03T05:27:23.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1e 2023-03-03T05:27:23.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-03-03T05:27:23.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx 2023-03-03T05:27:23.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-03-03T05:27:23.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/genet 2023-03-03T05:27:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-03-03T05:27:23.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-03-03T05:27:23.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade 2023-03-03T05:27:23.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade/bna 2023-03-03T05:27:23.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-03-03T05:27:23.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio 2023-03-03T05:27:23.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb 2023-03-03T05:27:23.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-03-03T05:27:23.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-03-03T05:27:23.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-03-03T05:27:23.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-03-03T05:27:23.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-03-03T05:27:23.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-03-03T05:27:23.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-03-03T05:27:23.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-03-03T05:27:23.913 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-03-03T05:27:23.913 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco 2023-03-03T05:27:23.913 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco/enic 2023-03-03T05:27:23.913 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-03-03T05:27:23.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec 2023-03-03T05:27:23.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip 2023-03-03T05:27:23.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-03-03T05:27:23.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-03-03T05:27:23.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-03-03T05:27:23.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-03-03T05:27:23.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-03-03T05:27:23.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-03-03T05:27:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink 2023-03-03T05:27:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-03-03T05:27:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-03-03T05:27:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/dnet.ko 2023-03-03T05:27:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex 2023-03-03T05:27:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex/benet 2023-03-03T05:27:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-03-03T05:27:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ethoc.ko 2023-03-03T05:27:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel 2023-03-03T05:27:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e100.ko 2023-03-03T05:27:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000 2023-03-03T05:27:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-03-03T05:27:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000e 2023-03-03T05:27:23.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-03-03T05:27:23.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/fm10k 2023-03-03T05:27:23.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-03-03T05:27:23.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/i40e 2023-03-03T05:27:23.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-03-03T05:27:23.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/iavf 2023-03-03T05:27:23.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igb 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igbvf 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgb 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbe 2023-03-03T05:27:23.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-03-03T05:27:23.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbevf 2023-03-03T05:27:23.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-03-03T05:27:23.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/jme.ko 2023-03-03T05:27:23.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell 2023-03-03T05:27:23.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-03-03T05:27:23.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/skge.ko 2023-03-03T05:27:23.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-03-03T05:27:23.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox 2023-03-03T05:27:23.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4 2023-03-03T05:27:23.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-03-03T05:27:23.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-03-03T05:27:23.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5 2023-03-03T05:27:23.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-03-03T05:27:23.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-03-03T05:27:23.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/micrel 2023-03-03T05:27:23.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-03-03T05:27:23.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom 2023-03-03T05:27:23.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom/myri10ge 2023-03-03T05:27:23.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-03-03T05:27:23.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi 2023-03-03T05:27:23.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-03-03T05:27:23.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-03-03T05:27:23.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/neterion 2023-03-03T05:27:23.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-03-03T05:27:23.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/nvidia 2023-03-03T05:27:23.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-03-03T05:27:23.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines 2023-03-03T05:27:23.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-03-03T05:27:23.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-03-03T05:27:23.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic 2023-03-03T05:27:23.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/netxen 2023-03-03T05:27:23.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rdc 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-03-03T05:27:23.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-03-03T05:27:23.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/atp.ko 2023-03-03T05:27:23.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-03-03T05:27:23.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rocker 2023-03-03T05:27:23.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-03-03T05:27:23.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sfc 2023-03-03T05:27:23.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/silan 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis/sis190.ko 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sis/sis900.ko 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-03-03T05:27:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-03-03T05:27:23.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-03-03T05:27:23.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-03-03T05:27:23.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro 2023-03-03T05:27:23.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac 2023-03-03T05:27:23.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-03-03T05:27:23.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-03-03T05:27:23.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun 2023-03-03T05:27:23.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/cassini.ko 2023-03-03T05:27:23.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/niu.ko 2023-03-03T05:27:23.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/sungem.ko 2023-03-03T05:27:23.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-03-03T05:27:23.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/tehuti 2023-03-03T05:27:23.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-03-03T05:27:23.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ti 2023-03-03T05:27:23.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/ti/tlan.ko 2023-03-03T05:27:23.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via 2023-03-03T05:27:23.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-03-03T05:27:23.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-03-03T05:27:23.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/xircom 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/geneve.ko 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/hyperv 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-03-03T05:27:23.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ieee802154 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ieee802154/fakelb.ko 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ifb.ko 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ipvlan 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ipvlan/ipvlan.ko 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/macvlan.ko 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/macvtap.ko 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio 2023-03-03T05:27:23.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio.ko 2023-03-03T05:27:23.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-03-03T05:27:23.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-03-03T05:27:23.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/mii.ko 2023-03-03T05:27:23.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/net_failover.ko 2023-03-03T05:27:23.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/netconsole.ko 2023-03-03T05:27:23.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/nlmon.ko 2023-03-03T05:27:23.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs 2023-03-03T05:27:23.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs/pcs-altera-tse.ko 2023-03-03T05:27:23.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-03-03T05:27:23.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy 2023-03-03T05:27:23.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/amd.ko 2023-03-03T05:27:23.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-03-03T05:27:23.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-03-03T05:27:23.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm7xxx.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/bcm87xx.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/broadcom.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/cicada.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/davicom.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/dp83640.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/et1011c.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/icplus.ko 2023-03-03T05:27:23.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/lxt.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/marvell.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/micrel.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/national.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/phylink.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/qsemi.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/realtek.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/smsc.ko 2023-03-03T05:27:23.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/ste10Xp.ko 2023-03-03T05:27:23.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/phy/vitesse.ko 2023-03-03T05:27:23.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp 2023-03-03T05:27:23.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/bsd_comp.ko 2023-03-03T05:27:23.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_async.ko 2023-03-03T05:27:23.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_deflate.ko 2023-03-03T05:27:23.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_generic.ko 2023-03-03T05:27:23.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_mppe.ko 2023-03-03T05:27:23.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/ppp_synctty.ko 2023-03-03T05:27:23.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pppoe.ko 2023-03-03T05:27:23.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pppox.ko 2023-03-03T05:27:23.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/ppp/pptp.ko 2023-03-03T05:27:23.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip 2023-03-03T05:27:23.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip/slhc.ko 2023-03-03T05:27:23.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/slip/slip.ko 2023-03-03T05:27:23.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/sungem_phy.ko 2023-03-03T05:27:23.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/tap.ko 2023-03-03T05:27:23.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team 2023-03-03T05:27:23.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team.ko 2023-03-03T05:27:23.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_activebackup.ko 2023-03-03T05:27:23.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_broadcast.ko 2023-03-03T05:27:23.940 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-03-03T05:27:23.940 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_random.ko 2023-03-03T05:27:23.940 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-03-03T05:27:23.940 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/tun.ko 2023-03-03T05:27:23.940 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/veth.ko 2023-03-03T05:27:23.940 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/virtio_net.ko 2023-03-03T05:27:23.940 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vmxnet3 2023-03-03T05:27:23.941 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-03-03T05:27:23.941 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vxlan 2023-03-03T05:27:23.941 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/vxlan/vxlan.ko 2023-03-03T05:27:23.941 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netback 2023-03-03T05:27:23.941 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netback/xen-netback.ko 2023-03-03T05:27:23.941 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/net/xen-netfront.ko 2023-03-03T05:27:23.941 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ntb 2023-03-03T05:27:23.942 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ntb/ntb.ko 2023-03-03T05:27:23.942 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme 2023-03-03T05:27:23.942 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host 2023-03-03T05:27:23.942 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host/nvme-core.ko 2023-03-03T05:27:23.942 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/nvme/host/nvme.ko 2023-03-03T05:27:23.942 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport.ko 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_cs.ko 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_pc.ko 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/parport/parport_serial.ko 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/hotplug 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-03-03T05:27:23.943 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/pcie 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/pcie/aer_inject.ko 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pci/xen-pcifront.ko 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/i82092.ko 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/pd6729.ko 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pcmcia/yenta_socket.ko 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl 2023-03-03T05:27:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl/intel 2023-03-03T05:27:23.945 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-03-03T05:27:23.945 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform 2023-03-03T05:27:23.945 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86 2023-03-03T05:27:23.945 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/acer-wmi.ko 2023-03-03T05:27:23.945 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/acerhdf.ko 2023-03-03T05:27:23.945 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/apple-gmux.ko 2023-03-03T05:27:23.945 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-laptop.ko 2023-03-03T05:27:23.946 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-03-03T05:27:23.946 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/asus-wmi.ko 2023-03-03T05:27:23.946 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/classmate-laptop.ko 2023-03-03T05:27:23.946 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-03-03T05:27:23.946 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-03-03T05:27:23.946 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/hdaps.ko 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/intel_ips.ko 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/msi-wmi.ko 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/mxm-wmi.ko 2023-03-03T05:27:23.947 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-03-03T05:27:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/samsung-laptop.ko 2023-03-03T05:27:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/samsung-q10.ko 2023-03-03T05:27:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/topstar-laptop.ko 2023-03-03T05:27:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-03-03T05:27:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/toshiba_haps.ko 2023-03-03T05:27:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/wmi-bmof.ko 2023-03-03T05:27:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/platform/x86/wmi.ko 2023-03-03T05:27:23.949 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap 2023-03-03T05:27:23.949 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap/intel_rapl_common.ko 2023-03-03T05:27:23.949 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/powercap/intel_rapl_msr.ko 2023-03-03T05:27:23.949 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps 2023-03-03T05:27:23.949 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients 2023-03-03T05:27:23.949 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps-gpio.ko 2023-03-03T05:27:23.950 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps-ldisc.ko 2023-03-03T05:27:23.950 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/clients/pps_parport.ko 2023-03-03T05:27:23.950 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/pps/pps_core.ko 2023-03-03T05:27:23.950 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp 2023-03-03T05:27:23.950 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp/ptp.ko 2023-03-03T05:27:23.950 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ptp/ptp_kvm.ko 2023-03-03T05:27:23.950 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc 2023-03-03T05:27:23.951 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-bq32k.ko 2023-03-03T05:27:23.951 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-bq4802.ko 2023-03-03T05:27:23.951 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1286.ko 2023-03-03T05:27:23.951 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1307.ko 2023-03-03T05:27:23.951 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1374.ko 2023-03-03T05:27:23.952 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1511.ko 2023-03-03T05:27:23.952 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1553.ko 2023-03-03T05:27:23.952 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1672.ko 2023-03-03T05:27:23.952 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds1742.ko 2023-03-03T05:27:23.952 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds2404.ko 2023-03-03T05:27:23.953 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-ds3232.ko 2023-03-03T05:27:23.953 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-em3027.ko 2023-03-03T05:27:23.953 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-fm3130.ko 2023-03-03T05:27:23.953 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-isl12022.ko 2023-03-03T05:27:23.954 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-isl1208.ko 2023-03-03T05:27:23.954 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m41t80.ko 2023-03-03T05:27:23.954 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m48t35.ko 2023-03-03T05:27:23.954 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-m48t59.ko 2023-03-03T05:27:23.955 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-max6900.ko 2023-03-03T05:27:23.955 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-msm6242.ko 2023-03-03T05:27:23.955 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf2127.ko 2023-03-03T05:27:23.955 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf85063.ko 2023-03-03T05:27:23.956 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8523.ko 2023-03-03T05:27:23.956 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8563.ko 2023-03-03T05:27:23.956 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-pcf8583.ko 2023-03-03T05:27:23.956 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rp5c01.ko 2023-03-03T05:27:23.957 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rs5c372.ko 2023-03-03T05:27:23.957 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-03-03T05:27:23.957 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rx8025.ko 2023-03-03T05:27:23.957 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-rx8581.ko 2023-03-03T05:27:23.958 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-03-03T05:27:23.958 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-v3020.ko 2023-03-03T05:27:23.958 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/rtc/rtc-x1205.ko 2023-03-03T05:27:23.958 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi 2023-03-03T05:27:23.958 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-9xxx.ko 2023-03-03T05:27:23.959 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-sas.ko 2023-03-03T05:27:23.959 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/3w-xxxx.ko 2023-03-03T05:27:23.959 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/BusLogic.ko 2023-03-03T05:27:23.960 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/a100u2w.ko 2023-03-03T05:27:23.960 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aacraid 2023-03-03T05:27:23.960 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aacraid/aacraid.ko 2023-03-03T05:27:23.960 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/advansys.ko 2023-03-03T05:27:23.961 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx 2023-03-03T05:27:23.961 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-03-03T05:27:23.961 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-03-03T05:27:23.961 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/am53c974.ko 2023-03-03T05:27:23.961 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/arcmsr 2023-03-03T05:27:23.961 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-03-03T05:27:23.961 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/atp870u.ko 2023-03-03T05:27:23.962 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/be2iscsi 2023-03-03T05:27:23.962 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-03-03T05:27:23.962 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bfa 2023-03-03T05:27:23.962 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bfa/bfa.ko 2023-03-03T05:27:23.962 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2fc 2023-03-03T05:27:23.962 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-03-03T05:27:23.962 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2i 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ch.ko 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/csiostor 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/csiostor/csiostor.ko 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb3i 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-03-03T05:27:23.963 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb4i 2023-03-03T05:27:23.964 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-03-03T05:27:23.964 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-03-03T05:27:23.964 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/dc395x.ko 2023-03-03T05:27:23.964 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler 2023-03-03T05:27:23.964 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-03-03T05:27:23.965 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-03-03T05:27:23.965 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-03-03T05:27:23.965 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-03-03T05:27:23.965 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/dmx3191d.ko 2023-03-03T05:27:23.965 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esas2r 2023-03-03T05:27:23.965 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esas2r/esas2r.ko 2023-03-03T05:27:23.965 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/esp_scsi.ko 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe/fcoe.ko 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fnic 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/fnic/fnic.ko 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hpsa.ko 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hptiop.ko 2023-03-03T05:27:23.966 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/hv_storvsc.ko 2023-03-03T05:27:23.967 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/initio.ko 2023-03-03T05:27:23.967 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ipr.ko 2023-03-03T05:27:23.967 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ips.ko 2023-03-03T05:27:23.967 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/isci 2023-03-03T05:27:23.967 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/isci/isci.ko 2023-03-03T05:27:23.967 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-03-03T05:27:23.967 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/iscsi_tcp.ko 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libfc 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libfc/libfc.ko 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libiscsi.ko 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libiscsi_tcp.ko 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libsas 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/libsas/libsas.ko 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/lpfc 2023-03-03T05:27:23.968 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/lpfc/lpfc.ko 2023-03-03T05:27:23.969 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid 2023-03-03T05:27:23.969 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid.ko 2023-03-03T05:27:23.969 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-03-03T05:27:23.969 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-03-03T05:27:23.969 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-03-03T05:27:23.969 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mpt3sas 2023-03-03T05:27:23.969 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvsas 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvsas/mvsas.ko 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/mvumi.ko 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pm8001 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/pmcraid.ko 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla1280.ko 2023-03-03T05:27:23.970 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx 2023-03-03T05:27:23.971 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-03-03T05:27:23.971 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-03-03T05:27:23.971 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla4xxx 2023-03-03T05:27:23.971 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-03-03T05:27:23.971 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/raid_class.ko 2023-03-03T05:27:23.971 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_debug.ko 2023-03-03T05:27:23.971 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_fc.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_sas.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_spi.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/scsi_transport_srp.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/ses.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/st.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/stex.ko 2023-03-03T05:27:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/sym53c8xx_2 2023-03-03T05:27:23.973 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-03-03T05:27:23.973 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/virtio_scsi.ko 2023-03-03T05:27:23.973 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/vmw_pvscsi.ko 2023-03-03T05:27:23.973 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/scsi/wd719x.ko 2023-03-03T05:27:23.973 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ssb 2023-03-03T05:27:23.973 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ssb/ssb.ko 2023-03-03T05:27:23.973 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target 2023-03-03T05:27:23.974 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/iscsi 2023-03-03T05:27:23.974 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-03-03T05:27:23.974 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/loopback 2023-03-03T05:27:23.974 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/loopback/tcm_loop.ko 2023-03-03T05:27:23.974 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_file.ko 2023-03-03T05:27:23.974 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_iblock.ko 2023-03-03T05:27:23.974 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_mod.ko 2023-03-03T05:27:23.975 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_pscsi.ko 2023-03-03T05:27:23.975 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/target_core_user.ko 2023-03-03T05:27:23.975 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/tcm_fc 2023-03-03T05:27:23.975 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-03-03T05:27:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal 2023-03-03T05:27:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel 2023-03-03T05:27:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal 2023-03-03T05:27:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-03-03T05:27:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-03-03T05:27:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-03-03T05:27:23.976 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-03-03T05:27:23.977 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-03-03T05:27:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-03-03T05:27:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-03-03T05:27:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-03-03T05:27:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty 2023-03-03T05:27:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/ipwireless 2023-03-03T05:27:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-03-03T05:27:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/n_gsm.ko 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/n_hdlc.ko 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/nozomi.ko 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/8250 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/arc_uart.ko 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/jsm 2023-03-03T05:27:23.979 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/serial/jsm/jsm.ko 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/tty/synclink_gt.ko 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/core 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/core/ufshcd-core.ko 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/host 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio 2023-03-03T05:27:23.980 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio.ko 2023-03-03T05:27:23.981 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_aec.ko 2023-03-03T05:27:23.981 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_cif.ko 2023-03-03T05:27:23.981 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_pci_generic.ko 2023-03-03T05:27:23.981 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/uio/uio_sercos3.ko 2023-03-03T05:27:23.981 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb 2023-03-03T05:27:23.981 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class 2023-03-03T05:27:23.981 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/cdc-acm.ko 2023-03-03T05:27:23.982 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/cdc-wdm.ko 2023-03-03T05:27:23.982 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/usblp.ko 2023-03-03T05:27:23.982 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/class/usbtmc.ko 2023-03-03T05:27:23.982 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host 2023-03-03T05:27:23.982 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host/sl811-hcd.ko 2023-03-03T05:27:23.982 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/host/u132-hcd.ko 2023-03-03T05:27:23.982 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image/mdc800.ko 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/image/microtek.ko 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/adutux.ko 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/appledisplay.ko 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/emi26.ko 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/emi62.ko 2023-03-03T05:27:23.983 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ezusb.ko 2023-03-03T05:27:23.984 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ftdi-elan.ko 2023-03-03T05:27:23.984 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/idmouse.ko 2023-03-03T05:27:23.984 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/iowarrior.ko 2023-03-03T05:27:23.984 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/isight_firmware.ko 2023-03-03T05:27:23.984 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/ldusb.ko 2023-03-03T05:27:23.984 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/legousbtower.ko 2023-03-03T05:27:23.984 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/sisusbvga 2023-03-03T05:27:23.985 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-03-03T05:27:23.985 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/trancevibrator.ko 2023-03-03T05:27:23.985 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usb3503.ko 2023-03-03T05:27:23.985 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usblcd.ko 2023-03-03T05:27:23.985 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/usbsevseg.ko 2023-03-03T05:27:23.985 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/uss720.ko 2023-03-03T05:27:23.985 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/misc/yurex.ko 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/phy 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/phy/phy-generic.ko 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/aircable.ko 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ark3116.ko 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/belkin_sa.ko 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ch341.ko 2023-03-03T05:27:23.986 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cp210x.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cyberjack.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/cypress_m8.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/digi_acceleport.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/empeg.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ftdi_sio.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/garmin_gps.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/io_edgeport.ko 2023-03-03T05:27:23.987 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/io_ti.ko 2023-03-03T05:27:23.988 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ipaq.ko 2023-03-03T05:27:23.988 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ipw.ko 2023-03-03T05:27:23.988 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ir-usb.ko 2023-03-03T05:27:23.988 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-03-03T05:27:23.988 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/keyspan.ko 2023-03-03T05:27:23.988 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/keyspan_pda.ko 2023-03-03T05:27:23.988 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/kl5kusb105.ko 2023-03-03T05:27:23.989 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/kobil_sct.ko 2023-03-03T05:27:23.989 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mct_u232.ko 2023-03-03T05:27:23.989 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mos7720.ko 2023-03-03T05:27:23.989 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/mos7840.ko 2023-03-03T05:27:23.989 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/navman.ko 2023-03-03T05:27:23.989 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/omninet.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/opticon.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/option.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/oti6858.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/pl2303.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/qcaux.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/qcserial.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/quatech2.ko 2023-03-03T05:27:23.990 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/safe_serial.ko 2023-03-03T05:27:23.991 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/sierra.ko 2023-03-03T05:27:23.991 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/spcp8x5.ko 2023-03-03T05:27:23.991 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ssu100.ko 2023-03-03T05:27:23.991 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/symbolserial.ko 2023-03-03T05:27:23.991 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-03-03T05:27:23.991 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-03-03T05:27:23.991 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb_debug.ko 2023-03-03T05:27:23.992 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/usb_wwan.ko 2023-03-03T05:27:23.992 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/visor.ko 2023-03-03T05:27:23.992 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/whiteheat.ko 2023-03-03T05:27:23.992 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/serial/xsens_mt.ko 2023-03-03T05:27:23.992 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage 2023-03-03T05:27:23.992 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/uas.ko 2023-03-03T05:27:23.992 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-alauda.ko 2023-03-03T05:27:23.993 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-cypress.ko 2023-03-03T05:27:23.993 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-datafab.ko 2023-03-03T05:27:23.993 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-03-03T05:27:23.993 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-freecom.ko 2023-03-03T05:27:23.993 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-isd200.ko 2023-03-03T05:27:23.993 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-03-03T05:27:23.993 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-karma.ko 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-onetouch.ko 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-realtek.ko 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-sddr09.ko 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-sddr55.ko 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/ums-usbat.ko 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/storage/usb-storage.ko 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip 2023-03-03T05:27:23.994 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/usbip-core.ko 2023-03-03T05:27:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/usbip-host.ko 2023-03-03T05:27:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-03-03T05:27:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio 2023-03-03T05:27:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci 2023-03-03T05:27:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-03-03T05:27:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/pci/vfio-pci.ko 2023-03-03T05:27:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/vfio.ko 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost.ko 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_iotlb.ko 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_net.ko 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/vhost/vhost_scsi.ko 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video 2023-03-03T05:27:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight 2023-03-03T05:27:23.997 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/apple_bl.ko 2023-03-03T05:27:23.997 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/lcd.ko 2023-03-03T05:27:23.997 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/video/backlight/platform_lcd.ko 2023-03-03T05:27:23.997 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio 2023-03-03T05:27:23.997 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio.ko 2023-03-03T05:27:23.997 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_balloon.ko 2023-03-03T05:27:23.997 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_mmio.ko 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci.ko 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/virtio/virtio_ring.ko 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds1wm.ko 2023-03-03T05:27:23.998 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds2482.ko 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/masters/ds2490.ko 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-03-03T05:27:23.999 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-03-03T05:27:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-03-03T05:27:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-03-03T05:27:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-03-03T05:27:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_smem.ko 2023-03-03T05:27:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/slaves/w1_therm.ko 2023-03-03T05:27:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/w1/wire.ko 2023-03-03T05:27:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog 2023-03-03T05:27:24.001 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/alim1535_wdt.ko 2023-03-03T05:27:24.001 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/alim7101_wdt.ko 2023-03-03T05:27:24.001 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/f71808e_wdt.ko 2023-03-03T05:27:24.001 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/hpwdt.ko 2023-03-03T05:27:24.001 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/i6300esb.ko 2023-03-03T05:27:24.001 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-03-03T05:27:24.001 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/iTCO_wdt.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ib700wdt.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ibmasr.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/it8712f_wdt.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/it87_wdt.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/machzwd.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/nv_tco.ko 2023-03-03T05:27:24.002 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/pcwd_pci.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/pcwd_usb.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sch311x_wdt.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/softdog.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/sp5100_tco.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/via_wdt.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-03-03T05:27:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83877f_wdt.ko 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/w83977f_wdt.ko 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/wdt_pci.ko 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/watchdog/xen_wdt.ko 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-acpi-processor.ko 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-evtchn.ko 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-gntalloc.ko 2023-03-03T05:27:24.004 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-gntdev.ko 2023-03-03T05:27:24.005 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-pciback 2023-03-03T05:27:24.005 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-03-03T05:27:24.005 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xen-privcmd.ko 2023-03-03T05:27:24.005 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xenfs 2023-03-03T05:27:24.005 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/drivers/xen/xenfs/xenfs.ko 2023-03-03T05:27:24.005 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs 2023-03-03T05:27:24.005 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/9p 2023-03-03T05:27:24.006 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/9p/9p.ko 2023-03-03T05:27:24.006 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/affs 2023-03-03T05:27:24.006 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/affs/affs.ko 2023-03-03T05:27:24.006 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/befs 2023-03-03T05:27:24.006 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/befs/befs.ko 2023-03-03T05:27:24.006 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/binfmt_misc.ko 2023-03-03T05:27:24.006 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/btrfs 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/btrfs/btrfs.ko 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cachefiles 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cachefiles/cachefiles.ko 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ceph 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ceph/ceph.ko 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cifs 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cifs/cifs.ko 2023-03-03T05:27:24.007 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/coda 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/coda/coda.ko 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cramfs 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/cramfs/cramfs.ko 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/dlm 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/dlm/dlm.ko 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ecryptfs 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ecryptfs/ecryptfs.ko 2023-03-03T05:27:24.008 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/f2fs 2023-03-03T05:27:24.009 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/f2fs/f2fs.ko 2023-03-03T05:27:24.009 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat 2023-03-03T05:27:24.009 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/fat.ko 2023-03-03T05:27:24.009 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/msdos.ko 2023-03-03T05:27:24.009 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fat/vfat.ko 2023-03-03T05:27:24.009 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fscache 2023-03-03T05:27:24.009 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fscache/fscache.ko 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse/cuse.ko 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/fuse/fuse.ko 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/gfs2 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/gfs2/gfs2.ko 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfs 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfs/hfs.ko 2023-03-03T05:27:24.010 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfsplus 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/hfsplus/hfsplus.ko 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/isofs 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/isofs/isofs.ko 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/jfs 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/jfs/jfs.ko 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/lockd 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/lockd/lockd.ko 2023-03-03T05:27:24.011 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/minix 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/minix/minix.ko 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/netfs 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/netfs/netfs.ko 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/blocklayout 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/filelayout 2023-03-03T05:27:24.012 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-03-03T05:27:24.013 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/flexfilelayout 2023-03-03T05:27:24.013 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-03-03T05:27:24.013 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfs.ko 2023-03-03T05:27:24.013 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfsv3.ko 2023-03-03T05:27:24.013 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs/nfsv4.ko 2023-03-03T05:27:24.013 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common 2023-03-03T05:27:24.013 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common/grace.ko 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfs_common/nfs_acl.ko 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfsd 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nfsd/nfsd.ko 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nilfs2 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nilfs2/nilfs2.ko 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-celtic.ko 2023-03-03T05:27:24.014 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-centeuro.ko 2023-03-03T05:27:24.015 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-croatian.ko 2023-03-03T05:27:24.015 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-cyrillic.ko 2023-03-03T05:27:24.015 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-gaelic.ko 2023-03-03T05:27:24.015 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-greek.ko 2023-03-03T05:27:24.015 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-iceland.ko 2023-03-03T05:27:24.015 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-inuit.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-roman.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-romanian.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/mac-turkish.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1250.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1251.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp1255.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp737.ko 2023-03-03T05:27:24.016 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp775.ko 2023-03-03T05:27:24.017 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp850.ko 2023-03-03T05:27:24.017 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp852.ko 2023-03-03T05:27:24.017 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp855.ko 2023-03-03T05:27:24.017 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp857.ko 2023-03-03T05:27:24.017 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp860.ko 2023-03-03T05:27:24.017 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp861.ko 2023-03-03T05:27:24.017 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp862.ko 2023-03-03T05:27:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp863.ko 2023-03-03T05:27:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp864.ko 2023-03-03T05:27:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp865.ko 2023-03-03T05:27:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp866.ko 2023-03-03T05:27:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp869.ko 2023-03-03T05:27:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp874.ko 2023-03-03T05:27:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp932.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp936.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp949.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_cp950.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_euc-jp.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-1.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-13.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-14.ko 2023-03-03T05:27:24.019 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-15.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-2.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-3.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-4.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-5.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-6.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-7.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_iso8859-9.ko 2023-03-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-r.ko 2023-03-03T05:27:24.021 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-ru.ko 2023-03-03T05:27:24.021 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_koi8-u.ko 2023-03-03T05:27:24.021 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/nls/nls_utf8.ko 2023-03-03T05:27:24.021 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2 2023-03-03T05:27:24.021 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/cluster 2023-03-03T05:27:24.021 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-03-03T05:27:24.021 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlm 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlmfs 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2.ko 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-03-03T05:27:24.022 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/overlayfs 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/overlayfs/overlay.ko 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/pstore 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/pstore/ramoops.ko 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/reiserfs 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/reiserfs/reiserfs.ko 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/romfs 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/romfs/romfs.ko 2023-03-03T05:27:24.023 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common 2023-03-03T05:27:24.024 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common/cifs_arc4.ko 2023-03-03T05:27:24.024 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/smbfs_common/cifs_md4.ko 2023-03-03T05:27:24.024 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/squashfs 2023-03-03T05:27:24.024 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/squashfs/squashfs.ko 2023-03-03T05:27:24.024 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/sysv 2023-03-03T05:27:24.024 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/sysv/sysv.ko 2023-03-03T05:27:24.024 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/udf 2023-03-03T05:27:24.025 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/udf/udf.ko 2023-03-03T05:27:24.025 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ufs 2023-03-03T05:27:24.025 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/ufs/ufs.ko 2023-03-03T05:27:24.025 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/xfs 2023-03-03T05:27:24.025 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/fs/xfs/xfs.ko 2023-03-03T05:27:24.025 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel 2023-03-03T05:27:24.025 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/rcu 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/rcu/rcutorture.ko 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/torture.ko 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/trace 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/asn1_encoder.ko 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crc-itu-t.ko 2023-03-03T05:27:24.026 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crc8.ko 2023-03-03T05:27:24.027 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto 2023-03-03T05:27:24.027 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto/libarc4.ko 2023-03-03T05:27:24.027 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/crypto/libdes.ko 2023-03-03T05:27:24.027 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/libcrc32c.ko 2023-03-03T05:27:24.027 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lru_cache.ko 2023-03-03T05:27:24.027 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4 2023-03-03T05:27:24.028 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4/lz4_compress.ko 2023-03-03T05:27:24.028 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/lz4/lz4hc_compress.ko 2023-03-03T05:27:24.028 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/math 2023-03-03T05:27:24.028 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/math/cordic.ko 2023-03-03T05:27:24.028 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/raid6 2023-03-03T05:27:24.028 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/raid6/raid6_pq.ko 2023-03-03T05:27:24.028 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/reed_solomon 2023-03-03T05:27:24.029 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/reed_solomon/reed_solomon.ko 2023-03-03T05:27:24.029 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_bm.ko 2023-03-03T05:27:24.029 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_fsm.ko 2023-03-03T05:27:24.029 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/ts_kmp.ko 2023-03-03T05:27:24.029 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/zstd 2023-03-03T05:27:24.029 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/lib/zstd/zstd_compress.ko 2023-03-03T05:27:24.030 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/mm 2023-03-03T05:27:24.030 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/mm/hwpoison-inject.ko 2023-03-03T05:27:24.030 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net 2023-03-03T05:27:24.030 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan 2023-03-03T05:27:24.030 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/6lowpan.ko 2023-03-03T05:27:24.030 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_dest.ko 2023-03-03T05:27:24.030 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_fragment.ko 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_hop.ko 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_ipv6.ko 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_mobility.ko 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_routing.ko 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/6lowpan/nhc_udp.ko 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/garp.ko 2023-03-03T05:27:24.031 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/mrp.ko 2023-03-03T05:27:24.032 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/p8022.ko 2023-03-03T05:27:24.032 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/psnap.ko 2023-03-03T05:27:24.032 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/802/stp.ko 2023-03-03T05:27:24.032 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/8021q 2023-03-03T05:27:24.032 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/8021q/8021q.ko 2023-03-03T05:27:24.032 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p 2023-03-03T05:27:24.032 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet.ko 2023-03-03T05:27:24.033 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_fd.ko 2023-03-03T05:27:24.033 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_rdma.ko 2023-03-03T05:27:24.033 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/9p/9pnet_virtio.ko 2023-03-03T05:27:24.033 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/batman-adv 2023-03-03T05:27:24.033 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/batman-adv/batman-adv.ko 2023-03-03T05:27:24.033 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge 2023-03-03T05:27:24.033 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/br_netfilter.ko 2023-03-03T05:27:24.034 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/bridge.ko 2023-03-03T05:27:24.034 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter 2023-03-03T05:27:24.034 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-03-03T05:27:24.034 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_among.ko 2023-03-03T05:27:24.034 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_arp.ko 2023-03-03T05:27:24.034 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-03-03T05:27:24.034 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_ip.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_limit.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_log.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_mark.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-03-03T05:27:24.035 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_snat.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_stp.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-03-03T05:27:24.036 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/ebtables.ko 2023-03-03T05:27:24.037 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-03-03T05:27:24.037 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-03-03T05:27:24.037 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-03-03T05:27:24.037 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ceph 2023-03-03T05:27:24.038 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ceph/libceph.ko 2023-03-03T05:27:24.038 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core 2023-03-03T05:27:24.038 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core/failover.ko 2023-03-03T05:27:24.038 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/core/pktgen.ko 2023-03-03T05:27:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp 2023-03-03T05:27:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp.ko 2023-03-03T05:27:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_diag.ko 2023-03-03T05:27:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_ipv4.ko 2023-03-03T05:27:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dccp/dccp_ipv6.ko 2023-03-03T05:27:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dns_resolver 2023-03-03T05:27:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dns_resolver/dns_resolver.ko 2023-03-03T05:27:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa 2023-03-03T05:27:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/dsa_core.ko 2023-03-03T05:27:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_brcm.ko 2023-03-03T05:27:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_dsa.ko 2023-03-03T05:27:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_none.ko 2023-03-03T05:27:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/dsa/tag_trailer.ko 2023-03-03T05:27:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154 2023-03-03T05:27:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/6lowpan 2023-03-03T05:27:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-03-03T05:27:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/ieee802154.ko 2023-03-03T05:27:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ieee802154/ieee802154_socket.ko 2023-03-03T05:27:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4 2023-03-03T05:27:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ah4.ko 2023-03-03T05:27:24.044 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/esp4.ko 2023-03-03T05:27:24.044 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/fou.ko 2023-03-03T05:27:24.044 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/gre.ko 2023-03-03T05:27:24.044 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/inet_diag.ko 2023-03-03T05:27:24.044 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_gre.ko 2023-03-03T05:27:24.044 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_tunnel.ko 2023-03-03T05:27:24.044 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ip_vti.ko 2023-03-03T05:27:24.045 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ipcomp.ko 2023-03-03T05:27:24.045 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/ipip.ko 2023-03-03T05:27:24.045 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter 2023-03-03T05:27:24.045 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arp_tables.ko 2023-03-03T05:27:24.045 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-03-03T05:27:24.045 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-03-03T05:27:24.045 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-03-03T05:27:24.046 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/iptable_security.ko 2023-03-03T05:27:24.047 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-03-03T05:27:24.047 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-03-03T05:27:24.047 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-03-03T05:27:24.047 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-03-03T05:27:24.047 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-03-03T05:27:24.047 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-03-03T05:27:24.047 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_bic.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_dctcp.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_diag.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_highspeed.ko 2023-03-03T05:27:24.048 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_htcp.ko 2023-03-03T05:27:24.049 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_hybla.ko 2023-03-03T05:27:24.049 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_illinois.ko 2023-03-03T05:27:24.049 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_lp.ko 2023-03-03T05:27:24.049 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_scalable.ko 2023-03-03T05:27:24.049 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_vegas.ko 2023-03-03T05:27:24.049 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_veno.ko 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_westwood.ko 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tcp_yeah.ko 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/tunnel4.ko 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/udp_diag.ko 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/udp_tunnel.ko 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv4/xfrm4_tunnel.ko 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6 2023-03-03T05:27:24.050 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ah6.ko 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/esp6.ko 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/fou6.ko 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_tunnel.ko 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ip6_vti.ko 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/ipcomp6.ko 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter 2023-03-03T05:27:24.051 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-03-03T05:27:24.052 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-03-03T05:27:24.053 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-03-03T05:27:24.053 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-03-03T05:27:24.053 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-03-03T05:27:24.053 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-03-03T05:27:24.053 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-03-03T05:27:24.053 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-03-03T05:27:24.053 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-03-03T05:27:24.054 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-03-03T05:27:24.055 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/sit.ko 2023-03-03T05:27:24.055 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/tunnel6.ko 2023-03-03T05:27:24.055 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/ipv6/xfrm6_tunnel.ko 2023-03-03T05:27:24.055 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/key 2023-03-03T05:27:24.055 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/key/af_key.ko 2023-03-03T05:27:24.055 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp 2023-03-03T05:27:24.055 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_core.ko 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_debugfs.ko 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_eth.ko 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ip.ko 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ip6.ko 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_netlink.ko 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/l2tp/l2tp_ppp.ko 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/llc 2023-03-03T05:27:24.056 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/llc/llc.ko 2023-03-03T05:27:24.057 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mac802154 2023-03-03T05:27:24.057 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mac802154/mac802154.ko 2023-03-03T05:27:24.057 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mpls 2023-03-03T05:27:24.057 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/mpls/mpls_gso.ko 2023-03-03T05:27:24.057 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter 2023-03-03T05:27:24.057 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset 2023-03-03T05:27:24.057 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set.ko 2023-03-03T05:27:24.058 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-03-03T05:27:24.058 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-03-03T05:27:24.058 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-03-03T05:27:24.058 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-03-03T05:27:24.058 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-03-03T05:27:24.058 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-03-03T05:27:24.058 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-03-03T05:27:24.059 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs.ko 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-03-03T05:27:24.060 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-03-03T05:27:24.061 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-03-03T05:27:24.061 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-03-03T05:27:24.061 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-03-03T05:27:24.061 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-03-03T05:27:24.061 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-03-03T05:27:24.061 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-03-03T05:27:24.061 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conncount.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_h323.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_irc.ko 2023-03-03T05:27:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-03-03T05:27:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-03-03T05:27:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-03-03T05:27:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_sane.ko 2023-03-03T05:27:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_sip.ko 2023-03-03T05:27:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-03-03T05:27:24.063 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-03-03T05:27:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_dup_netdev.ko 2023-03-03T05:27:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_flow_table.ko 2023-03-03T05:27:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_flow_table_inet.ko 2023-03-03T05:27:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_log_syslog.ko 2023-03-03T05:27:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat.ko 2023-03-03T05:27:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_amanda.ko 2023-03-03T05:27:24.064 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_ftp.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_irc.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_sip.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_nat_tftp.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_synproxy_core.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nf_tables.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_acct.ko 2023-03-03T05:27:24.065 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_log.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_osf.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nfnetlink_queue.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_chain_nat.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_compat.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_ct.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_dup_netdev.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_fib.ko 2023-03-03T05:27:24.066 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_fwd_netdev.ko 2023-03-03T05:27:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_hash.ko 2023-03-03T05:27:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_limit.ko 2023-03-03T05:27:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_log.ko 2023-03-03T05:27:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_masq.ko 2023-03-03T05:27:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_nat.ko 2023-03-03T05:27:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_numgen.ko 2023-03-03T05:27:24.067 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_queue.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_quota.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_redir.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_reject.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_reject_inet.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_socket.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/nft_tunnel.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_AUDIT.ko 2023-03-03T05:27:24.068 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CHECKSUM.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CLASSIFY.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_CT.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_DSCP.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_HL.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_HMARK.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_IDLETIMER.ko 2023-03-03T05:27:24.069 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_LED.ko 2023-03-03T05:27:24.070 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_LOG.ko 2023-03-03T05:27:24.070 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_MASQUERADE.ko 2023-03-03T05:27:24.070 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NETMAP.ko 2023-03-03T05:27:24.070 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NFLOG.ko 2023-03-03T05:27:24.070 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_NFQUEUE.ko 2023-03-03T05:27:24.070 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_RATEEST.ko 2023-03-03T05:27:24.070 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_REDIRECT.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_SECMARK.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TCPMSS.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TEE.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TPROXY.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_TRACE.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_addrtype.ko 2023-03-03T05:27:24.071 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_bpf.ko 2023-03-03T05:27:24.072 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cgroup.ko 2023-03-03T05:27:24.072 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cluster.ko 2023-03-03T05:27:24.072 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_comment.ko 2023-03-03T05:27:24.072 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connbytes.ko 2023-03-03T05:27:24.072 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connlabel.ko 2023-03-03T05:27:24.072 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connlimit.ko 2023-03-03T05:27:24.072 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_connmark.ko 2023-03-03T05:27:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_conntrack.ko 2023-03-03T05:27:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_cpu.ko 2023-03-03T05:27:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_dccp.ko 2023-03-03T05:27:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_devgroup.ko 2023-03-03T05:27:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_dscp.ko 2023-03-03T05:27:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ecn.ko 2023-03-03T05:27:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_esp.ko 2023-03-03T05:27:24.074 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_hashlimit.ko 2023-03-03T05:27:24.074 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_helper.ko 2023-03-03T05:27:24.074 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_hl.ko 2023-03-03T05:27:24.074 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ipcomp.ko 2023-03-03T05:27:24.074 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_iprange.ko 2023-03-03T05:27:24.074 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_ipvs.ko 2023-03-03T05:27:24.074 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_l2tp.ko 2023-03-03T05:27:24.075 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_length.ko 2023-03-03T05:27:24.075 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_limit.ko 2023-03-03T05:27:24.075 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_mac.ko 2023-03-03T05:27:24.075 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_mark.ko 2023-03-03T05:27:24.075 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_multiport.ko 2023-03-03T05:27:24.075 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_nat.ko 2023-03-03T05:27:24.075 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_nfacct.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_osf.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_owner.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_physdev.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_pkttype.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_policy.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_quota.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_rateest.ko 2023-03-03T05:27:24.076 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_realm.ko 2023-03-03T05:27:24.077 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_recent.ko 2023-03-03T05:27:24.077 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_sctp.ko 2023-03-03T05:27:24.077 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_set.ko 2023-03-03T05:27:24.077 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_socket.ko 2023-03-03T05:27:24.077 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_state.ko 2023-03-03T05:27:24.077 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_statistic.ko 2023-03-03T05:27:24.077 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_string.ko 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_tcpmss.ko 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_time.ko 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netfilter/xt_u32.ko 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netlink 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/netlink/netlink_diag.ko 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/nsh 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/nsh/nsh.ko 2023-03-03T05:27:24.078 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch 2023-03-03T05:27:24.079 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/openvswitch.ko 2023-03-03T05:27:24.079 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-geneve.ko 2023-03-03T05:27:24.079 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-gre.ko 2023-03-03T05:27:24.079 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/openvswitch/vport-vxlan.ko 2023-03-03T05:27:24.079 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/packet 2023-03-03T05:27:24.079 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/packet/af_packet_diag.ko 2023-03-03T05:27:24.079 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds 2023-03-03T05:27:24.080 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds.ko 2023-03-03T05:27:24.080 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds_rdma.ko 2023-03-03T05:27:24.080 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/rds/rds_tcp.ko 2023-03-03T05:27:24.080 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched 2023-03-03T05:27:24.081 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_csum.ko 2023-03-03T05:27:24.081 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_gact.ko 2023-03-03T05:27:24.081 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_ipt.ko 2023-03-03T05:27:24.081 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_mirred.ko 2023-03-03T05:27:24.082 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_nat.ko 2023-03-03T05:27:24.082 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_pedit.ko 2023-03-03T05:27:24.082 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_police.ko 2023-03-03T05:27:24.082 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_simple.ko 2023-03-03T05:27:24.083 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_skbedit.ko 2023-03-03T05:27:24.083 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/act_vlan.ko 2023-03-03T05:27:24.083 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_basic.ko 2023-03-03T05:27:24.083 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_bpf.ko 2023-03-03T05:27:24.084 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_flow.ko 2023-03-03T05:27:24.084 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_fw.ko 2023-03-03T05:27:24.084 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_route.ko 2023-03-03T05:27:24.084 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_rsvp.ko 2023-03-03T05:27:24.085 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_rsvp6.ko 2023-03-03T05:27:24.085 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_tcindex.ko 2023-03-03T05:27:24.085 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/cls_u32.ko 2023-03-03T05:27:24.085 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_cmp.ko 2023-03-03T05:27:24.085 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_ipset.ko 2023-03-03T05:27:24.085 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_meta.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_nbyte.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_text.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/em_u32.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_cbq.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_choke.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_codel.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_drr.ko 2023-03-03T05:27:24.086 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_dsmark.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_fq.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_gred.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_hfsc.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_hhf.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_htb.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_ingress.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_mqprio.ko 2023-03-03T05:27:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_multiq.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_netem.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_pie.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_plug.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_prio.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_qfq.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_red.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_sfb.ko 2023-03-03T05:27:24.088 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_sfq.ko 2023-03-03T05:27:24.089 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_tbf.ko 2023-03-03T05:27:24.089 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sched/sch_teql.ko 2023-03-03T05:27:24.089 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp 2023-03-03T05:27:24.089 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp/sctp.ko 2023-03-03T05:27:24.089 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sctp/sctp_diag.ko 2023-03-03T05:27:24.089 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc 2023-03-03T05:27:24.089 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/sunrpc.ko 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/xprtrdma 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc/diag.ko 2023-03-03T05:27:24.090 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/tipc/tipc.ko 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/unix 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/unix/unix_diag.ko 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock.ko 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock_diag.ko 2023-03-03T05:27:24.091 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/vmw_vsock/vsock_loopback.ko 2023-03-03T05:27:24.092 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/xfrm 2023-03-03T05:27:24.092 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/net/xfrm/xfrm_ipcomp.ko 2023-03-03T05:27:24.092 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security 2023-03-03T05:27:24.092 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys 2023-03-03T05:27:24.092 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/encrypted-keys 2023-03-03T05:27:24.092 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-03-03T05:27:24.092 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/trusted-keys 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/security/keys/trusted-keys/trusted.ko 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt/lib 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/kernel/virt/lib/irqbypass.ko 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.alias 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.alias.bin 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin 2023-03-03T05:27:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin.bin 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.builtin.modinfo 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.dep 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.dep.bin 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.devname 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.order 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.softdep 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.symbols 2023-03-03T05:27:24.094 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.2.0-ceph-g75534f74ac3d/modules.symbols.bin 2023-03-03T05:27:24.095 DEBUG:teuthology.task.kernel:get_image_version: 6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:24.096 DEBUG:teuthology.orchestra.run.smithi102:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-03-03T05:27:24.119 INFO:teuthology.orchestra.run.smithi102.stdout:postinstall scriptlet (using /bin/sh): 2023-03-03T05:27:24.119 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d -a -r /boot/System.map-6.2.0-ceph-g75534f74ac3d ]; then 2023-03-03T05:27:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:cp /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T05:27:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:cp /boot/System.map-6.2.0-ceph-g75534f74ac3d /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T05:27:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:rm -f /boot/vmlinuz-6.2.0-ceph-g75534f74ac3d /boot/System.map-6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/installkernel 6.2.0-ceph-g75534f74ac3d /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T05:27:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:rm -f /boot/.vmlinuz-6.2.0-ceph-g75534f74ac3d-rpm /boot/.System.map-6.2.0-ceph-g75534f74ac3d-rpm 2023-03-03T05:27:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2023-03-03T05:27:24.120 INFO:teuthology.orchestra.run.smithi102.stdout:preuninstall scriptlet (using /bin/sh): 2023-03-03T05:27:24.121 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-03-03T05:27:24.121 INFO:teuthology.orchestra.run.smithi102.stdout:new-kernel-pkg --remove 6.2.0-ceph-g75534f74ac3d --rminitrd --initrdfile=/boot/initramfs-6.2.0-ceph-g75534f74ac3d.img 2023-03-03T05:27:24.121 INFO:teuthology.orchestra.run.smithi102.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-03-03T05:27:24.121 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-install remove 6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:24.121 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2023-03-03T05:27:24.121 INFO:teuthology.orchestra.run.smithi102.stdout:postuninstall scriptlet (using /bin/sh): 2023-03-03T05:27:24.121 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /sbin/update-bootloader ]; then 2023-03-03T05:27:24.122 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/update-bootloader --remove 6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:24.122 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2023-03-03T05:27:24.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2023-03-03T05:27:24.231 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2023-03-03T05:27:24.231 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2023-03-03T05:27:24.231 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.02 2023-03-03T05:27:24.231 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 129.el8 2023-03-03T05:27:24.231 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:Group : System Environment/Base 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 9265586 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-03-03T05:27:24.232 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : kbuilder.bsys.centos.org 2023-03-03T05:27:24.233 INFO:teuthology.orchestra.run.smithi102.stdout:Relocations : (not relocatable) 2023-03-03T05:27:24.233 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : CentOS BuildSystem 2023-03-03T05:27:24.233 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2023-03-03T05:27:24.233 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2023-03-03T05:27:24.233 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2023-03-03T05:27:24.233 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2023-03-03T05:27:24.233 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:27:24.234 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-03-03T05:27:24.234 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2023-03-03T05:27:24.234 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2023-03-03T05:27:24.234 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2023-03-03T05:27:24.234 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:27:24.234 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2023-03-03T05:27:24.235 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-03-03T05:27:24.235 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:24.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-03-03T05:27:29.733 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2023-03-03T05:27:32.220 INFO:teuthology.orchestra.run.smithi102.stderr:done 2023-03-03T05:27:32.222 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2023-03-03T05:27:32.246 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.Z98raWs2cd 2023-03-03T05:27:32.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z98raWs2cd 2023-03-03T05:27:32.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.Z98raWs2cd 2023-03-03T05:27:32.480 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.Z98raWs2cd is 6KB 2023-03-03T05:27:32.527 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.Z98raWs2cd 2023-03-03T05:27:32.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2023-03-03T05:27:32.609 INFO:teuthology.orchestra.run.smithi102.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-03-03T05:27:32.609 INFO:teuthology.orchestra.run.smithi102.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-03-03T05:27:32.609 INFO:teuthology.orchestra.run.smithi102.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-03-03T05:27:32.609 INFO:teuthology.orchestra.run.smithi102.stdout:2ff282740cf84d4fa02ff60ed95fbbff-6.2.0-ceph-g75534f74ac3d.conf 2023-03-03T05:27:32.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-6.2.0-ceph-g75534f74ac3d 2023-03-03T05:27:32.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2023-03-03T05:28:02.780 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-03-03T05:28:02.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2023-03-03T05:28:02.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T05:28:21.253 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2023-03-03T05:28:51.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2023-03-03T05:28:51.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T05:29:09.637 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2023-03-03T05:29:39.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2023-03-03T05:29:39.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T05:29:40.143 DEBUG:teuthology.orchestra.run.smithi102:> true 2023-03-03T05:29:40.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2023-03-03T05:29:40.638 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.2.0-ceph-g75534f74ac3d"... 2023-03-03T05:29:40.638 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2023-03-03T05:29:40.705 INFO:teuthology.orchestra.run.smithi102.stdout:6.2.0-ceph-g75534f74ac3d 2023-03-03T05:29:40.706 DEBUG:teuthology.task.kernel:current kernel version is 6.2.0-ceph-g75534f74ac3d vs 6.2.0-ceph-g75534f74ac3d 2023-03-03T05:29:40.706 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-03-03T05:29:40.706 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-03-03T05:29:41.707 DEBUG:teuthology.parallel:result is None 2023-03-03T05:29:41.708 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-03T05:29:41.734 INFO:teuthology.task.internal:Creating test directory... 2023-03-03T05:29:41.735 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T05:29:41.740 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T05:29:41.806 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-03T05:29:41.820 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-03T05:29:41.853 INFO:teuthology.task.internal:Creating archive directory... 2023-03-03T05:29:41.854 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T05:29:41.859 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T05:29:42.199 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-03T05:29:42.207 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-03T05:29:42.207 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T05:29:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T05:29:42.226 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T05:29:42.231 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T05:29:42.306 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T05:29:42.322 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T05:29:42.400 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T05:29:42.445 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T05:29:42.451 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-03T05:29:42.471 INFO:teuthology.task.internal:Configuring sudo... 2023-03-03T05:29:42.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T05:29:42.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T05:29:42.574 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-03T05:29:42.589 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-03T05:29:42.590 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T05:29:42.595 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T05:29:42.657 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T05:29:42.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T05:29:42.857 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T05:29:42.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T05:29:43.056 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:29:43.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T05:29:43.156 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T05:29:43.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T05:29:43.255 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T05:29:43.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T05:29:43.380 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:29:43.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T05:29:43.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2023-03-03T05:29:43.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-03-03T05:29:43.555 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T05:29:43.603 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T05:29:43.985 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-03T05:29:43.993 INFO:teuthology.task.internal:Starting timer... 2023-03-03T05:29:43.993 INFO:teuthology.run_tasks:Running task pcp... 2023-03-03T05:29:44.008 INFO:teuthology.run_tasks:Running task selinux... 2023-03-03T05:29:44.022 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-03-03T05:29:44.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2023-03-03T05:29:44.175 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2023-03-03T05:29:44.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2023-03-03T05:29:44.251 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2023-03-03T05:29:44.253 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-03T05:29:44.254 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2023-03-03T05:29:44.324 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2023-03-03T05:29:44.325 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2023-03-03T05:29:44.394 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2023-03-03T05:29:44.394 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2023-03-03T05:29:44.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T05:29:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T05:29:44.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T05:29:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T05:29:44.524 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-03T05:29:44.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2023-03-03T05:29:44.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2023-03-03T05:29:44.651 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-03T05:29:44.660 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-03-03T05:29:44.661 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-03T05:29:44.679 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-03T05:29:44.680 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-03T05:41:48.649 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2023-03-03T05:41:48.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2023-03-03T05:41:48.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T05:41:48.752 DEBUG:teuthology.orchestra.run.smithi102:> true 2023-03-03T05:41:48.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-03-03T05:41:48.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T05:41:48.964 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-03-03T05:41:49.033 INFO:teuthology.run_tasks:Running task clock... 2023-03-03T05:41:49.102 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-03T05:41:49.103 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T05:41:49.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T05:41:49.109 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T05:41:49.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T05:41:49.146 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T05:41:49.161 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T05:41:49.209 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T05:41:49.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2023-03-03T05:41:49.235 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2023-03-03T05:41:49.250 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T05:41:49.253 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T05:41:49.263 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T05:41:49.331 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-03-03T05:41:49.334 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T05:41:49.335 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-03-03T05:41:49.335 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.335 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.335 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.336 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2023-03-03T05:41:49.404 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2023-03-03T05:41:49.448 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T05:41:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T05:41:49.611 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2023-03-03T05:41:49.621 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T05:41:49.621 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2023-03-03T05:41:49.622 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.622 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.622 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.622 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T05:41:49.623 INFO:teuthology.run_tasks:Running task install... 2023-03-03T05:41:49.690 DEBUG:teuthology.task.install:project ceph 2023-03-03T05:41:49.691 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}} 2023-03-03T05:41:49.691 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'} 2023-03-03T05:41:49.691 INFO:teuthology.task.install:Using flavor: default 2023-03-03T05:41:49.697 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-03T05:41:49.698 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-03-03T05:41:49.698 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T05:41:49.699 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T05:41:49.701 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T05:41:49.701 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T05:41:49.845 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T05:41:49.845 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T05:41:49.861 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T05:41:49.862 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T05:41:49.951 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T05:41:49.952 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:41:49.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T05:41:49.965 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T05:41:49.966 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:41:49.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T05:41:49.995 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T05:41:49.995 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T05:41:50.053 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T05:41:50.053 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T05:41:50.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T05:41:50.139 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T05:41:50.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T05:41:50.205 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2023-03-03T05:41:50.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2023-03-03T05:41:50.328 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T05:41:50.427 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2023-03-03T05:41:50.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2023-03-03T05:41:50.469 INFO:teuthology.orchestra.run.smithi172.stdout:71 files removed 2023-03-03T05:41:50.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T05:41:50.842 INFO:teuthology.orchestra.run.smithi102.stdout:71 files removed 2023-03-03T05:41:50.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T05:41:51.113 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 332 kB/s | 80 kB 00:00 2023-03-03T05:41:51.381 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 41 kB/s | 9.4 kB 00:00 2023-03-03T05:41:51.631 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 6.6 kB/s | 1.5 kB 00:00 2023-03-03T05:41:51.771 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 337 kB/s | 80 kB 00:00 2023-03-03T05:41:52.038 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 41 kB/s | 9.4 kB 00:00 2023-03-03T05:41:52.297 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 6.4 kB/s | 1.5 kB 00:00 2023-03-03T05:41:52.326 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - AppStream 42 MB/s | 28 MB 00:00 2023-03-03T05:41:53.025 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - AppStream 40 MB/s | 28 MB 00:00 2023-03-03T05:41:58.201 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - BaseOS 42 MB/s | 27 MB 00:00 2023-03-03T05:41:59.426 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - BaseOS 36 MB/s | 27 MB 00:00 2023-03-03T05:42:02.516 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - Extras 143 kB/s | 18 kB 00:00 2023-03-03T05:42:02.944 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - PowerTools 19 MB/s | 5.7 MB 00:00 2023-03-03T05:42:04.024 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Extras 151 kB/s | 18 kB 00:00 2023-03-03T05:42:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - PowerTools 20 MB/s | 5.7 MB 00:00 2023-03-03T05:42:06.576 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-03-03T05:42:07.601 INFO:teuthology.orchestra.run.smithi102.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2023-03-03T05:42:09.097 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 14 MB 00:02 2023-03-03T05:42:09.710 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 14 MB 00:01 2023-03-03T05:42:12.601 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 362 kB/s | 24 kB 00:00 2023-03-03T05:42:13.944 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 419 kB/s | 24 kB 00:00 2023-03-03T05:42:16.097 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T05:42:16.098 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T05:42:16.109 INFO:teuthology.orchestra.run.smithi172.stdout:Package NetworkManager-1:1.39.12-1.el8.x86_64 is already installed. 2023-03-03T05:42:16.109 INFO:teuthology.orchestra.run.smithi172.stdout:Package NetworkManager-initscripts-updown-1:1.39.12-1.el8.noarch is already installed. 2023-03-03T05:42:16.110 INFO:teuthology.orchestra.run.smithi172.stdout:Package iproute-5.18.0-1.el8.x86_64 is already installed. 2023-03-03T05:42:16.111 INFO:teuthology.orchestra.run.smithi172.stdout:Package util-linux-2.32.1-38.el8.x86_64 is already installed. 2023-03-03T05:42:16.112 INFO:teuthology.orchestra.run.smithi172.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T05:42:16.113 INFO:teuthology.orchestra.run.smithi172.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T05:42:16.116 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T05:42:16.117 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T05:42:16.246 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-03-03T05:42:16.259 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================================= 2023-03-03T05:42:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-03-03T05:42:16.259 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================================= 2023-03-03T05:42:16.260 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-03-03T05:42:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: bison x86_64 3.0.4-10.el8 CentOS-AppStream 688 k 2023-03-03T05:42:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T05:42:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T05:42:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T05:42:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T05:42:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T05:42:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T05:42:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T05:42:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T05:42:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T05:42:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T05:42:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T05:42:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel x86_64 0.188-3.el8 CentOS-BaseOS 61 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: flex x86_64 2.6.1-9.el8 CentOS-AppStream 320 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 19 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T05:42:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: libtool x86_64 2.4.6-25.el8 CentOS-AppStream 709 k 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel x86_64 2.32.1-40.el8 CentOS-BaseOS 99 k 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel x86_64 1:1.1.1k-7.el8 CentOS-BaseOS 2.3 M 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 1.5 M 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 807 k 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 5.1 M 2023-03-03T05:42:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T05:42:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T05:42:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T05:42:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T05:42:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T05:42:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T05:42:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T05:42:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 CentOS-BaseOS 72 k 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-initscripts-updown noarch 1:1.40.12-1.el8 CentOS-BaseOS 142 k 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-debuginfod-client x86_64 0.188-3.el8 CentOS-BaseOS 75 k 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf x86_64 0.188-3.el8 CentOS-BaseOS 231 k 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libs x86_64 0.188-3.el8 CentOS-BaseOS 299 k 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: libblkid x86_64 2.32.1-40.el8 CentOS-BaseOS 220 k 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: libfdisk x86_64 2.32.1-40.el8 CentOS-BaseOS 253 k 2023-03-03T05:42:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: libmount x86_64 2.32.1-40.el8 CentOS-BaseOS 236 k 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: libsmartcols x86_64 2.32.1-40.el8 CentOS-BaseOS 179 k 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid x86_64 2.32.1-40.el8 CentOS-BaseOS 98 k 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux x86_64 2.32.1-40.el8 CentOS-BaseOS 2.5 M 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-user x86_64 2.32.1-40.el8 CentOS-BaseOS 102 k 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-03-03T05:42:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: automake noarch 1.16.1-7.el8 CentOS-AppStream 713 k 2023-03-03T05:42:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T05:42:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T05:42:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T05:42:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T05:42:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T05:42:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T05:42:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 CentOS-BaseOS 48 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: krb5-devel x86_64 1.18.2-22.el8 CentOS-BaseOS 561 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T05:42:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: libcom_err-devel x86_64 1.45.6-5.el8 CentOS-BaseOS 39 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: libpq x86_64 13.5-1.el8 CentOS-AppStream 198 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-03-03T05:42:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T05:42:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: libselinux-devel x86_64 2.9-6.el8 CentOS-BaseOS 200 k 2023-03-03T05:42:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: libsepol-devel x86_64 2.9-3.el8 CentOS-BaseOS 87 k 2023-03-03T05:42:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T05:42:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: libverto-devel x86_64 0.3.2-2.el8 CentOS-BaseOS 18 k 2023-03-03T05:42:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-03-03T05:42:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-03-03T05:42:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-devel x86_64 10.32-3.el8 CentOS-BaseOS 605 k 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf16 x86_64 10.32-3.el8 CentOS-BaseOS 229 k 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf32 x86_64 10.32-3.el8 CentOS-BaseOS 220 k 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T05:42:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T05:42:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T05:42:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T05:42:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T05:42:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T05:42:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-03-03T05:42:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T05:42:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-03-03T05:42:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.0.6-14.el8 CentOS-AppStream 157 k 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-03-03T05:42:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-03-03T05:42:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-03-03T05:42:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-03-03T05:42:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T05:42:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-03-03T05:42:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T05:42:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T05:42:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-03-03T05:42:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T05:42:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T05:42:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T05:42:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T05:42:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T05:42:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T05:42:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T05:42:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: uuid x86_64 1.6.2-43.el8 CentOS-AppStream 64 k 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel x86_64 1.2.11-19.el8 CentOS-BaseOS 58 k 2023-03-03T05:42:16.278 INFO:teuthology.orchestra.run.smithi172.stdout:Enabling module streams: 2023-03-03T05:42:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql 10 2023-03-03T05:42:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:42:16.279 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-03-03T05:42:16.279 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================================= 2023-03-03T05:42:16.279 INFO:teuthology.orchestra.run.smithi172.stdout:Install 127 Packages 2023-03-03T05:42:16.279 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 13 Packages 2023-03-03T05:42:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:42:16.280 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 206 M 2023-03-03T05:42:16.280 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-03-03T05:42:16.385 INFO:teuthology.orchestra.run.smithi172.stdout:(1/140): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 59 kB/s | 6.4 kB 00:00 2023-03-03T05:42:16.652 INFO:teuthology.orchestra.run.smithi172.stdout:(2/140): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 3.1 MB/s | 845 kB 00:00 2023-03-03T05:42:16.745 INFO:teuthology.orchestra.run.smithi172.stdout:(3/140): ceph-immutable-object-cache-18.0.0-273 1.8 MB/s | 166 kB 00:00 2023-03-03T05:42:16.878 INFO:teuthology.orchestra.run.smithi172.stdout:(4/140): ceph-base-18.0.0-2739.g472aaef4.el8.x8 8.9 MB/s | 5.4 MB 00:00 2023-03-03T05:42:17.079 INFO:teuthology.orchestra.run.smithi172.stdout:(5/140): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 6.8 MB/s | 2.3 MB 00:00 2023-03-03T05:42:17.112 INFO:teuthology.orchestra.run.smithi172.stdout:(6/140): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 6.6 MB/s | 1.5 MB 00:00 2023-03-03T05:42:17.569 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T05:42:17.570 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T05:42:17.582 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-1:1.39.12-1.el8.x86_64 is already installed. 2023-03-03T05:42:17.583 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-initscripts-updown-1:1.39.12-1.el8.noarch is already installed. 2023-03-03T05:42:17.584 INFO:teuthology.orchestra.run.smithi102.stdout:Package iproute-5.18.0-1.el8.x86_64 is already installed. 2023-03-03T05:42:17.584 INFO:teuthology.orchestra.run.smithi102.stdout:Package util-linux-2.32.1-38.el8.x86_64 is already installed. 2023-03-03T05:42:17.585 INFO:teuthology.orchestra.run.smithi102.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T05:42:17.587 INFO:teuthology.orchestra.run.smithi102.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T05:42:17.590 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T05:42:17.591 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T05:42:17.732 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2023-03-03T05:42:17.745 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================================= 2023-03-03T05:42:17.746 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2023-03-03T05:42:17.746 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================================= 2023-03-03T05:42:17.746 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2023-03-03T05:42:17.746 INFO:teuthology.orchestra.run.smithi102.stdout: bison x86_64 3.0.4-10.el8 CentOS-AppStream 688 k 2023-03-03T05:42:17.746 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T05:42:17.746 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T05:42:17.747 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T05:42:17.747 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T05:42:17.747 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T05:42:17.747 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T05:42:17.747 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T05:42:17.748 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T05:42:17.749 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel x86_64 0.188-3.el8 CentOS-BaseOS 61 k 2023-03-03T05:42:17.749 INFO:teuthology.orchestra.run.smithi102.stdout: flex x86_64 2.6.1-9.el8 CentOS-AppStream 320 k 2023-03-03T05:42:17.749 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 19 k 2023-03-03T05:42:17.749 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T05:42:17.750 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T05:42:17.750 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T05:42:17.750 INFO:teuthology.orchestra.run.smithi102.stdout: libtool x86_64 2.4.6-25.el8 CentOS-AppStream 709 k 2023-03-03T05:42:17.751 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel x86_64 2.32.1-40.el8 CentOS-BaseOS 99 k 2023-03-03T05:42:17.751 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel x86_64 1:1.1.1k-7.el8 CentOS-BaseOS 2.3 M 2023-03-03T05:42:17.752 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 1.5 M 2023-03-03T05:42:17.752 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 807 k 2023-03-03T05:42:17.752 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server x86_64 10.19-1.module_el8.6.0+1047+4202cf9a CentOS-AppStream 5.1 M 2023-03-03T05:42:17.752 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T05:42:17.753 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T05:42:17.753 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T05:42:17.753 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T05:42:17.754 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T05:42:17.754 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T05:42:17.754 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T05:42:17.755 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 CentOS-BaseOS 72 k 2023-03-03T05:42:17.755 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2023-03-03T05:42:17.755 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-initscripts-updown noarch 1:1.40.12-1.el8 CentOS-BaseOS 142 k 2023-03-03T05:42:17.756 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client x86_64 0.188-3.el8 CentOS-BaseOS 75 k 2023-03-03T05:42:17.756 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf x86_64 0.188-3.el8 CentOS-BaseOS 231 k 2023-03-03T05:42:17.756 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs x86_64 0.188-3.el8 CentOS-BaseOS 299 k 2023-03-03T05:42:17.756 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid x86_64 2.32.1-40.el8 CentOS-BaseOS 220 k 2023-03-03T05:42:17.757 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk x86_64 2.32.1-40.el8 CentOS-BaseOS 253 k 2023-03-03T05:42:17.757 INFO:teuthology.orchestra.run.smithi102.stdout: libmount x86_64 2.32.1-40.el8 CentOS-BaseOS 236 k 2023-03-03T05:42:17.757 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T05:42:17.757 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T05:42:17.758 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols x86_64 2.32.1-40.el8 CentOS-BaseOS 179 k 2023-03-03T05:42:17.758 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid x86_64 2.32.1-40.el8 CentOS-BaseOS 98 k 2023-03-03T05:42:17.758 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux x86_64 2.32.1-40.el8 CentOS-BaseOS 2.5 M 2023-03-03T05:42:17.758 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user x86_64 2.32.1-40.el8 CentOS-BaseOS 102 k 2023-03-03T05:42:17.758 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2023-03-03T05:42:17.759 INFO:teuthology.orchestra.run.smithi102.stdout: automake noarch 1.16.1-7.el8 CentOS-AppStream 713 k 2023-03-03T05:42:17.759 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T05:42:17.759 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T05:42:17.759 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T05:42:17.760 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T05:42:17.760 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T05:42:17.760 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T05:42:17.760 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T05:42:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T05:42:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-03-03T05:42:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T05:42:17.761 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-03-03T05:42:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 CentOS-BaseOS 48 k 2023-03-03T05:42:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: krb5-devel x86_64 1.18.2-22.el8 CentOS-BaseOS 561 k 2023-03-03T05:42:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T05:42:17.762 INFO:teuthology.orchestra.run.smithi102.stdout: libcom_err-devel x86_64 1.45.6-5.el8 CentOS-BaseOS 39 k 2023-03-03T05:42:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-03-03T05:42:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T05:42:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: libpq x86_64 13.5-1.el8 CentOS-AppStream 198 k 2023-03-03T05:42:17.763 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-03-03T05:42:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-03-03T05:42:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T05:42:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-03-03T05:42:17.764 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T05:42:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: libselinux-devel x86_64 2.9-6.el8 CentOS-BaseOS 200 k 2023-03-03T05:42:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: libsepol-devel x86_64 2.9-3.el8 CentOS-BaseOS 87 k 2023-03-03T05:42:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T05:42:17.766 INFO:teuthology.orchestra.run.smithi102.stdout: libverto-devel x86_64 0.3.2-2.el8 CentOS-BaseOS 18 k 2023-03-03T05:42:17.766 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-03-03T05:42:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-03-03T05:42:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-03-03T05:42:17.767 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-03-03T05:42:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-devel x86_64 10.32-3.el8 CentOS-BaseOS 605 k 2023-03-03T05:42:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf16 x86_64 10.32-3.el8 CentOS-BaseOS 229 k 2023-03-03T05:42:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf32 x86_64 10.32-3.el8 CentOS-BaseOS 220 k 2023-03-03T05:42:17.768 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T05:42:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-03-03T05:42:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T05:42:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T05:42:17.769 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T05:42:17.770 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T05:42:17.770 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T05:42:17.770 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T05:42:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-03-03T05:42:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T05:42:17.771 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T05:42:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-03-03T05:42:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T05:42:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T05:42:17.772 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T05:42:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-03-03T05:42:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-03-03T05:42:17.773 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-03-03T05:42:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-03-03T05:42:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T05:42:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T05:42:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.0.6-14.el8 CentOS-AppStream 157 k 2023-03-03T05:42:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-03-03T05:42:17.774 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T05:42:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T05:42:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-03-03T05:42:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-03-03T05:42:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-03-03T05:42:17.775 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T05:42:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T05:42:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-03-03T05:42:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-03-03T05:42:17.776 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-03-03T05:42:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-03-03T05:42:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-03-03T05:42:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-03-03T05:42:17.777 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T05:42:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-03-03T05:42:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T05:42:17.778 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T05:42:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-03-03T05:42:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T05:42:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T05:42:17.779 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T05:42:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T05:42:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T05:42:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T05:42:17.780 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T05:42:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T05:42:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-03-03T05:42:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T05:42:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-03-03T05:42:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T05:42:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: uuid x86_64 1.6.2-43.el8 CentOS-AppStream 64 k 2023-03-03T05:42:17.782 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-03-03T05:42:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel x86_64 1.2.11-19.el8 CentOS-BaseOS 58 k 2023-03-03T05:42:17.783 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling module streams: 2023-03-03T05:42:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql 10 2023-03-03T05:42:17.783 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:42:17.784 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2023-03-03T05:42:17.784 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================================= 2023-03-03T05:42:17.784 INFO:teuthology.orchestra.run.smithi102.stdout:Install 127 Packages 2023-03-03T05:42:17.784 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 13 Packages 2023-03-03T05:42:17.785 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:42:17.785 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 206 M 2023-03-03T05:42:17.786 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2023-03-03T05:42:17.812 INFO:teuthology.orchestra.run.smithi172.stdout:(7/140): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 6.3 MB/s | 4.6 MB 00:00 2023-03-03T05:42:18.004 INFO:teuthology.orchestra.run.smithi102.stdout:(1/140): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 28 kB/s | 6.4 kB 00:00 2023-03-03T05:42:18.529 INFO:teuthology.orchestra.run.smithi172.stdout:(8/140): ceph-common-18.0.0-2739.g472aaef4.el8. 9.5 MB/s | 22 MB 00:02 2023-03-03T05:42:18.624 INFO:teuthology.orchestra.run.smithi172.stdout:(9/140): ceph-selinux-18.0.0-2739.g472aaef4.el8 266 kB/s | 24 kB 00:00 2023-03-03T05:42:18.813 INFO:teuthology.orchestra.run.smithi172.stdout:(10/140): ceph-osd-18.0.0-2739.g472aaef4.el8.x8 10 MB/s | 18 MB 00:01 2023-03-03T05:42:19.172 INFO:teuthology.orchestra.run.smithi172.stdout:(11/140): cephfs-mirror-18.0.0-2739.g472aaef4.e 718 kB/s | 257 kB 00:00 2023-03-03T05:42:19.255 INFO:teuthology.orchestra.run.smithi172.stdout:(12/140): libcephfs-devel-18.0.0-2739.g472aaef4 367 kB/s | 30 kB 00:00 2023-03-03T05:42:19.480 INFO:teuthology.orchestra.run.smithi172.stdout:(13/140): libcephfs2-18.0.0-2739.g472aaef4.el8. 3.3 MB/s | 767 kB 00:00 2023-03-03T05:42:19.487 INFO:teuthology.orchestra.run.smithi102.stdout:(2/140): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 570 kB/s | 845 kB 00:01 2023-03-03T05:42:19.564 INFO:teuthology.orchestra.run.smithi172.stdout:(14/140): libcephsqlite-18.0.0-2739.g472aaef4.e 1.6 MB/s | 137 kB 00:00 2023-03-03T05:42:19.647 INFO:teuthology.orchestra.run.smithi172.stdout:(15/140): librados-devel-18.0.0-2739.g472aaef4. 1.6 MB/s | 139 kB 00:00 2023-03-03T05:42:19.754 INFO:teuthology.orchestra.run.smithi102.stdout:(3/140): ceph-immutable-object-cache-18.0.0-273 622 kB/s | 166 kB 00:00 2023-03-03T05:42:19.856 INFO:teuthology.orchestra.run.smithi172.stdout:(16/140): ceph-radosgw-18.0.0-2739.g472aaef4.el 6.4 MB/s | 13 MB 00:02 2023-03-03T05:42:19.890 INFO:teuthology.orchestra.run.smithi172.stdout:(17/140): libradosstriper1-18.0.0-2739.g472aaef 1.9 MB/s | 474 kB 00:00 2023-03-03T05:42:19.982 INFO:teuthology.orchestra.run.smithi172.stdout:(18/140): python3-ceph-argparse-18.0.0-2739.g47 487 kB/s | 45 kB 00:00 2023-03-03T05:42:20.057 INFO:teuthology.orchestra.run.smithi172.stdout:(19/140): python3-ceph-common-18.0.0-2739.g472a 1.5 MB/s | 117 kB 00:00 2023-03-03T05:42:20.127 INFO:teuthology.orchestra.run.smithi172.stdout:(20/140): python3-cephfs-18.0.0-2739.g472aaef4. 2.8 MB/s | 190 kB 00:00 2023-03-03T05:42:20.224 INFO:teuthology.orchestra.run.smithi172.stdout:(21/140): python3-rados-18.0.0-2739.g472aaef4.e 3.6 MB/s | 371 kB 00:00 2023-03-03T05:42:20.262 INFO:teuthology.orchestra.run.smithi102.stdout:(4/140): ceph-common-18.0.0-2739.g472aaef4.el8. 8.6 MB/s | 22 MB 00:02 2023-03-03T05:42:20.283 INFO:teuthology.orchestra.run.smithi172.stdout:(22/140): python3-rbd-18.0.0-2739.g472aaef4.el8 6.1 MB/s | 364 kB 00:00 2023-03-03T05:42:20.358 INFO:teuthology.orchestra.run.smithi172.stdout:(23/140): python3-rgw-18.0.0-2739.g472aaef4.el8 1.4 MB/s | 107 kB 00:00 2023-03-03T05:42:20.441 INFO:teuthology.orchestra.run.smithi172.stdout:(24/140): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 1.1 MB/s | 94 kB 00:00 2023-03-03T05:42:20.471 INFO:teuthology.orchestra.run.smithi102.stdout:(5/140): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 7.4 MB/s | 1.5 MB 00:00 2023-03-03T05:42:20.758 INFO:teuthology.orchestra.run.smithi172.stdout:(25/140): librgw2-18.0.0-2739.g472aaef4.el8.x86 7.2 MB/s | 6.5 MB 00:00 2023-03-03T05:42:20.850 INFO:teuthology.orchestra.run.smithi172.stdout:(26/140): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 1.9 MB/s | 180 kB 00:00 2023-03-03T05:42:20.952 INFO:teuthology.orchestra.run.smithi172.stdout:(27/140): ceph-grafana-dashboards-18.0.0-2739.g 231 kB/s | 23 kB 00:00 2023-03-03T05:42:20.965 INFO:teuthology.orchestra.run.smithi102.stdout:(6/140): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 9.4 MB/s | 4.6 MB 00:00 2023-03-03T05:42:21.009 INFO:teuthology.orchestra.run.smithi172.stdout:(28/140): ceph-mgr-cephadm-18.0.0-2739.g472aaef 2.1 MB/s | 123 kB 00:00 2023-03-03T05:42:21.085 INFO:teuthology.orchestra.run.smithi172.stdout:(29/140): rbd-mirror-18.0.0-2739.g472aaef4.el8. 5.4 MB/s | 3.4 MB 00:00 2023-03-03T05:42:21.343 INFO:teuthology.orchestra.run.smithi172.stdout:(30/140): ceph-mgr-dashboard-18.0.0-2739.g472aa 5.2 MB/s | 1.7 MB 00:00 2023-03-03T05:42:21.443 INFO:teuthology.orchestra.run.smithi172.stdout:(31/140): ceph-mgr-modules-core-18.0.0-2739.g47 2.3 MB/s | 236 kB 00:00 2023-03-03T05:42:21.527 INFO:teuthology.orchestra.run.smithi172.stdout:(32/140): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 568 kB/s | 47 kB 00:00 2023-03-03T05:42:21.610 INFO:teuthology.orchestra.run.smithi172.stdout:(33/140): ceph-prometheus-alerts-18.0.0-2739.g4 175 kB/s | 15 kB 00:00 2023-03-03T05:42:21.711 INFO:teuthology.orchestra.run.smithi172.stdout:(34/140): ceph-volume-18.0.0-2739.g472aaef4.el8 2.6 MB/s | 263 kB 00:00 2023-03-03T05:42:21.803 INFO:teuthology.orchestra.run.smithi172.stdout:(35/140): cephadm-18.0.0-2739.g472aaef4.el8.noa 902 kB/s | 83 kB 00:00 2023-03-03T05:42:21.887 INFO:teuthology.orchestra.run.smithi172.stdout:(36/140): cephfs-top-18.0.0-2739.g472aaef4.el8. 217 kB/s | 18 kB 00:00 2023-03-03T05:42:21.962 INFO:teuthology.orchestra.run.smithi172.stdout:(37/140): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-03-03T05:42:22.064 INFO:teuthology.orchestra.run.smithi172.stdout:(38/140): bison-3.0.4-10.el8.x86_64.rpm 6.6 MB/s | 688 kB 00:00 2023-03-03T05:42:22.129 INFO:teuthology.orchestra.run.smithi172.stdout:(39/140): flex-2.6.1-9.el8.x86_64.rpm 4.9 MB/s | 320 kB 00:00 2023-03-03T05:42:22.146 INFO:teuthology.orchestra.run.smithi172.stdout:(40/140): jq-1.6-6.el8.x86_64.rpm 12 MB/s | 202 kB 00:00 2023-03-03T05:42:22.147 INFO:teuthology.orchestra.run.smithi102.stdout:(7/140): ceph-base-18.0.0-2739.g472aaef4.el8.x8 1.2 MB/s | 5.4 MB 00:04 2023-03-03T05:42:22.171 INFO:teuthology.orchestra.run.smithi172.stdout:(41/140): libpq-13.5-1.el8.x86_64.rpm 7.8 MB/s | 198 kB 00:00 2023-03-03T05:42:22.196 INFO:teuthology.orchestra.run.smithi102.stdout:(8/140): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 953 kB/s | 2.3 MB 00:02 2023-03-03T05:42:22.205 INFO:teuthology.orchestra.run.smithi172.stdout:(42/140): librdkafka-0.11.4-3.el8.x86_64.rpm 10 MB/s | 354 kB 00:00 2023-03-03T05:42:22.239 INFO:teuthology.orchestra.run.smithi172.stdout:(43/140): libtool-2.4.6-25.el8.x86_64.rpm 21 MB/s | 709 kB 00:00 2023-03-03T05:42:22.264 INFO:teuthology.orchestra.run.smithi172.stdout:(44/140): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2023-03-03T05:42:22.272 INFO:teuthology.orchestra.run.smithi102.stdout:(9/140): ceph-selinux-18.0.0-2739.g472aaef4.el8 324 kB/s | 24 kB 00:00 2023-03-03T05:42:22.289 INFO:teuthology.orchestra.run.smithi172.stdout:(45/140): oniguruma-6.8.2-2.el8.x86_64.rpm 7.4 MB/s | 187 kB 00:00 2023-03-03T05:42:22.427 INFO:teuthology.orchestra.run.smithi172.stdout:(46/140): openblas-0.3.15-4.el8.x86_64.rpm 34 MB/s | 4.7 MB 00:00 2023-03-03T05:42:22.515 INFO:teuthology.orchestra.run.smithi172.stdout:(47/140): ceph-mgr-diskprediction-local-18.0.0- 5.2 MB/s | 7.4 MB 00:01 2023-03-03T05:42:22.606 INFO:teuthology.orchestra.run.smithi102.stdout:(10/140): ceph-osd-18.0.0-2739.g472aaef4.el8.x8 11 MB/s | 18 MB 00:01 2023-03-03T05:42:22.640 INFO:teuthology.orchestra.run.smithi172.stdout:(48/140): openblas-threads-0.3.15-4.el8.x86_64. 23 MB/s | 4.9 MB 00:00 2023-03-03T05:42:22.656 INFO:teuthology.orchestra.run.smithi102.stdout:(11/140): cephfs-mirror-18.0.0-2739.g472aaef4.e 5.1 MB/s | 257 kB 00:00 2023-03-03T05:42:22.674 INFO:teuthology.orchestra.run.smithi172.stdout:(49/140): postgresql-contrib-10.19-1.module_el8 24 MB/s | 807 kB 00:00 2023-03-03T05:42:22.698 INFO:teuthology.orchestra.run.smithi102.stdout:(12/140): libcephfs-devel-18.0.0-2739.g472aaef4 737 kB/s | 30 kB 00:00 2023-03-03T05:42:22.797 INFO:teuthology.orchestra.run.smithi172.stdout:(50/140): postgresql-10.19-1.module_el8.6.0+104 5.3 MB/s | 1.5 MB 00:00 2023-03-03T05:42:22.799 INFO:teuthology.orchestra.run.smithi102.stdout:(13/140): libcephfs2-18.0.0-2739.g472aaef4.el8. 7.5 MB/s | 767 kB 00:00 2023-03-03T05:42:22.856 INFO:teuthology.orchestra.run.smithi102.stdout:(14/140): libcephsqlite-18.0.0-2739.g472aaef4.e 2.3 MB/s | 137 kB 00:00 2023-03-03T05:42:22.882 INFO:teuthology.orchestra.run.smithi172.stdout:(51/140): postgresql-server-10.19-1.module_el8. 24 MB/s | 5.1 MB 00:00 2023-03-03T05:42:22.915 INFO:teuthology.orchestra.run.smithi102.stdout:(15/140): librados-devel-18.0.0-2739.g472aaef4. 2.3 MB/s | 139 kB 00:00 2023-03-03T05:42:22.916 INFO:teuthology.orchestra.run.smithi172.stdout:(52/140): python3-jinja2-2.10.1-3.el8.noarch.rp 16 MB/s | 538 kB 00:00 2023-03-03T05:42:22.933 INFO:teuthology.orchestra.run.smithi172.stdout:(53/140): python3-jsonpatch-1.21-2.el8.noarch.r 1.6 MB/s | 27 kB 00:00 2023-03-03T05:42:22.950 INFO:teuthology.orchestra.run.smithi172.stdout:(54/140): python3-jsonpointer-1.10-11.el8.noarc 1.2 MB/s | 20 kB 00:00 2023-03-03T05:42:22.975 INFO:teuthology.orchestra.run.smithi172.stdout:(55/140): python3-mako-1.0.6-14.el8.noarch.rpm 6.2 MB/s | 157 kB 00:00 2023-03-03T05:42:22.991 INFO:teuthology.orchestra.run.smithi102.stdout:(16/140): libradosstriper1-18.0.0-2739.g472aaef 6.2 MB/s | 474 kB 00:00 2023-03-03T05:42:22.992 INFO:teuthology.orchestra.run.smithi172.stdout:(56/140): python3-markupsafe-0.23-19.el8.x86_64 2.3 MB/s | 39 kB 00:00 2023-03-03T05:42:23.101 INFO:teuthology.orchestra.run.smithi172.stdout:(57/140): python3-numpy-1.14.3-10.el8.x86_64.rp 34 MB/s | 3.7 MB 00:00 2023-03-03T05:42:23.135 INFO:teuthology.orchestra.run.smithi172.stdout:(58/140): python3-numpy-f2py-1.14.3-10.el8.x86_ 6.6 MB/s | 225 kB 00:00 2023-03-03T05:42:23.193 INFO:teuthology.orchestra.run.smithi172.stdout:(59/140): python3-prettytable-0.7.2-14.el8.noar 755 kB/s | 44 kB 00:00 2023-03-03T05:42:23.218 INFO:teuthology.orchestra.run.smithi172.stdout:(60/140): python3-pyOpenSSL-19.0.0-1.el8.noarch 4.0 MB/s | 103 kB 00:00 2023-03-03T05:42:23.285 INFO:teuthology.orchestra.run.smithi172.stdout:(61/140): python3-babel-2.5.1-7.el8.noarch.rpm 9.8 MB/s | 4.8 MB 00:00 2023-03-03T05:42:23.311 INFO:teuthology.orchestra.run.smithi172.stdout:(62/140): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.3 MB/s | 126 kB 00:00 2023-03-03T05:42:23.336 INFO:teuthology.orchestra.run.smithi172.stdout:(63/140): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-03-03T05:42:23.370 INFO:teuthology.orchestra.run.smithi172.stdout:(64/140): python3-pytz-2017.2-9.el8.noarch.rpm 923 kB/s | 54 kB 00:00 2023-03-03T05:42:23.403 INFO:teuthology.orchestra.run.smithi172.stdout:(65/140): python3-werkzeug-0.12.2-4.el8.noarch. 13 MB/s | 457 kB 00:00 2023-03-03T05:42:23.445 INFO:teuthology.orchestra.run.smithi172.stdout:(66/140): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-03-03T05:42:23.470 INFO:teuthology.orchestra.run.smithi172.stdout:(67/140): uuid-1.6.2-43.el8.x86_64.rpm 2.5 MB/s | 64 kB 00:00 2023-03-03T05:42:23.474 INFO:teuthology.orchestra.run.smithi102.stdout:(17/140): librgw2-18.0.0-2739.g472aaef4.el8.x86 13 MB/s | 6.5 MB 00:00 2023-03-03T05:42:23.496 INFO:teuthology.orchestra.run.smithi172.stdout:(68/140): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-03-03T05:42:23.533 INFO:teuthology.orchestra.run.smithi102.stdout:(18/140): python3-ceph-argparse-18.0.0-2739.g47 766 kB/s | 45 kB 00:00 2023-03-03T05:42:23.563 INFO:teuthology.orchestra.run.smithi172.stdout:(69/140): elfutils-libelf-devel-0.188-3.el8.x86 918 kB/s | 61 kB 00:00 2023-03-03T05:42:23.583 INFO:teuthology.orchestra.run.smithi102.stdout:(19/140): python3-ceph-common-18.0.0-2739.g472a 2.3 MB/s | 117 kB 00:00 2023-03-03T05:42:23.588 INFO:teuthology.orchestra.run.smithi172.stdout:(70/140): fuse-2.9.7-16.el8.x86_64.rpm 3.2 MB/s | 83 kB 00:00 2023-03-03T05:42:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:(20/140): python3-cephfs-18.0.0-2739.g472aaef4. 3.7 MB/s | 190 kB 00:00 2023-03-03T05:42:23.646 INFO:teuthology.orchestra.run.smithi172.stdout:(71/140): keyutils-libs-devel-1.5.10-9.el8.x86_ 827 kB/s | 48 kB 00:00 2023-03-03T05:42:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:(21/140): python3-rados-18.0.0-2739.g472aaef4.e 6.2 MB/s | 371 kB 00:00 2023-03-03T05:42:23.714 INFO:teuthology.orchestra.run.smithi172.stdout:(72/140): krb5-devel-1.18.2-22.el8.x86_64.rpm 8.2 MB/s | 561 kB 00:00 2023-03-03T05:42:23.739 INFO:teuthology.orchestra.run.smithi172.stdout:(73/140): libaio-devel-0.3.112-1.el8.x86_64.rpm 751 kB/s | 19 kB 00:00 2023-03-03T05:42:23.750 INFO:teuthology.orchestra.run.smithi102.stdout:(22/140): python3-rbd-18.0.0-2739.g472aaef4.el8 6.1 MB/s | 364 kB 00:00 2023-03-03T05:42:23.789 INFO:teuthology.orchestra.run.smithi172.stdout:(74/140): libcom_err-devel-1.45.6-5.el8.x86_64. 778 kB/s | 39 kB 00:00 2023-03-03T05:42:23.801 INFO:teuthology.orchestra.run.smithi102.stdout:(23/140): python3-rgw-18.0.0-2739.g472aaef4.el8 2.1 MB/s | 107 kB 00:00 2023-03-03T05:42:23.851 INFO:teuthology.orchestra.run.smithi102.stdout:(24/140): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 1.8 MB/s | 94 kB 00:00 2023-03-03T05:42:23.965 INFO:teuthology.orchestra.run.smithi172.stdout:(75/140): python3-scipy-1.0.0-21.module_el8.5.0 22 MB/s | 14 MB 00:00 2023-03-03T05:42:23.998 INFO:teuthology.orchestra.run.smithi172.stdout:(76/140): libgfortran-8.5.0-18.el8.x86_64.rpm 3.0 MB/s | 645 kB 00:00 2023-03-03T05:42:24.023 INFO:teuthology.orchestra.run.smithi172.stdout:(77/140): libquadmath-8.5.0-18.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2023-03-03T05:42:24.048 INFO:teuthology.orchestra.run.smithi172.stdout:(78/140): librabbitmq-0.9.0-3.el8.x86_64.rpm 935 kB/s | 47 kB 00:00 2023-03-03T05:42:24.093 INFO:teuthology.orchestra.run.smithi102.stdout:(25/140): rbd-mirror-18.0.0-2739.g472aaef4.el8. 14 MB/s | 3.4 MB 00:00 2023-03-03T05:42:24.135 INFO:teuthology.orchestra.run.smithi102.stdout:(26/140): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 4.2 MB/s | 180 kB 00:00 2023-03-03T05:42:24.169 INFO:teuthology.orchestra.run.smithi102.stdout:(27/140): ceph-grafana-dashboards-18.0.0-2739.g 696 kB/s | 23 kB 00:00 2023-03-03T05:42:24.219 INFO:teuthology.orchestra.run.smithi102.stdout:(28/140): ceph-mgr-cephadm-18.0.0-2739.g472aaef 2.4 MB/s | 123 kB 00:00 2023-03-03T05:42:24.353 INFO:teuthology.orchestra.run.smithi102.stdout:(29/140): ceph-mgr-dashboard-18.0.0-2739.g472aa 13 MB/s | 1.7 MB 00:00 2023-03-03T05:42:24.516 INFO:teuthology.orchestra.run.smithi172.stdout:(79/140): ceph-test-18.0.0-2739.g472aaef4.el8.x 7.9 MB/s | 47 MB 00:05 2023-03-03T05:42:24.541 INFO:teuthology.orchestra.run.smithi172.stdout:(80/140): libsepol-devel-2.9-3.el8.x86_64.rpm 177 kB/s | 87 kB 00:00 2023-03-03T05:42:24.574 INFO:teuthology.orchestra.run.smithi172.stdout:(81/140): libselinux-devel-2.9-6.el8.x86_64.rpm 363 kB/s | 200 kB 00:00 2023-03-03T05:42:24.600 INFO:teuthology.orchestra.run.smithi172.stdout:(82/140): libverto-devel-0.3.2-2.el8.x86_64.rpm 332 kB/s | 18 kB 00:00 2023-03-03T05:42:24.625 INFO:teuthology.orchestra.run.smithi172.stdout:(83/140): libuuid-devel-2.32.1-40.el8.x86_64.rp 912 kB/s | 99 kB 00:00 2023-03-03T05:42:24.658 INFO:teuthology.orchestra.run.smithi172.stdout:(84/140): pcre2-utf16-10.32-3.el8.x86_64.rpm 6.7 MB/s | 229 kB 00:00 2023-03-03T05:42:24.730 INFO:teuthology.orchestra.run.smithi172.stdout:(85/140): pcre2-utf32-10.32-3.el8.x86_64.rpm 3.0 MB/s | 220 kB 00:00 2023-03-03T05:42:24.767 INFO:teuthology.orchestra.run.smithi172.stdout:(86/140): python3-cffi-1.11.5-5.el8.x86_64.rpm 6.3 MB/s | 237 kB 00:00 2023-03-03T05:42:24.795 INFO:teuthology.orchestra.run.smithi102.stdout:(30/140): ceph-mgr-diskprediction-local-18.0.0- 17 MB/s | 7.4 MB 00:00 2023-03-03T05:42:24.845 INFO:teuthology.orchestra.run.smithi102.stdout:(31/140): ceph-mgr-modules-core-18.0.0-2739.g47 4.7 MB/s | 236 kB 00:00 2023-03-03T05:42:24.876 INFO:teuthology.orchestra.run.smithi172.stdout:(87/140): pcre2-devel-10.32-3.el8.x86_64.rpm 2.1 MB/s | 605 kB 00:00 2023-03-03T05:42:24.887 INFO:teuthology.orchestra.run.smithi102.stdout:(32/140): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 1.1 MB/s | 47 kB 00:00 2023-03-03T05:42:24.910 INFO:teuthology.orchestra.run.smithi172.stdout:(88/140): python3-cryptography-3.2.1-5.el8.x86_ 3.8 MB/s | 559 kB 00:00 2023-03-03T05:42:24.929 INFO:teuthology.orchestra.run.smithi102.stdout:(33/140): ceph-prometheus-alerts-18.0.0-2739.g4 349 kB/s | 15 kB 00:00 2023-03-03T05:42:24.935 INFO:teuthology.orchestra.run.smithi172.stdout:(89/140): python3-jwt-1.6.1-2.el8.noarch.rpm 730 kB/s | 43 kB 00:00 2023-03-03T05:42:24.960 INFO:teuthology.orchestra.run.smithi172.stdout:(90/140): python3-pycparser-2.14-14.el8.noarch. 4.3 MB/s | 109 kB 00:00 2023-03-03T05:42:24.986 INFO:teuthology.orchestra.run.smithi172.stdout:(91/140): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2023-03-03T05:42:24.987 INFO:teuthology.orchestra.run.smithi102.stdout:(34/140): ceph-volume-18.0.0-2739.g472aaef4.el8 4.4 MB/s | 263 kB 00:00 2023-03-03T05:42:25.002 INFO:teuthology.orchestra.run.smithi172.stdout:(92/140): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-03-03T05:42:25.036 INFO:teuthology.orchestra.run.smithi172.stdout:(93/140): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.4 MB/s | 72 kB 00:00 2023-03-03T05:42:25.038 INFO:teuthology.orchestra.run.smithi102.stdout:(35/140): cephadm-18.0.0-2739.g472aaef4.el8.noa 1.6 MB/s | 83 kB 00:00 2023-03-03T05:42:25.053 INFO:teuthology.orchestra.run.smithi172.stdout:(94/140): zlib-devel-1.2.11-19.el8.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-03-03T05:42:25.089 INFO:teuthology.orchestra.run.smithi102.stdout:(36/140): cephfs-top-18.0.0-2739.g472aaef4.el8. 363 kB/s | 18 kB 00:00 2023-03-03T05:42:25.129 INFO:teuthology.orchestra.run.smithi172.stdout:(95/140): openssl-devel-1.1.1k-7.el8.x86_64.rpm 4.2 MB/s | 2.3 MB 00:00 2023-03-03T05:42:25.164 INFO:teuthology.orchestra.run.smithi102.stdout:(37/140): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-03-03T05:42:25.206 INFO:teuthology.orchestra.run.smithi102.stdout:(38/140): bison-3.0.4-10.el8.x86_64.rpm 16 MB/s | 688 kB 00:00 2023-03-03T05:42:25.240 INFO:teuthology.orchestra.run.smithi102.stdout:(39/140): flex-2.6.1-9.el8.x86_64.rpm 9.4 MB/s | 320 kB 00:00 2023-03-03T05:42:25.265 INFO:teuthology.orchestra.run.smithi102.stdout:(40/140): jq-1.6-6.el8.x86_64.rpm 8.0 MB/s | 202 kB 00:00 2023-03-03T05:42:25.279 INFO:teuthology.orchestra.run.smithi172.stdout:(96/140): liboath-2.6.2-3.el8.x86_64.rpm 261 kB/s | 59 kB 00:00 2023-03-03T05:42:25.291 INFO:teuthology.orchestra.run.smithi102.stdout:(41/140): libpq-13.5-1.el8.x86_64.rpm 7.8 MB/s | 198 kB 00:00 2023-03-03T05:42:25.333 INFO:teuthology.orchestra.run.smithi102.stdout:(42/140): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-03-03T05:42:25.337 INFO:teuthology.orchestra.run.smithi172.stdout:(97/140): libunwind-1.3.1-3.el8.x86_64.rpm 360 kB/s | 75 kB 00:00 2023-03-03T05:42:25.374 INFO:teuthology.orchestra.run.smithi102.stdout:(43/140): libtool-2.4.6-25.el8.x86_64.rpm 17 MB/s | 709 kB 00:00 2023-03-03T05:42:25.407 INFO:teuthology.orchestra.run.smithi102.stdout:(44/140): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.7 MB/s | 259 kB 00:00 2023-03-03T05:42:25.413 INFO:teuthology.orchestra.run.smithi172.stdout:(98/140): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 582 kB/s | 44 kB 00:00 2023-03-03T05:42:25.433 INFO:teuthology.orchestra.run.smithi102.stdout:(45/140): oniguruma-6.8.2-2.el8.x86_64.rpm 7.4 MB/s | 187 kB 00:00 2023-03-03T05:42:25.463 INFO:teuthology.orchestra.run.smithi172.stdout:(99/140): gperftools-libs-2.7-9.el8.x86_64.rpm 718 kB/s | 306 kB 00:00 2023-03-03T05:42:25.530 INFO:teuthology.orchestra.run.smithi172.stdout:(100/140): python3-asyncssh-2.7.0-2.el8.noarch. 1.9 MB/s | 476 kB 00:00 2023-03-03T05:42:25.555 INFO:teuthology.orchestra.run.smithi172.stdout:(101/140): python3-cachetools-3.1.1-4.el8.noarc 361 kB/s | 33 kB 00:00 2023-03-03T05:42:25.567 INFO:teuthology.orchestra.run.smithi102.stdout:(46/140): openblas-0.3.15-4.el8.x86_64.rpm 36 MB/s | 4.7 MB 00:00 2023-03-03T05:42:25.589 INFO:teuthology.orchestra.run.smithi172.stdout:(102/140): python3-beautifulsoup4-4.6.3-2.el8.1 1.0 MB/s | 185 kB 00:00 2023-03-03T05:42:25.619 INFO:teuthology.orchestra.run.smithi172.stdout:(103/140): python3-certifi-2018.10.15-7.el8.noa 195 kB/s | 16 kB 00:00 2023-03-03T05:42:25.648 INFO:teuthology.orchestra.run.smithi172.stdout:(104/140): python3-cheroot-8.5.2-1.el8.noarch.r 1.8 MB/s | 173 kB 00:00 2023-03-03T05:42:25.673 INFO:teuthology.orchestra.run.smithi172.stdout:(105/140): python3-google-auth-1.1.1-10.el8.noa 1.4 MB/s | 81 kB 00:00 2023-03-03T05:42:25.683 INFO:teuthology.orchestra.run.smithi102.stdout:(47/140): openblas-threads-0.3.15-4.el8.x86_64. 42 MB/s | 4.9 MB 00:00 2023-03-03T05:42:25.715 INFO:teuthology.orchestra.run.smithi172.stdout:(106/140): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-03-03T05:42:25.740 INFO:teuthology.orchestra.run.smithi172.stdout:(107/140): python3-jaraco-functools-2.0-4.el8.n 263 kB/s | 18 kB 00:00 2023-03-03T05:42:25.742 INFO:teuthology.orchestra.run.smithi102.stdout:(48/140): postgresql-10.19-1.module_el8.6.0+104 26 MB/s | 1.5 MB 00:00 2023-03-03T05:42:25.782 INFO:teuthology.orchestra.run.smithi172.stdout:(108/140): python3-cherrypy-18.4.0-1.el8.noarch 1.9 MB/s | 384 kB 00:00 2023-03-03T05:42:25.793 INFO:teuthology.orchestra.run.smithi102.stdout:(49/140): postgresql-contrib-10.19-1.module_el8 16 MB/s | 807 kB 00:00 2023-03-03T05:42:25.842 INFO:teuthology.orchestra.run.smithi172.stdout:(109/140): python3-logutils-0.3.5-11.el8.noarch 478 kB/s | 49 kB 00:00 2023-03-03T05:42:25.874 INFO:teuthology.orchestra.run.smithi172.stdout:(110/140): python3-more-itertools-7.2.0-3.el8.n 640 kB/s | 59 kB 00:00 2023-03-03T05:42:25.899 INFO:teuthology.orchestra.run.smithi172.stdout:(111/140): python3-natsort-7.1.1-2.el8.noarch.r 1.0 MB/s | 60 kB 00:00 2023-03-03T05:42:25.918 INFO:teuthology.orchestra.run.smithi102.stdout:(50/140): postgresql-server-10.19-1.module_el8. 41 MB/s | 5.1 MB 00:00 2023-03-03T05:42:25.967 INFO:teuthology.orchestra.run.smithi172.stdout:(112/140): python3-kubernetes-11.0.0-6.el8.noar 4.6 MB/s | 1.2 MB 00:00 2023-03-03T05:42:25.992 INFO:teuthology.orchestra.run.smithi172.stdout:(113/140): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-03-03T05:42:26.035 INFO:teuthology.orchestra.run.smithi172.stdout:(114/140): python3-repoze-lru-0.7-6.el8.noarch. 503 kB/s | 34 kB 00:00 2023-03-03T05:42:26.059 INFO:teuthology.orchestra.run.smithi172.stdout:(115/140): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2023-03-03T05:42:26.069 INFO:teuthology.orchestra.run.smithi102.stdout:(51/140): python3-babel-2.5.1-7.el8.noarch.rpm 32 MB/s | 4.8 MB 00:00 2023-03-03T05:42:26.084 INFO:teuthology.orchestra.run.smithi172.stdout:(116/140): python3-routes-2.4.1-12.el8.noarch.r 2.1 MB/s | 196 kB 00:00 2023-03-03T05:42:26.102 INFO:teuthology.orchestra.run.smithi102.stdout:(52/140): python3-jinja2-2.10.1-3.el8.noarch.rp 16 MB/s | 538 kB 00:00 2023-03-03T05:42:26.113 INFO:teuthology.orchestra.run.smithi172.stdout:(117/140): python3-rsa-4.8-1.el8.noarch.rpm 816 kB/s | 61 kB 00:00 2023-03-03T05:42:26.129 INFO:teuthology.orchestra.run.smithi102.stdout:(53/140): python3-jsonpatch-1.21-2.el8.noarch.r 1.0 MB/s | 27 kB 00:00 2023-03-03T05:42:26.135 INFO:teuthology.orchestra.run.smithi172.stdout:(118/140): python3-simplegeneric-0.8.1-17.el8.n 248 kB/s | 19 kB 00:00 2023-03-03T05:42:26.160 INFO:teuthology.orchestra.run.smithi172.stdout:(119/140): python3-singledispatch-3.4.0.3-18.el 317 kB/s | 24 kB 00:00 2023-03-03T05:42:26.219 INFO:teuthology.orchestra.run.smithi172.stdout:(120/140): python3-tempora-1.14.1-5.el8.noarch. 268 kB/s | 29 kB 00:00 2023-03-03T05:42:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:(121/140): python3-trustme-0.6.0-4.el8.noarch.r 253 kB/s | 27 kB 00:00 2023-03-03T05:42:26.269 INFO:teuthology.orchestra.run.smithi172.stdout:(122/140): python3-waitress-1.2.1-2.el8.1.noarc 2.2 MB/s | 241 kB 00:00 2023-03-03T05:42:26.270 INFO:teuthology.orchestra.run.smithi102.stdout:(54/140): ceph-radosgw-18.0.0-2739.g472aaef4.el 3.2 MB/s | 13 MB 00:04 2023-03-03T05:42:26.294 INFO:teuthology.orchestra.run.smithi172.stdout:(123/140): python3-websocket-client-0.56.0-5.el 1.2 MB/s | 61 kB 00:00 2023-03-03T05:42:26.295 INFO:teuthology.orchestra.run.smithi102.stdout:(55/140): python3-jsonpointer-1.10-11.el8.noarc 119 kB/s | 20 kB 00:00 2023-03-03T05:42:26.320 INFO:teuthology.orchestra.run.smithi102.stdout:(56/140): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2023-03-03T05:42:26.320 INFO:teuthology.orchestra.run.smithi172.stdout:(124/140): python3-webob-1.8.5-1.el8.1.noarch.r 2.4 MB/s | 251 kB 00:00 2023-03-03T05:42:26.345 INFO:teuthology.orchestra.run.smithi172.stdout:(125/140): python3-webtest-2.0.33-1.el8.noarch. 1.1 MB/s | 85 kB 00:00 2023-03-03T05:42:26.370 INFO:teuthology.orchestra.run.smithi172.stdout:(126/140): python3-zc-lockfile-2.0-2.el8.noarch 303 kB/s | 23 kB 00:00 2023-03-03T05:42:26.387 INFO:teuthology.orchestra.run.smithi102.stdout:(57/140): python3-markupsafe-0.23-19.el8.x86_64 423 kB/s | 39 kB 00:00 2023-03-03T05:42:26.420 INFO:teuthology.orchestra.run.smithi102.stdout:(58/140): python3-numpy-f2py-1.14.3-10.el8.x86_ 6.6 MB/s | 225 kB 00:00 2023-03-03T05:42:26.446 INFO:teuthology.orchestra.run.smithi102.stdout:(59/140): python3-prettytable-0.7.2-14.el8.noar 1.7 MB/s | 44 kB 00:00 2023-03-03T05:42:26.513 INFO:teuthology.orchestra.run.smithi102.stdout:(60/140): python3-numpy-1.14.3-10.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2023-03-03T05:42:26.538 INFO:teuthology.orchestra.run.smithi102.stdout:(61/140): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.1 MB/s | 103 kB 00:00 2023-03-03T05:42:26.563 INFO:teuthology.orchestra.run.smithi102.stdout:(62/140): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-03-03T05:42:26.589 INFO:teuthology.orchestra.run.smithi102.stdout:(63/140): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-03-03T05:42:26.604 INFO:teuthology.orchestra.run.smithi172.stdout:(127/140): thrift-0.13.0-2.el8.x86_64.rpm 6.1 MB/s | 1.7 MB 00:00 2023-03-03T05:42:26.605 INFO:teuthology.orchestra.run.smithi102.stdout:(64/140): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-03-03T05:42:26.646 INFO:teuthology.orchestra.run.smithi172.stdout:(128/140): NetworkManager-initscripts-updown-1. 3.4 MB/s | 142 kB 00:00 2023-03-03T05:42:26.648 INFO:teuthology.orchestra.run.smithi102.stdout:(65/140): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-03-03T05:42:26.671 INFO:teuthology.orchestra.run.smithi172.stdout:(129/140): elfutils-debuginfod-client-0.188-3.e 3.0 MB/s | 75 kB 00:00 2023-03-03T05:42:26.681 INFO:teuthology.orchestra.run.smithi102.stdout:(66/140): socat-1.7.4.1-1.el8.x86_64.rpm 9.6 MB/s | 323 kB 00:00 2023-03-03T05:42:26.705 INFO:teuthology.orchestra.run.smithi172.stdout:(130/140): elfutils-libelf-0.188-3.el8.x86_64.r 6.8 MB/s | 231 kB 00:00 2023-03-03T05:42:26.706 INFO:teuthology.orchestra.run.smithi102.stdout:(67/140): uuid-1.6.2-43.el8.x86_64.rpm 2.5 MB/s | 64 kB 00:00 2023-03-03T05:42:26.732 INFO:teuthology.orchestra.run.smithi102.stdout:(68/140): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-03-03T05:42:26.757 INFO:teuthology.orchestra.run.smithi102.stdout:(69/140): elfutils-libelf-devel-0.188-3.el8.x86 2.4 MB/s | 61 kB 00:00 2023-03-03T05:42:26.763 INFO:teuthology.orchestra.run.smithi172.stdout:(131/140): librados2-18.0.0-2739.g472aaef4.el8. 8.7 MB/s | 3.7 MB 00:00 2023-03-03T05:42:26.783 INFO:teuthology.orchestra.run.smithi102.stdout:(70/140): fuse-2.9.7-16.el8.x86_64.rpm 3.3 MB/s | 83 kB 00:00 2023-03-03T05:42:26.788 INFO:teuthology.orchestra.run.smithi172.stdout:(132/140): elfutils-libs-0.188-3.el8.x86_64.rpm 3.5 MB/s | 299 kB 00:00 2023-03-03T05:42:26.808 INFO:teuthology.orchestra.run.smithi102.stdout:(71/140): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.9 MB/s | 48 kB 00:00 2023-03-03T05:42:26.814 INFO:teuthology.orchestra.run.smithi172.stdout:(133/140): libblkid-2.32.1-40.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-03T05:42:26.858 INFO:teuthology.orchestra.run.smithi102.stdout:(72/140): krb5-devel-1.18.2-22.el8.x86_64.rpm 11 MB/s | 561 kB 00:00 2023-03-03T05:42:26.872 INFO:teuthology.orchestra.run.smithi172.stdout:(134/140): librbd1-18.0.0-2739.g472aaef4.el8.x8 7.1 MB/s | 3.6 MB 00:00 2023-03-03T05:42:26.883 INFO:teuthology.orchestra.run.smithi102.stdout:(73/140): libaio-devel-0.3.112-1.el8.x86_64.rpm 752 kB/s | 19 kB 00:00 2023-03-03T05:42:26.898 INFO:teuthology.orchestra.run.smithi172.stdout:(135/140): libfdisk-2.32.1-40.el8.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2023-03-03T05:42:26.909 INFO:teuthology.orchestra.run.smithi102.stdout:(74/140): libcom_err-devel-1.45.6-5.el8.x86_64. 1.5 MB/s | 39 kB 00:00 2023-03-03T05:42:26.947 INFO:teuthology.orchestra.run.smithi172.stdout:(136/140): libmount-2.32.1-40.el8.x86_64.rpm 1.8 MB/s | 236 kB 00:00 2023-03-03T05:42:26.951 INFO:teuthology.orchestra.run.smithi102.stdout:(75/140): libgfortran-8.5.0-18.el8.x86_64.rpm 15 MB/s | 645 kB 00:00 2023-03-03T05:42:26.982 INFO:teuthology.orchestra.run.smithi172.stdout:(137/140): libsmartcols-2.32.1-40.el8.x86_64.rp 1.6 MB/s | 179 kB 00:00 2023-03-03T05:42:26.985 INFO:teuthology.orchestra.run.smithi102.stdout:(76/140): libquadmath-8.5.0-18.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2023-03-03T05:42:27.009 INFO:teuthology.orchestra.run.smithi172.stdout:(138/140): libuuid-2.32.1-40.el8.x86_64.rpm 904 kB/s | 98 kB 00:00 2023-03-03T05:42:27.010 INFO:teuthology.orchestra.run.smithi102.stdout:(77/140): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2023-03-03T05:42:27.032 INFO:teuthology.orchestra.run.smithi172.stdout:(139/140): util-linux-user-2.32.1-40.el8.x86_64 2.0 MB/s | 102 kB 00:00 2023-03-03T05:42:27.044 INFO:teuthology.orchestra.run.smithi102.stdout:(78/140): libselinux-devel-2.9-6.el8.x86_64.rpm 5.9 MB/s | 200 kB 00:00 2023-03-03T05:42:27.069 INFO:teuthology.orchestra.run.smithi102.stdout:(79/140): libsepol-devel-2.9-3.el8.x86_64.rpm 3.4 MB/s | 87 kB 00:00 2023-03-03T05:42:27.094 INFO:teuthology.orchestra.run.smithi102.stdout:(80/140): libuuid-devel-2.32.1-40.el8.x86_64.rp 3.9 MB/s | 99 kB 00:00 2023-03-03T05:42:27.099 INFO:teuthology.orchestra.run.smithi172.stdout:(140/140): util-linux-2.32.1-40.el8.x86_64.rpm 16 MB/s | 2.5 MB 00:00 2023-03-03T05:42:27.105 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-03-03T05:42:27.106 INFO:teuthology.orchestra.run.smithi172.stdout:Total 19 MB/s | 206 MB 00:10 2023-03-03T05:42:27.120 INFO:teuthology.orchestra.run.smithi102.stdout:(81/140): libverto-devel-0.3.2-2.el8.x86_64.rpm 729 kB/s | 18 kB 00:00 2023-03-03T05:42:27.212 INFO:teuthology.orchestra.run.smithi102.stdout:(82/140): openssl-devel-1.1.1k-7.el8.x86_64.rpm 25 MB/s | 2.3 MB 00:00 2023-03-03T05:42:27.254 INFO:teuthology.orchestra.run.smithi102.stdout:(83/140): pcre2-devel-10.32-3.el8.x86_64.rpm 14 MB/s | 605 kB 00:00 2023-03-03T05:42:27.288 INFO:teuthology.orchestra.run.smithi102.stdout:(84/140): pcre2-utf16-10.32-3.el8.x86_64.rpm 6.7 MB/s | 229 kB 00:00 2023-03-03T05:42:27.323 INFO:teuthology.orchestra.run.smithi102.stdout:(85/140): pcre2-utf32-10.32-3.el8.x86_64.rpm 6.6 MB/s | 220 kB 00:00 2023-03-03T05:42:27.355 INFO:teuthology.orchestra.run.smithi102.stdout:(86/140): python3-cffi-1.11.5-5.el8.x86_64.rpm 7.0 MB/s | 237 kB 00:00 2023-03-03T05:42:27.397 INFO:teuthology.orchestra.run.smithi102.stdout:(87/140): python3-cryptography-3.2.1-5.el8.x86_ 13 MB/s | 559 kB 00:00 2023-03-03T05:42:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:(88/140): python3-jwt-1.6.1-2.el8.noarch.rpm 1.7 MB/s | 43 kB 00:00 2023-03-03T05:42:27.449 INFO:teuthology.orchestra.run.smithi102.stdout:(89/140): python3-oauthlib-2.1.0-1.el8.noarch.r 6.1 MB/s | 155 kB 00:00 2023-03-03T05:42:27.539 INFO:teuthology.orchestra.run.smithi102.stdout:(90/140): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2023-03-03T05:42:27.565 INFO:teuthology.orchestra.run.smithi102.stdout:(91/140): python3-requests-oauthlib-1.0.0-1.el8 1.7 MB/s | 43 kB 00:00 2023-03-03T05:42:27.590 INFO:teuthology.orchestra.run.smithi102.stdout:(92/140): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 2.8 MB/s | 72 kB 00:00 2023-03-03T05:42:27.616 INFO:teuthology.orchestra.run.smithi102.stdout:(93/140): zlib-devel-1.2.11-19.el8.x86_64.rpm 2.3 MB/s | 58 kB 00:00 2023-03-03T05:42:27.799 INFO:teuthology.orchestra.run.smithi102.stdout:(94/140): python3-scipy-1.0.0-21.module_el8.5.0 11 MB/s | 14 MB 00:01 2023-03-03T05:42:27.908 INFO:teuthology.orchestra.run.smithi102.stdout:(95/140): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2023-03-03T05:42:27.933 INFO:teuthology.orchestra.run.smithi102.stdout:(96/140): liboath-2.6.2-3.el8.x86_64.rpm 442 kB/s | 59 kB 00:00 2023-03-03T05:42:27.958 INFO:teuthology.orchestra.run.smithi102.stdout:(97/140): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-03-03T05:42:27.982 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-03-03T05:42:28.000 INFO:teuthology.orchestra.run.smithi102.stdout:(98/140): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2023-03-03T05:42:28.059 INFO:teuthology.orchestra.run.smithi102.stdout:(99/140): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2023-03-03T05:42:28.084 INFO:teuthology.orchestra.run.smithi102.stdout:(100/140): python3-asyncssh-2.7.0-2.el8.noarch. 3.1 MB/s | 476 kB 00:00 2023-03-03T05:42:28.118 INFO:teuthology.orchestra.run.smithi102.stdout:(101/140): python3-cachetools-3.1.1-4.el8.noarc 564 kB/s | 33 kB 00:00 2023-03-03T05:42:28.143 INFO:teuthology.orchestra.run.smithi102.stdout:(102/140): python3-certifi-2018.10.15-7.el8.noa 279 kB/s | 16 kB 00:00 2023-03-03T05:42:28.177 INFO:teuthology.orchestra.run.smithi102.stdout:(103/140): python3-cheroot-8.5.2-1.el8.noarch.r 2.9 MB/s | 173 kB 00:00 2023-03-03T05:42:28.214 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-03-03T05:42:28.214 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-03-03T05:42:28.227 INFO:teuthology.orchestra.run.smithi102.stdout:(104/140): python3-cherrypy-18.4.0-1.el8.noarch 4.5 MB/s | 384 kB 00:00 2023-03-03T05:42:28.253 INFO:teuthology.orchestra.run.smithi102.stdout:(105/140): python3-google-auth-1.1.1-10.el8.noa 1.1 MB/s | 81 kB 00:00 2023-03-03T05:42:28.278 INFO:teuthology.orchestra.run.smithi102.stdout:(106/140): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-03-03T05:42:28.303 INFO:teuthology.orchestra.run.smithi102.stdout:(107/140): python3-jaraco-functools-2.0-4.el8.n 350 kB/s | 18 kB 00:00 2023-03-03T05:42:28.353 INFO:teuthology.orchestra.run.smithi102.stdout:(108/140): python3-logutils-0.3.5-11.el8.noarch 982 kB/s | 49 kB 00:00 2023-03-03T05:42:28.404 INFO:teuthology.orchestra.run.smithi102.stdout:(109/140): python3-more-itertools-7.2.0-3.el8.n 1.1 MB/s | 59 kB 00:00 2023-03-03T05:42:28.454 INFO:teuthology.orchestra.run.smithi102.stdout:(110/140): python3-natsort-7.1.1-2.el8.noarch.r 1.2 MB/s | 60 kB 00:00 2023-03-03T05:42:28.521 INFO:teuthology.orchestra.run.smithi102.stdout:(111/140): python3-kubernetes-11.0.0-6.el8.noar 4.8 MB/s | 1.2 MB 00:00 2023-03-03T05:42:28.546 INFO:teuthology.orchestra.run.smithi102.stdout:(112/140): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-03-03T05:42:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:(113/140): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-03-03T05:42:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:(114/140): python3-repoze-lru-0.7-6.el8.noarch. 671 kB/s | 34 kB 00:00 2023-03-03T05:42:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:(115/140): python3-routes-2.4.1-12.el8.noarch.r 3.3 MB/s | 196 kB 00:00 2023-03-03T05:42:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:(116/140): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2023-03-03T05:42:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:(117/140): python3-simplegeneric-0.8.1-17.el8.n 372 kB/s | 19 kB 00:00 2023-03-03T05:42:28.706 INFO:teuthology.orchestra.run.smithi102.stdout:(118/140): python3-singledispatch-3.4.0.3-18.el 479 kB/s | 24 kB 00:00 2023-03-03T05:42:28.740 INFO:teuthology.orchestra.run.smithi102.stdout:(119/140): python3-tempora-1.14.1-5.el8.noarch. 502 kB/s | 29 kB 00:00 2023-03-03T05:42:28.770 INFO:teuthology.orchestra.run.smithi102.stdout:(120/140): python3-trustme-0.6.0-4.el8.noarch.r 468 kB/s | 27 kB 00:00 2023-03-03T05:42:28.799 INFO:teuthology.orchestra.run.smithi102.stdout:(121/140): python3-waitress-1.2.1-2.el8.1.noarc 4.0 MB/s | 241 kB 00:00 2023-03-03T05:42:28.849 INFO:teuthology.orchestra.run.smithi102.stdout:(122/140): python3-webob-1.8.5-1.el8.1.noarch.r 2.9 MB/s | 251 kB 00:00 2023-03-03T05:42:28.875 INFO:teuthology.orchestra.run.smithi102.stdout:(123/140): python3-websocket-client-0.56.0-5.el 815 kB/s | 61 kB 00:00 2023-03-03T05:42:28.910 INFO:teuthology.orchestra.run.smithi102.stdout:(124/140): python3-webtest-2.0.33-1.el8.noarch. 1.4 MB/s | 85 kB 00:00 2023-03-03T05:42:28.934 INFO:teuthology.orchestra.run.smithi102.stdout:(125/140): python3-zc-lockfile-2.0-2.el8.noarch 392 kB/s | 23 kB 00:00 2023-03-03T05:42:29.167 INFO:teuthology.orchestra.run.smithi102.stdout:(126/140): thrift-0.13.0-2.el8.x86_64.rpm 6.7 MB/s | 1.7 MB 00:00 2023-03-03T05:42:29.518 INFO:teuthology.orchestra.run.smithi102.stdout:(127/140): librbd1-18.0.0-2739.g472aaef4.el8.x8 10 MB/s | 3.6 MB 00:00 2023-03-03T05:42:29.551 INFO:teuthology.orchestra.run.smithi102.stdout:(128/140): NetworkManager-initscripts-updown-1. 4.2 MB/s | 142 kB 00:00 2023-03-03T05:42:29.576 INFO:teuthology.orchestra.run.smithi102.stdout:(129/140): elfutils-debuginfod-client-0.188-3.e 3.0 MB/s | 75 kB 00:00 2023-03-03T05:42:29.602 INFO:teuthology.orchestra.run.smithi102.stdout:(130/140): elfutils-libelf-0.188-3.el8.x86_64.r 9.0 MB/s | 231 kB 00:00 2023-03-03T05:42:29.636 INFO:teuthology.orchestra.run.smithi102.stdout:(131/140): elfutils-libs-0.188-3.el8.x86_64.rpm 8.9 MB/s | 299 kB 00:00 2023-03-03T05:42:29.668 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-03-03T05:42:29.669 INFO:teuthology.orchestra.run.smithi102.stdout:(132/140): libblkid-2.32.1-40.el8.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2023-03-03T05:42:29.672 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-03-03T05:42:29.728 INFO:teuthology.orchestra.run.smithi102.stdout:(133/140): librados2-18.0.0-2739.g472aaef4.el8. 4.6 MB/s | 3.7 MB 00:00 2023-03-03T05:42:29.762 INFO:teuthology.orchestra.run.smithi102.stdout:(134/140): libfdisk-2.32.1-40.el8.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2023-03-03T05:42:29.787 INFO:teuthology.orchestra.run.smithi102.stdout:(135/140): libmount-2.32.1-40.el8.x86_64.rpm 4.0 MB/s | 236 kB 00:00 2023-03-03T05:42:29.820 INFO:teuthology.orchestra.run.smithi102.stdout:(136/140): libsmartcols-2.32.1-40.el8.x86_64.rp 3.0 MB/s | 179 kB 00:00 2023-03-03T05:42:29.846 INFO:teuthology.orchestra.run.smithi102.stdout:(137/140): libuuid-2.32.1-40.el8.x86_64.rpm 1.6 MB/s | 98 kB 00:00 2023-03-03T05:42:29.871 INFO:teuthology.orchestra.run.smithi102.stdout:(138/140): util-linux-user-2.32.1-40.el8.x86_64 4.0 MB/s | 102 kB 00:00 2023-03-03T05:42:29.938 INFO:teuthology.orchestra.run.smithi102.stdout:(139/140): util-linux-2.32.1-40.el8.x86_64.rpm 21 MB/s | 2.5 MB 00:00 2023-03-03T05:42:30.539 INFO:teuthology.orchestra.run.smithi102.stdout:(140/140): ceph-test-18.0.0-2739.g472aaef4.el8. 5.6 MB/s | 47 MB 00:08 2023-03-03T05:42:30.543 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2023-03-03T05:42:30.544 INFO:teuthology.orchestra.run.smithi102.stdout:Total 16 MB/s | 206 MB 00:12 2023-03-03T05:42:33.646 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2023-03-03T05:42:33.938 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2023-03-03T05:42:33.939 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2023-03-03T05:42:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-03-03T05:42:35.552 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: thrift-0.13.0-2.el8.x86_64 1/1 2023-03-03T05:42:35.557 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2023-03-03T05:42:35.562 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2023-03-03T05:42:36.030 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/153 2023-03-03T05:42:36.159 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T05:42:36.247 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T05:42:36.417 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T05:42:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T05:42:36.705 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/153 2023-03-03T05:42:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/153 2023-03-03T05:42:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T05:42:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T05:42:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T05:42:38.158 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T05:42:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T05:42:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T05:42:38.641 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T05:42:38.720 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T05:42:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T05:42:39.108 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T05:42:39.200 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T05:42:39.470 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-libelf-0.188-3.el8.x86_64 12/153 2023-03-03T05:42:39.729 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/153 2023-03-03T05:42:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 14/153 2023-03-03T05:42:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpq-13.5-1.el8.x86_64 15/153 2023-03-03T05:42:40.570 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 16/153 2023-03-03T05:42:40.766 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T05:42:40.868 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T05:42:40.903 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2023-03-03T05:42:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T05:42:41.125 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T05:42:41.145 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: thrift-0.13.0-2.el8.x86_64 1/1 2023-03-03T05:42:41.351 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/153 2023-03-03T05:42:41.498 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T05:42:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/153 2023-03-03T05:42:41.628 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libuuid-2.32.1-40.el8.x86_64 2/153 2023-03-03T05:42:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : zlib-devel-1.2.11-19.el8.x86_64 20/153 2023-03-03T05:42:41.782 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T05:42:41.874 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T05:42:41.916 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libblkid-2.32.1-40.el8.x86_64 3/153 2023-03-03T05:42:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T05:42:42.104 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/153 2023-03-03T05:42:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T05:42:42.319 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/153 2023-03-03T05:42:42.515 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T05:42:42.793 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T05:42:42.996 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/153 2023-03-03T05:42:42.999 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T05:42:43.165 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T05:42:43.599 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T05:42:43.642 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T05:42:43.774 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T05:42:43.815 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T05:42:43.983 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T05:42:44.182 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T05:42:44.327 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T05:42:44.466 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T05:42:44.527 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T05:42:44.673 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T05:42:44.823 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 11/153 2023-03-03T05:42:44.842 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/153 2023-03-03T05:42:45.063 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libelf-0.188-3.el8.x86_64 12/153 2023-03-03T05:42:45.072 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/153 2023-03-03T05:42:45.286 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 27/153 2023-03-03T05:42:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/153 2023-03-03T05:42:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/153 2023-03-03T05:42:45.526 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 14/153 2023-03-03T05:42:45.714 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 29/153 2023-03-03T05:42:45.875 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 30/153 2023-03-03T05:42:46.012 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpq-13.5-1.el8.x86_64 15/153 2023-03-03T05:42:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 31/153 2023-03-03T05:42:46.262 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 16/153 2023-03-03T05:42:46.464 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T05:42:46.619 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 17/153 2023-03-03T05:42:46.756 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T05:42:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T05:42:47.169 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/153 2023-03-03T05:42:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/153 2023-03-03T05:42:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : zlib-devel-1.2.11-19.el8.x86_64 20/153 2023-03-03T05:42:47.489 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T05:42:47.629 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/153 2023-03-03T05:42:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 21/153 2023-03-03T05:42:47.816 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T05:42:47.995 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/153 2023-03-03T05:42:48.248 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T05:42:48.556 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 22/153 2023-03-03T05:42:48.714 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T05:42:49.484 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/153 2023-03-03T05:42:49.647 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T05:42:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T05:42:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/153 2023-03-03T05:42:50.569 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-03-03T05:42:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/153 2023-03-03T05:42:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.8-1.el8.noarch 38/153 2023-03-03T05:42:50.949 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/153 2023-03-03T05:42:51.066 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-libs-0.188-3.el8.x86_64 39/153 2023-03-03T05:42:51.183 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/153 2023-03-03T05:42:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-debuginfod-client-0.188-3.el8.x86_64 40/153 2023-03-03T05:42:51.378 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 27/153 2023-03-03T05:42:51.486 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 41/153 2023-03-03T05:42:51.630 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/153 2023-03-03T05:42:51.702 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 42/153 2023-03-03T05:42:51.855 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 29/153 2023-03-03T05:42:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T05:42:51.993 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 30/153 2023-03-03T05:42:51.998 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T05:42:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T05:42:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T05:42:52.228 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 31/153 2023-03-03T05:42:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libuuid-devel-2.32.1-40.el8.x86_64 45/153 2023-03-03T05:42:52.531 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T05:42:53.018 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T05:42:53.340 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T05:42:53.354 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T05:42:53.426 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/153 2023-03-03T05:42:53.576 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/153 2023-03-03T05:42:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/153 2023-03-03T05:42:53.866 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/153 2023-03-03T05:42:54.078 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 50/153 2023-03-03T05:42:54.217 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/153 2023-03-03T05:42:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 51/153 2023-03-03T05:42:54.455 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T05:42:54.524 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 52/153 2023-03-03T05:42:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 53/153 2023-03-03T05:42:55.033 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 54/153 2023-03-03T05:42:55.273 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 55/153 2023-03-03T05:42:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/153 2023-03-03T05:42:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/153 2023-03-03T05:42:55.874 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/153 2023-03-03T05:42:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 59/153 2023-03-03T05:42:56.279 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-2.6-1.el8.noarch 60/153 2023-03-03T05:42:56.505 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 61/153 2023-03-03T05:42:56.710 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-03-03T05:42:56.756 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 62/153 2023-03-03T05:42:57.064 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/153 2023-03-03T05:42:57.080 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 63/153 2023-03-03T05:42:57.293 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/153 2023-03-03T05:42:57.334 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 64/153 2023-03-03T05:42:57.537 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.8-1.el8.noarch 38/153 2023-03-03T05:42:57.719 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libs-0.188-3.el8.x86_64 39/153 2023-03-03T05:42:57.730 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 65/153 2023-03-03T05:42:57.995 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-debuginfod-client-0.188-3.el8.x86_64 40/153 2023-03-03T05:42:58.052 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 66/153 2023-03-03T05:42:58.188 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 41/153 2023-03-03T05:42:58.303 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 67/153 2023-03-03T05:42:58.376 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 42/153 2023-03-03T05:42:58.589 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T05:42:58.710 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libfdisk-2.32.1-40.el8.x86_64 43/153 2023-03-03T05:42:58.887 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T05:42:58.986 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libmount-2.32.1-40.el8.x86_64 44/153 2023-03-03T05:42:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 68/153 2023-03-03T05:42:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libuuid-devel-2.32.1-40.el8.x86_64 45/153 2023-03-03T05:42:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T05:42:59.321 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T05:42:59.350 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T05:42:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 70/153 2023-03-03T05:42:59.873 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 71/153 2023-03-03T05:42:59.965 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libsmartcols-2.32.1-40.el8.x86_64 46/153 2023-03-03T05:43:00.255 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T05:43:00.307 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: util-linux-2.32.1-40.el8.x86_64 47/153 2023-03-03T05:43:00.313 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 72/153 2023-03-03T05:43:00.531 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/153 2023-03-03T05:43:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 73/153 2023-03-03T05:43:00.808 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/153 2023-03-03T05:43:00.918 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 74/153 2023-03-03T05:43:01.063 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 50/153 2023-03-03T05:43:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 75/153 2023-03-03T05:43:01.218 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 51/153 2023-03-03T05:43:01.434 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 76/153 2023-03-03T05:43:01.443 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 52/153 2023-03-03T05:43:01.714 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 53/153 2023-03-03T05:43:01.965 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 54/153 2023-03-03T05:43:02.172 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 55/153 2023-03-03T05:43:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/153 2023-03-03T05:43:02.577 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/153 2023-03-03T05:43:02.621 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/153 2023-03-03T05:43:02.846 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/153 2023-03-03T05:43:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 78/153 2023-03-03T05:43:03.054 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 59/153 2023-03-03T05:43:03.274 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-2.6-1.el8.noarch 60/153 2023-03-03T05:43:03.370 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 79/153 2023-03-03T05:43:03.543 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 61/153 2023-03-03T05:43:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 80/153 2023-03-03T05:43:03.812 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 62/153 2023-03-03T05:43:04.011 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 81/153 2023-03-03T05:43:04.090 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 63/153 2023-03-03T05:43:04.289 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/153 2023-03-03T05:43:04.357 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 64/153 2023-03-03T05:43:04.523 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/153 2023-03-03T05:43:04.748 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 84/153 2023-03-03T05:43:04.769 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 65/153 2023-03-03T05:43:05.034 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 66/153 2023-03-03T05:43:05.051 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 85/153 2023-03-03T05:43:05.261 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 67/153 2023-03-03T05:43:05.281 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 86/153 2023-03-03T05:43:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/153 2023-03-03T05:43:05.777 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/153 2023-03-03T05:43:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libselinux-devel-2.9-6.el8.x86_64 89/153 2023-03-03T05:43:06.158 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 68/153 2023-03-03T05:43:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/153 2023-03-03T05:43:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T05:43:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/153 2023-03-03T05:43:06.516 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 69/153 2023-03-03T05:43:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/153 2023-03-03T05:43:06.773 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 70/153 2023-03-03T05:43:07.086 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 71/153 2023-03-03T05:43:07.158 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.7-16.el8.x86_64 93/153 2023-03-03T05:43:07.560 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 72/153 2023-03-03T05:43:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/153 2023-03-03T05:43:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T05:43:07.838 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 73/153 2023-03-03T05:43:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T05:43:08.079 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 96/153 2023-03-03T05:43:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 74/153 2023-03-03T05:43:08.262 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 97/153 2023-03-03T05:43:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 75/153 2023-03-03T05:43:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 76/153 2023-03-03T05:43:10.480 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/153 2023-03-03T05:43:10.936 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 78/153 2023-03-03T05:43:11.239 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 79/153 2023-03-03T05:43:11.528 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 80/153 2023-03-03T05:43:11.746 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T05:43:11.879 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 81/153 2023-03-03T05:43:12.028 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T05:43:12.140 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/153 2023-03-03T05:43:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/153 2023-03-03T05:43:12.610 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T05:43:12.643 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 84/153 2023-03-03T05:43:12.991 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 85/153 2023-03-03T05:43:13.239 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 86/153 2023-03-03T05:43:13.432 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/153 2023-03-03T05:43:13.780 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/153 2023-03-03T05:43:13.856 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T05:43:13.999 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T05:43:14.033 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libselinux-devel-2.9-6.el8.x86_64 89/153 2023-03-03T05:43:14.194 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T05:43:14.506 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/153 2023-03-03T05:43:14.793 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/153 2023-03-03T05:43:15.040 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/153 2023-03-03T05:43:15.257 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.7-16.el8.x86_64 93/153 2023-03-03T05:43:15.438 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/153 2023-03-03T05:43:15.643 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T05:43:15.799 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 95/153 2023-03-03T05:43:15.980 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 96/153 2023-03-03T05:43:16.127 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 97/153 2023-03-03T05:43:21.464 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T05:43:21.686 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T05:43:22.421 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 98/153 2023-03-03T05:43:22.765 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T05:43:23.034 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 99/153 2023-03-03T05:43:23.226 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T05:43:33.366 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T05:43:33.366 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2023-03-03T05:43:33.366 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2023-03-03T05:43:33.366 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-03-03T05:43:33.367 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /run 2023-03-03T05:43:33.367 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2023-03-03T05:43:33.367 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2023-03-03T05:43:33.367 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2023-03-03T05:43:33.367 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2023-03-03T05:43:33.367 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-03-03T05:43:33.367 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:43:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T05:43:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T05:43:35.618 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T05:43:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T05:43:36.082 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T05:43:36.533 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T05:43:36.748 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T05:43:36.772 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T05:43:37.046 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 105/153 2023-03-03T05:43:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 106/153 2023-03-03T05:43:37.456 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T05:43:37.624 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T05:43:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-6.el8.x86_64 108/153 2023-03-03T05:43:38.200 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : automake-1.16.1-7.el8.noarch 109/153 2023-03-03T05:43:39.762 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T05:43:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T05:43:40.095 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 111/153 2023-03-03T05:43:40.312 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 112/153 2023-03-03T05:43:40.572 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T05:43:40.785 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T05:43:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T05:43:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T05:43:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T05:43:42.245 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T05:43:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 116/153 2023-03-03T05:43:42.867 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T05:43:42.903 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T05:43:43.122 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T05:43:43.133 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T05:43:43.380 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 119/153 2023-03-03T05:43:43.563 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T05:43:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 100/153 2023-03-03T05:43:45.219 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2023-03-03T05:43:45.219 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2023-03-03T05:43:45.219 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2023-03-03T05:43:45.219 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /run 2023-03-03T05:43:45.219 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2023-03-03T05:43:45.220 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2023-03-03T05:43:45.220 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2023-03-03T05:43:45.220 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2023-03-03T05:43:45.220 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2023-03-03T05:43:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:43:45.665 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T05:43:47.511 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 101/153 2023-03-03T05:43:47.745 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T05:43:47.977 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T05:43:48.058 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 102/153 2023-03-03T05:43:48.273 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T05:43:48.853 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 103/153 2023-03-03T05:43:49.040 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T05:43:49.122 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 104/153 2023-03-03T05:43:49.341 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 105/153 2023-03-03T05:43:49.419 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 121/153 2023-03-03T05:43:49.555 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 106/153 2023-03-03T05:43:49.706 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T05:43:49.706 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T05:43:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T05:43:49.846 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 107/153 2023-03-03T05:43:49.937 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T05:43:49.979 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T05:43:50.150 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-6.el8.x86_64 108/153 2023-03-03T05:43:50.212 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T05:43:50.416 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : automake-1.16.1-7.el8.noarch 109/153 2023-03-03T05:43:50.558 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T05:43:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T05:43:51.015 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T05:43:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 126/153 2023-03-03T05:43:52.301 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 127/153 2023-03-03T05:43:52.643 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openssl-devel-1:1.1.1k-7.el8.x86_64 128/153 2023-03-03T05:43:52.890 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-user-2.32.1-40.el8.x86_64 129/153 2023-03-03T05:43:53.133 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 130/153 2023-03-03T05:43:53.358 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 131/153 2023-03-03T05:43:53.529 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : elfutils-libelf-devel-0.188-3.el8.x86_64 132/153 2023-03-03T05:43:53.566 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T05:43:53.750 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 110/153 2023-03-03T05:43:53.880 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 111/153 2023-03-03T05:43:54.126 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 112/153 2023-03-03T05:43:54.347 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T05:43:54.676 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 113/153 2023-03-03T05:43:54.977 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T05:43:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T05:43:55.878 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T05:43:55.895 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T05:43:56.179 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 134/153 2023-03-03T05:43:56.197 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 114/153 2023-03-03T05:43:56.396 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T05:43:56.428 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 135/153 2023-03-03T05:43:56.559 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 115/153 2023-03-03T05:43:56.697 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 136/153 2023-03-03T05:43:56.861 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T05:43:56.877 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T05:43:56.967 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 116/153 2023-03-03T05:43:57.103 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 138/153 2023-03-03T05:43:57.177 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T05:43:57.278 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 117/153 2023-03-03T05:43:57.294 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T05:43:57.378 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T05:43:57.463 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T05:43:57.516 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 118/153 2023-03-03T05:43:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T05:43:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T05:43:57.732 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 119/153 2023-03-03T05:43:57.854 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-debuginfod-client-0.187-4.el8.x86_64 141/153 2023-03-03T05:43:57.938 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T05:43:58.056 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-libs-0.187-4.el8.x86_64 142/153 2023-03-03T05:43:58.057 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T05:43:58.314 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T05:43:58.318 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T05:43:58.439 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T05:43:58.440 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T05:43:58.713 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T05:43:58.911 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-user-2.32.1-38.el8.x86_64 146/153 2023-03-03T05:43:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-2.32.1-38.el8.x86_64 147/153 2023-03-03T05:43:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T05:43:59.388 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T05:43:59.388 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T05:43:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T05:43:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T05:43:59.854 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T05:43:59.855 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T05:44:00.113 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T05:44:00.113 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T05:44:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T05:44:00.492 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T05:44:01.082 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T05:44:01.083 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/153 2023-03-03T05:44:01.083 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/153 2023-03-03T05:44:01.083 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/153 2023-03-03T05:44:01.083 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/153 2023-03-03T05:44:01.083 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/153 2023-03-03T05:44:01.083 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/153 2023-03-03T05:44:01.084 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T05:44:01.084 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T05:44:01.084 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T05:44:01.084 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T05:44:01.084 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/153 2023-03-03T05:44:01.084 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/153 2023-03-03T05:44:01.084 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/153 2023-03-03T05:44:01.085 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/153 2023-03-03T05:44:01.085 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/153 2023-03-03T05:44:01.085 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/153 2023-03-03T05:44:01.085 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/153 2023-03-03T05:44:01.086 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T05:44:01.086 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/153 2023-03-03T05:44:01.086 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/153 2023-03-03T05:44:01.087 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/153 2023-03-03T05:44:01.087 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/153 2023-03-03T05:44:01.087 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/153 2023-03-03T05:44:01.087 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/153 2023-03-03T05:44:01.087 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/153 2023-03-03T05:44:01.095 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/153 2023-03-03T05:44:01.096 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/153 2023-03-03T05:44:01.096 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/153 2023-03-03T05:44:01.096 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/153 2023-03-03T05:44:01.096 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/153 2023-03-03T05:44:01.097 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/153 2023-03-03T05:44:01.097 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/153 2023-03-03T05:44:01.097 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : automake-1.16.1-7.el8.noarch 39/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/153 2023-03-03T05:44:01.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/153 2023-03-03T05:44:01.099 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-6.el8.x86_64 42/153 2023-03-03T05:44:01.099 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/153 2023-03-03T05:44:01.099 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 44/153 2023-03-03T05:44:01.100 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/153 2023-03-03T05:44:01.100 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/153 2023-03-03T05:44:01.100 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 47/153 2023-03-03T05:44:01.100 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/153 2023-03-03T05:44:01.100 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/153 2023-03-03T05:44:01.100 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 50/153 2023-03-03T05:44:01.100 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 51/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-server-10.19-1.module_el8.6.0+1047+42 52/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 54/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/153 2023-03-03T05:44:01.101 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/153 2023-03-03T05:44:01.102 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/153 2023-03-03T05:44:01.102 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/153 2023-03-03T05:44:01.102 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/153 2023-03-03T05:44:01.102 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/153 2023-03-03T05:44:01.102 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/153 2023-03-03T05:44:01.102 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 65/153 2023-03-03T05:44:01.102 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 66/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-devel-0.188-3.el8.x86_64 71/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 72/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/153 2023-03-03T05:44:01.103 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/153 2023-03-03T05:44:01.104 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/153 2023-03-03T05:44:01.104 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/153 2023-03-03T05:44:01.104 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 77/153 2023-03-03T05:44:01.104 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 78/153 2023-03-03T05:44:01.104 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 79/153 2023-03-03T05:44:01.104 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libselinux-devel-2.9-6.el8.x86_64 80/153 2023-03-03T05:44:01.104 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/153 2023-03-03T05:44:01.105 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-devel-2.32.1-40.el8.x86_64 82/153 2023-03-03T05:44:01.105 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/153 2023-03-03T05:44:01.105 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8.x86_64 84/153 2023-03-03T05:44:01.105 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 85/153 2023-03-03T05:44:01.106 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 86/153 2023-03-03T05:44:01.106 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 87/153 2023-03-03T05:44:01.106 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 88/153 2023-03-03T05:44:01.106 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 89/153 2023-03-03T05:44:01.107 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 90/153 2023-03-03T05:44:01.107 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 91/153 2023-03-03T05:44:01.107 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 92/153 2023-03-03T05:44:01.107 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 93/153 2023-03-03T05:44:01.108 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 94/153 2023-03-03T05:44:01.108 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : zlib-devel-1.2.11-19.el8.x86_64 95/153 2023-03-03T05:44:01.108 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 96/153 2023-03-03T05:44:01.108 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 97/153 2023-03-03T05:44:01.108 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 98/153 2023-03-03T05:44:01.108 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 99/153 2023-03-03T05:44:01.108 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 100/153 2023-03-03T05:44:01.109 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 101/153 2023-03-03T05:44:01.109 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 102/153 2023-03-03T05:44:01.109 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 103/153 2023-03-03T05:44:01.109 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 104/153 2023-03-03T05:44:01.109 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 105/153 2023-03-03T05:44:01.109 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 106/153 2023-03-03T05:44:01.110 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 107/153 2023-03-03T05:44:01.110 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 108/153 2023-03-03T05:44:01.110 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 109/153 2023-03-03T05:44:01.110 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 110/153 2023-03-03T05:44:01.110 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 111/153 2023-03-03T05:44:01.110 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 112/153 2023-03-03T05:44:01.110 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 113/153 2023-03-03T05:44:01.111 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-2.6-1.el8.noarch 114/153 2023-03-03T05:44:01.111 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 115/153 2023-03-03T05:44:01.111 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 116/153 2023-03-03T05:44:01.111 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 117/153 2023-03-03T05:44:01.111 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 118/153 2023-03-03T05:44:01.111 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 119/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 120/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 121/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 122/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 123/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 124/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 125/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 126/153 2023-03-03T05:44:01.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 127/153 2023-03-03T05:44:01.113 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 128/153 2023-03-03T05:44:01.113 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 129/153 2023-03-03T05:44:01.113 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 130/153 2023-03-03T05:44:01.113 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 131/153 2023-03-03T05:44:01.113 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.12-1.el 132/153 2023-03-03T05:44:01.113 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-initscripts-updown-1:1.39.12-1.el 133/153 2023-03-03T05:44:01.113 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-debuginfod-client-0.188-3.el8.x86_64 134/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-debuginfod-client-0.187-4.el8.x86_64 135/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-0.188-3.el8.x86_64 136/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-0.187-4.el8.x86_64 137/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libs-0.188-3.el8.x86_64 138/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libs-0.187-4.el8.x86_64 139/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libblkid-2.32.1-40.el8.x86_64 140/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libblkid-2.32.1-38.el8.x86_64 141/153 2023-03-03T05:44:01.114 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libfdisk-2.32.1-40.el8.x86_64 142/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libfdisk-2.32.1-38.el8.x86_64 143/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libmount-2.32.1-40.el8.x86_64 144/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libmount-2.32.1-38.el8.x86_64 145/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsmartcols-2.32.1-40.el8.x86_64 146/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsmartcols-2.32.1-38.el8.x86_64 147/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-2.32.1-40.el8.x86_64 148/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-2.32.1-38.el8.x86_64 149/153 2023-03-03T05:44:01.115 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-2.32.1-40.el8.x86_64 150/153 2023-03-03T05:44:01.116 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-2.32.1-38.el8.x86_64 151/153 2023-03-03T05:44:01.116 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-user-2.32.1-40.el8.x86_64 152/153 2023-03-03T05:44:03.069 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/153 2023-03-03T05:44:03.376 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-user-2.32.1-38.el8.x86_64 153/153 2023-03-03T05:44:03.376 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:03.376 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2023-03-03T05:44:03.376 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-initscripts-updown-1:1.40.12-1.el8.noarch 2023-03-03T05:44:03.376 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-debuginfod-client-0.188-3.el8.x86_64 2023-03-03T05:44:03.377 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-0.188-3.el8.x86_64 2023-03-03T05:44:03.377 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libs-0.188-3.el8.x86_64 2023-03-03T05:44:03.377 INFO:teuthology.orchestra.run.smithi172.stdout: libblkid-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.377 INFO:teuthology.orchestra.run.smithi172.stdout: libfdisk-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.377 INFO:teuthology.orchestra.run.smithi172.stdout: libmount-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.377 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.377 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.378 INFO:teuthology.orchestra.run.smithi172.stdout: libsmartcols-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.378 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.378 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.378 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-user-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.378 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-03-03T05:44:03.378 INFO:teuthology.orchestra.run.smithi172.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T05:44:03.378 INFO:teuthology.orchestra.run.smithi172.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.379 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.380 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.380 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.380 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.380 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.380 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.380 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.380 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.381 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:03.382 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel-0.188-3.el8.x86_64 2023-03-03T05:44:03.382 INFO:teuthology.orchestra.run.smithi172.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T05:44:03.382 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-2.9.7-16.el8.x86_64 2023-03-03T05:44:03.382 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T05:44:03.382 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-6.el8.x86_64 2023-03-03T05:44:03.382 INFO:teuthology.orchestra.run.smithi172.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T05:44:03.382 INFO:teuthology.orchestra.run.smithi172.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T05:44:03.383 INFO:teuthology.orchestra.run.smithi172.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: libselinux-devel-2.9-6.el8.x86_64 2023-03-03T05:44:03.384 INFO:teuthology.orchestra.run.smithi172.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T05:44:03.385 INFO:teuthology.orchestra.run.smithi172.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T05:44:03.385 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T05:44:03.385 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel-2.32.1-40.el8.x86_64 2023-03-03T05:44:03.385 INFO:teuthology.orchestra.run.smithi172.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-03-03T05:44:03.385 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T05:44:03.385 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T05:44:03.385 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.15-4.el8.x86_64 2023-03-03T05:44:03.386 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-03-03T05:44:03.386 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel-1:1.1.1k-7.el8.x86_64 2023-03-03T05:44:03.386 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-03-03T05:44:03.386 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-03-03T05:44:03.386 INFO:teuthology.orchestra.run.smithi172.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-03-03T05:44:03.387 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T05:44:03.387 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T05:44:03.387 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T05:44:03.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T05:44:03.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T05:44:03.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T05:44:03.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T05:44:03.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T05:44:03.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T05:44:03.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T05:44:03.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T05:44:03.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T05:44:03.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T05:44:03.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T05:44:03.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T05:44:03.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T05:44:03.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T05:44:03.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T05:44:03.391 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T05:44:03.391 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T05:44:03.391 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T05:44:03.391 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T05:44:03.391 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.0.6-14.el8.noarch 2023-03-03T05:44:03.392 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T05:44:03.392 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T05:44:03.392 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T05:44:03.392 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T05:44:03.392 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T05:44:03.392 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T05:44:03.393 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T05:44:03.393 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T05:44:03.393 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T05:44:03.393 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T05:44:03.393 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T05:44:03.394 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T05:44:03.394 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T05:44:03.394 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T05:44:03.394 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.394 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.395 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T05:44:03.395 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T05:44:03.395 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.395 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T05:44:03.395 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T05:44:03.396 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-03-03T05:44:03.396 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T05:44:03.396 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T05:44:03.396 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T05:44:03.396 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T05:44:03.396 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T05:44:03.397 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T05:44:03.397 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T05:44:03.397 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T05:44:03.397 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T05:44:03.398 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T05:44:03.398 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.398 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.398 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:03.398 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T05:44:03.398 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T05:44:03.399 INFO:teuthology.orchestra.run.smithi172.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T05:44:03.399 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T05:44:03.399 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T05:44:03.399 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel-1.2.11-19.el8.x86_64 2023-03-03T05:44:03.400 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:03.400 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-03-03T05:44:03.534 DEBUG:teuthology.parallel:result is None 2023-03-03T05:44:04.629 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 121/153 2023-03-03T05:44:04.874 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T05:44:04.969 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 122/153 2023-03-03T05:44:05.205 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T05:44:05.308 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 123/153 2023-03-03T05:44:05.529 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T05:44:05.991 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 124/153 2023-03-03T05:44:06.196 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T05:44:06.542 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 125/153 2023-03-03T05:44:06.912 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 126/153 2023-03-03T05:44:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 127/153 2023-03-03T05:44:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openssl-devel-1:1.1.1k-7.el8.x86_64 128/153 2023-03-03T05:44:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-user-2.32.1-40.el8.x86_64 129/153 2023-03-03T05:44:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 130/153 2023-03-03T05:44:09.343 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 131/153 2023-03-03T05:44:09.563 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : elfutils-libelf-devel-0.188-3.el8.x86_64 132/153 2023-03-03T05:44:13.420 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T05:44:13.720 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T05:44:13.784 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-10.19-1.module_el8.6.0+1047+42 133/153 2023-03-03T05:44:14.038 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 134/153 2023-03-03T05:44:14.272 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 135/153 2023-03-03T05:44:14.497 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 136/153 2023-03-03T05:44:14.670 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T05:44:14.725 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.12-1.el 137/153 2023-03-03T05:44:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 138/153 2023-03-03T05:44:15.161 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T05:44:15.305 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 139/153 2023-03-03T05:44:15.511 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T05:44:15.562 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 140/153 2023-03-03T05:44:15.746 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-debuginfod-client-0.187-4.el8.x86_64 141/153 2023-03-03T05:44:15.951 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libs-0.187-4.el8.x86_64 142/153 2023-03-03T05:44:15.951 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T05:44:16.239 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 143/153 2023-03-03T05:44:16.275 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T05:44:16.394 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.39.12-1.el 144/153 2023-03-03T05:44:16.395 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T05:44:16.689 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 145/153 2023-03-03T05:44:16.866 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-user-2.32.1-38.el8.x86_64 146/153 2023-03-03T05:44:17.138 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-2.32.1-38.el8.x86_64 147/153 2023-03-03T05:44:17.138 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T05:44:17.397 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libfdisk-2.32.1-38.el8.x86_64 148/153 2023-03-03T05:44:17.397 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T05:44:17.664 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libmount-2.32.1-38.el8.x86_64 149/153 2023-03-03T05:44:17.665 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T05:44:17.914 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libblkid-2.32.1-38.el8.x86_64 150/153 2023-03-03T05:44:17.914 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T05:44:19.557 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libuuid-2.32.1-38.el8.x86_64 151/153 2023-03-03T05:44:19.557 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T05:44:19.822 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libsmartcols-2.32.1-38.el8.x86_64 152/153 2023-03-03T05:44:20.012 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T05:44:20.345 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: elfutils-libelf-0.187-4.el8.x86_64 153/153 2023-03-03T05:44:20.345 INFO:teuthology.orchestra.run.smithi102.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-03-03T05:44:20.345 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:21.262 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/153 2023-03-03T05:44:21.262 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/153 2023-03-03T05:44:21.262 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/153 2023-03-03T05:44:21.262 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/153 2023-03-03T05:44:21.262 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/153 2023-03-03T05:44:21.263 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/153 2023-03-03T05:44:21.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/153 2023-03-03T05:44:21.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/153 2023-03-03T05:44:21.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/153 2023-03-03T05:44:21.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/153 2023-03-03T05:44:21.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/153 2023-03-03T05:44:21.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/153 2023-03-03T05:44:21.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/153 2023-03-03T05:44:21.265 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/153 2023-03-03T05:44:21.265 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/153 2023-03-03T05:44:21.265 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/153 2023-03-03T05:44:21.265 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/153 2023-03-03T05:44:21.265 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/153 2023-03-03T05:44:21.265 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/153 2023-03-03T05:44:21.265 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/153 2023-03-03T05:44:21.273 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/153 2023-03-03T05:44:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/153 2023-03-03T05:44:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/153 2023-03-03T05:44:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/153 2023-03-03T05:44:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/153 2023-03-03T05:44:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/153 2023-03-03T05:44:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/153 2023-03-03T05:44:21.274 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/153 2023-03-03T05:44:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/153 2023-03-03T05:44:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/153 2023-03-03T05:44:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/153 2023-03-03T05:44:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : automake-1.16.1-7.el8.noarch 39/153 2023-03-03T05:44:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/153 2023-03-03T05:44:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/153 2023-03-03T05:44:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-6.el8.x86_64 42/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 44/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 47/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/153 2023-03-03T05:44:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-10.19-1.module_el8.6.0+1047+4202cf9a. 50/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-contrib-10.19-1.module_el8.6.0+1047+4 51/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-server-10.19-1.module_el8.6.0+1047+42 52/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 54/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/153 2023-03-03T05:44:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 65/153 2023-03-03T05:44:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 66/153 2023-03-03T05:44:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/153 2023-03-03T05:44:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/153 2023-03-03T05:44:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/153 2023-03-03T05:44:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/153 2023-03-03T05:44:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-devel-0.188-3.el8.x86_64 71/153 2023-03-03T05:44:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 72/153 2023-03-03T05:44:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 77/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 78/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 79/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libselinux-devel-2.9-6.el8.x86_64 80/153 2023-03-03T05:44:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/153 2023-03-03T05:44:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-devel-2.32.1-40.el8.x86_64 82/153 2023-03-03T05:44:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/153 2023-03-03T05:44:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8.x86_64 84/153 2023-03-03T05:44:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 85/153 2023-03-03T05:44:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 86/153 2023-03-03T05:44:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 87/153 2023-03-03T05:44:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 88/153 2023-03-03T05:44:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 89/153 2023-03-03T05:44:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 90/153 2023-03-03T05:44:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 91/153 2023-03-03T05:44:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 92/153 2023-03-03T05:44:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 93/153 2023-03-03T05:44:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 94/153 2023-03-03T05:44:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : zlib-devel-1.2.11-19.el8.x86_64 95/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 96/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 97/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 98/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 99/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 100/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 101/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 102/153 2023-03-03T05:44:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 103/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 104/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 105/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 106/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 107/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 108/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 109/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 110/153 2023-03-03T05:44:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 111/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 112/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 113/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-2.6-1.el8.noarch 114/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 115/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 116/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 117/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 118/153 2023-03-03T05:44:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 119/153 2023-03-03T05:44:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 120/153 2023-03-03T05:44:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 121/153 2023-03-03T05:44:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 122/153 2023-03-03T05:44:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 123/153 2023-03-03T05:44:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 124/153 2023-03-03T05:44:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 125/153 2023-03-03T05:44:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 126/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 127/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 128/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 129/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 130/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 131/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.12-1.el 132/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-initscripts-updown-1:1.39.12-1.el 133/153 2023-03-03T05:44:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.188-3.el8.x86_64 134/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.187-4.el8.x86_64 135/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.188-3.el8.x86_64 136/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.187-4.el8.x86_64 137/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.188-3.el8.x86_64 138/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.187-4.el8.x86_64 139/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.32.1-40.el8.x86_64 140/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.32.1-38.el8.x86_64 141/153 2023-03-03T05:44:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.32.1-40.el8.x86_64 142/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.32.1-38.el8.x86_64 143/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.32.1-40.el8.x86_64 144/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.32.1-38.el8.x86_64 145/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.32.1-40.el8.x86_64 146/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.32.1-38.el8.x86_64 147/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.32.1-40.el8.x86_64 148/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.32.1-38.el8.x86_64 149/153 2023-03-03T05:44:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.32.1-40.el8.x86_64 150/153 2023-03-03T05:44:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.32.1-38.el8.x86_64 151/153 2023-03-03T05:44:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.32.1-40.el8.x86_64 152/153 2023-03-03T05:44:23.311 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.32.1-38.el8.x86_64 153/153 2023-03-03T05:44:23.311 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:23.311 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2023-03-03T05:44:23.311 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-initscripts-updown-1:1.40.12-1.el8.noarch 2023-03-03T05:44:23.311 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client-0.188-3.el8.x86_64 2023-03-03T05:44:23.312 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-0.188-3.el8.x86_64 2023-03-03T05:44:23.312 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs-0.188-3.el8.x86_64 2023-03-03T05:44:23.312 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.312 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.312 INFO:teuthology.orchestra.run.smithi102.stdout: libmount-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.312 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.312 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.313 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.313 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.313 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.313 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.313 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2023-03-03T05:44:23.313 INFO:teuthology.orchestra.run.smithi102.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T05:44:23.313 INFO:teuthology.orchestra.run.smithi102.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T05:44:23.314 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.314 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.314 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.314 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.318 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.318 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.318 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.318 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.318 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T05:44:23.318 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel-0.188-3.el8.x86_64 2023-03-03T05:44:23.318 INFO:teuthology.orchestra.run.smithi102.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T05:44:23.319 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-2.9.7-16.el8.x86_64 2023-03-03T05:44:23.319 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T05:44:23.319 INFO:teuthology.orchestra.run.smithi102.stdout: jq-1.6-6.el8.x86_64 2023-03-03T05:44:23.319 INFO:teuthology.orchestra.run.smithi102.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T05:44:23.319 INFO:teuthology.orchestra.run.smithi102.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-03-03T05:44:23.319 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T05:44:23.319 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-03-03T05:44:23.320 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T05:44:23.321 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.321 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.321 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T05:44:23.321 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.321 INFO:teuthology.orchestra.run.smithi102.stdout: libselinux-devel-2.9-6.el8.x86_64 2023-03-03T05:44:23.321 INFO:teuthology.orchestra.run.smithi102.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T05:44:23.321 INFO:teuthology.orchestra.run.smithi102.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T05:44:23.322 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T05:44:23.322 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel-2.32.1-40.el8.x86_64 2023-03-03T05:44:23.322 INFO:teuthology.orchestra.run.smithi102.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-03-03T05:44:23.322 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T05:44:23.322 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T05:44:23.322 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.15-4.el8.x86_64 2023-03-03T05:44:23.322 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-03-03T05:44:23.323 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel-1:1.1.1k-7.el8.x86_64 2023-03-03T05:44:23.323 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-03-03T05:44:23.323 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-03-03T05:44:23.323 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-03-03T05:44:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T05:44:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T05:44:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server-10.19-1.module_el8.6.0+1047+4202cf9a.x86_64 2023-03-03T05:44:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T05:44:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T05:44:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T05:44:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T05:44:23.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T05:44:23.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T05:44:23.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T05:44:23.325 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T05:44:23.326 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T05:44:23.327 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T05:44:23.327 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T05:44:23.327 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T05:44:23.327 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.0.6-14.el8.noarch 2023-03-03T05:44:23.327 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T05:44:23.327 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T05:44:23.327 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T05:44:23.328 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T05:44:23.329 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T05:44:23.329 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T05:44:23.329 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T05:44:23.329 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.329 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.330 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T05:44:23.330 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T05:44:23.330 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.330 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T05:44:23.330 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T05:44:23.330 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-03-03T05:44:23.330 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T05:44:23.331 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T05:44:23.331 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T05:44:23.331 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T05:44:23.331 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T05:44:23.331 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T05:44:23.331 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T05:44:23.331 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T05:44:23.332 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T05:44:23.332 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T05:44:23.332 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.332 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.332 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T05:44:23.332 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T05:44:23.332 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T05:44:23.333 INFO:teuthology.orchestra.run.smithi102.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T05:44:23.333 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T05:44:23.333 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T05:44:23.333 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel-1.2.11-19.el8.x86_64 2023-03-03T05:44:23.334 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:23.334 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2023-03-03T05:44:23.721 DEBUG:teuthology.parallel:result is None 2023-03-03T05:44:23.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T05:44:23.862 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T05:44:23.944 INFO:teuthology.orchestra.run.smithi102.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T05:44:23.945 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T05:44:23.946 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T05:44:23.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T05:44:24.084 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T05:44:24.111 INFO:teuthology.orchestra.run.smithi172.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T05:44:24.112 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T05:44:24.112 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T05:44:24.112 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-03T05:44:24.113 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:24.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T05:44:24.222 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:44:24.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T05:44:24.256 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-03T05:44:24.257 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:24.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T05:44:24.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T05:44:24.466 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:44:24.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T05:44:24.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T05:44:24.565 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-03T05:44:24.566 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:24.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T05:44:24.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T05:44:24.816 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:44:24.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T05:44:24.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T05:44:24.912 INFO:teuthology.run_tasks:Running task ceph... 2023-03-03T05:44:24.990 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-03-03T05:44:24.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /var/log/ceph 2023-03-03T05:44:24.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2023-03-03T05:44:25.092 INFO:tasks.ceph:Disabling ceph logrotate... 2023-03-03T05:44:25.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T05:44:25.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T05:44:25.197 INFO:tasks.ceph:Creating extra log directories... 2023-03-03T05:44:25.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T05:44:25.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T05:44:25.312 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-03-03T05:44:25.313 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-03-03T05:44:25.313 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-03-03T05:44:25.314 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:25.314 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T05:44:25.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T05:44:25.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T05:44:25.747 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:44:25.748 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T05:44:25.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T05:44:25.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T05:44:25.919 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-03-03T05:44:25.919 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-03-03T05:44:25.920 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192093', 'branch': 'wip-vshankar-testing-20230303.023823', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7192093', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20230303.023823'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20230303.023823', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20230303.023823', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi102.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOZkGy1FXGwI7zar+JgG/pI7NbBPl0NOU0x42EcareR46Y0bdu2QAQdWacApFIsIq4S32kMe+audUeP0nlFn4Y=', 'smithi172.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1SyCVoQPMW4XOWT/PxlpBz+odYm8m6q9lBogugRDxqVJ5FiaELoLATbj70I+3K/pcqEXXmsrIxYFalyDvRhIqJ5tUsqy/rinKy/S6kyX9ESiUmKWQHEOHe2Yyoy5YrjH18+LG9bgYyR8nI+0ifqdEJ0+t+nzPkN50DDJMPYoSdB7tvCJDeLNz8vOJ9nX7jqtTyWg2dtSFW/Ah0YoOTDieL3p67NoJ0Gf6uSHR8p0J0to4hC90yY/UAG4EQBtOMM4ZN/0DdPnKyQvuN9sGQ+XpTqQ+ok0HUulGbn0oEaovOL7glS1eipn+aqV0BNdlyldxTjQju5kdw7KVL6WFrH1cydV7KAnT04x6uv7dCGFj5199v2HZg47aAj9AYV7K08JUUsbPPiL7WolOXjcJbUAvLVlyrNDPoxGOtPYv0Fnq2oNCj6bK6IoN0vH0lXipVf4BjdIYxXxaPS9GXZGV2iQ2anuGvi2HBJ+KlYFSJgrqYFLLH6dqmyRApgmGknw1XcU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, {'ceph': None}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6', 'timestamp': '2023-03-03_04:39:14', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.31660'} 2023-03-03T05:44:25.921 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T05:44:25.926 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T05:44:25.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T05:44:25.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T05:44:26.098 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:26.099 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T05:44:26.168 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T05:44:26.168 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2023-03-03T05:44:26.237 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T05:44:26.238 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:26.238 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 448 Links: 1 2023-03-03T05:44:26.238 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:26.238 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:26.238 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-03-03 05:44:20.592209605 +0000 2023-03-03T05:44:26.239 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-03-03 05:40:54.280226627 +0000 2023-03-03T05:44:26.239 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-03-03 05:40:54.280226627 +0000 2023-03-03T05:44:26.239 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2023-03-03 05:40:54.243227381 +0000 2023-03-03T05:44:26.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T05:44:26.336 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-03-03T05:44:26.336 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-03-03T05:44:26.337 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000318189 s, 1.6 MB/s 2023-03-03T05:44:26.341 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T05:44:26.416 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2023-03-03T05:44:26.487 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T05:44:26.487 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:26.487 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 454 Links: 1 2023-03-03T05:44:26.487 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:26.488 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:26.488 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-03-03 05:44:20.592209605 +0000 2023-03-03T05:44:26.488 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-03-03 05:40:55.379204224 +0000 2023-03-03T05:44:26.488 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-03-03 05:40:55.379204224 +0000 2023-03-03T05:44:26.488 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2023-03-03 05:40:55.348204856 +0000 2023-03-03T05:44:26.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T05:44:26.588 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-03-03T05:44:26.588 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-03-03T05:44:26.588 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000321388 s, 1.6 MB/s 2023-03-03T05:44:26.592 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T05:44:26.665 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2023-03-03T05:44:26.737 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T05:44:26.737 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:26.737 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 460 Links: 1 2023-03-03T05:44:26.737 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:26.738 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:26.738 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-03-03 05:44:20.592209605 +0000 2023-03-03T05:44:26.738 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-03-03 05:40:56.490181577 +0000 2023-03-03T05:44:26.738 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-03-03 05:40:56.490181577 +0000 2023-03-03T05:44:26.738 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2023-03-03 05:40:56.458182230 +0000 2023-03-03T05:44:26.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T05:44:26.840 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-03-03T05:44:26.841 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-03-03T05:44:26.841 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000356741 s, 1.4 MB/s 2023-03-03T05:44:26.844 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T05:44:26.920 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2023-03-03T05:44:26.990 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T05:44:26.991 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:26.991 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 466 Links: 1 2023-03-03T05:44:26.991 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:26.991 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:26.991 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-03-03 05:44:20.593209585 +0000 2023-03-03T05:44:26.991 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-03-03 05:40:57.584159277 +0000 2023-03-03T05:44:26.992 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-03-03 05:40:57.584159277 +0000 2023-03-03T05:44:26.992 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2023-03-03 05:40:57.550159970 +0000 2023-03-03T05:44:26.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T05:44:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-03-03T05:44:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-03-03T05:44:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000351417 s, 1.5 MB/s 2023-03-03T05:44:27.100 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T05:44:27.174 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:27.174 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:44:27.174 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T05:44:27.194 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T05:44:27.195 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2023-03-03T05:44:27.255 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T05:44:27.255 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 385554 Links: 1 2023-03-03T05:44:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:27.256 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:27.256 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-03-03 05:44:00.702509555 +0000 2023-03-03T05:44:27.256 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-03-03 05:36:38.288825464 +0000 2023-03-03T05:44:27.256 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-03-03 05:36:38.288825464 +0000 2023-03-03T05:44:27.256 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-03-03T05:44:27.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T05:44:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-03-03T05:44:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-03-03T05:44:27.325 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.00023355 s, 2.2 MB/s 2023-03-03T05:44:27.326 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T05:44:27.386 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2023-03-03T05:44:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T05:44:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:27.446 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 386639 Links: 1 2023-03-03T05:44:27.447 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:27.447 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:27.447 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-03-03 05:44:00.702509555 +0000 2023-03-03T05:44:27.447 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-03-03 05:36:38.800813013 +0000 2023-03-03T05:44:27.447 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-03-03 05:36:38.800813013 +0000 2023-03-03T05:44:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-03-03T05:44:27.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T05:44:27.512 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-03-03T05:44:27.513 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-03-03T05:44:27.513 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.00021888 s, 2.3 MB/s 2023-03-03T05:44:27.514 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T05:44:27.573 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2023-03-03T05:44:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T05:44:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:27.632 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 380601 Links: 1 2023-03-03T05:44:27.632 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:27.632 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:27.633 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-03-03 05:44:00.702509555 +0000 2023-03-03T05:44:27.633 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-03-03 05:36:39.319800393 +0000 2023-03-03T05:44:27.633 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-03-03 05:36:39.319800393 +0000 2023-03-03T05:44:27.633 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-03-03T05:44:27.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T05:44:27.700 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-03-03T05:44:27.700 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-03-03T05:44:27.700 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000214313 s, 2.4 MB/s 2023-03-03T05:44:27.701 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T05:44:27.759 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2023-03-03T05:44:27.817 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T05:44:27.817 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T05:44:27.817 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 388470 Links: 1 2023-03-03T05:44:27.817 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:44:27.818 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T05:44:27.818 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-03-03 05:44:00.702509555 +0000 2023-03-03T05:44:27.818 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-03-03 05:36:39.847787553 +0000 2023-03-03T05:44:27.818 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-03-03 05:36:39.847787553 +0000 2023-03-03T05:44:27.818 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-03-03T05:44:27.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T05:44:27.883 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-03-03T05:44:27.883 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-03-03T05:44:27.884 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000208851 s, 2.5 MB/s 2023-03-03T05:44:27.884 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T05:44:27.944 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:27.944 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T05:44:27.944 INFO:tasks.ceph:Generating config... 2023-03-03T05:44:27.946 INFO:tasks.ceph:[client] client mount timeout = 600 2023-03-03T05:44:27.946 INFO:tasks.ceph:[client] debug client = 20 2023-03-03T05:44:27.946 INFO:tasks.ceph:[client] debug ms = 1 2023-03-03T05:44:27.946 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-03-03T05:44:27.947 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-03-03T05:44:27.947 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-03-03T05:44:27.947 INFO:tasks.ceph:[mds] debug mds = 20 2023-03-03T05:44:27.947 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-03-03T05:44:27.948 INFO:tasks.ceph:[mds] debug ms = 1 2023-03-03T05:44:27.948 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-03-03T05:44:27.948 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-03-03T05:44:27.948 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-03-03T05:44:27.948 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-03-03T05:44:27.949 INFO:tasks.ceph:[mds] mds debug frag = True 2023-03-03T05:44:27.949 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-03-03T05:44:27.949 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-03-03T05:44:27.949 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-03-03T05:44:27.949 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-03-03T05:44:27.949 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2023-03-03T05:44:27.951 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-03-03T05:44:27.951 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-03-03T05:44:27.952 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-03-03T05:44:27.952 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-03-03T05:44:27.952 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-03-03T05:44:27.952 INFO:tasks.ceph:[mgr] debug ms = 1 2023-03-03T05:44:27.953 INFO:tasks.ceph:[mon] debug mon = 20 2023-03-03T05:44:27.953 INFO:tasks.ceph:[mon] debug ms = 1 2023-03-03T05:44:27.953 INFO:tasks.ceph:[mon] debug paxos = 20 2023-03-03T05:44:27.953 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-03-03T05:44:27.953 INFO:tasks.ceph:[osd] bdev async discard = True 2023-03-03T05:44:27.954 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-03-03T05:44:27.954 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-03-03T05:44:27.954 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-03-03T05:44:27.954 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-03-03T05:44:27.954 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-03-03T05:44:27.955 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-03-03T05:44:27.955 INFO:tasks.ceph:[osd] debug ms = 1 2023-03-03T05:44:27.955 INFO:tasks.ceph:[osd] debug osd = 20 2023-03-03T05:44:27.955 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-03-03T05:44:27.955 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-03-03T05:44:27.956 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-03-03T05:44:27.956 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-03-03T05:44:27.956 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-03-03T05:44:27.956 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-03-03T05:44:27.956 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-03-03T05:44:27.957 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-03-03T05:44:27.957 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-03-03T05:44:27.957 INFO:tasks.ceph:Setting up mon.a... 2023-03-03T05:44:27.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-03-03T05:44:28.116 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.keyring 2023-03-03T05:44:28.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-03-03T05:44:28.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T05:44:28.382 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.102'), ('mon.b', '172.21.15.172'), ('mon.c', '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]')] 2023-03-03T05:44:28.382 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.102,172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-03-03T05:44:28.383 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:28.384 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T05:44:28.456 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.102 --add b 172.21.15.172 --addv c '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: generated fsid 56ef27ff-a10a-47b1-b557-af27e3b0b6e2 2023-03-03T05:44:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = pacific 2023-03-03T05:44:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 0 2023-03-03T05:44:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:fsid 56ef27ff-a10a-47b1-b557-af27e3b0b6e2 2023-03-03T05:44:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:last_changed 2023-03-03T05:44:28.575168+0000 2023-03-03T05:44:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:created 2023-03-03T05:44:28.575168+0000 2023-03-03T05:44:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:min_mon_release 16 (pacific) 2023-03-03T05:44:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:election_strategy: 1 2023-03-03T05:44:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2023-03-03T05:44:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2023-03-03T05:44:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:2: [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] mon.c 2023-03-03T05:44:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-03-03T05:44:28.583 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T05:44:28.655 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 56ef27ff-a10a-47b1-b557-af27e3b0b6e2... 2023-03-03T05:44:28.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T05:44:28.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T05:44:28.709 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-03-03T05:44:28.709 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2023-03-03T05:44:28.709 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T05:44:28.710 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2023-03-03T05:44:28.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.710 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2023-03-03T05:44:28.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.710 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2023-03-03T05:44:28.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.711 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2023-03-03T05:44:28.711 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2023-03-03T05:44:28.711 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.711 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2023-03-03T05:44:28.711 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2023-03-03T05:44:28.711 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T05:44:28.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2023-03-03T05:44:28.712 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.712 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T05:44:28.712 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T05:44:28.712 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.712 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2023-03-03T05:44:28.713 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2023-03-03T05:44:28.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2023-03-03T05:44:28.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2023-03-03T05:44:28.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2023-03-03T05:44:28.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2023-03-03T05:44:28.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2023-03-03T05:44:28.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2023-03-03T05:44:28.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T05:44:28.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T05:44:28.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2023-03-03T05:44:28.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T05:44:28.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2023-03-03T05:44:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "172.21.15.102,172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790]" 2023-03-03T05:44:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on skipped message = False 2023-03-03T05:44:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 56ef27ff-a10a-47b1-b557-af27e3b0b6e2 2023-03-03T05:44:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.716 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2023-03-03T05:44:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2023-03-03T05:44:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2023-03-03T05:44:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2023-03-03T05:44:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2023-03-03T05:44:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2023-03-03T05:44:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2023-03-03T05:44:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2023-03-03T05:44:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2023-03-03T05:44:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2023-03-03T05:44:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2023-03-03T05:44:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2023-03-03T05:44:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2023-03-03T05:44:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2023-03-03T05:44:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2023-03-03T05:44:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2023-03-03T05:44:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2023-03-03T05:44:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2023-03-03T05:44:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: bdev async discard = True 2023-03-03T05:44:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: bdev enable discard = True 2023-03-03T05:44:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore allocator = bitmap 2023-03-03T05:44:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2023-03-03T05:44:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2023-03-03T05:44:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 1/20 2023-03-03T05:44:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 1/20 2023-03-03T05:44:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-03-03T05:44:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2023-03-03T05:44:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 4/10 2023-03-03T05:44:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T05:44:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2023-03-03T05:44:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2023-03-03T05:44:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: osd blocked scrub grace period = 3600 2023-03-03T05:44:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2023-03-03T05:44:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: osd max markdown count = 1000 2023-03-03T05:44:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2023-03-03T05:44:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: osd op complaint time = 180 2023-03-03T05:44:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.726 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2023-03-03T05:44:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-03-03T05:44:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2023-03-03T05:44:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-03-03T05:44:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-03-03T05:44:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-03-03T05:44:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-03-03T05:44:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2023-03-03T05:44:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.728 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2023-03-03T05:44:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-03-03T05:44:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-03-03T05:44:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2023-03-03T05:44:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-03-03T05:44:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2023-03-03T05:44:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2023-03-03T05:44:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-03-03T05:44:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2023-03-03T05:44:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2023-03-03T05:44:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-03-03T05:44:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.731 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T05:44:28.731 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T05:44:28.732 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T05:44:28.732 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.733 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T05:44:28.733 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T05:44:28.733 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T05:44:28.734 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.734 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2023-03-03T05:44:28.735 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2023-03-03T05:44:28.735 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.735 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2023-03-03T05:44:28.736 INFO:teuthology.orchestra.run.smithi172.stdout: mon op complaint time = 120 2023-03-03T05:44:28.736 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-03-03T05:44:28.736 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2023-03-03T05:44:28.737 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2023-03-03T05:44:28.737 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2023-03-03T05:44:28.737 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2023-03-03T05:44:28.738 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T05:44:28.738 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T05:44:28.739 INFO:teuthology.orchestra.run.smithi172.stdout: client mount timeout = 600 2023-03-03T05:44:28.739 INFO:teuthology.orchestra.run.smithi172.stdout: debug client = 20 2023-03-03T05:44:28.739 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-03-03T05:44:28.739 INFO:teuthology.orchestra.run.smithi172.stdout: rados mon op timeout = 900 2023-03-03T05:44:28.740 INFO:teuthology.orchestra.run.smithi172.stdout: rados osd op timeout = 900 2023-03-03T05:44:28.740 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2023-03-03T05:44:28.740 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2023-03-03T05:44:28.741 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2023-03-03T05:44:28.741 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.a] 2023-03-03T05:44:28.741 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.c] 2023-03-03T05:44:28.741 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.e] 2023-03-03T05:44:28.741 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.b] 2023-03-03T05:44:28.742 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.d] 2023-03-03T05:44:28.742 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.f] 2023-03-03T05:44:28.742 INFO:teuthology.orchestra.run.smithi172.stdout:[mds] 2023-03-03T05:44:28.742 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds = 20 2023-03-03T05:44:28.742 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds balancer = 20 2023-03-03T05:44:28.743 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-03-03T05:44:28.743 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal fragment size max = 10000 2023-03-03T05:44:28.743 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal merge size = 5 2023-03-03T05:44:28.743 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal split bits = 3 2023-03-03T05:44:28.743 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal split size = 100 2023-03-03T05:44:28.744 INFO:teuthology.orchestra.run.smithi172.stdout: mds debug frag = True 2023-03-03T05:44:28.744 INFO:teuthology.orchestra.run.smithi172.stdout: mds debug scatterstat = True 2023-03-03T05:44:28.744 INFO:teuthology.orchestra.run.smithi172.stdout: mds oft prefetch dirfrags = True 2023-03-03T05:44:28.744 INFO:teuthology.orchestra.run.smithi172.stdout: mds op complaint time = 180 2023-03-03T05:44:28.744 INFO:teuthology.orchestra.run.smithi172.stdout: mds verify scatter = True 2023-03-03T05:44:28.745 INFO:teuthology.orchestra.run.smithi172.stdout: mds_dir_prefetch = True 2023-03-03T05:44:28.745 INFO:teuthology.orchestra.run.smithi172.stdout: mds_sleep_rank_change = 5000000.0 2023-03-03T05:44:28.745 INFO:teuthology.orchestra.run.smithi172.stdout: osd op complaint time = 180 2023-03-03T05:44:28.745 INFO:teuthology.orchestra.run.smithi172.stdout: rados mon op timeout = 900 2023-03-03T05:44:28.745 INFO:teuthology.orchestra.run.smithi172.stdout: rados osd op timeout = 900 2023-03-03T05:44:28.827 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2023-03-03T05:44:28.827 INFO:teuthology.orchestra.run.smithi102.stdout: chdir = "" 2023-03-03T05:44:28.827 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T05:44:28.827 INFO:teuthology.orchestra.run.smithi102.stdout: auth supported = cephx 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: filestore xattr use omap = true 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: mon clock drift allowed = 1.000 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: osd crush chooseleaf type = 0 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: auth debug = true 2023-03-03T05:44:28.828 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.829 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on old message = true 2023-03-03T05:44:28.829 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on bug = true 2023-03-03T05:44:28.829 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.829 INFO:teuthology.orchestra.run.smithi102.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T05:44:28.829 INFO:teuthology.orchestra.run.smithi102.stdout: mon pg warn max object skew = 0 2023-03-03T05:44:28.829 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.829 INFO:teuthology.orchestra.run.smithi102.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default size = 2 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow primary affinity = true 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow pg remap = true 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on legacy crush tunables = false 2023-03-03T05:44:28.830 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on crush straw calc version zero = false 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on no sortbitwise = false 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on osd down out interval zero = false 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on too few osds = false 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: mon_allow_pool_size_one = true 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.831 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T05:44:28.832 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.832 INFO:teuthology.orchestra.run.smithi102.stdout: osd default data pool replay window = 5 2023-03-03T05:44:28.832 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.832 INFO:teuthology.orchestra.run.smithi102.stdout: mon allow pool delete = true 2023-03-03T05:44:28.832 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.832 INFO:teuthology.orchestra.run.smithi102.stdout: mon cluster log file level = debug 2023-03-03T05:44:28.832 INFO:teuthology.orchestra.run.smithi102.stdout: debug asserts on shutdown = true 2023-03-03T05:44:28.833 INFO:teuthology.orchestra.run.smithi102.stdout: mon health detail to clog = false 2023-03-03T05:44:28.833 INFO:teuthology.orchestra.run.smithi102.stdout: mon host = "172.21.15.102,172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790]" 2023-03-03T05:44:28.833 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on skipped message = False 2023-03-03T05:44:28.833 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 56ef27ff-a10a-47b1-b557-af27e3b0b6e2 2023-03-03T05:44:28.833 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.833 INFO:teuthology.orchestra.run.smithi102.stdout:[osd] 2023-03-03T05:44:28.833 INFO:teuthology.orchestra.run.smithi102.stdout: osd journal size = 100 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub load threshold = 5.0 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub max interval = 600 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: osd recover clone overlap = true 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: osd recovery max chunk = 1048576 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.834 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug shutdown = true 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug op order = true 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify stray on activate = true 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: osd open classes on start = true 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug pg log writeout = true 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: osd deep scrub update digest min age = 30 2023-03-03T05:44:28.835 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: osd map max advance = 10 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: journal zero on create = true 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: filestore ondisk finisher threads = 3 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: filestore apply finisher threads = 3 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.836 INFO:teuthology.orchestra.run.smithi102.stdout: bdev debug aio = true 2023-03-03T05:44:28.837 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug misdirected ops = true 2023-03-03T05:44:28.837 INFO:teuthology.orchestra.run.smithi102.stdout: bdev async discard = True 2023-03-03T05:44:28.837 INFO:teuthology.orchestra.run.smithi102.stdout: bdev enable discard = True 2023-03-03T05:44:28.837 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore allocator = bitmap 2023-03-03T05:44:28.837 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore block size = 96636764160 2023-03-03T05:44:28.838 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore fsck on mount = True 2023-03-03T05:44:28.838 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluefs = 1/20 2023-03-03T05:44:28.838 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluestore = 1/20 2023-03-03T05:44:28.838 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2023-03-03T05:44:28.838 INFO:teuthology.orchestra.run.smithi102.stdout: debug osd = 20 2023-03-03T05:44:28.838 INFO:teuthology.orchestra.run.smithi102.stdout: debug rocksdb = 4/10 2023-03-03T05:44:28.838 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T05:44:28.839 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd full ratio = 0.9 2023-03-03T05:44:28.839 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd nearfull ratio = 0.8 2023-03-03T05:44:28.839 INFO:teuthology.orchestra.run.smithi102.stdout: osd blocked scrub grace period = 3600 2023-03-03T05:44:28.839 INFO:teuthology.orchestra.run.smithi102.stdout: osd failsafe full ratio = 0.95 2023-03-03T05:44:28.839 INFO:teuthology.orchestra.run.smithi102.stdout: osd max markdown count = 1000 2023-03-03T05:44:28.839 INFO:teuthology.orchestra.run.smithi102.stdout: osd objectstore = bluestore 2023-03-03T05:44:28.840 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2023-03-03T05:44:28.840 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.840 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr] 2023-03-03T05:44:28.840 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2023-03-03T05:44:28.840 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgr = 20 2023-03-03T05:44:28.840 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2023-03-03T05:44:28.840 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2023-03-03T05:44:28.841 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2023-03-03T05:44:28.841 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2023-03-03T05:44:28.841 INFO:teuthology.orchestra.run.smithi102.stdout: mgr/telemetry/nag = false 2023-03-03T05:44:28.841 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.841 INFO:teuthology.orchestra.run.smithi102.stdout:[mon] 2023-03-03T05:44:28.841 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2023-03-03T05:44:28.841 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2023-03-03T05:44:28.842 INFO:teuthology.orchestra.run.smithi102.stdout: debug paxos = 20 2023-03-03T05:44:28.842 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2023-03-03T05:44:28.842 INFO:teuthology.orchestra.run.smithi102.stdout: mon data avail warn = 5 2023-03-03T05:44:28.842 INFO:teuthology.orchestra.run.smithi102.stdout: mon mgr mkfs grace = 240 2023-03-03T05:44:28.842 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2023-03-03T05:44:28.842 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd reporter subtree level = osd 2023-03-03T05:44:28.843 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd prime pg temp = true 2023-03-03T05:44:28.843 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2023-03-03T05:44:28.843 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.843 INFO:teuthology.orchestra.run.smithi102.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T05:44:28.843 INFO:teuthology.orchestra.run.smithi102.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T05:44:28.844 INFO:teuthology.orchestra.run.smithi102.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T05:44:28.844 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.844 INFO:teuthology.orchestra.run.smithi102.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T05:44:28.845 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T05:44:28.845 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T05:44:28.845 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.845 INFO:teuthology.orchestra.run.smithi102.stdout: # 1m isn't quite enough 2023-03-03T05:44:28.846 INFO:teuthology.orchestra.run.smithi102.stdout: mon_down_mkfs_grace = 2m 2023-03-03T05:44:28.846 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.846 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_filestore_osds = false 2023-03-03T05:44:28.846 INFO:teuthology.orchestra.run.smithi102.stdout: mon op complaint time = 120 2023-03-03T05:44:28.847 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:44:28.847 INFO:teuthology.orchestra.run.smithi102.stdout:[client] 2023-03-03T05:44:28.847 INFO:teuthology.orchestra.run.smithi102.stdout: rgw cache enabled = true 2023-03-03T05:44:28.847 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable ops log = true 2023-03-03T05:44:28.848 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable usage log = true 2023-03-03T05:44:28.848 INFO:teuthology.orchestra.run.smithi102.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T05:44:28.848 INFO:teuthology.orchestra.run.smithi102.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T05:44:28.848 INFO:teuthology.orchestra.run.smithi102.stdout: client mount timeout = 600 2023-03-03T05:44:28.849 INFO:teuthology.orchestra.run.smithi102.stdout: debug client = 20 2023-03-03T05:44:28.849 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2023-03-03T05:44:28.849 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2023-03-03T05:44:28.849 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2023-03-03T05:44:28.850 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.a] 2023-03-03T05:44:28.850 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.b] 2023-03-03T05:44:28.850 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.c] 2023-03-03T05:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2023-03-03T05:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2023-03-03T05:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.e] 2023-03-03T05:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2023-03-03T05:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2023-03-03T05:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.f] 2023-03-03T05:44:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:[mds] 2023-03-03T05:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds = 20 2023-03-03T05:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds balancer = 20 2023-03-03T05:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2023-03-03T05:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: mds bal fragment size max = 10000 2023-03-03T05:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: mds bal merge size = 5 2023-03-03T05:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: mds bal split bits = 3 2023-03-03T05:44:28.852 INFO:teuthology.orchestra.run.smithi102.stdout: mds bal split size = 100 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug frag = True 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug scatterstat = True 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: mds oft prefetch dirfrags = True 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: mds op complaint time = 180 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: mds verify scatter = True 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: mds_dir_prefetch = True 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: mds_sleep_rank_change = 5000000.0 2023-03-03T05:44:28.853 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2023-03-03T05:44:28.854 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2023-03-03T05:44:28.854 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2023-03-03T05:44:28.870 INFO:tasks.ceph:Creating admin key on mon.a... 2023-03-03T05:44:28.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-03-03T05:44:29.024 INFO:tasks.ceph:Copying monmap to all nodes... 2023-03-03T05:44:29.025 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:29.025 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-03-03T05:44:29.097 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:29.098 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-03-03T05:44:29.166 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi102.front.sepia.ceph.com 2023-03-03T05:44:29.167 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:29.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T05:44:29.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T05:44:29.307 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:44:29.307 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:29.414 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2023-03-03T05:44:29.416 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:44:29.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T05:44:29.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T05:44:29.454 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:44:29.455 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:29.513 INFO:tasks.ceph:Setting up mon nodes... 2023-03-03T05:44:29.514 INFO:tasks.ceph:Setting up mgr nodes... 2023-03-03T05:44:29.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T05:44:29.686 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T05:44:29.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T05:44:29.754 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T05:44:29.756 INFO:tasks.ceph:Setting up mds nodes... 2023-03-03T05:44:29.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T05:44:29.929 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T05:44:29.938 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-03-03T05:44:30.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T05:44:30.203 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T05:44:30.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-03-03T05:44:30.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-03-03T05:44:30.478 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-03-03T05:44:30.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-03-03T05:44:30.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T05:44:30.641 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T05:44:30.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-03-03T05:44:30.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T05:44:30.766 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T05:44:30.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-03-03T05:44:30.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-03-03T05:44:30.889 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-03-03T05:44:30.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-03-03T05:44:30.918 INFO:tasks.ceph_client:Setting up client nodes... 2023-03-03T05:44:30.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-03T05:44:31.057 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-03-03T05:44:31.103 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-03-03T05:44:31.103 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T05:44:31.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-03-03T05:44:31.212 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:31.212 INFO:tasks.ceph:role: osd.0 2023-03-03T05:44:31.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com 2023-03-03T05:44:31.213 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T05:44:31.326 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:31.327 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:31.327 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:31.327 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:31.327 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:31.327 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:31.327 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:31.328 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:31.328 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:31.328 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:31.754 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2023-03-03T05:44:31.761 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2023-03-03T05:44:31.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-03-03T05:44:31.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-03-03T05:44:32.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-03-03T05:44:32.218 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:32.219 INFO:tasks.ceph:role: osd.1 2023-03-03T05:44:32.219 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com 2023-03-03T05:44:32.219 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T05:44:32.329 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:32.329 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:32.329 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:32.329 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:32.330 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:32.330 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:32.330 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:32.330 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:32.330 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:32.330 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:32.771 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2023-03-03T05:44:32.779 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2023-03-03T05:44:32.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-03-03T05:44:32.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-03-03T05:44:33.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-03-03T05:44:33.185 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:33.185 INFO:tasks.ceph:role: osd.2 2023-03-03T05:44:33.186 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com 2023-03-03T05:44:33.186 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T05:44:33.293 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:33.293 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:33.293 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:33.294 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:33.294 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:33.294 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:33.294 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:33.294 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:33.294 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:33.294 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:33.720 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2023-03-03T05:44:33.727 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2023-03-03T05:44:33.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-03-03T05:44:33.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-03-03T05:44:34.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-03-03T05:44:34.170 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:34.171 INFO:tasks.ceph:role: osd.3 2023-03-03T05:44:34.171 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com 2023-03-03T05:44:34.171 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T05:44:34.280 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:34.280 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:34.281 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:34.281 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:34.281 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:34.281 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:34.281 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:34.281 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:34.281 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:34.282 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:34.704 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2023-03-03T05:44:34.712 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2023-03-03T05:44:34.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-03-03T05:44:34.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-03-03T05:44:35.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:35.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:35.184+0000 7f7ef3456640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-03-03T05:44:35.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:35.187+0000 7f7ef3456640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-03-03T05:44:35.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:35.187+0000 7f7ef3456640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T05:44:35.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:35.187+0000 7f7ef3456640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T05:44:35.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:35.187+0000 7f7ef3456640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T05:44:35.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:35.193+0000 7f7ef3456640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-03-03T05:44:38.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-03T05:44:38.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:38.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:38.666+0000 7efeb1d62640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-03-03T05:44:38.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:38.666+0000 7efeb1d62640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-03-03T05:44:38.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:38.666+0000 7efeb1d62640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T05:44:38.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:38.667+0000 7efeb1d62640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T05:44:38.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:38.667+0000 7efeb1d62640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T05:44:38.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:38.668+0000 7efeb1d62640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-03-03T05:44:41.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-03T05:44:41.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:42.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:42.090+0000 7f36febd6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-03-03T05:44:42.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:42.091+0000 7f36febd6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-03-03T05:44:42.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:42.091+0000 7f36febd6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T05:44:42.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:42.091+0000 7f36febd6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T05:44:42.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:42.091+0000 7f36febd6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T05:44:42.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:42.093+0000 7f36febd6640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-03-03T05:44:45.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-03T05:44:45.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:45.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:45.274+0000 7ffa86d30640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-03-03T05:44:45.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:45.274+0000 7ffa86d30640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-03-03T05:44:45.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:45.274+0000 7ffa86d30640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T05:44:45.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:45.275+0000 7ffa86d30640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T05:44:45.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:45.275+0000 7ffa86d30640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T05:44:45.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:44:45.277+0000 7ffa86d30640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-03-03T05:44:48.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-03T05:44:48.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-03-03T05:44:48.343 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:48.344 INFO:tasks.ceph:role: osd.4 2023-03-03T05:44:48.344 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2023-03-03T05:44:48.344 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T05:44:48.422 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:48.424 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:48.424 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:48.424 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:48.425 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:48.848 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-03-03T05:44:48.850 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-03-03T05:44:48.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-03-03T05:44:49.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-03-03T05:44:49.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-03-03T05:44:49.120 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:49.121 INFO:tasks.ceph:role: osd.5 2023-03-03T05:44:49.121 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2023-03-03T05:44:49.121 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T05:44:49.196 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:49.197 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:49.197 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:49.197 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:49.197 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:49.614 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-03-03T05:44:49.616 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-03-03T05:44:49.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-03-03T05:44:49.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-03-03T05:44:49.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-03-03T05:44:49.842 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:49.842 INFO:tasks.ceph:role: osd.6 2023-03-03T05:44:49.842 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2023-03-03T05:44:49.843 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T05:44:49.918 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:49.918 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:49.918 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:49.918 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:49.919 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:49.919 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:49.919 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:49.919 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:49.919 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:49.919 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:50.367 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-03-03T05:44:50.369 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-03-03T05:44:50.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-03-03T05:44:50.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-03-03T05:44:50.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-03-03T05:44:50.631 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T05:44:50.631 INFO:tasks.ceph:role: osd.7 2023-03-03T05:44:50.631 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2023-03-03T05:44:50.632 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T05:44:50.706 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T05:44:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T05:44:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T05:44:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T05:44:50.707 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T05:44:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-03-03T05:44:50.707 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T05:44:50.707 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T05:44:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T05:44:50.707 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T05:44:51.141 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-03-03T05:44:51.142 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-03-03T05:44:51.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-03-03T05:44:51.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-03-03T05:44:51.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:51.459 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:51.454+0000 7fc42f2dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-03-03T05:44:51.459 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:51.455+0000 7fc42f2dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-03-03T05:44:51.459 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:51.455+0000 7fc42f2dd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T05:44:51.460 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:51.455+0000 7fc42f2dd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T05:44:51.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:51.455+0000 7fc42f2dd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T05:44:51.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:51.455+0000 7fc42f2dd640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-03-03T05:44:54.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-03T05:44:54.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:54.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:54.420+0000 7fdfd3cfd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-03-03T05:44:54.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:54.420+0000 7fdfd3cfd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-03-03T05:44:54.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:54.420+0000 7fdfd3cfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T05:44:54.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:54.420+0000 7fdfd3cfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T05:44:54.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:54.420+0000 7fdfd3cfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T05:44:54.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:54.421+0000 7fdfd3cfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-03-03T05:44:55.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:44:55.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:44:57.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-03-03T05:44:57.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:44:57.392 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:57.390+0000 7f694a3b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-03-03T05:44:57.392 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:57.390+0000 7f694a3b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-03-03T05:44:57.392 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:57.390+0000 7f694a3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T05:44:57.392 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:57.390+0000 7f694a3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T05:44:57.392 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:57.390+0000 7f694a3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T05:44:57.393 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:44:57.391+0000 7f694a3b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-03-03T05:45:00.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-03-03T05:45:00.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:45:00.371 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:00.369+0000 7f5170cfb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-03-03T05:45:00.371 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:00.369+0000 7f5170cfb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-03-03T05:45:00.371 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:00.369+0000 7f5170cfb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T05:45:00.371 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:00.369+0000 7f5170cfb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T05:45:00.372 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:00.369+0000 7f5170cfb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T05:45:00.372 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:00.370+0000 7f5170cfb640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-03-03T05:45:03.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-03T05:45:03.243 INFO:tasks.ceph:Reading keys from all nodes... 2023-03-03T05:45:03.244 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-03-03T05:45:03.350 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-03-03T05:45:03.451 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-03-03T05:45:03.550 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-03-03T05:45:03.649 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-03-03T05:45:03.748 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-03-03T05:45:03.847 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-03-03T05:45:03.946 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:03.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-03-03T05:45:04.046 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-03-03T05:45:04.074 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-03-03T05:45:04.141 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-03-03T05:45:04.207 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-03-03T05:45:04.275 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-03-03T05:45:04.341 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-03-03T05:45:04.408 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-03-03T05:45:04.475 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:04.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-03-03T05:45:04.541 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:04.542 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-03-03T05:45:04.610 INFO:tasks.ceph:Adding keys to all mons... 2023-03-03T05:45:04.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T05:45:04.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T05:45:04.641 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2023-03-03T05:45:04.641 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA9iQFkPq7cKBAAEfeuyRasjIrJaxwo28Hczg== 2023-03-03T05:45:04.641 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.a] 2023-03-03T05:45:04.641 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA9iQFkD7RbNxAAa94pUVdS2dc5E7Y9gSh0nw== 2023-03-03T05:45:04.641 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.c] 2023-03-03T05:45:04.642 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA+iQFkQvUYDBAA234yMWMDIdVMVas4dsWbdg== 2023-03-03T05:45:04.642 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.e] 2023-03-03T05:45:04.642 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA+iQFkOXh2HBAAae8yCoSMs41aKdrgR9nfEg== 2023-03-03T05:45:04.642 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-03-03T05:45:04.643 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBDiQFkdXkUCxAAvKboBqVBF/GF/z+bfbh4Rg== 2023-03-03T05:45:04.643 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-03-03T05:45:04.643 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBGiQFkphjPJxAAJw6+sGhV2cYbCuX24nMIFQ== 2023-03-03T05:45:04.643 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-03-03T05:45:04.643 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBKiQFk0QeDBRAAfdTNyIpuYuUDfrZAHjX/aw== 2023-03-03T05:45:04.643 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2023-03-03T05:45:04.643 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBNiQFkBxp1EBAAxretHsW8mKH1CeNfdW+S7g== 2023-03-03T05:45:04.644 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2023-03-03T05:45:04.644 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA9iQFkp3jkLBAAGiOYNrMv+Gr3uu/J0nFUew== 2023-03-03T05:45:04.644 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.b] 2023-03-03T05:45:04.644 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA+iQFk77IrJhAAzMwk8MeFoiGPt/1Hjm76lQ== 2023-03-03T05:45:04.644 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.d] 2023-03-03T05:45:04.644 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA+iQFkD8CaLRAA5+2zah8BKwqj9FOvUdhFWQ== 2023-03-03T05:45:04.644 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.f] 2023-03-03T05:45:04.645 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA+iQFk5xTsNBAATABwD9hvGs4GHc8SMeskjg== 2023-03-03T05:45:04.645 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2023-03-03T05:45:04.645 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBTiQFk+GUiGxAAuBUn+SY1Frn7fN562YLQew== 2023-03-03T05:45:04.645 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2023-03-03T05:45:04.645 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBWiQFkSKYSGRAAxhJ4/o9zc/fLrbZx6fxfog== 2023-03-03T05:45:04.645 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2023-03-03T05:45:04.645 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBZiQFkQzFMFxAAuaYlWLPZDoRTnwuJF/sPkw== 2023-03-03T05:45:04.646 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2023-03-03T05:45:04.646 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBciQFktdUMFhAA51+1Q3j6VK2EoHhLwBS4yw== 2023-03-03T05:45:04.646 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-03-03T05:45:04.646 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQA/iQFkneRdAxAAcca5wcTcgQhMOQr6WhMyHg== 2023-03-03T05:45:04.703 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.y] 2023-03-03T05:45:04.704 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA9iQFkPq7cKBAAEfeuyRasjIrJaxwo28Hczg== 2023-03-03T05:45:04.704 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2023-03-03T05:45:04.704 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA9iQFkD7RbNxAAa94pUVdS2dc5E7Y9gSh0nw== 2023-03-03T05:45:04.704 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2023-03-03T05:45:04.704 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA+iQFkQvUYDBAA234yMWMDIdVMVas4dsWbdg== 2023-03-03T05:45:04.704 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.e] 2023-03-03T05:45:04.704 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA+iQFkOXh2HBAAae8yCoSMs41aKdrgR9nfEg== 2023-03-03T05:45:04.705 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.0] 2023-03-03T05:45:04.705 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBDiQFkdXkUCxAAvKboBqVBF/GF/z+bfbh4Rg== 2023-03-03T05:45:04.705 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.1] 2023-03-03T05:45:04.705 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBGiQFkphjPJxAAJw6+sGhV2cYbCuX24nMIFQ== 2023-03-03T05:45:04.705 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.2] 2023-03-03T05:45:04.705 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBKiQFk0QeDBRAAfdTNyIpuYuUDfrZAHjX/aw== 2023-03-03T05:45:04.705 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.3] 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBNiQFkBxp1EBAAxretHsW8mKH1CeNfdW+S7g== 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.x] 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA9iQFkp3jkLBAAGiOYNrMv+Gr3uu/J0nFUew== 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA+iQFk77IrJhAAzMwk8MeFoiGPt/1Hjm76lQ== 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA+iQFkD8CaLRAA5+2zah8BKwqj9FOvUdhFWQ== 2023-03-03T05:45:04.706 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.f] 2023-03-03T05:45:04.707 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA+iQFk5xTsNBAATABwD9hvGs4GHc8SMeskjg== 2023-03-03T05:45:04.707 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.4] 2023-03-03T05:45:04.707 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBTiQFk+GUiGxAAuBUn+SY1Frn7fN562YLQew== 2023-03-03T05:45:04.707 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.5] 2023-03-03T05:45:04.707 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBWiQFkSKYSGRAAxhJ4/o9zc/fLrbZx6fxfog== 2023-03-03T05:45:04.707 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.6] 2023-03-03T05:45:04.707 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBZiQFkQzFMFxAAuaYlWLPZDoRTnwuJF/sPkw== 2023-03-03T05:45:04.708 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.7] 2023-03-03T05:45:04.708 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBciQFktdUMFhAA51+1Q3j6VK2EoHhLwBS4yw== 2023-03-03T05:45:04.708 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2023-03-03T05:45:04.708 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQA/iQFkneRdAxAAcca5wcTcgQhMOQr6WhMyHg== 2023-03-03T05:45:04.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T05:45:04.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T05:45:04.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:04.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:05.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:05.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:05.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:05.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:05.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:05.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T05:45:05.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T05:45:06.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:06.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:06.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:06.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:06.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:06.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T05:45:06.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:06.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:06.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:06.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:06.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:06.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:06.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:06.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T05:45:07.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T05:45:07.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T05:45:07.259 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-03-03T05:45:07.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-03-03T05:45:07.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T05:45:07.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-03-03T05:45:07.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-03-03T05:45:07.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T05:45:07.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-03-03T05:45:07.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-03-03T05:45:07.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T05:45:07.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-03-03T05:45:07.977 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:45:07.983 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T05:45:08.044 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-03-03T05:45:08.045 INFO:tasks.ceph.mon.a:Restarting daemon 2023-03-03T05:45:08.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-03-03T05:45:08.052 INFO:tasks.ceph.mon.a:Started 2023-03-03T05:45:08.052 INFO:tasks.ceph.mon.b:Restarting daemon 2023-03-03T05:45:08.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-03-03T05:45:08.076 INFO:tasks.ceph.mon.b:Started 2023-03-03T05:45:08.077 INFO:tasks.ceph.mon.c:Restarting daemon 2023-03-03T05:45:08.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-03-03T05:45:08.079 INFO:tasks.ceph.mon.c:Started 2023-03-03T05:45:08.080 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-03-03T05:45:08.080 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-03-03T05:45:08.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-03-03T05:45:08.086 INFO:tasks.ceph.mgr.y:Started 2023-03-03T05:45:08.087 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-03-03T05:45:08.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-03-03T05:45:08.089 INFO:tasks.ceph.mgr.x:Started 2023-03-03T05:45:08.089 INFO:tasks.ceph:Setting crush tunables to default 2023-03-03T05:45:08.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd crush tunables default 2023-03-03T05:45:08.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.343+0000 7f0d8e9ff700 1 Processor -- start 2023-03-03T05:45:08.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.343+0000 7f0d8e9ff700 1 -- start start 2023-03-03T05:45:08.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d8e9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d88153450 0x7f0d880cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:08.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d8e9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d8815a380 0x7f0d88159940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:08.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d8e9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d88159e80 con 0x7f0d880cf9a0 2023-03-03T05:45:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d8e9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d880cae70 con 0x7f0d8815a380 2023-03-03T05:45:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d8e9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d880cb110 con 0x7f0d88153450 2023-03-03T05:45:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d87fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d88153450 0x7f0d880cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d877fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d8815a380 0x7f0d88159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d87fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d88153450 0x7f0d880cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54972/0 (socket says 172.21.15.102:54972) 2023-03-03T05:45:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.345+0000 7f0d87fff700 1 -- 172.21.15.102:0/3382209653 learned_addr learned my addr 172.21.15.102:0/3382209653 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.346+0000 7f0d87fff700 1 -- 172.21.15.102:0/3382209653 >> v2:172.21.15.102:3300/0 conn(0x7f0d8815a380 msgr2=0x7f0d88159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:08.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.346+0000 7f0d87fff700 1 --2- 172.21.15.102:0/3382209653 >> v2:172.21.15.102:3300/0 conn(0x7f0d8815a380 0x7f0d88159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.346+0000 7f0d87fff700 1 -- 172.21.15.102:0/3382209653 >> v1:172.21.15.172:6789/0 conn(0x7f0d880cf9a0 legacy=0x7f0d880ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:08.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.346+0000 7f0d87fff700 1 -- 172.21.15.102:0/3382209653 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880cb3f0 con 0x7f0d88153450 2023-03-03T05:45:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.346+0000 7f0d87fff700 1 --2- 172.21.15.102:0/3382209653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d88153450 0x7f0d880cc1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d880ced50 tx=0x7f0d78009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c078ef5d85755f72 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d86ffd700 1 -- 172.21.15.102:0/3382209653 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78011830 con 0x7f0d88153450 2023-03-03T05:45:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d86ffd700 1 -- 172.21.15.102:0/3382209653 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d78011990 con 0x7f0d88153450 2023-03-03T05:45:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d86ffd700 1 -- 172.21.15.102:0/3382209653 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78011d20 con 0x7f0d88153450 2023-03-03T05:45:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/3382209653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d88153450 msgr2=0x7f0d880cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 --2- 172.21.15.102:0/3382209653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d88153450 0x7f0d880cc1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d880ced50 tx=0x7f0d78009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/3382209653 shutdown_connections 2023-03-03T05:45:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 --2- 172.21.15.102:0/3382209653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d88153450 0x7f0d880cc1b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 --2- 172.21.15.102:0/3382209653 >> v2:172.21.15.102:3300/0 conn(0x7f0d8815a380 0x7f0d88159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/3382209653 >> 172.21.15.102:0/3382209653 conn(0x7f0d88065fb0 msgr2=0x7f0d880663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/3382209653 shutdown_connections 2023-03-03T05:45:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.347+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/3382209653 wait complete. 2023-03-03T05:45:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.350+0000 7f0d8e9ff700 1 Processor -- start 2023-03-03T05:45:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.351+0000 7f0d8e9ff700 1 -- start start 2023-03-03T05:45:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d8e9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d880cf9a0 0x7f0d8814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d8e9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d88153450 0x7f0d881e2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:08.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d8e9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d8815a380 0x7f0d881e2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:08.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d8e9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d881f2130 con 0x7f0d88153450 2023-03-03T05:45:08.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d8e9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8814e7b0 con 0x7f0d8815a380 2023-03-03T05:45:08.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d8e9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8814fb30 con 0x7f0d880cf9a0 2023-03-03T05:45:08.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d87fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d880cf9a0 0x7f0d8814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:08.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d87fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d880cf9a0 0x7f0d8814cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57538/0 (socket says 172.21.15.102:57538) 2023-03-03T05:45:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d87fff700 1 -- 172.21.15.102:0/681056986 learned_addr learned my addr 172.21.15.102:0/681056986 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d877fe700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d88153450 0x7f0d881e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.352+0000 7f0d8cf9c700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d8815a380 0x7f0d881e2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d87fff700 1 -- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d8815a380 msgr2=0x7f0d881e2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d87fff700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d8815a380 0x7f0d881e2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d87fff700 1 -- 172.21.15.102:0/681056986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d88153450 msgr2=0x7f0d881e2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d87fff700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d88153450 0x7f0d881e2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d87fff700 1 -- 172.21.15.102:0/681056986 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78009550 con 0x7f0d880cf9a0 2023-03-03T05:45:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d87fff700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d880cf9a0 0x7f0d8814cc70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0d78000c00 tx=0x7f0d78004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d84ff9700 1 -- 172.21.15.102:0/681056986 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78004380 con 0x7f0d880cf9a0 2023-03-03T05:45:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d84ff9700 1 -- 172.21.15.102:0/681056986 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7801f070 con 0x7f0d880cf9a0 2023-03-03T05:45:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d880cb9b0 con 0x7f0d880cf9a0 2023-03-03T05:45:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d84ff9700 1 -- 172.21.15.102:0/681056986 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d780036a0 con 0x7f0d880cf9a0 2023-03-03T05:45:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.353+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d880cb6d0 con 0x7f0d880cf9a0 2023-03-03T05:45:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.354+0000 7f0d84ff9700 1 -- 172.21.15.102:0/681056986 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f0d7802a020 con 0x7f0d880cf9a0 2023-03-03T05:45:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.354+0000 7f0d84ff9700 1 -- 172.21.15.102:0/681056986 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0d7801a6b0 con 0x7f0d880cf9a0 2023-03-03T05:45:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.354+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d880cbc90 con 0x7f0d880cf9a0 2023-03-03T05:45:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.357+0000 7f0d84ff9700 1 -- 172.21.15.102:0/681056986 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f0d78015020 con 0x7f0d880cf9a0 2023-03-03T05:45:08.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.437+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0d881580b0 con 0x7f0d880cf9a0 2023-03-03T05:45:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.445+0000 7f0d84ff9700 1 -- 172.21.15.102:0/681056986 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0d780038d0 con 0x7f0d880cf9a0 2023-03-03T05:45:08.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.446+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d880cf9a0 msgr2=0x7f0d8814cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:08.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.446+0000 7f0d8e9ff700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d880cf9a0 0x7f0d8814cc70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0d78000c00 tx=0x7f0d78004750 comp rx=0 tx=0).stop 2023-03-03T05:45:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.447+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 shutdown_connections 2023-03-03T05:45:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.447+0000 7f0d8e9ff700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d880cf9a0 0x7f0d8814cc70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.447+0000 7f0d8e9ff700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d8815a380 0x7f0d881e2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.447+0000 7f0d8e9ff700 1 --2- 172.21.15.102:0/681056986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d88153450 0x7f0d881e2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.447+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 >> 172.21.15.102:0/681056986 conn(0x7f0d88065fb0 msgr2=0x7f0d8815b740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.447+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 shutdown_connections 2023-03-03T05:45:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:08.447+0000 7f0d8e9ff700 1 -- 172.21.15.102:0/681056986 wait complete. 2023-03-03T05:45:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2023-03-03T05:45:08.474 INFO:tasks.ceph:check_enable_crimson: False 2023-03-03T05:45:08.475 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-03-03T05:45:08.475 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:08.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-03-03T05:45:08.580 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:08.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-03-03T05:45:08.681 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:08.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-03-03T05:45:08.782 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:08.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-03-03T05:45:08.882 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:08.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-03-03T05:45:08.910 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:08.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-03-03T05:45:08.977 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:08.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-03-03T05:45:09.043 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-03-03T05:45:09.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-03-03T05:45:09.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new d23900ad-c97a-46fb-b261-8cf9f0f56dbf 0 2023-03-03T05:45:09.276 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.273+0000 7fc09c027700 1 Processor -- start 2023-03-03T05:45:09.276 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc09c027700 1 -- start start 2023-03-03T05:45:09.278 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc09c027700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc09406a300 0x7fc09406a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.278 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc09c027700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc0940c6770 0x7fc09406f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.278 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc09c027700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc094070230 con 0x7fc094159440 2023-03-03T05:45:09.278 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc09c027700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09406ac10 con 0x7fc09406a300 2023-03-03T05:45:09.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc09c027700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc094071cf0 con 0x7fc0940c6770 2023-03-03T05:45:09.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc0995c2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc0940c6770 0x7fc09406f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.274+0000 7fc0995c2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc0940c6770 0x7fc09406f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:55820/0 (socket says 172.21.15.172:55820) 2023-03-03T05:45:09.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc0995c2700 1 -- 172.21.15.172:0/3734912098 learned_addr learned my addr 172.21.15.172:0/3734912098 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:09.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc0995c2700 1 -- 172.21.15.172:0/3734912098 >> v2:172.21.15.102:3300/0 conn(0x7fc09406a300 msgr2=0x7fc09406a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:09.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc0995c2700 1 --2- 172.21.15.172:0/3734912098 >> v2:172.21.15.102:3300/0 conn(0x7fc09406a300 0x7fc09406a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.279 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc0995c2700 1 -- 172.21.15.172:0/3734912098 >> v1:172.21.15.102:6789/0 conn(0x7fc094159440 legacy=0x7fc09415c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc0995c2700 1 -- 172.21.15.172:0/3734912098 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc094071fd0 con 0x7fc0940c6770 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc0995c2700 1 --2- 172.21.15.172:0/3734912098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0940c6770 0x7fc09406f400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc09406c320 tx=0x7fc084009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=889af0a945f26384 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc098dc1700 1 -- 172.21.15.172:0/3734912098 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084004230 con 0x7fc0940c6770 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc098dc1700 1 -- 172.21.15.172:0/3734912098 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc084004390 con 0x7fc0940c6770 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc098dc1700 1 -- 172.21.15.172:0/3734912098 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0840036a0 con 0x7fc0940c6770 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc09c027700 1 -- 172.21.15.172:0/3734912098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0940c6770 msgr2=0x7fc09406f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.275+0000 7fc09c027700 1 --2- 172.21.15.172:0/3734912098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0940c6770 0x7fc09406f400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc09406c320 tx=0x7fc084009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.280 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.276+0000 7fc09c027700 1 -- 172.21.15.172:0/3734912098 shutdown_connections 2023-03-03T05:45:09.281 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.276+0000 7fc09c027700 1 --2- 172.21.15.172:0/3734912098 >> v2:172.21.15.102:3300/0 conn(0x7fc09406a300 0x7fc09406a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.281 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.276+0000 7fc09c027700 1 --2- 172.21.15.172:0/3734912098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0940c6770 0x7fc09406f400 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.281 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.276+0000 7fc09c027700 1 -- 172.21.15.172:0/3734912098 >> 172.21.15.172:0/3734912098 conn(0x7fc09406b410 msgr2=0x7fc09406b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:09.281 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.276+0000 7fc09c027700 1 -- 172.21.15.172:0/3734912098 shutdown_connections 2023-03-03T05:45:09.281 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.276+0000 7fc09c027700 1 -- 172.21.15.172:0/3734912098 wait complete. 2023-03-03T05:45:09.282 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:09.277+0000 7fc9d9f52200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T05:45:09.283 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.277+0000 7fc09c027700 1 Processor -- start 2023-03-03T05:45:09.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.277+0000 7fc09c027700 1 -- start start 2023-03-03T05:45:09.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.277+0000 7fc09c027700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc09406a300 0x7fc0941e5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.277+0000 7fc09c027700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0940c6770 0x7fc0940cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.277+0000 7fc09c027700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc094159440 0x7fc0941efc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09c027700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0940c7de0 con 0x7fc09406a300 2023-03-03T05:45:09.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09c027700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0940cafc0 con 0x7fc094159440 2023-03-03T05:45:09.284 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09c027700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0940caa00 con 0x7fc0940c6770 2023-03-03T05:45:09.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc0995c2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0940c6770 0x7fc0940cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09a5c4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc094159440 0x7fc0941efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc0995c2700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0940c6770 0x7fc0940cbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:50968/0 (socket says 172.21.15.172:50968) 2023-03-03T05:45:09.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc0995c2700 1 -- 172.21.15.172:0/192698779 learned_addr learned my addr 172.21.15.172:0/192698779 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:09.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc099dc3700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc09406a300 0x7fc0941e5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09a5c4700 1 -- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0940c6770 msgr2=0x7fc0940cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.285 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09a5c4700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0940c6770 0x7fc0940cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09a5c4700 1 -- 172.21.15.172:0/192698779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc09406a300 msgr2=0x7fc0941e5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09a5c4700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc09406a300 0x7fc0941e5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09a5c4700 1 -- 172.21.15.172:0/192698779 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084009550 con 0x7fc094159440 2023-03-03T05:45:09.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc0995c2700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0940c6770 0x7fc0940cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:09.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc099dc3700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc09406a300 0x7fc0941e5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T05:45:09.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09a5c4700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc094159440 0x7fc0941efc10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00be20 tx=0x7fc08c009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:09.286 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc08a7fc700 1 -- 172.21.15.172:0/192698779 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c00cc90 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc08a7fc700 1 -- 172.21.15.172:0/192698779 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c00ce30 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc094072590 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc08a7fc700 1 -- 172.21.15.172:0/192698779 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c014680 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.278+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0940722b0 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.279+0000 7fc08a7fc700 1 -- 172.21.15.172:0/192698779 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fc08c014ad0 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.279+0000 7fc08a7fc700 1 -- 172.21.15.172:0/192698779 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc08c01db70 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.279+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc094158f20 con 0x7fc094159440 2023-03-03T05:45:09.287 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.281+0000 7fc08a7fc700 1 -- 172.21.15.172:0/192698779 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fc08c022060 con 0x7fc094159440 2023-03-03T05:45:09.352 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:09.350+0000 7fc9d9f52200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T05:45:09.367 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.365+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d23900ad-c97a-46fb-b261-8cf9f0f56dbf", "id": 0} v 0) v1 -- 0x7fc0940cb170 con 0x7fc094159440 2023-03-03T05:45:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.373+0000 7fc08a7fc700 1 -- 172.21.15.172:0/192698779 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d23900ad-c97a-46fb-b261-8cf9f0f56dbf", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc0940cb170 con 0x7fc094159440 2023-03-03T05:45:09.376 INFO:teuthology.orchestra.run.smithi172.stdout:0 2023-03-03T05:45:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc094159440 msgr2=0x7fc0941efc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc094159440 0x7fc0941efc10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00be20 tx=0x7fc08c009ef0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 shutdown_connections 2023-03-03T05:45:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc094159440 0x7fc0941efc10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc09406a300 0x7fc0941e5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 --2- 172.21.15.172:0/192698779 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0940c6770 0x7fc0940cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 >> 172.21.15.172:0/192698779 conn(0x7fc09406b410 msgr2=0x7fc09414de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 shutdown_connections 2023-03-03T05:45:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.374+0000 7fc09c027700 1 -- 172.21.15.172:0/192698779 wait complete. 2023-03-03T05:45:09.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new ad558fcb-2250-48d2-8a46-7bce17eeb6f3 1 2023-03-03T05:45:09.450 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:09.448+0000 7fc9d9f52200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T05:45:09.511 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.509+0000 7f10953a9700 1 Processor -- start 2023-03-03T05:45:09.512 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.509+0000 7f10953a9700 1 -- start start 2023-03-03T05:45:09.514 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f10953a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1090153260 0x7f10900ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.514 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f10953a9700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f10900c9cb0 0x7f10900cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.514 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f10953a9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10900cf940 con 0x7f10900cac20 2023-03-03T05:45:09.514 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f10953a9700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10900cfe70 con 0x7f10900c9cb0 2023-03-03T05:45:09.514 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f10953a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1090158d70 con 0x7f1090153260 2023-03-03T05:45:09.514 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f10900cac20 0x7f109015c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:41488/0 (socket says 172.21.15.172:41488) 2023-03-03T05:45:09.514 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1090153260 0x7f10900ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108f7fe700 1 -- 172.21.15.172:0/2583890764 learned_addr learned my addr 172.21.15.172:0/2583890764 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-03-03T05:45:09.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108dffb700 1 -- 172.21.15.172:0/2583890764 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135473860 0 0) 0x7f10900cf940 con 0x7f10900cac20 2023-03-03T05:45:09.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108dffb700 1 -- 172.21.15.172:0/2583890764 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10780035b0 con 0x7f10900cac20 2023-03-03T05:45:09.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108e7fc700 1 --2- 172.21.15.172:0/2583890764 >> v2:172.21.15.102:3300/0 conn(0x7f10900c9cb0 0x7f10900cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108effd700 1 -- 172.21.15.172:0/2583890764 >> v2:172.21.15.102:3300/0 conn(0x7f10900c9cb0 msgr2=0x7f10900cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:09.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108effd700 1 --2- 172.21.15.172:0/2583890764 >> v2:172.21.15.102:3300/0 conn(0x7f10900c9cb0 0x7f10900cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108effd700 1 -- 172.21.15.172:0/2583890764 >> v1:172.21.15.172:6789/0 conn(0x7f10900cac20 legacy=0x7f109015c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108effd700 1 -- 172.21.15.172:0/2583890764 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10900cfb90 con 0x7f1090153260 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108e7fc700 1 --2- 172.21.15.172:0/2583890764 >> v2:172.21.15.102:3300/0 conn(0x7f10900c9cb0 0x7f10900cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108effd700 1 --2- 172.21.15.172:0/2583890764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1090153260 0x7f10900ca6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1084009840 tx=0x7f1084009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b31cb7d57933ac server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108dffb700 1 -- 172.21.15.172:0/2583890764 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084011830 con 0x7f1090153260 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108dffb700 1 -- 172.21.15.172:0/2583890764 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1084011990 con 0x7f1090153260 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.510+0000 7f108dffb700 1 -- 172.21.15.172:0/2583890764 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084011d20 con 0x7f1090153260 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 -- 172.21.15.172:0/2583890764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1090153260 msgr2=0x7f10900ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 --2- 172.21.15.172:0/2583890764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1090153260 0x7f10900ca6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1084009840 tx=0x7f1084009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 -- 172.21.15.172:0/2583890764 shutdown_connections 2023-03-03T05:45:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 --2- 172.21.15.172:0/2583890764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1090153260 0x7f10900ca6e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 --2- 172.21.15.172:0/2583890764 >> v2:172.21.15.102:3300/0 conn(0x7f10900c9cb0 0x7f10900cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 -- 172.21.15.172:0/2583890764 >> 172.21.15.172:0/2583890764 conn(0x7f1090066c00 msgr2=0x7f10900ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 -- 172.21.15.172:0/2583890764 shutdown_connections 2023-03-03T05:45:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.511+0000 7f10953a9700 1 -- 172.21.15.172:0/2583890764 wait complete. 2023-03-03T05:45:09.519 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.512+0000 7f10953a9700 1 Processor -- start 2023-03-03T05:45:09.519 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.512+0000 7f10953a9700 1 -- start start 2023-03-03T05:45:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f10953a9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10900cac20 0x7f10900cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f10953a9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090153260 0x7f10900c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f10953a9700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10900c3ae0 0x7f10901f1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f10953a9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900c7be0 con 0x7f1090153260 2023-03-03T05:45:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f10953a9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900c81a0 con 0x7f10900cac20 2023-03-03T05:45:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f10953a9700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1090073ea0 con 0x7f10900c3ae0 2023-03-03T05:45:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f108effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10900cac20 0x7f10900cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f108f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10900c3ae0 0x7f10901f1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f108effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10900cac20 0x7f10900cd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45254/0 (socket says 172.21.15.172:45254) 2023-03-03T05:45:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f108effd700 1 -- 172.21.15.172:0/438404473 learned_addr learned my addr 172.21.15.172:0/438404473 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f108f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10900c3ae0 0x7f10901f1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40012/0 (socket says 172.21.15.172:40012) 2023-03-03T05:45:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.513+0000 7f108e7fc700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090153260 0x7f10900c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f108f7fe700 1 -- 172.21.15.172:0/438404473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10900cac20 msgr2=0x7f10900cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f108f7fe700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10900cac20 0x7f10900cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f108f7fe700 1 -- 172.21.15.172:0/438404473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090153260 msgr2=0x7f10900c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f108f7fe700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090153260 0x7f10900c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f108f7fe700 1 -- 172.21.15.172:0/438404473 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10840094a0 con 0x7f10900c3ae0 2023-03-03T05:45:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f108f7fe700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10900c3ae0 0x7f10901f1d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f108800ca80 tx=0x7f108800cd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f10777fe700 1 -- 172.21.15.172:0/438404473 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108800a9e0 con 0x7f10900c3ae0 2023-03-03T05:45:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f10777fe700 1 -- 172.21.15.172:0/438404473 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108800ab40 con 0x7f10900c3ae0 2023-03-03T05:45:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1090159260 con 0x7f10900c3ae0 2023-03-03T05:45:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f10777fe700 1 -- 172.21.15.172:0/438404473 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088012400 con 0x7f10900c3ae0 2023-03-03T05:45:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.514+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1090158f80 con 0x7f10900c3ae0 2023-03-03T05:45:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.515+0000 7f10777fe700 1 -- 172.21.15.172:0/438404473 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f1088019030 con 0x7f10900c3ae0 2023-03-03T05:45:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.515+0000 7f10777fe700 1 -- 172.21.15.172:0/438404473 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f108801c7f0 con 0x7f10900c3ae0 2023-03-03T05:45:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.515+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1090159540 con 0x7f10900c3ae0 2023-03-03T05:45:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.517+0000 7f10777fe700 1 -- 172.21.15.172:0/438404473 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f1088021060 con 0x7f10900c3ae0 2023-03-03T05:45:09.538 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:09.537+0000 7f7c6fdc8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T05:45:09.603 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.600+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ad558fcb-2250-48d2-8a46-7bce17eeb6f3", "id": 1} v 0) v1 -- 0x7f10900c8310 con 0x7f10900c3ae0 2023-03-03T05:45:09.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.608+0000 7f10777fe700 1 -- 172.21.15.172:0/438404473 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad558fcb-2250-48d2-8a46-7bce17eeb6f3", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1088017070 con 0x7f10900c3ae0 2023-03-03T05:45:09.611 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-03-03T05:45:09.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10900c3ae0 msgr2=0x7f10901f1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10900c3ae0 0x7f10901f1d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f108800ca80 tx=0x7f108800cd90 comp rx=0 tx=0).stop 2023-03-03T05:45:09.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 shutdown_connections 2023-03-03T05:45:09.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10900c3ae0 0x7f10901f1d30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10900cac20 0x7f10900cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 --2- 172.21.15.172:0/438404473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090153260 0x7f10900c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 >> 172.21.15.172:0/438404473 conn(0x7f10900c9cb0 msgr2=0x7f10901576f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:09.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 shutdown_connections 2023-03-03T05:45:09.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.609+0000 7f10953a9700 1 -- 172.21.15.172:0/438404473 wait complete. 2023-03-03T05:45:09.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 086ff50f-40e5-4583-83eb-bc1cded78708 2 2023-03-03T05:45:09.643 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:09.641+0000 7f7c6fdc8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T05:45:09.651 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:09.648+0000 7fc9d9f52200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T05:45:09.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.742+0000 7fec32763700 1 Processor -- start 2023-03-03T05:45:09.745 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.742+0000 7fec32763700 1 -- start start 2023-03-03T05:45:09.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec32763700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fec2c0c9cb0 0x7fec2c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec32763700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c0cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec32763700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0cf940 con 0x7fec2c153260 2023-03-03T05:45:09.749 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec32763700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2c0cfe70 con 0x7fec2c0c9cb0 2023-03-03T05:45:09.749 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec32763700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c158d70 con 0x7fec2c0ca190 2023-03-03T05:45:09.749 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c0cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.749 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec2b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fec2c0c9cb0 0x7fec2c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.750 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c0cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40022/0 (socket says 172.21.15.172:40022) 2023-03-03T05:45:09.750 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 -- 172.21.15.172:0/4156493537 learned_addr learned my addr 172.21.15.172:0/4156493537 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:09.750 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 -- 172.21.15.172:0/4156493537 >> v2:172.21.15.172:3300/0 conn(0x7fec2c0c9cb0 msgr2=0x7fec2c0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:09.750 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 --2- 172.21.15.172:0/4156493537 >> v2:172.21.15.172:3300/0 conn(0x7fec2c0c9cb0 0x7fec2c0cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 -- 172.21.15.172:0/4156493537 >> v1:172.21.15.102:6789/0 conn(0x7fec2c153260 legacy=0x7fec2c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:09.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 -- 172.21.15.172:0/4156493537 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0cfb90 con 0x7fec2c0ca190 2023-03-03T05:45:09.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec30d00700 1 --2- 172.21.15.172:0/4156493537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c0cbdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fec24009e50 tx=0x7fec2400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b957d37384a58f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:09.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.743+0000 7fec2affd700 1 -- 172.21.15.172:0/4156493537 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2400e070 con 0x7fec2c0ca190 2023-03-03T05:45:09.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec2affd700 1 -- 172.21.15.172:0/4156493537 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec24003a20 con 0x7fec2c0ca190 2023-03-03T05:45:09.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec2affd700 1 -- 172.21.15.172:0/4156493537 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24003dd0 con 0x7fec2c0ca190 2023-03-03T05:45:09.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 -- 172.21.15.172:0/4156493537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 msgr2=0x7fec2c0cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 --2- 172.21.15.172:0/4156493537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c0cbdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fec24009e50 tx=0x7fec2400b330 comp rx=0 tx=0).stop 2023-03-03T05:45:09.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 -- 172.21.15.172:0/4156493537 shutdown_connections 2023-03-03T05:45:09.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 --2- 172.21.15.172:0/4156493537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c0cbdc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 --2- 172.21.15.172:0/4156493537 >> v2:172.21.15.172:3300/0 conn(0x7fec2c0c9cb0 0x7fec2c0cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 -- 172.21.15.172:0/4156493537 >> 172.21.15.172:0/4156493537 conn(0x7fec2c066c00 msgr2=0x7fec2c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:09.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 -- 172.21.15.172:0/4156493537 shutdown_connections 2023-03-03T05:45:09.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.744+0000 7fec32763700 1 -- 172.21.15.172:0/4156493537 wait complete. 2023-03-03T05:45:09.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.745+0000 7fec32763700 1 Processor -- start 2023-03-03T05:45:09.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec32763700 1 -- start start 2023-03-03T05:45:09.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec32763700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec32763700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c153260 0x7fec2c1516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec32763700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec2c158040 0x7fec2c154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:09.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec32763700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c15a0b0 con 0x7fec2c153260 2023-03-03T05:45:09.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec32763700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c1ee990 con 0x7fec2c158040 2023-03-03T05:45:09.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec32763700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c1eec10 con 0x7fec2c0ca190 2023-03-03T05:45:09.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec30d00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec2c158040 0x7fec2c154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec2bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec30d00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec2c158040 0x7fec2c154a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45274/0 (socket says 172.21.15.172:45274) 2023-03-03T05:45:09.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec30d00700 1 -- 172.21.15.172:0/1916368099 learned_addr learned my addr 172.21.15.172:0/1916368099 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:09.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.746+0000 7fec2bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40036/0 (socket says 172.21.15.172:40036) 2023-03-03T05:45:09.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec2b7fe700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c153260 0x7fec2c1516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:09.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec30d00700 1 -- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 msgr2=0x7fec2c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec30d00700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec30d00700 1 -- 172.21.15.172:0/1916368099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c153260 msgr2=0x7fec2c1516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec30d00700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c153260 0x7fec2c1516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec30d00700 1 -- 172.21.15.172:0/1916368099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec24009b30 con 0x7fec2c158040 2023-03-03T05:45:09.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec2b7fe700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c153260 0x7fec2c1516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T05:45:09.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec2bfff700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:09.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec30d00700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec2c158040 0x7fec2c154a90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fec2c158eb0 tx=0x7fec24004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:09.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec28ff9700 1 -- 172.21.15.172:0/1916368099 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24004440 con 0x7fec2c158040 2023-03-03T05:45:09.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec28ff9700 1 -- 172.21.15.172:0/1916368099 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec240092e0 con 0x7fec2c158040 2023-03-03T05:45:09.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec28ff9700 1 -- 172.21.15.172:0/1916368099 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24004800 con 0x7fec2c158040 2023-03-03T05:45:09.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c159260 con 0x7fec2c158040 2023-03-03T05:45:09.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c158f80 con 0x7fec2c158040 2023-03-03T05:45:09.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec28ff9700 1 -- 172.21.15.172:0/1916368099 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fec24022030 con 0x7fec2c158040 2023-03-03T05:45:09.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.747+0000 7fec28ff9700 1 -- 172.21.15.172:0/1916368099 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fec24012850 con 0x7fec2c158040 2023-03-03T05:45:09.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.748+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c159540 con 0x7fec2c158040 2023-03-03T05:45:09.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.750+0000 7fec28ff9700 1 -- 172.21.15.172:0/1916368099 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fec24020070 con 0x7fec2c158040 2023-03-03T05:45:09.767 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:09.757+0000 7f7c6fdc8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T05:45:09.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.832+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "086ff50f-40e5-4583-83eb-bc1cded78708", "id": 2} v 0) v1 -- 0x7fec2c0b18d0 con 0x7fec2c158040 2023-03-03T05:45:09.841 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.838+0000 7fec28ff9700 1 -- 172.21.15.172:0/1916368099 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "086ff50f-40e5-4583-83eb-bc1cded78708", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fec240330b0 con 0x7fec2c158040 2023-03-03T05:45:09.841 INFO:teuthology.orchestra.run.smithi172.stdout:2 2023-03-03T05:45:09.842 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec2c158040 msgr2=0x7fec2c154a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:09.842 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec2c158040 0x7fec2c154a90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fec2c158eb0 tx=0x7fec24004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.842 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 shutdown_connections 2023-03-03T05:45:09.843 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fec2c0ca190 0x7fec2c1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.843 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fec2c158040 0x7fec2c154a90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.843 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 --2- 172.21.15.172:0/1916368099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c153260 0x7fec2c1516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:09.843 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 >> 172.21.15.172:0/1916368099 conn(0x7fec2c0c9cb0 msgr2=0x7fec2c0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:09.843 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 shutdown_connections 2023-03-03T05:45:09.843 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:09.839+0000 7fec32763700 1 -- 172.21.15.172:0/1916368099 wait complete. 2023-03-03T05:45:09.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 35415ce3-4ea3-4de5-921b-cf70bacdc9f6 3 2023-03-03T05:45:10.014 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.012+0000 7f043232f700 1 Processor -- start 2023-03-03T05:45:10.014 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.012+0000 7f043232f700 1 -- start start 2023-03-03T05:45:10.017 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.012+0000 7f043232f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f042c150ea0 0x7f042c06b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.018 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.012+0000 7f043232f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.018 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f043232f700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042c0c5010 con 0x7f042c06a300 2023-03-03T05:45:10.018 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f043232f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042c0c5c50 con 0x7f042c150ea0 2023-03-03T05:45:10.018 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f043232f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0c5e60 con 0x7f042c157230 2023-03-03T05:45:10.018 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f042bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f042c150ea0 0x7f042c06b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.018 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f04308cc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.018 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f042bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f042c150ea0 0x7f042c06b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45276/0 (socket says 172.21.15.172:45276) 2023-03-03T05:45:10.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f04308cc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c06b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40052/0 (socket says 172.21.15.172:40052) 2023-03-03T05:45:10.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f042bfff700 1 -- 172.21.15.172:0/2985399760 learned_addr learned my addr 172.21.15.172:0/2985399760 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f04308cc700 1 -- 172.21.15.172:0/2985399760 >> v2:172.21.15.172:3300/0 conn(0x7f042c150ea0 msgr2=0x7f042c06b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f04308cc700 1 --2- 172.21.15.172:0/2985399760 >> v2:172.21.15.172:3300/0 conn(0x7f042c150ea0 0x7f042c06b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f04308cc700 1 -- 172.21.15.172:0/2985399760 >> v1:172.21.15.102:6789/0 conn(0x7f042c06a300 legacy=0x7f042c156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f04308cc700 1 -- 172.21.15.172:0/2985399760 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c0c6140 con 0x7f042c157230 2023-03-03T05:45:10.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f04308cc700 1 --2- 172.21.15.172:0/2985399760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c06b810 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0424009d90 tx=0x7f042400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37906e81f84d33df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f042affd700 1 -- 172.21.15.172:0/2985399760 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042400e070 con 0x7f042c157230 2023-03-03T05:45:10.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f042affd700 1 -- 172.21.15.172:0/2985399760 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042400bae0 con 0x7f042c157230 2023-03-03T05:45:10.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.013+0000 7f042affd700 1 -- 172.21.15.172:0/2985399760 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042400be90 con 0x7f042c157230 2023-03-03T05:45:10.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 -- 172.21.15.172:0/2985399760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 msgr2=0x7f042c06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 --2- 172.21.15.172:0/2985399760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c06b810 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0424009d90 tx=0x7f042400b330 comp rx=0 tx=0).stop 2023-03-03T05:45:10.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 -- 172.21.15.172:0/2985399760 shutdown_connections 2023-03-03T05:45:10.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 --2- 172.21.15.172:0/2985399760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c06b810 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.021 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 --2- 172.21.15.172:0/2985399760 >> v2:172.21.15.172:3300/0 conn(0x7f042c150ea0 0x7f042c06b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.021 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 -- 172.21.15.172:0/2985399760 >> 172.21.15.172:0/2985399760 conn(0x7f042c06f7a0 msgr2=0x7f042c0cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.021 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 -- 172.21.15.172:0/2985399760 shutdown_connections 2023-03-03T05:45:10.021 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.014+0000 7f043232f700 1 -- 172.21.15.172:0/2985399760 wait complete. 2023-03-03T05:45:10.021 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.015+0000 7f043232f700 1 Processor -- start 2023-03-03T05:45:10.021 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.015+0000 7f043232f700 1 -- start start 2023-03-03T05:45:10.023 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f043232f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f042c06a300 0x7f042c152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.023 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f043232f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042c150ea0 0x7f042c14eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.024 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f043232f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c1efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.024 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f043232f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c151830 con 0x7f042c150ea0 2023-03-03T05:45:10.024 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f043232f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c1f0d40 con 0x7f042c06a300 2023-03-03T05:45:10.024 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f043232f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c1ec840 con 0x7f042c157230 2023-03-03T05:45:10.024 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c1efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.024 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f042bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f042c06a300 0x7f042c152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.024 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c1efdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40062/0 (socket says 172.21.15.172:40062) 2023-03-03T05:45:10.025 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 -- 172.21.15.172:0/577584856 learned_addr learned my addr 172.21.15.172:0/577584856 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.025 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f042b7fe700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042c150ea0 0x7f042c14eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.025 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 -- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f042c06a300 msgr2=0x7f042c152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.025 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f042c06a300 0x7f042c152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.025 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 -- 172.21.15.172:0/577584856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042c150ea0 msgr2=0x7f042c14eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.026 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042c150ea0 0x7f042c14eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.026 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f04308cc700 1 -- 172.21.15.172:0/577584856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0424009a70 con 0x7f042c157230 2023-03-03T05:45:10.026 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f042bfff700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f042c06a300 0x7f042c152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:10.026 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.016+0000 7f042b7fe700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042c150ea0 0x7f042c14eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T05:45:10.026 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f04308cc700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c1efdb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0424004000 tx=0x7f0424004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.026 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f0428ff9700 1 -- 172.21.15.172:0/577584856 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04240045f0 con 0x7f042c157230 2023-03-03T05:45:10.026 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f0428ff9700 1 -- 172.21.15.172:0/577584856 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0424004750 con 0x7f042c157230 2023-03-03T05:45:10.027 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042c0c4ca0 con 0x7f042c157230 2023-03-03T05:45:10.027 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f0428ff9700 1 -- 172.21.15.172:0/577584856 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424012430 con 0x7f042c157230 2023-03-03T05:45:10.027 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042c0c49c0 con 0x7f042c157230 2023-03-03T05:45:10.027 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f0428ff9700 1 -- 172.21.15.172:0/577584856 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f0424012880 con 0x7f042c157230 2023-03-03T05:45:10.027 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.017+0000 7f0428ff9700 1 -- 172.21.15.172:0/577584856 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f042401b3a0 con 0x7f042c157230 2023-03-03T05:45:10.027 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.018+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042c065fb0 con 0x7f042c157230 2023-03-03T05:45:10.028 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.020+0000 7f0428ff9700 1 -- 172.21.15.172:0/577584856 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f0424012da0 con 0x7f042c157230 2023-03-03T05:45:10.028 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:10.023+0000 7f7c6fdc8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T05:45:10.104 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.102+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd new", "uuid": "35415ce3-4ea3-4de5-921b-cf70bacdc9f6", "id": 3} v 0) v1 -- 0x7f042c06ad60 con 0x7f042c157230 2023-03-03T05:45:10.110 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.108+0000 7f0428ff9700 1 -- 172.21.15.172:0/577584856 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35415ce3-4ea3-4de5-921b-cf70bacdc9f6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f042401c540 con 0x7f042c157230 2023-03-03T05:45:10.111 INFO:teuthology.orchestra.run.smithi172.stdout:3 2023-03-03T05:45:10.112 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 msgr2=0x7f042c1efdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.113 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c1efdb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0424004000 tx=0x7f0424004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.113 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 shutdown_connections 2023-03-03T05:45:10.113 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f042c157230 0x7f042c1efdb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.113 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f042c06a300 0x7f042c152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.114 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 --2- 172.21.15.172:0/577584856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042c150ea0 0x7f042c14eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.114 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 >> 172.21.15.172:0/577584856 conn(0x7f042c06f7a0 msgr2=0x7f042c14fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.114 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 shutdown_connections 2023-03-03T05:45:10.114 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.109+0000 7f043232f700 1 -- 172.21.15.172:0/577584856 wait complete. 2023-03-03T05:45:10.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 873b86f0-b407-479c-9074-12767a5419cc 4 2023-03-03T05:45:10.246 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.243+0000 7fcd58294700 1 Processor -- start 2023-03-03T05:45:10.246 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.243+0000 7fcd58294700 1 -- start start 2023-03-03T05:45:10.249 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.244+0000 7fcd58294700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcd50151120 0x7fcd5006fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.249 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.244+0000 7fcd58294700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500b1ac0 0x7fcd500c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.249 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.244+0000 7fcd58294700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5006e740 con 0x7fcd50150d50 2023-03-03T05:45:10.249 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.244+0000 7fcd58294700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5015ac90 con 0x7fcd50151120 2023-03-03T05:45:10.250 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.244+0000 7fcd58294700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5015af70 con 0x7fcd500b1ac0 2023-03-03T05:45:10.250 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.244+0000 7fcd56831700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500b1ac0 0x7fcd500c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.250 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd56831700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500b1ac0 0x7fcd500c39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40068/0 (socket says 172.21.15.172:40068) 2023-03-03T05:45:10.250 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd56831700 1 -- 172.21.15.172:0/2908156265 learned_addr learned my addr 172.21.15.172:0/2908156265 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.250 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd56831700 1 -- 172.21.15.172:0/2908156265 >> v2:172.21.15.102:3300/0 conn(0x7fcd50151120 msgr2=0x7fcd5006fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.251 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd5582f700 1 --2- 172.21.15.172:0/2908156265 >> v2:172.21.15.102:3300/0 conn(0x7fcd50151120 0x7fcd5006fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.251 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd56831700 1 --2- 172.21.15.172:0/2908156265 >> v2:172.21.15.102:3300/0 conn(0x7fcd50151120 0x7fcd5006fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.251 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd56831700 1 -- 172.21.15.172:0/2908156265 >> v1:172.21.15.102:6789/0 conn(0x7fcd50150d50 legacy=0x7fcd5015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.251 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd56831700 1 -- 172.21.15.172:0/2908156265 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd50070c50 con 0x7fcd500b1ac0 2023-03-03T05:45:10.251 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd56831700 1 --2- 172.21.15.172:0/2908156265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500b1ac0 0x7fcd500c39a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009e50 tx=0x7fcd4800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d76cadc2e25870d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.251 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd5502e700 1 -- 172.21.15.172:0/2908156265 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800e070 con 0x7fcd500b1ac0 2023-03-03T05:45:10.252 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd5502e700 1 -- 172.21.15.172:0/2908156265 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd48003a20 con 0x7fcd500b1ac0 2023-03-03T05:45:10.252 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.245+0000 7fcd5502e700 1 -- 172.21.15.172:0/2908156265 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48003dd0 con 0x7fcd500b1ac0 2023-03-03T05:45:10.252 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 -- 172.21.15.172:0/2908156265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500b1ac0 msgr2=0x7fcd500c39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.252 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 --2- 172.21.15.172:0/2908156265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500b1ac0 0x7fcd500c39a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009e50 tx=0x7fcd4800b330 comp rx=0 tx=0).stop 2023-03-03T05:45:10.252 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 -- 172.21.15.172:0/2908156265 shutdown_connections 2023-03-03T05:45:10.253 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 --2- 172.21.15.172:0/2908156265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500b1ac0 0x7fcd500c39a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.253 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 --2- 172.21.15.172:0/2908156265 >> v2:172.21.15.102:3300/0 conn(0x7fcd50151120 0x7fcd5006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.253 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 -- 172.21.15.172:0/2908156265 >> 172.21.15.172:0/2908156265 conn(0x7fcd50066c00 msgr2=0x7fcd50070540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.253 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 -- 172.21.15.172:0/2908156265 shutdown_connections 2023-03-03T05:45:10.255 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.246+0000 7fcd58294700 1 -- 172.21.15.172:0/2908156265 wait complete. 2023-03-03T05:45:10.258 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.248+0000 7fcd58294700 1 Processor -- start 2023-03-03T05:45:10.258 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.248+0000 7fcd58294700 1 -- start start 2023-03-03T05:45:10.258 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd58294700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd50150d50 0x7fcd50143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.258 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd58294700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd50151120 0x7fcd501ecda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.259 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd58294700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5014bd90 0x7fcd501597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.259 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd58294700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd501e3f80 con 0x7fcd50150d50 2023-03-03T05:45:10.259 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd58294700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50143ea0 con 0x7fcd5014bd90 2023-03-03T05:45:10.260 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd58294700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd501e3ca0 con 0x7fcd50151120 2023-03-03T05:45:10.260 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd50151120 0x7fcd501ecda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.260 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd50151120 0x7fcd501ecda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40072/0 (socket says 172.21.15.172:40072) 2023-03-03T05:45:10.260 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 -- 172.21.15.172:0/3310486854 learned_addr learned my addr 172.21.15.172:0/3310486854 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.260 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd56831700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5014bd90 0x7fcd501597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.261 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd56030700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd50150d50 0x7fcd50143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.261 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 -- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5014bd90 msgr2=0x7fcd501597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.261 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5014bd90 0x7fcd501597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.261 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 -- 172.21.15.172:0/3310486854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd50150d50 msgr2=0x7fcd50143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.261 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd50150d50 0x7fcd50143960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.262 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 -- 172.21.15.172:0/3310486854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48009b30 con 0x7fcd50151120 2023-03-03T05:45:10.262 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd56030700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd50150d50 0x7fcd50143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:10.262 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.249+0000 7fcd5582f700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd50151120 0x7fcd501ecda0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000c820 tx=0x7fcd4000cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.262 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.250+0000 7fcd46ffd700 1 -- 172.21.15.172:0/3310486854 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40011800 con 0x7fcd50151120 2023-03-03T05:45:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.250+0000 7fcd46ffd700 1 -- 172.21.15.172:0/3310486854 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd40011960 con 0x7fcd50151120 2023-03-03T05:45:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.250+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd50071210 con 0x7fcd50151120 2023-03-03T05:45:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.250+0000 7fcd46ffd700 1 -- 172.21.15.172:0/3310486854 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40011cf0 con 0x7fcd50151120 2023-03-03T05:45:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.250+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd50070f30 con 0x7fcd50151120 2023-03-03T05:45:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.251+0000 7fcd46ffd700 1 -- 172.21.15.172:0/3310486854 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fcd40021070 con 0x7fcd50151120 2023-03-03T05:45:10.264 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.251+0000 7fcd46ffd700 1 -- 172.21.15.172:0/3310486854 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fcd40018870 con 0x7fcd50151120 2023-03-03T05:45:10.264 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.251+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5015a870 con 0x7fcd50151120 2023-03-03T05:45:10.264 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.253+0000 7fcd46ffd700 1 -- 172.21.15.172:0/3310486854 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fcd4001f060 con 0x7fcd50151120 2023-03-03T05:45:10.371 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.369+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd new", "uuid": "873b86f0-b407-479c-9074-12767a5419cc", "id": 4} v 0) v1 -- 0x7fcd500677f0 con 0x7fcd50151120 2023-03-03T05:45:10.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.376+0000 7fcd46ffd700 1 -- 172.21.15.172:0/3310486854 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "873b86f0-b407-479c-9074-12767a5419cc", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcd40015070 con 0x7fcd50151120 2023-03-03T05:45:10.379 INFO:teuthology.orchestra.run.smithi172.stdout:4 2023-03-03T05:45:10.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd50151120 msgr2=0x7fcd501ecda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd50151120 0x7fcd501ecda0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000c820 tx=0x7fcd4000cb30 comp rx=0 tx=0).stop 2023-03-03T05:45:10.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 shutdown_connections 2023-03-03T05:45:10.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5014bd90 0x7fcd501597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd50150d50 0x7fcd50143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 --2- 172.21.15.172:0/3310486854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd50151120 0x7fcd501ecda0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 >> 172.21.15.172:0/3310486854 conn(0x7fcd500b1ac0 msgr2=0x7fcd5014f440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 shutdown_connections 2023-03-03T05:45:10.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.377+0000 7fcd58294700 1 -- 172.21.15.172:0/3310486854 wait complete. 2023-03-03T05:45:10.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 312da3e9-4358-4b98-bb71-8ef9d455a191 5 2023-03-03T05:45:10.451 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:10.449+0000 7fc9d9f52200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T05:45:10.513 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63ad4bf700 1 Processor -- start 2023-03-03T05:45:10.513 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63ad4bf700 1 -- start start 2023-03-03T05:45:10.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63ad4bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80ccc40 0x7f63a80cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.515 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63ad4bf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63a80ce410 0x7f63a8163720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63ad4bf700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a8153630 con 0x7f63a80cd760 2023-03-03T05:45:10.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63ad4bf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a80ca7a0 con 0x7f63a80ce410 2023-03-03T05:45:10.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63ad4bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a80cd550 con 0x7f63a80ccc40 2023-03-03T05:45:10.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.511+0000 7f63a6ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80ccc40 0x7f63a80cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a77fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63a80ce410 0x7f63a8163720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a6ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80ccc40 0x7f63a80cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40076/0 (socket says 172.21.15.172:40076) 2023-03-03T05:45:10.516 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a6ffd700 1 -- 172.21.15.172:0/3258412061 learned_addr learned my addr 172.21.15.172:0/3258412061 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a77fe700 1 -- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80ccc40 msgr2=0x7f63a80cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a77fe700 1 --2- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80ccc40 0x7f63a80cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a77fe700 1 -- 172.21.15.172:0/3258412061 >> v1:172.21.15.102:6789/0 conn(0x7f63a80cd760 legacy=0x7f63a80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a77fe700 1 -- 172.21.15.172:0/3258412061 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a815d1d0 con 0x7f63a80ce410 2023-03-03T05:45:10.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a6ffd700 1 --2- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80ccc40 0x7f63a80cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:10.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a77fe700 1 --2- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ce410 0x7f63a8163720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f63a0009d90 tx=0x7f63a000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81143c44aab2c93f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.517 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a5ffb700 1 -- 172.21.15.172:0/3258412061 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a000e070 con 0x7f63a80ce410 2023-03-03T05:45:10.518 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a5ffb700 1 -- 172.21.15.172:0/3258412061 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a000bae0 con 0x7f63a80ce410 2023-03-03T05:45:10.518 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.512+0000 7f63a5ffb700 1 -- 172.21.15.172:0/3258412061 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a000be90 con 0x7f63a80ce410 2023-03-03T05:45:10.518 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 -- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ce410 msgr2=0x7f63a8163720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.518 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 --2- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ce410 0x7f63a8163720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f63a0009d90 tx=0x7f63a000b330 comp rx=0 tx=0).stop 2023-03-03T05:45:10.518 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 -- 172.21.15.172:0/3258412061 shutdown_connections 2023-03-03T05:45:10.518 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 --2- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ce410 0x7f63a8163720 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.518 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 --2- 172.21.15.172:0/3258412061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80ccc40 0x7f63a80cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.519 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 -- 172.21.15.172:0/3258412061 >> 172.21.15.172:0/3258412061 conn(0x7f63a8065fb0 msgr2=0x7f63a80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.519 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 -- 172.21.15.172:0/3258412061 shutdown_connections 2023-03-03T05:45:10.519 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.513+0000 7f63ad4bf700 1 -- 172.21.15.172:0/3258412061 wait complete. 2023-03-03T05:45:10.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.514+0000 7f63ad4bf700 1 Processor -- start 2023-03-03T05:45:10.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.514+0000 7f63ad4bf700 1 -- start start 2023-03-03T05:45:10.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63ad4bf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ccc40 0x7f63a8156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63ad4bf700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80cd760 0x7f63a815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63ad4bf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a80ce410 0x7f63a8158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63ad4bf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a815a840 con 0x7f63a80ce410 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63ad4bf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8157070 con 0x7f63a80ccc40 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63ad4bf700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a815b1a0 con 0x7f63a80cd760 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ccc40 0x7f63a8156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ccc40 0x7f63a8156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45318/0 (socket says 172.21.15.172:45318) 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 -- 172.21.15.172:0/2424129796 learned_addr learned my addr 172.21.15.172:0/2424129796 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a67fc700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80cd760 0x7f63a815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a77fe700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a80ce410 0x7f63a8158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 -- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80cd760 msgr2=0x7f63a815ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80cd760 0x7f63a815ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 -- 172.21.15.172:0/2424129796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a80ce410 msgr2=0x7f63a8158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a80ce410 0x7f63a8158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 -- 172.21.15.172:0/2424129796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6398009550 con 0x7f63a80ccc40 2023-03-03T05:45:10.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a77fe700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a80ce410 0x7f63a8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:10.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f63a6ffd700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ccc40 0x7f63a8156b30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f639800ebe0 tx=0x7f639800eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f638f7fe700 1 -- 172.21.15.172:0/2424129796 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639800c850 con 0x7f63a80ccc40 2023-03-03T05:45:10.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.515+0000 7f638f7fe700 1 -- 172.21.15.172:0/2424129796 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639800c9b0 con 0x7f63a80ccc40 2023-03-03T05:45:10.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.516+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a0009a70 con 0x7f63a80ccc40 2023-03-03T05:45:10.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.516+0000 7f638f7fe700 1 -- 172.21.15.172:0/2424129796 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639800cd20 con 0x7f63a80ccc40 2023-03-03T05:45:10.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.516+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a81594b0 con 0x7f63a80ccc40 2023-03-03T05:45:10.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.516+0000 7f638f7fe700 1 -- 172.21.15.172:0/2424129796 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f6398023070 con 0x7f63a80ccc40 2023-03-03T05:45:10.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.516+0000 7f638f7fe700 1 -- 172.21.15.172:0/2424129796 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f6398013d30 con 0x7f63a80ccc40 2023-03-03T05:45:10.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.517+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a81591d0 con 0x7f63a80ccc40 2023-03-03T05:45:10.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.519+0000 7f638f7fe700 1 -- 172.21.15.172:0/2424129796 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f6398021060 con 0x7f63a80ccc40 2023-03-03T05:45:10.599 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:10.597+0000 7fc9d9f52200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T05:45:10.604 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.602+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "312da3e9-4358-4b98-bb71-8ef9d455a191", "id": 5} v 0) v1 -- 0x7f63a8159790 con 0x7f63a80ccc40 2023-03-03T05:45:10.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.609+0000 7f638f7fe700 1 -- 172.21.15.172:0/2424129796 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "312da3e9-4358-4b98-bb71-8ef9d455a191", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6398017070 con 0x7f63a80ccc40 2023-03-03T05:45:10.614 INFO:teuthology.orchestra.run.smithi172.stdout:5 2023-03-03T05:45:10.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ccc40 msgr2=0x7f63a8156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ccc40 0x7f63a8156b30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f639800ebe0 tx=0x7f639800eef0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 shutdown_connections 2023-03-03T05:45:10.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a80ce410 0x7f63a8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63a80ccc40 0x7f63a8156b30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 --2- 172.21.15.172:0/2424129796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f63a80cd760 0x7f63a815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 >> 172.21.15.172:0/2424129796 conn(0x7f63a8065fb0 msgr2=0x7f63a8151bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 shutdown_connections 2023-03-03T05:45:10.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.610+0000 7f63ad4bf700 1 -- 172.21.15.172:0/2424129796 wait complete. 2023-03-03T05:45:10.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new c9e44237-2654-4425-95bd-31ee7706f4ff 6 2023-03-03T05:45:10.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.749+0000 7f2e6c100700 1 Processor -- start 2023-03-03T05:45:10.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.749+0000 7f2e6c100700 1 -- start start 2023-03-03T05:45:10.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6c100700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e641530a0 0x7f2e641594a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6c100700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64163650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6c100700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e640cd870 con 0x7f2e640c9220 2023-03-03T05:45:10.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6c100700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6406b410 con 0x7f2e641530a0 2023-03-03T05:45:10.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6c100700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64158d90 con 0x7f2e640c79f0 2023-03-03T05:45:10.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6a69d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64163650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6a69d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64163650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40096/0 (socket says 172.21.15.172:40096) 2023-03-03T05:45:10.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6a69d700 1 -- 172.21.15.172:0/3092425727 learned_addr learned my addr 172.21.15.172:0/3092425727 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e69e9c700 1 --2- 172.21.15.172:0/3092425727 >> v2:172.21.15.172:3300/0 conn(0x7f2e641530a0 0x7f2e641594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e69e9c700 1 -- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 msgr2=0x7f2e64163650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e69e9c700 1 --2- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64163650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e69e9c700 1 -- 172.21.15.172:0/3092425727 >> v1:172.21.15.102:6789/0 conn(0x7f2e640c9220 legacy=0x7f2e640ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e69e9c700 1 -- 172.21.15.172:0/3092425727 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e640cdac0 con 0x7f2e641530a0 2023-03-03T05:45:10.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e6a69d700 1 --2- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64163650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:10.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e69e9c700 1 --2- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e641530a0 0x7f2e641594a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2e54009840 tx=0x7f2e54009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd9f81c86f88f3bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.750+0000 7f2e68e9a700 1 -- 172.21.15.172:0/3092425727 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54004230 con 0x7f2e641530a0 2023-03-03T05:45:10.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e68e9a700 1 -- 172.21.15.172:0/3092425727 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54004390 con 0x7f2e641530a0 2023-03-03T05:45:10.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e68e9a700 1 -- 172.21.15.172:0/3092425727 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e540036a0 con 0x7f2e641530a0 2023-03-03T05:45:10.757 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 -- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e641530a0 msgr2=0x7f2e641594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.758 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 --2- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e641530a0 0x7f2e641594a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2e54009840 tx=0x7f2e54009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:10.758 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 -- 172.21.15.172:0/3092425727 shutdown_connections 2023-03-03T05:45:10.758 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 --2- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64163650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.758 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 --2- 172.21.15.172:0/3092425727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e641530a0 0x7f2e641594a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.758 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 -- 172.21.15.172:0/3092425727 >> 172.21.15.172:0/3092425727 conn(0x7f2e64065fb0 msgr2=0x7f2e640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.758 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 -- 172.21.15.172:0/3092425727 shutdown_connections 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.751+0000 7f2e6c100700 1 -- 172.21.15.172:0/3092425727 wait complete. 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.752+0000 7f2e6c100700 1 Processor -- start 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.752+0000 7f2e6c100700 1 -- start start 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6c100700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6c100700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640c9220 0x7f2e6415c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6c100700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641530a0 0x7f2e640c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6c100700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640c7080 con 0x7f2e641530a0 2023-03-03T05:45:10.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6c100700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640c4900 con 0x7f2e640c9220 2023-03-03T05:45:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6c100700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640c6540 con 0x7f2e640c79f0 2023-03-03T05:45:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6969b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640c9220 0x7f2e6415c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e69e9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6969b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640c9220 0x7f2e6415c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45334/0 (socket says 172.21.15.172:45334) 2023-03-03T05:45:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6969b700 1 -- 172.21.15.172:0/1078866407 learned_addr learned my addr 172.21.15.172:0/1078866407 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.753+0000 7f2e6a69d700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641530a0 0x7f2e640c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e69e9c700 1 -- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640c9220 msgr2=0x7f2e6415c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e69e9c700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640c9220 0x7f2e6415c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e69e9c700 1 -- 172.21.15.172:0/1078866407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641530a0 msgr2=0x7f2e640c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e69e9c700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641530a0 0x7f2e640c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e69e9c700 1 -- 172.21.15.172:0/1078866407 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c009b30 con 0x7f2e640c79f0 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e6969b700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640c9220 0x7f2e6415c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e69e9c700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64152140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e54013000 tx=0x7f2e54004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e5a7fc700 1 -- 172.21.15.172:0/1078866407 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54003d60 con 0x7f2e640c79f0 2023-03-03T05:45:10.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e5a7fc700 1 -- 172.21.15.172:0/1078866407 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54011630 con 0x7f2e640c79f0 2023-03-03T05:45:10.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e540094a0 con 0x7f2e640c79f0 2023-03-03T05:45:10.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e5a7fc700 1 -- 172.21.15.172:0/1078866407 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e540119e0 con 0x7f2e640c79f0 2023-03-03T05:45:10.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.754+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e64158f80 con 0x7f2e640c79f0 2023-03-03T05:45:10.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.755+0000 7f2e5a7fc700 1 -- 172.21.15.172:0/1078866407 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f2e5402d070 con 0x7f2e640c79f0 2023-03-03T05:45:10.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.755+0000 7f2e5a7fc700 1 -- 172.21.15.172:0/1078866407 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f2e5401a9f0 con 0x7f2e640c79f0 2023-03-03T05:45:10.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.755+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6415cff0 con 0x7f2e640c79f0 2023-03-03T05:45:10.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.757+0000 7f2e5a7fc700 1 -- 172.21.15.172:0/1078866407 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2e5402b060 con 0x7f2e640c79f0 2023-03-03T05:45:10.834 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:10.832+0000 7fc9d9f52200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T05:45:10.843 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.840+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c9e44237-2654-4425-95bd-31ee7706f4ff", "id": 6} v 0) v1 -- 0x7f2e64153470 con 0x7f2e640c79f0 2023-03-03T05:45:10.848 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.846+0000 7f2e5a7fc700 1 -- 172.21.15.172:0/1078866407 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c9e44237-2654-4425-95bd-31ee7706f4ff", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2e54021070 con 0x7f2e640c79f0 2023-03-03T05:45:10.849 INFO:teuthology.orchestra.run.smithi172.stdout:6 2023-03-03T05:45:10.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 msgr2=0x7f2e64152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64152140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e54013000 tx=0x7f2e54004750 comp rx=0 tx=0).stop 2023-03-03T05:45:10.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 shutdown_connections 2023-03-03T05:45:10.851 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e640c79f0 0x7f2e64152140 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.851 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641530a0 0x7f2e640c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.851 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 --2- 172.21.15.172:0/1078866407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640c9220 0x7f2e6415c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.851 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 >> 172.21.15.172:0/1078866407 conn(0x7f2e64065fb0 msgr2=0x7f2e640ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.851 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 shutdown_connections 2023-03-03T05:45:10.851 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.847+0000 7f2e6c100700 1 -- 172.21.15.172:0/1078866407 wait complete. 2023-03-03T05:45:10.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new c2898a5d-7f10-454d-b3b7-22034f852296 7 2023-03-03T05:45:10.910 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:10.908+0000 7fc9d9f52200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T05:45:10.983 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.980+0000 7f7b9f3cc700 1 Processor -- start 2023-03-03T05:45:10.984 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.981+0000 7f7b9f3cc700 1 -- start start 2023-03-03T05:45:10.990 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.981+0000 7f7b9f3cc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b9806a300 0x7f7b9806a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.990 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.981+0000 7f7b9f3cc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b98159440 0x7f7b981598c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.991 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.981+0000 7f7b9f3cc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b98070230 con 0x7f7b980c6770 2023-03-03T05:45:10.991 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.981+0000 7f7b9f3cc700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b98159e00 con 0x7f7b9806a300 2023-03-03T05:45:10.991 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.981+0000 7f7b9f3cc700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b9806ac10 con 0x7f7b98159440 2023-03-03T05:45:10.991 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9d969700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b98159440 0x7f7b981598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:10.992 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9c967700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7b980c6770 0x7f7b9806f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:41490/0 (socket says 172.21.15.172:41490) 2023-03-03T05:45:10.992 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9c967700 1 -- 172.21.15.172:0/2420362898 learned_addr learned my addr 172.21.15.172:0/2420362898 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-03-03T05:45:10.992 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b8ffff700 1 -- 172.21.15.172:0/2420362898 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046942465 0 0) 0x7f7b98070230 con 0x7f7b980c6770 2023-03-03T05:45:10.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b8ffff700 1 -- 172.21.15.172:0/2420362898 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b840035b0 con 0x7f7b980c6770 2023-03-03T05:45:10.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9d969700 1 -- 172.21.15.172:0/2420362898 >> v2:172.21.15.102:3300/0 conn(0x7f7b9806a300 msgr2=0x7f7b9806a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:10.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9d969700 1 --2- 172.21.15.172:0/2420362898 >> v2:172.21.15.102:3300/0 conn(0x7f7b9806a300 0x7f7b9806a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9d969700 1 -- 172.21.15.172:0/2420362898 >> v1:172.21.15.172:6789/0 conn(0x7f7b980c6770 legacy=0x7f7b9806f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.994 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9d969700 1 -- 172.21.15.172:0/2420362898 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9806fb10 con 0x7f7b98159440 2023-03-03T05:45:10.994 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b9d969700 1 --2- 172.21.15.172:0/2420362898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b98159440 0x7f7b981598c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7b9806c490 tx=0x7f7b9000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c46a715f3ed3f13 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:10.994 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b8ffff700 1 -- 172.21.15.172:0/2420362898 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000e070 con 0x7f7b98159440 2023-03-03T05:45:10.994 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b8ffff700 1 -- 172.21.15.172:0/2420362898 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9000bb20 con 0x7f7b98159440 2023-03-03T05:45:10.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.982+0000 7f7b8ffff700 1 -- 172.21.15.172:0/2420362898 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b90012430 con 0x7f7b98159440 2023-03-03T05:45:10.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/2420362898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b98159440 msgr2=0x7f7b981598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:10.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 --2- 172.21.15.172:0/2420362898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b98159440 0x7f7b981598c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7b9806c490 tx=0x7f7b9000b330 comp rx=0 tx=0).stop 2023-03-03T05:45:10.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/2420362898 shutdown_connections 2023-03-03T05:45:10.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 --2- 172.21.15.172:0/2420362898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b98159440 0x7f7b981598c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 --2- 172.21.15.172:0/2420362898 >> v2:172.21.15.102:3300/0 conn(0x7f7b9806a300 0x7f7b9806a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:10.997 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/2420362898 >> 172.21.15.172:0/2420362898 conn(0x7f7b9806b410 msgr2=0x7f7b9806b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:10.997 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/2420362898 shutdown_connections 2023-03-03T05:45:10.997 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.983+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/2420362898 wait complete. 2023-03-03T05:45:10.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.984+0000 7f7b9f3cc700 1 Processor -- start 2023-03-03T05:45:10.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.984+0000 7f7b9f3cc700 1 -- start start 2023-03-03T05:45:10.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9f3cc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806a300 0x7f7b980cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9f3cc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b980c6770 0x7f7b980c3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9f3cc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b98159440 0x7f7b98158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:10.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9f3cc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980cbda0 con 0x7f7b98159440 2023-03-03T05:45:10.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9f3cc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980ca8a0 con 0x7f7b9806a300 2023-03-03T05:45:10.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9f3cc700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980c5b40 con 0x7f7b980c6770 2023-03-03T05:45:11.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9d168700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806a300 0x7f7b980cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:11.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9c967700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b980c6770 0x7f7b980c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:11.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9d168700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806a300 0x7f7b980cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45346/0 (socket says 172.21.15.172:45346) 2023-03-03T05:45:11.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9c967700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b980c6770 0x7f7b980c3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:40112/0 (socket says 172.21.15.172:40112) 2023-03-03T05:45:11.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.985+0000 7f7b9d168700 1 -- 172.21.15.172:0/178148998 learned_addr learned my addr 172.21.15.172:0/178148998 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-03-03T05:45:11.001 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d969700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b98159440 0x7f7b98158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:11.001 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d168700 1 -- 172.21.15.172:0/178148998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b980c6770 msgr2=0x7f7b980c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.001 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d168700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b980c6770 0x7f7b980c3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d168700 1 -- 172.21.15.172:0/178148998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b98159440 msgr2=0x7f7b98158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d168700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b98159440 0x7f7b98158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d168700 1 -- 172.21.15.172:0/178148998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b90009a70 con 0x7f7b9806a300 2023-03-03T05:45:11.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d969700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b98159440 0x7f7b98158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:11.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9d168700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806a300 0x7f7b980cb5d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400b4a0 tx=0x7f7b9400b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:11.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b8dffb700 1 -- 172.21.15.172:0/178148998 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94011830 con 0x7f7b9806a300 2023-03-03T05:45:11.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b8dffb700 1 -- 172.21.15.172:0/178148998 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b94011990 con 0x7f7b9806a300 2023-03-03T05:45:11.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b98071fd0 con 0x7f7b9806a300 2023-03-03T05:45:11.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b8dffb700 1 -- 172.21.15.172:0/178148998 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94011d40 con 0x7f7b9806a300 2023-03-03T05:45:11.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b8dffb700 1 -- 172.21.15.172:0/178148998 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f7b9400f050 con 0x7f7b9806a300 2023-03-03T05:45:11.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b98071cf0 con 0x7f7b9806a300 2023-03-03T05:45:11.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.986+0000 7f7b8dffb700 1 -- 172.21.15.172:0/178148998 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7b9401ad10 con 0x7f7b9806a300 2023-03-03T05:45:11.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.987+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b980722b0 con 0x7f7b9806a300 2023-03-03T05:45:11.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:10.989+0000 7f7b8dffb700 1 -- 172.21.15.172:0/178148998 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7b94039b00 con 0x7f7b9806a300 2023-03-03T05:45:11.053 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:11.051+0000 7fc9d9f52200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T05:45:11.075 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.073+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c2898a5d-7f10-454d-b3b7-22034f852296", "id": 7} v 0) v1 -- 0x7f7b980c7e10 con 0x7f7b9806a300 2023-03-03T05:45:11.080 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.078+0000 7f7b8dffb700 1 -- 172.21.15.172:0/178148998 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2898a5d-7f10-454d-b3b7-22034f852296", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7b94028070 con 0x7f7b9806a300 2023-03-03T05:45:11.081 INFO:teuthology.orchestra.run.smithi172.stdout:7 2023-03-03T05:45:11.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806a300 msgr2=0x7f7b980cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806a300 0x7f7b980cb5d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400b4a0 tx=0x7f7b9400b7b0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 shutdown_connections 2023-03-03T05:45:11.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b98159440 0x7f7b98158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806a300 0x7f7b980cb5d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 --2- 172.21.15.172:0/178148998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b980c6770 0x7f7b980c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 >> 172.21.15.172:0/178148998 conn(0x7f7b9806b410 msgr2=0x7f7b98152070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:11.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 shutdown_connections 2023-03-03T05:45:11.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-03-03T05:45:11.079+0000 7f7b9f3cc700 1 -- 172.21.15.172:0/178148998 wait complete. 2023-03-03T05:45:11.097 INFO:tasks.ceph.osd.0:Restarting daemon 2023-03-03T05:45:11.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-03-03T05:45:11.103 INFO:tasks.ceph.osd.0:Started 2023-03-03T05:45:11.104 INFO:tasks.ceph.osd.1:Restarting daemon 2023-03-03T05:45:11.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-03-03T05:45:11.109 INFO:tasks.ceph.osd.1:Started 2023-03-03T05:45:11.110 INFO:tasks.ceph.osd.2:Restarting daemon 2023-03-03T05:45:11.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-03-03T05:45:11.116 INFO:tasks.ceph.osd.2:Started 2023-03-03T05:45:11.116 INFO:tasks.ceph.osd.3:Restarting daemon 2023-03-03T05:45:11.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-03-03T05:45:11.122 INFO:tasks.ceph.osd.3:Started 2023-03-03T05:45:11.122 INFO:tasks.ceph.osd.4:Restarting daemon 2023-03-03T05:45:11.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-03-03T05:45:11.125 INFO:tasks.ceph.osd.4:Started 2023-03-03T05:45:11.125 INFO:tasks.ceph.osd.5:Restarting daemon 2023-03-03T05:45:11.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-03-03T05:45:11.127 INFO:tasks.ceph.osd.5:Started 2023-03-03T05:45:11.127 INFO:tasks.ceph.osd.6:Restarting daemon 2023-03-03T05:45:11.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-03-03T05:45:11.130 INFO:tasks.ceph.osd.6:Started 2023-03-03T05:45:11.130 INFO:tasks.ceph.osd.7:Restarting daemon 2023-03-03T05:45:11.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-03-03T05:45:11.133 INFO:tasks.ceph.osd.7:Started 2023-03-03T05:45:11.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:11.375 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:11.373+0000 7f7c6fdc8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T05:45:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.507+0000 7f7b6fbff700 1 Processor -- start 2023-03-03T05:45:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.507+0000 7f7b6fbff700 1 -- start start 2023-03-03T05:45:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.509+0000 7f7b6fbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b681597f0 0x7f7b680cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6fbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b680cb9f0 con 0x7f7b680cf9a0 2023-03-03T05:45:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6fbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b68159c80 con 0x7f7b680ced80 2023-03-03T05:45:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6fbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b680ca750 con 0x7f7b681597f0 2023-03-03T05:45:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6d99b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7b680ced80 0x7f7b680cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33008/0 (socket says 172.21.15.102:33008) 2023-03-03T05:45:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6d99b700 1 -- 172.21.15.102:0/2038691171 learned_addr learned my addr 172.21.15.102:0/2038691171 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6d19a700 1 --2- 172.21.15.102:0/2038691171 >> v2:172.21.15.172:3300/0 conn(0x7f7b681597f0 0x7f7b680cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:11.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6c999700 1 -- 172.21.15.102:0/2038691171 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276190349 0 0) 0x7f7b68159c80 con 0x7f7b680ced80 2023-03-03T05:45:11.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6c999700 1 -- 172.21.15.102:0/2038691171 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b540035b0 con 0x7f7b680ced80 2023-03-03T05:45:11.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6d19a700 1 -- 172.21.15.102:0/2038691171 >> v1:172.21.15.172:6789/0 conn(0x7f7b680ced80 legacy=0x7f7b680cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6d19a700 1 -- 172.21.15.102:0/2038691171 >> v1:172.21.15.102:6789/0 conn(0x7f7b680cf9a0 legacy=0x7f7b68163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.510+0000 7f7b6d19a700 1 -- 172.21.15.102:0/2038691171 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b680cade0 con 0x7f7b681597f0 2023-03-03T05:45:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.511+0000 7f7b6d19a700 1 --2- 172.21.15.102:0/2038691171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b681597f0 0x7f7b680cf150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009840 tx=0x7f7b58009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2f52b3e1001ee44 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.511+0000 7f7b6c999700 1 -- 172.21.15.102:0/2038691171 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58011830 con 0x7f7b681597f0 2023-03-03T05:45:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.511+0000 7f7b6c999700 1 -- 172.21.15.102:0/2038691171 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b58011990 con 0x7f7b681597f0 2023-03-03T05:45:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.511+0000 7f7b6c999700 1 -- 172.21.15.102:0/2038691171 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58011d20 con 0x7f7b681597f0 2023-03-03T05:45:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.512+0000 7f7b6fbff700 1 -- 172.21.15.102:0/2038691171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b681597f0 msgr2=0x7f7b680cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.512+0000 7f7b6fbff700 1 --2- 172.21.15.102:0/2038691171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b681597f0 0x7f7b680cf150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009840 tx=0x7f7b58009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.512+0000 7f7b6fbff700 1 -- 172.21.15.102:0/2038691171 shutdown_connections 2023-03-03T05:45:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.512+0000 7f7b6fbff700 1 --2- 172.21.15.102:0/2038691171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b681597f0 0x7f7b680cf150 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.512+0000 7f7b6fbff700 1 -- 172.21.15.102:0/2038691171 >> 172.21.15.102:0/2038691171 conn(0x7f7b68065fb0 msgr2=0x7f7b680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.512+0000 7f7b6fbff700 1 -- 172.21.15.102:0/2038691171 shutdown_connections 2023-03-03T05:45:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.512+0000 7f7b6fbff700 1 -- 172.21.15.102:0/2038691171 wait complete. 2023-03-03T05:45:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.514+0000 7f7b6fbff700 1 Processor -- start 2023-03-03T05:45:11.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.514+0000 7f7b6fbff700 1 -- start start 2023-03-03T05:45:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6fbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b680ced80 0x7f7b680c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6fbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b680cf9a0 0x7f7b6814d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6fbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b681597f0 0x7f7b6814d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6fbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6814ff00 con 0x7f7b681597f0 2023-03-03T05:45:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6fbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680b44e0 con 0x7f7b680ced80 2023-03-03T05:45:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6fbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6814f8b0 con 0x7f7b680cf9a0 2023-03-03T05:45:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6d99b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b680ced80 0x7f7b680c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:11.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6d19a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b680cf9a0 0x7f7b6814d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:11.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6e19c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b681597f0 0x7f7b6814d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:11.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6d19a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b680cf9a0 0x7f7b6814d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57564/0 (socket says 172.21.15.102:57564) 2023-03-03T05:45:11.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6d99b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b680ced80 0x7f7b680c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55018/0 (socket says 172.21.15.102:55018) 2023-03-03T05:45:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.515+0000 7f7b6d19a700 1 -- 172.21.15.102:0/1258409637 learned_addr learned my addr 172.21.15.102:0/1258409637 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b6d19a700 1 -- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b680ced80 msgr2=0x7f7b680c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b6d19a700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b680ced80 0x7f7b680c7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b6d19a700 1 -- 172.21.15.102:0/1258409637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b681597f0 msgr2=0x7f7b6814d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b6d19a700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b681597f0 0x7f7b6814d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b6d19a700 1 -- 172.21.15.102:0/1258409637 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b580094a0 con 0x7f7b680cf9a0 2023-03-03T05:45:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b6d99b700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b680ced80 0x7f7b680c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b6d19a700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b680cf9a0 0x7f7b6814d2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009b90 tx=0x7f7b58004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:11.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.516+0000 7f7b5e7fc700 1 -- 172.21.15.102:0/1258409637 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b580036a0 con 0x7f7b680cf9a0 2023-03-03T05:45:11.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.517+0000 7f7b5e7fc700 1 -- 172.21.15.102:0/1258409637 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5801e070 con 0x7f7b680cf9a0 2023-03-03T05:45:11.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.517+0000 7f7b5e7fc700 1 -- 172.21.15.102:0/1258409637 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58003a60 con 0x7f7b680cf9a0 2023-03-03T05:45:11.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.517+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b680cb3a0 con 0x7f7b680cf9a0 2023-03-03T05:45:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.517+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b680cb0c0 con 0x7f7b680cf9a0 2023-03-03T05:45:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.517+0000 7f7b5e7fc700 1 -- 172.21.15.102:0/1258409637 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f7b5800f050 con 0x7f7b680cf9a0 2023-03-03T05:45:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.517+0000 7f7b5e7fc700 1 -- 172.21.15.102:0/1258409637 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7b5801a420 con 0x7f7b680cf9a0 2023-03-03T05:45:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.518+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b680cb680 con 0x7f7b680cf9a0 2023-03-03T05:45:11.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.521+0000 7f7b5e7fc700 1 -- 172.21.15.102:0/1258409637 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7b58020070 con 0x7f7b680cf9a0 2023-03-03T05:45:11.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.600+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b680677f0 con 0x7f7b680cf9a0 2023-03-03T05:45:11.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.601+0000 7f7b5e7fc700 1 -- 172.21.15.102:0/1258409637 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f7b58015020 con 0x7f7b680cf9a0 2023-03-03T05:45:11.603 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:11.604 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:11.075336+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:11.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b680cf9a0 msgr2=0x7f7b6814d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:11.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b680cf9a0 0x7f7b6814d2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009b90 tx=0x7f7b58004750 comp rx=0 tx=0).stop 2023-03-03T05:45:11.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 shutdown_connections 2023-03-03T05:45:11.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b681597f0 0x7f7b6814d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b680ced80 0x7f7b680c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 --2- 172.21.15.102:0/1258409637 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b680cf9a0 0x7f7b6814d2f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:11.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 >> 172.21.15.102:0/1258409637 conn(0x7f7b68065fb0 msgr2=0x7f7b68158520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:11.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 shutdown_connections 2023-03-03T05:45:11.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:11.602+0000 7f7b6fbff700 1 -- 172.21.15.102:0/1258409637 wait complete. 2023-03-03T05:45:11.627 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:11.625+0000 7fc9d9f52200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T05:45:11.631 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-03-03T05:45:11.631 INFO:tasks.ceph:Waiting for OSDs to come up 2023-03-03T05:45:11.647 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:11.645+0000 7f7c6fdc8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T05:45:11.701 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:11.698+0000 7fc9d9f52200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T05:45:11.743 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-03-03T05:45:11.741+0000 7f1d48252640 -1 Falling back to public interface 2023-03-03T05:45:11.744 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-03-03T05:45:11.741+0000 7fba43c5f640 -1 Falling back to public interface 2023-03-03T05:45:11.744 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-03-03T05:45:11.741+0000 7fba3b00d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-03-03T05:45:11.746 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-03-03T05:45:11.744+0000 7f9d80950640 -1 Falling back to public interface 2023-03-03T05:45:11.751 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-03-03T05:45:11.749+0000 7fa1e7961640 -1 Falling back to public interface 2023-03-03T05:45:11.846 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:11.843+0000 7fc9d9f52200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T05:45:11.898 INFO:tasks.ceph.osd.3.smithi102.stderr:2023-03-03T05:45:11.896+0000 7f90a9ad4640 -1 Falling back to public interface 2023-03-03T05:45:11.905 INFO:tasks.ceph.osd.1.smithi102.stderr:2023-03-03T05:45:11.903+0000 7fbaaa901640 -1 Falling back to public interface 2023-03-03T05:45:11.920 INFO:tasks.ceph.osd.0.smithi102.stderr:2023-03-03T05:45:11.918+0000 7f9876a87640 -1 Falling back to public interface 2023-03-03T05:45:11.921 INFO:tasks.ceph.osd.2.smithi102.stderr:2023-03-03T05:45:11.919+0000 7fdcc74d8640 -1 Falling back to public interface 2023-03-03T05:45:11.971 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:11.969+0000 7f7c6fdc8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T05:45:12.089 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:12.087+0000 7f7c6fdc8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T05:45:12.100 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:12.097+0000 7fc9d9f52200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T05:45:12.240 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:12.237+0000 7fc9d9f52200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T05:45:12.320 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:12.317+0000 7fc9d9f52200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T05:45:12.322 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:12.320+0000 7f7c6fdc8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T05:45:12.400 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:12.398+0000 7fc9d9f52200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T05:45:12.440 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:12.479 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:12.476+0000 7fc9d9f52200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T05:45:12.660 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.658+0000 7f8cf43ff700 1 Processor -- start 2023-03-03T05:45:12.660 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.658+0000 7f8cf43ff700 1 -- start start 2023-03-03T05:45:12.661 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.659+0000 7f8cf43ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8cec153540 0x7f8cec06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:12.661 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.659+0000 7f8cf43ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8cec0cfc20 0x7f8cec06c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:12.661 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.659+0000 7f8cf43ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cda10 0x7f8cec15c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:12.661 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.659+0000 7f8cf43ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0cd0a0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.661 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.659+0000 7f8cf43ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0cd5d0 con 0x7f8cec153540 2023-03-03T05:45:12.662 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.659+0000 7f8cf43ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec159ae0 con 0x7f8cec0cda10 2023-03-03T05:45:12.663 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf299c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cda10 0x7f8cec15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:12.664 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8cec0cfc20 0x7f8cec06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:12.664 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf219b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8cec153540 0x7f8cec06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:12.664 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8cec0cfc20 0x7f8cec06c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52060/0 (socket says 172.21.15.102:52060) 2023-03-03T05:45:12.664 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 -- 172.21.15.102:0/4102156450 learned_addr learned my addr 172.21.15.102:0/4102156450 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:12.664 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf299c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cda10 0x7f8cec15c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57572/0 (socket says 172.21.15.102:57572) 2023-03-03T05:45:12.664 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 -- 172.21.15.102:0/4102156450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cda10 msgr2=0x7f8cec15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:12.664 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 --2- 172.21.15.102:0/4102156450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cda10 0x7f8cec15c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.665 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 -- 172.21.15.102:0/4102156450 >> v2:172.21.15.172:3300/0 conn(0x7f8cec153540 msgr2=0x7f8cec06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:12.665 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 --2- 172.21.15.102:0/4102156450 >> v2:172.21.15.172:3300/0 conn(0x7f8cec153540 0x7f8cec06bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.665 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 -- 172.21.15.102:0/4102156450 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec159dc0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.665 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.660+0000 7f8cf199a700 1 --2- 172.21.15.102:0/4102156450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cfc20 0x7f8cec06c510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8cec06a300 tx=0x7f8cdc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da3377bd693405e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:12.665 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.661+0000 7f8cf1199700 1 -- 172.21.15.102:0/4102156450 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc011830 con 0x7f8cec0cfc20 2023-03-03T05:45:12.665 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.661+0000 7f8cf1199700 1 -- 172.21.15.102:0/4102156450 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc011990 con 0x7f8cec0cfc20 2023-03-03T05:45:12.666 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.661+0000 7f8cf1199700 1 -- 172.21.15.102:0/4102156450 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc011d20 con 0x7f8cec0cfc20 2023-03-03T05:45:12.666 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.661+0000 7f8cf43ff700 1 -- 172.21.15.102:0/4102156450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cfc20 msgr2=0x7f8cec06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:12.666 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.661+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/4102156450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cfc20 0x7f8cec06c510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8cec06a300 tx=0x7f8cdc009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.666 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.662+0000 7f8cf43ff700 1 -- 172.21.15.102:0/4102156450 shutdown_connections 2023-03-03T05:45:12.666 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.662+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/4102156450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cda10 0x7f8cec15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.667 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.662+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/4102156450 >> v2:172.21.15.172:3300/0 conn(0x7f8cec153540 0x7f8cec06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.667 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.662+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/4102156450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cfc20 0x7f8cec06c510 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.667 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.662+0000 7f8cf43ff700 1 -- 172.21.15.102:0/4102156450 >> 172.21.15.102:0/4102156450 conn(0x7f8cec0cef80 msgr2=0x7f8cec0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:12.667 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.662+0000 7f8cf43ff700 1 -- 172.21.15.102:0/4102156450 shutdown_connections 2023-03-03T05:45:12.667 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.662+0000 7f8cf43ff700 1 -- 172.21.15.102:0/4102156450 wait complete. 2023-03-03T05:45:12.670 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.664+0000 7f8cf43ff700 1 Processor -- start 2023-03-03T05:45:12.670 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.664+0000 7f8cf43ff700 1 -- start start 2023-03-03T05:45:12.670 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf43ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cda10 0x7f8cec1ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:12.670 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf43ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cfc20 0x7f8cec154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:12.671 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf43ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cec153540 0x7f8cec1552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:12.671 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf43ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec155e90 con 0x7f8cec0cda10 2023-03-03T05:45:12.671 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf43ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec1efae0 con 0x7f8cec153540 2023-03-03T05:45:12.671 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf43ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec1548c0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.671 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf299c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cec153540 0x7f8cec1552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:12.671 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf199a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cfc20 0x7f8cec154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:12.671 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf219b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cda10 0x7f8cec1ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:12.672 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf299c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cec153540 0x7f8cec1552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55034/0 (socket says 172.21.15.102:55034) 2023-03-03T05:45:12.672 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.665+0000 7f8cf299c700 1 -- 172.21.15.102:0/660357062 learned_addr learned my addr 172.21.15.102:0/660357062 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:12.672 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.666+0000 7f8cf199a700 1 -- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cec153540 msgr2=0x7f8cec1552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:12.672 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.666+0000 7f8cf199a700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cec153540 0x7f8cec1552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.672 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.666+0000 7f8cf199a700 1 -- 172.21.15.102:0/660357062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cda10 msgr2=0x7f8cec1ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:12.672 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.666+0000 7f8cf199a700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cda10 0x7f8cec1ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.673 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.666+0000 7f8cf199a700 1 -- 172.21.15.102:0/660357062 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc009550 con 0x7f8cec0cfc20 2023-03-03T05:45:12.673 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.666+0000 7f8cf299c700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cec153540 0x7f8cec1552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:12.673 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.666+0000 7f8cf199a700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cfc20 0x7f8cec154db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc016000 tx=0x7f8cdc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:12.673 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.667+0000 7f8ce2ffd700 1 -- 172.21.15.102:0/660357062 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc004450 con 0x7f8cec0cfc20 2023-03-03T05:45:12.674 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.667+0000 7f8ce2ffd700 1 -- 172.21.15.102:0/660357062 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc01e070 con 0x7f8cec0cfc20 2023-03-03T05:45:12.674 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.667+0000 7f8ce2ffd700 1 -- 172.21.15.102:0/660357062 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc0038e0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.674 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.667+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec06bab0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.675 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.667+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec0cd2f0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.675 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.667+0000 7f8ce2ffd700 1 -- 172.21.15.102:0/660357062 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f8cdc00f050 con 0x7f8cec0cfc20 2023-03-03T05:45:12.675 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.667+0000 7f8ce2ffd700 1 -- 172.21.15.102:0/660357062 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8cdc02dc10 con 0x7f8cec0cfc20 2023-03-03T05:45:12.675 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.668+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec06d2d0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.676 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.671+0000 7f8ce2ffd700 1 -- 172.21.15.102:0/660357062 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f8cdc020070 con 0x7f8cec0cfc20 2023-03-03T05:45:12.747 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.745+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cec0748c0 con 0x7f8cec0cfc20 2023-03-03T05:45:12.747 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.746+0000 7f8ce2ffd700 1 -- 172.21.15.102:0/660357062 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8cdc015020 con 0x7f8cec0cfc20 2023-03-03T05:45:12.748 INFO:teuthology.misc.health.smithi102.stdout: 2023-03-03T05:45:12.748 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:11.075336+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:12.749 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cfc20 msgr2=0x7f8cec154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:12.750 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cfc20 0x7f8cec154db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc016000 tx=0x7f8cdc004750 comp rx=0 tx=0).stop 2023-03-03T05:45:12.750 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 shutdown_connections 2023-03-03T05:45:12.751 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cec153540 0x7f8cec1552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.751 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cec0cda10 0x7f8cec1ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.751 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 --2- 172.21.15.102:0/660357062 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cec0cfc20 0x7f8cec154db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:12.752 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 >> 172.21.15.102:0/660357062 conn(0x7f8cec0cef80 msgr2=0x7f8cec157f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:12.752 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 shutdown_connections 2023-03-03T05:45:12.752 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:12.747+0000 7f8cf43ff700 1 -- 172.21.15.102:0/660357062 wait complete. 2023-03-03T05:45:12.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-03T05:45:13.118 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:13.116+0000 7f7c6fdc8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T05:45:13.247 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:13.245+0000 7f7c6fdc8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T05:45:13.265 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:13.263+0000 7fc9d9f52200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T05:45:13.347 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:13.345+0000 7fc9d9f52200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T05:45:13.424 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:13.422+0000 7fc9d9f52200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T05:45:13.500 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:13.498+0000 7f7c6fdc8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T05:45:13.508 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:13.506+0000 7fc9d9f52200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T05:45:13.691 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:13.689+0000 7fc9d9f52200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T05:45:13.764 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:13.762+0000 7fc9d9f52200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T05:45:13.850 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:13.848+0000 7f7c6fdc8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T05:45:13.917 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:13.914+0000 7fc9d9f52200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T05:45:14.107 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:14.105+0000 7f7c6fdc8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T05:45:14.249 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:14.247+0000 7f7c6fdc8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T05:45:14.391 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:14.389+0000 7f7c6fdc8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T05:45:14.572 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:14.570+0000 7f7c6fdc8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T05:45:14.948 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:14.945+0000 7fc993a6d700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-03-03T05:45:14.948 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-03-03T05:45:14.946+0000 7fc993a6d700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-03-03T05:45:15.251 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-03-03T05:45:15.248+0000 7fa1e7961640 -1 osd.5 0 log_to_monitors true 2023-03-03T05:45:15.253 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-03-03T05:45:15.251+0000 7f1d48252640 -1 osd.6 0 log_to_monitors true 2023-03-03T05:45:15.282 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-03-03T05:45:15.280+0000 7f9d80950640 -1 osd.4 0 log_to_monitors true 2023-03-03T05:45:15.284 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-03-03T05:45:15.282+0000 7fba43c5f640 -1 osd.7 0 log_to_monitors true 2023-03-03T05:45:15.848 INFO:tasks.ceph.osd.1.smithi102.stderr:2023-03-03T05:45:15.846+0000 7fbaaa901640 -1 osd.1 0 log_to_monitors true 2023-03-03T05:45:15.862 INFO:tasks.ceph.osd.0.smithi102.stderr:2023-03-03T05:45:15.860+0000 7f9876a87640 -1 osd.0 0 log_to_monitors true 2023-03-03T05:45:15.879 INFO:tasks.ceph.osd.2.smithi102.stderr:2023-03-03T05:45:15.878+0000 7fdcc74d8640 -1 osd.2 0 log_to_monitors true 2023-03-03T05:45:15.913 INFO:tasks.ceph.osd.3.smithi102.stderr:2023-03-03T05:45:15.911+0000 7f90a9ad4640 -1 osd.3 0 log_to_monitors true 2023-03-03T05:45:15.928 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:15.926+0000 7f7c6fdc8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T05:45:16.082 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:16.080+0000 7f7c6fdc8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T05:45:16.223 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:16.221+0000 7f7c6fdc8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T05:45:16.374 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:16.372+0000 7f7c6fdc8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T05:45:16.699 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:16.697+0000 7f7c6fdc8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T05:45:16.836 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:16.834+0000 7f7c6fdc8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T05:45:17.234 INFO:tasks.ceph.mgr.y.smithi102.stderr:2023-03-03T05:45:17.232+0000 7f7c6fdc8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T05:45:18.834 INFO:tasks.ceph.osd.1.smithi102.stderr:2023-03-03T05:45:18.831+0000 7fba98ff6700 -1 osd.1 0 waiting for initial osdmap 2023-03-03T05:45:18.920 INFO:tasks.ceph.osd.3.smithi102.stderr:2023-03-03T05:45:18.918+0000 7f90981f6700 -1 osd.3 0 waiting for initial osdmap 2023-03-03T05:45:18.962 INFO:tasks.ceph.osd.1.smithi102.stderr:2023-03-03T05:45:18.958+0000 7fba90ffb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:18.971 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-03-03T05:45:18.968+0000 7fba39009700 -1 osd.7 0 waiting for initial osdmap 2023-03-03T05:45:18.987 INFO:tasks.ceph.osd.3.smithi102.stderr:2023-03-03T05:45:18.984+0000 7f90901fb700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:18.988 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-03-03T05:45:18.985+0000 7fba2fbe1700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:19.018 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-03-03T05:45:19.015+0000 7f1d3bdf9700 -1 osd.6 0 waiting for initial osdmap 2023-03-03T05:45:19.031 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-03-03T05:45:19.028+0000 7f9d744f7700 -1 osd.4 0 waiting for initial osdmap 2023-03-03T05:45:19.032 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-03-03T05:45:19.030+0000 7f1d341d4700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:19.034 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-03-03T05:45:19.032+0000 7fa1db508700 -1 osd.5 0 waiting for initial osdmap 2023-03-03T05:45:19.040 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-03-03T05:45:19.038+0000 7f9d6c8d2700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:19.052 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-03-03T05:45:19.050+0000 7fa1d38e3700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:19.579 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:19.865 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.862+0000 7fbfbed9d700 1 Processor -- start 2023-03-03T05:45:19.865 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.862+0000 7fbfbed9d700 1 -- start start 2023-03-03T05:45:19.868 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.864+0000 7fbfbed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb8152d80 0x7fbfb8158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:19.868 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.864+0000 7fbfbed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbfb80ce510 0x7fbfb815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:19.868 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.864+0000 7fbfbed9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb806cde0 con 0x7fbfb806c880 2023-03-03T05:45:19.868 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.864+0000 7fbfbed9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb806d080 con 0x7fbfb80ce510 2023-03-03T05:45:19.868 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.864+0000 7fbfbed9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb815c110 con 0x7fbfb8152d80 2023-03-03T05:45:19.869 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbd59a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbfb80ce510 0x7fbfb815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:19.869 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbd59a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbfb80ce510 0x7fbfb815cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:49960/0 (socket says 172.21.15.102:49960) 2023-03-03T05:45:19.869 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbd59a700 1 -- 172.21.15.102:0/2968743116 learned_addr learned my addr 172.21.15.102:0/2968743116 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:19.869 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbdd9b700 1 --2- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb8152d80 0x7fbfb8158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:19.870 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbd59a700 1 -- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb8152d80 msgr2=0x7fbfb8158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:19.870 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbd59a700 1 --2- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb8152d80 0x7fbfb8158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:19.871 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbd59a700 1 -- 172.21.15.102:0/2968743116 >> v1:172.21.15.102:6789/0 conn(0x7fbfb806c880 legacy=0x7fbfb80cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:19.872 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.865+0000 7fbfbd59a700 1 -- 172.21.15.102:0/2968743116 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb815c3f0 con 0x7fbfb80ce510 2023-03-03T05:45:19.872 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.866+0000 7fbfbd59a700 1 --2- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 0x7fbfb815cbf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8006fd0 tx=0x7fbfa8009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966618d13838c6e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:19.872 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.866+0000 7fbfbcd99700 1 -- 172.21.15.102:0/2968743116 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8005e30 con 0x7fbfb80ce510 2023-03-03T05:45:19.873 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.866+0000 7fbfbcd99700 1 -- 172.21.15.102:0/2968743116 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8004030 con 0x7fbfb80ce510 2023-03-03T05:45:19.873 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.866+0000 7fbfbcd99700 1 -- 172.21.15.102:0/2968743116 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa80043e0 con 0x7fbfb80ce510 2023-03-03T05:45:19.873 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 msgr2=0x7fbfb815cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:19.873 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 0x7fbfb815cbf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8006fd0 tx=0x7fbfa8009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:19.874 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2968743116 shutdown_connections 2023-03-03T05:45:19.874 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb8152d80 0x7fbfb8158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:19.874 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2968743116 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 0x7fbfb815cbf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:19.874 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2968743116 >> 172.21.15.102:0/2968743116 conn(0x7fbfb8060d50 msgr2=0x7fbfb8061150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:19.874 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2968743116 shutdown_connections 2023-03-03T05:45:19.875 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.867+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2968743116 wait complete. 2023-03-03T05:45:19.876 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.870+0000 7fbfbed9d700 1 Processor -- start 2023-03-03T05:45:19.876 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.870+0000 7fbfbed9d700 1 -- start start 2023-03-03T05:45:19.877 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.871+0000 7fbfbed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb806c880 0x7fbfb80b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:19.877 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.871+0000 7fbfbed9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 0x7fbfb80c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:19.877 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfb8152d80 0x7fbfb80c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:19.877 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbed9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb814f390 con 0x7fbfb8152d80 2023-03-03T05:45:19.878 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbed9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb814ee10 con 0x7fbfb80ce510 2023-03-03T05:45:19.878 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbed9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80b47c0 con 0x7fbfb806c880 2023-03-03T05:45:19.878 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbe59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfb8152d80 0x7fbfb80c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:19.878 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbe59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfb8152d80 0x7fbfb80c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37646/0 (socket says 172.21.15.102:37646) 2023-03-03T05:45:19.878 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbe59c700 1 -- 172.21.15.102:0/2512647476 learned_addr learned my addr 172.21.15.102:0/2512647476 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:19.879 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbdd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb806c880 0x7fbfb80b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:19.879 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbd59a700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 0x7fbfb80c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:19.879 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.872+0000 7fbfbdd9b700 1 -- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 msgr2=0x7fbfb80c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:19.879 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfbdd9b700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 0x7fbfb80c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:19.880 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfbdd9b700 1 -- 172.21.15.102:0/2512647476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfb8152d80 msgr2=0x7fbfb80c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:19.880 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfbdd9b700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfb8152d80 0x7fbfb80c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:19.880 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfbdd9b700 1 -- 172.21.15.102:0/2512647476 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8009500 con 0x7fbfb806c880 2023-03-03T05:45:19.880 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfbdd9b700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb806c880 0x7fbfb80b3e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400e870 tx=0x7fbfb400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:19.881 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfae7fc700 1 -- 172.21.15.102:0/2512647476 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4013680 con 0x7fbfb806c880 2023-03-03T05:45:19.881 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfae7fc700 1 -- 172.21.15.102:0/2512647476 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb40137e0 con 0x7fbfb806c880 2023-03-03T05:45:19.881 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfae7fc700 1 -- 172.21.15.102:0/2512647476 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4013b90 con 0x7fbfb806c880 2023-03-03T05:45:19.881 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.873+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb8153150 con 0x7fbfb806c880 2023-03-03T05:45:19.882 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.874+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb815c6d0 con 0x7fbfb806c880 2023-03-03T05:45:19.883 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.875+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb806d360 con 0x7fbfb806c880 2023-03-03T05:45:19.883 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.876+0000 7fbfae7fc700 1 -- 172.21.15.102:0/2512647476 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 5) v1 ==== 86375+0+0 (secure 0 0 0) 0x7fbfb4019030 con 0x7fbfb806c880 2023-03-03T05:45:19.883 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.878+0000 7fbfae7fc700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf9c0682a0 0x7fbf9c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:19.884 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.878+0000 7fbfae7fc700 1 -- 172.21.15.102:0/2512647476 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fbfb4084a80 con 0x7fbfb806c880 2023-03-03T05:45:19.884 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.878+0000 7fbfbd59a700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf9c0682a0 0x7fbf9c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:19.884 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.879+0000 7fbfbd59a700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf9c0682a0 0x7fbf9c06a750 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8006fd0 tx=0x7fbfa8008070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:19.885 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:19.882+0000 7fbfae7fc700 1 -- 172.21.15.102:0/2512647476 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbfb4052130 con 0x7fbfb806c880 2023-03-03T05:45:20.021 INFO:tasks.ceph.osd.0.smithi102.stderr:2023-03-03T05:45:20.018+0000 7f98651f6700 -1 osd.0 0 waiting for initial osdmap 2023-03-03T05:45:20.049 INFO:tasks.ceph.osd.0.smithi102.stderr:2023-03-03T05:45:20.046+0000 7f985d1fb700 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:20.116 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.113+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb8156c90 con 0x7fbfb806c880 2023-03-03T05:45:20.117 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.114+0000 7fbfae7fc700 1 -- 172.21.15.102:0/2512647476 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+9708 (secure 0 0 0) 0x7fbfb4017090 con 0x7fbfb806c880 2023-03-03T05:45:20.117 INFO:teuthology.misc.health.smithi102.stdout: 2023-03-03T05:45:20.117 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":13,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:19.921722+0000","last_up_change":"2023-03-03T05:45:19.921722+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:20.119 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf9c0682a0 msgr2=0x7fbf9c06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:20.120 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf9c0682a0 0x7fbf9c06a750 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8006fd0 tx=0x7fbfa8008070 comp rx=0 tx=0).stop 2023-03-03T05:45:20.120 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb806c880 msgr2=0x7fbfb80b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:20.120 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb806c880 0x7fbfb80b3e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400e870 tx=0x7fbfb400eb80 comp rx=0 tx=0).stop 2023-03-03T05:45:20.120 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 shutdown_connections 2023-03-03T05:45:20.120 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf9c0682a0 0x7fbf9c06a750 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:20.120 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfb806c880 0x7fbfb80b3e00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:20.121 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfb8152d80 0x7fbfb80c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:20.121 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 --2- 172.21.15.102:0/2512647476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfb80ce510 0x7fbfb80c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:20.121 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 >> 172.21.15.102:0/2512647476 conn(0x7fbfb8060d50 msgr2=0x7fbfb815a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:20.121 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 shutdown_connections 2023-03-03T05:45:20.121 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:20.117+0000 7fbfbed9d700 1 -- 172.21.15.102:0/2512647476 wait complete. 2023-03-03T05:45:20.143 DEBUG:teuthology.misc:6 of 8 OSDs are up 2023-03-03T05:45:20.982 INFO:tasks.ceph.osd.2.smithi102.stderr:2023-03-03T05:45:20.980+0000 7fdcb73f9700 -1 osd.2 0 waiting for initial osdmap 2023-03-03T05:45:21.001 INFO:tasks.ceph.osd.2.smithi102.stderr:2023-03-03T05:45:20.997+0000 7fdcadbfb700 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T05:45:26.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:45:26.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:45:26.951 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:27.182 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.179+0000 7f10aedff700 1 Processor -- start 2023-03-03T05:45:27.182 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.179+0000 7f10aedff700 1 -- start start 2023-03-03T05:45:27.183 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.181+0000 7f10aedff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10a815c4e0 0x7f10a806c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.184 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.181+0000 7f10aedff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a80cbf40 con 0x7f10a815a430 2023-03-03T05:45:27.184 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.181+0000 7f10aedff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a806c8d0 con 0x7f10a815c8b0 2023-03-03T05:45:27.184 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.181+0000 7f10aedff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a80ca390 con 0x7f10a815c4e0 2023-03-03T05:45:27.184 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.181+0000 7f10a7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f10a815a430 0x7f10a80cc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39650/0 (socket says 172.21.15.102:39650) 2023-03-03T05:45:27.184 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.181+0000 7f10a7fff700 1 -- 172.21.15.102:0/925903482 learned_addr learned my addr 172.21.15.102:0/925903482 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:27.184 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.181+0000 7f10acb9b700 1 --2- 172.21.15.102:0/925903482 >> v2:172.21.15.172:3300/0 conn(0x7f10a815c4e0 0x7f10a806c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.185 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10a77fe700 1 -- 172.21.15.102:0/925903482 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093159322 0 0) 0x7f10a80cbf40 con 0x7f10a815a430 2023-03-03T05:45:27.186 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10acb9b700 1 -- 172.21.15.102:0/925903482 >> v1:172.21.15.172:6789/0 conn(0x7f10a815c8b0 legacy=0x7f10a815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.186 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10acb9b700 1 -- 172.21.15.102:0/925903482 >> v1:172.21.15.102:6789/0 conn(0x7f10a815a430 legacy=0x7f10a80cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.186 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10acb9b700 1 -- 172.21.15.102:0/925903482 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a80cb050 con 0x7f10a815c4e0 2023-03-03T05:45:27.186 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10acb9b700 1 --2- 172.21.15.102:0/925903482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a806c390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f109c009840 tx=0x7f109c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=206e5dced9e86dc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:27.186 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10a77fe700 1 -- 172.21.15.102:0/925903482 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c011830 con 0x7f10a815c4e0 2023-03-03T05:45:27.186 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10a77fe700 1 -- 172.21.15.102:0/925903482 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c011990 con 0x7f10a815c4e0 2023-03-03T05:45:27.186 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.182+0000 7f10a77fe700 1 -- 172.21.15.102:0/925903482 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c011d20 con 0x7f10a815c4e0 2023-03-03T05:45:27.187 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.183+0000 7f10aedff700 1 -- 172.21.15.102:0/925903482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 msgr2=0x7f10a806c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.187 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.183+0000 7f10aedff700 1 --2- 172.21.15.102:0/925903482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a806c390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f109c009840 tx=0x7f109c009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:27.187 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.183+0000 7f10aedff700 1 -- 172.21.15.102:0/925903482 shutdown_connections 2023-03-03T05:45:27.187 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.183+0000 7f10aedff700 1 --2- 172.21.15.102:0/925903482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a806c390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.187 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.183+0000 7f10aedff700 1 -- 172.21.15.102:0/925903482 >> 172.21.15.102:0/925903482 conn(0x7f10a80cef80 msgr2=0x7f10a80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:27.187 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.183+0000 7f10aedff700 1 -- 172.21.15.102:0/925903482 shutdown_connections 2023-03-03T05:45:27.188 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.183+0000 7f10aedff700 1 -- 172.21.15.102:0/925903482 wait complete. 2023-03-03T05:45:27.189 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.185+0000 7f10aedff700 1 Processor -- start 2023-03-03T05:45:27.189 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.185+0000 7f10aedff700 1 -- start start 2023-03-03T05:45:27.189 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.186+0000 7f10aedff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10a815a430 0x7f10a81519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.189 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.186+0000 7f10aedff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a81462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.189 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.186+0000 7f10aedff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a815c8b0 0x7f10a8146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.189 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.186+0000 7f10aedff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80c8d60 con 0x7f10a815c8b0 2023-03-03T05:45:27.190 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.186+0000 7f10aedff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80b2630 con 0x7f10a815c4e0 2023-03-03T05:45:27.190 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.186+0000 7f10aedff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80c78d0 con 0x7f10a815a430 2023-03-03T05:45:27.192 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a81462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.192 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10acb9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10a815a430 0x7f10a81519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.193 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a81462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50356/0 (socket says 172.21.15.102:50356) 2023-03-03T05:45:27.193 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 -- 172.21.15.102:0/3326185666 learned_addr learned my addr 172.21.15.102:0/3326185666 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:27.193 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10acb9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10a815a430 0x7f10a81519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60700/0 (socket says 172.21.15.102:60700) 2023-03-03T05:45:27.193 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10ad39c700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a815c8b0 0x7f10a8146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.193 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 -- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10a815a430 msgr2=0x7f10a81519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.193 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10a815a430 0x7f10a81519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.193 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 -- 172.21.15.102:0/3326185666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a815c8b0 msgr2=0x7f10a8146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.194 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a815c8b0 0x7f10a8146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.194 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 -- 172.21.15.102:0/3326185666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c0094a0 con 0x7f10a815c4e0 2023-03-03T05:45:27.194 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a7fff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a81462c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f109400c550 tx=0x7f109400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:27.194 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.187+0000 7f10a57fa700 1 -- 172.21.15.102:0/3326185666 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094004c80 con 0x7f10a815c4e0 2023-03-03T05:45:27.194 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.188+0000 7f10a57fa700 1 -- 172.21.15.102:0/3326185666 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1094004e20 con 0x7f10a815c4e0 2023-03-03T05:45:27.194 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.188+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a80cb610 con 0x7f10a815c4e0 2023-03-03T05:45:27.194 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.188+0000 7f10a57fa700 1 -- 172.21.15.102:0/3326185666 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094007310 con 0x7f10a815c4e0 2023-03-03T05:45:27.195 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.188+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a80cb330 con 0x7f10a815c4e0 2023-03-03T05:45:27.195 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.189+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a80cb8f0 con 0x7f10a815c4e0 2023-03-03T05:45:27.195 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.190+0000 7f10a57fa700 1 -- 172.21.15.102:0/3326185666 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f109401a020 con 0x7f10a815c4e0 2023-03-03T05:45:27.195 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.191+0000 7f10a57fa700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f108c068320 0x7f108c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.196 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.191+0000 7f10a57fa700 1 -- 172.21.15.102:0/3326185666 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1094085390 con 0x7f10a815c4e0 2023-03-03T05:45:27.196 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.192+0000 7f10acb9b700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f108c068320 0x7f108c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.196 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.192+0000 7f10acb9b700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f108c068320 0x7f108c06a7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f109c0090b0 tx=0x7f109c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:27.198 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.196+0000 7f10a57fa700 1 -- 172.21.15.102:0/3326185666 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1094051f20 con 0x7f10a815c4e0 2023-03-03T05:45:27.394 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.391+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10a80af4f0 con 0x7f10a815c4e0 2023-03-03T05:45:27.395 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.392+0000 7f10a57fa700 1 -- 172.21.15.102:0/3326185666 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12526 (secure 0 0 0) 0x7f1094082100 con 0x7f10a815c4e0 2023-03-03T05:45:27.395 INFO:teuthology.misc.health.smithi102.stdout: 2023-03-03T05:45:27.396 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":16,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:22.944545+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:27.398 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f108c068320 msgr2=0x7f108c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.398 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f108c068320 0x7f108c06a7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f109c0090b0 tx=0x7f109c01c040 comp rx=0 tx=0).stop 2023-03-03T05:45:27.399 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 msgr2=0x7f10a81462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.399 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a81462c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f109400c550 tx=0x7f109400c860 comp rx=0 tx=0).stop 2023-03-03T05:45:27.399 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 shutdown_connections 2023-03-03T05:45:27.399 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f108c068320 0x7f108c06a7d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.399 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10a815a430 0x7f10a81519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.399 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a815c8b0 0x7f10a8146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.399 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 --2- 172.21.15.102:0/3326185666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10a815c4e0 0x7f10a81462c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.400 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 >> 172.21.15.102:0/3326185666 conn(0x7f10a80cef80 msgr2=0x7f10a80afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:27.400 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 shutdown_connections 2023-03-03T05:45:27.400 INFO:teuthology.misc.health.smithi102.stderr:2023-03-03T05:45:27.395+0000 7f10aedff700 1 -- 172.21.15.102:0/3326185666 wait complete. 2023-03-03T05:45:27.421 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-03-03T05:45:27.422 INFO:tasks.ceph:Creating RBD pool 2023-03-03T05:45:27.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd pool create rbd 8 2023-03-03T05:45:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.654+0000 7f39197ff700 1 Processor -- start 2023-03-03T05:45:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.654+0000 7f39197ff700 1 -- start start 2023-03-03T05:45:27.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.655+0000 7f39197ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f39140ce410 0x7f39140cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.655+0000 7f39197ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3914153630 con 0x7f39140ccc40 2023-03-03T05:45:27.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.655+0000 7f39197ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f391415d1d0 con 0x7f39140cd760 2023-03-03T05:45:27.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.655+0000 7f39197ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39140ca7a0 con 0x7f39140ce410 2023-03-03T05:45:27.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.656+0000 7f3912ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f39140ccc40 0x7f39140cdd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39652/0 (socket says 172.21.15.102:39652) 2023-03-03T05:45:27.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.656+0000 7f3912ffd700 1 -- 172.21.15.102:0/2935059050 learned_addr learned my addr 172.21.15.102:0/2935059050 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:27.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.656+0000 7f39127fc700 1 --2- 172.21.15.102:0/2935059050 >> v2:172.21.15.172:3300/0 conn(0x7f39140ce410 0x7f39140cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.656+0000 7f39127fc700 1 -- 172.21.15.102:0/2935059050 >> v1:172.21.15.172:6789/0 conn(0x7f39140cd760 legacy=0x7f3914163720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.656+0000 7f39127fc700 1 -- 172.21.15.102:0/2935059050 >> v1:172.21.15.102:6789/0 conn(0x7f39140ccc40 legacy=0x7f39140cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.656+0000 7f3911ffb700 1 -- 172.21.15.102:0/2935059050 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946919352 0 0) 0x7f3914153630 con 0x7f39140ccc40 2023-03-03T05:45:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.656+0000 7f39127fc700 1 -- 172.21.15.102:0/2935059050 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39141591d0 con 0x7f39140ce410 2023-03-03T05:45:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f39127fc700 1 --2- 172.21.15.102:0/2935059050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140ce410 0x7f39140cd010 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f39080097c0 tx=0x7f3908009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=160240bd83c80781 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f3911ffb700 1 -- 172.21.15.102:0/2935059050 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908004230 con 0x7f39140ce410 2023-03-03T05:45:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f3911ffb700 1 -- 172.21.15.102:0/2935059050 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3908004390 con 0x7f39140ce410 2023-03-03T05:45:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f3911ffb700 1 -- 172.21.15.102:0/2935059050 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39080036a0 con 0x7f39140ce410 2023-03-03T05:45:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f39197ff700 1 -- 172.21.15.102:0/2935059050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140ce410 msgr2=0x7f39140cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f39197ff700 1 --2- 172.21.15.102:0/2935059050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140ce410 0x7f39140cd010 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f39080097c0 tx=0x7f3908009ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f39197ff700 1 -- 172.21.15.102:0/2935059050 shutdown_connections 2023-03-03T05:45:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f39197ff700 1 --2- 172.21.15.102:0/2935059050 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140ce410 0x7f39140cd010 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.657+0000 7f39197ff700 1 -- 172.21.15.102:0/2935059050 >> 172.21.15.102:0/2935059050 conn(0x7f3914065fb0 msgr2=0x7f39140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.658+0000 7f39197ff700 1 -- 172.21.15.102:0/2935059050 shutdown_connections 2023-03-03T05:45:27.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.658+0000 7f39197ff700 1 -- 172.21.15.102:0/2935059050 wait complete. 2023-03-03T05:45:27.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.659+0000 7f39197ff700 1 Processor -- start 2023-03-03T05:45:27.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.660+0000 7f39197ff700 1 -- start start 2023-03-03T05:45:27.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39197ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39140ccc40 0x7f39141f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39197ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140cd760 0x7f39141ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39197ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39140ce410 0x7f391414c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39197ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39141f2390 con 0x7f39140ccc40 2023-03-03T05:45:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39197ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39140b22b0 con 0x7f39140cd760 2023-03-03T05:45:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39197ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39140aeb70 con 0x7f39140ce410 2023-03-03T05:45:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39140ccc40 0x7f39141f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39140ccc40 0x7f39141f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43520/0 (socket says 172.21.15.102:43520) 2023-03-03T05:45:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 -- 172.21.15.102:0/4015541397 learned_addr learned my addr 172.21.15.102:0/4015541397 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39127fc700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140cd760 0x7f39141ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39137fe700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39140ce410 0x7f391414c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 -- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39140ce410 msgr2=0x7f391414c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39140ce410 0x7f391414c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 -- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140cd760 msgr2=0x7f39141ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:27.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140cd760 0x7f39141ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:27.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f3912ffd700 1 -- 172.21.15.102:0/4015541397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080094a0 con 0x7f39140ccc40 2023-03-03T05:45:27.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.661+0000 7f39127fc700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140cd760 0x7f39141ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:27.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.662+0000 7f3912ffd700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39140ccc40 0x7f39141f27b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3904001f80 tx=0x7f390400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:27.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.662+0000 7f38fb7fe700 1 -- 172.21.15.102:0/4015541397 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390400acd0 con 0x7f39140ccc40 2023-03-03T05:45:27.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.662+0000 7f38fb7fe700 1 -- 172.21.15.102:0/4015541397 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390400ae50 con 0x7f39140ccc40 2023-03-03T05:45:27.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.662+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3914159790 con 0x7f39140ccc40 2023-03-03T05:45:27.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.662+0000 7f38fb7fe700 1 -- 172.21.15.102:0/4015541397 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3904007310 con 0x7f39140ccc40 2023-03-03T05:45:27.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.662+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39141594b0 con 0x7f39140ccc40 2023-03-03T05:45:27.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.663+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3914159a70 con 0x7f39140ccc40 2023-03-03T05:45:27.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.665+0000 7f38fb7fe700 1 -- 172.21.15.102:0/4015541397 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f390401a030 con 0x7f39140ccc40 2023-03-03T05:45:27.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.667+0000 7f38fb7fe700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39000682f0 0x7f390006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:27.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.667+0000 7f38fb7fe700 1 -- 172.21.15.102:0/4015541397 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f39040855d0 con 0x7f39140ccc40 2023-03-03T05:45:27.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.667+0000 7f39127fc700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39000682f0 0x7f390006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:27.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.668+0000 7f39127fc700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39000682f0 0x7f390006a7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3908009790 tx=0x7f3908008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:27.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.672+0000 7f38fb7fe700 1 -- 172.21.15.102:0/4015541397 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3904052190 con 0x7f39140ccc40 2023-03-03T05:45:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:27.869+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f39141f24d0 con 0x7f39140ccc40 2023-03-03T05:45:28.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.043+0000 7f38fb7fe700 1 -- 172.21.15.102:0/4015541397 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f39141f24d0 con 0x7f39140ccc40 2023-03-03T05:45:28.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39000682f0 msgr2=0x7f390006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39000682f0 0x7f390006a7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3908009790 tx=0x7f3908008030 comp rx=0 tx=0).stop 2023-03-03T05:45:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39140ccc40 msgr2=0x7f39141f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39140ccc40 0x7f39141f27b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3904001f80 tx=0x7f390400a340 comp rx=0 tx=0).stop 2023-03-03T05:45:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 shutdown_connections 2023-03-03T05:45:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39000682f0 0x7f390006a7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39140ce410 0x7f391414c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39140ccc40 0x7f39141f27b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 --2- 172.21.15.102:0/4015541397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39140cd760 0x7f39141ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.046+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 >> 172.21.15.102:0/4015541397 conn(0x7f3914065fb0 msgr2=0x7f391406c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.047+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 shutdown_connections 2023-03-03T05:45:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.047+0000 7f39197ff700 1 -- 172.21.15.102:0/4015541397 wait complete. 2023-03-03T05:45:28.052 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'rbd' created 2023-03-03T05:45:28.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-03-03T05:45:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.315+0000 7f1f873ff700 1 Processor -- start 2023-03-03T05:45:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.315+0000 7f1f873ff700 1 -- start start 2023-03-03T05:45:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.316+0000 7f1f873ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f800cbf40 0x7f1f800cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.316+0000 7f1f873ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f8015c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f873ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8015a0f0 con 0x7f1f80153050 2023-03-03T05:45:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f873ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8006c870 con 0x7f1f800cbf40 2023-03-03T05:45:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f873ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8006c660 con 0x7f1f8015c1f0 2023-03-03T05:45:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8499a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f800cbf40 0x7f1f800cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8599c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f8015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8519b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1f80153050 0x7f1f8006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57828/0 (socket says 172.21.15.102:57828) 2023-03-03T05:45:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8499a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f800cbf40 0x7f1f800cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50366/0 (socket says 172.21.15.102:50366) 2023-03-03T05:45:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8519b700 1 -- 172.21.15.102:0/3140353023 learned_addr learned my addr 172.21.15.102:0/3140353023 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8599c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f8015c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60712/0 (socket says 172.21.15.102:60712) 2023-03-03T05:45:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f77fff700 1 -- 172.21.15.102:0/3140353023 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566510272 0 0) 0x7f1f8015a0f0 con 0x7f1f80153050 2023-03-03T05:45:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f77fff700 1 -- 172.21.15.102:0/3140353023 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c0035b0 con 0x7f1f80153050 2023-03-03T05:45:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8599c700 1 -- 172.21.15.102:0/3140353023 >> v2:172.21.15.172:3300/0 conn(0x7f1f800cbf40 msgr2=0x7f1f800cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8599c700 1 --2- 172.21.15.102:0/3140353023 >> v2:172.21.15.172:3300/0 conn(0x7f1f800cbf40 0x7f1f800cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8599c700 1 -- 172.21.15.102:0/3140353023 >> v1:172.21.15.172:6789/0 conn(0x7f1f80153050 legacy=0x7f1f8006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8599c700 1 -- 172.21.15.102:0/3140353023 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80153420 con 0x7f1f8015c1f0 2023-03-03T05:45:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.317+0000 7f1f8499a700 1 --2- 172.21.15.102:0/3140353023 >> v2:172.21.15.172:3300/0 conn(0x7f1f800cbf40 0x7f1f800cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.318+0000 7f1f8599c700 1 --2- 172.21.15.102:0/3140353023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f8015c610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009d90 tx=0x7f1f7800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eafac33e3e68ed16 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.318+0000 7f1f77fff700 1 -- 172.21.15.102:0/3140353023 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7800e070 con 0x7f1f8015c1f0 2023-03-03T05:45:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.318+0000 7f1f77fff700 1 -- 172.21.15.102:0/3140353023 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7800bb20 con 0x7f1f8015c1f0 2023-03-03T05:45:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.318+0000 7f1f77fff700 1 -- 172.21.15.102:0/3140353023 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78012430 con 0x7f1f8015c1f0 2023-03-03T05:45:28.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.318+0000 7f1f873ff700 1 -- 172.21.15.102:0/3140353023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 msgr2=0x7f1f8015c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:28.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.318+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3140353023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f8015c610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009d90 tx=0x7f1f7800b330 comp rx=0 tx=0).stop 2023-03-03T05:45:28.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.319+0000 7f1f873ff700 1 -- 172.21.15.102:0/3140353023 shutdown_connections 2023-03-03T05:45:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.319+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3140353023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f8015c610 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.319+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3140353023 >> v2:172.21.15.172:3300/0 conn(0x7f1f800cbf40 0x7f1f800cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.319+0000 7f1f873ff700 1 -- 172.21.15.102:0/3140353023 >> 172.21.15.102:0/3140353023 conn(0x7f1f800b08c0 msgr2=0x7f1f800b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.319+0000 7f1f873ff700 1 -- 172.21.15.102:0/3140353023 shutdown_connections 2023-03-03T05:45:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.319+0000 7f1f873ff700 1 -- 172.21.15.102:0/3140353023 wait complete. 2023-03-03T05:45:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.321+0000 7f1f873ff700 1 Processor -- start 2023-03-03T05:45:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.321+0000 7f1f873ff700 1 -- start start 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f873ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f800cbf40 0x7f1f80155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f873ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f80153050 0x7f1f800b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f873ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f800b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f873ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800b2710 con 0x7f1f800cbf40 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f873ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800b4210 con 0x7f1f80153050 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f873ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800b44f0 con 0x7f1f8015c1f0 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f8519b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f800cbf40 0x7f1f80155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f8519b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f800cbf40 0x7f1f80155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43534/0 (socket says 172.21.15.102:43534) 2023-03-03T05:45:28.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f8519b700 1 -- 172.21.15.102:0/3137096387 learned_addr learned my addr 172.21.15.102:0/3137096387 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:28.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f8499a700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f80153050 0x7f1f800b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:28.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.322+0000 7f1f8599c700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f800b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:28.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f8519b700 1 -- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 msgr2=0x7f1f800b2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:28.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f8519b700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f800b2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f8519b700 1 -- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f80153050 msgr2=0x7f1f800b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:28.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f8519b700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f80153050 0x7f1f800b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:28.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f8519b700 1 -- 172.21.15.102:0/3137096387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f700094a0 con 0x7f1f800cbf40 2023-03-03T05:45:28.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f8499a700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f80153050 0x7f1f800b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:28.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f8519b700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f800cbf40 0x7f1f80155760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00c700 tx=0x7f1f7c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:28.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f75ffb700 1 -- 172.21.15.102:0/3137096387 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c005d70 con 0x7f1f800cbf40 2023-03-03T05:45:28.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f75ffb700 1 -- 172.21.15.102:0/3137096387 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0070d0 con 0x7f1f800cbf40 2023-03-03T05:45:28.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f75ffb700 1 -- 172.21.15.102:0/3137096387 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c007460 con 0x7f1f800cbf40 2023-03-03T05:45:28.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.323+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f78009a70 con 0x7f1f800cbf40 2023-03-03T05:45:28.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.324+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f800caea0 con 0x7f1f800cbf40 2023-03-03T05:45:28.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.325+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8006d360 con 0x7f1f800cbf40 2023-03-03T05:45:28.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.326+0000 7f1f75ffb700 1 -- 172.21.15.102:0/3137096387 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f7c019030 con 0x7f1f800cbf40 2023-03-03T05:45:28.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.328+0000 7f1f75ffb700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f640682f0 0x7f1f6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:28.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.328+0000 7f1f75ffb700 1 -- 172.21.15.102:0/3137096387 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f1f7c084dc0 con 0x7f1f800cbf40 2023-03-03T05:45:28.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.328+0000 7f1f8499a700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f640682f0 0x7f1f6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:28.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.329+0000 7f1f8499a700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f640682f0 0x7f1f6406a7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f800cbd40 tx=0x7f1f70009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:28.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.333+0000 7f1f75ffb700 1 -- 172.21.15.102:0/3137096387 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f7c051800 con 0x7f1f800cbf40 2023-03-03T05:45:28.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:28.525+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1f800cb180 con 0x7f1f800cbf40 2023-03-03T05:45:29.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.045+0000 7f1f75ffb700 1 -- 172.21.15.102:0/3137096387 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f1f7c017030 con 0x7f1f800cbf40 2023-03-03T05:45:29.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f640682f0 msgr2=0x7f1f6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f640682f0 0x7f1f6406a7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f800cbd40 tx=0x7f1f70009210 comp rx=0 tx=0).stop 2023-03-03T05:45:29.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f800cbf40 msgr2=0x7f1f80155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f800cbf40 0x7f1f80155760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00c700 tx=0x7f1f7c00ca10 comp rx=0 tx=0).stop 2023-03-03T05:45:29.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 shutdown_connections 2023-03-03T05:45:29.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f640682f0 0x7f1f6406a7a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f8015c1f0 0x7f1f800b2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f800cbf40 0x7f1f80155760 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 --2- 172.21.15.102:0/3137096387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f80153050 0x7f1f800b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.047+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 >> 172.21.15.102:0/3137096387 conn(0x7f1f800b08c0 msgr2=0x7f1f801525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:29.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.048+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 shutdown_connections 2023-03-03T05:45:29.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.048+0000 7f1f873ff700 1 -- 172.21.15.102:0/3137096387 wait complete. 2023-03-03T05:45:29.052 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rbd' on pool 'rbd' 2023-03-03T05:45:29.080 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-03-03T05:45:29.082 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T05:45:29.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T05:45:29.087 INFO:tasks.ceph.mds.a:Started 2023-03-03T05:45:29.088 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T05:45:29.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T05:45:29.093 INFO:tasks.ceph.mds.c:Started 2023-03-03T05:45:29.094 INFO:tasks.ceph.mds.e:Restarting daemon 2023-03-03T05:45:29.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-03-03T05:45:29.101 INFO:tasks.ceph.mds.e:Started 2023-03-03T05:45:29.101 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T05:45:29.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T05:45:29.104 INFO:tasks.ceph.mds.b:Started 2023-03-03T05:45:29.105 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T05:45:29.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T05:45:29.109 INFO:tasks.ceph.mds.d:Started 2023-03-03T05:45:29.110 INFO:tasks.ceph.mds.f:Restarting daemon 2023-03-03T05:45:29.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-03-03T05:45:29.112 INFO:tasks.ceph.mds.f:Started 2023-03-03T05:45:29.112 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-03-03T05:45:29.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:29.194 INFO:tasks.ceph.mds.b.smithi172.stdout:starting mds.b at 2023-03-03T05:45:29.195 INFO:tasks.ceph.mds.d.smithi172.stdout:starting mds.d at 2023-03-03T05:45:29.200 INFO:tasks.ceph.mds.f.smithi172.stdout:starting mds.f at 2023-03-03T05:45:29.304 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2023-03-03T05:45:29.322 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2023-03-03T05:45:29.324 INFO:tasks.ceph.mds.e.smithi102.stdout:starting mds.e at 2023-03-03T05:45:29.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.394+0000 7ff8fc1ff700 1 Processor -- start 2023-03-03T05:45:29.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.394+0000 7ff8fc1ff700 1 -- start start 2023-03-03T05:45:29.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8fc1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff8f40ced80 0x7ff8f40cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8fc1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff8f40cf9a0 0x7ff8f4163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8fc1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40cb9f0 con 0x7ff8f41597f0 2023-03-03T05:45:29.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8fc1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f40ca750 con 0x7ff8f40ced80 2023-03-03T05:45:29.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8fc1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f415a230 con 0x7ff8f40cf9a0 2023-03-03T05:45:29.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8f9f9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff8f40ced80 0x7ff8f40cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8f9f9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff8f40ced80 0x7ff8f40cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43590/0 (socket says 172.21.15.102:43590) 2023-03-03T05:45:29.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/299060201 learned_addr learned my addr 172.21.15.102:0/299060201 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:29.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8fa79c700 1 --2- 172.21.15.102:0/299060201 >> v2:172.21.15.172:3300/0 conn(0x7ff8f40cf9a0 0x7ff8f4163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/299060201 >> v2:172.21.15.172:3300/0 conn(0x7ff8f40cf9a0 msgr2=0x7ff8f4163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:29.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8f9f9b700 1 --2- 172.21.15.102:0/299060201 >> v2:172.21.15.172:3300/0 conn(0x7ff8f40cf9a0 0x7ff8f4163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.396+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/299060201 >> v1:172.21.15.102:6789/0 conn(0x7ff8f41597f0 legacy=0x7ff8f40cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.397+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/299060201 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4159c80 con 0x7ff8f40ced80 2023-03-03T05:45:29.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.397+0000 7ff8f9f9b700 1 --2- 172.21.15.102:0/299060201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40ced80 0x7ff8f40cf150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff8f415d3d0 tx=0x7ff8f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f61c2123892031a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:29.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.397+0000 7ff8f8f99700 1 -- 172.21.15.102:0/299060201 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0011830 con 0x7ff8f40ced80 2023-03-03T05:45:29.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.397+0000 7ff8f8f99700 1 -- 172.21.15.102:0/299060201 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f0011990 con 0x7ff8f40ced80 2023-03-03T05:45:29.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.397+0000 7ff8f8f99700 1 -- 172.21.15.102:0/299060201 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0011d20 con 0x7ff8f40ced80 2023-03-03T05:45:29.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/299060201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40ced80 msgr2=0x7ff8f40cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/299060201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40ced80 0x7ff8f40cf150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff8f415d3d0 tx=0x7ff8f0009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/299060201 shutdown_connections 2023-03-03T05:45:29.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/299060201 >> v2:172.21.15.172:3300/0 conn(0x7ff8f40cf9a0 0x7ff8f4163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/299060201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40ced80 0x7ff8f40cf150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/299060201 >> 172.21.15.102:0/299060201 conn(0x7ff8f4065fb0 msgr2=0x7ff8f40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:29.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/299060201 shutdown_connections 2023-03-03T05:45:29.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.398+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/299060201 wait complete. 2023-03-03T05:45:29.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.400+0000 7ff8fc1ff700 1 Processor -- start 2023-03-03T05:45:29.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.400+0000 7ff8fc1ff700 1 -- start start 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.401+0000 7ff8fc1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8f40ced80 0x7ff8f40b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.401+0000 7ff8fc1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40cf9a0 0x7ff8f40b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.401+0000 7ff8fc1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8f41597f0 0x7ff8f40c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.401+0000 7ff8fc1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40b4ba0 con 0x7ff8f40cf9a0 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.401+0000 7ff8fc1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40c6c80 con 0x7ff8f41597f0 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.401+0000 7ff8fc1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40b0650 con 0x7ff8f40ced80 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8f40ced80 0x7ff8f40b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.401+0000 7ff8f979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40cf9a0 0x7ff8f40b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8f40ced80 0x7ff8f40b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60770/0 (socket says 172.21.15.102:60770) 2023-03-03T05:45:29.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/2144658563 learned_addr learned my addr 172.21.15.102:0/2144658563 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:29.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8fa79c700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8f41597f0 0x7ff8f40c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8f41597f0 msgr2=0x7ff8f40c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8f41597f0 0x7ff8f40c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/2144658563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40cf9a0 msgr2=0x7ff8f40b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40cf9a0 0x7ff8f40b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 -- 172.21.15.102:0/2144658563 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0009550 con 0x7ff8f40ced80 2023-03-03T05:45:29.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8f9f9b700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8f40ced80 0x7ff8f40b0950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0009c10 tx=0x7ff8f0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:29.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.402+0000 7ff8ea7fc700 1 -- 172.21.15.102:0/2144658563 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0004510 con 0x7ff8f40ced80 2023-03-03T05:45:29.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.403+0000 7ff8ea7fc700 1 -- 172.21.15.102:0/2144658563 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f001e070 con 0x7ff8f40ced80 2023-03-03T05:45:29.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.403+0000 7ff8ea7fc700 1 -- 172.21.15.102:0/2144658563 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00038e0 con 0x7ff8f40ced80 2023-03-03T05:45:29.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.403+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f40cb0c0 con 0x7ff8f40ced80 2023-03-03T05:45:29.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.403+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f40cade0 con 0x7ff8f40ced80 2023-03-03T05:45:29.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.404+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f40cb3a0 con 0x7ff8f40ced80 2023-03-03T05:45:29.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.405+0000 7ff8ea7fc700 1 -- 172.21.15.102:0/2144658563 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8f000f050 con 0x7ff8f40ced80 2023-03-03T05:45:29.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.406+0000 7ff8ea7fc700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff8d8068320 0x7ff8d806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.406+0000 7ff8ea7fc700 1 -- 172.21.15.102:0/2144658563 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff8f0086820 con 0x7ff8f40ced80 2023-03-03T05:45:29.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.406+0000 7ff8f979a700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff8d8068320 0x7ff8d806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.407+0000 7ff8f979a700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff8d8068320 0x7ff8d806a7d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4158b40 tx=0x7ff8e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:29.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.411+0000 7ff8ea7fc700 1 -- 172.21.15.102:0/2144658563 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8f00532e0 con 0x7ff8f40ced80 2023-03-03T05:45:29.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.603+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8f40677f0 con 0x7ff8f40ced80 2023-03-03T05:45:29.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.605+0000 7ff8ea7fc700 1 -- 172.21.15.102:0/2144658563 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14351 (secure 0 0 0) 0x7ff8f0020090 con 0x7ff8f40ced80 2023-03-03T05:45:29.608 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:29.608 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":18,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:29.042299+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:29.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff8d8068320 msgr2=0x7ff8d806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff8d8068320 0x7ff8d806a7d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4158b40 tx=0x7ff8e4009040 comp rx=0 tx=0).stop 2023-03-03T05:45:29.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8f40ced80 msgr2=0x7ff8f40b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8f40ced80 0x7ff8f40b0950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0009c10 tx=0x7ff8f0004750 comp rx=0 tx=0).stop 2023-03-03T05:45:29.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 shutdown_connections 2023-03-03T05:45:29.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff8d8068320 0x7ff8d806a7d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff8f40ced80 0x7ff8f40b0950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8f41597f0 0x7ff8f40c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 --2- 172.21.15.102:0/2144658563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f40cf9a0 0x7ff8f40b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 >> 172.21.15.102:0/2144658563 conn(0x7ff8f4065fb0 msgr2=0x7ff8f40cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:29.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 shutdown_connections 2023-03-03T05:45:29.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.607+0000 7ff8fc1ff700 1 -- 172.21.15.102:0/2144658563 wait complete. 2023-03-03T05:45:29.637 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T05:45:20.032740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T05:45:27.930463+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T05:45:29.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T05:45:29.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.883+0000 7f6811dff700 1 Processor -- start 2023-03-03T05:45:29.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.883+0000 7f6811dff700 1 -- start start 2023-03-03T05:45:29.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.884+0000 7f6811dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f680c15a380 0x7f680c0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.884+0000 7f6811dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680c0cae70 con 0x7f680c0cf9a0 2023-03-03T05:45:29.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.884+0000 7f6811dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680c0cb110 con 0x7f680c153450 2023-03-03T05:45:29.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.884+0000 7f6811dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680c0cb3f0 con 0x7f680c15a380 2023-03-03T05:45:29.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.885+0000 7f680b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f680c153450 0x7f680c159940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57848/0 (socket says 172.21.15.102:57848) 2023-03-03T05:45:29.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.885+0000 7f680b7fe700 1 -- 172.21.15.102:0/927413180 learned_addr learned my addr 172.21.15.102:0/927413180 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:29.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.885+0000 7f680affd700 1 --2- 172.21.15.102:0/927413180 >> v2:172.21.15.172:3300/0 conn(0x7f680c15a380 0x7f680c0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.885+0000 7f680a7fc700 1 -- 172.21.15.102:0/927413180 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210906850 0 0) 0x7f680c0cb110 con 0x7f680c153450 2023-03-03T05:45:29.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.885+0000 7f680affd700 1 -- 172.21.15.102:0/927413180 >> v1:172.21.15.172:6789/0 conn(0x7f680c153450 legacy=0x7f680c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.885+0000 7f680affd700 1 -- 172.21.15.102:0/927413180 >> v1:172.21.15.102:6789/0 conn(0x7f680c0cf9a0 legacy=0x7f680c0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.885+0000 7f680affd700 1 -- 172.21.15.102:0/927413180 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c0cb6d0 con 0x7f680c15a380 2023-03-03T05:45:29.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f680affd700 1 --2- 172.21.15.102:0/927413180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0cc1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6800009840 tx=0x7f6800009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=796bbfd83c5302e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:29.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f680a7fc700 1 -- 172.21.15.102:0/927413180 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800004230 con 0x7f680c15a380 2023-03-03T05:45:29.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f680a7fc700 1 -- 172.21.15.102:0/927413180 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6800004390 con 0x7f680c15a380 2023-03-03T05:45:29.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f680a7fc700 1 -- 172.21.15.102:0/927413180 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68000036a0 con 0x7f680c15a380 2023-03-03T05:45:29.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f6811dff700 1 -- 172.21.15.102:0/927413180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 msgr2=0x7f680c0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f6811dff700 1 --2- 172.21.15.102:0/927413180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0cc1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6800009840 tx=0x7f6800009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:29.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f6811dff700 1 -- 172.21.15.102:0/927413180 shutdown_connections 2023-03-03T05:45:29.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f6811dff700 1 --2- 172.21.15.102:0/927413180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0cc1b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.886+0000 7f6811dff700 1 -- 172.21.15.102:0/927413180 >> 172.21.15.102:0/927413180 conn(0x7f680c065fb0 msgr2=0x7f680c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:29.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.887+0000 7f6811dff700 1 -- 172.21.15.102:0/927413180 shutdown_connections 2023-03-03T05:45:29.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.887+0000 7f6811dff700 1 -- 172.21.15.102:0/927413180 wait complete. 2023-03-03T05:45:29.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.889+0000 7f6811dff700 1 Processor -- start 2023-03-03T05:45:29.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.889+0000 7f6811dff700 1 -- start start 2023-03-03T05:45:29.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f6811dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f680c0cf9a0 0x7f680c0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f6811dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f680c153450 0x7f680c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f6811dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f6811dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0b1840 con 0x7f680c153450 2023-03-03T05:45:29.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f6811dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c158060 con 0x7f680c15a380 2023-03-03T05:45:29.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f6811dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0b0a20 con 0x7f680c0cf9a0 2023-03-03T05:45:29.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f680b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f680c0cf9a0 0x7f680c0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f680bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f680affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f680c153450 0x7f680c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f680bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50468/0 (socket says 172.21.15.102:50468) 2023-03-03T05:45:29.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.890+0000 7f680bfff700 1 -- 172.21.15.102:0/2942400074 learned_addr learned my addr 172.21.15.102:0/2942400074 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:29.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680b7fe700 1 -- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 msgr2=0x7f680c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680b7fe700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680b7fe700 1 -- 172.21.15.102:0/2942400074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f680c153450 msgr2=0x7f680c0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:29.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680b7fe700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f680c153450 0x7f680c0cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:29.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680b7fe700 1 -- 172.21.15.102:0/2942400074 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000094a0 con 0x7f680c0cf9a0 2023-03-03T05:45:29.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680bfff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680affd700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f680c153450 0x7f680c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f680b7fe700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f680c0cf9a0 0x7f680c0c72c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c4b0 tx=0x7f67fc00c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f67f3fff700 1 -- 172.21.15.102:0/2942400074 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc004c80 con 0x7f680c0cf9a0 2023-03-03T05:45:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c0cbc90 con 0x7f680c0cf9a0 2023-03-03T05:45:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.891+0000 7f67f3fff700 1 -- 172.21.15.102:0/2942400074 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc004e00 con 0x7f680c0cf9a0 2023-03-03T05:45:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.892+0000 7f67f3fff700 1 -- 172.21.15.102:0/2942400074 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0072d0 con 0x7f680c0cf9a0 2023-03-03T05:45:29.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.892+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c0cb9b0 con 0x7f680c0cf9a0 2023-03-03T05:45:29.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.892+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c0cf490 con 0x7f680c0cf9a0 2023-03-03T05:45:29.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.894+0000 7f67f3fff700 1 -- 172.21.15.102:0/2942400074 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f67fc01a020 con 0x7f680c0cf9a0 2023-03-03T05:45:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.895+0000 7f67f3fff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67f8068320 0x7f67f806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.895+0000 7f67f3fff700 1 -- 172.21.15.102:0/2942400074 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f67fc085330 con 0x7f680c0cf9a0 2023-03-03T05:45:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.895+0000 7f680affd700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67f8068320 0x7f67f806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.896+0000 7f680affd700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67f8068320 0x7f67f806a7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f68000090b0 tx=0x7f6800008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:29.899+0000 7f67f3fff700 1 -- 172.21.15.102:0/2942400074 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f67fc051e30 con 0x7f680c0cf9a0 2023-03-03T05:45:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.094+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f680c0677f0 con 0x7f680c0cf9a0 2023-03-03T05:45:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.094+0000 7f67f3fff700 1 -- 172.21.15.102:0/2942400074 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f67fc055450 con 0x7f680c0cf9a0 2023-03-03T05:45:30.098 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2023-03-03T05:45:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67f8068320 msgr2=0x7f67f806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67f8068320 0x7f67f806a7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f68000090b0 tx=0x7f6800008030 comp rx=0 tx=0).stop 2023-03-03T05:45:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f680c0cf9a0 msgr2=0x7f680c0c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f680c0cf9a0 0x7f680c0c72c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c4b0 tx=0x7f67fc00c7c0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 shutdown_connections 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67f8068320 0x7f67f806a7d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f680c0cf9a0 0x7f680c0c72c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f680c15a380 0x7f680c0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 --2- 172.21.15.102:0/2942400074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f680c153450 0x7f680c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 >> 172.21.15.102:0/2942400074 conn(0x7f680c065fb0 msgr2=0x7f680c0ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 shutdown_connections 2023-03-03T05:45:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.097+0000 7f6811dff700 1 -- 172.21.15.102:0/2942400074 wait complete. 2023-03-03T05:45:30.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T05:45:30.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.375+0000 7ffb909ff700 1 Processor -- start 2023-03-03T05:45:30.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.375+0000 7ffb909ff700 1 -- start start 2023-03-03T05:45:30.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.376+0000 7ffb909ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb8c153450 0x7ffb8c0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.376+0000 7ffb909ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c159940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.376+0000 7ffb909ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffb8c0cf9a0 0x7ffb8c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb909ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb8c159e80 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb909ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb8c0cae70 con 0x7ffb8c153450 2023-03-03T05:45:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb909ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c0cb110 con 0x7ffb8c15a380 2023-03-03T05:45:30.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb89d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb8c153450 0x7ffb8c0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb8c153450 0x7ffb8c0cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50482/0 (socket says 172.21.15.102:50482) 2023-03-03T05:45:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 -- 172.21.15.102:0/3880997630 learned_addr learned my addr 172.21.15.102:0/3880997630 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8ad9d700 1 --2- 172.21.15.102:0/3880997630 >> v2:172.21.15.102:3300/0 conn(0x7ffb8c0cf9a0 0x7ffb8c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 -- 172.21.15.102:0/3880997630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 msgr2=0x7ffb8c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 --2- 172.21.15.102:0/3880997630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 -- 172.21.15.102:0/3880997630 >> v2:172.21.15.102:3300/0 conn(0x7ffb8c0cf9a0 msgr2=0x7ffb8c0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:30.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 --2- 172.21.15.102:0/3880997630 >> v2:172.21.15.102:3300/0 conn(0x7ffb8c0cf9a0 0x7ffb8c0ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.377+0000 7ffb8a59c700 1 -- 172.21.15.102:0/3880997630 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c0cb3f0 con 0x7ffb8c153450 2023-03-03T05:45:30.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.378+0000 7ffb8a59c700 1 --2- 172.21.15.102:0/3880997630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c153450 0x7ffb8c0cc1b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c0ced50 tx=0x7ffb7c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=852347e009fcbadc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:30.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.378+0000 7ffb8959a700 1 -- 172.21.15.102:0/3880997630 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c011830 con 0x7ffb8c153450 2023-03-03T05:45:30.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.378+0000 7ffb8959a700 1 -- 172.21.15.102:0/3880997630 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c011990 con 0x7ffb8c153450 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.378+0000 7ffb8959a700 1 -- 172.21.15.102:0/3880997630 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c011d20 con 0x7ffb8c153450 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.378+0000 7ffb909ff700 1 -- 172.21.15.102:0/3880997630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c153450 msgr2=0x7ffb8c0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.378+0000 7ffb909ff700 1 --2- 172.21.15.102:0/3880997630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c153450 0x7ffb8c0cc1b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c0ced50 tx=0x7ffb7c009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.379+0000 7ffb909ff700 1 -- 172.21.15.102:0/3880997630 shutdown_connections 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.379+0000 7ffb909ff700 1 --2- 172.21.15.102:0/3880997630 >> v2:172.21.15.102:3300/0 conn(0x7ffb8c0cf9a0 0x7ffb8c0ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.379+0000 7ffb909ff700 1 --2- 172.21.15.102:0/3880997630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c153450 0x7ffb8c0cc1b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.379+0000 7ffb909ff700 1 --2- 172.21.15.102:0/3880997630 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.379+0000 7ffb909ff700 1 -- 172.21.15.102:0/3880997630 >> 172.21.15.102:0/3880997630 conn(0x7ffb8c065fb0 msgr2=0x7ffb8c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.379+0000 7ffb909ff700 1 -- 172.21.15.102:0/3880997630 shutdown_connections 2023-03-03T05:45:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.379+0000 7ffb909ff700 1 -- 172.21.15.102:0/3880997630 wait complete. 2023-03-03T05:45:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.381+0000 7ffb909ff700 1 Processor -- start 2023-03-03T05:45:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.381+0000 7ffb909ff700 1 -- start start 2023-03-03T05:45:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb909ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c0cf9a0 0x7ffb8c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb909ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb8c153450 0x7ffb8c0b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb909ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c0c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb909ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c0c7340 con 0x7ffb8c153450 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb909ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c0b2f00 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb909ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c0c6250 con 0x7ffb8c15a380 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb8a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c0cf9a0 0x7ffb8c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb8a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c0cf9a0 0x7ffb8c0b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50494/0 (socket says 172.21.15.102:50494) 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb8a59c700 1 -- 172.21.15.102:0/600296316 learned_addr learned my addr 172.21.15.102:0/600296316 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.382+0000 7ffb8ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c0c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb89d9b700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb8c153450 0x7ffb8c0b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb8a59c700 1 -- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 msgr2=0x7ffb8c0c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb8a59c700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c0c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb8a59c700 1 -- 172.21.15.102:0/600296316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb8c153450 msgr2=0x7ffb8c0b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb8a59c700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb8c153450 0x7ffb8c0b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb8a59c700 1 -- 172.21.15.102:0/600296316 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c009550 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb8a59c700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c0cf9a0 0x7ffb8c0b1c20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c014040 tx=0x7ffb7c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb7affd700 1 -- 172.21.15.102:0/600296316 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c004450 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb7affd700 1 -- 172.21.15.102:0/600296316 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c01e070 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb7affd700 1 -- 172.21.15.102:0/600296316 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c0038e0 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.383+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c0cb9b0 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.384+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c0cb6d0 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.385+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c0cbc70 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.386+0000 7ffb7affd700 1 -- 172.21.15.102:0/600296316 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb7c00f050 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.387+0000 7ffb7affd700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb6c068320 0x7ffb6c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.387+0000 7ffb7affd700 1 -- 172.21.15.102:0/600296316 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffb7c0868e0 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.387+0000 7ffb89d9b700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb6c068320 0x7ffb6c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.388+0000 7ffb89d9b700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb6c068320 0x7ffb6c06a7d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c0c91d0 tx=0x7ffb80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:30.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.392+0000 7ffb7affd700 1 -- 172.21.15.102:0/600296316 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb7c0533a0 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.584+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ffb8c0677f0 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.585+0000 7ffb7affd700 1 -- 172.21.15.102:0/600296316 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffb7c0569c0 con 0x7ffb8c0cf9a0 2023-03-03T05:45:30.587 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2023-03-03T05:45:30.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb6c068320 msgr2=0x7ffb6c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb6c068320 0x7ffb6c06a7d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c0c91d0 tx=0x7ffb80008040 comp rx=0 tx=0).stop 2023-03-03T05:45:30.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c0cf9a0 msgr2=0x7ffb8c0b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c0cf9a0 0x7ffb8c0b1c20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c014040 tx=0x7ffb7c004750 comp rx=0 tx=0).stop 2023-03-03T05:45:30.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 shutdown_connections 2023-03-03T05:45:30.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb6c068320 0x7ffb6c06a7d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb8c15a380 0x7ffb8c0c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb8c0cf9a0 0x7ffb8c0b1c20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 --2- 172.21.15.102:0/600296316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb8c153450 0x7ffb8c0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.587+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 >> 172.21.15.102:0/600296316 conn(0x7ffb8c065fb0 msgr2=0x7ffb8c156d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.588+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 shutdown_connections 2023-03-03T05:45:30.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.588+0000 7ffb909ff700 1 -- 172.21.15.102:0/600296316 wait complete. 2023-03-03T05:45:30.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:45:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.859+0000 7fdbcf39d700 1 Processor -- start 2023-03-03T05:45:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.860+0000 7fdbcf39d700 1 -- start start 2023-03-03T05:45:30.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbcf39d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbc81564b0 0x7fdbc80cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbcf39d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbc80c9610 0x7fdbc80cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbcf39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc80c9ca0 con 0x7fdbc80ca440 2023-03-03T05:45:30.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbcf39d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc80b4920 con 0x7fdbc80c9610 2023-03-03T05:45:30.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbcf39d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc80b4c00 con 0x7fdbc81564b0 2023-03-03T05:45:30.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbcdb9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbc80c9610 0x7fdbc80cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbce39b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbc81564b0 0x7fdbc80cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbce39b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbc81564b0 0x7fdbc80cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50500/0 (socket says 172.21.15.102:50500) 2023-03-03T05:45:30.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbcdb9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbc80c9610 0x7fdbc80cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43632/0 (socket says 172.21.15.102:43632) 2023-03-03T05:45:30.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.861+0000 7fdbce39b700 1 -- 172.21.15.102:0/3722871756 learned_addr learned my addr 172.21.15.102:0/3722871756 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbce39b700 1 -- 172.21.15.102:0/3722871756 >> v2:172.21.15.102:3300/0 conn(0x7fdbc80c9610 msgr2=0x7fdbc80cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbce39b700 1 --2- 172.21.15.102:0/3722871756 >> v2:172.21.15.102:3300/0 conn(0x7fdbc80c9610 0x7fdbc80cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbce39b700 1 -- 172.21.15.102:0/3722871756 >> v1:172.21.15.102:6789/0 conn(0x7fdbc80ca440 legacy=0x7fdbc80ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbce39b700 1 -- 172.21.15.102:0/3722871756 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80b46f0 con 0x7fdbc81564b0 2023-03-03T05:45:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbcdb9a700 1 --2- 172.21.15.102:0/3722871756 >> v2:172.21.15.102:3300/0 conn(0x7fdbc80c9610 0x7fdbc80cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:30.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbce39b700 1 --2- 172.21.15.102:0/3722871756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 0x7fdbc80cb660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdbc808c600 tx=0x7fdbc4009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884c42656c077931 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:30.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbcd399700 1 -- 172.21.15.102:0/3722871756 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4011830 con 0x7fdbc81564b0 2023-03-03T05:45:30.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.862+0000 7fdbcd399700 1 -- 172.21.15.102:0/3722871756 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc4011990 con 0x7fdbc81564b0 2023-03-03T05:45:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcd399700 1 -- 172.21.15.102:0/3722871756 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4011d20 con 0x7fdbc81564b0 2023-03-03T05:45:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 -- 172.21.15.102:0/3722871756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 msgr2=0x7fdbc80cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/3722871756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 0x7fdbc80cb660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdbc808c600 tx=0x7fdbc4009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 -- 172.21.15.102:0/3722871756 shutdown_connections 2023-03-03T05:45:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/3722871756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 0x7fdbc80cb660 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/3722871756 >> v2:172.21.15.102:3300/0 conn(0x7fdbc80c9610 0x7fdbc80cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 -- 172.21.15.102:0/3722871756 >> 172.21.15.102:0/3722871756 conn(0x7fdbc8065fb0 msgr2=0x7fdbc80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:30.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 -- 172.21.15.102:0/3722871756 shutdown_connections 2023-03-03T05:45:30.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.863+0000 7fdbcf39d700 1 -- 172.21.15.102:0/3722871756 wait complete. 2023-03-03T05:45:30.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.865+0000 7fdbcf39d700 1 Processor -- start 2023-03-03T05:45:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.866+0000 7fdbcf39d700 1 -- start start 2023-03-03T05:45:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcf39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80c9610 0x7fdbc80c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcf39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc80ca440 0x7fdbc814e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcf39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 0x7fdbc814e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcf39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc81f2210 con 0x7fdbc80ca440 2023-03-03T05:45:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcf39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc814ecd0 con 0x7fdbc81564b0 2023-03-03T05:45:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcf39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc815c240 con 0x7fdbc80c9610 2023-03-03T05:45:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc80ca440 0x7fdbc814e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc80ca440 0x7fdbc814e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43634/0 (socket says 172.21.15.102:43634) 2023-03-03T05:45:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 -- 172.21.15.102:0/894111610 learned_addr learned my addr 172.21.15.102:0/894111610 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbce39b700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80c9610 0x7fdbc80c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbceb9c700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 0x7fdbc814e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 -- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80c9610 msgr2=0x7fdbc80c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80c9610 0x7fdbc80c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 -- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 msgr2=0x7fdbc814e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 0x7fdbc814e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.867+0000 7fdbcdb9a700 1 -- 172.21.15.102:0/894111610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80094a0 con 0x7fdbc80ca440 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.868+0000 7fdbcdb9a700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc80ca440 0x7fdbc814e250 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800ec10 tx=0x7fdbb800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.868+0000 7fdbbeffd700 1 -- 172.21.15.102:0/894111610 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80097a0 con 0x7fdbc80ca440 2023-03-03T05:45:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.868+0000 7fdbbeffd700 1 -- 172.21.15.102:0/894111610 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb800c850 con 0x7fdbc80ca440 2023-03-03T05:45:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.868+0000 7fdbbeffd700 1 -- 172.21.15.102:0/894111610 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb800cbe0 con 0x7fdbc80ca440 2023-03-03T05:45:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.868+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc4009550 con 0x7fdbc80ca440 2023-03-03T05:45:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.868+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc80b2210 con 0x7fdbc80ca440 2023-03-03T05:45:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.869+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc80b1f30 con 0x7fdbc80ca440 2023-03-03T05:45:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.871+0000 7fdbbeffd700 1 -- 172.21.15.102:0/894111610 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdbb8019030 con 0x7fdbc80ca440 2023-03-03T05:45:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.872+0000 7fdbbeffd700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbac0682f0 0x7fdbac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.872+0000 7fdbbeffd700 1 -- 172.21.15.102:0/894111610 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdbb8085550 con 0x7fdbc80ca440 2023-03-03T05:45:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.872+0000 7fdbce39b700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbac0682f0 0x7fdbac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.873+0000 7fdbce39b700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbac0682f0 0x7fdbac06a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4014040 tx=0x7fdbc4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:30.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:30.876+0000 7fdbbeffd700 1 -- 172.21.15.102:0/894111610 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdbb80520a0 con 0x7fdbc80ca440 2023-03-03T05:45:31.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.113+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbc8156880 con 0x7fdbc80ca440 2023-03-03T05:45:31.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.114+0000 7fdbbeffd700 1 -- 172.21.15.102:0/894111610 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5054 (secure 0 0 0) 0x7fdbb8021020 con 0x7fdbc80ca440 2023-03-03T05:45:31.116 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:31.117 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4315,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1262584332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":1262584332},{"type":"v1","addr":"172.21.15.102:6837","nonce":1262584332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-03-03T05:45:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbac0682f0 msgr2=0x7fdbac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbac0682f0 0x7fdbac06a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4014040 tx=0x7fdbc4008030 comp rx=0 tx=0).stop 2023-03-03T05:45:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc80ca440 msgr2=0x7fdbc814e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc80ca440 0x7fdbc814e250 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800ec10 tx=0x7fdbb800ef20 comp rx=0 tx=0).stop 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 shutdown_connections 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbac0682f0 0x7fdbac06a7a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbc80c9610 0x7fdbc80c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbc81564b0 0x7fdbc814e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 --2- 172.21.15.102:0/894111610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc80ca440 0x7fdbc814e250 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 >> 172.21.15.102:0/894111610 conn(0x7fdbc8065fb0 msgr2=0x7fdbc8151e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 shutdown_connections 2023-03-03T05:45:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.116+0000 7fdbcf39d700 1 -- 172.21.15.102:0/894111610 wait complete. 2023-03-03T05:45:31.121 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 3 2023-03-03T05:45:31.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:31.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.388+0000 7fcee91ff700 1 Processor -- start 2023-03-03T05:45:31.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.388+0000 7fcee91ff700 1 -- start start 2023-03-03T05:45:31.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.389+0000 7fcee91ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee40ced80 0x7fcee40cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.389+0000 7fcee91ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcee41597f0 0x7fcee40cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.389+0000 7fcee91ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40cb9f0 con 0x7fcee40cf9a0 2023-03-03T05:45:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.389+0000 7fcee91ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee40ca750 con 0x7fcee41597f0 2023-03-03T05:45:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee91ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee415a230 con 0x7fcee40ced80 2023-03-03T05:45:31.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee40ced80 0x7fcee40cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcee41597f0 0x7fcee40cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee40ced80 0x7fcee40cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60808/0 (socket says 172.21.15.102:60808) 2023-03-03T05:45:31.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee2d9d700 1 -- 172.21.15.102:0/2772303711 learned_addr learned my addr 172.21.15.102:0/2772303711 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:31.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee359e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcee40cf9a0 0x7fcee415c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57854/0 (socket says 172.21.15.102:57854) 2023-03-03T05:45:31.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee2d9d700 1 -- 172.21.15.102:0/2772303711 >> v2:172.21.15.172:3300/0 conn(0x7fcee41597f0 msgr2=0x7fcee40cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:31.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee2d9d700 1 --2- 172.21.15.102:0/2772303711 >> v2:172.21.15.172:3300/0 conn(0x7fcee41597f0 0x7fcee40cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee2d9d700 1 -- 172.21.15.102:0/2772303711 >> v1:172.21.15.172:6789/0 conn(0x7fcee40cf9a0 legacy=0x7fcee415c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee2d9d700 1 -- 172.21.15.102:0/2772303711 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee4159c80 con 0x7fcee40ced80 2023-03-03T05:45:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.390+0000 7fcee259c700 1 --2- 172.21.15.102:0/2772303711 >> v2:172.21.15.172:3300/0 conn(0x7fcee41597f0 0x7fcee40cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.391+0000 7fcee2d9d700 1 --2- 172.21.15.102:0/2772303711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee40ced80 0x7fcee40cf150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fced40098f0 tx=0x7fced4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3837ce0fce947d97 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.391+0000 7fcee1d9b700 1 -- 172.21.15.102:0/2772303711 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4011830 con 0x7fcee40ced80 2023-03-03T05:45:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.391+0000 7fcee1d9b700 1 -- 172.21.15.102:0/2772303711 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced4011990 con 0x7fcee40ced80 2023-03-03T05:45:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.391+0000 7fcee1d9b700 1 -- 172.21.15.102:0/2772303711 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4011d20 con 0x7fcee40ced80 2023-03-03T05:45:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.391+0000 7fcee91ff700 1 -- 172.21.15.102:0/2772303711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee40ced80 msgr2=0x7fcee40cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.391+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2772303711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee40ced80 0x7fcee40cf150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fced40098f0 tx=0x7fced4009c00 comp rx=0 tx=0).stop 2023-03-03T05:45:31.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.392+0000 7fcee91ff700 1 -- 172.21.15.102:0/2772303711 shutdown_connections 2023-03-03T05:45:31.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.392+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2772303711 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee40ced80 0x7fcee40cf150 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.392+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2772303711 >> v2:172.21.15.172:3300/0 conn(0x7fcee41597f0 0x7fcee40cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.392+0000 7fcee91ff700 1 -- 172.21.15.102:0/2772303711 >> 172.21.15.102:0/2772303711 conn(0x7fcee4065fb0 msgr2=0x7fcee40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:31.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.392+0000 7fcee91ff700 1 -- 172.21.15.102:0/2772303711 shutdown_connections 2023-03-03T05:45:31.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.392+0000 7fcee91ff700 1 -- 172.21.15.102:0/2772303711 wait complete. 2023-03-03T05:45:31.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.394+0000 7fcee91ff700 1 Processor -- start 2023-03-03T05:45:31.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.394+0000 7fcee91ff700 1 -- start start 2023-03-03T05:45:31.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee91ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee40ced80 0x7fcee4155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee91ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee40cf9a0 0x7fcee40b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee91ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee41597f0 0x7fcee41ef990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee91ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4156700 con 0x7fcee40cf9a0 2023-03-03T05:45:31.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee91ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee41543f0 con 0x7fcee40ced80 2023-03-03T05:45:31.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee91ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee41546d0 con 0x7fcee41597f0 2023-03-03T05:45:31.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee40cf9a0 0x7fcee40b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.395+0000 7fcee2d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee40ced80 0x7fcee4155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee40cf9a0 0x7fcee40b3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43640/0 (socket says 172.21.15.102:43640) 2023-03-03T05:45:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 -- 172.21.15.102:0/2612602978 learned_addr learned my addr 172.21.15.102:0/2612602978 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee2d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee40ced80 0x7fcee4155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50516/0 (socket says 172.21.15.102:50516) 2023-03-03T05:45:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee359e700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee41597f0 0x7fcee41ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 -- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee41597f0 msgr2=0x7fcee41ef990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee41597f0 0x7fcee41ef990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 -- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee40ced80 msgr2=0x7fcee4155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee40ced80 0x7fcee4155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 -- 172.21.15.102:0/2612602978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80094a0 con 0x7fcee40cf9a0 2023-03-03T05:45:31.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee2d9d700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee40ced80 0x7fcee4155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:31.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.396+0000 7fcee259c700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee40cf9a0 0x7fcee40b3a20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fced800ebe0 tx=0x7fced800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:31.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.397+0000 7fcecb7fe700 1 -- 172.21.15.102:0/2612602978 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8009760 con 0x7fcee40cf9a0 2023-03-03T05:45:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.397+0000 7fcecb7fe700 1 -- 172.21.15.102:0/2612602978 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced800c850 con 0x7fcee40cf9a0 2023-03-03T05:45:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.397+0000 7fcecb7fe700 1 -- 172.21.15.102:0/2612602978 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800cbe0 con 0x7fcee40cf9a0 2023-03-03T05:45:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.397+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced4009550 con 0x7fcee40cf9a0 2023-03-03T05:45:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.397+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee40cb0c0 con 0x7fcee40cf9a0 2023-03-03T05:45:31.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.398+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee40cade0 con 0x7fcee40cf9a0 2023-03-03T05:45:31.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.400+0000 7fcecb7fe700 1 -- 172.21.15.102:0/2612602978 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fced8019030 con 0x7fcee40cf9a0 2023-03-03T05:45:31.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.401+0000 7fcecb7fe700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fced00682f0 0x7fced006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.401+0000 7fcecb7fe700 1 -- 172.21.15.102:0/2612602978 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fced8085920 con 0x7fcee40cf9a0 2023-03-03T05:45:31.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.401+0000 7fcee2d9d700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fced00682f0 0x7fced006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.402+0000 7fcee2d9d700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fced00682f0 0x7fced006a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fced4000c00 tx=0x7fced4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:31.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.405+0000 7fcecb7fe700 1 -- 172.21.15.102:0/2612602978 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fced8052350 con 0x7fcee40cf9a0 2023-03-03T05:45:31.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.599+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcee40cb3a0 con 0x7fcee40cf9a0 2023-03-03T05:45:31.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.600+0000 7fcecb7fe700 1 -- 172.21.15.102:0/2612602978 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14338 (secure 0 0 0) 0x7fced8017090 con 0x7fcee40cf9a0 2023-03-03T05:45:31.603 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:31.603 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":19,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:30.053615+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:31.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fced00682f0 msgr2=0x7fced006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fced00682f0 0x7fced006a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fced4000c00 tx=0x7fced4008030 comp rx=0 tx=0).stop 2023-03-03T05:45:31.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee40cf9a0 msgr2=0x7fcee40b3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee40cf9a0 0x7fcee40b3a20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fced800ebe0 tx=0x7fced800eef0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 shutdown_connections 2023-03-03T05:45:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fced00682f0 0x7fced006a7a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee41597f0 0x7fcee41ef990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee40ced80 0x7fcee4155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 --2- 172.21.15.102:0/2612602978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee40cf9a0 0x7fcee40b3a20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 >> 172.21.15.102:0/2612602978 conn(0x7fcee4065fb0 msgr2=0x7fcee40cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 shutdown_connections 2023-03-03T05:45:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.603+0000 7fcee91ff700 1 -- 172.21.15.102:0/2612602978 wait complete. 2023-03-03T05:45:31.633 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T05:45:20.032740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T05:45:27.930463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T05:45:31.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T05:45:31.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.877+0000 7ff32b59e700 1 Processor -- start 2023-03-03T05:45:31.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.878+0000 7ff32b59e700 1 -- start start 2023-03-03T05:45:31.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32b59e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff32c0c4390 0x7ff32c0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32b59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c159140 con 0x7ff32c153180 2023-03-03T05:45:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32a59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff32c153180 0x7ff32c0c9640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39704/0 (socket says 172.21.15.102:39704) 2023-03-03T05:45:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32a59c700 1 -- 172.21.15.102:0/2276613096 learned_addr learned my addr 172.21.15.102:0/2276613096 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32b59e700 1 -- 172.21.15.102:0/2276613096 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c159420 con 0x7ff32c158cb0 2023-03-03T05:45:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32b59e700 1 -- 172.21.15.102:0/2276613096 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32c159cc0 con 0x7ff32c0c4390 2023-03-03T05:45:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32ad9d700 1 --2- 172.21.15.102:0/2276613096 >> v2:172.21.15.102:3300/0 conn(0x7ff32c0c4390 0x7ff32c0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.880+0000 7ff32959a700 1 -- 172.21.15.102:0/2276613096 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426477141 0 0) 0x7ff32c159140 con 0x7ff32c153180 2023-03-03T05:45:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32959a700 1 -- 172.21.15.102:0/2276613096 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3140035b0 con 0x7ff32c153180 2023-03-03T05:45:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32959a700 1 -- 172.21.15.102:0/2276613096 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264279771 0 0) 0x7ff32c159420 con 0x7ff32c158cb0 2023-03-03T05:45:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32ad9d700 1 -- 172.21.15.102:0/2276613096 >> v1:172.21.15.172:6789/0 conn(0x7ff32c158cb0 legacy=0x7ff32c15c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32ad9d700 1 -- 172.21.15.102:0/2276613096 >> v1:172.21.15.102:6789/0 conn(0x7ff32c153180 legacy=0x7ff32c0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32ad9d700 1 -- 172.21.15.102:0/2276613096 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c159700 con 0x7ff32c0c4390 2023-03-03T05:45:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32ad9d700 1 --2- 172.21.15.102:0/2276613096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c0c4390 0x7ff32c0c47d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff324009d90 tx=0x7ff32400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f37668a3f7d99be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32959a700 1 -- 172.21.15.102:0/2276613096 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32400e070 con 0x7ff32c0c4390 2023-03-03T05:45:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32959a700 1 -- 172.21.15.102:0/2276613096 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32400bb20 con 0x7ff32c0c4390 2023-03-03T05:45:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.881+0000 7ff32959a700 1 -- 172.21.15.102:0/2276613096 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32400bed0 con 0x7ff32c0c4390 2023-03-03T05:45:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.882+0000 7ff32b59e700 1 -- 172.21.15.102:0/2276613096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c0c4390 msgr2=0x7ff32c0c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.882+0000 7ff32b59e700 1 --2- 172.21.15.102:0/2276613096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c0c4390 0x7ff32c0c47d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff324009d90 tx=0x7ff32400b330 comp rx=0 tx=0).stop 2023-03-03T05:45:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.882+0000 7ff32b59e700 1 -- 172.21.15.102:0/2276613096 shutdown_connections 2023-03-03T05:45:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.882+0000 7ff32b59e700 1 --2- 172.21.15.102:0/2276613096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c0c4390 0x7ff32c0c47d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.882+0000 7ff32b59e700 1 -- 172.21.15.102:0/2276613096 >> 172.21.15.102:0/2276613096 conn(0x7ff32c071a30 msgr2=0x7ff32c065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.882+0000 7ff32b59e700 1 -- 172.21.15.102:0/2276613096 shutdown_connections 2023-03-03T05:45:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.882+0000 7ff32b59e700 1 -- 172.21.15.102:0/2276613096 wait complete. 2023-03-03T05:45:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.884+0000 7ff32b59e700 1 Processor -- start 2023-03-03T05:45:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.884+0000 7ff32b59e700 1 -- start start 2023-03-03T05:45:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.885+0000 7ff32b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff32c0c4390 0x7ff32c15b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.885+0000 7ff32b59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c153180 0x7ff32c154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.885+0000 7ff32b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff32c158cb0 0x7ff32c14e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.885+0000 7ff32b59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c0cd3a0 con 0x7ff32c153180 2023-03-03T05:45:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.885+0000 7ff32b59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c1545a0 con 0x7ff32c158cb0 2023-03-03T05:45:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.885+0000 7ff32b59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c153d40 con 0x7ff32c0c4390 2023-03-03T05:45:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff32c0c4390 0x7ff32c15b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff329d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c153180 0x7ff32c154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff32c0c4390 0x7ff32c15b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60812/0 (socket says 172.21.15.102:60812) 2023-03-03T05:45:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff329d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c153180 0x7ff32c154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43658/0 (socket says 172.21.15.102:43658) 2023-03-03T05:45:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 -- 172.21.15.102:0/725930233 learned_addr learned my addr 172.21.15.102:0/725930233 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32ad9d700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff32c158cb0 0x7ff32c14e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 -- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff32c158cb0 msgr2=0x7ff32c14e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff32c158cb0 0x7ff32c14e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 -- 172.21.15.102:0/725930233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c153180 msgr2=0x7ff32c154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:31.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c153180 0x7ff32c154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:31.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 -- 172.21.15.102:0/725930233 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff324009a70 con 0x7ff32c0c4390 2023-03-03T05:45:31.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.886+0000 7ff32a59c700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff32c0c4390 0x7ff32c15b410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00c500 tx=0x7ff31c00c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:31.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.887+0000 7ff31affd700 1 -- 172.21.15.102:0/725930233 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c005d70 con 0x7ff32c0c4390 2023-03-03T05:45:31.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.887+0000 7ff31affd700 1 -- 172.21.15.102:0/725930233 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c0070d0 con 0x7ff32c0c4390 2023-03-03T05:45:31.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.887+0000 7ff31affd700 1 -- 172.21.15.102:0/725930233 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c007440 con 0x7ff32c0c4390 2023-03-03T05:45:31.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.887+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c15c3c0 con 0x7ff32c0c4390 2023-03-03T05:45:31.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.887+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c1599e0 con 0x7ff32c0c4390 2023-03-03T05:45:31.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.888+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c0c5050 con 0x7ff32c0c4390 2023-03-03T05:45:31.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.889+0000 7ff31affd700 1 -- 172.21.15.102:0/725930233 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff31c019030 con 0x7ff32c0c4390 2023-03-03T05:45:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.891+0000 7ff31affd700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff30c0682f0 0x7ff30c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.891+0000 7ff31affd700 1 -- 172.21.15.102:0/725930233 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff31c084b70 con 0x7ff32c0c4390 2023-03-03T05:45:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.891+0000 7ff329d9b700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff30c0682f0 0x7ff30c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.891+0000 7ff329d9b700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff30c0682f0 0x7ff30c06a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff32c15d290 tx=0x7ff320009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:31.895+0000 7ff31affd700 1 -- 172.21.15.102:0/725930233 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff31c04b5e0 con 0x7ff32c0c4390 2023-03-03T05:45:32.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.089+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff32c154880 con 0x7ff32c0c4390 2023-03-03T05:45:32.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.089+0000 7ff31affd700 1 -- 172.21.15.102:0/725930233 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff31c017020 con 0x7ff32c0c4390 2023-03-03T05:45:32.095 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2023-03-03T05:45:32.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff30c0682f0 msgr2=0x7ff30c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff30c0682f0 0x7ff30c06a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff32c15d290 tx=0x7ff320009040 comp rx=0 tx=0).stop 2023-03-03T05:45:32.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff32c0c4390 msgr2=0x7ff32c15b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff32c0c4390 0x7ff32c15b410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00c500 tx=0x7ff31c00c810 comp rx=0 tx=0).stop 2023-03-03T05:45:32.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 shutdown_connections 2023-03-03T05:45:32.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff30c0682f0 0x7ff30c06a7a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff32c0c4390 0x7ff32c15b410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff32c158cb0 0x7ff32c14e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 --2- 172.21.15.102:0/725930233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff32c153180 0x7ff32c154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 >> 172.21.15.102:0/725930233 conn(0x7ff32c071a30 msgr2=0x7ff32c15c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:32.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 shutdown_connections 2023-03-03T05:45:32.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.092+0000 7ff32b59e700 1 -- 172.21.15.102:0/725930233 wait complete. 2023-03-03T05:45:32.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T05:45:32.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.369+0000 7f224e1ff700 1 Processor -- start 2023-03-03T05:45:32.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.369+0000 7f224e1ff700 1 -- start start 2023-03-03T05:45:32.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f224e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224815a380 0x7f22480cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f224e1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480cae70 con 0x7f22480cf9a0 2023-03-03T05:45:32.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f224e1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480cb110 con 0x7f2248153450 2023-03-03T05:45:32.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f224e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22480cb3f0 con 0x7f224815a380 2023-03-03T05:45:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f2247fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f22480cf9a0 0x7f22480ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39716/0 (socket says 172.21.15.102:39716) 2023-03-03T05:45:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f2247fff700 1 -- 172.21.15.102:0/3679033324 learned_addr learned my addr 172.21.15.102:0/3679033324 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f2246ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224815a380 0x7f22480cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.371+0000 7f22477fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2248153450 0x7f2248159940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57870/0 (socket says 172.21.15.102:57870) 2023-03-03T05:45:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f22467fc700 1 -- 172.21.15.102:0/3679033324 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470552112 0 0) 0x7f22480cb110 con 0x7f2248153450 2023-03-03T05:45:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f22467fc700 1 -- 172.21.15.102:0/3679033324 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22300035b0 con 0x7f2248153450 2023-03-03T05:45:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f22467fc700 1 -- 172.21.15.102:0/3679033324 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525498035 0 0) 0x7f22480cae70 con 0x7f22480cf9a0 2023-03-03T05:45:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f2246ffd700 1 -- 172.21.15.102:0/3679033324 >> v1:172.21.15.172:6789/0 conn(0x7f2248153450 legacy=0x7f2248159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f2246ffd700 1 -- 172.21.15.102:0/3679033324 >> v1:172.21.15.102:6789/0 conn(0x7f22480cf9a0 legacy=0x7f22480ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f2246ffd700 1 -- 172.21.15.102:0/3679033324 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22480cb6d0 con 0x7f224815a380 2023-03-03T05:45:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f2246ffd700 1 --2- 172.21.15.102:0/3679033324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224815a380 0x7f22480cc1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f223c009870 tx=0x7f223c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e06e880d6f95b31a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f22467fc700 1 -- 172.21.15.102:0/3679033324 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c011830 con 0x7f224815a380 2023-03-03T05:45:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f22467fc700 1 -- 172.21.15.102:0/3679033324 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c011990 con 0x7f224815a380 2023-03-03T05:45:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.372+0000 7f22467fc700 1 -- 172.21.15.102:0/3679033324 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c011d20 con 0x7f224815a380 2023-03-03T05:45:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.373+0000 7f224e1ff700 1 -- 172.21.15.102:0/3679033324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224815a380 msgr2=0x7f22480cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.373+0000 7f224e1ff700 1 --2- 172.21.15.102:0/3679033324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224815a380 0x7f22480cc1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f223c009870 tx=0x7f223c009b80 comp rx=0 tx=0).stop 2023-03-03T05:45:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.373+0000 7f224e1ff700 1 -- 172.21.15.102:0/3679033324 shutdown_connections 2023-03-03T05:45:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.373+0000 7f224e1ff700 1 --2- 172.21.15.102:0/3679033324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224815a380 0x7f22480cc1b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.373+0000 7f224e1ff700 1 -- 172.21.15.102:0/3679033324 >> 172.21.15.102:0/3679033324 conn(0x7f2248065fb0 msgr2=0x7f22480663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.373+0000 7f224e1ff700 1 -- 172.21.15.102:0/3679033324 shutdown_connections 2023-03-03T05:45:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.373+0000 7f224e1ff700 1 -- 172.21.15.102:0/3679033324 wait complete. 2023-03-03T05:45:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.376+0000 7f224e1ff700 1 Processor -- start 2023-03-03T05:45:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.376+0000 7f224e1ff700 1 -- start start 2023-03-03T05:45:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.377+0000 7f224e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22480cf9a0 0x7f22480cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.377+0000 7f224e1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2248153450 0x7f22481f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.377+0000 7f224e1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224815a380 0x7f2248149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f224e1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22480b2000 con 0x7f2248153450 2023-03-03T05:45:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f224e1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22480c3750 con 0x7f224815a380 2023-03-03T05:45:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f224e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22481f2a90 con 0x7f22480cf9a0 2023-03-03T05:45:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2248153450 0x7f22481f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f22477fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22480cf9a0 0x7f22480cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2248153450 0x7f22481f1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43674/0 (socket says 172.21.15.102:43674) 2023-03-03T05:45:32.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 -- 172.21.15.102:0/1365957283 learned_addr learned my addr 172.21.15.102:0/1365957283 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:32.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2247fff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224815a380 0x7f2248149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 -- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22480cf9a0 msgr2=0x7f22480cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22480cf9a0 0x7f22480cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 -- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224815a380 msgr2=0x7f2248149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224815a380 0x7f2248149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.378+0000 7f2246ffd700 1 -- 172.21.15.102:0/1365957283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c0094a0 con 0x7f2248153450 2023-03-03T05:45:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.379+0000 7f2246ffd700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2248153450 0x7f22481f1fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f223c0097c0 tx=0x7f223c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.379+0000 7f222ffff700 1 -- 172.21.15.102:0/1365957283 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c01b7f0 con 0x7f2248153450 2023-03-03T05:45:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.379+0000 7f222ffff700 1 -- 172.21.15.102:0/1365957283 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c01b970 con 0x7f2248153450 2023-03-03T05:45:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.379+0000 7f222ffff700 1 -- 172.21.15.102:0/1365957283 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c01bd20 con 0x7f2248153450 2023-03-03T05:45:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.379+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22480cbc90 con 0x7f2248153450 2023-03-03T05:45:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.379+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22480cb9b0 con 0x7f2248153450 2023-03-03T05:45:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.380+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22480cf490 con 0x7f2248153450 2023-03-03T05:45:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.381+0000 7f222ffff700 1 -- 172.21.15.102:0/1365957283 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f223c00f050 con 0x7f2248153450 2023-03-03T05:45:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.382+0000 7f222ffff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22340682f0 0x7f223406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.383+0000 7f222ffff700 1 -- 172.21.15.102:0/1365957283 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f223c086920 con 0x7f2248153450 2023-03-03T05:45:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.383+0000 7f22477fe700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22340682f0 0x7f223406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.384+0000 7f22477fe700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22340682f0 0x7f223406a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2238007fd0 tx=0x7f2238007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:32.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.386+0000 7f222ffff700 1 -- 172.21.15.102:0/1365957283 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f223c0533e0 con 0x7f2248153450 2023-03-03T05:45:32.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.575+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f22480677f0 con 0x7f2248153450 2023-03-03T05:45:32.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.576+0000 7f222ffff700 1 -- 172.21.15.102:0/1365957283 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f223c0150a0 con 0x7f2248153450 2023-03-03T05:45:32.579 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2023-03-03T05:45:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.578+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22340682f0 msgr2=0x7f223406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.578+0000 7f224e1ff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22340682f0 0x7f223406a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2238007fd0 tx=0x7f2238007dc0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.578+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2248153450 msgr2=0x7f22481f1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.578+0000 7f224e1ff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2248153450 0x7f22481f1fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f223c0097c0 tx=0x7f223c004750 comp rx=0 tx=0).stop 2023-03-03T05:45:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 shutdown_connections 2023-03-03T05:45:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22340682f0 0x7f223406a7a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f22480cf9a0 0x7f22480cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224815a380 0x7f2248149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 --2- 172.21.15.102:0/1365957283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2248153450 0x7f22481f1fb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 >> 172.21.15.102:0/1365957283 conn(0x7f2248065fb0 msgr2=0x7f22480ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 shutdown_connections 2023-03-03T05:45:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.579+0000 7f224e1ff700 1 -- 172.21.15.102:0/1365957283 wait complete. 2023-03-03T05:45:32.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T05:45:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.853+0000 7feb487ff700 1 Processor -- start 2023-03-03T05:45:32.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.853+0000 7feb487ff700 1 -- start start 2023-03-03T05:45:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb487ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb400ced80 0x7feb400cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb487ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb400cf9a0 0x7feb40163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb487ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400cb9f0 con 0x7feb401597f0 2023-03-03T05:45:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb487ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb400ca750 con 0x7feb400ced80 2023-03-03T05:45:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb487ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4015a230 con 0x7feb400cf9a0 2023-03-03T05:45:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb45d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7feb401597f0 0x7feb400cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57882/0 (socket says 172.21.15.102:57882) 2023-03-03T05:45:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb45d9a700 1 -- 172.21.15.102:0/407170676 learned_addr learned my addr 172.21.15.102:0/407170676 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.854+0000 7feb46d9c700 1 --2- 172.21.15.102:0/407170676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb400cf9a0 0x7feb40163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb4659b700 1 --2- 172.21.15.102:0/407170676 >> v2:172.21.15.102:3300/0 conn(0x7feb400ced80 0x7feb400cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb45599700 1 -- 172.21.15.102:0/407170676 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002202336 0 0) 0x7feb400cb9f0 con 0x7feb401597f0 2023-03-03T05:45:32.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb45599700 1 -- 172.21.15.102:0/407170676 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2c0035b0 con 0x7feb401597f0 2023-03-03T05:45:32.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb46d9c700 1 -- 172.21.15.102:0/407170676 >> v2:172.21.15.102:3300/0 conn(0x7feb400ced80 msgr2=0x7feb400cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb46d9c700 1 --2- 172.21.15.102:0/407170676 >> v2:172.21.15.102:3300/0 conn(0x7feb400ced80 0x7feb400cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb46d9c700 1 -- 172.21.15.102:0/407170676 >> v1:172.21.15.172:6789/0 conn(0x7feb401597f0 legacy=0x7feb400cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb46d9c700 1 -- 172.21.15.102:0/407170676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40159c80 con 0x7feb400cf9a0 2023-03-03T05:45:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.855+0000 7feb46d9c700 1 --2- 172.21.15.102:0/407170676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb400cf9a0 0x7feb40163660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb38009d90 tx=0x7feb3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e7877e1142b7d01 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb45599700 1 -- 172.21.15.102:0/407170676 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800e070 con 0x7feb400cf9a0 2023-03-03T05:45:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb45599700 1 -- 172.21.15.102:0/407170676 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3800bb20 con 0x7feb400cf9a0 2023-03-03T05:45:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb45599700 1 -- 172.21.15.102:0/407170676 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb38012430 con 0x7feb400cf9a0 2023-03-03T05:45:32.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 -- 172.21.15.102:0/407170676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb400cf9a0 msgr2=0x7feb40163660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 --2- 172.21.15.102:0/407170676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb400cf9a0 0x7feb40163660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb38009d90 tx=0x7feb3800b330 comp rx=0 tx=0).stop 2023-03-03T05:45:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 -- 172.21.15.102:0/407170676 shutdown_connections 2023-03-03T05:45:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 --2- 172.21.15.102:0/407170676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb400cf9a0 0x7feb40163660 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 --2- 172.21.15.102:0/407170676 >> v2:172.21.15.102:3300/0 conn(0x7feb400ced80 0x7feb400cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 -- 172.21.15.102:0/407170676 >> 172.21.15.102:0/407170676 conn(0x7feb40065fb0 msgr2=0x7feb400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 -- 172.21.15.102:0/407170676 shutdown_connections 2023-03-03T05:45:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.856+0000 7feb487ff700 1 -- 172.21.15.102:0/407170676 wait complete. 2023-03-03T05:45:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.858+0000 7feb487ff700 1 Processor -- start 2023-03-03T05:45:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.858+0000 7feb487ff700 1 -- start start 2023-03-03T05:45:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.859+0000 7feb487ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ced80 0x7feb40148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.859+0000 7feb487ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb400cf9a0 0x7feb4014adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.859+0000 7feb487ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb401597f0 0x7feb400b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.859+0000 7feb487ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb40149cf0 con 0x7feb400ced80 2023-03-03T05:45:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.859+0000 7feb487ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb401494c0 con 0x7feb400cf9a0 2023-03-03T05:45:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.859+0000 7feb487ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb400af940 con 0x7feb401597f0 2023-03-03T05:45:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ced80 0x7feb40148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ced80 0x7feb40148e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43696/0 (socket says 172.21.15.102:43696) 2023-03-03T05:45:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 -- 172.21.15.102:0/534454935 learned_addr learned my addr 172.21.15.102:0/534454935 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:32.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb46d9c700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb401597f0 0x7feb400b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb45d9a700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb400cf9a0 0x7feb4014adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 -- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb401597f0 msgr2=0x7feb400b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb401597f0 0x7feb400b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 -- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb400cf9a0 msgr2=0x7feb4014adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:32.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb400cf9a0 0x7feb4014adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 -- 172.21.15.102:0/534454935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38009a70 con 0x7feb400ced80 2023-03-03T05:45:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.860+0000 7feb4659b700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ced80 0x7feb40148e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00e870 tx=0x7feb3c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.861+0000 7feb36ffd700 1 -- 172.21.15.102:0/534454935 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c009590 con 0x7feb400ced80 2023-03-03T05:45:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.861+0000 7feb36ffd700 1 -- 172.21.15.102:0/534454935 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c013680 con 0x7feb400ced80 2023-03-03T05:45:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.861+0000 7feb36ffd700 1 -- 172.21.15.102:0/534454935 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0139f0 con 0x7feb400ced80 2023-03-03T05:45:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.861+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb400cb0c0 con 0x7feb400ced80 2023-03-03T05:45:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.861+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb400cade0 con 0x7feb400ced80 2023-03-03T05:45:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.862+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb400cb3a0 con 0x7feb400ced80 2023-03-03T05:45:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.863+0000 7feb36ffd700 1 -- 172.21.15.102:0/534454935 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb3c019030 con 0x7feb400ced80 2023-03-03T05:45:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.864+0000 7feb36ffd700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb240682f0 0x7feb2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.864+0000 7feb36ffd700 1 -- 172.21.15.102:0/534454935 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feb3c0856c0 con 0x7feb400ced80 2023-03-03T05:45:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.865+0000 7feb45d9a700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb240682f0 0x7feb2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.865+0000 7feb45d9a700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb240682f0 0x7feb2406a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feb401576b0 tx=0x7feb30007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:32.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:32.868+0000 7feb36ffd700 1 -- 172.21.15.102:0/534454935 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb3c052180 con 0x7feb400ced80 2023-03-03T05:45:33.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.063+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feb400677f0 con 0x7feb400ced80 2023-03-03T05:45:33.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.063+0000 7feb36ffd700 1 -- 172.21.15.102:0/534454935 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7feb3c017090 con 0x7feb400ced80 2023-03-03T05:45:33.067 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:33.067 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2023-03-03T05:45:33.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2023-03-03T05:45:33.067 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2023-03-03T05:45:33.067 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2023-03-03T05:45:33.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2023-03-03T05:45:33.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2023-03-03T05:45:33.068 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 2, 2023-03-03T05:45:33.068 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": "rbd" 2023-03-03T05:45:33.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2023-03-03T05:45:33.068 INFO:teuthology.orchestra.run.smithi102.stdout:] 2023-03-03T05:45:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb240682f0 msgr2=0x7feb2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb240682f0 0x7feb2406a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feb401576b0 tx=0x7feb30007dc0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ced80 msgr2=0x7feb40148e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ced80 0x7feb40148e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00e870 tx=0x7feb3c00eb80 comp rx=0 tx=0).stop 2023-03-03T05:45:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 shutdown_connections 2023-03-03T05:45:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb240682f0 0x7feb2406a7a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb401597f0 0x7feb400b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb400ced80 0x7feb40148e80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 --2- 172.21.15.102:0/534454935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb400cf9a0 0x7feb4014adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 >> 172.21.15.102:0/534454935 conn(0x7feb40065fb0 msgr2=0x7feb4015a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 shutdown_connections 2023-03-03T05:45:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.066+0000 7feb487ff700 1 -- 172.21.15.102:0/534454935 wait complete. 2023-03-03T05:45:33.096 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T05:45:33.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T05:45:33.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.346+0000 7f022359e700 1 Processor -- start 2023-03-03T05:45:33.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.346+0000 7f022359e700 1 -- start start 2023-03-03T05:45:33.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.347+0000 7f022359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f02240c9690 0x7f02240b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:33.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.347+0000 7f022359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240cf030 0x7f02241635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.347+0000 7f022359e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022406c7d0 con 0x7f02240cec60 2023-03-03T05:45:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.347+0000 7f022359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02240c47c0 con 0x7f02240c9690 2023-03-03T05:45:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.347+0000 7f022359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02240b4220 con 0x7f02240cf030 2023-03-03T05:45:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f022259c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f02240cec60 0x7f0224159610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57884/0 (socket says 172.21.15.102:57884) 2023-03-03T05:45:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f0222d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240cf030 0x7f02241635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f022259c700 1 -- 172.21.15.102:0/3751774735 learned_addr learned my addr 172.21.15.102:0/3751774735 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f0221d9b700 1 --2- 172.21.15.102:0/3751774735 >> v2:172.21.15.172:3300/0 conn(0x7f02240c9690 0x7f02240b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f022159a700 1 -- 172.21.15.102:0/3751774735 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132841491 0 0) 0x7f022406c7d0 con 0x7f02240cec60 2023-03-03T05:45:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f0222d9d700 1 -- 172.21.15.102:0/3751774735 >> v2:172.21.15.172:3300/0 conn(0x7f02240c9690 msgr2=0x7f02240b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f0222d9d700 1 --2- 172.21.15.102:0/3751774735 >> v2:172.21.15.172:3300/0 conn(0x7f02240c9690 0x7f02240b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f0222d9d700 1 -- 172.21.15.102:0/3751774735 >> v1:172.21.15.172:6789/0 conn(0x7f02240cec60 legacy=0x7f0224159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f0222d9d700 1 -- 172.21.15.102:0/3751774735 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022406c210 con 0x7f02240cf030 2023-03-03T05:45:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.348+0000 7f0222d9d700 1 --2- 172.21.15.102:0/3751774735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240cf030 0x7f02241635f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f021c009d90 tx=0x7f021c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=116a04fbc932b85c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022159a700 1 -- 172.21.15.102:0/3751774735 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c00e070 con 0x7f02240cf030 2023-03-03T05:45:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022159a700 1 -- 172.21.15.102:0/3751774735 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c00bb20 con 0x7f02240cf030 2023-03-03T05:45:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022159a700 1 -- 172.21.15.102:0/3751774735 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c012430 con 0x7f02240cf030 2023-03-03T05:45:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 -- 172.21.15.102:0/3751774735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240cf030 msgr2=0x7f02241635f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 --2- 172.21.15.102:0/3751774735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240cf030 0x7f02241635f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f021c009d90 tx=0x7f021c00b330 comp rx=0 tx=0).stop 2023-03-03T05:45:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 -- 172.21.15.102:0/3751774735 shutdown_connections 2023-03-03T05:45:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 --2- 172.21.15.102:0/3751774735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240cf030 0x7f02241635f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 --2- 172.21.15.102:0/3751774735 >> v2:172.21.15.172:3300/0 conn(0x7f02240c9690 0x7f02240b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 -- 172.21.15.102:0/3751774735 >> 172.21.15.102:0/3751774735 conn(0x7f022408b1c0 msgr2=0x7f0224071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 -- 172.21.15.102:0/3751774735 shutdown_connections 2023-03-03T05:45:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.349+0000 7f022359e700 1 -- 172.21.15.102:0/3751774735 wait complete. 2023-03-03T05:45:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.351+0000 7f022359e700 1 Processor -- start 2023-03-03T05:45:33.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.351+0000 7f022359e700 1 -- start start 2023-03-03T05:45:33.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.352+0000 7f022359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240c9690 0x7f0224151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:33.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.352+0000 7f022359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02240cec60 0x7f022414b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:33.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.352+0000 7f022359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02240cf030 0x7f0224145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:33.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.352+0000 7f022359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022415ac90 con 0x7f02240cec60 2023-03-03T05:45:33.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.352+0000 7f022359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0224157a70 con 0x7f02240cf030 2023-03-03T05:45:33.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.352+0000 7f022359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02240c6010 con 0x7f02240c9690 2023-03-03T05:45:33.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02240cec60 0x7f022414b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:33.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f022259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240c9690 0x7f0224151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:33.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02240cec60 0x7f022414b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43700/0 (socket says 172.21.15.102:43700) 2023-03-03T05:45:33.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 -- 172.21.15.102:0/233604507 learned_addr learned my addr 172.21.15.102:0/233604507 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:33.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0222d9d700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02240cf030 0x7f0224145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:33.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 -- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240c9690 msgr2=0x7f0224151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:33.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240c9690 0x7f0224151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 -- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02240cf030 msgr2=0x7f0224145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:33.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02240cf030 0x7f0224145ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:33.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f0221d9b700 1 -- 172.21.15.102:0/233604507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c009a70 con 0x7f02240cec60 2023-03-03T05:45:33.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.353+0000 7f022259c700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240c9690 0x7f0224151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:33.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.354+0000 7f0221d9b700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02240cec60 0x7f022414b500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f021800e9b0 tx=0x7f021800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:33.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.354+0000 7f0212ffd700 1 -- 172.21.15.102:0/233604507 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218009710 con 0x7f02240cec60 2023-03-03T05:45:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.354+0000 7f0212ffd700 1 -- 172.21.15.102:0/233604507 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021800ee50 con 0x7f02240cec60 2023-03-03T05:45:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.354+0000 7f0212ffd700 1 -- 172.21.15.102:0/233604507 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021800ca50 con 0x7f02240cec60 2023-03-03T05:45:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.354+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0224158e10 con 0x7f02240cec60 2023-03-03T05:45:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.354+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022406c4f0 con 0x7f02240cec60 2023-03-03T05:45:33.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.355+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02241590f0 con 0x7f02240cec60 2023-03-03T05:45:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.356+0000 7f0212ffd700 1 -- 172.21.15.102:0/233604507 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0218019030 con 0x7f02240cec60 2023-03-03T05:45:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.358+0000 7f0212ffd700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0204068320 0x7f020406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.358+0000 7f0212ffd700 1 -- 172.21.15.102:0/233604507 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0218085870 con 0x7f02240cec60 2023-03-03T05:45:33.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.358+0000 7f022259c700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0204068320 0x7f020406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:33.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.359+0000 7f022259c700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0204068320 0x7f020406a7d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0214007fd0 tx=0x7f0214007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:33.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.362+0000 7f0212ffd700 1 -- 172.21.15.102:0/233604507 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0218052330 con 0x7f02240cec60 2023-03-03T05:45:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:33.552+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0224074990 con 0x7f02240cec60 2023-03-03T05:45:34.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.283+0000 7f0212ffd700 1 -- 172.21.15.102:0/233604507 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0218055950 con 0x7f02240cec60 2023-03-03T05:45:34.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.288+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0204068320 msgr2=0x7f020406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:34.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.288+0000 7f022359e700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0204068320 0x7f020406a7d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0214007fd0 tx=0x7f0214007de0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.288+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02240cec60 msgr2=0x7f022414b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:34.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.288+0000 7f022359e700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02240cec60 0x7f022414b500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f021800e9b0 tx=0x7f021800ecc0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 shutdown_connections 2023-03-03T05:45:34.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0204068320 0x7f020406a7d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02240c9690 0x7f0224151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02240cf030 0x7f0224145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 --2- 172.21.15.102:0/233604507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02240cec60 0x7f022414b500 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 >> 172.21.15.102:0/233604507 conn(0x7f022408b1c0 msgr2=0x7f022414d370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:34.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 shutdown_connections 2023-03-03T05:45:34.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.289+0000 7f022359e700 1 -- 172.21.15.102:0/233604507 wait complete. 2023-03-03T05:45:34.293 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2023-03-03T05:45:34.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T05:45:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.572+0000 7fe97f5ff700 1 Processor -- start 2023-03-03T05:45:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.572+0000 7fe97f5ff700 1 -- start start 2023-03-03T05:45:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.573+0000 7fe97f5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780ccc40 0x7fe9780cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97f5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9780cd760 0x7fe9780cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97f5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe978153630 con 0x7fe9780ce240 2023-03-03T05:45:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97f5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9780ca7a0 con 0x7fe9780cd760 2023-03-03T05:45:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97f5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780cd550 con 0x7fe9780ccc40 2023-03-03T05:45:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97cb9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9780cd760 0x7fe9780cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97cb9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9780cd760 0x7fe9780cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43710/0 (socket says 172.21.15.102:43710) 2023-03-03T05:45:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97cb9a700 1 -- 172.21.15.102:0/985821534 learned_addr learned my addr 172.21.15.102:0/985821534 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97d39b700 1 --2- 172.21.15.102:0/985821534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780ccc40 0x7fe9780cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97d39b700 1 -- 172.21.15.102:0/985821534 >> v2:172.21.15.102:3300/0 conn(0x7fe9780cd760 msgr2=0x7fe9780cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97d39b700 1 --2- 172.21.15.102:0/985821534 >> v2:172.21.15.102:3300/0 conn(0x7fe9780cd760 0x7fe9780cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97d39b700 1 -- 172.21.15.102:0/985821534 >> v1:172.21.15.172:6789/0 conn(0x7fe9780ce240 legacy=0x7fe9780b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe97d39b700 1 -- 172.21.15.102:0/985821534 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780ce6c0 con 0x7fe9780ccc40 2023-03-03T05:45:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.574+0000 7fe96ffff700 1 -- 172.21.15.102:0/985821534 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333809225 0 0) 0x7fe978153630 con 0x7fe9780ce240 2023-03-03T05:45:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.575+0000 7fe97d39b700 1 --2- 172.21.15.102:0/985821534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780ccc40 0x7fe9780cd010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe974009840 tx=0x7fe974009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f8a92f6274417c4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.575+0000 7fe96ffff700 1 -- 172.21.15.102:0/985821534 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974011830 con 0x7fe9780ccc40 2023-03-03T05:45:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.575+0000 7fe96ffff700 1 -- 172.21.15.102:0/985821534 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe974011990 con 0x7fe9780ccc40 2023-03-03T05:45:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.575+0000 7fe96ffff700 1 -- 172.21.15.102:0/985821534 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974011d20 con 0x7fe9780ccc40 2023-03-03T05:45:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 -- 172.21.15.102:0/985821534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780ccc40 msgr2=0x7fe9780cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/985821534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780ccc40 0x7fe9780cd010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe974009840 tx=0x7fe974009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 -- 172.21.15.102:0/985821534 shutdown_connections 2023-03-03T05:45:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/985821534 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780ccc40 0x7fe9780cd010 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/985821534 >> v2:172.21.15.102:3300/0 conn(0x7fe9780cd760 0x7fe9780cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 -- 172.21.15.102:0/985821534 >> 172.21.15.102:0/985821534 conn(0x7fe978065fb0 msgr2=0x7fe9780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:34.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 -- 172.21.15.102:0/985821534 shutdown_connections 2023-03-03T05:45:34.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.576+0000 7fe97f5ff700 1 -- 172.21.15.102:0/985821534 wait complete. 2023-03-03T05:45:34.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.578+0000 7fe97f5ff700 1 Processor -- start 2023-03-03T05:45:34.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.578+0000 7fe97f5ff700 1 -- start start 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97f5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9780ccc40 0x7fe97815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97f5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780cd760 0x7fe97815b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97f5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9780ce240 0x7fe9780b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97f5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9781f2740 con 0x7fe9780ccc40 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97f5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9781f1b30 con 0x7fe9780ce240 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97f5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9781f1e10 con 0x7fe9780cd760 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97d39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9780ccc40 0x7fe97815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97d39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9780ccc40 0x7fe97815bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43722/0 (socket says 172.21.15.102:43722) 2023-03-03T05:45:34.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97d39b700 1 -- 172.21.15.102:0/1119346203 learned_addr learned my addr 172.21.15.102:0/1119346203 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:34.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97cb9a700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780cd760 0x7fe97815b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:34.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.579+0000 7fe97db9c700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9780ce240 0x7fe9780b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:34.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97d39b700 1 -- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780cd760 msgr2=0x7fe97815b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:34.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97d39b700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780cd760 0x7fe97815b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97d39b700 1 -- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9780ce240 msgr2=0x7fe9780b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:34.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97d39b700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9780ce240 0x7fe9780b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:34.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97d39b700 1 -- 172.21.15.102:0/1119346203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9740094a0 con 0x7fe9780ccc40 2023-03-03T05:45:34.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97d39b700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9780ccc40 0x7fe97815bd80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe974000c00 tx=0x7fe974004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:34.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe96dffb700 1 -- 172.21.15.102:0/1119346203 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9740036a0 con 0x7fe9780ccc40 2023-03-03T05:45:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe96dffb700 1 -- 172.21.15.102:0/1119346203 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97401f070 con 0x7fe9780ccc40 2023-03-03T05:45:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe96dffb700 1 -- 172.21.15.102:0/1119346203 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974003a20 con 0x7fe9780ccc40 2023-03-03T05:45:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9781591d0 con 0x7fe9780ccc40 2023-03-03T05:45:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.580+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97815d1d0 con 0x7fe9780ccc40 2023-03-03T05:45:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.582+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9781594b0 con 0x7fe9780ccc40 2023-03-03T05:45:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.583+0000 7fe96dffb700 1 -- 172.21.15.102:0/1119346203 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe97400f050 con 0x7fe9780ccc40 2023-03-03T05:45:34.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.584+0000 7fe96dffb700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe95c0682f0 0x7fe95c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:34.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.584+0000 7fe96dffb700 1 -- 172.21.15.102:0/1119346203 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fe974021070 con 0x7fe9780ccc40 2023-03-03T05:45:34.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.585+0000 7fe97cb9a700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe95c0682f0 0x7fe95c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:34.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.585+0000 7fe97cb9a700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe95c0682f0 0x7fe95c06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe97814c3b0 tx=0x7fe9680092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:34.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.589+0000 7fe96dffb700 1 -- 172.21.15.102:0/1119346203 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe974053cb0 con 0x7fe9780ccc40 2023-03-03T05:45:34.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:34.782+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe9780b1c70 con 0x7fe9780ccc40 2023-03-03T05:45:35.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.287+0000 7fe96dffb700 1 -- 172.21.15.102:0/1119346203 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe97402b0d0 con 0x7fe9780ccc40 2023-03-03T05:45:35.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.295+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe95c0682f0 msgr2=0x7fe95c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.295+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe95c0682f0 0x7fe95c06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe97814c3b0 tx=0x7fe9680092c0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.295+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9780ccc40 msgr2=0x7fe97815bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.295+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9780ccc40 0x7fe97815bd80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe974000c00 tx=0x7fe974004750 comp rx=0 tx=0).stop 2023-03-03T05:45:35.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.299+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 shutdown_connections 2023-03-03T05:45:35.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.299+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9780ce240 0x7fe9780b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.299+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9780ccc40 0x7fe97815bd80 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe974000c00 tx=0x7fe974004750 comp rx=0 tx=0).stop 2023-03-03T05:45:35.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.299+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe95c0682f0 0x7fe95c06a7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.299+0000 7fe97f5ff700 1 --2- 172.21.15.102:0/1119346203 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9780cd760 0x7fe97815b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.299+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 >> 172.21.15.102:0/1119346203 conn(0x7fe978065fb0 msgr2=0x7fe97806c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:35.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.306+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 shutdown_connections 2023-03-03T05:45:35.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.306+0000 7fe97f5ff700 1 -- 172.21.15.102:0/1119346203 wait complete. 2023-03-03T05:45:35.313 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2023-03-03T05:45:35.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T05:45:35.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.585+0000 7f611bdff700 1 Processor -- start 2023-03-03T05:45:35.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.585+0000 7f611bdff700 1 -- start start 2023-03-03T05:45:35.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.586+0000 7f611bdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6114153450 0x7f61140cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:35.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.586+0000 7f611bdff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f611415a380 0x7f6114159940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:35.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.586+0000 7f611bdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6114159e80 con 0x7f61140cf9a0 2023-03-03T05:45:35.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.586+0000 7f611bdff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61140cae70 con 0x7f611415a380 2023-03-03T05:45:35.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.586+0000 7f611bdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61140cb110 con 0x7f6114153450 2023-03-03T05:45:35.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f6119b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6114153450 0x7f61140cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:35.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f611939a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f611415a380 0x7f6114159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:35.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f6119b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6114153450 0x7f61140cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60912/0 (socket says 172.21.15.102:60912) 2023-03-03T05:45:35.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f6119b9b700 1 -- 172.21.15.102:0/3454679912 learned_addr learned my addr 172.21.15.102:0/3454679912 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:35.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f611a39c700 1 --1- 172.21.15.102:0/3454679912 >> v1:172.21.15.102:6789/0 conn(0x7f61140cf9a0 0x7f61140ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39724/0 (socket says 172.21.15.102:39724) 2023-03-03T05:45:35.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f6119b9b700 1 -- 172.21.15.102:0/3454679912 >> v2:172.21.15.172:3300/0 conn(0x7f611415a380 msgr2=0x7f6114159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:35.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f6119b9b700 1 --2- 172.21.15.102:0/3454679912 >> v2:172.21.15.172:3300/0 conn(0x7f611415a380 0x7f6114159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f6119b9b700 1 -- 172.21.15.102:0/3454679912 >> v1:172.21.15.102:6789/0 conn(0x7f61140cf9a0 legacy=0x7f61140ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.587+0000 7f6119b9b700 1 -- 172.21.15.102:0/3454679912 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61140cb3f0 con 0x7f6114153450 2023-03-03T05:45:35.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f6119b9b700 1 --2- 172.21.15.102:0/3454679912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6114153450 0x7f61140cc1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61100098f0 tx=0x7f6110009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c59ff2d3f9a7793 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:35.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f6118b99700 1 -- 172.21.15.102:0/3454679912 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110011830 con 0x7f6114153450 2023-03-03T05:45:35.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f6118b99700 1 -- 172.21.15.102:0/3454679912 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6110011990 con 0x7f6114153450 2023-03-03T05:45:35.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f6118b99700 1 -- 172.21.15.102:0/3454679912 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110011d20 con 0x7f6114153450 2023-03-03T05:45:35.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f611bdff700 1 -- 172.21.15.102:0/3454679912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6114153450 msgr2=0x7f61140cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f611bdff700 1 --2- 172.21.15.102:0/3454679912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6114153450 0x7f61140cc1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61100098f0 tx=0x7f6110009c00 comp rx=0 tx=0).stop 2023-03-03T05:45:35.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f611bdff700 1 -- 172.21.15.102:0/3454679912 shutdown_connections 2023-03-03T05:45:35.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f611bdff700 1 --2- 172.21.15.102:0/3454679912 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6114153450 0x7f61140cc1b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f611bdff700 1 --2- 172.21.15.102:0/3454679912 >> v2:172.21.15.172:3300/0 conn(0x7f611415a380 0x7f6114159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f611bdff700 1 -- 172.21.15.102:0/3454679912 >> 172.21.15.102:0/3454679912 conn(0x7f6114065fb0 msgr2=0x7f61140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:35.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.588+0000 7f611bdff700 1 -- 172.21.15.102:0/3454679912 shutdown_connections 2023-03-03T05:45:35.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.589+0000 7f611bdff700 1 -- 172.21.15.102:0/3454679912 wait complete. 2023-03-03T05:45:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.590+0000 7f611bdff700 1 Processor -- start 2023-03-03T05:45:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.591+0000 7f611bdff700 1 -- start start 2023-03-03T05:45:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.591+0000 7f611bdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61140cf9a0 0x7f61140b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611bdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6114153450 0x7f61141f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611bdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f611415a380 0x7f6114150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611bdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61141f2a50 con 0x7f6114153450 2023-03-03T05:45:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611bdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6114155b80 con 0x7f61140cf9a0 2023-03-03T05:45:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611bdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61140b2160 con 0x7f611415a380 2023-03-03T05:45:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6114153450 0x7f61141f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6114153450 0x7f61141f1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43732/0 (socket says 172.21.15.102:43732) 2023-03-03T05:45:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611939a700 1 -- 172.21.15.102:0/3025792713 learned_addr learned my addr 172.21.15.102:0/3025792713 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f6119b9b700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61140cf9a0 0x7f61140b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611a39c700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f611415a380 0x7f6114150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f6119b9b700 1 -- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f611415a380 msgr2=0x7f6114150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f6119b9b700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f611415a380 0x7f6114150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f6119b9b700 1 -- 172.21.15.102:0/3025792713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6114153450 msgr2=0x7f61141f1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f6119b9b700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6114153450 0x7f61141f1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f6119b9b700 1 -- 172.21.15.102:0/3025792713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6110009550 con 0x7f61140cf9a0 2023-03-03T05:45:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.592+0000 7f611939a700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6114153450 0x7f61141f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.593+0000 7f6119b9b700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61140cf9a0 0x7f61140b1c20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6110016000 tx=0x7f6110004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.593+0000 7f610a7fc700 1 -- 172.21.15.102:0/3025792713 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110004510 con 0x7f61140cf9a0 2023-03-03T05:45:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.593+0000 7f610a7fc700 1 -- 172.21.15.102:0/3025792713 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611001e070 con 0x7f61140cf9a0 2023-03-03T05:45:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.593+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61140cb9b0 con 0x7f61140cf9a0 2023-03-03T05:45:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.593+0000 7f610a7fc700 1 -- 172.21.15.102:0/3025792713 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61100038c0 con 0x7f61140cf9a0 2023-03-03T05:45:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.593+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61140cb6d0 con 0x7f61140cf9a0 2023-03-03T05:45:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.594+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61140cbc90 con 0x7f61140cf9a0 2023-03-03T05:45:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.596+0000 7f610a7fc700 1 -- 172.21.15.102:0/3025792713 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f611000f050 con 0x7f61140cf9a0 2023-03-03T05:45:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.597+0000 7f610a7fc700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f60f8068320 0x7f60f806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.597+0000 7f610a7fc700 1 -- 172.21.15.102:0/3025792713 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f6110086b10 con 0x7f61140cf9a0 2023-03-03T05:45:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.597+0000 7f611939a700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f60f8068320 0x7f60f806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:35.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.598+0000 7f611939a700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f60f8068320 0x7f60f806a7d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61140c9da0 tx=0x7f6104009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:35.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.601+0000 7f610a7fc700 1 -- 172.21.15.102:0/3025792713 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6110053260 con 0x7f61140cf9a0 2023-03-03T05:45:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.834+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f61140677f0 con 0x7f61140cf9a0 2023-03-03T05:45:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.842+0000 7f610a7fc700 1 -- 172.21.15.102:0/3025792713 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f61100150c0 con 0x7f61140cf9a0 2023-03-03T05:45:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f60f8068320 msgr2=0x7f60f806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f60f8068320 0x7f60f806a7d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61140c9da0 tx=0x7f6104009210 comp rx=0 tx=0).stop 2023-03-03T05:45:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61140cf9a0 msgr2=0x7f61140b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61140cf9a0 0x7f61140b1c20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6110016000 tx=0x7f6110004750 comp rx=0 tx=0).stop 2023-03-03T05:45:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 shutdown_connections 2023-03-03T05:45:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f60f8068320 0x7f60f806a7d0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61140c9da0 tx=0x7f6104009210 comp rx=0 tx=0).stop 2023-03-03T05:45:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f611415a380 0x7f6114150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61140cf9a0 0x7f61140b1c20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 --2- 172.21.15.102:0/3025792713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6114153450 0x7f61141f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.847+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 >> 172.21.15.102:0/3025792713 conn(0x7f6114065fb0 msgr2=0x7f61141524d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.850+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 shutdown_connections 2023-03-03T05:45:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:35.851+0000 7f611bdff700 1 -- 172.21.15.102:0/3025792713 wait complete. 2023-03-03T05:45:35.862 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 3 and data pool 4 2023-03-03T05:45:35.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:36.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.136+0000 7fcfc93ff700 1 Processor -- start 2023-03-03T05:45:36.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.137+0000 7fcfc93ff700 1 -- start start 2023-03-03T05:45:36.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc93ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfc4153290 0x7fcfc4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc93ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcfc41593c0 0x7fcfc40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc93ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc4153660 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc93ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc406c360 con 0x7fcfc4153290 2023-03-03T05:45:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc93ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc415cea0 con 0x7fcfc41593c0 2023-03-03T05:45:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc2ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfc4153290 0x7fcfc4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc2ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfc4153290 0x7fcfc4158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43738/0 (socket says 172.21.15.102:43738) 2023-03-03T05:45:36.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/673647206 learned_addr learned my addr 172.21.15.102:0/673647206 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:36.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.138+0000 7fcfc27fc700 1 --2- 172.21.15.102:0/673647206 >> v2:172.21.15.172:3300/0 conn(0x7fcfc41593c0 0x7fcfc40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/673647206 >> v2:172.21.15.172:3300/0 conn(0x7fcfc41593c0 msgr2=0x7fcfc40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:36.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc2ffd700 1 --2- 172.21.15.102:0/673647206 >> v2:172.21.15.172:3300/0 conn(0x7fcfc41593c0 0x7fcfc40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/673647206 >> v1:172.21.15.102:6789/0 conn(0x7fcfc40cd8e0 legacy=0x7fcfc40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/673647206 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc406c7f0 con 0x7fcfc4153290 2023-03-03T05:45:36.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc1ffb700 1 -- 172.21.15.102:0/673647206 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654679926 0 0) 0x7fcfc4153660 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc2ffd700 1 --2- 172.21.15.102:0/673647206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc4153290 0x7fcfc4158e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4159fb0 tx=0x7fcfb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e214ae406f28d889 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc1ffb700 1 -- 172.21.15.102:0/673647206 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4011830 con 0x7fcfc4153290 2023-03-03T05:45:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.139+0000 7fcfc1ffb700 1 -- 172.21.15.102:0/673647206 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb4011990 con 0x7fcfc4153290 2023-03-03T05:45:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc1ffb700 1 -- 172.21.15.102:0/673647206 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4011d20 con 0x7fcfc4153290 2023-03-03T05:45:36.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 -- 172.21.15.102:0/673647206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc4153290 msgr2=0x7fcfc4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/673647206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc4153290 0x7fcfc4158e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4159fb0 tx=0x7fcfb4009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 -- 172.21.15.102:0/673647206 shutdown_connections 2023-03-03T05:45:36.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/673647206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc4153290 0x7fcfc4158e80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/673647206 >> v2:172.21.15.172:3300/0 conn(0x7fcfc41593c0 0x7fcfc40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 -- 172.21.15.102:0/673647206 >> 172.21.15.102:0/673647206 conn(0x7fcfc4065fb0 msgr2=0x7fcfc40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:36.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 -- 172.21.15.102:0/673647206 shutdown_connections 2023-03-03T05:45:36.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.140+0000 7fcfc93ff700 1 -- 172.21.15.102:0/673647206 wait complete. 2023-03-03T05:45:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.143+0000 7fcfc93ff700 1 Processor -- start 2023-03-03T05:45:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.143+0000 7fcfc93ff700 1 -- start start 2023-03-03T05:45:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc93ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfc40cd8e0 0x7fcfc40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc93ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc4153290 0x7fcfc40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc93ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc41593c0 0x7fcfc40b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc93ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40b4c50 con 0x7fcfc41593c0 2023-03-03T05:45:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc93ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4156140 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc93ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4156740 con 0x7fcfc4153290 2023-03-03T05:45:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfc40cd8e0 0x7fcfc40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfc40cd8e0 0x7fcfc40cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50602/0 (socket says 172.21.15.102:50602) 2023-03-03T05:45:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/3343501928 learned_addr learned my addr 172.21.15.102:0/3343501928 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc27fc700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc4153290 0x7fcfc40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc37fe700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc41593c0 0x7fcfc40b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc4153290 msgr2=0x7fcfc40b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc4153290 0x7fcfc40b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/3343501928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc41593c0 msgr2=0x7fcfc40b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc41593c0 0x7fcfc40b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.145+0000 7fcfc2ffd700 1 -- 172.21.15.102:0/3343501928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb4009550 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.146+0000 7fcfc2ffd700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfc40cd8e0 0x7fcfc40cf0b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009c10 tx=0x7fcfb4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:36.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.146+0000 7fcfab7fe700 1 -- 172.21.15.102:0/3343501928 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4004510 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.146+0000 7fcfab7fe700 1 -- 172.21.15.102:0/3343501928 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb40036a0 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.146+0000 7fcfab7fe700 1 -- 172.21.15.102:0/3343501928 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4003a10 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.146+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc40ccde0 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.146+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc415d160 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.147+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc40cd0c0 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.149+0000 7fcfab7fe700 1 -- 172.21.15.102:0/3343501928 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcfb402d070 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.150+0000 7fcfab7fe700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfb00682f0 0x7fcfb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.150+0000 7fcfab7fe700 1 -- 172.21.15.102:0/3343501928 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcfb4021030 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.150+0000 7fcfc27fc700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfb00682f0 0x7fcfb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.151+0000 7fcfc27fc700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfb00682f0 0x7fcfb006a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4066f70 tx=0x7fcfb8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.154+0000 7fcfab7fe700 1 -- 172.21.15.102:0/3343501928 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcfb4053c50 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.349+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfc40b1e50 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.352+0000 7fcfab7fe700 1 -- 172.21.15.102:0/3343501928 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+18066 (secure 0 0 0) 0x7fcfb4057270 con 0x7fcfc40cd8e0 2023-03-03T05:45:36.354 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:36.355 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:35.837317+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:36.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfb00682f0 msgr2=0x7fcfb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfb00682f0 0x7fcfb006a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4066f70 tx=0x7fcfb8009210 comp rx=0 tx=0).stop 2023-03-03T05:45:36.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfc40cd8e0 msgr2=0x7fcfc40cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfc40cd8e0 0x7fcfc40cf0b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009c10 tx=0x7fcfb4004750 comp rx=0 tx=0).stop 2023-03-03T05:45:36.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 shutdown_connections 2023-03-03T05:45:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfc41593c0 0x7fcfc40b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfc40cd8e0 0x7fcfc40cf0b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfb00682f0 0x7fcfb006a7a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 --2- 172.21.15.102:0/3343501928 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfc4153290 0x7fcfc40b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 >> 172.21.15.102:0/3343501928 conn(0x7fcfc4065fb0 msgr2=0x7fcfc4157700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 shutdown_connections 2023-03-03T05:45:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.354+0000 7fcfc93ff700 1 -- 172.21.15.102:0/3343501928 wait complete. 2023-03-03T05:45:36.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.624+0000 7ff6661ff700 1 Processor -- start 2023-03-03T05:45:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.624+0000 7ff6661ff700 1 -- start start 2023-03-03T05:45:36.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.625+0000 7ff6661ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff6601597f0 0x7ff6600cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.625+0000 7ff6661ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff6600cf9a0 0x7ff660163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff6661ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6600cb9f0 con 0x7ff6600ced80 2023-03-03T05:45:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff6661ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6600ca750 con 0x7ff6601597f0 2023-03-03T05:45:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff6661ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff66015a230 con 0x7ff6600cf9a0 2023-03-03T05:45:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff6601597f0 0x7ff6600cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff6600ced80 0x7ff6600cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45072/0 (socket says 172.21.15.102:45072) 2023-03-03T05:45:36.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65f7fe700 1 -- 172.21.15.102:0/2680914239 learned_addr learned my addr 172.21.15.102:0/2680914239 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:36.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65ffff700 1 --2- 172.21.15.102:0/2680914239 >> v2:172.21.15.172:3300/0 conn(0x7ff6600cf9a0 0x7ff660163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65effd700 1 -- 172.21.15.102:0/2680914239 >> v2:172.21.15.172:3300/0 conn(0x7ff6600cf9a0 msgr2=0x7ff660163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65effd700 1 --2- 172.21.15.102:0/2680914239 >> v2:172.21.15.172:3300/0 conn(0x7ff6600cf9a0 0x7ff660163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65effd700 1 -- 172.21.15.102:0/2680914239 >> v1:172.21.15.102:6789/0 conn(0x7ff6600ced80 legacy=0x7ff6600cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65effd700 1 -- 172.21.15.102:0/2680914239 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff660159c80 con 0x7ff6601597f0 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.626+0000 7ff65e7fc700 1 -- 172.21.15.102:0/2680914239 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307958430 0 0) 0x7ff6600cb9f0 con 0x7ff6600ced80 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.627+0000 7ff65effd700 1 --2- 172.21.15.102:0/2680914239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 0x7ff6600cf150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff66015d3d0 tx=0x7ff654009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da96a76fc73e2715 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.627+0000 7ff65e7fc700 1 -- 172.21.15.102:0/2680914239 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654011830 con 0x7ff6601597f0 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.627+0000 7ff65e7fc700 1 -- 172.21.15.102:0/2680914239 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff654011990 con 0x7ff6601597f0 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.627+0000 7ff65e7fc700 1 -- 172.21.15.102:0/2680914239 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654011d20 con 0x7ff6601597f0 2023-03-03T05:45:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 -- 172.21.15.102:0/2680914239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 msgr2=0x7ff6600cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 --2- 172.21.15.102:0/2680914239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 0x7ff6600cf150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff66015d3d0 tx=0x7ff654009b20 comp rx=0 tx=0).stop 2023-03-03T05:45:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 -- 172.21.15.102:0/2680914239 shutdown_connections 2023-03-03T05:45:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 --2- 172.21.15.102:0/2680914239 >> v2:172.21.15.172:3300/0 conn(0x7ff6600cf9a0 0x7ff660163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 --2- 172.21.15.102:0/2680914239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 0x7ff6600cf150 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 -- 172.21.15.102:0/2680914239 >> 172.21.15.102:0/2680914239 conn(0x7ff660065fb0 msgr2=0x7ff6600663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 -- 172.21.15.102:0/2680914239 shutdown_connections 2023-03-03T05:45:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.628+0000 7ff6661ff700 1 -- 172.21.15.102:0/2680914239 wait complete. 2023-03-03T05:45:36.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.630+0000 7ff6661ff700 1 Processor -- start 2023-03-03T05:45:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.630+0000 7ff6661ff700 1 -- start start 2023-03-03T05:45:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff6661ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6600ced80 0x7ff6600b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff6661ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6600cf9a0 0x7ff66014d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff6661ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 0x7ff66014d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff6661ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6600c81e0 con 0x7ff6601597f0 2023-03-03T05:45:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff6661ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6600c8d40 con 0x7ff6600cf9a0 2023-03-03T05:45:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff6661ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66015ad80 con 0x7ff6600ced80 2023-03-03T05:45:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff65effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6600cf9a0 0x7ff66014d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff65f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6600ced80 0x7ff6600b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff65ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 0x7ff66014d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff65effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6600cf9a0 0x7ff66014d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59132/0 (socket says 172.21.15.102:59132) 2023-03-03T05:45:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.631+0000 7ff65effd700 1 -- 172.21.15.102:0/1676742558 learned_addr learned my addr 172.21.15.102:0/1676742558 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff65effd700 1 -- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6600ced80 msgr2=0x7ff6600b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff65effd700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6600ced80 0x7ff6600b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff65effd700 1 -- 172.21.15.102:0/1676742558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 msgr2=0x7ff66014d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff65effd700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 0x7ff66014d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff65effd700 1 -- 172.21.15.102:0/1676742558 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6540094a0 con 0x7ff6600cf9a0 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff65effd700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6600cf9a0 0x7ff66014d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6540090b0 tx=0x7ff654004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff647fff700 1 -- 172.21.15.102:0/1676742558 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6540036a0 con 0x7ff6600cf9a0 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff647fff700 1 -- 172.21.15.102:0/1676742558 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65401e070 con 0x7ff6600cf9a0 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff647fff700 1 -- 172.21.15.102:0/1676742558 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654003a20 con 0x7ff6600cf9a0 2023-03-03T05:45:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.632+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6600cb0c0 con 0x7ff6600cf9a0 2023-03-03T05:45:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.633+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6600cade0 con 0x7ff6600cf9a0 2023-03-03T05:45:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.634+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6600cb3a0 con 0x7ff6600cf9a0 2023-03-03T05:45:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.635+0000 7ff647fff700 1 -- 172.21.15.102:0/1676742558 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff65400f050 con 0x7ff6600cf9a0 2023-03-03T05:45:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.636+0000 7ff647fff700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.636+0000 7ff647fff700 1 -- 172.21.15.102:0/1676742558 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff654086d10 con 0x7ff6600cf9a0 2023-03-03T05:45:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.636+0000 7ff65f7fe700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.637+0000 7ff65f7fe700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff650000f60 tx=0x7ff650007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.641+0000 7ff647fff700 1 -- 172.21.15.102:0/1676742558 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff65408c020 con 0x7ff6600cf9a0 2023-03-03T05:45:36.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.835+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6600c8e80 con 0x7ff6600cf9a0 2023-03-03T05:45:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.836+0000 7ff647fff700 1 -- 172.21.15.102:0/1676742558 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+18066 (secure 0 0 0) 0x7ff654015090 con 0x7ff6600cf9a0 2023-03-03T05:45:36.839 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:36.840 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:35.837317+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:36.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 msgr2=0x7ff64c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff650000f60 tx=0x7ff650007e70 comp rx=0 tx=0).stop 2023-03-03T05:45:36.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6600cf9a0 msgr2=0x7ff66014d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:36.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6600cf9a0 0x7ff66014d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6540090b0 tx=0x7ff654004750 comp rx=0 tx=0).stop 2023-03-03T05:45:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 shutdown_connections 2023-03-03T05:45:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6600ced80 0x7ff6600b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6601597f0 0x7ff66014d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 --2- 172.21.15.102:0/1676742558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6600cf9a0 0x7ff66014d150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 >> 172.21.15.102:0/1676742558 conn(0x7ff660065fb0 msgr2=0x7ff660066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 shutdown_connections 2023-03-03T05:45:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:36.839+0000 7ff6661ff700 1 -- 172.21.15.102:0/1676742558 wait complete. 2023-03-03T05:45:36.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T05:45:37.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.116+0000 7fb2d03ff700 1 Processor -- start 2023-03-03T05:45:37.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.116+0000 7fb2d03ff700 1 -- start start 2023-03-03T05:45:37.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.117+0000 7fb2d03ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c815a380 0x7fb2c80cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:37.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.117+0000 7fb2d03ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c80cae70 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.117+0000 7fb2d03ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c80cb110 con 0x7fb2c8153450 2023-03-03T05:45:37.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.117+0000 7fb2d03ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c80cb3f0 con 0x7fb2c815a380 2023-03-03T05:45:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2cd99a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c815a380 0x7fb2c80cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2ce99c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb2c80cf9a0 0x7fb2c80ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45078/0 (socket says 172.21.15.102:45078) 2023-03-03T05:45:37.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2ce99c700 1 -- 172.21.15.102:0/2146132572 learned_addr learned my addr 172.21.15.102:0/2146132572 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2cd99a700 1 -- 172.21.15.102:0/2146132572 >> v1:172.21.15.172:6789/0 conn(0x7fb2c8153450 legacy=0x7fb2c8159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2cd99a700 1 -- 172.21.15.102:0/2146132572 >> v1:172.21.15.102:6789/0 conn(0x7fb2c80cf9a0 legacy=0x7fb2c80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:37.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2cd199700 1 -- 172.21.15.102:0/2146132572 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886430885 0 0) 0x7fb2c80cae70 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2cd99a700 1 -- 172.21.15.102:0/2146132572 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c80cb6d0 con 0x7fb2c815a380 2023-03-03T05:45:37.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.118+0000 7fb2cd99a700 1 --2- 172.21.15.102:0/2146132572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c815a380 0x7fb2c80cc1b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097f0 tx=0x7fb2b8009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34482daf0c3e7a83 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:37.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2cd199700 1 -- 172.21.15.102:0/2146132572 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8004230 con 0x7fb2c815a380 2023-03-03T05:45:37.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2cd199700 1 -- 172.21.15.102:0/2146132572 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b8004390 con 0x7fb2c815a380 2023-03-03T05:45:37.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2cd199700 1 -- 172.21.15.102:0/2146132572 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80036a0 con 0x7fb2c815a380 2023-03-03T05:45:37.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2146132572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c815a380 msgr2=0x7fb2c80cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:37.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2146132572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c815a380 0x7fb2c80cc1b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097f0 tx=0x7fb2b8009b00 comp rx=0 tx=0).stop 2023-03-03T05:45:37.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2146132572 shutdown_connections 2023-03-03T05:45:37.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2146132572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c815a380 0x7fb2c80cc1b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:37.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2146132572 >> 172.21.15.102:0/2146132572 conn(0x7fb2c8065fb0 msgr2=0x7fb2c80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:37.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.119+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2146132572 shutdown_connections 2023-03-03T05:45:37.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.120+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2146132572 wait complete. 2023-03-03T05:45:37.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.121+0000 7fb2d03ff700 1 Processor -- start 2023-03-03T05:45:37.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.122+0000 7fb2d03ff700 1 -- start start 2023-03-03T05:45:37.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2d03ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c80cf9a0 0x7fb2c80b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:37.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2d03ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2c8153450 0x7fb2c81f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:37.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2d03ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2c815a380 0x7fb2c80b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:37.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2d03ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c80b1c20 con 0x7fb2c815a380 2023-03-03T05:45:37.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2d03ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c81f2460 con 0x7fb2c8153450 2023-03-03T05:45:37.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2d03ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c81f1e50 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2ce19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c80cf9a0 0x7fb2c80b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2ce19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c80cf9a0 0x7fb2c80b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36654/0 (socket says 172.21.15.102:36654) 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2cd99a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2c8153450 0x7fb2c81f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2ce19b700 1 -- 172.21.15.102:0/2207838036 learned_addr learned my addr 172.21.15.102:0/2207838036 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2ce99c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2c815a380 0x7fb2c80b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2ce19b700 1 -- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2c8153450 msgr2=0x7fb2c81f28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.123+0000 7fb2ce19b700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2c8153450 0x7fb2c81f28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2ce19b700 1 -- 172.21.15.102:0/2207838036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2c815a380 msgr2=0x7fb2c80b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:37.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2ce19b700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2c815a380 0x7fb2c80b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:37.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2ce19b700 1 -- 172.21.15.102:0/2207838036 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b80094a0 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2ce19b700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c80cf9a0 0x7fb2c80b4740 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400c750 tx=0x7fb2c400ca60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:37.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2beffd700 1 -- 172.21.15.102:0/2207838036 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40054e0 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2beffd700 1 -- 172.21.15.102:0/2207838036 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c4006840 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2beffd700 1 -- 172.21.15.102:0/2207838036 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4006bf0 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c80cbc90 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.124+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c80cb9b0 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.125+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c80cf490 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.127+0000 7fb2beffd700 1 -- 172.21.15.102:0/2207838036 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb2c4010070 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.128+0000 7fb2beffd700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2ac0682f0 0x7fb2ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:37.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.128+0000 7fb2beffd700 1 -- 172.21.15.102:0/2207838036 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb2c4085590 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.128+0000 7fb2cd99a700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2ac0682f0 0x7fb2ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:37.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.129+0000 7fb2cd99a700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2ac0682f0 0x7fb2ac06a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097c0 tx=0x7fb2b8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:37.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.133+0000 7fb2beffd700 1 -- 172.21.15.102:0/2207838036 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2c4051c70 con 0x7fb2c80cf9a0 2023-03-03T05:45:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:37.372+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb2c808c7d0 con 0x7fb2c80cf9a0 2023-03-03T05:45:38.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.320+0000 7fb2beffd700 1 -- 172.21.15.102:0/2207838036 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb2c4055290 con 0x7fb2c80cf9a0 2023-03-03T05:45:38.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2ac0682f0 msgr2=0x7fb2ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:38.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2ac0682f0 0x7fb2ac06a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097c0 tx=0x7fb2b8008030 comp rx=0 tx=0).stop 2023-03-03T05:45:38.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c80cf9a0 msgr2=0x7fb2c80b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:38.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c80cf9a0 0x7fb2c80b4740 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400c750 tx=0x7fb2c400ca60 comp rx=0 tx=0).stop 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 shutdown_connections 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2ac0682f0 0x7fb2ac06a7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2c80cf9a0 0x7fb2c80b4740 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2c815a380 0x7fb2c80b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 --2- 172.21.15.102:0/2207838036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2c8153450 0x7fb2c81f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 >> 172.21.15.102:0/2207838036 conn(0x7fb2c8065fb0 msgr2=0x7fb2c80ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 shutdown_connections 2023-03-03T05:45:38.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.323+0000 7fb2d03ff700 1 -- 172.21.15.102:0/2207838036 wait complete. 2023-03-03T05:45:38.354 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2023-03-03T05:45:38.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T05:45:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.598+0000 7fe86e39d700 1 Processor -- start 2023-03-03T05:45:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.599+0000 7fe86e39d700 1 -- start start 2023-03-03T05:45:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.600+0000 7fe86e39d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe8680c4390 0x7fe8680c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.600+0000 7fe86e39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe868159140 con 0x7fe868153180 2023-03-03T05:45:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.600+0000 7fe86e39d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe868159420 con 0x7fe868158cb0 2023-03-03T05:45:38.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.600+0000 7fe86e39d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe868159cc0 con 0x7fe8680c4390 2023-03-03T05:45:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.600+0000 7fe86d39b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe868153180 0x7fe8680c9640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45086/0 (socket says 172.21.15.102:45086) 2023-03-03T05:45:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.600+0000 7fe86d39b700 1 -- 172.21.15.102:0/1373538678 learned_addr learned my addr 172.21.15.102:0/1373538678 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe86db9c700 1 --2- 172.21.15.102:0/1373538678 >> v2:172.21.15.172:3300/0 conn(0x7fe8680c4390 0x7fe8680c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe85ffff700 1 -- 172.21.15.102:0/1373538678 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700160279 0 0) 0x7fe868159140 con 0x7fe868153180 2023-03-03T05:45:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe85ffff700 1 -- 172.21.15.102:0/1373538678 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8540035b0 con 0x7fe868153180 2023-03-03T05:45:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe86db9c700 1 -- 172.21.15.102:0/1373538678 >> v1:172.21.15.172:6789/0 conn(0x7fe868158cb0 legacy=0x7fe86815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe85ffff700 1 -- 172.21.15.102:0/1373538678 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778539575 0 0) 0x7fe868159420 con 0x7fe868158cb0 2023-03-03T05:45:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe86db9c700 1 -- 172.21.15.102:0/1373538678 >> v1:172.21.15.102:6789/0 conn(0x7fe868153180 legacy=0x7fe8680c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe86db9c700 1 -- 172.21.15.102:0/1373538678 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe868159700 con 0x7fe8680c4390 2023-03-03T05:45:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.601+0000 7fe86db9c700 1 --2- 172.21.15.102:0/1373538678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe8680c47d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe860009d90 tx=0x7fe86000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e055a0c72a0a06c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe85ffff700 1 -- 172.21.15.102:0/1373538678 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86000e070 con 0x7fe8680c4390 2023-03-03T05:45:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe85ffff700 1 -- 172.21.15.102:0/1373538678 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86000bb20 con 0x7fe8680c4390 2023-03-03T05:45:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe85ffff700 1 -- 172.21.15.102:0/1373538678 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86000bed0 con 0x7fe8680c4390 2023-03-03T05:45:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe86e39d700 1 -- 172.21.15.102:0/1373538678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 msgr2=0x7fe8680c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1373538678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe8680c47d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe860009d90 tx=0x7fe86000b330 comp rx=0 tx=0).stop 2023-03-03T05:45:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe86e39d700 1 -- 172.21.15.102:0/1373538678 shutdown_connections 2023-03-03T05:45:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1373538678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe8680c47d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe86e39d700 1 -- 172.21.15.102:0/1373538678 >> 172.21.15.102:0/1373538678 conn(0x7fe868071a30 msgr2=0x7fe868065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe86e39d700 1 -- 172.21.15.102:0/1373538678 shutdown_connections 2023-03-03T05:45:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.602+0000 7fe86e39d700 1 -- 172.21.15.102:0/1373538678 wait complete. 2023-03-03T05:45:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.604+0000 7fe86e39d700 1 Processor -- start 2023-03-03T05:45:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.604+0000 7fe86e39d700 1 -- start start 2023-03-03T05:45:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.605+0000 7fe86e39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe86815b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.605+0000 7fe86e39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe868153180 0x7fe868154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.605+0000 7fe86e39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe868158cb0 0x7fe86814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.605+0000 7fe86e39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680cd3a0 con 0x7fe868158cb0 2023-03-03T05:45:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.605+0000 7fe86e39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8681545a0 con 0x7fe8680c4390 2023-03-03T05:45:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.605+0000 7fe86e39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe868153d40 con 0x7fe868153180 2023-03-03T05:45:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86d39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe86815b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86db9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe868158cb0 0x7fe86814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86cb9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe868153180 0x7fe868154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86d39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe86815b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59160/0 (socket says 172.21.15.102:59160) 2023-03-03T05:45:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86cb9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe868153180 0x7fe868154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36662/0 (socket says 172.21.15.102:36662) 2023-03-03T05:45:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86d39b700 1 -- 172.21.15.102:0/1914027652 learned_addr learned my addr 172.21.15.102:0/1914027652 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86db9c700 1 -- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe868153180 msgr2=0x7fe868154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86db9c700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe868153180 0x7fe868154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86db9c700 1 -- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 msgr2=0x7fe86815b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86db9c700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe86815b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.606+0000 7fe86db9c700 1 -- 172.21.15.102:0/1914027652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe860009a70 con 0x7fe868158cb0 2023-03-03T05:45:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.607+0000 7fe86db9c700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe868158cb0 0x7fe86814e430 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe860016000 tx=0x7fe860004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.607+0000 7fe85dffb700 1 -- 172.21.15.102:0/1914027652 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86001c810 con 0x7fe868158cb0 2023-03-03T05:45:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.607+0000 7fe85dffb700 1 -- 172.21.15.102:0/1914027652 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe860020070 con 0x7fe868158cb0 2023-03-03T05:45:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.607+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86815c3c0 con 0x7fe868158cb0 2023-03-03T05:45:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.607+0000 7fe85dffb700 1 -- 172.21.15.102:0/1914027652 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86001cbd0 con 0x7fe868158cb0 2023-03-03T05:45:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.607+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8681599e0 con 0x7fe868158cb0 2023-03-03T05:45:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.608+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8680c5050 con 0x7fe868158cb0 2023-03-03T05:45:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.609+0000 7fe85dffb700 1 -- 172.21.15.102:0/1914027652 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe86001cd30 con 0x7fe868158cb0 2023-03-03T05:45:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.611+0000 7fe85dffb700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe84c0682f0 0x7fe84c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.611+0000 7fe85dffb700 1 -- 172.21.15.102:0/1914027652 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe860022070 con 0x7fe868158cb0 2023-03-03T05:45:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.611+0000 7fe86d39b700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe84c0682f0 0x7fe84c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.612+0000 7fe86d39b700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe84c0682f0 0x7fe84c06a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe864005910 tx=0x7fe864009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.615+0000 7fe85dffb700 1 -- 172.21.15.102:0/1914027652 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8600547d0 con 0x7fe868158cb0 2023-03-03T05:45:38.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:38.852+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe868154750 con 0x7fe868158cb0 2023-03-03T05:45:39.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.321+0000 7fe85dffb700 1 -- 172.21.15.102:0/1914027652 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe868154750 con 0x7fe868158cb0 2023-03-03T05:45:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe84c0682f0 msgr2=0x7fe84c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe84c0682f0 0x7fe84c06a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe864005910 tx=0x7fe864009040 comp rx=0 tx=0).stop 2023-03-03T05:45:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe868158cb0 msgr2=0x7fe86814e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe868158cb0 0x7fe86814e430 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe860016000 tx=0x7fe860004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 shutdown_connections 2023-03-03T05:45:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe868158cb0 0x7fe86814e430 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8680c4390 0x7fe86815b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe84c0682f0 0x7fe84c06a7a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 --2- 172.21.15.102:0/1914027652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe868153180 0x7fe868154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 >> 172.21.15.102:0/1914027652 conn(0x7fe868071a30 msgr2=0x7fe86815c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:39.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 shutdown_connections 2023-03-03T05:45:39.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.324+0000 7fe86e39d700 1 -- 172.21.15.102:0/1914027652 wait complete. 2023-03-03T05:45:39.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-03-03T05:45:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.599+0000 7fda869ff700 1 Processor -- start 2023-03-03T05:45:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.599+0000 7fda869ff700 1 -- start start 2023-03-03T05:45:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.600+0000 7fda869ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fda8015a380 0x7fda800cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.600+0000 7fda869ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda800cf9a0 0x7fda800ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.600+0000 7fda869ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda800cae70 con 0x7fda80153450 2023-03-03T05:45:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.600+0000 7fda869ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda800cb110 con 0x7fda8015a380 2023-03-03T05:45:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.600+0000 7fda869ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda800cb3f0 con 0x7fda800cf9a0 2023-03-03T05:45:39.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.600+0000 7fda7f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fda8015a380 0x7fda800cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:39.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.600+0000 7fda84f9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda800cf9a0 0x7fda800ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:39.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7ffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fda80153450 0x7fda80159940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36374/0 (socket says 172.21.15.102:36374) 2023-03-03T05:45:39.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7ffff700 1 -- 172.21.15.102:0/3826732285 learned_addr learned my addr 172.21.15.102:0/3826732285 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:39.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7effd700 1 -- 172.21.15.102:0/3826732285 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204033876 0 0) 0x7fda800cae70 con 0x7fda80153450 2023-03-03T05:45:39.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7effd700 1 -- 172.21.15.102:0/3826732285 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda680035b0 con 0x7fda80153450 2023-03-03T05:45:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7f7fe700 1 -- 172.21.15.102:0/3826732285 >> v2:172.21.15.172:3300/0 conn(0x7fda800cf9a0 msgr2=0x7fda800ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7f7fe700 1 --2- 172.21.15.102:0/3826732285 >> v2:172.21.15.172:3300/0 conn(0x7fda800cf9a0 0x7fda800ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7f7fe700 1 -- 172.21.15.102:0/3826732285 >> v1:172.21.15.172:6789/0 conn(0x7fda80153450 legacy=0x7fda80159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7f7fe700 1 -- 172.21.15.102:0/3826732285 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800cb6d0 con 0x7fda8015a380 2023-03-03T05:45:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda84f9c700 1 --2- 172.21.15.102:0/3826732285 >> v2:172.21.15.172:3300/0 conn(0x7fda800cf9a0 0x7fda800ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.601+0000 7fda7f7fe700 1 --2- 172.21.15.102:0/3826732285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda800cc1b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda800744d0 tx=0x7fda74009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f456222e346779b5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda7effd700 1 -- 172.21.15.102:0/3826732285 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74011830 con 0x7fda8015a380 2023-03-03T05:45:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda7effd700 1 -- 172.21.15.102:0/3826732285 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74011990 con 0x7fda8015a380 2023-03-03T05:45:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda7effd700 1 -- 172.21.15.102:0/3826732285 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74011d00 con 0x7fda8015a380 2023-03-03T05:45:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 -- 172.21.15.102:0/3826732285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 msgr2=0x7fda800cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 --2- 172.21.15.102:0/3826732285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda800cc1b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda800744d0 tx=0x7fda74009b20 comp rx=0 tx=0).stop 2023-03-03T05:45:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 -- 172.21.15.102:0/3826732285 shutdown_connections 2023-03-03T05:45:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 --2- 172.21.15.102:0/3826732285 >> v2:172.21.15.172:3300/0 conn(0x7fda800cf9a0 0x7fda800ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 --2- 172.21.15.102:0/3826732285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda800cc1b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 -- 172.21.15.102:0/3826732285 >> 172.21.15.102:0/3826732285 conn(0x7fda80065fb0 msgr2=0x7fda800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 -- 172.21.15.102:0/3826732285 shutdown_connections 2023-03-03T05:45:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.602+0000 7fda869ff700 1 -- 172.21.15.102:0/3826732285 wait complete. 2023-03-03T05:45:39.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.604+0000 7fda869ff700 1 Processor -- start 2023-03-03T05:45:39.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.605+0000 7fda869ff700 1 -- start start 2023-03-03T05:45:39.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.605+0000 7fda869ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda800cf9a0 0x7fda800c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:39.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.605+0000 7fda869ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda80153450 0x7fda800c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:39.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.605+0000 7fda869ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda801f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:39.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda869ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda800c7a30 con 0x7fda8015a380 2023-03-03T05:45:39.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda869ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8014ecd0 con 0x7fda800cf9a0 2023-03-03T05:45:39.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda869ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda800ae850 con 0x7fda80153450 2023-03-03T05:45:39.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda7f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda80153450 0x7fda800c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda7ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda800cf9a0 0x7fda800c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda84f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda801f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda7f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda80153450 0x7fda800c8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36672/0 (socket says 172.21.15.102:36672) 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda7f7fe700 1 -- 172.21.15.102:0/1921449722 learned_addr learned my addr 172.21.15.102:0/1921449722 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda84f9c700 1 -- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda80153450 msgr2=0x7fda800c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda84f9c700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda80153450 0x7fda800c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda84f9c700 1 -- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda800cf9a0 msgr2=0x7fda800c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:39.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda84f9c700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda800cf9a0 0x7fda800c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:39.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda84f9c700 1 -- 172.21.15.102:0/1921449722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda78009a70 con 0x7fda8015a380 2023-03-03T05:45:39.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.606+0000 7fda7f7fe700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda80153450 0x7fda800c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:39.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.607+0000 7fda84f9c700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda801f2120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fda7800efc0 tx=0x7fda7800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:39.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.607+0000 7fda7cff9700 1 -- 172.21.15.102:0/1921449722 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78009e80 con 0x7fda8015a380 2023-03-03T05:45:39.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.607+0000 7fda7cff9700 1 -- 172.21.15.102:0/1921449722 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7800f040 con 0x7fda8015a380 2023-03-03T05:45:39.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.607+0000 7fda7cff9700 1 -- 172.21.15.102:0/1921449722 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7800ccd0 con 0x7fda8015a380 2023-03-03T05:45:39.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.607+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda740094a0 con 0x7fda8015a380 2023-03-03T05:45:39.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.607+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda800cbc90 con 0x7fda8015a380 2023-03-03T05:45:39.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.608+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda800cb9b0 con 0x7fda8015a380 2023-03-03T05:45:39.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.609+0000 7fda7cff9700 1 -- 172.21.15.102:0/1921449722 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda78014480 con 0x7fda8015a380 2023-03-03T05:45:39.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.611+0000 7fda7cff9700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda6c0682f0 0x7fda6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:39.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.611+0000 7fda7cff9700 1 -- 172.21.15.102:0/1921449722 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fda780863f0 con 0x7fda8015a380 2023-03-03T05:45:39.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.612+0000 7fda7ffff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda6c0682f0 0x7fda6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:39.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.612+0000 7fda7ffff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda6c0682f0 0x7fda6c06a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fda70007fd0 tx=0x7fda70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:39.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.615+0000 7fda7cff9700 1 -- 172.21.15.102:0/1921449722 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda78052a70 con 0x7fda8015a380 2023-03-03T05:45:39.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:39.854+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fda80066ac0 con 0x7fda8015a380 2023-03-03T05:45:40.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.327+0000 7fda7cff9700 1 -- 172.21.15.102:0/1921449722 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) v1 ==== 115+0+0 (secure 0 0 0) 0x7fda78056090 con 0x7fda8015a380 2023-03-03T05:45:40.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda6c0682f0 msgr2=0x7fda6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda6c0682f0 0x7fda6c06a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fda70007fd0 tx=0x7fda70009040 comp rx=0 tx=0).stop 2023-03-03T05:45:40.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 msgr2=0x7fda801f2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda801f2120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fda7800efc0 tx=0x7fda7800c3a0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 shutdown_connections 2023-03-03T05:45:40.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda8015a380 0x7fda801f2120 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda800cf9a0 0x7fda800c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda6c0682f0 0x7fda6c06a7a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.330+0000 7fda869ff700 1 --2- 172.21.15.102:0/1921449722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda80153450 0x7fda800c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.331+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 >> 172.21.15.102:0/1921449722 conn(0x7fda80065fb0 msgr2=0x7fda801523b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:40.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.331+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 shutdown_connections 2023-03-03T05:45:40.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.331+0000 7fda869ff700 1 -- 172.21.15.102:0/1921449722 wait complete. 2023-03-03T05:45:40.360 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T05:45:40.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:45:40.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.603+0000 7f0118bff700 1 Processor -- start 2023-03-03T05:45:40.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.603+0000 7f0118bff700 1 -- start start 2023-03-03T05:45:40.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.604+0000 7f0118bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f01141597f0 0x7f01140cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:40.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f0118bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140cb9f0 con 0x7f01140ced80 2023-03-03T05:45:40.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f0118bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114159c80 con 0x7f01140cf9a0 2023-03-03T05:45:40.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f0118bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01140ca750 con 0x7f01141597f0 2023-03-03T05:45:40.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f0111d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f01141597f0 0x7f01140cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:40.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f011259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f01140ced80 0x7f01140cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45088/0 (socket says 172.21.15.102:45088) 2023-03-03T05:45:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f011259c700 1 -- 172.21.15.102:0/3940464597 learned_addr learned my addr 172.21.15.102:0/3940464597 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f0111d9b700 1 -- 172.21.15.102:0/3940464597 >> v1:172.21.15.172:6789/0 conn(0x7f01140cf9a0 legacy=0x7f0114163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f0111d9b700 1 -- 172.21.15.102:0/3940464597 >> v1:172.21.15.102:6789/0 conn(0x7f01140ced80 legacy=0x7f01140cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.605+0000 7f0111d9b700 1 -- 172.21.15.102:0/3940464597 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140cade0 con 0x7f01141597f0 2023-03-03T05:45:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.606+0000 7f0111d9b700 1 --2- 172.21.15.102:0/3940464597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01141597f0 0x7f01140cf150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0108009840 tx=0x7f0108009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a609dc4f0bba9b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.606+0000 7f011159a700 1 -- 172.21.15.102:0/3940464597 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011830 con 0x7f01141597f0 2023-03-03T05:45:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.606+0000 7f011159a700 1 -- 172.21.15.102:0/3940464597 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108011990 con 0x7f01141597f0 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.606+0000 7f011159a700 1 -- 172.21.15.102:0/3940464597 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011d20 con 0x7f01141597f0 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.606+0000 7f0118bff700 1 -- 172.21.15.102:0/3940464597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01141597f0 msgr2=0x7f01140cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.606+0000 7f0118bff700 1 --2- 172.21.15.102:0/3940464597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01141597f0 0x7f01140cf150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0108009840 tx=0x7f0108009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.607+0000 7f0118bff700 1 -- 172.21.15.102:0/3940464597 shutdown_connections 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.607+0000 7f0118bff700 1 --2- 172.21.15.102:0/3940464597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01141597f0 0x7f01140cf150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.607+0000 7f0118bff700 1 -- 172.21.15.102:0/3940464597 >> 172.21.15.102:0/3940464597 conn(0x7f0114065fb0 msgr2=0x7f01140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.607+0000 7f0118bff700 1 -- 172.21.15.102:0/3940464597 shutdown_connections 2023-03-03T05:45:40.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.607+0000 7f0118bff700 1 -- 172.21.15.102:0/3940464597 wait complete. 2023-03-03T05:45:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.609+0000 7f0118bff700 1 Processor -- start 2023-03-03T05:45:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.609+0000 7f0118bff700 1 -- start start 2023-03-03T05:45:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0118bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140ced80 0x7f01140c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0118bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01140cf9a0 0x7f01140c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0118bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141597f0 0x7f01140c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0118bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c72e0 con 0x7f01140cf9a0 2023-03-03T05:45:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0118bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c75c0 con 0x7f01141597f0 2023-03-03T05:45:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0118bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c4b10 con 0x7f01140ced80 2023-03-03T05:45:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0111d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01140cf9a0 0x7f01140c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0111d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01140cf9a0 0x7f01140c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36162/0 (socket says 172.21.15.102:36162) 2023-03-03T05:45:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0111d9b700 1 -- 172.21.15.102:0/2892752595 learned_addr learned my addr 172.21.15.102:0/2892752595 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f011259c700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140ced80 0x7f01140c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0112d9d700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141597f0 0x7f01140c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0111d9b700 1 -- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140ced80 msgr2=0x7f01140c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0111d9b700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140ced80 0x7f01140c7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0111d9b700 1 -- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141597f0 msgr2=0x7f01140c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.610+0000 7f0111d9b700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141597f0 0x7f01140c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.611+0000 7f0111d9b700 1 -- 172.21.15.102:0/2892752595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080094a0 con 0x7f01140cf9a0 2023-03-03T05:45:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.611+0000 7f0111d9b700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01140cf9a0 0x7f01140c7f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01080090b0 tx=0x7f0108004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.611+0000 7f0102ffd700 1 -- 172.21.15.102:0/2892752595 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01080036a0 con 0x7f01140cf9a0 2023-03-03T05:45:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.611+0000 7f0102ffd700 1 -- 172.21.15.102:0/2892752595 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010801e070 con 0x7f01140cf9a0 2023-03-03T05:45:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.611+0000 7f0102ffd700 1 -- 172.21.15.102:0/2892752595 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108003a60 con 0x7f01140cf9a0 2023-03-03T05:45:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.611+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01140cb3a0 con 0x7f01140cf9a0 2023-03-03T05:45:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.611+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01140cb0c0 con 0x7f01140cf9a0 2023-03-03T05:45:40.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.612+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01140cb680 con 0x7f01140cf9a0 2023-03-03T05:45:40.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.614+0000 7f0102ffd700 1 -- 172.21.15.102:0/2892752595 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f010800f050 con 0x7f01140cf9a0 2023-03-03T05:45:40.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.615+0000 7f0102ffd700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00f40682f0 0x7f00f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:40.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.615+0000 7f0102ffd700 1 -- 172.21.15.102:0/2892752595 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0108086d60 con 0x7f01140cf9a0 2023-03-03T05:45:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.615+0000 7f011259c700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00f40682f0 0x7f00f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.616+0000 7f011259c700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00f40682f0 0x7f00f406a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0104005910 tx=0x7f0104009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:40.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.619+0000 7f0102ffd700 1 -- 172.21.15.102:0/2892752595 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01080534d0 con 0x7f01140cf9a0 2023-03-03T05:45:40.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.864+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01140cdfb0 con 0x7f01140cf9a0 2023-03-03T05:45:40.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.865+0000 7f0102ffd700 1 -- 172.21.15.102:0/2892752595 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6175 (secure 0 0 0) 0x7f0108015090 con 0x7f01140cf9a0 2023-03-03T05:45:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:40.867 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:45:40.321230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4315},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4315":{"gid":4315,"name":"e","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.102:6837/1262584332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":1262584332},{"type":"v1","addr":"172.21.15.102:6837","nonce":1262584332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:45:40.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00f40682f0 msgr2=0x7f00f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00f40682f0 0x7f00f406a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0104005910 tx=0x7f0104009040 comp rx=0 tx=0).stop 2023-03-03T05:45:40.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01140cf9a0 msgr2=0x7f01140c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01140cf9a0 0x7f01140c7f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01080090b0 tx=0x7f0108004750 comp rx=0 tx=0).stop 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 shutdown_connections 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00f40682f0 0x7f00f406a7a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140ced80 0x7f01140c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141597f0 0x7f01140c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 --2- 172.21.15.102:0/2892752595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01140cf9a0 0x7f01140c7f90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 >> 172.21.15.102:0/2892752595 conn(0x7f0114065fb0 msgr2=0x7f01140ccbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:40.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.867+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 shutdown_connections 2023-03-03T05:45:40.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:40.868+0000 7f0118bff700 1 -- 172.21.15.102:0/2892752595 wait complete. 2023-03-03T05:45:40.871 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 7 2023-03-03T05:45:40.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:41.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.136+0000 7fccb5f9d700 1 Processor -- start 2023-03-03T05:45:41.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.136+0000 7fccb5f9d700 1 -- start start 2023-03-03T05:45:41.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccb5f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb00ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccb5f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb00c9ca0 con 0x7fccb00ca980 2023-03-03T05:45:41.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccb5f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb00b46f0 con 0x7fccb01564b0 2023-03-03T05:45:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccb5f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00cadc0 con 0x7fccb00c9610 2023-03-03T05:45:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccb4f9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fccb01564b0 0x7fccb00cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36392/0 (socket says 172.21.15.102:36392) 2023-03-03T05:45:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccb4f9b700 1 -- 172.21.15.102:0/3750307368 learned_addr learned my addr 172.21.15.102:0/3750307368 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccaffff700 1 --2- 172.21.15.102:0/3750307368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb00ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.138+0000 7fccaf7fe700 1 -- 172.21.15.102:0/3750307368 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869339773 0 0) 0x7fccb00b46f0 con 0x7fccb01564b0 2023-03-03T05:45:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaf7fe700 1 -- 172.21.15.102:0/3750307368 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca00035b0 con 0x7fccb01564b0 2023-03-03T05:45:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaffff700 1 -- 172.21.15.102:0/3750307368 >> v1:172.21.15.172:6789/0 conn(0x7fccb01564b0 legacy=0x7fccb00cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaffff700 1 -- 172.21.15.102:0/3750307368 >> v1:172.21.15.102:6789/0 conn(0x7fccb00ca980 legacy=0x7fccb01636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaf7fe700 1 -- 172.21.15.102:0/3750307368 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490809458 0 0) 0x7fccb00c9ca0 con 0x7fccb00ca980 2023-03-03T05:45:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaffff700 1 -- 172.21.15.102:0/3750307368 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb00b4920 con 0x7fccb00c9610 2023-03-03T05:45:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaffff700 1 --2- 172.21.15.102:0/3750307368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb00ca440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009870 tx=0x7fcc9c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37edafb34b9e610e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaf7fe700 1 -- 172.21.15.102:0/3750307368 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c011830 con 0x7fccb00c9610 2023-03-03T05:45:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaf7fe700 1 -- 172.21.15.102:0/3750307368 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c011990 con 0x7fccb00c9610 2023-03-03T05:45:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.139+0000 7fccaf7fe700 1 -- 172.21.15.102:0/3750307368 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c011d20 con 0x7fccb00c9610 2023-03-03T05:45:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.140+0000 7fccb5f9d700 1 -- 172.21.15.102:0/3750307368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 msgr2=0x7fccb00ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.140+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/3750307368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb00ca440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009870 tx=0x7fcc9c009b80 comp rx=0 tx=0).stop 2023-03-03T05:45:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.140+0000 7fccb5f9d700 1 -- 172.21.15.102:0/3750307368 shutdown_connections 2023-03-03T05:45:41.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.140+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/3750307368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb00ca440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.140+0000 7fccb5f9d700 1 -- 172.21.15.102:0/3750307368 >> 172.21.15.102:0/3750307368 conn(0x7fccb0065fb0 msgr2=0x7fccb00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:41.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.140+0000 7fccb5f9d700 1 -- 172.21.15.102:0/3750307368 shutdown_connections 2023-03-03T05:45:41.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.140+0000 7fccb5f9d700 1 -- 172.21.15.102:0/3750307368 wait complete. 2023-03-03T05:45:41.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.142+0000 7fccb5f9d700 1 Processor -- start 2023-03-03T05:45:41.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.142+0000 7fccb5f9d700 1 -- start start 2023-03-03T05:45:41.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.143+0000 7fccb5f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb0151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.143+0000 7fccb5f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb00ca980 0x7fccb014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.143+0000 7fccb5f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccb01564b0 0x7fccb014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.143+0000 7fccb5f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb015b860 con 0x7fccb01564b0 2023-03-03T05:45:41.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.143+0000 7fccb5f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb014e730 con 0x7fccb00ca980 2023-03-03T05:45:41.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.143+0000 7fccb5f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb014ea10 con 0x7fccb00c9610 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb0151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb0151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36694/0 (socket says 172.21.15.102:36694) 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 -- 172.21.15.102:0/1180198032 learned_addr learned my addr 172.21.15.102:0/1180198032 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccaffff700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb00ca980 0x7fccb014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb579c700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccb01564b0 0x7fccb014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 -- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb00ca980 msgr2=0x7fccb014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb00ca980 0x7fccb014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 -- 172.21.15.102:0/1180198032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccb01564b0 msgr2=0x7fccb014f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccb01564b0 0x7fccb014f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 -- 172.21.15.102:0/1180198032 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c0094a0 con 0x7fccb00c9610 2023-03-03T05:45:41.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.144+0000 7fccb4f9b700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb0151300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcca400c500 tx=0x7fcca400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:41.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.145+0000 7fccad7fa700 1 -- 172.21.15.102:0/1180198032 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4005d70 con 0x7fccb00c9610 2023-03-03T05:45:41.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.145+0000 7fccad7fa700 1 -- 172.21.15.102:0/1180198032 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca40070d0 con 0x7fccb00c9610 2023-03-03T05:45:41.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.145+0000 7fccad7fa700 1 -- 172.21.15.102:0/1180198032 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4007480 con 0x7fccb00c9610 2023-03-03T05:45:41.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.145+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb00b1f30 con 0x7fccb00c9610 2023-03-03T05:45:41.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.145+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb00b4c00 con 0x7fccb00c9610 2023-03-03T05:45:41.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.146+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb00b2210 con 0x7fccb00c9610 2023-03-03T05:45:41.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.147+0000 7fccad7fa700 1 -- 172.21.15.102:0/1180198032 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcca4019030 con 0x7fccb00c9610 2023-03-03T05:45:41.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.148+0000 7fccad7fa700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc94068320 0x7fcc9406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.148+0000 7fccad7fa700 1 -- 172.21.15.102:0/1180198032 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcca4084f80 con 0x7fccb00c9610 2023-03-03T05:45:41.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.149+0000 7fccaffff700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc94068320 0x7fcc9406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.149+0000 7fccaffff700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc94068320 0x7fcc9406a7d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0097c0 tx=0x7fcc9c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:41.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.152+0000 7fccad7fa700 1 -- 172.21.15.102:0/1180198032 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcca4051690 con 0x7fccb00c9610 2023-03-03T05:45:41.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.351+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb00677f0 con 0x7fccb00c9610 2023-03-03T05:45:41.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.352+0000 7fccad7fa700 1 -- 172.21.15.102:0/1180198032 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7fcca4017020 con 0x7fccb00c9610 2023-03-03T05:45:41.355 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:41.355 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:41.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc94068320 msgr2=0x7fcc9406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc94068320 0x7fcc9406a7d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0097c0 tx=0x7fcc9c01c040 comp rx=0 tx=0).stop 2023-03-03T05:45:41.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 msgr2=0x7fccb0151300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb0151300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcca400c500 tx=0x7fcca400c810 comp rx=0 tx=0).stop 2023-03-03T05:45:41.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 shutdown_connections 2023-03-03T05:45:41.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc94068320 0x7fcc9406a7d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccb00c9610 0x7fccb0151300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccb01564b0 0x7fccb014f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.355+0000 7fccb5f9d700 1 --2- 172.21.15.102:0/1180198032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccb00ca980 0x7fccb014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.356+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 >> 172.21.15.102:0/1180198032 conn(0x7fccb0065fb0 msgr2=0x7fccb0154f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:41.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.356+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 shutdown_connections 2023-03-03T05:45:41.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.356+0000 7fccb5f9d700 1 -- 172.21.15.102:0/1180198032 wait complete. 2023-03-03T05:45:41.385 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T05:45:41.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T05:45:41.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.626+0000 7f9682d9d700 1 Processor -- start 2023-03-03T05:45:41.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.626+0000 7f9682d9d700 1 -- start start 2023-03-03T05:45:41.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f9682d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f967c0c9610 0x7f967c0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f9682d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c0ca980 0x7f967c1636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f9682d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c0c9ca0 con 0x7f967c1564b0 2023-03-03T05:45:41.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f9682d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c0b4920 con 0x7f967c0c9610 2023-03-03T05:45:41.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f9682d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0b4c00 con 0x7f967c0ca980 2023-03-03T05:45:41.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f968159a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f967c0c9610 0x7f967c0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f9681d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f967c1564b0 0x7f967c0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45102/0 (socket says 172.21.15.102:45102) 2023-03-03T05:45:41.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f9681d9b700 1 -- 172.21.15.102:0/3000416046 learned_addr learned my addr 172.21.15.102:0/3000416046 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:41.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.628+0000 7f968259c700 1 --2- 172.21.15.102:0/3000416046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c0ca980 0x7f967c1636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f968159a700 1 -- 172.21.15.102:0/3000416046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c0ca980 msgr2=0x7f967c1636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:41.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f968159a700 1 --2- 172.21.15.102:0/3000416046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c0ca980 0x7f967c1636a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f968159a700 1 -- 172.21.15.102:0/3000416046 >> v1:172.21.15.102:6789/0 conn(0x7f967c1564b0 legacy=0x7f967c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f968159a700 1 -- 172.21.15.102:0/3000416046 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0b46f0 con 0x7f967c0c9610 2023-03-03T05:45:41.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f9680d99700 1 -- 172.21.15.102:0/3000416046 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68880049 0 0) 0x7f967c0c9ca0 con 0x7f967c1564b0 2023-03-03T05:45:41.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f968159a700 1 --2- 172.21.15.102:0/3000416046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0c9610 0x7f967c0ca440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f966c009840 tx=0x7f966c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9311dc4ba6e6b2b9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:41.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f9680d99700 1 -- 172.21.15.102:0/3000416046 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c011830 con 0x7f967c0c9610 2023-03-03T05:45:41.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f9680d99700 1 -- 172.21.15.102:0/3000416046 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c011990 con 0x7f967c0c9610 2023-03-03T05:45:41.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.629+0000 7f9680d99700 1 -- 172.21.15.102:0/3000416046 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c011d20 con 0x7f967c0c9610 2023-03-03T05:45:41.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 -- 172.21.15.102:0/3000416046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0c9610 msgr2=0x7f967c0ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3000416046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0c9610 0x7f967c0ca440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f966c009840 tx=0x7f966c009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:41.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 -- 172.21.15.102:0/3000416046 shutdown_connections 2023-03-03T05:45:41.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3000416046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c0ca980 0x7f967c1636a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3000416046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0c9610 0x7f967c0ca440 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 -- 172.21.15.102:0/3000416046 >> 172.21.15.102:0/3000416046 conn(0x7f967c065fb0 msgr2=0x7f967c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:41.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 -- 172.21.15.102:0/3000416046 shutdown_connections 2023-03-03T05:45:41.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.630+0000 7f9682d9d700 1 -- 172.21.15.102:0/3000416046 wait complete. 2023-03-03T05:45:41.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.632+0000 7f9682d9d700 1 Processor -- start 2023-03-03T05:45:41.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.632+0000 7f9682d9d700 1 -- start start 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.633+0000 7f9682d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f967c0c9610 0x7f967c1f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.633+0000 7f9682d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0ca980 0x7f967c15a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.633+0000 7f9682d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c1564b0 0x7f967c15a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.633+0000 7f9682d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0c62b0 con 0x7f967c0ca980 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.633+0000 7f9682d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c15cad0 con 0x7f967c0c9610 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.633+0000 7f9682d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c15cd70 con 0x7f967c1564b0 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0ca980 0x7f967c15a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f9681d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f967c0c9610 0x7f967c1f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0ca980 0x7f967c15a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36188/0 (socket says 172.21.15.102:36188) 2023-03-03T05:45:41.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c1564b0 0x7f967c15a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f9681d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f967c0c9610 0x7f967c1f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59198/0 (socket says 172.21.15.102:59198) 2023-03-03T05:45:41.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 -- 172.21.15.102:0/3534257717 learned_addr learned my addr 172.21.15.102:0/3534257717 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:41.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 -- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c1564b0 msgr2=0x7f967c15a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c1564b0 0x7f967c15a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 -- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f967c0c9610 msgr2=0x7f967c1f1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f967c0c9610 0x7f967c1f1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 -- 172.21.15.102:0/3534257717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0094a0 con 0x7f967c0ca980 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.634+0000 7f968159a700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0ca980 0x7f967c15a1d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f966c009fd0 tx=0x7f966c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.635+0000 7f96727fc700 1 -- 172.21.15.102:0/3534257717 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c0036a0 con 0x7f967c0ca980 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.635+0000 7f96727fc700 1 -- 172.21.15.102:0/3534257717 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c01f070 con 0x7f967c0ca980 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.635+0000 7f96727fc700 1 -- 172.21.15.102:0/3534257717 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c003a60 con 0x7f967c0ca980 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.635+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c0b2210 con 0x7f967c0ca980 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.635+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c0b1f30 con 0x7f967c0ca980 2023-03-03T05:45:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.636+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c156880 con 0x7f967c0ca980 2023-03-03T05:45:41.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.637+0000 7f96727fc700 1 -- 172.21.15.102:0/3534257717 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f966c00f050 con 0x7f967c0ca980 2023-03-03T05:45:41.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.638+0000 7f96727fc700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9660068320 0x7f966006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:41.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.638+0000 7f96727fc700 1 -- 172.21.15.102:0/3534257717 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f966c021070 con 0x7f967c0ca980 2023-03-03T05:45:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.639+0000 7f9681d9b700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9660068320 0x7f966006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.639+0000 7f9681d9b700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9660068320 0x7f966006a7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9678000f60 tx=0x7f9678007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.642+0000 7f96727fc700 1 -- 172.21.15.102:0/3534257717 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f966c053cf0 con 0x7f967c0ca980 2023-03-03T05:45:41.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.829+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 --> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f967c04e6e0 con 0x7f9660068320 2023-03-03T05:45:41.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.839+0000 7f96727fc700 1 -- 172.21.15.102:0/3534257717 <== mgr.4105 v2:172.21.15.172:6832/114058 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278389 (secure 0 0 0) 0x7f967c04e6e0 con 0x7f9660068320 2023-03-03T05:45:41.843 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:41.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.843+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9660068320 msgr2=0x7f966006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9660068320 0x7f966006a7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9678000f60 tx=0x7f9678007e70 comp rx=0 tx=0).stop 2023-03-03T05:45:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0ca980 msgr2=0x7f967c15a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0ca980 0x7f967c15a1d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f966c009fd0 tx=0x7f966c004750 comp rx=0 tx=0).stop 2023-03-03T05:45:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 shutdown_connections 2023-03-03T05:45:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9660068320 0x7f966006a7d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f967c1564b0 0x7f967c15a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f967c0c9610 0x7f967c1f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 --2- 172.21.15.102:0/3534257717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f967c0ca980 0x7f967c15a1d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:41.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 >> 172.21.15.102:0/3534257717 conn(0x7f967c065fb0 msgr2=0x7f967c156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:41.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 shutdown_connections 2023-03-03T05:45:41.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:41.844+0000 7f9682d9d700 1 -- 172.21.15.102:0/3534257717 wait complete. 2023-03-03T05:45:41.848 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2023-03-03T05:45:41.873 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-03-03T05:45:39.943419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65020,"kb_used_data":2748,"kb_used_omap":0,"kb_used_meta":62208,"kb_avail":754909700,"statfs":{"total":773094113280,"available":773027532800,"internally_reserved":0,"allocated":2813952,"data_stored":1741898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63700992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.665301"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850859+0000","last_change":"2023-03-03T05:45:35.862354+0000","last_active":"2023-03-03T05:45:36.850859+0000","last_peered":"2023-03-03T05:45:36.850859+0000","last_clean":"2023-03-03T05:45:36.850859+0000","last_became_active":"2023-03-03T05:45:35.862016+0000","last_became_peered":"2023-03-03T05:45:35.862016+0000","last_unstale":"2023-03-03T05:45:36.850859+0000","last_undegraded":"2023-03-03T05:45:36.850859+0000","last_fullsized":"2023-03-03T05:45:36.850859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.871294+0000","last_change":"2023-03-03T05:45:35.864429+0000","last_active":"2023-03-03T05:45:35.871294+0000","last_peered":"2023-03-03T05:45:35.871294+0000","last_clean":"2023-03-03T05:45:35.871294+0000","last_became_active":"2023-03-03T05:45:35.864091+0000","last_became_peered":"2023-03-03T05:45:35.864091+0000","last_unstale":"2023-03-03T05:45:35.871294+0000","last_undegraded":"2023-03-03T05:45:35.871294+0000","last_fullsized":"2023-03-03T05:45:35.871294+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.871339+0000","last_change":"2023-03-03T05:45:35.861374+0000","last_active":"2023-03-03T05:45:35.871339+0000","last_peered":"2023-03-03T05:45:35.871339+0000","last_clean":"2023-03-03T05:45:35.871339+0000","last_became_active":"2023-03-03T05:45:35.860993+0000","last_became_peered":"2023-03-03T05:45:35.860993+0000","last_unstale":"2023-03-03T05:45:35.871339+0000","last_undegraded":"2023-03-03T05:45:35.871339+0000","last_fullsized":"2023-03-03T05:45:35.871339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.871702+0000","last_change":"2023-03-03T05:45:35.862593+0000","last_active":"2023-03-03T05:45:35.871702+0000","last_peered":"2023-03-03T05:45:35.871702+0000","last_clean":"2023-03-03T05:45:35.871702+0000","last_became_active":"2023-03-03T05:45:35.862342+0000","last_became_peered":"2023-03-03T05:45:35.862342+0000","last_unstale":"2023-03-03T05:45:35.871702+0000","last_undegraded":"2023-03-03T05:45:35.871702+0000","last_fullsized":"2023-03-03T05:45:35.871702+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.871149+0000","last_change":"2023-03-03T05:45:35.861090+0000","last_active":"2023-03-03T05:45:35.871149+0000","last_peered":"2023-03-03T05:45:35.871149+0000","last_clean":"2023-03-03T05:45:35.871149+0000","last_became_active":"2023-03-03T05:45:35.860814+0000","last_became_peered":"2023-03-03T05:45:35.860814+0000","last_unstale":"2023-03-03T05:45:35.871149+0000","last_undegraded":"2023-03-03T05:45:35.871149+0000","last_fullsized":"2023-03-03T05:45:35.871149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.281015+0000","last_change":"2023-03-03T05:45:34.281015+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.281015+0000","last_undegraded":"2023-03-03T05:45:34.281015+0000","last_fullsized":"2023-03-03T05:45:34.281015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.286542+0000","last_change":"2023-03-03T05:45:34.286542+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.286542+0000","last_undegraded":"2023-03-03T05:45:34.286542+0000","last_fullsized":"2023-03-03T05:45:34.286542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:32.932001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.283743+0000","last_change":"2023-03-03T05:45:34.283743+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.283743+0000","last_undegraded":"2023-03-03T05:45:34.283743+0000","last_fullsized":"2023-03-03T05:45:34.283743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.283296+0000","last_change":"2023-03-03T05:45:34.283296+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.283296+0000","last_undegraded":"2023-03-03T05:45:34.283296+0000","last_fullsized":"2023-03-03T05:45:34.283296+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:17:39.778447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851170+0000","last_change":"2023-03-03T05:45:35.304400+0000","last_active":"2023-03-03T05:45:36.851170+0000","last_peered":"2023-03-03T05:45:36.851170+0000","last_clean":"2023-03-03T05:45:36.851170+0000","last_became_active":"2023-03-03T05:45:35.304093+0000","last_became_peered":"2023-03-03T05:45:35.304093+0000","last_unstale":"2023-03-03T05:45:36.851170+0000","last_undegraded":"2023-03-03T05:45:36.851170+0000","last_fullsized":"2023-03-03T05:45:36.851170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.853998+0000","last_change":"2023-03-03T05:45:35.302282+0000","last_active":"2023-03-03T05:45:35.853998+0000","last_peered":"2023-03-03T05:45:35.853998+0000","last_clean":"2023-03-03T05:45:35.853998+0000","last_became_active":"2023-03-03T05:45:35.302002+0000","last_became_peered":"2023-03-03T05:45:35.302002+0000","last_unstale":"2023-03-03T05:45:35.853998+0000","last_undegraded":"2023-03-03T05:45:35.853998+0000","last_fullsized":"2023-03-03T05:45:35.853998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.281623+0000","last_change":"2023-03-03T05:45:34.281623+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.281623+0000","last_undegraded":"2023-03-03T05:45:34.281623+0000","last_fullsized":"2023-03-03T05:45:34.281623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:30.692527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850547+0000","last_change":"2023-03-03T05:45:35.856887+0000","last_active":"2023-03-03T05:45:36.850547+0000","last_peered":"2023-03-03T05:45:36.850547+0000","last_clean":"2023-03-03T05:45:36.850547+0000","last_became_active":"2023-03-03T05:45:35.856498+0000","last_became_peered":"2023-03-03T05:45:35.856498+0000","last_unstale":"2023-03-03T05:45:36.850547+0000","last_undegraded":"2023-03-03T05:45:36.850547+0000","last_fullsized":"2023-03-03T05:45:36.850547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.281044+0000","last_change":"2023-03-03T05:45:34.281044+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.281044+0000","last_undegraded":"2023-03-03T05:45:34.281044+0000","last_fullsized":"2023-03-03T05:45:34.281044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:48.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.287265+0000","last_change":"2023-03-03T05:45:34.287265+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.287265+0000","last_undegraded":"2023-03-03T05:45:34.287265+0000","last_fullsized":"2023-03-03T05:45:34.287265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.283571+0000","last_change":"2023-03-03T05:45:34.283571+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.283571+0000","last_undegraded":"2023-03-03T05:45:34.283571+0000","last_fullsized":"2023-03-03T05:45:34.283571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:08.368039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.280925+0000","last_change":"2023-03-03T05:45:34.280925+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.280925+0000","last_undegraded":"2023-03-03T05:45:34.280925+0000","last_fullsized":"2023-03-03T05:45:34.280925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.853965+0000","last_change":"2023-03-03T05:45:35.308976+0000","last_active":"2023-03-03T05:45:35.853965+0000","last_peered":"2023-03-03T05:45:35.853965+0000","last_clean":"2023-03-03T05:45:35.853965+0000","last_became_active":"2023-03-03T05:45:35.308707+0000","last_became_peered":"2023-03-03T05:45:35.308707+0000","last_unstale":"2023-03-03T05:45:35.853965+0000","last_undegraded":"2023-03-03T05:45:35.853965+0000","last_fullsized":"2023-03-03T05:45:35.853965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851097+0000","last_change":"2023-03-03T05:45:35.303791+0000","last_active":"2023-03-03T05:45:36.851097+0000","last_peered":"2023-03-03T05:45:36.851097+0000","last_clean":"2023-03-03T05:45:36.851097+0000","last_became_active":"2023-03-03T05:45:35.303482+0000","last_became_peered":"2023-03-03T05:45:35.303482+0000","last_unstale":"2023-03-03T05:45:36.851097+0000","last_undegraded":"2023-03-03T05:45:36.851097+0000","last_fullsized":"2023-03-03T05:45:36.851097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:31.933113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.281068+0000","last_change":"2023-03-03T05:45:34.281068+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.281068+0000","last_undegraded":"2023-03-03T05:45:34.281068+0000","last_fullsized":"2023-03-03T05:45:34.281068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:46.322139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850472+0000","last_change":"2023-03-03T05:45:35.856376+0000","last_active":"2023-03-03T05:45:36.850472+0000","last_peered":"2023-03-03T05:45:36.850472+0000","last_clean":"2023-03-03T05:45:36.850472+0000","last_became_active":"2023-03-03T05:45:35.856064+0000","last_became_peered":"2023-03-03T05:45:35.856064+0000","last_unstale":"2023-03-03T05:45:36.850472+0000","last_undegraded":"2023-03-03T05:45:36.850472+0000","last_fullsized":"2023-03-03T05:45:36.850472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.854163+0000","last_change":"2023-03-03T05:45:35.295520+0000","last_active":"2023-03-03T05:45:35.854163+0000","last_peered":"2023-03-03T05:45:35.854163+0000","last_clean":"2023-03-03T05:45:35.854163+0000","last_became_active":"2023-03-03T05:45:35.295203+0000","last_became_peered":"2023-03-03T05:45:35.295203+0000","last_unstale":"2023-03-03T05:45:35.854163+0000","last_undegraded":"2023-03-03T05:45:35.854163+0000","last_fullsized":"2023-03-03T05:45:35.854163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.285835+0000","last_change":"2023-03-03T05:45:34.285835+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.285835+0000","last_undegraded":"2023-03-03T05:45:34.285835+0000","last_fullsized":"2023-03-03T05:45:34.285835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.871322+0000","last_change":"2023-03-03T05:45:35.861231+0000","last_active":"2023-03-03T05:45:35.871322+0000","last_peered":"2023-03-03T05:45:35.871322+0000","last_clean":"2023-03-03T05:45:35.871322+0000","last_became_active":"2023-03-03T05:45:35.860675+0000","last_became_peered":"2023-03-03T05:45:35.860675+0000","last_unstale":"2023-03-03T05:45:35.871322+0000","last_undegraded":"2023-03-03T05:45:35.871322+0000","last_fullsized":"2023-03-03T05:45:35.871322+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.281348+0000","last_change":"2023-03-03T05:45:34.281348+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.281348+0000","last_undegraded":"2023-03-03T05:45:34.281348+0000","last_fullsized":"2023-03-03T05:45:34.281348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:22.294783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.282804+0000","last_change":"2023-03-03T05:45:34.282804+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.282804+0000","last_undegraded":"2023-03-03T05:45:34.282804+0000","last_fullsized":"2023-03-03T05:45:34.282804+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:16.655836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850964+0000","last_change":"2023-03-03T05:45:35.857451+0000","last_active":"2023-03-03T05:45:36.850964+0000","last_peered":"2023-03-03T05:45:36.850964+0000","last_clean":"2023-03-03T05:45:36.850964+0000","last_became_active":"2023-03-03T05:45:35.857087+0000","last_became_peered":"2023-03-03T05:45:35.857087+0000","last_unstale":"2023-03-03T05:45:36.850964+0000","last_undegraded":"2023-03-03T05:45:36.850964+0000","last_fullsized":"2023-03-03T05:45:36.850964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.286305+0000","last_change":"2023-03-03T05:45:34.286305+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.286305+0000","last_undegraded":"2023-03-03T05:45:34.286305+0000","last_fullsized":"2023-03-03T05:45:34.286305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.372984+0000","last_change":"2023-03-03T05:45:29.189033+0000","last_active":"2023-03-03T05:45:30.372984+0000","last_peered":"2023-03-03T05:45:30.372984+0000","last_clean":"2023-03-03T05:45:30.372984+0000","last_became_active":"2023-03-03T05:45:29.188719+0000","last_became_peered":"2023-03-03T05:45:29.188719+0000","last_unstale":"2023-03-03T05:45:30.372984+0000","last_undegraded":"2023-03-03T05:45:30.372984+0000","last_fullsized":"2023-03-03T05:45:30.372984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850356+0000","last_change":"2023-03-03T05:45:35.303756+0000","last_active":"2023-03-03T05:45:36.850356+0000","last_peered":"2023-03-03T05:45:36.850356+0000","last_clean":"2023-03-03T05:45:36.850356+0000","last_became_active":"2023-03-03T05:45:35.303448+0000","last_became_peered":"2023-03-03T05:45:35.303448+0000","last_unstale":"2023-03-03T05:45:36.850356+0000","last_undegraded":"2023-03-03T05:45:36.850356+0000","last_fullsized":"2023-03-03T05:45:36.850356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.373300+0000","last_change":"2023-03-03T05:45:29.189075+0000","last_active":"2023-03-03T05:45:30.373300+0000","last_peered":"2023-03-03T05:45:30.373300+0000","last_clean":"2023-03-03T05:45:30.373300+0000","last_became_active":"2023-03-03T05:45:29.188778+0000","last_became_peered":"2023-03-03T05:45:29.188778+0000","last_unstale":"2023-03-03T05:45:30.373300+0000","last_undegraded":"2023-03-03T05:45:30.373300+0000","last_fullsized":"2023-03-03T05:45:30.373300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:54:35.536172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.853930+0000","last_change":"2023-03-03T05:45:35.309781+0000","last_active":"2023-03-03T05:45:35.853930+0000","last_peered":"2023-03-03T05:45:35.853930+0000","last_clean":"2023-03-03T05:45:35.853930+0000","last_became_active":"2023-03-03T05:45:35.309497+0000","last_became_peered":"2023-03-03T05:45:35.309497+0000","last_unstale":"2023-03-03T05:45:35.853930+0000","last_undegraded":"2023-03-03T05:45:35.853930+0000","last_fullsized":"2023-03-03T05:45:35.853930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.373034+0000","last_change":"2023-03-03T05:45:29.057308+0000","last_active":"2023-03-03T05:45:30.373034+0000","last_peered":"2023-03-03T05:45:30.373034+0000","last_clean":"2023-03-03T05:45:30.373034+0000","last_became_active":"2023-03-03T05:45:29.056828+0000","last_became_peered":"2023-03-03T05:45:29.056828+0000","last_unstale":"2023-03-03T05:45:30.373034+0000","last_undegraded":"2023-03-03T05:45:30.373034+0000","last_fullsized":"2023-03-03T05:45:30.373034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:56.079893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.283209+0000","last_change":"2023-03-03T05:45:34.283209+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.283209+0000","last_undegraded":"2023-03-03T05:45:34.283209+0000","last_fullsized":"2023-03-03T05:45:34.283209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.373390+0000","last_change":"2023-03-03T05:45:29.057646+0000","last_active":"2023-03-03T05:45:30.373390+0000","last_peered":"2023-03-03T05:45:30.373390+0000","last_clean":"2023-03-03T05:45:30.373390+0000","last_became_active":"2023-03-03T05:45:29.057330+0000","last_became_peered":"2023-03-03T05:45:29.057330+0000","last_unstale":"2023-03-03T05:45:30.373390+0000","last_undegraded":"2023-03-03T05:45:30.373390+0000","last_fullsized":"2023-03-03T05:45:30.373390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:22:57.039574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.373242+0000","last_change":"2023-03-03T05:45:22.224435+0000","last_active":"2023-03-03T05:45:30.373242+0000","last_peered":"2023-03-03T05:45:30.373242+0000","last_clean":"2023-03-03T05:45:30.373242+0000","last_became_active":"2023-03-03T05:45:22.222958+0000","last_became_peered":"2023-03-03T05:45:22.222958+0000","last_unstale":"2023-03-03T05:45:30.373242+0000","last_undegraded":"2023-03-03T05:45:30.373242+0000","last_fullsized":"2023-03-03T05:45:30.373242+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_clean_scrub_stamp":"2023-03-03T05:45:20.925942+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:10:20.426026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.286961+0000","last_change":"2023-03-03T05:45:34.286961+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.286961+0000","last_undegraded":"2023-03-03T05:45:34.286961+0000","last_fullsized":"2023-03-03T05:45:34.286961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.373302+0000","last_change":"2023-03-03T05:45:29.056874+0000","last_active":"2023-03-03T05:45:30.373302+0000","last_peered":"2023-03-03T05:45:30.373302+0000","last_clean":"2023-03-03T05:45:30.373302+0000","last_became_active":"2023-03-03T05:45:29.056438+0000","last_became_peered":"2023-03-03T05:45:29.056438+0000","last_unstale":"2023-03-03T05:45:30.373302+0000","last_undegraded":"2023-03-03T05:45:30.373302+0000","last_fullsized":"2023-03-03T05:45:30.373302+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:08.113953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.287987+0000","last_change":"2023-03-03T05:45:34.287987+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.287987+0000","last_undegraded":"2023-03-03T05:45:34.287987+0000","last_fullsized":"2023-03-03T05:45:34.287987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.373364+0000","last_change":"2023-03-03T05:45:29.056891+0000","last_active":"2023-03-03T05:45:30.373364+0000","last_peered":"2023-03-03T05:45:30.373364+0000","last_clean":"2023-03-03T05:45:30.373364+0000","last_became_active":"2023-03-03T05:45:29.056483+0000","last_became_peered":"2023-03-03T05:45:29.056483+0000","last_unstale":"2023-03-03T05:45:30.373364+0000","last_undegraded":"2023-03-03T05:45:30.373364+0000","last_fullsized":"2023-03-03T05:45:30.373364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:25.701585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.287647+0000","last_change":"2023-03-03T05:45:34.287647+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.287647+0000","last_undegraded":"2023-03-03T05:45:34.287647+0000","last_fullsized":"2023-03-03T05:45:34.287647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.286751+0000","last_change":"2023-03-03T05:45:34.286751+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.286751+0000","last_undegraded":"2023-03-03T05:45:34.286751+0000","last_fullsized":"2023-03-03T05:45:34.286751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.060518+0000","last_change":"2023-03-03T05:45:29.193601+0000","last_active":"2023-03-03T05:45:30.060518+0000","last_peered":"2023-03-03T05:45:30.060518+0000","last_clean":"2023-03-03T05:45:30.060518+0000","last_became_active":"2023-03-03T05:45:29.193262+0000","last_became_peered":"2023-03-03T05:45:29.193262+0000","last_unstale":"2023-03-03T05:45:30.060518+0000","last_undegraded":"2023-03-03T05:45:30.060518+0000","last_fullsized":"2023-03-03T05:45:30.060518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-03-03T05:45:30.373362+0000","last_change":"2023-03-03T05:45:29.193546+0000","last_active":"2023-03-03T05:45:30.373362+0000","last_peered":"2023-03-03T05:45:30.373362+0000","last_clean":"2023-03-03T05:45:30.373362+0000","last_became_active":"2023-03-03T05:45:29.193172+0000","last_became_peered":"2023-03-03T05:45:29.193172+0000","last_unstale":"2023-03-03T05:45:30.373362+0000","last_undegraded":"2023-03-03T05:45:30.373362+0000","last_fullsized":"2023-03-03T05:45:30.373362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:07:42.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.286287+0000","last_change":"2023-03-03T05:45:34.286287+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.286287+0000","last_undegraded":"2023-03-03T05:45:34.286287+0000","last_fullsized":"2023-03-03T05:45:34.286287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850872+0000","last_change":"2023-03-03T05:45:35.856468+0000","last_active":"2023-03-03T05:45:36.850872+0000","last_peered":"2023-03-03T05:45:36.850872+0000","last_clean":"2023-03-03T05:45:36.850872+0000","last_became_active":"2023-03-03T05:45:35.855970+0000","last_became_peered":"2023-03-03T05:45:35.855970+0000","last_unstale":"2023-03-03T05:45:36.850872+0000","last_undegraded":"2023-03-03T05:45:36.850872+0000","last_fullsized":"2023-03-03T05:45:36.850872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.287052+0000","last_change":"2023-03-03T05:45:34.287052+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.287052+0000","last_undegraded":"2023-03-03T05:45:34.287052+0000","last_fullsized":"2023-03-03T05:45:34.287052+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:42.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.281877+0000","last_change":"2023-03-03T05:45:34.281877+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.281877+0000","last_undegraded":"2023-03-03T05:45:34.281877+0000","last_fullsized":"2023-03-03T05:45:34.281877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850462+0000","last_change":"2023-03-03T05:45:35.861316+0000","last_active":"2023-03-03T05:45:36.850462+0000","last_peered":"2023-03-03T05:45:36.850462+0000","last_clean":"2023-03-03T05:45:36.850462+0000","last_became_active":"2023-03-03T05:45:35.861052+0000","last_became_peered":"2023-03-03T05:45:35.861052+0000","last_unstale":"2023-03-03T05:45:36.850462+0000","last_undegraded":"2023-03-03T05:45:36.850462+0000","last_fullsized":"2023-03-03T05:45:36.850462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.282303+0000","last_change":"2023-03-03T05:45:34.282303+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.282303+0000","last_undegraded":"2023-03-03T05:45:34.282303+0000","last_fullsized":"2023-03-03T05:45:34.282303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:54.389717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850392+0000","last_change":"2023-03-03T05:45:35.857442+0000","last_active":"2023-03-03T05:45:36.850392+0000","last_peered":"2023-03-03T05:45:36.850392+0000","last_clean":"2023-03-03T05:45:36.850392+0000","last_became_active":"2023-03-03T05:45:35.857130+0000","last_became_peered":"2023-03-03T05:45:35.857130+0000","last_unstale":"2023-03-03T05:45:36.850392+0000","last_undegraded":"2023-03-03T05:45:36.850392+0000","last_fullsized":"2023-03-03T05:45:36.850392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.283265+0000","last_change":"2023-03-03T05:45:34.283265+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.283265+0000","last_undegraded":"2023-03-03T05:45:34.283265+0000","last_fullsized":"2023-03-03T05:45:34.283265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2023-03-03T05:45:34.283867+0000","last_change":"2023-03-03T05:45:34.283867+0000","last_active":"2023-03-03T05:45:34.268835+0000","last_peered":"2023-03-03T05:45:34.268835+0000","last_clean":"2023-03-03T05:45:34.268835+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:34.283867+0000","last_undegraded":"2023-03-03T05:45:34.283867+0000","last_fullsized":"2023-03-03T05:45:34.283867+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850763+0000","last_change":"2023-03-03T05:45:35.850670+0000","last_active":"2023-03-03T05:45:36.850763+0000","last_peered":"2023-03-03T05:45:36.850763+0000","last_clean":"2023-03-03T05:45:36.850763+0000","last_became_active":"2023-03-03T05:45:35.850380+0000","last_became_peered":"2023-03-03T05:45:35.850380+0000","last_unstale":"2023-03-03T05:45:36.850763+0000","last_undegraded":"2023-03-03T05:45:36.850763+0000","last_fullsized":"2023-03-03T05:45:36.850763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.870786+0000","last_change":"2023-03-03T05:45:35.860508+0000","last_active":"2023-03-03T05:45:35.870786+0000","last_peered":"2023-03-03T05:45:35.870786+0000","last_clean":"2023-03-03T05:45:35.870786+0000","last_became_active":"2023-03-03T05:45:35.860137+0000","last_became_peered":"2023-03-03T05:45:35.860137+0000","last_unstale":"2023-03-03T05:45:35.870786+0000","last_undegraded":"2023-03-03T05:45:35.870786+0000","last_fullsized":"2023-03-03T05:45:35.870786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-03-03T05:45:35.277217+0000","last_change":"2023-03-03T05:45:35.277217+0000","last_active":"2023-03-03T05:45:35.277217+0000","last_peered":"2023-03-03T05:45:35.277217+0000","last_clean":"2023-03-03T05:45:35.277217+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-03-03T05:45:35.277217+0000","last_undegraded":"2023-03-03T05:45:35.277217+0000","last_fullsized":"2023-03-03T05:45:35.277217+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":28,"acting":28,"num_store_stats":2},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7972,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363868,"statfs":{"total":96636764160,"available":96628600832,"internally_reserved":0,"allocated":225280,"data_stored":82106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7820,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94364020,"statfs":{"total":96636764160,"available":96628756480,"internally_reserved":0,"allocated":200704,"data_stored":68350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8912,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362928,"statfs":{"total":96636764160,"available":96627638272,"internally_reserved":0,"allocated":794624,"data_stored":658718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7808,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94364032,"statfs":{"total":96636764160,"available":96628768768,"internally_reserved":0,"allocated":188416,"data_stored":62180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8896,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362944,"statfs":{"total":96636764160,"available":96627654656,"internally_reserved":0,"allocated":778240,"data_stored":651255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7820,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94364020,"statfs":{"total":96636764160,"available":96628756480,"internally_reserved":0,"allocated":200704,"data_stored":68350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7808,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94364032,"statfs":{"total":96636764160,"available":96628768768,"internally_reserved":0,"allocated":188416,"data_stored":62180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T05:45:41.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T05:45:42.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.159+0000 7fe2e359d700 1 Processor -- start 2023-03-03T05:45:42.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.159+0000 7fe2e359d700 1 -- start start 2023-03-03T05:45:42.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.160+0000 7fe2e359d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2dc1564b0 0x7fe2dc0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e359d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0c9ca0 con 0x7fe2dc0c9610 2023-03-03T05:45:42.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e359d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0b46f0 con 0x7fe2dc0cab50 2023-03-03T05:45:42.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e359d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc0b4920 con 0x7fe2dc1564b0 2023-03-03T05:45:42.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e259b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2dc1564b0 0x7fe2dc0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e259b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2dc1564b0 0x7fe2dc0cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36200/0 (socket says 172.21.15.102:36200) 2023-03-03T05:45:42.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e259b700 1 -- 172.21.15.102:0/1583588522 learned_addr learned my addr 172.21.15.102:0/1583588522 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:42.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e1d9a700 1 --1- 172.21.15.102:0/1583588522 >> v1:172.21.15.102:6789/0 conn(0x7fe2dc0c9610 0x7fe2dc0ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45116/0 (socket says 172.21.15.102:45116) 2023-03-03T05:45:42.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e259b700 1 -- 172.21.15.102:0/1583588522 >> v1:172.21.15.172:6789/0 conn(0x7fe2dc0cab50 legacy=0x7fe2dc1636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e259b700 1 -- 172.21.15.102:0/1583588522 >> v1:172.21.15.102:6789/0 conn(0x7fe2dc0c9610 legacy=0x7fe2dc0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e259b700 1 -- 172.21.15.102:0/1583588522 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0b4c00 con 0x7fe2dc1564b0 2023-03-03T05:45:42.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.161+0000 7fe2e1599700 1 -- 172.21.15.102:0/1583588522 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638180440 0 0) 0x7fe2dc0c9ca0 con 0x7fe2dc0c9610 2023-03-03T05:45:42.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.162+0000 7fe2e259b700 1 --2- 172.21.15.102:0/1583588522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc1564b0 0x7fe2dc0cb660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80098f0 tx=0x7fe2d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40aa432dd1c038d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.162+0000 7fe2e1599700 1 -- 172.21.15.102:0/1583588522 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800b8a0 con 0x7fe2dc1564b0 2023-03-03T05:45:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.162+0000 7fe2e1599700 1 -- 172.21.15.102:0/1583588522 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d800ba00 con 0x7fe2dc1564b0 2023-03-03T05:45:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.162+0000 7fe2e1599700 1 -- 172.21.15.102:0/1583588522 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800bd90 con 0x7fe2dc1564b0 2023-03-03T05:45:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.163+0000 7fe2e359d700 1 -- 172.21.15.102:0/1583588522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc1564b0 msgr2=0x7fe2dc0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.163+0000 7fe2e359d700 1 --2- 172.21.15.102:0/1583588522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc1564b0 0x7fe2dc0cb660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80098f0 tx=0x7fe2d8009c00 comp rx=0 tx=0).stop 2023-03-03T05:45:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.163+0000 7fe2e359d700 1 -- 172.21.15.102:0/1583588522 shutdown_connections 2023-03-03T05:45:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.163+0000 7fe2e359d700 1 --2- 172.21.15.102:0/1583588522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc1564b0 0x7fe2dc0cb660 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.163+0000 7fe2e359d700 1 -- 172.21.15.102:0/1583588522 >> 172.21.15.102:0/1583588522 conn(0x7fe2dc065fb0 msgr2=0x7fe2dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:42.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.163+0000 7fe2e359d700 1 -- 172.21.15.102:0/1583588522 shutdown_connections 2023-03-03T05:45:42.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.163+0000 7fe2e359d700 1 -- 172.21.15.102:0/1583588522 wait complete. 2023-03-03T05:45:42.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.165+0000 7fe2e359d700 1 Processor -- start 2023-03-03T05:45:42.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.165+0000 7fe2e359d700 1 -- start start 2023-03-03T05:45:42.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e359d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2dc0c9610 0x7fe2dc1f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e359d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc0cab50 0x7fe2dc15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e359d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2dc1564b0 0x7fe2dc14e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e359d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1f1940 con 0x7fe2dc0cab50 2023-03-03T05:45:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e359d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1f2890 con 0x7fe2dc1564b0 2023-03-03T05:45:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e359d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc159590 con 0x7fe2dc0c9610 2023-03-03T05:45:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e1d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc0cab50 0x7fe2dc15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e259b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2dc0c9610 0x7fe2dc1f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e1d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc0cab50 0x7fe2dc15b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36204/0 (socket says 172.21.15.102:36204) 2023-03-03T05:45:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e1d9a700 1 -- 172.21.15.102:0/216743835 learned_addr learned my addr 172.21.15.102:0/216743835 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.166+0000 7fe2e2d9c700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2dc1564b0 0x7fe2dc14e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e259b700 1 -- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2dc1564b0 msgr2=0x7fe2dc14e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e259b700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2dc1564b0 0x7fe2dc14e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e259b700 1 -- 172.21.15.102:0/216743835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc0cab50 msgr2=0x7fe2dc15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e259b700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc0cab50 0x7fe2dc15b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e259b700 1 -- 172.21.15.102:0/216743835 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d8009550 con 0x7fe2dc0c9610 2023-03-03T05:45:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e1d9a700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc0cab50 0x7fe2dc15b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e259b700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2dc0c9610 0x7fe2dc1f2350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8016000 tx=0x7fe2d8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2d2ffd700 1 -- 172.21.15.102:0/216743835 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d801a900 con 0x7fe2dc0c9610 2023-03-03T05:45:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2d2ffd700 1 -- 172.21.15.102:0/216743835 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d801f070 con 0x7fe2dc0c9610 2023-03-03T05:45:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2d2ffd700 1 -- 172.21.15.102:0/216743835 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d801aca0 con 0x7fe2dc0c9610 2023-03-03T05:45:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.167+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc0b2210 con 0x7fe2dc0c9610 2023-03-03T05:45:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.168+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc0b1f30 con 0x7fe2dc0c9610 2023-03-03T05:45:42.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.169+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc156880 con 0x7fe2dc0c9610 2023-03-03T05:45:42.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.170+0000 7fe2d2ffd700 1 -- 172.21.15.102:0/216743835 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2d800f050 con 0x7fe2dc0c9610 2023-03-03T05:45:42.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.171+0000 7fe2d2ffd700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2c0068320 0x7fe2c006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.171+0000 7fe2d2ffd700 1 -- 172.21.15.102:0/216743835 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe2d8021070 con 0x7fe2dc0c9610 2023-03-03T05:45:42.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.171+0000 7fe2e1d9a700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2c0068320 0x7fe2c006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.172+0000 7fe2e1d9a700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2c0068320 0x7fe2c006a7d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc007ca0 tx=0x7fe2cc007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:42.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.175+0000 7fe2d2ffd700 1 -- 172.21.15.102:0/216743835 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2d8053d70 con 0x7fe2dc0c9610 2023-03-03T05:45:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.366+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 --> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2dc0677f0 con 0x7fe2c0068320 2023-03-03T05:45:42.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.375+0000 7fe2d2ffd700 1 -- 172.21.15.102:0/216743835 <== mgr.4105 v2:172.21.15.172:6832/114058 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286699 (secure 0 0 0) 0x7fe2dc0677f0 con 0x7fe2c0068320 2023-03-03T05:45:42.379 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:42.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2c0068320 msgr2=0x7fe2c006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2c0068320 0x7fe2c006a7d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc007ca0 tx=0x7fe2cc007bf0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2dc0c9610 msgr2=0x7fe2dc1f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2dc0c9610 0x7fe2dc1f2350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8016000 tx=0x7fe2d8004750 comp rx=0 tx=0).stop 2023-03-03T05:45:42.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 shutdown_connections 2023-03-03T05:45:42.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2c0068320 0x7fe2c006a7d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2dc0c9610 0x7fe2dc1f2350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2dc1564b0 0x7fe2dc14e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 --2- 172.21.15.102:0/216743835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2dc0cab50 0x7fe2dc15b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 >> 172.21.15.102:0/216743835 conn(0x7fe2dc065fb0 msgr2=0x7fe2dc155660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 shutdown_connections 2023-03-03T05:45:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.380+0000 7fe2e359d700 1 -- 172.21.15.102:0/216743835 wait complete. 2023-03-03T05:45:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2023-03-03T05:45:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-03-03T05:45:41.943851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590480,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":164,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66128,"kb_used_data":3024,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":754908592,"statfs":{"total":773094113280,"available":773026398208,"internally_reserved":0,"allocated":3096576,"data_stored":1889661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.662444"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922943+0000","last_change":"2023-03-03T05:45:35.854816+0000","last_active":"2023-03-03T05:45:36.922943+0000","last_peered":"2023-03-03T05:45:36.922943+0000","last_clean":"2023-03-03T05:45:36.922943+0000","last_became_active":"2023-03-03T05:45:35.854447+0000","last_became_peered":"2023-03-03T05:45:35.854447+0000","last_unstale":"2023-03-03T05:45:36.922943+0000","last_undegraded":"2023-03-03T05:45:36.922943+0000","last_fullsized":"2023-03-03T05:45:36.922943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850859+0000","last_change":"2023-03-03T05:45:35.862354+0000","last_active":"2023-03-03T05:45:36.850859+0000","last_peered":"2023-03-03T05:45:36.850859+0000","last_clean":"2023-03-03T05:45:36.850859+0000","last_became_active":"2023-03-03T05:45:35.862016+0000","last_became_peered":"2023-03-03T05:45:35.862016+0000","last_unstale":"2023-03-03T05:45:36.850859+0000","last_undegraded":"2023-03-03T05:45:36.850859+0000","last_fullsized":"2023-03-03T05:45:36.850859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962731+0000","last_change":"2023-03-03T05:45:35.864429+0000","last_active":"2023-03-03T05:45:35.962731+0000","last_peered":"2023-03-03T05:45:35.962731+0000","last_clean":"2023-03-03T05:45:35.962731+0000","last_became_active":"2023-03-03T05:45:35.864091+0000","last_became_peered":"2023-03-03T05:45:35.864091+0000","last_unstale":"2023-03-03T05:45:35.962731+0000","last_undegraded":"2023-03-03T05:45:35.962731+0000","last_fullsized":"2023-03-03T05:45:35.962731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962683+0000","last_change":"2023-03-03T05:45:35.861374+0000","last_active":"2023-03-03T05:45:35.962683+0000","last_peered":"2023-03-03T05:45:35.962683+0000","last_clean":"2023-03-03T05:45:35.962683+0000","last_became_active":"2023-03-03T05:45:35.860993+0000","last_became_peered":"2023-03-03T05:45:35.860993+0000","last_unstale":"2023-03-03T05:45:35.962683+0000","last_undegraded":"2023-03-03T05:45:35.962683+0000","last_fullsized":"2023-03-03T05:45:35.962683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853383+0000","last_change":"2023-03-03T05:45:35.857703+0000","last_active":"2023-03-03T05:45:36.853383+0000","last_peered":"2023-03-03T05:45:36.853383+0000","last_clean":"2023-03-03T05:45:36.853383+0000","last_became_active":"2023-03-03T05:45:35.857421+0000","last_became_peered":"2023-03-03T05:45:35.857421+0000","last_unstale":"2023-03-03T05:45:36.853383+0000","last_undegraded":"2023-03-03T05:45:36.853383+0000","last_fullsized":"2023-03-03T05:45:36.853383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851248+0000","last_change":"2023-03-03T05:45:35.859892+0000","last_active":"2023-03-03T05:45:36.851248+0000","last_peered":"2023-03-03T05:45:36.851248+0000","last_clean":"2023-03-03T05:45:36.851248+0000","last_became_active":"2023-03-03T05:45:35.859617+0000","last_became_peered":"2023-03-03T05:45:35.859617+0000","last_unstale":"2023-03-03T05:45:36.851248+0000","last_undegraded":"2023-03-03T05:45:36.851248+0000","last_fullsized":"2023-03-03T05:45:36.851248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:23:53.405094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850434+0000","last_change":"2023-03-03T05:45:35.862939+0000","last_active":"2023-03-03T05:45:36.850434+0000","last_peered":"2023-03-03T05:45:36.850434+0000","last_clean":"2023-03-03T05:45:36.850434+0000","last_became_active":"2023-03-03T05:45:35.862681+0000","last_became_peered":"2023-03-03T05:45:35.862681+0000","last_unstale":"2023-03-03T05:45:36.850434+0000","last_undegraded":"2023-03-03T05:45:36.850434+0000","last_fullsized":"2023-03-03T05:45:36.850434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:58.535750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923559+0000","last_change":"2023-03-03T05:45:35.866302+0000","last_active":"2023-03-03T05:45:36.923559+0000","last_peered":"2023-03-03T05:45:36.923559+0000","last_clean":"2023-03-03T05:45:36.923559+0000","last_became_active":"2023-03-03T05:45:35.865627+0000","last_became_peered":"2023-03-03T05:45:35.865627+0000","last_unstale":"2023-03-03T05:45:36.923559+0000","last_undegraded":"2023-03-03T05:45:36.923559+0000","last_fullsized":"2023-03-03T05:45:36.923559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853588+0000","last_change":"2023-03-03T05:45:35.856762+0000","last_active":"2023-03-03T05:45:36.853588+0000","last_peered":"2023-03-03T05:45:36.853588+0000","last_clean":"2023-03-03T05:45:36.853588+0000","last_became_active":"2023-03-03T05:45:35.856458+0000","last_became_peered":"2023-03-03T05:45:35.856458+0000","last_unstale":"2023-03-03T05:45:36.853588+0000","last_undegraded":"2023-03-03T05:45:36.853588+0000","last_fullsized":"2023-03-03T05:45:36.853588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923093+0000","last_change":"2023-03-03T05:45:35.865258+0000","last_active":"2023-03-03T05:45:36.923093+0000","last_peered":"2023-03-03T05:45:36.923093+0000","last_clean":"2023-03-03T05:45:36.923093+0000","last_became_active":"2023-03-03T05:45:35.864889+0000","last_became_peered":"2023-03-03T05:45:35.864889+0000","last_unstale":"2023-03-03T05:45:36.923093+0000","last_undegraded":"2023-03-03T05:45:36.923093+0000","last_fullsized":"2023-03-03T05:45:36.923093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923368+0000","last_change":"2023-03-03T05:45:35.868970+0000","last_active":"2023-03-03T05:45:36.923368+0000","last_peered":"2023-03-03T05:45:36.923368+0000","last_clean":"2023-03-03T05:45:36.923368+0000","last_became_active":"2023-03-03T05:45:35.868555+0000","last_became_peered":"2023-03-03T05:45:35.868555+0000","last_unstale":"2023-03-03T05:45:36.923368+0000","last_undegraded":"2023-03-03T05:45:36.923368+0000","last_fullsized":"2023-03-03T05:45:36.923368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922618+0000","last_change":"2023-03-03T05:45:35.855291+0000","last_active":"2023-03-03T05:45:36.922618+0000","last_peered":"2023-03-03T05:45:36.922618+0000","last_clean":"2023-03-03T05:45:36.922618+0000","last_became_active":"2023-03-03T05:45:35.854925+0000","last_became_peered":"2023-03-03T05:45:35.854925+0000","last_unstale":"2023-03-03T05:45:36.922618+0000","last_undegraded":"2023-03-03T05:45:36.922618+0000","last_fullsized":"2023-03-03T05:45:36.922618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.924177+0000","last_change":"2023-03-03T05:45:35.853479+0000","last_active":"2023-03-03T05:45:36.924177+0000","last_peered":"2023-03-03T05:45:36.924177+0000","last_clean":"2023-03-03T05:45:36.924177+0000","last_became_active":"2023-03-03T05:45:35.853193+0000","last_became_peered":"2023-03-03T05:45:35.853193+0000","last_unstale":"2023-03-03T05:45:36.924177+0000","last_undegraded":"2023-03-03T05:45:36.924177+0000","last_fullsized":"2023-03-03T05:45:36.924177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923036+0000","last_change":"2023-03-03T05:45:35.865976+0000","last_active":"2023-03-03T05:45:36.923036+0000","last_peered":"2023-03-03T05:45:36.923036+0000","last_clean":"2023-03-03T05:45:36.923036+0000","last_became_active":"2023-03-03T05:45:35.865561+0000","last_became_peered":"2023-03-03T05:45:35.865561+0000","last_unstale":"2023-03-03T05:45:36.923036+0000","last_undegraded":"2023-03-03T05:45:36.923036+0000","last_fullsized":"2023-03-03T05:45:36.923036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962602+0000","last_change":"2023-03-03T05:45:35.862593+0000","last_active":"2023-03-03T05:45:35.962602+0000","last_peered":"2023-03-03T05:45:35.962602+0000","last_clean":"2023-03-03T05:45:35.962602+0000","last_became_active":"2023-03-03T05:45:35.862342+0000","last_became_peered":"2023-03-03T05:45:35.862342+0000","last_unstale":"2023-03-03T05:45:35.962602+0000","last_undegraded":"2023-03-03T05:45:35.962602+0000","last_fullsized":"2023-03-03T05:45:35.962602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850467+0000","last_change":"2023-03-03T05:45:35.857623+0000","last_active":"2023-03-03T05:45:36.850467+0000","last_peered":"2023-03-03T05:45:36.850467+0000","last_clean":"2023-03-03T05:45:36.850467+0000","last_became_active":"2023-03-03T05:45:35.857357+0000","last_became_peered":"2023-03-03T05:45:35.857357+0000","last_unstale":"2023-03-03T05:45:36.850467+0000","last_undegraded":"2023-03-03T05:45:36.850467+0000","last_fullsized":"2023-03-03T05:45:36.850467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:00.850961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923102+0000","last_change":"2023-03-03T05:45:35.859115+0000","last_active":"2023-03-03T05:45:36.923102+0000","last_peered":"2023-03-03T05:45:36.923102+0000","last_clean":"2023-03-03T05:45:36.923102+0000","last_became_active":"2023-03-03T05:45:35.858794+0000","last_became_peered":"2023-03-03T05:45:35.858794+0000","last_unstale":"2023-03-03T05:45:36.923102+0000","last_undegraded":"2023-03-03T05:45:36.923102+0000","last_fullsized":"2023-03-03T05:45:36.923102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962764+0000","last_change":"2023-03-03T05:45:35.861090+0000","last_active":"2023-03-03T05:45:35.962764+0000","last_peered":"2023-03-03T05:45:35.962764+0000","last_clean":"2023-03-03T05:45:35.962764+0000","last_became_active":"2023-03-03T05:45:35.860814+0000","last_became_peered":"2023-03-03T05:45:35.860814+0000","last_unstale":"2023-03-03T05:45:35.962764+0000","last_undegraded":"2023-03-03T05:45:35.962764+0000","last_fullsized":"2023-03-03T05:45:35.962764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923139+0000","last_change":"2023-03-03T05:45:35.855611+0000","last_active":"2023-03-03T05:45:36.923139+0000","last_peered":"2023-03-03T05:45:36.923139+0000","last_clean":"2023-03-03T05:45:36.923139+0000","last_became_active":"2023-03-03T05:45:35.855333+0000","last_became_peered":"2023-03-03T05:45:35.855333+0000","last_unstale":"2023-03-03T05:45:36.923139+0000","last_undegraded":"2023-03-03T05:45:36.923139+0000","last_fullsized":"2023-03-03T05:45:36.923139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923216+0000","last_change":"2023-03-03T05:45:35.865135+0000","last_active":"2023-03-03T05:45:36.923216+0000","last_peered":"2023-03-03T05:45:36.923216+0000","last_clean":"2023-03-03T05:45:36.923216+0000","last_became_active":"2023-03-03T05:45:35.864823+0000","last_became_peered":"2023-03-03T05:45:35.864823+0000","last_unstale":"2023-03-03T05:45:36.923216+0000","last_undegraded":"2023-03-03T05:45:36.923216+0000","last_fullsized":"2023-03-03T05:45:36.923216+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923121+0000","last_change":"2023-03-03T05:45:35.864573+0000","last_active":"2023-03-03T05:45:36.923121+0000","last_peered":"2023-03-03T05:45:36.923121+0000","last_clean":"2023-03-03T05:45:36.923121+0000","last_became_active":"2023-03-03T05:45:35.864297+0000","last_became_peered":"2023-03-03T05:45:35.864297+0000","last_unstale":"2023-03-03T05:45:36.923121+0000","last_undegraded":"2023-03-03T05:45:36.923121+0000","last_fullsized":"2023-03-03T05:45:36.923121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922982+0000","last_change":"2023-03-03T05:45:35.862720+0000","last_active":"2023-03-03T05:45:36.922982+0000","last_peered":"2023-03-03T05:45:36.922982+0000","last_clean":"2023-03-03T05:45:36.922982+0000","last_became_active":"2023-03-03T05:45:35.861282+0000","last_became_peered":"2023-03-03T05:45:35.861282+0000","last_unstale":"2023-03-03T05:45:36.922982+0000","last_undegraded":"2023-03-03T05:45:36.922982+0000","last_fullsized":"2023-03-03T05:45:36.922982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853407+0000","last_change":"2023-03-03T05:45:35.859828+0000","last_active":"2023-03-03T05:45:36.853407+0000","last_peered":"2023-03-03T05:45:36.853407+0000","last_clean":"2023-03-03T05:45:36.853407+0000","last_became_active":"2023-03-03T05:45:35.859568+0000","last_became_peered":"2023-03-03T05:45:35.859568+0000","last_unstale":"2023-03-03T05:45:36.853407+0000","last_undegraded":"2023-03-03T05:45:36.853407+0000","last_fullsized":"2023-03-03T05:45:36.853407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923461+0000","last_change":"2023-03-03T05:45:35.864435+0000","last_active":"2023-03-03T05:45:36.923461+0000","last_peered":"2023-03-03T05:45:36.923461+0000","last_clean":"2023-03-03T05:45:36.923461+0000","last_became_active":"2023-03-03T05:45:35.864129+0000","last_became_peered":"2023-03-03T05:45:35.864129+0000","last_unstale":"2023-03-03T05:45:36.923461+0000","last_undegraded":"2023-03-03T05:45:36.923461+0000","last_fullsized":"2023-03-03T05:45:36.923461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853682+0000","last_change":"2023-03-03T05:45:35.855909+0000","last_active":"2023-03-03T05:45:36.853682+0000","last_peered":"2023-03-03T05:45:36.853682+0000","last_clean":"2023-03-03T05:45:36.853682+0000","last_became_active":"2023-03-03T05:45:35.855388+0000","last_became_peered":"2023-03-03T05:45:35.855388+0000","last_unstale":"2023-03-03T05:45:36.853682+0000","last_undegraded":"2023-03-03T05:45:36.853682+0000","last_fullsized":"2023-03-03T05:45:36.853682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923513+0000","last_change":"2023-03-03T05:45:35.863167+0000","last_active":"2023-03-03T05:45:36.923513+0000","last_peered":"2023-03-03T05:45:36.923513+0000","last_clean":"2023-03-03T05:45:36.923513+0000","last_became_active":"2023-03-03T05:45:35.862519+0000","last_became_peered":"2023-03-03T05:45:35.862519+0000","last_unstale":"2023-03-03T05:45:36.923513+0000","last_undegraded":"2023-03-03T05:45:36.923513+0000","last_fullsized":"2023-03-03T05:45:36.923513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851022+0000","last_change":"2023-03-03T05:45:35.870150+0000","last_active":"2023-03-03T05:45:36.851022+0000","last_peered":"2023-03-03T05:45:36.851022+0000","last_clean":"2023-03-03T05:45:36.851022+0000","last_became_active":"2023-03-03T05:45:35.868786+0000","last_became_peered":"2023-03-03T05:45:35.868786+0000","last_unstale":"2023-03-03T05:45:36.851022+0000","last_undegraded":"2023-03-03T05:45:36.851022+0000","last_fullsized":"2023-03-03T05:45:36.851022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:52:23.931405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.855012+0000","last_change":"2023-03-03T05:45:35.310449+0000","last_active":"2023-03-03T05:45:35.855012+0000","last_peered":"2023-03-03T05:45:35.855012+0000","last_clean":"2023-03-03T05:45:35.855012+0000","last_became_active":"2023-03-03T05:45:35.310197+0000","last_became_peered":"2023-03-03T05:45:35.310197+0000","last_unstale":"2023-03-03T05:45:35.855012+0000","last_undegraded":"2023-03-03T05:45:35.855012+0000","last_fullsized":"2023-03-03T05:45:35.855012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923377+0000","last_change":"2023-03-03T05:45:35.861444+0000","last_active":"2023-03-03T05:45:36.923377+0000","last_peered":"2023-03-03T05:45:36.923377+0000","last_clean":"2023-03-03T05:45:36.923377+0000","last_became_active":"2023-03-03T05:45:35.861129+0000","last_became_peered":"2023-03-03T05:45:35.861129+0000","last_unstale":"2023-03-03T05:45:36.923377+0000","last_undegraded":"2023-03-03T05:45:36.923377+0000","last_fullsized":"2023-03-03T05:45:36.923377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843345+0000","last_change":"2023-03-03T05:45:35.308492+0000","last_active":"2023-03-03T05:45:35.843345+0000","last_peered":"2023-03-03T05:45:35.843345+0000","last_clean":"2023-03-03T05:45:35.843345+0000","last_became_active":"2023-03-03T05:45:35.308276+0000","last_became_peered":"2023-03-03T05:45:35.308276+0000","last_unstale":"2023-03-03T05:45:35.843345+0000","last_undegraded":"2023-03-03T05:45:35.843345+0000","last_fullsized":"2023-03-03T05:45:35.843345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:32.932001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853620+0000","last_change":"2023-03-03T05:45:35.866408+0000","last_active":"2023-03-03T05:45:36.853620+0000","last_peered":"2023-03-03T05:45:36.853620+0000","last_clean":"2023-03-03T05:45:36.853620+0000","last_became_active":"2023-03-03T05:45:35.866080+0000","last_became_peered":"2023-03-03T05:45:35.866080+0000","last_unstale":"2023-03-03T05:45:36.853620+0000","last_undegraded":"2023-03-03T05:45:36.853620+0000","last_fullsized":"2023-03-03T05:45:36.853620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:22.678556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848138+0000","last_change":"2023-03-03T05:45:35.293990+0000","last_active":"2023-03-03T05:45:35.848138+0000","last_peered":"2023-03-03T05:45:35.848138+0000","last_clean":"2023-03-03T05:45:35.848138+0000","last_became_active":"2023-03-03T05:45:35.293655+0000","last_became_peered":"2023-03-03T05:45:35.293655+0000","last_unstale":"2023-03-03T05:45:35.848138+0000","last_undegraded":"2023-03-03T05:45:35.848138+0000","last_fullsized":"2023-03-03T05:45:35.848138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850700+0000","last_change":"2023-03-03T05:45:35.852599+0000","last_active":"2023-03-03T05:45:36.850700+0000","last_peered":"2023-03-03T05:45:36.850700+0000","last_clean":"2023-03-03T05:45:36.850700+0000","last_became_active":"2023-03-03T05:45:35.852292+0000","last_became_peered":"2023-03-03T05:45:35.852292+0000","last_unstale":"2023-03-03T05:45:36.850700+0000","last_undegraded":"2023-03-03T05:45:36.850700+0000","last_fullsized":"2023-03-03T05:45:36.850700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:06.104140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848641+0000","last_change":"2023-03-03T05:45:35.297718+0000","last_active":"2023-03-03T05:45:35.848641+0000","last_peered":"2023-03-03T05:45:35.848641+0000","last_clean":"2023-03-03T05:45:35.848641+0000","last_became_active":"2023-03-03T05:45:35.297435+0000","last_became_peered":"2023-03-03T05:45:35.297435+0000","last_unstale":"2023-03-03T05:45:35.848641+0000","last_undegraded":"2023-03-03T05:45:35.848641+0000","last_fullsized":"2023-03-03T05:45:35.848641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:17:39.778447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923577+0000","last_change":"2023-03-03T05:45:35.871116+0000","last_active":"2023-03-03T05:45:36.923577+0000","last_peered":"2023-03-03T05:45:36.923577+0000","last_clean":"2023-03-03T05:45:36.923577+0000","last_became_active":"2023-03-03T05:45:35.870843+0000","last_became_peered":"2023-03-03T05:45:35.870843+0000","last_unstale":"2023-03-03T05:45:36.923577+0000","last_undegraded":"2023-03-03T05:45:36.923577+0000","last_fullsized":"2023-03-03T05:45:36.923577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851170+0000","last_change":"2023-03-03T05:45:35.304400+0000","last_active":"2023-03-03T05:45:36.851170+0000","last_peered":"2023-03-03T05:45:36.851170+0000","last_clean":"2023-03-03T05:45:36.851170+0000","last_became_active":"2023-03-03T05:45:35.304093+0000","last_became_peered":"2023-03-03T05:45:35.304093+0000","last_unstale":"2023-03-03T05:45:36.851170+0000","last_undegraded":"2023-03-03T05:45:36.851170+0000","last_fullsized":"2023-03-03T05:45:36.851170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853210+0000","last_change":"2023-03-03T05:45:35.862492+0000","last_active":"2023-03-03T05:45:36.853210+0000","last_peered":"2023-03-03T05:45:36.853210+0000","last_clean":"2023-03-03T05:45:36.853210+0000","last_became_active":"2023-03-03T05:45:35.862173+0000","last_became_peered":"2023-03-03T05:45:35.862173+0000","last_unstale":"2023-03-03T05:45:36.853210+0000","last_undegraded":"2023-03-03T05:45:36.853210+0000","last_fullsized":"2023-03-03T05:45:36.853210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.853604+0000","last_change":"2023-03-03T05:45:35.302282+0000","last_active":"2023-03-03T05:45:40.853604+0000","last_peered":"2023-03-03T05:45:40.853604+0000","last_clean":"2023-03-03T05:45:40.853604+0000","last_became_active":"2023-03-03T05:45:35.302002+0000","last_became_peered":"2023-03-03T05:45:35.302002+0000","last_unstale":"2023-03-03T05:45:40.853604+0000","last_undegraded":"2023-03-03T05:45:40.853604+0000","last_fullsized":"2023-03-03T05:45:40.853604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853414+0000","last_change":"2023-03-03T05:45:35.855928+0000","last_active":"2023-03-03T05:45:36.853414+0000","last_peered":"2023-03-03T05:45:36.853414+0000","last_clean":"2023-03-03T05:45:36.853414+0000","last_became_active":"2023-03-03T05:45:35.855612+0000","last_became_peered":"2023-03-03T05:45:35.855612+0000","last_unstale":"2023-03-03T05:45:36.853414+0000","last_undegraded":"2023-03-03T05:45:36.853414+0000","last_fullsized":"2023-03-03T05:45:36.853414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843369+0000","last_change":"2023-03-03T05:45:35.304856+0000","last_active":"2023-03-03T05:45:35.843369+0000","last_peered":"2023-03-03T05:45:35.843369+0000","last_clean":"2023-03-03T05:45:35.843369+0000","last_became_active":"2023-03-03T05:45:35.304439+0000","last_became_peered":"2023-03-03T05:45:35.304439+0000","last_unstale":"2023-03-03T05:45:35.843369+0000","last_undegraded":"2023-03-03T05:45:35.843369+0000","last_fullsized":"2023-03-03T05:45:35.843369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:30.692527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850547+0000","last_change":"2023-03-03T05:45:35.856887+0000","last_active":"2023-03-03T05:45:36.850547+0000","last_peered":"2023-03-03T05:45:36.850547+0000","last_clean":"2023-03-03T05:45:36.850547+0000","last_became_active":"2023-03-03T05:45:35.856498+0000","last_became_peered":"2023-03-03T05:45:35.856498+0000","last_unstale":"2023-03-03T05:45:36.850547+0000","last_undegraded":"2023-03-03T05:45:36.850547+0000","last_fullsized":"2023-03-03T05:45:36.850547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843003+0000","last_change":"2023-03-03T05:45:35.297635+0000","last_active":"2023-03-03T05:45:35.843003+0000","last_peered":"2023-03-03T05:45:35.843003+0000","last_clean":"2023-03-03T05:45:35.843003+0000","last_became_active":"2023-03-03T05:45:35.297373+0000","last_became_peered":"2023-03-03T05:45:35.297373+0000","last_unstale":"2023-03-03T05:45:35.843003+0000","last_undegraded":"2023-03-03T05:45:35.843003+0000","last_fullsized":"2023-03-03T05:45:35.843003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:48.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922855+0000","last_change":"2023-03-03T05:45:35.852113+0000","last_active":"2023-03-03T05:45:36.922855+0000","last_peered":"2023-03-03T05:45:36.922855+0000","last_clean":"2023-03-03T05:45:36.922855+0000","last_became_active":"2023-03-03T05:45:35.851799+0000","last_became_peered":"2023-03-03T05:45:35.851799+0000","last_unstale":"2023-03-03T05:45:36.922855+0000","last_undegraded":"2023-03-03T05:45:36.922855+0000","last_fullsized":"2023-03-03T05:45:36.922855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.845615+0000","last_change":"2023-03-03T05:45:35.292083+0000","last_active":"2023-03-03T05:45:35.845615+0000","last_peered":"2023-03-03T05:45:35.845615+0000","last_clean":"2023-03-03T05:45:35.845615+0000","last_became_active":"2023-03-03T05:45:35.291766+0000","last_became_peered":"2023-03-03T05:45:35.291766+0000","last_unstale":"2023-03-03T05:45:35.845615+0000","last_undegraded":"2023-03-03T05:45:35.845615+0000","last_fullsized":"2023-03-03T05:45:35.845615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851348+0000","last_change":"2023-03-03T05:45:35.870363+0000","last_active":"2023-03-03T05:45:36.851348+0000","last_peered":"2023-03-03T05:45:36.851348+0000","last_clean":"2023-03-03T05:45:36.851348+0000","last_became_active":"2023-03-03T05:45:35.868801+0000","last_became_peered":"2023-03-03T05:45:35.868801+0000","last_unstale":"2023-03-03T05:45:36.851348+0000","last_undegraded":"2023-03-03T05:45:36.851348+0000","last_fullsized":"2023-03-03T05:45:36.851348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:04.045341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844327+0000","last_change":"2023-03-03T05:45:35.304592+0000","last_active":"2023-03-03T05:45:35.844327+0000","last_peered":"2023-03-03T05:45:35.844327+0000","last_clean":"2023-03-03T05:45:35.844327+0000","last_became_active":"2023-03-03T05:45:35.304318+0000","last_became_peered":"2023-03-03T05:45:35.304318+0000","last_unstale":"2023-03-03T05:45:35.844327+0000","last_undegraded":"2023-03-03T05:45:35.844327+0000","last_fullsized":"2023-03-03T05:45:35.844327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:08.368039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922933+0000","last_change":"2023-03-03T05:45:35.857951+0000","last_active":"2023-03-03T05:45:36.922933+0000","last_peered":"2023-03-03T05:45:36.922933+0000","last_clean":"2023-03-03T05:45:36.922933+0000","last_became_active":"2023-03-03T05:45:35.857581+0000","last_became_peered":"2023-03-03T05:45:35.857581+0000","last_unstale":"2023-03-03T05:45:36.922933+0000","last_undegraded":"2023-03-03T05:45:36.922933+0000","last_fullsized":"2023-03-03T05:45:36.922933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.858197+0000","last_change":"2023-03-03T05:45:35.309150+0000","last_active":"2023-03-03T05:45:35.858197+0000","last_peered":"2023-03-03T05:45:35.858197+0000","last_clean":"2023-03-03T05:45:35.858197+0000","last_became_active":"2023-03-03T05:45:35.308840+0000","last_became_peered":"2023-03-03T05:45:35.308840+0000","last_unstale":"2023-03-03T05:45:35.858197+0000","last_undegraded":"2023-03-03T05:45:35.858197+0000","last_fullsized":"2023-03-03T05:45:35.858197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923050+0000","last_change":"2023-03-03T05:45:35.864919+0000","last_active":"2023-03-03T05:45:36.923050+0000","last_peered":"2023-03-03T05:45:36.923050+0000","last_clean":"2023-03-03T05:45:36.923050+0000","last_became_active":"2023-03-03T05:45:35.864595+0000","last_became_peered":"2023-03-03T05:45:35.864595+0000","last_unstale":"2023-03-03T05:45:36.923050+0000","last_undegraded":"2023-03-03T05:45:36.923050+0000","last_fullsized":"2023-03-03T05:45:36.923050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.308976+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.308707+0000","last_became_peered":"2023-03-03T05:45:35.308707+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850640+0000","last_change":"2023-03-03T05:45:35.868861+0000","last_active":"2023-03-03T05:45:36.850640+0000","last_peered":"2023-03-03T05:45:36.850640+0000","last_clean":"2023-03-03T05:45:36.850640+0000","last_became_active":"2023-03-03T05:45:35.868438+0000","last_became_peered":"2023-03-03T05:45:35.868438+0000","last_unstale":"2023-03-03T05:45:36.850640+0000","last_undegraded":"2023-03-03T05:45:36.850640+0000","last_fullsized":"2023-03-03T05:45:36.850640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:59.810575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851097+0000","last_change":"2023-03-03T05:45:35.303791+0000","last_active":"2023-03-03T05:45:36.851097+0000","last_peered":"2023-03-03T05:45:36.851097+0000","last_clean":"2023-03-03T05:45:36.851097+0000","last_became_active":"2023-03-03T05:45:35.303482+0000","last_became_peered":"2023-03-03T05:45:35.303482+0000","last_unstale":"2023-03-03T05:45:36.851097+0000","last_undegraded":"2023-03-03T05:45:36.851097+0000","last_fullsized":"2023-03-03T05:45:36.851097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:31.933113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923037+0000","last_change":"2023-03-03T05:45:35.864783+0000","last_active":"2023-03-03T05:45:36.923037+0000","last_peered":"2023-03-03T05:45:36.923037+0000","last_clean":"2023-03-03T05:45:36.923037+0000","last_became_active":"2023-03-03T05:45:35.864462+0000","last_became_peered":"2023-03-03T05:45:35.864462+0000","last_unstale":"2023-03-03T05:45:36.923037+0000","last_undegraded":"2023-03-03T05:45:36.923037+0000","last_fullsized":"2023-03-03T05:45:36.923037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844359+0000","last_change":"2023-03-03T05:45:35.310813+0000","last_active":"2023-03-03T05:45:35.844359+0000","last_peered":"2023-03-03T05:45:35.844359+0000","last_clean":"2023-03-03T05:45:35.844359+0000","last_became_active":"2023-03-03T05:45:35.310466+0000","last_became_peered":"2023-03-03T05:45:35.310466+0000","last_unstale":"2023-03-03T05:45:35.844359+0000","last_undegraded":"2023-03-03T05:45:35.844359+0000","last_fullsized":"2023-03-03T05:45:35.844359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:46.322139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850472+0000","last_change":"2023-03-03T05:45:35.856376+0000","last_active":"2023-03-03T05:45:36.850472+0000","last_peered":"2023-03-03T05:45:36.850472+0000","last_clean":"2023-03-03T05:45:36.850472+0000","last_became_active":"2023-03-03T05:45:35.856064+0000","last_became_peered":"2023-03-03T05:45:35.856064+0000","last_unstale":"2023-03-03T05:45:36.850472+0000","last_undegraded":"2023-03-03T05:45:36.850472+0000","last_fullsized":"2023-03-03T05:45:36.850472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962714+0000","last_change":"2023-03-03T05:45:35.295520+0000","last_active":"2023-03-03T05:45:35.962714+0000","last_peered":"2023-03-03T05:45:35.962714+0000","last_clean":"2023-03-03T05:45:35.962714+0000","last_became_active":"2023-03-03T05:45:35.295203+0000","last_became_peered":"2023-03-03T05:45:35.295203+0000","last_unstale":"2023-03-03T05:45:35.962714+0000","last_undegraded":"2023-03-03T05:45:35.962714+0000","last_fullsized":"2023-03-03T05:45:35.962714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923389+0000","last_change":"2023-03-03T05:45:35.852242+0000","last_active":"2023-03-03T05:45:36.923389+0000","last_peered":"2023-03-03T05:45:36.923389+0000","last_clean":"2023-03-03T05:45:36.923389+0000","last_became_active":"2023-03-03T05:45:35.851803+0000","last_became_peered":"2023-03-03T05:45:35.851803+0000","last_unstale":"2023-03-03T05:45:36.923389+0000","last_undegraded":"2023-03-03T05:45:36.923389+0000","last_fullsized":"2023-03-03T05:45:36.923389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:14.473334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.852256+0000","last_change":"2023-03-03T05:45:35.298542+0000","last_active":"2023-03-03T05:45:35.852256+0000","last_peered":"2023-03-03T05:45:35.852256+0000","last_clean":"2023-03-03T05:45:35.852256+0000","last_became_active":"2023-03-03T05:45:35.298219+0000","last_became_peered":"2023-03-03T05:45:35.298219+0000","last_unstale":"2023-03-03T05:45:35.852256+0000","last_undegraded":"2023-03-03T05:45:35.852256+0000","last_fullsized":"2023-03-03T05:45:35.852256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962699+0000","last_change":"2023-03-03T05:45:35.861231+0000","last_active":"2023-03-03T05:45:35.962699+0000","last_peered":"2023-03-03T05:45:35.962699+0000","last_clean":"2023-03-03T05:45:35.962699+0000","last_became_active":"2023-03-03T05:45:35.860675+0000","last_became_peered":"2023-03-03T05:45:35.860675+0000","last_unstale":"2023-03-03T05:45:35.962699+0000","last_undegraded":"2023-03-03T05:45:35.962699+0000","last_fullsized":"2023-03-03T05:45:35.962699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843762+0000","last_change":"2023-03-03T05:45:35.312143+0000","last_active":"2023-03-03T05:45:35.843762+0000","last_peered":"2023-03-03T05:45:35.843762+0000","last_clean":"2023-03-03T05:45:35.843762+0000","last_became_active":"2023-03-03T05:45:35.311848+0000","last_became_peered":"2023-03-03T05:45:35.311848+0000","last_unstale":"2023-03-03T05:45:35.843762+0000","last_undegraded":"2023-03-03T05:45:35.843762+0000","last_fullsized":"2023-03-03T05:45:35.843762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:22.294783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851303+0000","last_change":"2023-03-03T05:45:35.869075+0000","last_active":"2023-03-03T05:45:36.851303+0000","last_peered":"2023-03-03T05:45:36.851303+0000","last_clean":"2023-03-03T05:45:36.851303+0000","last_became_active":"2023-03-03T05:45:35.868413+0000","last_became_peered":"2023-03-03T05:45:35.868413+0000","last_unstale":"2023-03-03T05:45:36.851303+0000","last_undegraded":"2023-03-03T05:45:36.851303+0000","last_fullsized":"2023-03-03T05:45:36.851303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:34.743111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844226+0000","last_change":"2023-03-03T05:45:35.296611+0000","last_active":"2023-03-03T05:45:35.844226+0000","last_peered":"2023-03-03T05:45:35.844226+0000","last_clean":"2023-03-03T05:45:35.844226+0000","last_became_active":"2023-03-03T05:45:35.296294+0000","last_became_peered":"2023-03-03T05:45:35.296294+0000","last_unstale":"2023-03-03T05:45:35.844226+0000","last_undegraded":"2023-03-03T05:45:35.844226+0000","last_fullsized":"2023-03-03T05:45:35.844226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:16.655836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850964+0000","last_change":"2023-03-03T05:45:35.857451+0000","last_active":"2023-03-03T05:45:36.850964+0000","last_peered":"2023-03-03T05:45:36.850964+0000","last_clean":"2023-03-03T05:45:36.850964+0000","last_became_active":"2023-03-03T05:45:35.857087+0000","last_became_peered":"2023-03-03T05:45:35.857087+0000","last_unstale":"2023-03-03T05:45:36.850964+0000","last_undegraded":"2023-03-03T05:45:36.850964+0000","last_fullsized":"2023-03-03T05:45:36.850964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842797+0000","last_change":"2023-03-03T05:45:35.296114+0000","last_active":"2023-03-03T05:45:35.842797+0000","last_peered":"2023-03-03T05:45:35.842797+0000","last_clean":"2023-03-03T05:45:35.842797+0000","last_became_active":"2023-03-03T05:45:35.295855+0000","last_became_peered":"2023-03-03T05:45:35.295855+0000","last_unstale":"2023-03-03T05:45:35.842797+0000","last_undegraded":"2023-03-03T05:45:35.842797+0000","last_fullsized":"2023-03-03T05:45:35.842797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923246+0000","last_change":"2023-03-03T05:45:35.862249+0000","last_active":"2023-03-03T05:45:36.923246+0000","last_peered":"2023-03-03T05:45:36.923246+0000","last_clean":"2023-03-03T05:45:36.923246+0000","last_became_active":"2023-03-03T05:45:35.861773+0000","last_became_peered":"2023-03-03T05:45:35.861773+0000","last_unstale":"2023-03-03T05:45:36.923246+0000","last_undegraded":"2023-03-03T05:45:36.923246+0000","last_fullsized":"2023-03-03T05:45:36.923246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848485+0000","last_change":"2023-03-03T05:45:29.189033+0000","last_active":"2023-03-03T05:45:35.848485+0000","last_peered":"2023-03-03T05:45:35.848485+0000","last_clean":"2023-03-03T05:45:35.848485+0000","last_became_active":"2023-03-03T05:45:29.188719+0000","last_became_peered":"2023-03-03T05:45:29.188719+0000","last_unstale":"2023-03-03T05:45:35.848485+0000","last_undegraded":"2023-03-03T05:45:35.848485+0000","last_fullsized":"2023-03-03T05:45:35.848485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850356+0000","last_change":"2023-03-03T05:45:35.303756+0000","last_active":"2023-03-03T05:45:36.850356+0000","last_peered":"2023-03-03T05:45:36.850356+0000","last_clean":"2023-03-03T05:45:36.850356+0000","last_became_active":"2023-03-03T05:45:35.303448+0000","last_became_peered":"2023-03-03T05:45:35.303448+0000","last_unstale":"2023-03-03T05:45:36.850356+0000","last_undegraded":"2023-03-03T05:45:36.850356+0000","last_fullsized":"2023-03-03T05:45:36.850356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922899+0000","last_change":"2023-03-03T05:45:35.854166+0000","last_active":"2023-03-03T05:45:36.922899+0000","last_peered":"2023-03-03T05:45:36.922899+0000","last_clean":"2023-03-03T05:45:36.922899+0000","last_became_active":"2023-03-03T05:45:35.853873+0000","last_became_peered":"2023-03-03T05:45:35.853873+0000","last_unstale":"2023-03-03T05:45:36.922899+0000","last_undegraded":"2023-03-03T05:45:36.922899+0000","last_fullsized":"2023-03-03T05:45:36.922899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843699+0000","last_change":"2023-03-03T05:45:29.189075+0000","last_active":"2023-03-03T05:45:35.843699+0000","last_peered":"2023-03-03T05:45:35.843699+0000","last_clean":"2023-03-03T05:45:35.843699+0000","last_became_active":"2023-03-03T05:45:29.188778+0000","last_became_peered":"2023-03-03T05:45:29.188778+0000","last_unstale":"2023-03-03T05:45:35.843699+0000","last_undegraded":"2023-03-03T05:45:35.843699+0000","last_fullsized":"2023-03-03T05:45:35.843699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:54:35.536172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962645+0000","last_change":"2023-03-03T05:45:35.309781+0000","last_active":"2023-03-03T05:45:35.962645+0000","last_peered":"2023-03-03T05:45:35.962645+0000","last_clean":"2023-03-03T05:45:35.962645+0000","last_became_active":"2023-03-03T05:45:35.309497+0000","last_became_peered":"2023-03-03T05:45:35.309497+0000","last_unstale":"2023-03-03T05:45:35.962645+0000","last_undegraded":"2023-03-03T05:45:35.962645+0000","last_fullsized":"2023-03-03T05:45:35.962645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923571+0000","last_change":"2023-03-03T05:45:35.854721+0000","last_active":"2023-03-03T05:45:36.923571+0000","last_peered":"2023-03-03T05:45:36.923571+0000","last_clean":"2023-03-03T05:45:36.923571+0000","last_became_active":"2023-03-03T05:45:35.854423+0000","last_became_peered":"2023-03-03T05:45:35.854423+0000","last_unstale":"2023-03-03T05:45:36.923571+0000","last_undegraded":"2023-03-03T05:45:36.923571+0000","last_fullsized":"2023-03-03T05:45:36.923571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.852125+0000","last_change":"2023-03-03T05:45:29.057308+0000","last_active":"2023-03-03T05:45:35.852125+0000","last_peered":"2023-03-03T05:45:35.852125+0000","last_clean":"2023-03-03T05:45:35.852125+0000","last_became_active":"2023-03-03T05:45:29.056828+0000","last_became_peered":"2023-03-03T05:45:29.056828+0000","last_unstale":"2023-03-03T05:45:35.852125+0000","last_undegraded":"2023-03-03T05:45:35.852125+0000","last_fullsized":"2023-03-03T05:45:35.852125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:56.079893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848095+0000","last_change":"2023-03-03T05:45:35.295842+0000","last_active":"2023-03-03T05:45:35.848095+0000","last_peered":"2023-03-03T05:45:35.848095+0000","last_clean":"2023-03-03T05:45:35.848095+0000","last_became_active":"2023-03-03T05:45:35.295545+0000","last_became_peered":"2023-03-03T05:45:35.295545+0000","last_unstale":"2023-03-03T05:45:35.848095+0000","last_undegraded":"2023-03-03T05:45:35.848095+0000","last_fullsized":"2023-03-03T05:45:35.848095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923636+0000","last_change":"2023-03-03T05:45:35.862437+0000","last_active":"2023-03-03T05:45:36.923636+0000","last_peered":"2023-03-03T05:45:36.923636+0000","last_clean":"2023-03-03T05:45:36.923636+0000","last_became_active":"2023-03-03T05:45:35.862152+0000","last_became_peered":"2023-03-03T05:45:35.862152+0000","last_unstale":"2023-03-03T05:45:36.923636+0000","last_undegraded":"2023-03-03T05:45:36.923636+0000","last_fullsized":"2023-03-03T05:45:36.923636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842862+0000","last_change":"2023-03-03T05:45:29.057646+0000","last_active":"2023-03-03T05:45:35.842862+0000","last_peered":"2023-03-03T05:45:35.842862+0000","last_clean":"2023-03-03T05:45:35.842862+0000","last_became_active":"2023-03-03T05:45:29.057330+0000","last_became_peered":"2023-03-03T05:45:29.057330+0000","last_unstale":"2023-03-03T05:45:35.842862+0000","last_undegraded":"2023-03-03T05:45:35.842862+0000","last_fullsized":"2023-03-03T05:45:35.842862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:22:57.039574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.858102+0000","last_change":"2023-03-03T05:45:22.224435+0000","last_active":"2023-03-03T05:45:35.858102+0000","last_peered":"2023-03-03T05:45:35.858102+0000","last_clean":"2023-03-03T05:45:35.858102+0000","last_became_active":"2023-03-03T05:45:22.222958+0000","last_became_peered":"2023-03-03T05:45:22.222958+0000","last_unstale":"2023-03-03T05:45:35.858102+0000","last_undegraded":"2023-03-03T05:45:35.858102+0000","last_fullsized":"2023-03-03T05:45:35.858102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_clean_scrub_stamp":"2023-03-03T05:45:20.925942+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:10:20.426026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843695+0000","last_change":"2023-03-03T05:45:35.304483+0000","last_active":"2023-03-03T05:45:35.843695+0000","last_peered":"2023-03-03T05:45:35.843695+0000","last_clean":"2023-03-03T05:45:35.843695+0000","last_became_active":"2023-03-03T05:45:35.304202+0000","last_became_peered":"2023-03-03T05:45:35.304202+0000","last_unstale":"2023-03-03T05:45:35.843695+0000","last_undegraded":"2023-03-03T05:45:35.843695+0000","last_fullsized":"2023-03-03T05:45:35.843695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851295+0000","last_change":"2023-03-03T05:45:35.869049+0000","last_active":"2023-03-03T05:45:36.851295+0000","last_peered":"2023-03-03T05:45:36.851295+0000","last_clean":"2023-03-03T05:45:36.851295+0000","last_became_active":"2023-03-03T05:45:35.868376+0000","last_became_peered":"2023-03-03T05:45:35.868376+0000","last_unstale":"2023-03-03T05:45:36.851295+0000","last_undegraded":"2023-03-03T05:45:36.851295+0000","last_fullsized":"2023-03-03T05:45:36.851295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:42:55.207846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843590+0000","last_change":"2023-03-03T05:45:29.056874+0000","last_active":"2023-03-03T05:45:35.843590+0000","last_peered":"2023-03-03T05:45:35.843590+0000","last_clean":"2023-03-03T05:45:35.843590+0000","last_became_active":"2023-03-03T05:45:29.056438+0000","last_became_peered":"2023-03-03T05:45:29.056438+0000","last_unstale":"2023-03-03T05:45:35.843590+0000","last_undegraded":"2023-03-03T05:45:35.843590+0000","last_fullsized":"2023-03-03T05:45:35.843590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:08.113953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.846141+0000","last_change":"2023-03-03T05:45:35.304562+0000","last_active":"2023-03-03T05:45:35.846141+0000","last_peered":"2023-03-03T05:45:35.846141+0000","last_clean":"2023-03-03T05:45:35.846141+0000","last_became_active":"2023-03-03T05:45:35.304264+0000","last_became_peered":"2023-03-03T05:45:35.304264+0000","last_unstale":"2023-03-03T05:45:35.846141+0000","last_undegraded":"2023-03-03T05:45:35.846141+0000","last_fullsized":"2023-03-03T05:45:35.846141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923094+0000","last_change":"2023-03-03T05:45:35.859779+0000","last_active":"2023-03-03T05:45:36.923094+0000","last_peered":"2023-03-03T05:45:36.923094+0000","last_clean":"2023-03-03T05:45:36.923094+0000","last_became_active":"2023-03-03T05:45:35.857596+0000","last_became_peered":"2023-03-03T05:45:35.857596+0000","last_unstale":"2023-03-03T05:45:36.923094+0000","last_undegraded":"2023-03-03T05:45:36.923094+0000","last_fullsized":"2023-03-03T05:45:36.923094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844418+0000","last_change":"2023-03-03T05:45:29.056891+0000","last_active":"2023-03-03T05:45:35.844418+0000","last_peered":"2023-03-03T05:45:35.844418+0000","last_clean":"2023-03-03T05:45:35.844418+0000","last_became_active":"2023-03-03T05:45:29.056483+0000","last_became_peered":"2023-03-03T05:45:29.056483+0000","last_unstale":"2023-03-03T05:45:35.844418+0000","last_undegraded":"2023-03-03T05:45:35.844418+0000","last_fullsized":"2023-03-03T05:45:35.844418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:25.701585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.847358+0000","last_change":"2023-03-03T05:45:35.293821+0000","last_active":"2023-03-03T05:45:35.847358+0000","last_peered":"2023-03-03T05:45:35.847358+0000","last_clean":"2023-03-03T05:45:35.847358+0000","last_became_active":"2023-03-03T05:45:35.293533+0000","last_became_peered":"2023-03-03T05:45:35.293533+0000","last_unstale":"2023-03-03T05:45:35.847358+0000","last_undegraded":"2023-03-03T05:45:35.847358+0000","last_fullsized":"2023-03-03T05:45:35.847358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923917+0000","last_change":"2023-03-03T05:45:35.866830+0000","last_active":"2023-03-03T05:45:36.923917+0000","last_peered":"2023-03-03T05:45:36.923917+0000","last_clean":"2023-03-03T05:45:36.923917+0000","last_became_active":"2023-03-03T05:45:35.866509+0000","last_became_peered":"2023-03-03T05:45:35.866509+0000","last_unstale":"2023-03-03T05:45:36.923917+0000","last_undegraded":"2023-03-03T05:45:36.923917+0000","last_fullsized":"2023-03-03T05:45:36.923917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843688+0000","last_change":"2023-03-03T05:45:35.300794+0000","last_active":"2023-03-03T05:45:35.843688+0000","last_peered":"2023-03-03T05:45:35.843688+0000","last_clean":"2023-03-03T05:45:35.843688+0000","last_became_active":"2023-03-03T05:45:35.300531+0000","last_became_peered":"2023-03-03T05:45:35.300531+0000","last_unstale":"2023-03-03T05:45:35.843688+0000","last_undegraded":"2023-03-03T05:45:35.843688+0000","last_fullsized":"2023-03-03T05:45:35.843688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843343+0000","last_change":"2023-03-03T05:45:29.193601+0000","last_active":"2023-03-03T05:45:35.843343+0000","last_peered":"2023-03-03T05:45:35.843343+0000","last_clean":"2023-03-03T05:45:35.843343+0000","last_became_active":"2023-03-03T05:45:29.193262+0000","last_became_peered":"2023-03-03T05:45:29.193262+0000","last_unstale":"2023-03-03T05:45:35.843343+0000","last_undegraded":"2023-03-03T05:45:35.843343+0000","last_fullsized":"2023-03-03T05:45:35.843343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853228+0000","last_change":"2023-03-03T05:45:35.864517+0000","last_active":"2023-03-03T05:45:36.853228+0000","last_peered":"2023-03-03T05:45:36.853228+0000","last_clean":"2023-03-03T05:45:36.853228+0000","last_became_active":"2023-03-03T05:45:35.864193+0000","last_became_peered":"2023-03-03T05:45:35.864193+0000","last_unstale":"2023-03-03T05:45:36.853228+0000","last_undegraded":"2023-03-03T05:45:36.853228+0000","last_fullsized":"2023-03-03T05:45:36.853228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:11:08.433881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844985+0000","last_change":"2023-03-03T05:45:29.193546+0000","last_active":"2023-03-03T05:45:35.844985+0000","last_peered":"2023-03-03T05:45:35.844985+0000","last_clean":"2023-03-03T05:45:35.844985+0000","last_became_active":"2023-03-03T05:45:29.193172+0000","last_became_peered":"2023-03-03T05:45:29.193172+0000","last_unstale":"2023-03-03T05:45:35.844985+0000","last_undegraded":"2023-03-03T05:45:35.844985+0000","last_fullsized":"2023-03-03T05:45:35.844985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:07:42.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843485+0000","last_change":"2023-03-03T05:45:35.296693+0000","last_active":"2023-03-03T05:45:35.843485+0000","last_peered":"2023-03-03T05:45:35.843485+0000","last_clean":"2023-03-03T05:45:35.843485+0000","last_became_active":"2023-03-03T05:45:35.296469+0000","last_became_peered":"2023-03-03T05:45:35.296469+0000","last_unstale":"2023-03-03T05:45:35.843485+0000","last_undegraded":"2023-03-03T05:45:35.843485+0000","last_fullsized":"2023-03-03T05:45:35.843485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850872+0000","last_change":"2023-03-03T05:45:35.856468+0000","last_active":"2023-03-03T05:45:36.850872+0000","last_peered":"2023-03-03T05:45:36.850872+0000","last_clean":"2023-03-03T05:45:36.850872+0000","last_became_active":"2023-03-03T05:45:35.855970+0000","last_became_peered":"2023-03-03T05:45:35.855970+0000","last_unstale":"2023-03-03T05:45:36.850872+0000","last_undegraded":"2023-03-03T05:45:36.850872+0000","last_fullsized":"2023-03-03T05:45:36.850872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843006+0000","last_change":"2023-03-03T05:45:35.301855+0000","last_active":"2023-03-03T05:45:35.843006+0000","last_peered":"2023-03-03T05:45:35.843006+0000","last_clean":"2023-03-03T05:45:35.843006+0000","last_became_active":"2023-03-03T05:45:35.301622+0000","last_became_peered":"2023-03-03T05:45:35.301622+0000","last_unstale":"2023-03-03T05:45:35.843006+0000","last_undegraded":"2023-03-03T05:45:35.843006+0000","last_fullsized":"2023-03-03T05:45:35.843006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:42.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853538+0000","last_change":"2023-03-03T05:45:35.851780+0000","last_active":"2023-03-03T05:45:36.853538+0000","last_peered":"2023-03-03T05:45:36.853538+0000","last_clean":"2023-03-03T05:45:36.853538+0000","last_became_active":"2023-03-03T05:45:35.850190+0000","last_became_peered":"2023-03-03T05:45:35.850190+0000","last_unstale":"2023-03-03T05:45:36.853538+0000","last_undegraded":"2023-03-03T05:45:36.853538+0000","last_fullsized":"2023-03-03T05:45:36.853538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.853739+0000","last_change":"2023-03-03T05:45:35.309047+0000","last_active":"2023-03-03T05:45:35.853739+0000","last_peered":"2023-03-03T05:45:35.853739+0000","last_clean":"2023-03-03T05:45:35.853739+0000","last_became_active":"2023-03-03T05:45:35.308564+0000","last_became_peered":"2023-03-03T05:45:35.308564+0000","last_unstale":"2023-03-03T05:45:35.853739+0000","last_undegraded":"2023-03-03T05:45:35.853739+0000","last_fullsized":"2023-03-03T05:45:35.853739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850462+0000","last_change":"2023-03-03T05:45:35.861316+0000","last_active":"2023-03-03T05:45:36.850462+0000","last_peered":"2023-03-03T05:45:36.850462+0000","last_clean":"2023-03-03T05:45:36.850462+0000","last_became_active":"2023-03-03T05:45:35.861052+0000","last_became_peered":"2023-03-03T05:45:35.861052+0000","last_unstale":"2023-03-03T05:45:36.850462+0000","last_undegraded":"2023-03-03T05:45:36.850462+0000","last_fullsized":"2023-03-03T05:45:36.850462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844470+0000","last_change":"2023-03-03T05:45:35.303666+0000","last_active":"2023-03-03T05:45:35.844470+0000","last_peered":"2023-03-03T05:45:35.844470+0000","last_clean":"2023-03-03T05:45:35.844470+0000","last_became_active":"2023-03-03T05:45:35.303213+0000","last_became_peered":"2023-03-03T05:45:35.303213+0000","last_unstale":"2023-03-03T05:45:35.844470+0000","last_undegraded":"2023-03-03T05:45:35.844470+0000","last_fullsized":"2023-03-03T05:45:35.844470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:54.389717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850392+0000","last_change":"2023-03-03T05:45:35.857442+0000","last_active":"2023-03-03T05:45:36.850392+0000","last_peered":"2023-03-03T05:45:36.850392+0000","last_clean":"2023-03-03T05:45:36.850392+0000","last_became_active":"2023-03-03T05:45:35.857130+0000","last_became_peered":"2023-03-03T05:45:35.857130+0000","last_unstale":"2023-03-03T05:45:36.850392+0000","last_undegraded":"2023-03-03T05:45:36.850392+0000","last_fullsized":"2023-03-03T05:45:36.850392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848573+0000","last_change":"2023-03-03T05:45:35.305098+0000","last_active":"2023-03-03T05:45:35.848573+0000","last_peered":"2023-03-03T05:45:35.848573+0000","last_clean":"2023-03-03T05:45:35.848573+0000","last_became_active":"2023-03-03T05:45:35.304812+0000","last_became_peered":"2023-03-03T05:45:35.304812+0000","last_unstale":"2023-03-03T05:45:35.848573+0000","last_undegraded":"2023-03-03T05:45:35.848573+0000","last_fullsized":"2023-03-03T05:45:35.848573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923341+0000","last_change":"2023-03-03T05:45:35.858806+0000","last_active":"2023-03-03T05:45:36.923341+0000","last_peered":"2023-03-03T05:45:36.923341+0000","last_clean":"2023-03-03T05:45:36.923341+0000","last_became_active":"2023-03-03T05:45:35.858471+0000","last_became_peered":"2023-03-03T05:45:35.858471+0000","last_unstale":"2023-03-03T05:45:36.923341+0000","last_undegraded":"2023-03-03T05:45:36.923341+0000","last_fullsized":"2023-03-03T05:45:36.923341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848752+0000","last_change":"2023-03-03T05:45:35.310654+0000","last_active":"2023-03-03T05:45:35.848752+0000","last_peered":"2023-03-03T05:45:35.848752+0000","last_clean":"2023-03-03T05:45:35.848752+0000","last_became_active":"2023-03-03T05:45:35.310377+0000","last_became_peered":"2023-03-03T05:45:35.310377+0000","last_unstale":"2023-03-03T05:45:35.848752+0000","last_undegraded":"2023-03-03T05:45:35.848752+0000","last_fullsized":"2023-03-03T05:45:35.848752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850763+0000","last_change":"2023-03-03T05:45:35.850670+0000","last_active":"2023-03-03T05:45:36.850763+0000","last_peered":"2023-03-03T05:45:36.850763+0000","last_clean":"2023-03-03T05:45:36.850763+0000","last_became_active":"2023-03-03T05:45:35.850380+0000","last_became_peered":"2023-03-03T05:45:35.850380+0000","last_unstale":"2023-03-03T05:45:36.850763+0000","last_undegraded":"2023-03-03T05:45:36.850763+0000","last_fullsized":"2023-03-03T05:45:36.850763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923387+0000","last_change":"2023-03-03T05:45:35.863660+0000","last_active":"2023-03-03T05:45:36.923387+0000","last_peered":"2023-03-03T05:45:36.923387+0000","last_clean":"2023-03-03T05:45:36.923387+0000","last_became_active":"2023-03-03T05:45:35.863234+0000","last_became_peered":"2023-03-03T05:45:35.863234+0000","last_unstale":"2023-03-03T05:45:36.923387+0000","last_undegraded":"2023-03-03T05:45:36.923387+0000","last_fullsized":"2023-03-03T05:45:36.923387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922937+0000","last_change":"2023-03-03T05:45:35.851019+0000","last_active":"2023-03-03T05:45:36.922937+0000","last_peered":"2023-03-03T05:45:36.922937+0000","last_clean":"2023-03-03T05:45:36.922937+0000","last_became_active":"2023-03-03T05:45:35.850439+0000","last_became_peered":"2023-03-03T05:45:35.850439+0000","last_unstale":"2023-03-03T05:45:36.922937+0000","last_undegraded":"2023-03-03T05:45:36.922937+0000","last_fullsized":"2023-03-03T05:45:36.922937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:43.693100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962628+0000","last_change":"2023-03-03T05:45:35.860508+0000","last_active":"2023-03-03T05:45:35.962628+0000","last_peered":"2023-03-03T05:45:35.962628+0000","last_clean":"2023-03-03T05:45:35.962628+0000","last_became_active":"2023-03-03T05:45:35.860137+0000","last_became_peered":"2023-03-03T05:45:35.860137+0000","last_unstale":"2023-03-03T05:45:35.962628+0000","last_undegraded":"2023-03-03T05:45:35.962628+0000","last_fullsized":"2023-03-03T05:45:35.962628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923487+0000","last_change":"2023-03-03T05:45:35.864824+0000","last_active":"2023-03-03T05:45:36.923487+0000","last_peered":"2023-03-03T05:45:36.923487+0000","last_clean":"2023-03-03T05:45:36.923487+0000","last_became_active":"2023-03-03T05:45:35.864466+0000","last_became_peered":"2023-03-03T05:45:35.864466+0000","last_unstale":"2023-03-03T05:45:36.923487+0000","last_undegraded":"2023-03-03T05:45:36.923487+0000","last_fullsized":"2023-03-03T05:45:36.923487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922781+0000","last_change":"2023-03-03T05:45:35.859499+0000","last_active":"2023-03-03T05:45:36.922781+0000","last_peered":"2023-03-03T05:45:36.922781+0000","last_clean":"2023-03-03T05:45:36.922781+0000","last_became_active":"2023-03-03T05:45:35.859200+0000","last_became_peered":"2023-03-03T05:45:35.859200+0000","last_unstale":"2023-03-03T05:45:36.922781+0000","last_undegraded":"2023-03-03T05:45:36.922781+0000","last_fullsized":"2023-03-03T05:45:36.922781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":88905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8048,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363792,"statfs":{"total":96636764160,"available":96628523008,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9076,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362764,"statfs":{"total":96636764160,"available":96627470336,"internally_reserved":0,"allocated":831488,"data_stored":679127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9072,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362768,"statfs":{"total":96636764160,"available":96627474432,"internally_reserved":0,"allocated":827392,"data_stored":677834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T05:45:42.415 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T05:45:42.416 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-03-03T05:45:42.416 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-03-03T05:45:42.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-03-03T05:45:42.417 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-03-03T05:45:42.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.691+0000 7f4a0359e700 1 Processor -- start 2023-03-03T05:45:42.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.691+0000 7f4a0359e700 1 -- start start 2023-03-03T05:45:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.692+0000 7f4a0359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a040c9610 0x7f4a040ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.692+0000 7f4a0359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4a040ca980 0x7f4a041636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.692+0000 7f4a0359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a040c9ca0 con 0x7f4a041564b0 2023-03-03T05:45:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.692+0000 7f4a0359e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a040cadc0 con 0x7f4a040ca980 2023-03-03T05:45:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.692+0000 7f4a0359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a040b4920 con 0x7f4a040c9610 2023-03-03T05:45:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a0259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4a041564b0 0x7f4a040cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45128/0 (socket says 172.21.15.102:45128) 2023-03-03T05:45:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a0259c700 1 -- 172.21.15.102:0/3280431642 learned_addr learned my addr 172.21.15.102:0/3280431642 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a02d9d700 1 --2- 172.21.15.102:0/3280431642 >> v2:172.21.15.102:3300/0 conn(0x7f4a040ca980 0x7f4a041636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f49fbfff700 1 --2- 172.21.15.102:0/3280431642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a040c9610 0x7f4a040ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a02d9d700 1 -- 172.21.15.102:0/3280431642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a040c9610 msgr2=0x7f4a040ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a02d9d700 1 --2- 172.21.15.102:0/3280431642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a040c9610 0x7f4a040ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a02d9d700 1 -- 172.21.15.102:0/3280431642 >> v1:172.21.15.102:6789/0 conn(0x7f4a041564b0 legacy=0x7f4a040cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a01d9b700 1 -- 172.21.15.102:0/3280431642 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243393870 0 0) 0x7f4a040c9ca0 con 0x7f4a041564b0 2023-03-03T05:45:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a02d9d700 1 -- 172.21.15.102:0/3280431642 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a040b46f0 con 0x7f4a040ca980 2023-03-03T05:45:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.693+0000 7f4a02d9d700 1 --2- 172.21.15.102:0/3280431642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041636a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009d90 tx=0x7f49fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdada36e2ed69fca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:42.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a01d9b700 1 -- 172.21.15.102:0/3280431642 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00e070 con 0x7f4a040ca980 2023-03-03T05:45:42.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a01d9b700 1 -- 172.21.15.102:0/3280431642 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc00bb20 con 0x7f4a040ca980 2023-03-03T05:45:42.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a01d9b700 1 -- 172.21.15.102:0/3280431642 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc012430 con 0x7f4a040ca980 2023-03-03T05:45:42.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a0359e700 1 -- 172.21.15.102:0/3280431642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 msgr2=0x7f4a041636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a0359e700 1 --2- 172.21.15.102:0/3280431642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041636a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009d90 tx=0x7f49fc00b330 comp rx=0 tx=0).stop 2023-03-03T05:45:42.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a0359e700 1 -- 172.21.15.102:0/3280431642 shutdown_connections 2023-03-03T05:45:42.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a0359e700 1 --2- 172.21.15.102:0/3280431642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041636a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a0359e700 1 --2- 172.21.15.102:0/3280431642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a040c9610 0x7f4a040ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a0359e700 1 -- 172.21.15.102:0/3280431642 >> 172.21.15.102:0/3280431642 conn(0x7f4a04065fb0 msgr2=0x7f4a040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.694+0000 7f4a0359e700 1 -- 172.21.15.102:0/3280431642 shutdown_connections 2023-03-03T05:45:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.695+0000 7f4a0359e700 1 -- 172.21.15.102:0/3280431642 wait complete. 2023-03-03T05:45:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.696+0000 7f4a0359e700 1 Processor -- start 2023-03-03T05:45:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.697+0000 7f4a0359e700 1 -- start start 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a0359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a040c9610 0x7f4a040c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a0359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a0359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a041564b0 0x7f4a041e2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a0359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a041e2f80 con 0x7f4a040ca980 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a0359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a041f22f0 con 0x7f4a040c9610 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a0359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a040c7be0 con 0x7f4a041564b0 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a0259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a040c9610 0x7f4a040c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f49fbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a02d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a041564b0 0x7f4a041e2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f49fbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041ee590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36220/0 (socket says 172.21.15.102:36220) 2023-03-03T05:45:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a02d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a041564b0 0x7f4a041e2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36730/0 (socket says 172.21.15.102:36730) 2023-03-03T05:45:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f49fbfff700 1 -- 172.21.15.102:0/74301354 learned_addr learned my addr 172.21.15.102:0/74301354 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a02d9d700 1 -- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a040c9610 msgr2=0x7f4a040c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a02d9d700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a040c9610 0x7f4a040c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a02d9d700 1 -- 172.21.15.102:0/74301354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 msgr2=0x7f4a041ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a02d9d700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f4a02d9d700 1 -- 172.21.15.102:0/74301354 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc009a70 con 0x7f4a041564b0 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.698+0000 7f49fbfff700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.699+0000 7f4a02d9d700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a041564b0 0x7f4a041e2580 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f49fc016000 tx=0x7f49fc004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.699+0000 7f49faffd700 1 -- 172.21.15.102:0/74301354 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc012430 con 0x7f4a041564b0 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.699+0000 7f49faffd700 1 -- 172.21.15.102:0/74301354 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc020070 con 0x7f4a041564b0 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.699+0000 7f49faffd700 1 -- 172.21.15.102:0/74301354 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004280 con 0x7f4a041564b0 2023-03-03T05:45:42.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.699+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a040b1f30 con 0x7f4a041564b0 2023-03-03T05:45:42.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.699+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a040b4c00 con 0x7f4a041564b0 2023-03-03T05:45:42.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.700+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a040b2210 con 0x7f4a041564b0 2023-03-03T05:45:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.701+0000 7f49faffd700 1 -- 172.21.15.102:0/74301354 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f49fc02e070 con 0x7f4a041564b0 2023-03-03T05:45:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.702+0000 7f49faffd700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49e40682f0 0x7f49e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.703+0000 7f49faffd700 1 -- 172.21.15.102:0/74301354 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f49fc022070 con 0x7f4a041564b0 2023-03-03T05:45:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.703+0000 7f4a0259c700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49e40682f0 0x7f49e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.704+0000 7f4a0259c700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49e40682f0 0x7f49e406a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f49f4000f60 tx=0x7f49f4007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:42.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.707+0000 7f49faffd700 1 -- 172.21.15.102:0/74301354 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49fc0540b0 con 0x7f4a041564b0 2023-03-03T05:45:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.952+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4a041f24a0 con 0x7f4a041564b0 2023-03-03T05:45:42.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.956+0000 7f49faffd700 1 -- 172.21.15.102:0/74301354 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v6) v1 ==== 74+0+166386 (secure 0 0 0) 0x7f4a041f24a0 con 0x7f4a041564b0 2023-03-03T05:45:42.959 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:42.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49e40682f0 msgr2=0x7f49e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49e40682f0 0x7f49e406a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f49f4000f60 tx=0x7f49f4007e70 comp rx=0 tx=0).stop 2023-03-03T05:45:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a041564b0 msgr2=0x7f4a041e2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a041564b0 0x7f4a041e2580 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f49fc016000 tx=0x7f49fc004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 shutdown_connections 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49e40682f0 0x7f49e406a7a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a041564b0 0x7f4a041e2580 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a040c9610 0x7f4a040c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 --2- 172.21.15.102:0/74301354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a040ca980 0x7f4a041ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 >> 172.21.15.102:0/74301354 conn(0x7f4a04065fb0 msgr2=0x7f4a04066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 shutdown_connections 2023-03-03T05:45:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:42.960+0000 7f4a0359e700 1 -- 172.21.15.102:0/74301354 wait complete. 2023-03-03T05:45:42.989 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":6,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":114058},{"type":"v1","addr":"172.21.15.172:6833","nonce":114058}]},"active_addr":"172.21.15.172:6833/114058","active_change":"2023-03-03T05:45:13.930479+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2554687939}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1158372578}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2481203810}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1684134736}]}]} 2023-03-03T05:45:42.994 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-03-03T05:45:42.994 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-03-03T05:45:42.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:43.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.237+0000 7f1e9339d700 1 Processor -- start 2023-03-03T05:45:43.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.238+0000 7f1e9339d700 1 -- start start 2023-03-03T05:45:43.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.239+0000 7f1e9339d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1e8c0c9610 0x7f1e8c0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.239+0000 7f1e9339d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c0c9ca0 con 0x7f1e8c1564b0 2023-03-03T05:45:43.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.239+0000 7f1e9339d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c0b46f0 con 0x7f1e8c0ca980 2023-03-03T05:45:43.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.239+0000 7f1e9339d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b4920 con 0x7f1e8c0c9610 2023-03-03T05:45:43.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.239+0000 7f1e9239b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1e8c1564b0 0x7f1e8c0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45142/0 (socket says 172.21.15.102:45142) 2023-03-03T05:45:43.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.239+0000 7f1e9239b700 1 -- 172.21.15.102:0/528409682 learned_addr learned my addr 172.21.15.102:0/528409682 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:43.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.239+0000 7f1e91b9a700 1 --2- 172.21.15.102:0/528409682 >> v2:172.21.15.172:3300/0 conn(0x7f1e8c0c9610 0x7f1e8c0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91399700 1 -- 172.21.15.102:0/528409682 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096442122 0 0) 0x7f1e8c0c9ca0 con 0x7f1e8c1564b0 2023-03-03T05:45:43.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91399700 1 -- 172.21.15.102:0/528409682 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e780035b0 con 0x7f1e8c1564b0 2023-03-03T05:45:43.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91b9a700 1 -- 172.21.15.102:0/528409682 >> v1:172.21.15.172:6789/0 conn(0x7f1e8c0ca980 legacy=0x7f1e8c1636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91399700 1 -- 172.21.15.102:0/528409682 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180844247 0 0) 0x7f1e8c0b46f0 con 0x7f1e8c0ca980 2023-03-03T05:45:43.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91b9a700 1 -- 172.21.15.102:0/528409682 >> v1:172.21.15.102:6789/0 conn(0x7f1e8c1564b0 legacy=0x7f1e8c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91b9a700 1 -- 172.21.15.102:0/528409682 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c0b4c00 con 0x7f1e8c0c9610 2023-03-03T05:45:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91b9a700 1 --2- 172.21.15.102:0/528409682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c0ca440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c009840 tx=0x7f1e7c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e95dd28077da5c28 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91399700 1 -- 172.21.15.102:0/528409682 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c011830 con 0x7f1e8c0c9610 2023-03-03T05:45:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91399700 1 -- 172.21.15.102:0/528409682 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c011990 con 0x7f1e8c0c9610 2023-03-03T05:45:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.240+0000 7f1e91399700 1 -- 172.21.15.102:0/528409682 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c011d00 con 0x7f1e8c0c9610 2023-03-03T05:45:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.241+0000 7f1e9339d700 1 -- 172.21.15.102:0/528409682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 msgr2=0x7f1e8c0ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.241+0000 7f1e9339d700 1 --2- 172.21.15.102:0/528409682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c0ca440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c009840 tx=0x7f1e7c009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:43.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.241+0000 7f1e9339d700 1 -- 172.21.15.102:0/528409682 shutdown_connections 2023-03-03T05:45:43.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.241+0000 7f1e9339d700 1 --2- 172.21.15.102:0/528409682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c0ca440 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.241+0000 7f1e9339d700 1 -- 172.21.15.102:0/528409682 >> 172.21.15.102:0/528409682 conn(0x7f1e8c065fb0 msgr2=0x7f1e8c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:43.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.241+0000 7f1e9339d700 1 -- 172.21.15.102:0/528409682 shutdown_connections 2023-03-03T05:45:43.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.241+0000 7f1e9339d700 1 -- 172.21.15.102:0/528409682 wait complete. 2023-03-03T05:45:43.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.243+0000 7f1e9339d700 1 Processor -- start 2023-03-03T05:45:43.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.243+0000 7f1e9339d700 1 -- start start 2023-03-03T05:45:43.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.244+0000 7f1e9339d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c1f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.244+0000 7f1e9339d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e8c0ca980 0x7f1e8c1f0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e9339d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e8c1564b0 0x7f1e8c1e2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e9339d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c1e3600 con 0x7f1e8c0ca980 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e9339d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c159a50 con 0x7f1e8c0c9610 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e9339d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c1ef4e0 con 0x7f1e8c1564b0 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e91b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e8c0ca980 0x7f1e8c1f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e92b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e8c1564b0 0x7f1e8c1e2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e9239b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c1f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e92b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e8c1564b0 0x7f1e8c1e2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36732/0 (socket says 172.21.15.102:36732) 2023-03-03T05:45:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e92b9c700 1 -- 172.21.15.102:0/3188912546 learned_addr learned my addr 172.21.15.102:0/3188912546 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e91b9a700 1 -- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e8c1564b0 msgr2=0x7f1e8c1e2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e91b9a700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e8c1564b0 0x7f1e8c1e2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e91b9a700 1 -- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 msgr2=0x7f1e8c1f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e91b9a700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c1f0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e91b9a700 1 -- 172.21.15.102:0/3188912546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c0094a0 con 0x7f1e8c0ca980 2023-03-03T05:45:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e92b9c700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e8c1564b0 0x7f1e8c1e2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.245+0000 7f1e9239b700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c1f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:43.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.246+0000 7f1e91b9a700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e8c0ca980 0x7f1e8c1f0750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c016000 tx=0x7f1e7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:43.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.246+0000 7f1e82ffd700 1 -- 172.21.15.102:0/3188912546 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c004550 con 0x7f1e8c0ca980 2023-03-03T05:45:43.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.246+0000 7f1e82ffd700 1 -- 172.21.15.102:0/3188912546 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c01f070 con 0x7f1e8c0ca980 2023-03-03T05:45:43.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.246+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c0b2210 con 0x7f1e8c0ca980 2023-03-03T05:45:43.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.246+0000 7f1e82ffd700 1 -- 172.21.15.102:0/3188912546 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c0038e0 con 0x7f1e8c0ca980 2023-03-03T05:45:43.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.246+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c0b1f30 con 0x7f1e8c0ca980 2023-03-03T05:45:43.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.247+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c156880 con 0x7f1e8c0ca980 2023-03-03T05:45:43.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.249+0000 7f1e82ffd700 1 -- 172.21.15.102:0/3188912546 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1e7c00f050 con 0x7f1e8c0ca980 2023-03-03T05:45:43.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.251+0000 7f1e82ffd700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1e70068320 0x7f1e7006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.251+0000 7f1e82ffd700 1 -- 172.21.15.102:0/3188912546 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1e7c021070 con 0x7f1e8c0ca980 2023-03-03T05:45:43.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.251+0000 7f1e9239b700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1e70068320 0x7f1e7006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.252+0000 7f1e9239b700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1e70068320 0x7f1e7006a7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e88007fd0 tx=0x7f1e88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:43.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.255+0000 7f1e82ffd700 1 -- 172.21.15.102:0/3188912546 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1e7c053860 con 0x7f1e8c0ca980 2023-03-03T05:45:43.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.458+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e8c158350 con 0x7f1e8c0ca980 2023-03-03T05:45:43.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.459+0000 7f1e82ffd700 1 -- 172.21.15.102:0/3188912546 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f1e7c015020 con 0x7f1e8c0ca980 2023-03-03T05:45:43.461 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:43.462 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:43.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1e70068320 msgr2=0x7f1e7006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1e70068320 0x7f1e7006a7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e88007fd0 tx=0x7f1e88009040 comp rx=0 tx=0).stop 2023-03-03T05:45:43.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e8c0ca980 msgr2=0x7f1e8c1f0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e8c0ca980 0x7f1e8c1f0750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c016000 tx=0x7f1e7c004750 comp rx=0 tx=0).stop 2023-03-03T05:45:43.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 shutdown_connections 2023-03-03T05:45:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1e70068320 0x7f1e7006a7d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e8c1564b0 0x7f1e8c1e2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e8c0c9610 0x7f1e8c1f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 --2- 172.21.15.102:0/3188912546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e8c0ca980 0x7f1e8c1f0750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 >> 172.21.15.102:0/3188912546 conn(0x7f1e8c065fb0 msgr2=0x7f1e8c1550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 shutdown_connections 2023-03-03T05:45:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.462+0000 7f1e9339d700 1 -- 172.21.15.102:0/3188912546 wait complete. 2023-03-03T05:45:43.491 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-03-03T05:45:43.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:43.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.731+0000 7f1032bff700 1 Processor -- start 2023-03-03T05:45:43.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.732+0000 7f1032bff700 1 -- start start 2023-03-03T05:45:43.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.733+0000 7f1032bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f102c0cf9a0 0x7f102c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.733+0000 7f1032bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c0cb9f0 con 0x7f102c1597f0 2023-03-03T05:45:43.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.733+0000 7f1032bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c159c80 con 0x7f102c0ced80 2023-03-03T05:45:43.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.733+0000 7f1032bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c0ca750 con 0x7f102c0cf9a0 2023-03-03T05:45:43.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.733+0000 7f102bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f102c1597f0 0x7f102c161650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45156/0 (socket says 172.21.15.102:45156) 2023-03-03T05:45:43.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.733+0000 7f102bfff700 1 -- 172.21.15.102:0/253222817 learned_addr learned my addr 172.21.15.102:0/253222817 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:43.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.733+0000 7f103119c700 1 --2- 172.21.15.102:0/253222817 >> v2:172.21.15.172:3300/0 conn(0x7f102c0cf9a0 0x7f102c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f102b7fe700 1 -- 172.21.15.102:0/253222817 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925421004 0 0) 0x7f102c159c80 con 0x7f102c0ced80 2023-03-03T05:45:43.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f102b7fe700 1 -- 172.21.15.102:0/253222817 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c0035b0 con 0x7f102c0ced80 2023-03-03T05:45:43.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f102b7fe700 1 -- 172.21.15.102:0/253222817 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150202363 0 0) 0x7f102c0cb9f0 con 0x7f102c1597f0 2023-03-03T05:45:43.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f102b7fe700 1 -- 172.21.15.102:0/253222817 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c159c80 con 0x7f102c1597f0 2023-03-03T05:45:43.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f103119c700 1 -- 172.21.15.102:0/253222817 >> v1:172.21.15.172:6789/0 conn(0x7f102c0ced80 legacy=0x7f102c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f103119c700 1 -- 172.21.15.102:0/253222817 >> v1:172.21.15.102:6789/0 conn(0x7f102c1597f0 legacy=0x7f102c161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f103119c700 1 -- 172.21.15.102:0/253222817 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c0cade0 con 0x7f102c0cf9a0 2023-03-03T05:45:43.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f103119c700 1 --2- 172.21.15.102:0/253222817 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0cf9a0 0x7f102c0cf150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1024009d90 tx=0x7f102400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f2e91d7b818c69c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:43.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f102b7fe700 1 -- 172.21.15.102:0/253222817 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102400e070 con 0x7f102c0cf9a0 2023-03-03T05:45:43.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f102b7fe700 1 -- 172.21.15.102:0/253222817 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102400bb20 con 0x7f102c0cf9a0 2023-03-03T05:45:43.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.734+0000 7f102b7fe700 1 -- 172.21.15.102:0/253222817 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024012430 con 0x7f102c0cf9a0 2023-03-03T05:45:43.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.735+0000 7f1032bff700 1 -- 172.21.15.102:0/253222817 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0cf9a0 msgr2=0x7f102c0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.735+0000 7f1032bff700 1 --2- 172.21.15.102:0/253222817 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0cf9a0 0x7f102c0cf150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1024009d90 tx=0x7f102400b330 comp rx=0 tx=0).stop 2023-03-03T05:45:43.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.735+0000 7f1032bff700 1 -- 172.21.15.102:0/253222817 shutdown_connections 2023-03-03T05:45:43.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.735+0000 7f1032bff700 1 --2- 172.21.15.102:0/253222817 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0cf9a0 0x7f102c0cf150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.735+0000 7f1032bff700 1 -- 172.21.15.102:0/253222817 >> 172.21.15.102:0/253222817 conn(0x7f102c065fb0 msgr2=0x7f102c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:43.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.735+0000 7f1032bff700 1 -- 172.21.15.102:0/253222817 shutdown_connections 2023-03-03T05:45:43.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.735+0000 7f1032bff700 1 -- 172.21.15.102:0/253222817 wait complete. 2023-03-03T05:45:43.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.737+0000 7f1032bff700 1 Processor -- start 2023-03-03T05:45:43.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.737+0000 7f1032bff700 1 -- start start 2023-03-03T05:45:43.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.738+0000 7f1032bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0ced80 0x7f102c157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.738+0000 7f1032bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f102c0cf9a0 0x7f102c157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.738+0000 7f1032bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102c1597f0 0x7f102c1e32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.738+0000 7f1032bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c157b70 con 0x7f102c1597f0 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.738+0000 7f1032bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c0b3370 con 0x7f102c0ced80 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.738+0000 7f1032bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c158550 con 0x7f102c0cf9a0 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102c1597f0 0x7f102c1e32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102c1597f0 0x7f102c1e32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36244/0 (socket says 172.21.15.102:36244) 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 -- 172.21.15.102:0/3635232425 learned_addr learned my addr 172.21.15.102:0/3635232425 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 -- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f102c0cf9a0 msgr2=0x7f102c157550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f102c0cf9a0 0x7f102c157550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 -- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0ced80 msgr2=0x7f102c157010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:43.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0ced80 0x7f102c157010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 -- 172.21.15.102:0/3635232425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1024009a70 con 0x7f102c1597f0 2023-03-03T05:45:43.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.739+0000 7f103119c700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102c1597f0 0x7f102c1e32d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f102c0cc970 tx=0x7f1024004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.740+0000 7f10297fa700 1 -- 172.21.15.102:0/3635232425 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102401b810 con 0x7f102c1597f0 2023-03-03T05:45:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.740+0000 7f10297fa700 1 -- 172.21.15.102:0/3635232425 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1024020070 con 0x7f102c1597f0 2023-03-03T05:45:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.740+0000 7f10297fa700 1 -- 172.21.15.102:0/3635232425 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102401bbb0 con 0x7f102c1597f0 2023-03-03T05:45:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.740+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c0cb3a0 con 0x7f102c1597f0 2023-03-03T05:45:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.740+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c0cb0c0 con 0x7f102c1597f0 2023-03-03T05:45:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.741+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c0cb680 con 0x7f102c1597f0 2023-03-03T05:45:43.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.742+0000 7f10297fa700 1 -- 172.21.15.102:0/3635232425 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f102402e070 con 0x7f102c1597f0 2023-03-03T05:45:43.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.743+0000 7f10297fa700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10100682f0 0x7f101006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:43.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.743+0000 7f10297fa700 1 -- 172.21.15.102:0/3635232425 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1024022070 con 0x7f102c1597f0 2023-03-03T05:45:43.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.744+0000 7f103099b700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10100682f0 0x7f101006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:43.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.746+0000 7f103099b700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10100682f0 0x7f101006a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1020001570 tx=0x7f1020007910 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:43.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.747+0000 7f10297fa700 1 -- 172.21.15.102:0/3635232425 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1024054530 con 0x7f102c1597f0 2023-03-03T05:45:43.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.944+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f102c1e2ff0 con 0x7f102c1597f0 2023-03-03T05:45:43.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.946+0000 7f10297fa700 1 -- 172.21.15.102:0/3635232425 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f102c1e2ff0 con 0x7f102c1597f0 2023-03-03T05:45:43.948 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:43.948 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:43.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10100682f0 msgr2=0x7f101006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10100682f0 0x7f101006a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1020001570 tx=0x7f1020007910 comp rx=0 tx=0).stop 2023-03-03T05:45:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102c1597f0 msgr2=0x7f102c1e32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102c1597f0 0x7f102c1e32d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f102c0cc970 tx=0x7f1024004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 shutdown_connections 2023-03-03T05:45:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102c1597f0 0x7f102c1e32d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f102c0ced80 0x7f102c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10100682f0 0x7f101006a7a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 --2- 172.21.15.102:0/3635232425 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f102c0cf9a0 0x7f102c157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:43.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 >> 172.21.15.102:0/3635232425 conn(0x7f102c065fb0 msgr2=0x7f102c0ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:43.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 shutdown_connections 2023-03-03T05:45:43.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:43.949+0000 7f1032bff700 1 -- 172.21.15.102:0/3635232425 wait complete. 2023-03-03T05:45:43.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-03-03T05:45:43.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-03-03T05:45:43.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-03-03T05:45:43.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-03-03T05:45:43.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-03-03T05:45:43.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-03-03T05:45:43.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-03-03T05:45:43.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-03-03T05:45:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.428+0000 7ff3b8dff700 1 Processor -- start 2023-03-03T05:45:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.429+0000 7ff3b8dff700 1 -- start start 2023-03-03T05:45:44.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.432+0000 7ff3b8dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4070790 0x7ff3b406ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.432+0000 7ff3b8dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b40c5430 0x7ff3b40c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.432+0000 7ff3b8dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40c64a0 con 0x7ff3b40cb1c0 2023-03-03T05:45:44.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.432+0000 7ff3b8dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b40c2430 con 0x7ff3b40c5430 2023-03-03T05:45:44.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.432+0000 7ff3b8dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40c5ee0 con 0x7ff3b4070790 2023-03-03T05:45:44.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.433+0000 7ff3b2d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b40c5430 0x7ff3b40c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.433+0000 7ff3b2d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b40c5430 0x7ff3b40c58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36252/0 (socket says 172.21.15.102:36252) 2023-03-03T05:45:44.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.433+0000 7ff3b2d9d700 1 -- 172.21.15.102:0/1701445605 learned_addr learned my addr 172.21.15.102:0/1701445605 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.433+0000 7ff3abfff700 1 --2- 172.21.15.102:0/1701445605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4070790 0x7ff3b406ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.434+0000 7ff3b2d9d700 1 -- 172.21.15.102:0/1701445605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4070790 msgr2=0x7ff3b406ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.434+0000 7ff3b2d9d700 1 --2- 172.21.15.102:0/1701445605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4070790 0x7ff3b406ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.434+0000 7ff3b2d9d700 1 -- 172.21.15.102:0/1701445605 >> v1:172.21.15.102:6789/0 conn(0x7ff3b40cb1c0 legacy=0x7ff3b414f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.434+0000 7ff3b2d9d700 1 -- 172.21.15.102:0/1701445605 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b40c61c0 con 0x7ff3b40c5430 2023-03-03T05:45:44.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.434+0000 7ff3b2d9d700 1 --2- 172.21.15.102:0/1701445605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40c5430 0x7ff3b40c58a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00d150 tx=0x7ff3ac00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917dc80d6bbc99f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.435+0000 7fe352bff700 1 Processor -- start 2023-03-03T05:45:44.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.435+0000 7fe352bff700 1 -- start start 2023-03-03T05:45:44.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.435+0000 7ff3b1d9b700 1 -- 172.21.15.102:0/1701445605 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac010070 con 0x7ff3b40c5430 2023-03-03T05:45:44.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.435+0000 7ff3b1d9b700 1 -- 172.21.15.102:0/1701445605 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac00dd80 con 0x7ff3b40c5430 2023-03-03T05:45:44.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.435+0000 7ff3b1d9b700 1 -- 172.21.15.102:0/1701445605 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac00b310 con 0x7ff3b40c5430 2023-03-03T05:45:44.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.436+0000 7ff3b8dff700 1 -- 172.21.15.102:0/1701445605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40c5430 msgr2=0x7ff3b40c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.436+0000 7ff3b8dff700 1 --2- 172.21.15.102:0/1701445605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40c5430 0x7ff3b40c58a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00d150 tx=0x7ff3ac00d460 comp rx=0 tx=0).stop 2023-03-03T05:45:44.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.436+0000 7ff3b8dff700 1 -- 172.21.15.102:0/1701445605 shutdown_connections 2023-03-03T05:45:44.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.436+0000 7ff3b8dff700 1 --2- 172.21.15.102:0/1701445605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40c5430 0x7ff3b40c58a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.436+0000 7ff3b8dff700 1 --2- 172.21.15.102:0/1701445605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4070790 0x7ff3b406ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.436+0000 7ff3b8dff700 1 -- 172.21.15.102:0/1701445605 >> 172.21.15.102:0/1701445605 conn(0x7ff3b406af90 msgr2=0x7ff3b40caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.438+0000 7fe352bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe34c0717a0 0x7fe34c155870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.438+0000 7fe352bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c0cb2f0 con 0x7fe34c072090 2023-03-03T05:45:44.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.438+0000 7fe352bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c0cb590 con 0x7fe34c158ca0 2023-03-03T05:45:44.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.438+0000 7fe352bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34c15b5b0 con 0x7fe34c0717a0 2023-03-03T05:45:44.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.438+0000 7fe34bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe34c0717a0 0x7fe34c155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.439+0000 7fe34bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe34c0717a0 0x7fe34c155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59258/0 (socket says 172.21.15.102:59258) 2023-03-03T05:45:44.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.439+0000 7fe34bfff700 1 -- 172.21.15.102:0/2985303226 learned_addr learned my addr 172.21.15.102:0/2985303226 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.439+0000 7fe34bfff700 1 -- 172.21.15.102:0/2985303226 >> v1:172.21.15.172:6789/0 conn(0x7fe34c158ca0 legacy=0x7fe34c14ec90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T05:45:44.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.439+0000 7fe34bfff700 1 -- 172.21.15.102:0/2985303226 >> v1:172.21.15.102:6789/0 conn(0x7fe34c072090 legacy=0x7fe34c06e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.439+0000 7fe34bfff700 1 -- 172.21.15.102:0/2985303226 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c155070 con 0x7fe34c0717a0 2023-03-03T05:45:44.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.439+0000 7fe34bfff700 1 --2- 172.21.15.102:0/2985303226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34c0717a0 0x7fe34c155870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0098f0 tx=0x7fe33c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc3dd546ae061bde server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.441+0000 7ff3b8dff700 1 -- 172.21.15.102:0/1701445605 shutdown_connections 2023-03-03T05:45:44.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.441+0000 7fe34b7fe700 1 -- 172.21.15.102:0/2985303226 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c004230 con 0x7fe34c0717a0 2023-03-03T05:45:44.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.441+0000 7fe34b7fe700 1 -- 172.21.15.102:0/2985303226 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c004390 con 0x7fe34c0717a0 2023-03-03T05:45:44.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.441+0000 7fe34b7fe700 1 -- 172.21.15.102:0/2985303226 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0036a0 con 0x7fe34c0717a0 2023-03-03T05:45:44.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.442+0000 7fe352bff700 1 -- 172.21.15.102:0/2985303226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34c0717a0 msgr2=0x7fe34c155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.442+0000 7fe352bff700 1 --2- 172.21.15.102:0/2985303226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34c0717a0 0x7fe34c155870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0098f0 tx=0x7fe33c009c00 comp rx=0 tx=0).stop 2023-03-03T05:45:44.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.443+0000 7fe352bff700 1 -- 172.21.15.102:0/2985303226 shutdown_connections 2023-03-03T05:45:44.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.443+0000 7fe352bff700 1 --2- 172.21.15.102:0/2985303226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34c0717a0 0x7fe34c155870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.443+0000 7fe352bff700 1 -- 172.21.15.102:0/2985303226 >> 172.21.15.102:0/2985303226 conn(0x7fe34c06af90 msgr2=0x7fe34c0caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.443+0000 7ff3b8dff700 1 -- 172.21.15.102:0/1701445605 wait complete. 2023-03-03T05:45:44.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.444+0000 7fe352bff700 1 -- 172.21.15.102:0/2985303226 shutdown_connections 2023-03-03T05:45:44.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.445+0000 7fe352bff700 1 -- 172.21.15.102:0/2985303226 wait complete. 2023-03-03T05:45:44.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.447+0000 7ff3b8dff700 1 Processor -- start 2023-03-03T05:45:44.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.447+0000 7ff3b8dff700 1 -- start start 2023-03-03T05:45:44.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.449+0000 7fe352bff700 1 Processor -- start 2023-03-03T05:45:44.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7ff3b8dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b4070790 0x7ff3b40c09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7ff3b8dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40c5430 0x7ff3b4157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7ff3b8dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cb1c0 0x7ff3b41577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7ff3b8dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b41f6820 con 0x7ff3b40c5430 2023-03-03T05:45:44.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7ff3b8dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4156af0 con 0x7ff3b4070790 2023-03-03T05:45:44.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7ff3b8dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b41f6d10 con 0x7ff3b40cb1c0 2023-03-03T05:45:44.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7ff3b259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b4070790 0x7ff3b40c09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7ff3b259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b4070790 0x7ff3b40c09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59264/0 (socket says 172.21.15.102:59264) 2023-03-03T05:45:44.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7ff3b259c700 1 -- 172.21.15.102:0/3558279061 learned_addr learned my addr 172.21.15.102:0/3558279061 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7ff3b259c700 1 -- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cb1c0 msgr2=0x7ff3b41577a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:44.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7ff3b259c700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cb1c0 0x7ff3b41577a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7ff3b259c700 1 -- 172.21.15.102:0/3558279061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40c5430 msgr2=0x7ff3b4157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7ff3b259c700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40c5430 0x7ff3b4157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7ff3b259c700 1 -- 172.21.15.102:0/3558279061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac00bd40 con 0x7ff3b4070790 2023-03-03T05:45:44.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.449+0000 7fe352bff700 1 -- start start 2023-03-03T05:45:44.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe352bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe34c0717a0 0x7fe34c15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe352bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34c072090 0x7fe34c15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe352bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe34c14d7e0 0x7fe34c142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe352bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c155e80 con 0x7fe34c14d7e0 2023-03-03T05:45:44.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe352bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c14e720 con 0x7fe34c072090 2023-03-03T05:45:44.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe352bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c14c970 con 0x7fe34c0717a0 2023-03-03T05:45:44.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe35099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe34c0717a0 0x7fe34c15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe35099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe34c0717a0 0x7fe34c15ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36760/0 (socket says 172.21.15.102:36760) 2023-03-03T05:45:44.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.451+0000 7fe35099b700 1 -- 172.21.15.102:0/3575100997 learned_addr learned my addr 172.21.15.102:0/3575100997 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7fe35099b700 1 -- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34c072090 msgr2=0x7fe34c15a0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:44.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7fe35099b700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34c072090 0x7fe34c15a0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7fe35099b700 1 -- 172.21.15.102:0/3575100997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe34c14d7e0 msgr2=0x7fe34c142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7fe35099b700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe34c14d7e0 0x7fe34c142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7fe35099b700 1 -- 172.21.15.102:0/3575100997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c009550 con 0x7fe34c0717a0 2023-03-03T05:45:44.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7fe35099b700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe34c0717a0 0x7fe34c15ac90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe34400be80 tx=0x7fe34400bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34400cdc0 con 0x7fe34c0717a0 2023-03-03T05:45:44.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe344014440 con 0x7fe34c0717a0 2023-03-03T05:45:44.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.453+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3440147d0 con 0x7fe34c0717a0 2023-03-03T05:45:44.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.454+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c154d20 con 0x7fe34c0717a0 2023-03-03T05:45:44.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.454+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c155350 con 0x7fe34c0717a0 2023-03-03T05:45:44.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7f454e79d700 1 Processor -- start 2023-03-03T05:45:44.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.452+0000 7f454e79d700 1 -- start start 2023-03-03T05:45:44.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.455+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe34c14f3a0 con 0x7fe34c0717a0 2023-03-03T05:45:44.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.455+0000 7f1ce8bff700 1 Processor -- start 2023-03-03T05:45:44.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.456+0000 7f1ce8bff700 1 -- start start 2023-03-03T05:45:44.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.457+0000 7ff3b259c700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b4070790 0x7ff3b40c09f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff3a400c4a0 tx=0x7ff3a400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.458+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40054e0 con 0x7ff3b4070790 2023-03-03T05:45:44.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.458+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a4003680 con 0x7ff3b4070790 2023-03-03T05:45:44.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.458+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4003a30 con 0x7ff3b4070790 2023-03-03T05:45:44.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.459+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe34401b030 con 0x7fe34c0717a0 2023-03-03T05:45:44.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.457+0000 7f454e79d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45481572a0 0x7f45480c6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.457+0000 7f454e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4548070e60 0x7f45480712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.457+0000 7f454e79d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4548071eb0 con 0x7f45480c6c90 2023-03-03T05:45:44.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.457+0000 7f454e79d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454815cf90 con 0x7f45481572a0 2023-03-03T05:45:44.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.457+0000 7f454e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4548071bd0 con 0x7f4548070e60 2023-03-03T05:45:44.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.459+0000 7f454cf9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f45480c6c90 0x7f45480c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36438/0 (socket says 172.21.15.102:36438) 2023-03-03T05:45:44.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.459+0000 7f454cf9a700 1 -- 172.21.15.102:0/182101266 learned_addr learned my addr 172.21.15.102:0/182101266 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.459+0000 7f454d79b700 1 --2- 172.21.15.102:0/182101266 >> v2:172.21.15.102:3300/0 conn(0x7f45481572a0 0x7f45480c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.459+0000 7f454df9c700 1 --2- 172.21.15.102:0/182101266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4548070e60 0x7f45480712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b406e500 con 0x7ff3b4070790 2023-03-03T05:45:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b406e220 con 0x7ff3b4070790 2023-03-03T05:45:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7f50653ff700 1 Processor -- start 2023-03-03T05:45:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7f454d79b700 1 -- 172.21.15.102:0/182101266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4548070e60 msgr2=0x7f45480712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7f454d79b700 1 --2- 172.21.15.102:0/182101266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4548070e60 0x7f45480712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7f454d79b700 1 -- 172.21.15.102:0/182101266 >> v1:172.21.15.172:6789/0 conn(0x7f45480c6c90 legacy=0x7f45480c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7f454d79b700 1 -- 172.21.15.102:0/182101266 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454815d250 con 0x7f45481572a0 2023-03-03T05:45:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.460+0000 7f50653ff700 1 -- start start 2023-03-03T05:45:44.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.461+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff398000fc0 con 0x7ff3b4070790 2023-03-03T05:45:44.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7f454d79b700 1 --2- 172.21.15.102:0/182101266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45481572a0 0x7f45480c6750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4544009840 tx=0x7f4544009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df915b1001a0f870 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.458+0000 7f1ce8bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1ce406e360 0x7f1ce40c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.461+0000 7f1ce8bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ce4152b40 0x7f1ce4152f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.461+0000 7f1ce8bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce4071240 con 0x7f1ce406e730 2023-03-03T05:45:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.461+0000 7f1ce8bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce406f050 con 0x7f1ce4152b40 2023-03-03T05:45:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.461+0000 7f1ce8bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce4153490 con 0x7f1ce406e360 2023-03-03T05:45:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7f1ce1d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1ce406e730 0x7f1ce40709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36452/0 (socket says 172.21.15.102:36452) 2023-03-03T05:45:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7f1ce1d9b700 1 -- 172.21.15.102:0/1488630422 learned_addr learned my addr 172.21.15.102:0/1488630422 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7f1ce259c700 1 --2- 172.21.15.102:0/1488630422 >> v2:172.21.15.172:3300/0 conn(0x7f1ce406e360 0x7f1ce40c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7f1ce2d9d700 1 --2- 172.21.15.102:0/1488630422 >> v2:172.21.15.102:3300/0 conn(0x7f1ce4152b40 0x7f1ce4152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7ff39bfff700 1 Processor -- start 2023-03-03T05:45:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7ff39bfff700 1 -- start start 2023-03-03T05:45:44.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7fe3497fa700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe338068340 0x7fe33806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe344085c60 con 0x7fe34c0717a0 2023-03-03T05:45:44.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7fe3497fa700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.102:6808/109064,v1:172.21.15.102:6809/109064] conn(0x7fe338070bf0 0x7fe338073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 --> [v2:172.21.15.102:6808/109064,v1:172.21.15.102:6809/109064] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3380736b0 con 0x7fe338070bf0 2023-03-03T05:45:44.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.462+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe34408a2a0 con 0x7fe34c0717a0 2023-03-03T05:45:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.463+0000 7f1ce259c700 1 -- 172.21.15.102:0/1488630422 >> v2:172.21.15.102:3300/0 conn(0x7f1ce4152b40 msgr2=0x7f1ce4152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.463+0000 7f1ce259c700 1 --2- 172.21.15.102:0/1488630422 >> v2:172.21.15.102:3300/0 conn(0x7f1ce4152b40 0x7f1ce4152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.463+0000 7fe35119c700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.102:6808/109064,v1:172.21.15.102:6809/109064] conn(0x7fe338070bf0 0x7fe338073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.463+0000 7f453ffff700 1 -- 172.21.15.102:0/182101266 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544004230 con 0x7f45481572a0 2023-03-03T05:45:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.463+0000 7f453ffff700 1 -- 172.21.15.102:0/182101266 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4544004390 con 0x7f45481572a0 2023-03-03T05:45:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.463+0000 7f453ffff700 1 -- 172.21.15.102:0/182101266 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45440036a0 con 0x7f45481572a0 2023-03-03T05:45:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7fe35119c700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.102:6808/109064,v1:172.21.15.102:6809/109064] conn(0x7fe338070bf0 0x7fe338073000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7fe34bfff700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe338068340 0x7fe33806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f454e79d700 1 -- 172.21.15.102:0/182101266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45481572a0 msgr2=0x7f45480c6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f454e79d700 1 --2- 172.21.15.102:0/182101266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45481572a0 0x7f45480c6750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4544009840 tx=0x7f4544009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:44.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f1ce159a700 1 -- 172.21.15.102:0/1488630422 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539331009 0 0) 0x7f1ce4071240 con 0x7f1ce406e730 2023-03-03T05:45:44.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f1ce259c700 1 -- 172.21.15.102:0/1488630422 >> v1:172.21.15.172:6789/0 conn(0x7f1ce406e730 legacy=0x7f1ce40709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f1ce259c700 1 -- 172.21.15.102:0/1488630422 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40c6dd0 con 0x7f1ce406e360 2023-03-03T05:45:44.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7f1ce259c700 1 --2- 172.21.15.102:0/1488630422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce406e360 0x7f1ce40c6890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40c8560 tx=0x7f1cd4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be2a6f73af4f63c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7f1ce159a700 1 -- 172.21.15.102:0/1488630422 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4011830 con 0x7f1ce406e360 2023-03-03T05:45:44.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7f1ce159a700 1 -- 172.21.15.102:0/1488630422 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd4011990 con 0x7f1ce406e360 2023-03-03T05:45:44.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7f1ce159a700 1 -- 172.21.15.102:0/1488630422 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4011d20 con 0x7f1ce406e360 2023-03-03T05:45:44.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7ff39bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff394071cf0 0x7ff3940c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3940c4c30 0x7ff3940c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7ff39bfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3940722c0 con 0x7ff39414fd10 2023-03-03T05:45:44.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7ff39bfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3940c21e0 con 0x7ff394071cf0 2023-03-03T05:45:44.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7ff39bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3940725a0 con 0x7ff3940c4c30 2023-03-03T05:45:44.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7ff39a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3940c4c30 0x7ff3940c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7ff39a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3940c4c30 0x7ff3940c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36776/0 (socket says 172.21.15.102:36776) 2023-03-03T05:45:44.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7ff39a59c700 1 -- 172.21.15.102:0/1091474354 learned_addr learned my addr 172.21.15.102:0/1091474354 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7ff39a59c700 1 -- 172.21.15.102:0/1091474354 >> v2:172.21.15.102:3300/0 conn(0x7ff394071cf0 msgr2=0x7ff3940c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7ff39a59c700 1 --2- 172.21.15.102:0/1091474354 >> v2:172.21.15.102:3300/0 conn(0x7ff394071cf0 0x7ff3940c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7ff39a59c700 1 -- 172.21.15.102:0/1091474354 >> v1:172.21.15.172:6789/0 conn(0x7ff39414fd10 legacy=0x7ff3940c6540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T05:45:44.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7ff39a59c700 1 -- 172.21.15.102:0/1091474354 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394070df0 con 0x7ff3940c4c30 2023-03-03T05:45:44.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7ff39a59c700 1 --2- 172.21.15.102:0/1091474354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3940c4c30 0x7ff3940c38b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff38c014500 tx=0x7ff38c014810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c98f123079d094a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f50653ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f50600cb1c0 0x7f5060070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f50653ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f506006ed00 0x7f506014ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f50653ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50600c64a0 con 0x7f506014f480 2023-03-03T05:45:44.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f50653ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50600c2430 con 0x7f50600cb1c0 2023-03-03T05:45:44.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.464+0000 7f50653ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506006f110 con 0x7f506006ed00 2023-03-03T05:45:44.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7f505f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f506014f480 0x7f50600c5430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36466/0 (socket says 172.21.15.102:36466) 2023-03-03T05:45:44.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7f505f7fe700 1 -- 172.21.15.102:0/3708202340 learned_addr learned my addr 172.21.15.102:0/3708202340 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:44.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7f505effd700 1 --2- 172.21.15.102:0/3708202340 >> v2:172.21.15.102:3300/0 conn(0x7f50600cb1c0 0x7f5060070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.466+0000 7f505e7fc700 1 --2- 172.21.15.102:0/3708202340 >> v2:172.21.15.172:3300/0 conn(0x7f506006ed00 0x7f506014ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f505effd700 1 -- 172.21.15.102:0/3708202340 >> v2:172.21.15.172:3300/0 conn(0x7f506006ed00 msgr2=0x7f506014ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f505effd700 1 --2- 172.21.15.102:0/3708202340 >> v2:172.21.15.172:3300/0 conn(0x7f506006ed00 0x7f506014ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f505effd700 1 -- 172.21.15.102:0/3708202340 >> v1:172.21.15.172:6789/0 conn(0x7f506014f480 legacy=0x7f50600c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f505effd700 1 -- 172.21.15.102:0/3708202340 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600c61c0 con 0x7f50600cb1c0 2023-03-03T05:45:44.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f454e79d700 1 -- 172.21.15.102:0/182101266 shutdown_connections 2023-03-03T05:45:44.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f454e79d700 1 --2- 172.21.15.102:0/182101266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4548070e60 0x7f45480712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f454e79d700 1 --2- 172.21.15.102:0/182101266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45481572a0 0x7f45480c6750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f454e79d700 1 -- 172.21.15.102:0/182101266 >> 172.21.15.102:0/182101266 conn(0x7f4548065fb0 msgr2=0x7f454806a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f454e79d700 1 -- 172.21.15.102:0/182101266 shutdown_connections 2023-03-03T05:45:44.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f454e79d700 1 -- 172.21.15.102:0/182101266 wait complete. 2023-03-03T05:45:44.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7f505effd700 1 --2- 172.21.15.102:0/3708202340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50600cb1c0 0x7f5060070790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5060067140 tx=0x7f505800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce5819f0baea3e9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7ff398d99700 1 -- 172.21.15.102:0/1091474354 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c003c20 con 0x7ff3940c4c30 2023-03-03T05:45:44.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7ff398d99700 1 -- 172.21.15.102:0/1091474354 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c003d80 con 0x7ff3940c4c30 2023-03-03T05:45:44.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7ff398d99700 1 -- 172.21.15.102:0/1091474354 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c0187d0 con 0x7ff3940c4c30 2023-03-03T05:45:44.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.468+0000 7ff39bfff700 1 -- 172.21.15.102:0/1091474354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3940c4c30 msgr2=0x7ff3940c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.468+0000 7ff39bfff700 1 --2- 172.21.15.102:0/1091474354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3940c4c30 0x7ff3940c38b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff38c014500 tx=0x7ff38c014810 comp rx=0 tx=0).stop 2023-03-03T05:45:44.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 -- 172.21.15.102:0/1488630422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce406e360 msgr2=0x7f1ce40c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 --2- 172.21.15.102:0/1488630422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce406e360 0x7f1ce40c6890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40c8560 tx=0x7f1cd4009b20 comp rx=0 tx=0).stop 2023-03-03T05:45:44.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 -- 172.21.15.102:0/1488630422 shutdown_connections 2023-03-03T05:45:44.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 --2- 172.21.15.102:0/1488630422 >> v2:172.21.15.102:3300/0 conn(0x7f1ce4152b40 0x7f1ce4152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 --2- 172.21.15.102:0/1488630422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce406e360 0x7f1ce40c6890 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40c8560 tx=0x7f1cd4009b20 comp rx=0 tx=0).stop 2023-03-03T05:45:44.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 -- 172.21.15.102:0/1488630422 >> 172.21.15.102:0/1488630422 conn(0x7f1ce406af90 msgr2=0x7f1ce40caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7fe34bfff700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe338068340 0x7fe33806a7f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe34c1545e0 tx=0x7fe33c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 -- 172.21.15.102:0/1488630422 shutdown_connections 2023-03-03T05:45:44.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f1ce8bff700 1 -- 172.21.15.102:0/1488630422 wait complete. 2023-03-03T05:45:44.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f505dffb700 1 -- 172.21.15.102:0/3708202340 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505800e070 con 0x7f50600cb1c0 2023-03-03T05:45:44.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f505dffb700 1 -- 172.21.15.102:0/3708202340 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505800bad0 con 0x7f50600cb1c0 2023-03-03T05:45:44.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.469+0000 7f505dffb700 1 -- 172.21.15.102:0/3708202340 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505800be80 con 0x7f50600cb1c0 2023-03-03T05:45:44.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7f50653ff700 1 -- 172.21.15.102:0/3708202340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50600cb1c0 msgr2=0x7f5060070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7f50653ff700 1 --2- 172.21.15.102:0/3708202340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50600cb1c0 0x7f5060070790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5060067140 tx=0x7f505800b330 comp rx=0 tx=0).stop 2023-03-03T05:45:44.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.465+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff3a4019030 con 0x7ff3b4070790 2023-03-03T05:45:44.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.467+0000 7ff3aaffd700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3940683c0 0x7ff39406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.468+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff3a4085410 con 0x7ff3b4070790 2023-03-03T05:45:44.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.468+0000 7ff3aaffd700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.102:6816/109067,v1:172.21.15.102:6817/109067] conn(0x7ff394070c70 0x7ff394073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.468+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 --> [v2:172.21.15.102:6816/109067,v1:172.21.15.102:6817/109067] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff394073730 con 0x7ff394070c70 2023-03-03T05:45:44.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.468+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3a40857c0 con 0x7ff3b4070790 2023-03-03T05:45:44.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== osd.1 v2:172.21.15.102:6808/109064 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe3380736b0 con 0x7fe338070bf0 2023-03-03T05:45:44.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7ff39bfff700 1 -- 172.21.15.102:0/1091474354 shutdown_connections 2023-03-03T05:45:44.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7ff39bfff700 1 --2- 172.21.15.102:0/1091474354 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3940c4c30 0x7ff3940c38b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7ff39bfff700 1 --2- 172.21.15.102:0/1091474354 >> v2:172.21.15.102:3300/0 conn(0x7ff394071cf0 0x7ff3940c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7ff39bfff700 1 -- 172.21.15.102:0/1091474354 >> 172.21.15.102:0/1091474354 conn(0x7ff39406af90 msgr2=0x7ff3940caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7ff39bfff700 1 -- 172.21.15.102:0/1091474354 shutdown_connections 2023-03-03T05:45:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.470+0000 7ff39bfff700 1 -- 172.21.15.102:0/1091474354 wait complete. 2023-03-03T05:45:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.471+0000 7f454e79d700 1 Processor -- start 2023-03-03T05:45:44.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.471+0000 7f50653ff700 1 -- 172.21.15.102:0/3708202340 shutdown_connections 2023-03-03T05:45:44.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.471+0000 7f50653ff700 1 --2- 172.21.15.102:0/3708202340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50600cb1c0 0x7f5060070790 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.471+0000 7f50653ff700 1 --2- 172.21.15.102:0/3708202340 >> v2:172.21.15.172:3300/0 conn(0x7f506006ed00 0x7f506014ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.471+0000 7f50653ff700 1 -- 172.21.15.102:0/3708202340 >> 172.21.15.102:0/3708202340 conn(0x7f506006af90 msgr2=0x7f50600caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.471+0000 7ff3abfff700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3940683c0 0x7ff39406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.471+0000 7f454e79d700 1 -- start start 2023-03-03T05:45:44.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f454e79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4548070e60 0x7f45480be680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f454e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45480c6c90 0x7f45480c0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f454e79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45481572a0 0x7f45480c3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f454e79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45481f5e20 con 0x7f4548070e60 2023-03-03T05:45:44.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f454e79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45480bebc0 con 0x7f45481572a0 2023-03-03T05:45:44.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f454e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45480c0ea0 con 0x7f45480c6c90 2023-03-03T05:45:44.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f1ce8bff700 1 Processor -- start 2023-03-03T05:45:44.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7f1ce8bff700 1 -- start start 2023-03-03T05:45:44.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45481572a0 0x7f45480c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45481572a0 0x7f45480c3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59280/0 (socket says 172.21.15.102:59280) 2023-03-03T05:45:44.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 -- 172.21.15.102:0/407117681 learned_addr learned my addr 172.21.15.102:0/407117681 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454d79b700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4548070e60 0x7f45480be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454cf9a700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45480c6c90 0x7f45480c0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 -- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45480c6c90 msgr2=0x7f45480c0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45480c6c90 0x7f45480c0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 -- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4548070e60 msgr2=0x7f45480be680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4548070e60 0x7f45480be680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7f454df9c700 1 -- 172.21.15.102:0/407117681 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45440094a0 con 0x7f45481572a0 2023-03-03T05:45:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f454df9c700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45481572a0 0x7f45480c3950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f454000efc0 tx=0x7f454000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce8bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce406e730 0x7f1ce40c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce8bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce4152b40 0x7f1ce41e5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce8bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce41e2990 0x7f1ce414b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce8bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce40c4880 con 0x7f1ce406e730 2023-03-03T05:45:44.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce8bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41e2540 con 0x7f1ce4152b40 2023-03-03T05:45:44.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce8bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce40c45a0 con 0x7f1ce41e2990 2023-03-03T05:45:44.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.472+0000 7ff3b2d9d700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.102:6816/109067,v1:172.21.15.102:6817/109067] conn(0x7ff394070c70 0x7ff394073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7ff3b2d9d700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.102:6816/109067,v1:172.21.15.102:6817/109067] conn(0x7ff394070c70 0x7ff394073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== osd.0 v2:172.21.15.102:6816/109067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff394073730 con 0x7ff394070c70 2023-03-03T05:45:44.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce41e2990 0x7f1ce414b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce41e2990 0x7f1ce414b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36794/0 (socket says 172.21.15.102:36794) 2023-03-03T05:45:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f1ce2d9d700 1 -- 172.21.15.102:0/142085625 learned_addr learned my addr 172.21.15.102:0/142085625 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7f1ce1d9b700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce4152b40 0x7f1ce41e5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7f1ce259c700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce406e730 0x7f1ce40c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7f1ce2d9d700 1 -- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce4152b40 msgr2=0x7f1ce41e5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7f1ce2d9d700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ce4152b40 0x7f1ce41e5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7f1ce2d9d700 1 -- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce406e730 msgr2=0x7f1ce40c27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7f1ce2d9d700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce406e730 0x7f1ce40c27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7f1ce2d9d700 1 -- 172.21.15.102:0/142085625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40094a0 con 0x7f1ce41e2990 2023-03-03T05:45:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f1ce2d9d700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce41e2990 0x7f1ce414b170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00efc0 tx=0x7f1cdc00c0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7ff3abfff700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3940683c0 0x7ff39406a870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40c13a0 tx=0x7ff39c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00cad0 con 0x7f1ce41e2990 2023-03-03T05:45:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc00f040 con 0x7f1ce41e2990 2023-03-03T05:45:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00ce70 con 0x7f1ce41e2990 2023-03-03T05:45:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce4071790 con 0x7f1ce41e2990 2023-03-03T05:45:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.478+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce40714b0 con 0x7f1ce41e2990 2023-03-03T05:45:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454000cd40 con 0x7f45481572a0 2023-03-03T05:45:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454000f040 con 0x7f45481572a0 2023-03-03T05:45:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.477+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45400146c0 con 0x7f45481572a0 2023-03-03T05:45:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.479+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454815d810 con 0x7f45481572a0 2023-03-03T05:45:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.473+0000 7ff39bfff700 1 Processor -- start 2023-03-03T05:45:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.474+0000 7ff39bfff700 1 -- start start 2023-03-03T05:45:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff394071cf0 0x7ff394148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3940c4c30 0x7ff3941f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff39414fd10 0x7ff3940c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7ff39bfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3941f5a20 con 0x7ff39414fd10 2023-03-03T05:45:44.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7ff39bfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3941df060 con 0x7ff3940c4c30 2023-03-03T05:45:44.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.476+0000 7ff39bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3941fd3b0 con 0x7ff394071cf0 2023-03-03T05:45:44.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.478+0000 7ff39a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff39414fd10 0x7ff3940c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.478+0000 7ff39a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff39414fd10 0x7ff3940c3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36334/0 (socket says 172.21.15.102:36334) 2023-03-03T05:45:44.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.478+0000 7ff39a59c700 1 -- 172.21.15.102:0/1056362018 learned_addr learned my addr 172.21.15.102:0/1056362018 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f50653ff700 1 -- 172.21.15.102:0/3708202340 shutdown_connections 2023-03-03T05:45:44.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.475+0000 7f50653ff700 1 -- 172.21.15.102:0/3708202340 wait complete. 2023-03-03T05:45:44.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.478+0000 7f50653ff700 1 Processor -- start 2023-03-03T05:45:44.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.478+0000 7f50653ff700 1 -- start start 2023-03-03T05:45:44.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.479+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454815d530 con 0x7f45481572a0 2023-03-03T05:45:44.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.480+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f454815daf0 con 0x7f45481572a0 2023-03-03T05:45:44.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.481+0000 7ff399d9b700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff394071cf0 0x7ff394148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.481+0000 7ff39959a700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3940c4c30 0x7ff3941f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.481+0000 7ff39a59c700 1 -- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff394071cf0 msgr2=0x7ff394148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7ff39a59c700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff394071cf0 0x7ff394148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7ff39a59c700 1 -- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3940c4c30 msgr2=0x7ff3941f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7ff39a59c700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3940c4c30 0x7ff3941f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7ff39a59c700 1 -- 172.21.15.102:0/1056362018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c014160 con 0x7ff39414fd10 2023-03-03T05:45:44.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.480+0000 7f50653ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f506006ed00 0x7f5060155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.480+0000 7f50653ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50600cb1c0 0x7f5060157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.480+0000 7f50653ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506014f480 0x7f5060158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.480+0000 7f50653ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060147f80 con 0x7f50600cb1c0 2023-03-03T05:45:44.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.480+0000 7f50653ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060153a60 con 0x7f506014f480 2023-03-03T05:45:44.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.480+0000 7f50653ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5060155ca0 con 0x7f506006ed00 2023-03-03T05:45:44.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.481+0000 7f505effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f506006ed00 0x7f5060155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.481+0000 7f505effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f506006ed00 0x7f5060155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36820/0 (socket says 172.21.15.102:36820) 2023-03-03T05:45:44.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.481+0000 7f505effd700 1 -- 172.21.15.102:0/1841799798 learned_addr learned my addr 172.21.15.102:0/1841799798 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.481+0000 7f505f7fe700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506014f480 0x7f5060158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7f505effd700 1 -- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506014f480 msgr2=0x7f5060158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7f505effd700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506014f480 0x7f5060158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7f505effd700 1 -- 172.21.15.102:0/1841799798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50600cb1c0 msgr2=0x7f5060157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7f505effd700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50600cb1c0 0x7f5060157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7f505effd700 1 -- 172.21.15.102:0/1841799798 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5058009a70 con 0x7f506006ed00 2023-03-03T05:45:44.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1cc8000fc0 con 0x7f1ce41e2990 2023-03-03T05:45:44.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.482+0000 7f505effd700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f506006ed00 0x7f5060155760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5058000f80 tx=0x7f5058004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.483+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50580046a0 con 0x7f506006ed00 2023-03-03T05:45:44.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.483+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5058004800 con 0x7f506006ed00 2023-03-03T05:45:44.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.483+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058004970 con 0x7f506006ed00 2023-03-03T05:45:44.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.483+0000 7f50653ff700 1 -- 172.21.15.102:0/1841799798 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506006e500 con 0x7f506006ed00 2023-03-03T05:45:44.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.484+0000 7f50653ff700 1 -- 172.21.15.102:0/1841799798 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506006e220 con 0x7f506006ed00 2023-03-03T05:45:44.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.484+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1cdc014480 con 0x7f1ce41e2990 2023-03-03T05:45:44.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.484+0000 7ff39a59c700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff39414fd10 0x7ff3940c3a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3940726e0 tx=0x7ff38c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.485+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c003c20 con 0x7ff39414fd10 2023-03-03T05:45:44.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.485+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c004030 con 0x7ff39414fd10 2023-03-03T05:45:44.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.485+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c004400 con 0x7ff39414fd10 2023-03-03T05:45:44.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.486+0000 7f1cd2ffd700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cc40683c0 0x7f1cc406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.486+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1cdc086740 con 0x7f1ce41e2990 2023-03-03T05:45:44.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.486+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5038000fc0 con 0x7f506006ed00 2023-03-03T05:45:44.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.484+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4540014820 con 0x7f45481572a0 2023-03-03T05:45:44.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.486+0000 7f453dffb700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f452c0683c0 0x7f452c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.486+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f45400873e0 con 0x7f45481572a0 2023-03-03T05:45:44.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.486+0000 7f453dffb700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:6818/109068,v1:172.21.15.102:6821/109068] conn(0x7f452c070c70 0x7f452c073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 --> [v2:172.21.15.102:6818/109068,v1:172.21.15.102:6821/109068] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f452c073730 con 0x7f452c070c70 2023-03-03T05:45:44.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4540087790 con 0x7f45481572a0 2023-03-03T05:45:44.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f1cd2ffd700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:6800/109066,v1:172.21.15.102:6801/109066] conn(0x7f1cc4070c70 0x7f1cc4073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 --> [v2:172.21.15.102:6800/109066,v1:172.21.15.102:6801/109066] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cc4073730 con 0x7f1cc4070c70 2023-03-03T05:45:44.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cdc088d60 con 0x7f1ce41e2990 2023-03-03T05:45:44.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f1ce259c700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cc40683c0 0x7f1cc406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f454d79b700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f452c0683c0 0x7f452c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7f1ce1d9b700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:6800/109066,v1:172.21.15.102:6801/109066] conn(0x7f1cc4070c70 0x7f1cc4073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3940713b0 con 0x7ff39414fd10 2023-03-03T05:45:44.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.487+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3940710d0 con 0x7ff39414fd10 2023-03-03T05:45:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.488+0000 7f1ce259c700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cc40683c0 0x7f1cc406a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ce406fbe0 tx=0x7f1cd4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.488+0000 7f1ce1d9b700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:6800/109066,v1:172.21.15.102:6801/109066] conn(0x7f1cc4070c70 0x7f1cc4073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.489+0000 7f454cf9a700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:6818/109068,v1:172.21.15.102:6821/109068] conn(0x7f452c070c70 0x7f452c073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.490+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff394072880 con 0x7ff39414fd10 2023-03-03T05:45:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.490+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== osd.3 v2:172.21.15.102:6800/109066 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1cc4073730 con 0x7f1cc4070c70 2023-03-03T05:45:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.490+0000 7f56715ff700 1 Processor -- start 2023-03-03T05:45:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.491+0000 7f56715ff700 1 -- start start 2023-03-03T05:45:44.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.492+0000 7f5dd8fff700 1 Processor -- start 2023-03-03T05:45:44.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.490+0000 7f454d79b700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f452c0683c0 0x7f452c06a870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4544009840 tx=0x7f4544008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.492+0000 7f454cf9a700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:6818/109068,v1:172.21.15.102:6821/109068] conn(0x7f452c070c70 0x7f452c073080 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.491+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f505802d070 con 0x7f506006ed00 2023-03-03T05:45:44.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== osd.2 v2:172.21.15.102:6818/109068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f452c073730 con 0x7f452c070c70 2023-03-03T05:45:44.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.493+0000 7f50477fe700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f504c0683c0 0x7f504c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5058088870 con 0x7f506006ed00 2023-03-03T05:45:44.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f50477fe700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6824/114774,v1:172.21.15.172:6825/114774] conn(0x7f504c070c70 0x7f504c073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 --> [v2:172.21.15.172:6824/114774,v1:172.21.15.172:6825/114774] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f504c073730 con 0x7f504c070c70 2023-03-03T05:45:44.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5058088c20 con 0x7f506006ed00 2023-03-03T05:45:44.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.495+0000 7f505e7fc700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f504c0683c0 0x7f504c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.495+0000 7f505f7fe700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6824/114774,v1:172.21.15.172:6825/114774] conn(0x7f504c070c70 0x7f504c073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7f505e7fc700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f504c0683c0 0x7f504c06a870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f50600c91f0 tx=0x7f5054008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7f505f7fe700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6824/114774,v1:172.21.15.172:6825/114774] conn(0x7f504c070c70 0x7f504c073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.493+0000 7f5dd8fff700 1 -- start start 2023-03-03T05:45:44.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f56715ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f566c0cb6d0 0x7f566c157170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f56715ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f566c06eea0 0x7f566c1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f56715ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c06ecc0 con 0x7f566c06e360 2023-03-03T05:45:44.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f56715ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566c15e2b0 con 0x7f566c06eea0 2023-03-03T05:45:44.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7f56715ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566c15dd30 con 0x7f566c0cb6d0 2023-03-03T05:45:44.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7f5dd8fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5dd406ca80 0x7f5dd406ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7f56627fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f566c0cb6d0 0x7f566c157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.494+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff38c02c070 con 0x7ff39414fd10 2023-03-03T05:45:44.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7ff38a7fc700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3780683c0 0x7ff37806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff38c087ce0 con 0x7ff39414fd10 2023-03-03T05:45:44.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7ff38a7fc700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6816/114772,v1:172.21.15.172:6817/114772] conn(0x7ff378070c70 0x7ff378073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 --> [v2:172.21.15.172:6816/114772,v1:172.21.15.172:6817/114772] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff378073730 con 0x7ff378070c70 2023-03-03T05:45:44.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff38c088090 con 0x7ff39414fd10 2023-03-03T05:45:44.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7f56627fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f566c0cb6d0 0x7f566c157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59324/0 (socket says 172.21.15.102:59324) 2023-03-03T05:45:44.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.496+0000 7f56627fc700 1 -- 172.21.15.102:0/2737663574 learned_addr learned my addr 172.21.15.102:0/2737663574 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.497+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== osd.5 v2:172.21.15.172:6824/114774 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f504c073730 con 0x7f504c070c70 2023-03-03T05:45:44.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.497+0000 7f566b7fe700 1 --2- 172.21.15.102:0/2737663574 >> v2:172.21.15.102:3300/0 conn(0x7f566c06eea0 0x7f566c1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.497+0000 7f56627fc700 1 -- 172.21.15.102:0/2737663574 >> v2:172.21.15.102:3300/0 conn(0x7f566c06eea0 msgr2=0x7f566c1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.497+0000 7f56627fc700 1 --2- 172.21.15.102:0/2737663574 >> v2:172.21.15.102:3300/0 conn(0x7f566c06eea0 0x7f566c1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.497+0000 7f56627fc700 1 -- 172.21.15.102:0/2737663574 >> v1:172.21.15.102:6789/0 conn(0x7f566c06e360 legacy=0x7f566c0c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7ff39959a700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6816/114772,v1:172.21.15.172:6817/114772] conn(0x7ff378070c70 0x7ff378073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f56627fc700 1 -- 172.21.15.102:0/2737663574 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c15dfd0 con 0x7f566c0cb6d0 2023-03-03T05:45:44.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f566b7fe700 1 --2- 172.21.15.102:0/2737663574 >> v2:172.21.15.102:3300/0 conn(0x7f566c06eea0 0x7f566c1576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:44.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f5dd1d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5dd40ce790 0x7f5dd40cd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45196/0 (socket says 172.21.15.102:45196) 2023-03-03T05:45:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f5dd1d9b700 1 -- 172.21.15.102:0/623173338 learned_addr learned my addr 172.21.15.102:0/623173338 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd415d050 con 0x7f5dd40ce790 2023-03-03T05:45:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd40cce70 con 0x7f5dd40cda10 2023-03-03T05:45:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd406b580 con 0x7f5dd406ca80 2023-03-03T05:45:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.498+0000 7f5dd259c700 1 --2- 172.21.15.102:0/623173338 >> v2:172.21.15.102:3300/0 conn(0x7f5dd406ca80 0x7f5dd406ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565156332 0 0) 0x7f5dd40cce70 con 0x7f5dd40cda10 2023-03-03T05:45:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dbc0035b0 con 0x7f5dd40cda10 2023-03-03T05:45:44.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7ff399d9b700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3780683c0 0x7ff37806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7ff39959a700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6816/114772,v1:172.21.15.172:6817/114772] conn(0x7ff378070c70 0x7ff378073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f56627fc700 1 --2- 172.21.15.102:0/2737663574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c0cb6d0 0x7f566c157170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f566c06c230 tx=0x7f5654009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d41082bcbfafeba6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f566a7fc700 1 -- 172.21.15.102:0/2737663574 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654004230 con 0x7f566c0cb6d0 2023-03-03T05:45:44.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.500+0000 7f566a7fc700 1 -- 172.21.15.102:0/2737663574 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5654004390 con 0x7f566c0cb6d0 2023-03-03T05:45:44.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.500+0000 7f566a7fc700 1 -- 172.21.15.102:0/2737663574 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56540036a0 con 0x7f566c0cb6d0 2023-03-03T05:45:44.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 732469638 0 0) 0x7f5dbc0035b0 con 0x7f5dd40cda10 2023-03-03T05:45:44.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 >> v2:172.21.15.102:3300/0 conn(0x7f5dd406ca80 msgr2=0x7f5dd406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 --2- 172.21.15.102:0/623173338 >> v2:172.21.15.102:3300/0 conn(0x7f5dd406ca80 0x7f5dd406ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 >> v1:172.21.15.102:6789/0 conn(0x7f5dd40ce790 legacy=0x7f5dd40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd415c4d0 con 0x7f5dd40cda10 2023-03-03T05:45:44.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1894587018 0 0) 0x7f5dc8001670 con 0x7f5dd40cda10 2023-03-03T05:45:44.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 >> v1:172.21.15.172:6789/0 conn(0x7f5dd40cda10 legacy=0x7f5dd4163490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dbc003a00 0x7f5dbc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dbc004850 0x7f5dbc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc005200 0x7f5dbc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd406b580 con 0x7f5dbc004850 2023-03-03T05:45:44.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc00c080 con 0x7f5dbc005200 2023-03-03T05:45:44.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.499+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc00c240 con 0x7f5dbc003a00 2023-03-03T05:45:44.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.500+0000 7f5dd259c700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dbc003a00 0x7f5dbc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== osd.4 v2:172.21.15.172:6816/114772 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff378073730 con 0x7ff378070c70 2023-03-03T05:45:44.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7ff399d9b700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3780683c0 0x7ff37806a870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff390009910 tx=0x7ff390008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f56715ff700 1 -- 172.21.15.102:0/2737663574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c0cb6d0 msgr2=0x7f566c157170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f56715ff700 1 --2- 172.21.15.102:0/2737663574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c0cb6d0 0x7f566c157170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f566c06c230 tx=0x7f5654009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.500+0000 7f5dd2d9d700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc005200 0x7f5dbc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd1d9b700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dbc004850 0x7f5dbc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd259c700 1 -- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc005200 msgr2=0x7f5dbc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd259c700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc005200 0x7f5dbc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd259c700 1 -- 172.21.15.102:0/623173338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dbc004850 msgr2=0x7f5dbc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd259c700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dbc004850 0x7f5dbc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd259c700 1 -- 172.21.15.102:0/623173338 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd415c4d0 con 0x7f5dbc003a00 2023-03-03T05:45:44.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd2d9d700 1 -- 172.21.15.102:0/623173338 reap_dead start 2023-03-03T05:45:44.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd1d9b700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dbc004850 0x7f5dbc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:44.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.501+0000 7f5dd259c700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dbc003a00 0x7f5dbc003e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009b60 tx=0x7f5dbc005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d276b6af95f2a43 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.502+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc400e6c0 con 0x7f5dbc003a00 2023-03-03T05:45:44.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.502+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc400e820 con 0x7f5dbc003a00 2023-03-03T05:45:44.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.502+0000 7f5dd159a700 1 -- 172.21.15.102:0/623173338 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc400eb90 con 0x7f5dbc003a00 2023-03-03T05:45:44.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f56715ff700 1 -- 172.21.15.102:0/2737663574 shutdown_connections 2023-03-03T05:45:44.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f56715ff700 1 --2- 172.21.15.102:0/2737663574 >> v2:172.21.15.102:3300/0 conn(0x7f566c06eea0 0x7f566c1576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f56715ff700 1 --2- 172.21.15.102:0/2737663574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c0cb6d0 0x7f566c157170 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f56715ff700 1 -- 172.21.15.102:0/2737663574 >> 172.21.15.102:0/2737663574 conn(0x7f566c06af90 msgr2=0x7f566c0caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dbc003a00 msgr2=0x7f5dbc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f5dd8fff700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dbc003a00 0x7f5dbc003e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009b60 tx=0x7f5dbc005880 comp rx=0 tx=0).stop 2023-03-03T05:45:44.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f56715ff700 1 -- 172.21.15.102:0/2737663574 shutdown_connections 2023-03-03T05:45:44.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 shutdown_connections 2023-03-03T05:45:44.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.503+0000 7f5dd8fff700 1 --2- 172.21.15.102:0/623173338 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dbc003a00 0x7f5dbc003e10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.504+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 >> 172.21.15.102:0/623173338 conn(0x7f5dd4066bd0 msgr2=0x7f5dd40ceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.511+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 --> [v2:172.21.15.102:6808/109064,v1:172.21.15.102:6809/109064] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe34c14cab0 con 0x7fe338070bf0 2023-03-03T05:45:44.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.504+0000 7f56715ff700 1 -- 172.21.15.102:0/2737663574 wait complete. 2023-03-03T05:45:44.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.507+0000 7f56715ff700 1 Processor -- start 2023-03-03T05:45:44.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.508+0000 7f56715ff700 1 -- start start 2023-03-03T05:45:44.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.509+0000 7f56715ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c06e360 0x7f566c1f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.509+0000 7f56715ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c06eea0 0x7f566c1f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.509+0000 7f56715ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f566c0cb6d0 0x7f566c14efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.509+0000 7f56715ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c1565a0 con 0x7f566c06e360 2023-03-03T05:45:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.509+0000 7f56715ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c1e40d0 con 0x7f566c06eea0 2023-03-03T05:45:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.509+0000 7f56715ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c1f5bc0 con 0x7f566c0cb6d0 2023-03-03T05:45:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f566b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f566c0cb6d0 0x7f566c14efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f566b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f566c0cb6d0 0x7f566c14efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36846/0 (socket says 172.21.15.102:36846) 2023-03-03T05:45:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f566b7fe700 1 -- 172.21.15.102:0/2872952958 learned_addr learned my addr 172.21.15.102:0/2872952958 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.511+0000 7f56627fc700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c06eea0 0x7f566c1f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.511+0000 7f566b7fe700 1 -- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c06eea0 msgr2=0x7f566c1f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.511+0000 7f566b7fe700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f566c06eea0 0x7f566c1f6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.511+0000 7f566b7fe700 1 -- 172.21.15.102:0/2872952958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c06e360 msgr2=0x7f566c1f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.511+0000 7f566b7fe700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c06e360 0x7f566c1f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.511+0000 7f566b7fe700 1 -- 172.21.15.102:0/2872952958 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5654009550 con 0x7f566c0cb6d0 2023-03-03T05:45:44.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.512+0000 7f566b7fe700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f566c0cb6d0 0x7f566c14efb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f566400ce70 tx=0x7f566400cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.512+0000 7fe3497fa700 1 -- 172.21.15.102:0/3575100997 <== osd.1 v2:172.21.15.102:6808/109064 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe340009280 con 0x7fe338070bf0 2023-03-03T05:45:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.504+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 shutdown_connections 2023-03-03T05:45:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.504+0000 7f5dd8fff700 1 -- 172.21.15.102:0/623173338 wait complete. 2023-03-03T05:45:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.508+0000 7f5dd8fff700 1 Processor -- start 2023-03-03T05:45:44.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.508+0000 7f5dd8fff700 1 -- start start 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f5dd8fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc003a00 0x7f5dd40c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f5dd8fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dd40cc090 0x7f5dd40c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f5dd8fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c8070 0x7f5dd420f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f5dd8fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40b0330 con 0x7f5dd40c8070 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f5dd8fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40c84f0 con 0x7f5dbc003a00 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.510+0000 7f5dd8fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40c76a0 con 0x7f5dd40cc090 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.512+0000 7f5dd1d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dd40cc090 0x7f5dd40c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.512+0000 7f5dd2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c8070 0x7f5dd420f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.512+0000 7f5dd2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c8070 0x7f5dd420f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36386/0 (socket says 172.21.15.102:36386) 2023-03-03T05:45:44.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.512+0000 7f5dd2d9d700 1 -- 172.21.15.102:0/3542586739 learned_addr learned my addr 172.21.15.102:0/3542586739 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:44.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5dd259c700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc003a00 0x7f5dd40c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566400ae10 con 0x7f566c0cb6d0 2023-03-03T05:45:44.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5664003a20 con 0x7f566c0cb6d0 2023-03-03T05:45:44.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664003dd0 con 0x7f566c0cb6d0 2023-03-03T05:45:44.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c0c5890 con 0x7f566c0cb6d0 2023-03-03T05:45:44.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.514+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c0c55b0 con 0x7f566c0cb6d0 2023-03-03T05:45:44.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.515+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f566c0c5b70 con 0x7f566c0cb6d0 2023-03-03T05:45:44.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5dd2d9d700 1 -- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dd40cc090 msgr2=0x7f5dd40c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5dd2d9d700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dd40cc090 0x7f5dd40c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5dd2d9d700 1 -- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc003a00 msgr2=0x7f5dd40c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5dd2d9d700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dbc003a00 0x7f5dd40c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.513+0000 7f5dd2d9d700 1 -- 172.21.15.102:0/3542586739 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd415ca90 con 0x7f5dd40c8070 2023-03-03T05:45:44.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.514+0000 7f5dd2d9d700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c8070 0x7f5dd420f790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8000fb0 tx=0x7f5dc800e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.516+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 >> [v2:172.21.15.102:6808/109064,v1:172.21.15.102:6809/109064] conn(0x7fe338070bf0 msgr2=0x7fe338073000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.516+0000 7fe352bff700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.102:6808/109064,v1:172.21.15.102:6809/109064] conn(0x7fe338070bf0 0x7fe338073000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.517+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe338068340 msgr2=0x7fe33806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.517+0000 7fe352bff700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe338068340 0x7fe33806a7f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe34c1545e0 tx=0x7fe33c008030 comp rx=0 tx=0).stop 2023-03-03T05:45:44.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.517+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe34c0717a0 msgr2=0x7fe34c15ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.517+0000 7fe352bff700 1 --2- 172.21.15.102:0/3575100997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe34c0717a0 0x7fe34c15ac90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe34400be80 tx=0x7fe34400bf60 comp rx=0 tx=0).stop 2023-03-03T05:45:44.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.517+0000 7fe35119c700 1 -- 172.21.15.102:0/3575100997 reap_dead start 2023-03-03T05:45:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.517+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 shutdown_connections 2023-03-03T05:45:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.517+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 >> 172.21.15.102:0/3575100997 conn(0x7fe34c06af90 msgr2=0x7fe34c0c5330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.518+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 shutdown_connections 2023-03-03T05:45:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.518+0000 7fe352bff700 1 -- 172.21.15.102:0/3575100997 wait complete. 2023-03-03T05:45:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.519+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 --> [v2:172.21.15.102:6816/109067,v1:172.21.15.102:6817/109067] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff398002d40 con 0x7ff394070c70 2023-03-03T05:45:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.519+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3558279061 <== osd.0 v2:172.21.15.102:6816/109067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff398002d40 con 0x7ff394070c70 2023-03-03T05:45:44.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.515+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80052e0 con 0x7f5dd40c8070 2023-03-03T05:45:44.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.515+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc8005440 con 0x7f5dd40c8070 2023-03-03T05:45:44.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.515+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80038c0 con 0x7f5dd40c8070 2023-03-03T05:45:44.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 >> [v2:172.21.15.102:6816/109067,v1:172.21.15.102:6817/109067] conn(0x7ff394070c70 msgr2=0x7ff394073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7ff3b8dff700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.102:6816/109067,v1:172.21.15.102:6817/109067] conn(0x7ff394070c70 0x7ff394073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.515+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd415c7b0 con 0x7f5dd40c8070 2023-03-03T05:45:44.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.515+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd415cd70 con 0x7f5dd40c8070 2023-03-03T05:45:44.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.516+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5dd404e6e0 con 0x7f5dd40c8070 2023-03-03T05:45:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3940683c0 msgr2=0x7ff39406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7ff3b8dff700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3940683c0 0x7ff39406a870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40c13a0 tx=0x7ff39c008040 comp rx=0 tx=0).stop 2023-03-03T05:45:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b4070790 msgr2=0x7ff3b40c09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7ff3b8dff700 1 --2- 172.21.15.102:0/3558279061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b4070790 0x7ff3b40c09f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff3a400c4a0 tx=0x7ff3a400c860 comp rx=0 tx=0).stop 2023-03-03T05:45:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7ff3b2d9d700 1 -- 172.21.15.102:0/3558279061 reap_dead start 2023-03-03T05:45:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.522+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 shutdown_connections 2023-03-03T05:45:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.522+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 >> 172.21.15.102:0/3558279061 conn(0x7ff3b406af90 msgr2=0x7ff3b40caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.522+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 shutdown_connections 2023-03-03T05:45:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.522+0000 7ff3b8dff700 1 -- 172.21.15.102:0/3558279061 wait complete. 2023-03-03T05:45:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.518+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5664003f30 con 0x7f566c0cb6d0 2023-03-03T05:45:44.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7f5663fff700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f56500683c0 0x7f565006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5664086240 con 0x7f566c0cb6d0 2023-03-03T05:45:44.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7f5663fff700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6800/114763,v1:172.21.15.172:6802/114763] conn(0x7f5650070bf0 0x7f5650073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.521+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 --> [v2:172.21.15.172:6800/114763,v1:172.21.15.172:6802/114763] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56500736b0 con 0x7f5650070bf0 2023-03-03T05:45:44.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.522+0000 7f56627fc700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6800/114763,v1:172.21.15.172:6802/114763] conn(0x7f5650070bf0 0x7f5650073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.523+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56640529d0 con 0x7f566c0cb6d0 2023-03-03T05:45:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.523+0000 7f566affd700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f56500683c0 0x7f565006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.524+0000 7f56627fc700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6800/114763,v1:172.21.15.172:6802/114763] conn(0x7f5650070bf0 0x7f5650073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.525+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== osd.6 v2:172.21.15.172:6800/114763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f56500736b0 con 0x7f5650070bf0 2023-03-03T05:45:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.525+0000 7f566affd700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f56500683c0 0x7f565006a870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f565c003df0 tx=0x7f565c007a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.522+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5dc801b030 con 0x7f5dd40c8070 2023-03-03T05:45:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.524+0000 7f5dc2ffd700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5db40683c0 0x7f5db406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.524+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5dc8085ae0 con 0x7f5dd40c8070 2023-03-03T05:45:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.525+0000 7f5dc2ffd700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6801/114764,v1:172.21.15.172:6803/114764] conn(0x7f5db4070c70 0x7f5db4073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.525+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 --> [v2:172.21.15.172:6801/114764,v1:172.21.15.172:6803/114764] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5db4073730 con 0x7f5db4070c70 2023-03-03T05:45:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.525+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dc8085e90 con 0x7f5dd40c8070 2023-03-03T05:45:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.526+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 --> [v2:172.21.15.102:6800/109066,v1:172.21.15.102:6801/109066] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1cc8002d40 con 0x7f1cc4070c70 2023-03-03T05:45:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.526+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 --> [v2:172.21.15.102:6818/109068,v1:172.21.15.102:6821/109068] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f454804e6e0 con 0x7f452c070c70 2023-03-03T05:45:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.527+0000 7f1cd2ffd700 1 -- 172.21.15.102:0/142085625 <== osd.3 v2:172.21.15.102:6800/109066 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1cc8002d40 con 0x7f1cc4070c70 2023-03-03T05:45:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.527+0000 7f5dd1d9b700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6801/114764,v1:172.21.15.172:6803/114764] conn(0x7f5db4070c70 0x7f5db4073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.527+0000 7f453dffb700 1 -- 172.21.15.102:0/407117681 <== osd.2 v2:172.21.15.102:6818/109068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f454804e6e0 con 0x7f452c070c70 2023-03-03T05:45:44.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.527+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:6800/109066,v1:172.21.15.102:6801/109066] conn(0x7f1cc4070c70 msgr2=0x7f1cc4073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.527+0000 7f1ce8bff700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.102:6800/109066,v1:172.21.15.102:6801/109066] conn(0x7f1cc4070c70 0x7f1cc4073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f5dd259c700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5db40683c0 0x7f5db406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:44.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:6818/109068,v1:172.21.15.102:6821/109068] conn(0x7f452c070c70 msgr2=0x7f452c073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f454e79d700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.102:6818/109068,v1:172.21.15.102:6821/109068] conn(0x7f452c070c70 0x7f452c073080 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f452c0683c0 msgr2=0x7f452c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f454e79d700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f452c0683c0 0x7f452c06a870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4544009840 tx=0x7f4544008030 comp rx=0 tx=0).stop 2023-03-03T05:45:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45481572a0 msgr2=0x7f45480c3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f454e79d700 1 --2- 172.21.15.102:0/407117681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45481572a0 0x7f45480c3950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f454000efc0 tx=0x7f454000c3a0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cc40683c0 msgr2=0x7f1cc406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f1ce8bff700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cc40683c0 0x7f1cc406a870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ce406fbe0 tx=0x7f1cd4008030 comp rx=0 tx=0).stop 2023-03-03T05:45:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce41e2990 msgr2=0x7f1ce414b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f1ce8bff700 1 --2- 172.21.15.102:0/142085625 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ce41e2990 0x7f1ce414b170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00efc0 tx=0x7f1cdc00c0c0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.528+0000 7f1ce2d9d700 1 -- 172.21.15.102:0/142085625 reap_dead start 2023-03-03T05:45:44.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 shutdown_connections 2023-03-03T05:45:44.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 >> 172.21.15.102:0/142085625 conn(0x7f1ce406af90 msgr2=0x7f1ce4151cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f454df9c700 1 -- 172.21.15.102:0/407117681 reap_dead start 2023-03-03T05:45:44.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 shutdown_connections 2023-03-03T05:45:44.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 shutdown_connections 2023-03-03T05:45:44.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 >> 172.21.15.102:0/407117681 conn(0x7f4548065fb0 msgr2=0x7f45480c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f1ce8bff700 1 -- 172.21.15.102:0/142085625 wait complete. 2023-03-03T05:45:44.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 shutdown_connections 2023-03-03T05:45:44.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.529+0000 7f454e79d700 1 -- 172.21.15.102:0/407117681 wait complete. 2023-03-03T05:45:44.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.530+0000 7f5dd259c700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5db40683c0 0x7f5db406a870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000c00 tx=0x7f5dc4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.530+0000 7f5dd1d9b700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6801/114764,v1:172.21.15.172:6803/114764] conn(0x7f5db4070c70 0x7f5db4073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:44.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.531+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== osd.7 v2:172.21.15.172:6801/114764 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5db4073730 con 0x7f5db4070c70 2023-03-03T05:45:44.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.533+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 --> [v2:172.21.15.172:6824/114774,v1:172.21.15.172:6825/114774] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5038002d40 con 0x7f504c070c70 2023-03-03T05:45:44.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.534+0000 7f50477fe700 1 -- 172.21.15.102:0/1841799798 <== osd.5 v2:172.21.15.172:6824/114774 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5038002d40 con 0x7f504c070c70 2023-03-03T05:45:44.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.534+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6824/114774,v1:172.21.15.172:6825/114774] conn(0x7f504c070c70 msgr2=0x7f504c073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.534+0000 7f50457fa700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6824/114774,v1:172.21.15.172:6825/114774] conn(0x7f504c070c70 0x7f504c073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f504c0683c0 msgr2=0x7f504c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f504c0683c0 0x7f504c06a870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f50600c91f0 tx=0x7f5054008040 comp rx=0 tx=0).stop 2023-03-03T05:45:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f506006ed00 msgr2=0x7f5060155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 --2- 172.21.15.102:0/1841799798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f506006ed00 0x7f5060155760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5058000f80 tx=0x7f5058004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f505f7fe700 1 -- 172.21.15.102:0/1841799798 reap_dead start 2023-03-03T05:45:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 shutdown_connections 2023-03-03T05:45:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 >> 172.21.15.102:0/1841799798 conn(0x7f506006af90 msgr2=0x7f50600c5330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 shutdown_connections 2023-03-03T05:45:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.535+0000 7f50457fa700 1 -- 172.21.15.102:0/1841799798 wait complete. 2023-03-03T05:45:44.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.537+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 --> [v2:172.21.15.172:6816/114772,v1:172.21.15.172:6817/114772] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3941474d0 con 0x7ff378070c70 2023-03-03T05:45:44.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.537+0000 7ff38a7fc700 1 -- 172.21.15.102:0/1056362018 <== osd.4 v2:172.21.15.172:6816/114772 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff3941474d0 con 0x7ff378070c70 2023-03-03T05:45:44.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.539+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6816/114772,v1:172.21.15.172:6817/114772] conn(0x7ff378070c70 msgr2=0x7ff378073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.539+0000 7ff39bfff700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6816/114772,v1:172.21.15.172:6817/114772] conn(0x7ff378070c70 0x7ff378073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.540+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3780683c0 msgr2=0x7ff37806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.540+0000 7ff39bfff700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff3780683c0 0x7ff37806a870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff390009910 tx=0x7ff390008040 comp rx=0 tx=0).stop 2023-03-03T05:45:44.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.540+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff39414fd10 msgr2=0x7ff3940c3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.540+0000 7ff39bfff700 1 --2- 172.21.15.102:0/1056362018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff39414fd10 0x7ff3940c3a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3940726e0 tx=0x7ff38c004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.540+0000 7ff39a59c700 1 -- 172.21.15.102:0/1056362018 reap_dead start 2023-03-03T05:45:44.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.540+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 shutdown_connections 2023-03-03T05:45:44.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.540+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 >> 172.21.15.102:0/1056362018 conn(0x7ff39406af90 msgr2=0x7ff3940c5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.541+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 shutdown_connections 2023-03-03T05:45:44.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.541+0000 7ff39bfff700 1 -- 172.21.15.102:0/1056362018 wait complete. 2023-03-03T05:45:44.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.553+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 --> [v2:172.21.15.172:6800/114763,v1:172.21.15.172:6802/114763] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f566c04e6e0 con 0x7f5650070bf0 2023-03-03T05:45:44.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.554+0000 7f5663fff700 1 -- 172.21.15.102:0/2872952958 <== osd.6 v2:172.21.15.172:6800/114763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f566c04e6e0 con 0x7f5650070bf0 2023-03-03T05:45:44.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.555+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6800/114763,v1:172.21.15.172:6802/114763] conn(0x7f5650070bf0 msgr2=0x7f5650073000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.555+0000 7f56715ff700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6800/114763,v1:172.21.15.172:6802/114763] conn(0x7f5650070bf0 0x7f5650073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.557+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f56500683c0 msgr2=0x7f565006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.557+0000 7f56715ff700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f56500683c0 0x7f565006a870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f565c003df0 tx=0x7f565c007a40 comp rx=0 tx=0).stop 2023-03-03T05:45:44.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.557+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f566c0cb6d0 msgr2=0x7f566c14efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.557+0000 7f56715ff700 1 --2- 172.21.15.102:0/2872952958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f566c0cb6d0 0x7f566c14efb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f566400ce70 tx=0x7f566400cf50 comp rx=0 tx=0).stop 2023-03-03T05:45:44.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.559+0000 7f566b7fe700 1 -- 172.21.15.102:0/2872952958 reap_dead start 2023-03-03T05:45:44.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.560+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 shutdown_connections 2023-03-03T05:45:44.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.560+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 >> 172.21.15.102:0/2872952958 conn(0x7f566c06af90 msgr2=0x7f566c152a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.585 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2023-03-03T05:45:44.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.562+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 shutdown_connections 2023-03-03T05:45:44.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-03T05:45:44.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.565+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 --> [v2:172.21.15.172:6801/114764,v1:172.21.15.172:6803/114764] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5dd40c77e0 con 0x7f5db4070c70 2023-03-03T05:45:44.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.565+0000 7f56715ff700 1 -- 172.21.15.102:0/2872952958 wait complete. 2023-03-03T05:45:44.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.565+0000 7f5dc2ffd700 1 -- 172.21.15.102:0/3542586739 <== osd.7 v2:172.21.15.172:6801/114764 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5dd40c77e0 con 0x7f5db4070c70 2023-03-03T05:45:44.588 INFO:teuthology.orchestra.run.smithi102.stdout:55834574856 2023-03-03T05:45:44.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-03T05:45:44.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.566+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6801/114764,v1:172.21.15.172:6803/114764] conn(0x7f5db4070c70 msgr2=0x7f5db4073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.566+0000 7f5dd8fff700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6801/114764,v1:172.21.15.172:6803/114764] conn(0x7f5db4070c70 0x7f5db4073080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.566+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5db40683c0 msgr2=0x7f5db406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.566+0000 7f5dd8fff700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5db40683c0 0x7f5db406a870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000c00 tx=0x7f5dc4008030 comp rx=0 tx=0).stop 2023-03-03T05:45:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.567+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c8070 msgr2=0x7f5dd420f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.567+0000 7f5dd8fff700 1 --2- 172.21.15.102:0/3542586739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c8070 0x7f5dd420f790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8000fb0 tx=0x7f5dc800e9e0 comp rx=0 tx=0).stop 2023-03-03T05:45:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.567+0000 7f5dd2d9d700 1 -- 172.21.15.102:0/3542586739 reap_dead start 2023-03-03T05:45:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.567+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 shutdown_connections 2023-03-03T05:45:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.567+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 >> 172.21.15.102:0/3542586739 conn(0x7f5dd4066bd0 msgr2=0x7f5dd41522a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:44.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.567+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 shutdown_connections 2023-03-03T05:45:44.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:44.567+0000 7f5dd8fff700 1 -- 172.21.15.102:0/3542586739 wait complete. 2023-03-03T05:45:44.591 INFO:teuthology.orchestra.run.smithi102.stdout:55834574856 2023-03-03T05:45:44.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-03-03T05:45:44.592 INFO:teuthology.orchestra.run.smithi102.stdout:55834574857 2023-03-03T05:45:44.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-03-03T05:45:44.593 INFO:teuthology.orchestra.run.smithi102.stdout:55834574856 2023-03-03T05:45:44.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-03-03T05:45:44.595 INFO:teuthology.orchestra.run.smithi102.stdout:64424509449 2023-03-03T05:45:44.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-03T05:45:44.599 INFO:teuthology.orchestra.run.smithi102.stdout:55834574856 2023-03-03T05:45:44.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-03-03T05:45:44.605 INFO:teuthology.orchestra.run.smithi102.stdout:55834574857 2023-03-03T05:45:44.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-03-03T05:45:45.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.038+0000 7fed9679d700 1 Processor -- start 2023-03-03T05:45:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.038+0000 7fed9679d700 1 -- start start 2023-03-03T05:45:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.041+0000 7fed9679d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed9014ede0 0x7fed90071f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.041+0000 7fed9679d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fed900c2a10 0x7fed90158b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.041+0000 7fed9679d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900709c0 con 0x7fed900c5410 2023-03-03T05:45:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.041+0000 7fed9679d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed90159090 con 0x7fed9014ede0 2023-03-03T05:45:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.041+0000 7fed9679d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed900c2e90 con 0x7fed900c2a10 2023-03-03T05:45:45.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.042+0000 7fed94f9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fed900c5410 0x7fed900ca870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45210/0 (socket says 172.21.15.102:45210) 2023-03-03T05:45:45.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.042+0000 7fed94f9a700 1 -- 172.21.15.102:0/3880182265 learned_addr learned my addr 172.21.15.102:0/3880182265 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:45.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.043+0000 7fed95f9c700 1 --2- 172.21.15.102:0/3880182265 >> v2:172.21.15.172:3300/0 conn(0x7fed900c2a10 0x7fed90158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.043+0000 7fed9579b700 1 --2- 172.21.15.102:0/3880182265 >> v2:172.21.15.102:3300/0 conn(0x7fed9014ede0 0x7fed90071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.044+0000 7fed95f9c700 1 -- 172.21.15.102:0/3880182265 >> v2:172.21.15.102:3300/0 conn(0x7fed9014ede0 msgr2=0x7fed90071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.044+0000 7fed95f9c700 1 --2- 172.21.15.102:0/3880182265 >> v2:172.21.15.102:3300/0 conn(0x7fed9014ede0 0x7fed90071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.044+0000 7fed95f9c700 1 -- 172.21.15.102:0/3880182265 >> v1:172.21.15.102:6789/0 conn(0x7fed900c5410 legacy=0x7fed900ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.044+0000 7fed95f9c700 1 -- 172.21.15.102:0/3880182265 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed90070bf0 con 0x7fed900c2a10 2023-03-03T05:45:45.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.045+0000 7fed95f9c700 1 --2- 172.21.15.102:0/3880182265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90158b50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009840 tx=0x7fed8c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ec5d9902389cda server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.046+0000 7fed87fff700 1 -- 172.21.15.102:0/3880182265 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c004230 con 0x7fed900c2a10 2023-03-03T05:45:45.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.046+0000 7fed87fff700 1 -- 172.21.15.102:0/3880182265 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8c004390 con 0x7fed900c2a10 2023-03-03T05:45:45.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.046+0000 7fed87fff700 1 -- 172.21.15.102:0/3880182265 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c00f740 con 0x7fed900c2a10 2023-03-03T05:45:45.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.047+0000 7fed9679d700 1 -- 172.21.15.102:0/3880182265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 msgr2=0x7fed90158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.047+0000 7fed9679d700 1 --2- 172.21.15.102:0/3880182265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90158b50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009840 tx=0x7fed8c009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:45.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.047+0000 7f96159ff700 1 Processor -- start 2023-03-03T05:45:45.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.047+0000 7f96159ff700 1 -- start start 2023-03-03T05:45:45.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.049+0000 7fed9679d700 1 -- 172.21.15.102:0/3880182265 shutdown_connections 2023-03-03T05:45:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.049+0000 7fed9679d700 1 --2- 172.21.15.102:0/3880182265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90158b50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.049+0000 7fed9679d700 1 --2- 172.21.15.102:0/3880182265 >> v2:172.21.15.102:3300/0 conn(0x7fed9014ede0 0x7fed90071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.049+0000 7fed9679d700 1 -- 172.21.15.102:0/3880182265 >> 172.21.15.102:0/3880182265 conn(0x7fed90065fb0 msgr2=0x7fed9006a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.049+0000 7fed9679d700 1 -- 172.21.15.102:0/3880182265 shutdown_connections 2023-03-03T05:45:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.049+0000 7fed9679d700 1 -- 172.21.15.102:0/3880182265 wait complete. 2023-03-03T05:45:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.050+0000 7f96159ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f961006d460 0x7f9610155b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.050+0000 7f96159ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f96100c3ee0 0x7f9610071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.050+0000 7f96159ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96100c46e0 con 0x7f96100c3b10 2023-03-03T05:45:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.050+0000 7f96159ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96100c4ef0 con 0x7f961006d460 2023-03-03T05:45:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.050+0000 7f96159ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9610070f40 con 0x7f96100c3ee0 2023-03-03T05:45:45.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.052+0000 7f960e7fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f961006d460 0x7f9610155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.052+0000 7f960f7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f96100c3ee0 0x7f9610071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.052+0000 7f960e7fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f961006d460 0x7f9610155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36420/0 (socket says 172.21.15.102:36420) 2023-03-03T05:45:45.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.052+0000 7f960e7fc700 1 -- 172.21.15.102:0/3064020796 learned_addr learned my addr 172.21.15.102:0/3064020796 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.053+0000 7f960e7fc700 1 -- 172.21.15.102:0/3064020796 >> v2:172.21.15.172:3300/0 conn(0x7f96100c3ee0 msgr2=0x7f9610071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.053+0000 7f960e7fc700 1 --2- 172.21.15.102:0/3064020796 >> v2:172.21.15.172:3300/0 conn(0x7f96100c3ee0 0x7f9610071700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.053+0000 7f960e7fc700 1 -- 172.21.15.102:0/3064020796 >> v1:172.21.15.172:6789/0 conn(0x7f96100c3b10 legacy=0x7f961014f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.053+0000 7f960e7fc700 1 -- 172.21.15.102:0/3064020796 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96100c4930 con 0x7f961006d460 2023-03-03T05:45:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.054+0000 7fed9679d700 1 Processor -- start 2023-03-03T05:45:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.054+0000 7fed9679d700 1 -- start start 2023-03-03T05:45:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.054+0000 7f960e7fc700 1 --2- 172.21.15.102:0/3064020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961006d460 0x7f9610155b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9610072ad0 tx=0x7f960000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18fbd0241475b7d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.055+0000 7f960dffb700 1 -- 172.21.15.102:0/3064020796 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9600004230 con 0x7f961006d460 2023-03-03T05:45:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.055+0000 7f960dffb700 1 -- 172.21.15.102:0/3064020796 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9600004390 con 0x7f961006d460 2023-03-03T05:45:45.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.055+0000 7f960dffb700 1 -- 172.21.15.102:0/3064020796 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960000c830 con 0x7f961006d460 2023-03-03T05:45:45.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9679d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9679d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed900c5410 0x7fed90157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9679d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9014ede0 0x7fed90146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9679d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9014dbf0 con 0x7fed900c5410 2023-03-03T05:45:45.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9679d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9014e010 con 0x7fed900c2a10 2023-03-03T05:45:45.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9679d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed90155e30 con 0x7fed9014ede0 2023-03-03T05:45:45.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7f96159ff700 1 -- 172.21.15.102:0/3064020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961006d460 msgr2=0x7f9610155b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7f96159ff700 1 --2- 172.21.15.102:0/3064020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961006d460 0x7f9610155b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9610072ad0 tx=0x7f960000bc80 comp rx=0 tx=0).stop 2023-03-03T05:45:45.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7f96159ff700 1 -- 172.21.15.102:0/3064020796 shutdown_connections 2023-03-03T05:45:45.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7f96159ff700 1 --2- 172.21.15.102:0/3064020796 >> v2:172.21.15.172:3300/0 conn(0x7f96100c3ee0 0x7f9610071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7f96159ff700 1 --2- 172.21.15.102:0/3064020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961006d460 0x7f9610155b50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7f96159ff700 1 -- 172.21.15.102:0/3064020796 >> 172.21.15.102:0/3064020796 conn(0x7f961006af90 msgr2=0x7f96100caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9579b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9579b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59374/0 (socket says 172.21.15.102:59374) 2023-03-03T05:45:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.056+0000 7fed9579b700 1 -- 172.21.15.102:0/449082342 learned_addr learned my addr 172.21.15.102:0/449082342 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.057+0000 7fed94f9a700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed900c5410 0x7fed90157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.057+0000 7fed9579b700 1 -- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9014ede0 msgr2=0x7fed90146b80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.057+0000 7fed9579b700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9014ede0 0x7fed90146b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.057+0000 7fed9579b700 1 -- 172.21.15.102:0/449082342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed900c5410 msgr2=0x7fed90157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.057+0000 7fed9579b700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed900c5410 0x7fed90157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.057+0000 7fed9579b700 1 -- 172.21.15.102:0/449082342 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c0094a0 con 0x7fed900c2a10 2023-03-03T05:45:45.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.057+0000 7f28acbff700 1 Processor -- start 2023-03-03T05:45:45.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.058+0000 7fed9579b700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90156df0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fed8000e8a0 tx=0x7fed8000ec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.058+0000 7fed85ffb700 1 -- 172.21.15.102:0/449082342 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80009590 con 0x7fed900c2a10 2023-03-03T05:45:45.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.058+0000 7fed85ffb700 1 -- 172.21.15.102:0/449082342 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8000ee40 con 0x7fed900c2a10 2023-03-03T05:45:45.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.058+0000 7fed85ffb700 1 -- 172.21.15.102:0/449082342 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8000ca70 con 0x7fed900c2a10 2023-03-03T05:45:45.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.058+0000 7f28acbff700 1 -- start start 2023-03-03T05:45:45.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.060+0000 7f28acbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28a814f120 0x7f28a8070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.061+0000 7f28acbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f28a8158470 0x7f28a81588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.061+0000 7f28acbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80c64a0 con 0x7f28a806ed00 2023-03-03T05:45:45.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.061+0000 7f28acbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a80c2430 con 0x7f28a8158470 2023-03-03T05:45:45.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.061+0000 7f28acbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a806f110 con 0x7f28a814f120 2023-03-03T05:45:45.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.059+0000 7f96159ff700 1 -- 172.21.15.102:0/3064020796 shutdown_connections 2023-03-03T05:45:45.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.060+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed90071190 con 0x7fed900c2a10 2023-03-03T05:45:45.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.060+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed90070eb0 con 0x7fed900c2a10 2023-03-03T05:45:45.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.061+0000 7f28a5d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f28a806ed00 0x7f28a80c5430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36492/0 (socket says 172.21.15.102:36492) 2023-03-03T05:45:45.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.061+0000 7f28a5d9b700 1 -- 172.21.15.102:0/2252029325 learned_addr learned my addr 172.21.15.102:0/2252029325 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:45.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.061+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed90071470 con 0x7fed900c2a10 2023-03-03T05:45:45.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.062+0000 7f28a659c700 1 --2- 172.21.15.102:0/2252029325 >> v2:172.21.15.172:3300/0 conn(0x7f28a814f120 0x7f28a8070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a559a700 1 -- 172.21.15.102:0/2252029325 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246145089 0 0) 0x7f28a80c64a0 con 0x7f28a806ed00 2023-03-03T05:45:45.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a559a700 1 -- 172.21.15.102:0/2252029325 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28900057c0 con 0x7f28a806ed00 2023-03-03T05:45:45.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a6d9d700 1 --2- 172.21.15.102:0/2252029325 >> v2:172.21.15.102:3300/0 conn(0x7f28a8158470 0x7f28a81588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a659c700 1 -- 172.21.15.102:0/2252029325 >> v2:172.21.15.102:3300/0 conn(0x7f28a8158470 msgr2=0x7f28a81588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a659c700 1 --2- 172.21.15.102:0/2252029325 >> v2:172.21.15.102:3300/0 conn(0x7f28a8158470 0x7f28a81588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a659c700 1 -- 172.21.15.102:0/2252029325 >> v1:172.21.15.172:6789/0 conn(0x7f28a806ed00 legacy=0x7f28a80c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a659c700 1 -- 172.21.15.102:0/2252029325 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a80c5ee0 con 0x7f28a814f120 2023-03-03T05:45:45.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a6d9d700 1 --2- 172.21.15.102:0/2252029325 >> v2:172.21.15.102:3300/0 conn(0x7f28a8158470 0x7f28a81588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:45.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.063+0000 7f28a659c700 1 --2- 172.21.15.102:0/2252029325 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a814f120 0x7f28a8070790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28a80c26a0 tx=0x7f2898009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=325e0c492f799f10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.060+0000 7f96159ff700 1 -- 172.21.15.102:0/3064020796 wait complete. 2023-03-03T05:45:45.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.066+0000 7f96159ff700 1 Processor -- start 2023-03-03T05:45:45.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.066+0000 7f96159ff700 1 -- start start 2023-03-03T05:45:45.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.065+0000 7fed85ffb700 1 -- 172.21.15.102:0/449082342 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed80019030 con 0x7fed900c2a10 2023-03-03T05:45:45.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.068+0000 7fed85ffb700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed740683c0 0x7fed7406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.068+0000 7f28a559a700 1 -- 172.21.15.102:0/2252029325 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2898010620 con 0x7f28a814f120 2023-03-03T05:45:45.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.068+0000 7f28a559a700 1 -- 172.21.15.102:0/2252029325 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2898010780 con 0x7f28a814f120 2023-03-03T05:45:45.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.068+0000 7f28a559a700 1 -- 172.21.15.102:0/2252029325 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2898010b10 con 0x7f28a814f120 2023-03-03T05:45:45.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.068+0000 7fed85ffb700 1 -- 172.21.15.102:0/449082342 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fed80086300 con 0x7fed900c2a10 2023-03-03T05:45:45.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.069+0000 7fed94f9a700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed740683c0 0x7fed7406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.069+0000 7fed85ffb700 1 -- 172.21.15.102:0/449082342 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed80056520 con 0x7fed900c2a10 2023-03-03T05:45:45.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.069+0000 7f28acbff700 1 -- 172.21.15.102:0/2252029325 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a814f120 msgr2=0x7f28a8070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.069+0000 7f28acbff700 1 --2- 172.21.15.102:0/2252029325 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a814f120 0x7f28a8070790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28a80c26a0 tx=0x7f2898009b20 comp rx=0 tx=0).stop 2023-03-03T05:45:45.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7fed94f9a700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed740683c0 0x7fed7406a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fed901e6f90 tx=0x7fed88001ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f28acbff700 1 -- 172.21.15.102:0/2252029325 shutdown_connections 2023-03-03T05:45:45.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f28acbff700 1 --2- 172.21.15.102:0/2252029325 >> v2:172.21.15.102:3300/0 conn(0x7f28a8158470 0x7f28a81588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f28acbff700 1 --2- 172.21.15.102:0/2252029325 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a814f120 0x7f28a8070790 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f28acbff700 1 -- 172.21.15.102:0/2252029325 >> 172.21.15.102:0/2252029325 conn(0x7f28a806af90 msgr2=0x7f28a80caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f96159ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96100c3b10 0x7f9610153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f96159ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96100c3ee0 0x7f96101542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f96159ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9610153820 0x7f96101fed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f96159ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96101ff9f0 con 0x7f96100c3ee0 2023-03-03T05:45:45.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f96159ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9610159db0 con 0x7f96100c3b10 2023-03-03T05:45:45.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f96159ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9610155970 con 0x7f9610153820 2023-03-03T05:45:45.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f960f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9610153820 0x7f96101fed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f960f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9610153820 0x7f96101fed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36876/0 (socket says 172.21.15.102:36876) 2023-03-03T05:45:45.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.070+0000 7f960f7fe700 1 -- 172.21.15.102:0/1861540091 learned_addr learned my addr 172.21.15.102:0/1861540091 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f960effd700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96100c3b10 0x7f9610153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f28acbff700 1 -- 172.21.15.102:0/2252029325 shutdown_connections 2023-03-03T05:45:45.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f28acbff700 1 -- 172.21.15.102:0/2252029325 wait complete. 2023-03-03T05:45:45.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f960f7fe700 1 -- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96100c3b10 msgr2=0x7f9610153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f960f7fe700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96100c3b10 0x7f9610153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f960e7fc700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96100c3ee0 0x7f96101542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f960f7fe700 1 -- 172.21.15.102:0/1861540091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96100c3ee0 msgr2=0x7f96101542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f960f7fe700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96100c3ee0 0x7f96101542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.071+0000 7f960f7fe700 1 -- 172.21.15.102:0/1861540091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960000b600 con 0x7f9610153820 2023-03-03T05:45:45.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.072+0000 7f960e7fc700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96100c3ee0 0x7f96101542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:45.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.072+0000 7f960f7fe700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9610153820 0x7f96101fed00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9608007470 tx=0x7f9608007780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.072+0000 7f7705bff700 1 Processor -- start 2023-03-03T05:45:45.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.072+0000 7f9607fff700 1 -- 172.21.15.102:0/1861540091 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608007ea0 con 0x7f9610153820 2023-03-03T05:45:45.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.072+0000 7f9607fff700 1 -- 172.21.15.102:0/1861540091 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9608003c20 con 0x7f9610153820 2023-03-03T05:45:45.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.072+0000 7f9607fff700 1 -- 172.21.15.102:0/1861540091 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960800d110 con 0x7f9610153820 2023-03-03T05:45:45.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.073+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96100711e0 con 0x7f9610153820 2023-03-03T05:45:45.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.073+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96100c4c10 con 0x7f9610153820 2023-03-03T05:45:45.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.072+0000 7f7705bff700 1 -- start start 2023-03-03T05:45:45.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.074+0000 7fd7cadff700 1 Processor -- start 2023-03-03T05:45:45.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.074+0000 7fd7cadff700 1 -- start start 2023-03-03T05:45:45.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.074+0000 7f28acbff700 1 Processor -- start 2023-03-03T05:45:45.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.075+0000 7f28acbff700 1 -- start start 2023-03-03T05:45:45.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.074+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96100cb230 con 0x7f9610153820 2023-03-03T05:45:45.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f7705bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77000cb1c0 0x7f7700070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f7705bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f770006ed00 0x7f770014ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f7705bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77000c64a0 con 0x7f770014f480 2023-03-03T05:45:45.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f7705bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77000c2430 con 0x7f770006ed00 2023-03-03T05:45:45.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f7705bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770006f110 con 0x7f77000cb1c0 2023-03-03T05:45:45.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7fd7cadff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd7c406ca80 0x7fd7c406ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7fd7cadff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7c40cda10 0x7fd7c4163490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7fd7cadff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c415d050 con 0x7fd7c40ce790 2023-03-03T05:45:45.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7fd7cadff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c406b580 con 0x7fd7c40cda10 2023-03-03T05:45:45.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7fd7cadff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c406d390 con 0x7fd7c406ca80 2023-03-03T05:45:45.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f76fffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f770014f480 0x7f77000c5430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36494/0 (socket says 172.21.15.102:36494) 2023-03-03T05:45:45.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7f76fffff700 1 -- 172.21.15.102:0/132657472 learned_addr learned my addr 172.21.15.102:0/132657472 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:45.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f28acbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28a806ed00 0x7f28a81f6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.074+0000 7f2e8d3ff700 1 Processor -- start 2023-03-03T05:45:45.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f28acbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28a814f120 0x7f28a8151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f28acbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a8158470 0x7f28a81e87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f28acbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a80c9c70 con 0x7f28a806ed00 2023-03-03T05:45:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f28acbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8155ae0 con 0x7f28a8158470 2023-03-03T05:45:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.076+0000 7f28acbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8150490 con 0x7f28a814f120 2023-03-03T05:45:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7f28a5d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28a814f120 0x7f28a8151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7f28a5d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28a814f120 0x7f28a8151990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36894/0 (socket says 172.21.15.102:36894) 2023-03-03T05:45:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7f28a5d9b700 1 -- 172.21.15.102:0/2081085464 learned_addr learned my addr 172.21.15.102:0/2081085464 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7f28a6d9d700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a8158470 0x7f28a81e87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f28a659c700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28a806ed00 0x7f28a81f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7f28a659c700 1 -- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28a814f120 msgr2=0x7f28a8151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7f28a659c700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28a814f120 0x7f28a8151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7f28a659c700 1 -- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a8158470 msgr2=0x7f28a81e87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7f28a659c700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a8158470 0x7f28a81e87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7f28a659c700 1 -- 172.21.15.102:0/2081085464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28980094a0 con 0x7f28a806ed00 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7f28a6d9d700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a8158470 0x7f28a81e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:45.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7f28a659c700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28a806ed00 0x7f28a81f6900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2898016000 tx=0x7f2898004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7fd7c8b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd7c406ca80 0x7fd7c406ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7fd7c8b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd7c406ca80 0x7fd7c406ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59400/0 (socket says 172.21.15.102:59400) 2023-03-03T05:45:45.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7fd7c8b9b700 1 -- 172.21.15.102:0/3783486087 learned_addr learned my addr 172.21.15.102:0/3783486087 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7fd7c939c700 1 --2- 172.21.15.102:0/3783486087 >> v2:172.21.15.102:3300/0 conn(0x7fd7c40cda10 0x7fd7c4163490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7fd7c8b9b700 1 -- 172.21.15.102:0/3783486087 >> v2:172.21.15.102:3300/0 conn(0x7fd7c40cda10 msgr2=0x7fd7c4163490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7fd7c8b9b700 1 --2- 172.21.15.102:0/3783486087 >> v2:172.21.15.102:3300/0 conn(0x7fd7c40cda10 0x7fd7c4163490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7fd7c8b9b700 1 -- 172.21.15.102:0/3783486087 >> v1:172.21.15.172:6789/0 conn(0x7fd7c40ce790 legacy=0x7fd7c40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7fd7c8b9b700 1 -- 172.21.15.102:0/3783486087 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40cce70 con 0x7fd7c406ca80 2023-03-03T05:45:45.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.079+0000 7fd7c8b9b700 1 --2- 172.21.15.102:0/3783486087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c406ca80 0x7fd7c406ce50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7c406d5f0 tx=0x7fd7c000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e173d93973d383 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.080+0000 7fd7bb7fe700 1 -- 172.21.15.102:0/3783486087 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c000e070 con 0x7fd7c406ca80 2023-03-03T05:45:45.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.080+0000 7fd7bb7fe700 1 -- 172.21.15.102:0/3783486087 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c0003a20 con 0x7fd7c406ca80 2023-03-03T05:45:45.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.080+0000 7fd7bb7fe700 1 -- 172.21.15.102:0/3783486087 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0003dd0 con 0x7fd7c406ca80 2023-03-03T05:45:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7f76ff7fe700 1 --2- 172.21.15.102:0/132657472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77000cb1c0 0x7f7700070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.077+0000 7f76feffd700 1 --2- 172.21.15.102:0/132657472 >> v2:172.21.15.102:3300/0 conn(0x7f770006ed00 0x7f770014ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f76ff7fe700 1 -- 172.21.15.102:0/132657472 >> v2:172.21.15.102:3300/0 conn(0x7f770006ed00 msgr2=0x7f770014ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f76ff7fe700 1 --2- 172.21.15.102:0/132657472 >> v2:172.21.15.102:3300/0 conn(0x7f770006ed00 0x7f770014ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f76ff7fe700 1 -- 172.21.15.102:0/132657472 >> v1:172.21.15.172:6789/0 conn(0x7f770014f480 legacy=0x7f77000c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f76ff7fe700 1 -- 172.21.15.102:0/132657472 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77000c61c0 con 0x7f77000cb1c0 2023-03-03T05:45:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f76ff7fe700 1 --2- 172.21.15.102:0/132657472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77000cb1c0 0x7f7700070790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009840 tx=0x7f76f0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d922d5b39790a10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7fd7cadff700 1 -- 172.21.15.102:0/3783486087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c406ca80 msgr2=0x7fd7c406ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7fd7cadff700 1 --2- 172.21.15.102:0/3783486087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c406ca80 0x7fd7c406ce50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7c406d5f0 tx=0x7fd7c000b330 comp rx=0 tx=0).stop 2023-03-03T05:45:45.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f2896ffd700 1 -- 172.21.15.102:0/2081085464 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289801e070 con 0x7f28a806ed00 2023-03-03T05:45:45.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f2896ffd700 1 -- 172.21.15.102:0/2081085464 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2898004500 con 0x7f28a806ed00 2023-03-03T05:45:45.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f2896ffd700 1 -- 172.21.15.102:0/2081085464 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2898010b10 con 0x7f28a806ed00 2023-03-03T05:45:45.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f9607fff700 1 -- 172.21.15.102:0/1861540091 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9608014070 con 0x7f9610153820 2023-03-03T05:45:45.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.080+0000 7f9607fff700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95f00683c0 0x7f95f006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.080+0000 7f9607fff700 1 -- 172.21.15.102:0/1861540091 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f96080868b0 con 0x7f9610153820 2023-03-03T05:45:45.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f960effd700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95f00683c0 0x7f95f006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f9607fff700 1 -- 172.21.15.102:0/1861540091 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f96080530a0 con 0x7f9610153820 2023-03-03T05:45:45.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.078+0000 7f2e8d3ff700 1 -- start start 2023-03-03T05:45:45.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f2e8d3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e8806e360 0x7f2e8806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f2e8d3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e880c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f2e8d3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e880c4750 0x7f2e880c4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f2e8d3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e88157900 con 0x7f2e8806e360 2023-03-03T05:45:45.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f2e8d3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e880c4e60 con 0x7f2e880c4750 2023-03-03T05:45:45.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f2e8d3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8815d140 con 0x7f2e8806e730 2023-03-03T05:45:45.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f2e877fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e880c4750 0x7f2e880c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f2e877fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e880c4750 0x7f2e880c4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59428/0 (socket says 172.21.15.102:59428) 2023-03-03T05:45:45.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f2e877fe700 1 -- 172.21.15.102:0/1538341537 learned_addr learned my addr 172.21.15.102:0/1538341537 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f76fe7fc700 1 -- 172.21.15.102:0/132657472 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0004230 con 0x7f77000cb1c0 2023-03-03T05:45:45.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f76fe7fc700 1 -- 172.21.15.102:0/132657472 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f0004390 con 0x7f77000cb1c0 2023-03-03T05:45:45.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.081+0000 7f76fe7fc700 1 -- 172.21.15.102:0/132657472 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f00036a0 con 0x7f77000cb1c0 2023-03-03T05:45:45.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f7705bff700 1 -- 172.21.15.102:0/132657472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77000cb1c0 msgr2=0x7f7700070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.082+0000 7f7705bff700 1 --2- 172.21.15.102:0/132657472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77000cb1c0 0x7f7700070790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009840 tx=0x7f76f0009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:45.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f2e867fc700 1 --2- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e880c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f2e877fe700 1 -- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 msgr2=0x7f2e880c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f2e877fe700 1 --2- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e880c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f2e877fe700 1 -- 172.21.15.102:0/1538341537 >> v2:172.21.15.102:3300/0 conn(0x7f2e8806e360 msgr2=0x7f2e8806eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f2e877fe700 1 --2- 172.21.15.102:0/1538341537 >> v2:172.21.15.102:3300/0 conn(0x7f2e8806e360 0x7f2e8806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f2e877fe700 1 -- 172.21.15.102:0/1538341537 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8815d400 con 0x7f2e880c4750 2023-03-03T05:45:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a806e220 con 0x7f28a806ed00 2023-03-03T05:45:45.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.083+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a80c61c0 con 0x7f28a806ed00 2023-03-03T05:45:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7f2e877fe700 1 --2- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 0x7f2e880c4170 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e880c70a0 tx=0x7f2e70009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83b097a06ea506df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7f2e85ffb700 1 -- 172.21.15.102:0/1538341537 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7000b870 con 0x7f2e880c4750 2023-03-03T05:45:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7f2e85ffb700 1 -- 172.21.15.102:0/1538341537 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7000b9d0 con 0x7f2e880c4750 2023-03-03T05:45:45.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7f2e85ffb700 1 -- 172.21.15.102:0/1538341537 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7000bd60 con 0x7f2e880c4750 2023-03-03T05:45:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7fd7cadff700 1 -- 172.21.15.102:0/3783486087 shutdown_connections 2023-03-03T05:45:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7fd7cadff700 1 --2- 172.21.15.102:0/3783486087 >> v2:172.21.15.102:3300/0 conn(0x7fd7c40cda10 0x7fd7c4163490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7fd7cadff700 1 --2- 172.21.15.102:0/3783486087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c406ca80 0x7fd7c406ce50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.084+0000 7fd7cadff700 1 -- 172.21.15.102:0/3783486087 >> 172.21.15.102:0/3783486087 conn(0x7fd7c4066bd0 msgr2=0x7fd7c40ceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.085+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a806e500 con 0x7f28a806ed00 2023-03-03T05:45:45.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.085+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 msgr2=0x7f2e880c4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.085+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 0x7f2e880c4170 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e880c70a0 tx=0x7f2e70009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.085+0000 7fd7cadff700 1 -- 172.21.15.102:0/3783486087 shutdown_connections 2023-03-03T05:45:45.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.085+0000 7fd7cadff700 1 -- 172.21.15.102:0/3783486087 wait complete. 2023-03-03T05:45:45.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.087+0000 7f7705bff700 1 -- 172.21.15.102:0/132657472 shutdown_connections 2023-03-03T05:45:45.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.087+0000 7f7705bff700 1 --2- 172.21.15.102:0/132657472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77000cb1c0 0x7f7700070790 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.087+0000 7f7705bff700 1 --2- 172.21.15.102:0/132657472 >> v2:172.21.15.102:3300/0 conn(0x7f770006ed00 0x7f770014ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.087+0000 7f7705bff700 1 -- 172.21.15.102:0/132657472 >> 172.21.15.102:0/132657472 conn(0x7f770006af90 msgr2=0x7f77000caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.087+0000 7f7705bff700 1 -- 172.21.15.102:0/132657472 shutdown_connections 2023-03-03T05:45:45.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.087+0000 7f7705bff700 1 -- 172.21.15.102:0/132657472 wait complete. 2023-03-03T05:45:45.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.088+0000 7f2896ffd700 1 -- 172.21.15.102:0/2081085464 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f289802b030 con 0x7f28a806ed00 2023-03-03T05:45:45.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.089+0000 7fd7cadff700 1 Processor -- start 2023-03-03T05:45:45.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.089+0000 7fd7cadff700 1 -- start start 2023-03-03T05:45:45.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7fd7cadff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7c406ca80 0x7fd7c41ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7fd7cadff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c40cda10 0x7fd7c40b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7fd7cadff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7c40ce790 0x7fd7c40b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7fd7cadff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40b4350 con 0x7fd7c40ce790 2023-03-03T05:45:45.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7fd7cadff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40b2040 con 0x7fd7c40cda10 2023-03-03T05:45:45.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7fd7cadff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c414f1f0 con 0x7fd7c406ca80 2023-03-03T05:45:45.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7f7705bff700 1 Processor -- start 2023-03-03T05:45:45.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7fd7bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c40cda10 0x7fd7c40b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7fd7bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c40cda10 0x7fd7c40b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59440/0 (socket says 172.21.15.102:59440) 2023-03-03T05:45:45.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7fd7bbfff700 1 -- 172.21.15.102:0/1632230210 learned_addr learned my addr 172.21.15.102:0/1632230210 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1538341537 shutdown_connections 2023-03-03T05:45:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 0x7f2e880c4170 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1538341537 >> v2:172.21.15.102:3300/0 conn(0x7f2e8806e360 0x7f2e8806eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1538341537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e880c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1538341537 >> 172.21.15.102:0/1538341537 conn(0x7f2e8806af90 msgr2=0x7f2e880caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7fd7c8b9b700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7c406ca80 0x7fd7c41ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7fd7c8b9b700 1 -- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c40cda10 msgr2=0x7fd7c40b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7fd7c8b9b700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c40cda10 0x7fd7c40b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7fd7c8b9b700 1 -- 172.21.15.102:0/1632230210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7c40ce790 msgr2=0x7fd7c40b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7fd7c8b9b700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7c40ce790 0x7fd7c40b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7fd7c8b9b700 1 -- 172.21.15.102:0/1632230210 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0009b30 con 0x7fd7c406ca80 2023-03-03T05:45:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.091+0000 7f7705bff700 1 -- start start 2023-03-03T05:45:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.092+0000 7f7705bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770006ed00 0x7f7700158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f7705bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77000cb1c0 0x7f7700144370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f7705bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f770014f480 0x7f7700157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f7705bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77001579c0 con 0x7f770014f480 2023-03-03T05:45:45.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f7705bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700158320 con 0x7f77000cb1c0 2023-03-03T05:45:45.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f7705bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770014ee00 con 0x7f770006ed00 2023-03-03T05:45:45.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f76feffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77000cb1c0 0x7f7700144370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f76feffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77000cb1c0 0x7f7700144370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59456/0 (socket says 172.21.15.102:59456) 2023-03-03T05:45:45.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7f76feffd700 1 -- 172.21.15.102:0/2610665827 learned_addr learned my addr 172.21.15.102:0/2610665827 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7fd7c8b9b700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7c406ca80 0x7fd7c41ecfa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd7c406d5f0 tx=0x7fd7c0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.095+0000 7f960effd700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95f00683c0 0x7f95f006a870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f95f8007fd0 tx=0x7f95f8007ec0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7fd7b97fa700 1 -- 172.21.15.102:0/1632230210 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0004440 con 0x7fd7c406ca80 2023-03-03T05:45:45.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7fd7b97fa700 1 -- 172.21.15.102:0/1632230210 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c00045a0 con 0x7fd7c406ca80 2023-03-03T05:45:45.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7fd7b97fa700 1 -- 172.21.15.102:0/1632230210 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0004950 con 0x7fd7c406ca80 2023-03-03T05:45:45.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c415c7b0 con 0x7fd7c406ca80 2023-03-03T05:45:45.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c415c4d0 con 0x7fd7c406ca80 2023-03-03T05:45:45.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7f76feffd700 1 -- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770006ed00 msgr2=0x7f7700158dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:45.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7f76feffd700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770006ed00 0x7f7700158dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7f76feffd700 1 -- 172.21.15.102:0/2610665827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f770014f480 msgr2=0x7f7700157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7f76feffd700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f770014f480 0x7f7700157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.094+0000 7f76feffd700 1 -- 172.21.15.102:0/2610665827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00094a0 con 0x7f77000cb1c0 2023-03-03T05:45:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.095+0000 7f76feffd700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77000cb1c0 0x7f7700144370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f76f400e7e0 tx=0x7f76f400eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.095+0000 7f76e7fff700 1 -- 172.21.15.102:0/2610665827 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f4013610 con 0x7f77000cb1c0 2023-03-03T05:45:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.095+0000 7f76e7fff700 1 -- 172.21.15.102:0/2610665827 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f4013770 con 0x7f77000cb1c0 2023-03-03T05:45:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.095+0000 7f76e7fff700 1 -- 172.21.15.102:0/2610665827 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f4013ae0 con 0x7f77000cb1c0 2023-03-03T05:45:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.096+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770006e500 con 0x7f77000cb1c0 2023-03-03T05:45:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.096+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770006e220 con 0x7f77000cb1c0 2023-03-03T05:45:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7fb9271ff700 1 Processor -- start 2023-03-03T05:45:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.093+0000 7fb9271ff700 1 -- start start 2023-03-03T05:45:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.096+0000 7fb9271ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb920070ea0 0x7fb920166380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.096+0000 7fb9271ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9200c60e0 con 0x7fb92006e360 2023-03-03T05:45:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.096+0000 7fb9271ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92006f0e0 con 0x7fb92006e730 2023-03-03T05:45:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.096+0000 7fb9271ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9200725c0 con 0x7fb920070ea0 2023-03-03T05:45:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.097+0000 7fb92579c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb92006e730 0x7fb920170370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36518/0 (socket says 172.21.15.102:36518) 2023-03-03T05:45:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.097+0000 7fb92579c700 1 -- 172.21.15.102:0/90842764 learned_addr learned my addr 172.21.15.102:0/90842764 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:45.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.090+0000 7f2896ffd700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2888068370 0x7f288806a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.090+0000 7f2896ffd700 1 -- 172.21.15.102:0/2081085464 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2898086950 con 0x7f28a806ed00 2023-03-03T05:45:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.098+0000 7f28a5d9b700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2888068370 0x7f288806a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7f2896ffd700 1 -- 172.21.15.102:0/2081085464 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2898053140 con 0x7f28a806ed00 2023-03-03T05:45:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.101+0000 7f28a5d9b700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2888068370 0x7f288806a820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f28a8155f80 tx=0x7f289c005ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.099+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1538341537 shutdown_connections 2023-03-03T05:45:45.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.101+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1538341537 wait complete. 2023-03-03T05:45:45.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.097+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a40052f0 con 0x7fd7c406ca80 2023-03-03T05:45:45.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.098+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e00052f0 con 0x7f77000cb1c0 2023-03-03T05:45:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fd7b97fa700 1 -- 172.21.15.102:0/1632230210 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd7c0023030 con 0x7fd7c406ca80 2023-03-03T05:45:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fd7b97fa700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd7b00683c0 0x7fd7b006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.103+0000 7fd7b97fa700 1 -- 172.21.15.102:0/1632230210 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd7c0087690 con 0x7fd7c406ca80 2023-03-03T05:45:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.099+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701856292 0 0) 0x7fb92006f0e0 con 0x7fb92006e730 2023-03-03T05:45:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.099+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb908003590 con 0x7fb92006e730 2023-03-03T05:45:45.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 755183614 0 0) 0x7fb908003590 con 0x7fb92006e730 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 >> v2:172.21.15.102:3300/0 conn(0x7fb920070ea0 msgr2=0x7fb920166380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 --2- 172.21.15.102:0/90842764 >> v2:172.21.15.102:3300/0 conn(0x7fb920070ea0 0x7fb920166380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 >> v1:172.21.15.102:6789/0 conn(0x7fb92006e360 legacy=0x7fb920070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9200c54d0 con 0x7fb92006e730 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1894587018 0 0) 0x7fb9180019f0 con 0x7fb92006e730 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 >> v1:172.21.15.172:6789/0 conn(0x7fb92006e730 legacy=0x7fb920170370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb908003920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb908004120 0x7fb908009040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb908009580 0x7fb908009a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200725c0 con 0x7fb908004120 2023-03-03T05:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90800a250 con 0x7fb908009580 2023-03-03T05:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90800a410 con 0x7fb908004bf0 2023-03-03T05:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.100+0000 7fb924f9b700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb908003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.101+0000 7fb924f9b700 1 -- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb908009580 msgr2=0x7fb908009a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.101+0000 7fb92579c700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb908009580 0x7fb908009a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fb924f9b700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb908009580 0x7fb908009a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fb924f9b700 1 -- 172.21.15.102:0/90842764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb908004120 msgr2=0x7fb908009040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T05:45:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fb924f9b700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb908004120 0x7fb908009040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fb924f9b700 1 -- 172.21.15.102:0/90842764 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9200c54d0 con 0x7fb908004bf0 2023-03-03T05:45:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fb92579c700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb908009580 0x7fb908009a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fb92579c700 1 -- 172.21.15.102:0/90842764 reap_dead start 2023-03-03T05:45:45.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7fb924f9b700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb908003920 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9140017c0 tx=0x7fb908003840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e684a0d980a989f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.104+0000 7f2e8d3ff700 1 Processor -- start 2023-03-03T05:45:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.104+0000 7fd7b97fa700 1 -- 172.21.15.102:0/1632230210 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7c0053e80 con 0x7fd7c406ca80 2023-03-03T05:45:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.102+0000 7f76e7fff700 1 -- 172.21.15.102:0/2610665827 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f76f4023070 con 0x7f77000cb1c0 2023-03-03T05:45:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.104+0000 7f76e7fff700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f76e8068550 0x7f76e806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.104+0000 7f76e7fff700 1 -- 172.21.15.102:0/2610665827 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f76f40859e0 con 0x7f77000cb1c0 2023-03-03T05:45:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.105+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9140052d0 con 0x7fb908004bf0 2023-03-03T05:45:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.105+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb914005430 con 0x7fb908004bf0 2023-03-03T05:45:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.105+0000 7fb91f7fe700 1 -- 172.21.15.102:0/90842764 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9140038c0 con 0x7fb908004bf0 2023-03-03T05:45:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7fb9271ff700 1 -- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 msgr2=0x7fb908003920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7fb9271ff700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb908003920 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9140017c0 tx=0x7fb908003840 comp rx=0 tx=0).stop 2023-03-03T05:45:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.104+0000 7f2e8d3ff700 1 -- start start 2023-03-03T05:45:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7f2e8d3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e8806e360 0x7f2e88160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7f2e8d3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e8815bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7f2e8d3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 0x7f2e88146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7f2e8d3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88158830 con 0x7f2e8806e360 2023-03-03T05:45:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7f2e8d3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88160350 con 0x7f2e880c4750 2023-03-03T05:45:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7f2e8d3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88157fb0 con 0x7f2e8806e730 2023-03-03T05:45:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.106+0000 7fd7bbfff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd7b00683c0 0x7fd7b006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7f76ff7fe700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f76e8068550 0x7f76e806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7fb9271ff700 1 -- 172.21.15.102:0/90842764 shutdown_connections 2023-03-03T05:45:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7fb9271ff700 1 --2- 172.21.15.102:0/90842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb908003920 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7fb9271ff700 1 -- 172.21.15.102:0/90842764 >> 172.21.15.102:0/90842764 conn(0x7fb92006af90 msgr2=0x7fb9200caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7f2e86ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e8806e360 0x7f2e88160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7f2e86ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e8806e360 0x7f2e88160b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36508/0 (socket says 172.21.15.102:36508) 2023-03-03T05:45:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7f2e86ffd700 1 -- 172.21.15.102:0/1873404055 learned_addr learned my addr 172.21.15.102:0/1873404055 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7fd7bbfff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd7b00683c0 0x7fd7b006a870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40c5f50 tx=0x7fd7b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7f2e867fc700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e8815bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.108+0000 7f76e7fff700 1 -- 172.21.15.102:0/2610665827 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f76f4055c00 con 0x7f77000cb1c0 2023-03-03T05:45:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.109+0000 7f76ff7fe700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f76e8068550 0x7f76e806aa00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76f00090b0 tx=0x7f76f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.110+0000 7f2e86ffd700 1 -- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 msgr2=0x7f2e8815bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.110+0000 7f2e86ffd700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e8815bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.110+0000 7f2e86ffd700 1 -- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 msgr2=0x7f2e88146710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.110+0000 7f2e86ffd700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 0x7f2e88146710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.110+0000 7f2e86ffd700 1 -- 172.21.15.102:0/1873404055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e70009550 con 0x7f2e8806e360 2023-03-03T05:45:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.111+0000 7f2e86ffd700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e8806e360 0x7f2e88160b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000ba50 tx=0x7f2e8000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.110+0000 7fb9271ff700 1 -- 172.21.15.102:0/90842764 shutdown_connections 2023-03-03T05:45:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.110+0000 7fb9271ff700 1 -- 172.21.15.102:0/90842764 wait complete. 2023-03-03T05:45:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.113+0000 7f2e7f7fe700 1 -- 172.21.15.102:0/1873404055 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8000cab0 con 0x7f2e8806e360 2023-03-03T05:45:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.113+0000 7f2e7f7fe700 1 -- 172.21.15.102:0/1873404055 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8000cc30 con 0x7f2e8806e360 2023-03-03T05:45:45.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.113+0000 7f2e7f7fe700 1 -- 172.21.15.102:0/1873404055 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80014440 con 0x7f2e8806e360 2023-03-03T05:45:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.113+0000 7fb9271ff700 1 Processor -- start 2023-03-03T05:45:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.113+0000 7fb9271ff700 1 -- start start 2023-03-03T05:45:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.115+0000 7fb9271ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb9201f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.115+0000 7fb9271ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9200c15b0 0x7fb9201fba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.115+0000 7fb9271ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9200c77a0 0x7fb920161330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.115+0000 7fb9271ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9201ff040 con 0x7fb9200c77a0 2023-03-03T05:45:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.115+0000 7fb9271ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9201ff580 con 0x7fb9200c15b0 2023-03-03T05:45:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.115+0000 7fb9271ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9201fb2b0 con 0x7fb908004bf0 2023-03-03T05:45:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.116+0000 7fb92579c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9200c77a0 0x7fb920161330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.116+0000 7fb92579c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9200c77a0 0x7fb920161330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36516/0 (socket says 172.21.15.102:36516) 2023-03-03T05:45:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.116+0000 7fb92579c700 1 -- 172.21.15.102:0/2541076081 learned_addr learned my addr 172.21.15.102:0/2541076081 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.116+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8815d9c0 con 0x7f2e8806e360 2023-03-03T05:45:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.116+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8815d6e0 con 0x7f2e8806e360 2023-03-03T05:45:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.117+0000 7fb92579c700 1 -- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 msgr2=0x7fb9201f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.117+0000 7fb924f9b700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb9201f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.118+0000 7fb91ffff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9200c15b0 0x7fb9201fba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.118+0000 7fb92579c700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb9201f0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.118+0000 7fb92579c700 1 -- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9200c15b0 msgr2=0x7fb9201fba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.118+0000 7fb92579c700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9200c15b0 0x7fb9201fba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.118+0000 7fb92579c700 1 -- 172.21.15.102:0/2541076081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9200c5a90 con 0x7fb9200c77a0 2023-03-03T05:45:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.119+0000 7fb924f9b700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb9201f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.119+0000 7fb91ffff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9200c15b0 0x7fb9201fba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.120+0000 7fb92579c700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9200c77a0 0x7fb920161330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb9200c2150 tx=0x7fb91800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.120+0000 7fb91d7fa700 1 -- 172.21.15.102:0/2541076081 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800bda0 con 0x7fb9200c77a0 2023-03-03T05:45:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.120+0000 7fb91d7fa700 1 -- 172.21.15.102:0/2541076081 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9180070d0 con 0x7fb9200c77a0 2023-03-03T05:45:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.123+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8815dca0 con 0x7f2e8806e360 2023-03-03T05:45:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.126+0000 7f2e7f7fe700 1 -- 172.21.15.102:0/1873404055 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e8001b030 con 0x7f2e8806e360 2023-03-03T05:45:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.128+0000 7f2e7f7fe700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e680683c0 0x7f2e6806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.128+0000 7f2e7f7fe700 1 -- 172.21.15.102:0/1873404055 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2e80085990 con 0x7f2e8806e360 2023-03-03T05:45:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.129+0000 7f2e867fc700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e680683c0 0x7f2e6806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.120+0000 7fb91d7fa700 1 -- 172.21.15.102:0/2541076081 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918007460 con 0x7fb9200c77a0 2023-03-03T05:45:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.121+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9200c57b0 con 0x7fb9200c77a0 2023-03-03T05:45:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.125+0000 7fb91d7fa700 1 -- 172.21.15.102:0/2541076081 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb91801b030 con 0x7fb9200c77a0 2023-03-03T05:45:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.131+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9200c5d70 con 0x7fb9200c77a0 2023-03-03T05:45:45.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.133+0000 7f2e867fc700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e680683c0 0x7f2e6806a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e881679b0 tx=0x7f2e78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.133+0000 7f2e7f7fe700 1 -- 172.21.15.102:0/1873404055 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e80052180 con 0x7f2e8806e360 2023-03-03T05:45:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.135+0000 7fb91d7fa700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb90c068370 0x7fb90c06a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.135+0000 7fb91d7fa700 1 -- 172.21.15.102:0/2541076081 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb918085ba0 con 0x7fb9200c77a0 2023-03-03T05:45:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.137+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9201fb110 con 0x7fb9200c77a0 2023-03-03T05:45:45.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.137+0000 7fb924f9b700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb90c068370 0x7fb90c06a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.146+0000 7fb924f9b700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb90c068370 0x7fb90c06a820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb914005040 tx=0x7fb914009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.147+0000 7fb91d7fa700 1 -- 172.21.15.102:0/2541076081 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb9201fb110 con 0x7fb9200c77a0 2023-03-03T05:45:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.148+0000 7fcd561ff700 1 Processor -- start 2023-03-03T05:45:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.148+0000 7fcd561ff700 1 -- start start 2023-03-03T05:45:45.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.151+0000 7fcd561ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcd5006eea0 0x7fcd50168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.151+0000 7fcd561ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5006ecc0 con 0x7fcd5006e360 2023-03-03T05:45:45.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.151+0000 7fcd561ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5015dd30 con 0x7fcd500cb6d0 2023-03-03T05:45:45.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.151+0000 7fcd561ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5015e2b0 con 0x7fcd5006eea0 2023-03-03T05:45:45.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.153+0000 7fcd4f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcd5006e360 0x7fcd500c6090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45232/0 (socket says 172.21.15.102:45232) 2023-03-03T05:45:45.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.153+0000 7fcd4f7fe700 1 -- 172.21.15.102:0/1989903214 learned_addr learned my addr 172.21.15.102:0/1989903214 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.157+0000 7fcd4ffff700 1 --2- 172.21.15.102:0/1989903214 >> v2:172.21.15.172:3300/0 conn(0x7fcd5006eea0 0x7fcd50168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.157+0000 7fcd4e7fc700 1 -- 172.21.15.102:0/1989903214 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810901129 0 0) 0x7fcd5006ecc0 con 0x7fcd5006e360 2023-03-03T05:45:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.157+0000 7fcd4e7fc700 1 -- 172.21.15.102:0/1989903214 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd380035b0 con 0x7fcd5006e360 2023-03-03T05:45:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.157+0000 7fcd4e7fc700 1 -- 172.21.15.102:0/1989903214 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879015804 0 0) 0x7fcd5015dd30 con 0x7fcd500cb6d0 2023-03-03T05:45:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.158+0000 7fcd4e7fc700 1 -- 172.21.15.102:0/1989903214 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd5006ecc0 con 0x7fcd500cb6d0 2023-03-03T05:45:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.158+0000 7fcd4ffff700 1 -- 172.21.15.102:0/1989903214 >> v1:172.21.15.172:6789/0 conn(0x7fcd500cb6d0 legacy=0x7fcd50157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.158+0000 7fcd4ffff700 1 -- 172.21.15.102:0/1989903214 >> v1:172.21.15.102:6789/0 conn(0x7fcd5006e360 legacy=0x7fcd500c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.158+0000 7fcd4ffff700 1 -- 172.21.15.102:0/1989903214 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5015dfd0 con 0x7fcd5006eea0 2023-03-03T05:45:45.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.159+0000 7fcd4ffff700 1 --2- 172.21.15.102:0/1989903214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006eea0 0x7fcd50168b60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009d90 tx=0x7fcd4800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1fba9c8baa4b974 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.160+0000 7fcd4e7fc700 1 -- 172.21.15.102:0/1989903214 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800e070 con 0x7fcd5006eea0 2023-03-03T05:45:45.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.160+0000 7fcd4e7fc700 1 -- 172.21.15.102:0/1989903214 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4800bb20 con 0x7fcd5006eea0 2023-03-03T05:45:45.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.160+0000 7fcd4e7fc700 1 -- 172.21.15.102:0/1989903214 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48012430 con 0x7fcd5006eea0 2023-03-03T05:45:45.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.161+0000 7fcd561ff700 1 -- 172.21.15.102:0/1989903214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006eea0 msgr2=0x7fcd50168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.161+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1989903214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006eea0 0x7fcd50168b60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009d90 tx=0x7fcd4800b330 comp rx=0 tx=0).stop 2023-03-03T05:45:45.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.162+0000 7fcd561ff700 1 -- 172.21.15.102:0/1989903214 shutdown_connections 2023-03-03T05:45:45.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.162+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1989903214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006eea0 0x7fcd50168b60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.162+0000 7fcd561ff700 1 -- 172.21.15.102:0/1989903214 >> 172.21.15.102:0/1989903214 conn(0x7fcd5006af90 msgr2=0x7fcd500caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.162+0000 7fcd561ff700 1 -- 172.21.15.102:0/1989903214 shutdown_connections 2023-03-03T05:45:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.163+0000 7fcd561ff700 1 -- 172.21.15.102:0/1989903214 wait complete. 2023-03-03T05:45:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.167+0000 7fcd561ff700 1 Processor -- start 2023-03-03T05:45:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.167+0000 7fcd561ff700 1 -- start start 2023-03-03T05:45:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.169+0000 7fcd561ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006e360 0x7fcd500c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.169+0000 7fcd561ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd5006eea0 0x7fcd5020bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.169+0000 7fcd561ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500cb6d0 0x7fcd5020c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.169+0000 7fcd561ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5020bb20 con 0x7fcd5006eea0 2023-03-03T05:45:45.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.169+0000 7fcd561ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5020f250 con 0x7fcd5006e360 2023-03-03T05:45:45.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.169+0000 7fcd561ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5020f510 con 0x7fcd500cb6d0 2023-03-03T05:45:45.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.173+0000 7fcd4effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd5006eea0 0x7fcd5020bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.173+0000 7fcd4effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd5006eea0 0x7fcd5020bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36532/0 (socket says 172.21.15.102:36532) 2023-03-03T05:45:45.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.173+0000 7fcd4effd700 1 -- 172.21.15.102:0/1755186329 learned_addr learned my addr 172.21.15.102:0/1755186329 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:45.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.174+0000 7fcd4effd700 1 -- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500cb6d0 msgr2=0x7fcd5020c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:45.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.174+0000 7fcd4effd700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500cb6d0 0x7fcd5020c4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.174+0000 7fcd4effd700 1 -- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006e360 msgr2=0x7fcd500c3f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T05:45:45.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.174+0000 7fcd4effd700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006e360 0x7fcd500c3f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.174+0000 7fcd4effd700 1 -- 172.21.15.102:0/1755186329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48009a70 con 0x7fcd5006eea0 2023-03-03T05:45:45.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.175+0000 7fcd4effd700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd5006eea0 0x7fcd5020bfb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c500 tx=0x7fcd4400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.176+0000 7fcd37fff700 1 -- 172.21.15.102:0/1755186329 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd440052d0 con 0x7fcd5006eea0 2023-03-03T05:45:45.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.176+0000 7fcd37fff700 1 -- 172.21.15.102:0/1755186329 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44005430 con 0x7fcd5006eea0 2023-03-03T05:45:45.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.176+0000 7fcd37fff700 1 -- 172.21.15.102:0/1755186329 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd440038c0 con 0x7fcd5006eea0 2023-03-03T05:45:45.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.176+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd500c5890 con 0x7fcd5006eea0 2023-03-03T05:45:45.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.177+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd500c55b0 con 0x7fcd5006eea0 2023-03-03T05:45:45.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.181+0000 7fcd37fff700 1 -- 172.21.15.102:0/1755186329 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd44019030 con 0x7fcd5006eea0 2023-03-03T05:45:45.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.184+0000 7fcd37fff700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd3c0683f0 0x7fcd3c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:45.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.184+0000 7fcd37fff700 1 -- 172.21.15.102:0/1755186329 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcd44085030 con 0x7fcd5006eea0 2023-03-03T05:45:45.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.187+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd300052f0 con 0x7fcd5006eea0 2023-03-03T05:45:45.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.188+0000 7fcd4f7fe700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd3c0683f0 0x7fcd3c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:45.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.198+0000 7fcd37fff700 1 -- 172.21.15.102:0/1755186329 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcd44051820 con 0x7fcd5006eea0 2023-03-03T05:45:45.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.199+0000 7fcd4f7fe700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd3c0683f0 0x7fcd3c06a8a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcd40007fd0 tx=0x7fcd40007980 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:45.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.494+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fed9004e6e0 con 0x7fed900c2a10 2023-03-03T05:45:45.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.495+0000 7fed85ffb700 1 -- 172.21.15.102:0/449082342 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed80056110 con 0x7fed900c2a10 2023-03-03T05:45:45.501 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2023-03-03T05:45:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed740683c0 msgr2=0x7fed7406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed740683c0 0x7fed7406a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fed901e6f90 tx=0x7fed88001ac0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 msgr2=0x7fed90156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90156df0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fed8000e8a0 tx=0x7fed8000ec60 comp rx=0 tx=0).stop 2023-03-03T05:45:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 shutdown_connections 2023-03-03T05:45:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed740683c0 0x7fed7406a870 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fed901e6f90 tx=0x7fed88001ac0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9014ede0 0x7fed90146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed900c2a10 0x7fed90156df0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 --2- 172.21.15.102:0/449082342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed900c5410 0x7fed90157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.500+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 >> 172.21.15.102:0/449082342 conn(0x7fed90065fb0 msgr2=0x7fed900ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.501+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 shutdown_connections 2023-03-03T05:45:45.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.501+0000 7fed9679d700 1 -- 172.21.15.102:0/449082342 wait complete. 2023-03-03T05:45:45.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.528+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f96100c3580 con 0x7f9610153820 2023-03-03T05:45:45.535 INFO:teuthology.orchestra.run.smithi102.stdout:55834574855 2023-03-03T05:45:45.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.529+0000 7f9607fff700 1 -- 172.21.15.102:0/1861540091 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f96080566c0 con 0x7f9610153820 2023-03-03T05:45:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95f00683c0 msgr2=0x7f95f006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95f00683c0 0x7f95f006a870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f95f8007fd0 tx=0x7f95f8007ec0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9610153820 msgr2=0x7f96101fed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9610153820 0x7f96101fed00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9608007470 tx=0x7f9608007780 comp rx=0 tx=0).stop 2023-03-03T05:45:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 shutdown_connections 2023-03-03T05:45:45.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95f00683c0 0x7f95f006a870 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f95f8007fd0 tx=0x7f95f8007ec0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9610153820 0x7f96101fed00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96100c3b10 0x7f9610153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 --2- 172.21.15.102:0/1861540091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96100c3ee0 0x7f96101542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.536+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 >> 172.21.15.102:0/1861540091 conn(0x7f961006af90 msgr2=0x7f961014a530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.541+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 shutdown_connections 2023-03-03T05:45:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.541+0000 7f96159ff700 1 -- 172.21.15.102:0/1861540091 wait complete. 2023-03-03T05:45:45.544 INFO:teuthology.orchestra.run.smithi102.stdout:55834574855 2023-03-03T05:45:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.541+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f28a8070690 con 0x7f28a806ed00 2023-03-03T05:45:45.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.542+0000 7f2896ffd700 1 -- 172.21.15.102:0/2081085464 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2898056760 con 0x7f28a806ed00 2023-03-03T05:45:45.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.545+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f76e0005870 con 0x7f77000cb1c0 2023-03-03T05:45:45.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.545+0000 7f76e7fff700 1 -- 172.21.15.102:0/2610665827 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f76f40557f0 con 0x7f77000cb1c0 2023-03-03T05:45:45.548 INFO:teuthology.orchestra.run.smithi102.stdout:64424509448 2023-03-03T05:45:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2888068370 msgr2=0x7f288806a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2888068370 0x7f288806a820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f28a8155f80 tx=0x7f289c005ee0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28a806ed00 msgr2=0x7f28a81f6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28a806ed00 0x7f28a81f6900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2898016000 tx=0x7f2898004750 comp rx=0 tx=0).stop 2023-03-03T05:45:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 shutdown_connections 2023-03-03T05:45:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28a8158470 0x7f28a81e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28a806ed00 0x7f28a81f6900 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2898016000 tx=0x7f2898004750 comp rx=0 tx=0).stop 2023-03-03T05:45:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2888068370 0x7f288806a820 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 --2- 172.21.15.102:0/2081085464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f28a814f120 0x7f28a8151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.547+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 >> 172.21.15.102:0/2081085464 conn(0x7f28a806af90 msgr2=0x7f28a8071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 shutdown_connections 2023-03-03T05:45:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd7a4005870 con 0x7fd7c406ca80 2023-03-03T05:45:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7f28acbff700 1 -- 172.21.15.102:0/2081085464 wait complete. 2023-03-03T05:45:45.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7fd7b97fa700 1 -- 172.21.15.102:0/1632230210 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7c00574a0 con 0x7fd7c406ca80 2023-03-03T05:45:45.554 INFO:teuthology.orchestra.run.smithi102.stdout:55834574855 2023-03-03T05:45:45.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.0 2023-03-03T05:45:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f76e8068550 msgr2=0x7f76e806aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7f7705bff700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f76e8068550 0x7f76e806aa00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76f00090b0 tx=0x7f76f0008030 comp rx=0 tx=0).stop 2023-03-03T05:45:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77000cb1c0 msgr2=0x7f7700144370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.550+0000 7f7705bff700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77000cb1c0 0x7f7700144370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f76f400e7e0 tx=0x7f76f400eba0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.553+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 shutdown_connections 2023-03-03T05:45:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.553+0000 7f7705bff700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f76e8068550 0x7f76e806aa00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.553+0000 7f7705bff700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f770006ed00 0x7f7700158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.553+0000 7f7705bff700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f770014f480 0x7f7700157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.554+0000 7f7705bff700 1 --2- 172.21.15.102:0/2610665827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77000cb1c0 0x7f7700144370 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.554+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 >> 172.21.15.102:0/2610665827 conn(0x7f770006af90 msgr2=0x7f7700071890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.555+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 shutdown_connections 2023-03-03T05:45:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7f7705bff700 1 -- 172.21.15.102:0/2610665827 wait complete. 2023-03-03T05:45:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd7b00683c0 msgr2=0x7fd7b006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd7b00683c0 0x7fd7b006a870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40c5f50 tx=0x7fd7b4009190 comp rx=0 tx=0).stop 2023-03-03T05:45:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7c406ca80 msgr2=0x7fd7c41ecfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7c406ca80 0x7fd7c41ecfa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd7c406d5f0 tx=0x7fd7c0004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 shutdown_connections 2023-03-03T05:45:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd7b00683c0 0x7fd7b006a870 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7c406ca80 0x7fd7c41ecfa0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd7c406d5f0 tx=0x7fd7c0004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7c40ce790 0x7fd7c40b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 --2- 172.21.15.102:0/1632230210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7c40cda10 0x7fd7c40b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.561+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 >> 172.21.15.102:0/1632230210 conn(0x7fd7c4066bd0 msgr2=0x7fd7c40cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.562+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 shutdown_connections 2023-03-03T05:45:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.562+0000 7fd7cadff700 1 -- 172.21.15.102:0/1632230210 wait complete. 2023-03-03T05:45:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.570+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2e88160190 con 0x7f2e8806e360 2023-03-03T05:45:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.571+0000 7f2e7f7fe700 1 -- 172.21.15.102:0/1873404055 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e800557a0 con 0x7f2e8806e360 2023-03-03T05:45:45.576 INFO:teuthology.orchestra.run.smithi102.stdout:55834574856 2023-03-03T05:45:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.575+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e680683c0 msgr2=0x7f2e6806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.575+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e680683c0 0x7f2e6806a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e881679b0 tx=0x7f2e78008040 comp rx=0 tx=0).stop 2023-03-03T05:45:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.575+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e8806e360 msgr2=0x7f2e88160b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.575+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e8806e360 0x7f2e88160b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000ba50 tx=0x7f2e8000be10 comp rx=0 tx=0).stop 2023-03-03T05:45:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.576+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 shutdown_connections 2023-03-03T05:45:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.576+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e880c4750 0x7f2e88146710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.576+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e8806e360 0x7f2e88160b10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.576+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e680683c0 0x7f2e6806a870 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.576+0000 7f2e8d3ff700 1 --2- 172.21.15.102:0/1873404055 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e8806e730 0x7f2e8815bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.576+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 >> 172.21.15.102:0/1873404055 conn(0x7f2e8806af90 msgr2=0x7f2e8814a540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.577+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 shutdown_connections 2023-03-03T05:45:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.577+0000 7f2e8d3ff700 1 -- 172.21.15.102:0/1873404055 wait complete. 2023-03-03T05:45:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.581+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb9201fb110 con 0x7fb9200c77a0 2023-03-03T05:45:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.582+0000 7fb91d7fa700 1 -- 172.21.15.102:0/2541076081 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb9201fb110 con 0x7fb9200c77a0 2023-03-03T05:45:45.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2023-03-03T05:45:45.593 INFO:teuthology.orchestra.run.smithi102.stdout:55834574855 2023-03-03T05:45:45.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.592+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb90c068370 msgr2=0x7fb90c06a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.592+0000 7fb9271ff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb90c068370 0x7fb90c06a820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb914005040 tx=0x7fb914009030 comp rx=0 tx=0).stop 2023-03-03T05:45:45.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.592+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9200c77a0 msgr2=0x7fb920161330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.592+0000 7fb9271ff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9200c77a0 0x7fb920161330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb9200c2150 tx=0x7fb91800b3b0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 shutdown_connections 2023-03-03T05:45:45.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb90c068370 0x7fb90c06a820 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb908004bf0 0x7fb9201f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9200c77a0 0x7fb920161330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 --2- 172.21.15.102:0/2541076081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9200c15b0 0x7fb9201fba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 >> 172.21.15.102:0/2541076081 conn(0x7fb92006af90 msgr2=0x7fb920165580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 shutdown_connections 2023-03-03T05:45:45.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.593+0000 7fb9271ff700 1 -- 172.21.15.102:0/2541076081 wait complete. 2023-03-03T05:45:45.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2023-03-03T05:45:45.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.603+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fcd300055a0 con 0x7fcd5006eea0 2023-03-03T05:45:45.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.2 2023-03-03T05:45:45.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.604+0000 7fcd37fff700 1 -- 172.21.15.102:0/1755186329 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcd44017070 con 0x7fcd5006eea0 2023-03-03T05:45:45.606 INFO:teuthology.orchestra.run.smithi102.stdout:55834574856 2023-03-03T05:45:45.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd3c0683f0 msgr2=0x7fcd3c06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd3c0683f0 0x7fcd3c06a8a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcd40007fd0 tx=0x7fcd40007980 comp rx=0 tx=0).stop 2023-03-03T05:45:45.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd5006eea0 msgr2=0x7fcd5020bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:45.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd5006eea0 0x7fcd5020bfb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c500 tx=0x7fcd4400c8c0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 shutdown_connections 2023-03-03T05:45:45.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd3c0683f0 0x7fcd3c06a8a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd500cb6d0 0x7fcd5020c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd5006e360 0x7fcd500c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 --2- 172.21.15.102:0/1755186329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd5006eea0 0x7fcd5020bfb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:45.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 >> 172.21.15.102:0/1755186329 conn(0x7fcd5006af90 msgr2=0x7fcd500c63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:45.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 shutdown_connections 2023-03-03T05:45:45.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:45.607+0000 7fcd561ff700 1 -- 172.21.15.102:0/1755186329 wait complete. 2023-03-03T05:45:45.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2023-03-03T05:45:45.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2023-03-03T05:45:45.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2023-03-03T05:45:45.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2023-03-03T05:45:46.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-03T05:45:46.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-03-03T05:45:46.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-03T05:45:46.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-03T05:45:46.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-03-03T05:45:46.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-03-03T05:45:46.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-03-03T05:45:46.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-03-03T05:45:46.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.925+0000 7fc958fff700 1 Processor -- start 2023-03-03T05:45:46.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.927+0000 7fc958fff700 1 -- start start 2023-03-03T05:45:46.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.930+0000 7fc958fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc954070790 0x7fc95406ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:46.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.930+0000 7fc958fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9540c64a0 con 0x7fc954158470 2023-03-03T05:45:46.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.930+0000 7fc958fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9540c5ee0 con 0x7fc95414f120 2023-03-03T05:45:46.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.930+0000 7fc958fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9540c2430 con 0x7fc954070790 2023-03-03T05:45:46.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.932+0000 7fc95259c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc95414f120 0x7fc9540c5430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35920/0 (socket says 172.21.15.102:35920) 2023-03-03T05:45:46.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.932+0000 7fc95259c700 1 -- 172.21.15.102:0/1509322961 learned_addr learned my addr 172.21.15.102:0/1509322961 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:46.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.932+0000 7fc951d9b700 1 --2- 172.21.15.102:0/1509322961 >> v2:172.21.15.102:3300/0 conn(0x7fc954070790 0x7fc95406ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:46.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.933+0000 7fc951d9b700 1 -- 172.21.15.102:0/1509322961 >> v1:172.21.15.172:6789/0 conn(0x7fc95414f120 legacy=0x7fc9540c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:46.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.933+0000 7fc951d9b700 1 -- 172.21.15.102:0/1509322961 >> v1:172.21.15.102:6789/0 conn(0x7fc954158470 legacy=0x7fc9541588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:46.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.933+0000 7fc951d9b700 1 -- 172.21.15.102:0/1509322961 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9540c61c0 con 0x7fc954070790 2023-03-03T05:45:46.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.935+0000 7fc951d9b700 1 --2- 172.21.15.102:0/1509322961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954070790 0x7fc95406ed00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc9480091b0 tx=0x7fc9480094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5881337532af61d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:46.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.936+0000 7fc95159a700 1 -- 172.21.15.102:0/1509322961 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948004230 con 0x7fc954070790 2023-03-03T05:45:46.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.936+0000 7fc95159a700 1 -- 172.21.15.102:0/1509322961 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc948004390 con 0x7fc954070790 2023-03-03T05:45:46.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.936+0000 7fc95159a700 1 -- 172.21.15.102:0/1509322961 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480036a0 con 0x7fc954070790 2023-03-03T05:45:46.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.937+0000 7fc958fff700 1 -- 172.21.15.102:0/1509322961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954070790 msgr2=0x7fc95406ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:46.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.937+0000 7fc958fff700 1 --2- 172.21.15.102:0/1509322961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954070790 0x7fc95406ed00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc9480091b0 tx=0x7fc9480094c0 comp rx=0 tx=0).stop 2023-03-03T05:45:46.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.938+0000 7fc958fff700 1 -- 172.21.15.102:0/1509322961 shutdown_connections 2023-03-03T05:45:46.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.938+0000 7fc958fff700 1 --2- 172.21.15.102:0/1509322961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954070790 0x7fc95406ed00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:46.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.938+0000 7fc958fff700 1 -- 172.21.15.102:0/1509322961 >> 172.21.15.102:0/1509322961 conn(0x7fc95406af90 msgr2=0x7fc9540caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:46.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.938+0000 7fc958fff700 1 -- 172.21.15.102:0/1509322961 shutdown_connections 2023-03-03T05:45:46.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.939+0000 7fc958fff700 1 -- 172.21.15.102:0/1509322961 wait complete. 2023-03-03T05:45:46.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.942+0000 7fc958fff700 1 Processor -- start 2023-03-03T05:45:46.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.942+0000 7fc958fff700 1 -- start start 2023-03-03T05:45:46.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.944+0000 7fc958fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc954070790 0x7fc9541619c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:46.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.944+0000 7fc958fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc95414f120 0x7fc954152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:46.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.944+0000 7fc958fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954158470 0x7fc9541485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:46.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.944+0000 7fc958fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc954161f00 con 0x7fc954158470 2023-03-03T05:45:46.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.944+0000 7fc958fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9541627b0 con 0x7fc954070790 2023-03-03T05:45:46.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.944+0000 7fc958fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9541f2b00 con 0x7fc95414f120 2023-03-03T05:45:46.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.946+0000 7fc951d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc95414f120 0x7fc954152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:46.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.946+0000 7fc95259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc954070790 0x7fc9541619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:46.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.946+0000 7fc95259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc954070790 0x7fc9541619c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50332/0 (socket says 172.21.15.102:50332) 2023-03-03T05:45:46.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.946+0000 7fc95259c700 1 -- 172.21.15.102:0/1522855042 learned_addr learned my addr 172.21.15.102:0/1522855042 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:46.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.947+0000 7fc95259c700 1 -- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc95414f120 msgr2=0x7fc954152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:46.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.947+0000 7fc95259c700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc95414f120 0x7fc954152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:46.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.947+0000 7fc95259c700 1 -- 172.21.15.102:0/1522855042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954158470 msgr2=0x7fc9541485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:46.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.947+0000 7fc95259c700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954158470 0x7fc9541485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:46.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.947+0000 7fc95259c700 1 -- 172.21.15.102:0/1522855042 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948009d30 con 0x7fc954070790 2023-03-03T05:45:46.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.948+0000 7fc95259c700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc954070790 0x7fc9541619c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc94c002080 tx=0x7fc94c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:46.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.949+0000 7fc942ffd700 1 -- 172.21.15.102:0/1522855042 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c00ae00 con 0x7fc954070790 2023-03-03T05:45:46.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.949+0000 7fc942ffd700 1 -- 172.21.15.102:0/1522855042 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c0070d0 con 0x7fc954070790 2023-03-03T05:45:46.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.949+0000 7fc942ffd700 1 -- 172.21.15.102:0/1522855042 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c007480 con 0x7fc954070790 2023-03-03T05:45:46.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.949+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95406e500 con 0x7fc954070790 2023-03-03T05:45:46.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.949+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95406e220 con 0x7fc954070790 2023-03-03T05:45:46.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.957+0000 7fc942ffd700 1 -- 172.21.15.102:0/1522855042 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc94c01a030 con 0x7fc954070790 2023-03-03T05:45:46.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.959+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9380052f0 con 0x7fc954070790 2023-03-03T05:45:46.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.961+0000 7fc942ffd700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc934068390 0x7fc93406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:46.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.961+0000 7fc942ffd700 1 -- 172.21.15.102:0/1522855042 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc94c085cc0 con 0x7fc954070790 2023-03-03T05:45:46.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.962+0000 7fc951d9b700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc934068390 0x7fc93406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:46.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.964+0000 7fc951d9b700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc934068390 0x7fc93406a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc9480091b0 tx=0x7fc948008000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:46.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:46.965+0000 7fc942ffd700 1 -- 172.21.15.102:0/1522855042 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc94c0524b0 con 0x7fc954070790 2023-03-03T05:45:47.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.025+0000 7f671b1ff700 1 Processor -- start 2023-03-03T05:45:47.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.026+0000 7f671b1ff700 1 -- start start 2023-03-03T05:45:47.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.029+0000 7f671b1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f671406e360 0x7f671406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.029+0000 7f671b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f671406e730 0x7f67140c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.029+0000 7f671b1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f67140c4750 0x7f67140c4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.029+0000 7f671b1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6714157900 con 0x7f671406e360 2023-03-03T05:45:47.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.029+0000 7f671b1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67140c4e60 con 0x7f67140c4750 2023-03-03T05:45:47.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.029+0000 7f671b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671415d140 con 0x7f671406e730 2023-03-03T05:45:47.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.030+0000 7f6718f9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f671406e360 0x7f671406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.030+0000 7f6718f9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f671406e360 0x7f671406eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39462/0 (socket says 172.21.15.102:39462) 2023-03-03T05:45:47.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.030+0000 7f6718f9b700 1 -- 172.21.15.102:0/1043793015 learned_addr learned my addr 172.21.15.102:0/1043793015 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.030+0000 7f671979c700 1 --2- 172.21.15.102:0/1043793015 >> v2:172.21.15.172:3300/0 conn(0x7f67140c4750 0x7f67140c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.030+0000 7f6713fff700 1 --2- 172.21.15.102:0/1043793015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f671406e730 0x7f67140c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.031+0000 7f6718f9b700 1 -- 172.21.15.102:0/1043793015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f671406e730 msgr2=0x7f67140c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.031+0000 7f6718f9b700 1 --2- 172.21.15.102:0/1043793015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f671406e730 0x7f67140c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.031+0000 7f6718f9b700 1 -- 172.21.15.102:0/1043793015 >> v2:172.21.15.172:3300/0 conn(0x7f67140c4750 msgr2=0x7f67140c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.031+0000 7f6718f9b700 1 --2- 172.21.15.102:0/1043793015 >> v2:172.21.15.172:3300/0 conn(0x7f67140c4750 0x7f67140c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.031+0000 7f6718f9b700 1 -- 172.21.15.102:0/1043793015 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671415d400 con 0x7f671406e360 2023-03-03T05:45:47.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.032+0000 7f6718f9b700 1 --2- 172.21.15.102:0/1043793015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671406e360 0x7f671406eca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67140c70a0 tx=0x7f6704009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb4ea556a384cff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.033+0000 7f67137fe700 1 -- 172.21.15.102:0/1043793015 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704011830 con 0x7f671406e360 2023-03-03T05:45:47.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.033+0000 7f67137fe700 1 -- 172.21.15.102:0/1043793015 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6704011990 con 0x7f671406e360 2023-03-03T05:45:47.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.033+0000 7f67137fe700 1 -- 172.21.15.102:0/1043793015 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704011d20 con 0x7f671406e360 2023-03-03T05:45:47.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.034+0000 7f671b1ff700 1 -- 172.21.15.102:0/1043793015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671406e360 msgr2=0x7f671406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.034+0000 7f671b1ff700 1 --2- 172.21.15.102:0/1043793015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671406e360 0x7f671406eca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67140c70a0 tx=0x7f6704009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.031+0000 7fadd41ff700 1 Processor -- start 2023-03-03T05:45:47.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.031+0000 7fadd41ff700 1 -- start start 2023-03-03T05:45:47.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.034+0000 7fadd41ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc06e360 0x7fadcc0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.034+0000 7fadd41ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fadcc0cb6d0 0x7fadcc157170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.034+0000 7fadd41ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc06ecc0 con 0x7fadcc06eea0 2023-03-03T05:45:47.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.034+0000 7fadd41ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc15e2b0 con 0x7fadcc0cb6d0 2023-03-03T05:45:47.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.034+0000 7fadd41ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc15dd30 con 0x7fadcc06e360 2023-03-03T05:45:47.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7f671b1ff700 1 -- 172.21.15.102:0/1043793015 shutdown_connections 2023-03-03T05:45:47.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7f671b1ff700 1 --2- 172.21.15.102:0/1043793015 >> v2:172.21.15.172:3300/0 conn(0x7f67140c4750 0x7f67140c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7f671b1ff700 1 --2- 172.21.15.102:0/1043793015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671406e360 0x7f671406eca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7f671b1ff700 1 --2- 172.21.15.102:0/1043793015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f671406e730 0x7f67140c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7f671b1ff700 1 -- 172.21.15.102:0/1043793015 >> 172.21.15.102:0/1043793015 conn(0x7f671406af90 msgr2=0x7f67140caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7fadd279c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fadcc06eea0 0x7fadcc160820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42574/0 (socket says 172.21.15.102:42574) 2023-03-03T05:45:47.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7fadd279c700 1 -- 172.21.15.102:0/3147141798 learned_addr learned my addr 172.21.15.102:0/3147141798 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:47.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.036+0000 7fadd1f9b700 1 --2- 172.21.15.102:0/3147141798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc06e360 0x7fadcc0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.037+0000 7fadd179a700 1 --2- 172.21.15.102:0/3147141798 >> v2:172.21.15.172:3300/0 conn(0x7fadcc0cb6d0 0x7fadcc157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.037+0000 7fadd0f99700 1 -- 172.21.15.102:0/3147141798 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125323427 0 0) 0x7fadcc06ecc0 con 0x7fadcc06eea0 2023-03-03T05:45:47.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.038+0000 7fadd0f99700 1 -- 172.21.15.102:0/3147141798 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb80035b0 con 0x7fadcc06eea0 2023-03-03T05:45:47.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.038+0000 7fadd1f9b700 1 -- 172.21.15.102:0/3147141798 >> v2:172.21.15.172:3300/0 conn(0x7fadcc0cb6d0 msgr2=0x7fadcc157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.038+0000 7fadd1f9b700 1 --2- 172.21.15.102:0/3147141798 >> v2:172.21.15.172:3300/0 conn(0x7fadcc0cb6d0 0x7fadcc157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.038+0000 7fadd1f9b700 1 -- 172.21.15.102:0/3147141798 >> v1:172.21.15.102:6789/0 conn(0x7fadcc06eea0 legacy=0x7fadcc160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.038+0000 7fadd1f9b700 1 -- 172.21.15.102:0/3147141798 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc15dfd0 con 0x7fadcc06e360 2023-03-03T05:45:47.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.038+0000 7fadd1f9b700 1 --2- 172.21.15.102:0/3147141798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc06e360 0x7fadcc0c6090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009840 tx=0x7fadbc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f6059e17fda1a2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.039+0000 7f671b1ff700 1 -- 172.21.15.102:0/1043793015 shutdown_connections 2023-03-03T05:45:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.039+0000 7f671b1ff700 1 -- 172.21.15.102:0/1043793015 wait complete. 2023-03-03T05:45:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.039+0000 7fadd0f99700 1 -- 172.21.15.102:0/3147141798 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc011830 con 0x7fadcc06e360 2023-03-03T05:45:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.039+0000 7fadd0f99700 1 -- 172.21.15.102:0/3147141798 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc011990 con 0x7fadcc06e360 2023-03-03T05:45:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.039+0000 7fadd0f99700 1 -- 172.21.15.102:0/3147141798 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc011d20 con 0x7fadcc06e360 2023-03-03T05:45:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.040+0000 7fadd41ff700 1 -- 172.21.15.102:0/3147141798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc06e360 msgr2=0x7fadcc0c6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.040+0000 7fadd41ff700 1 --2- 172.21.15.102:0/3147141798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc06e360 0x7fadcc0c6090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009840 tx=0x7fadbc009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.040+0000 7fadd41ff700 1 -- 172.21.15.102:0/3147141798 shutdown_connections 2023-03-03T05:45:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.040+0000 7fadd41ff700 1 --2- 172.21.15.102:0/3147141798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc06e360 0x7fadcc0c6090 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009840 tx=0x7fadbc009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.040+0000 7fadd41ff700 1 --2- 172.21.15.102:0/3147141798 >> v2:172.21.15.172:3300/0 conn(0x7fadcc0cb6d0 0x7fadcc157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.040+0000 7fadd41ff700 1 -- 172.21.15.102:0/3147141798 >> 172.21.15.102:0/3147141798 conn(0x7fadcc06af90 msgr2=0x7fadcc0caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.042+0000 7f671b1ff700 1 Processor -- start 2023-03-03T05:45:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.043+0000 7f671b1ff700 1 -- start start 2023-03-03T05:45:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7f671b1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f671406e360 0x7f671415ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7f671b1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67140c4750 0x7f67141494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7f671b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6714147b50 0x7f67141443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7f671b1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671415fe10 con 0x7f67140c4750 2023-03-03T05:45:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7f671b1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671415f2d0 con 0x7f671406e360 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7f671b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6714160020 con 0x7f6714147b50 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.045+0000 7f6718f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f671406e360 0x7f671415ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.045+0000 7f6718f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f671406e360 0x7f671415ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50364/0 (socket says 172.21.15.102:50364) 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.045+0000 7f6718f9b700 1 -- 172.21.15.102:0/4064734514 learned_addr learned my addr 172.21.15.102:0/4064734514 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.046+0000 7f6718f9b700 1 -- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6714147b50 msgr2=0x7f67141443e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.046+0000 7f6718f9b700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6714147b50 0x7f67141443e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.046+0000 7f6718f9b700 1 -- 172.21.15.102:0/4064734514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67140c4750 msgr2=0x7f67141494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.046+0000 7f6718f9b700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67140c4750 0x7f67141494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.046+0000 7f6718f9b700 1 -- 172.21.15.102:0/4064734514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704009550 con 0x7f671406e360 2023-03-03T05:45:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.046+0000 7f6718f9b700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f671406e360 0x7f671415ecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6704014040 tx=0x7f6704004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7fadd41ff700 1 -- 172.21.15.102:0/3147141798 shutdown_connections 2023-03-03T05:45:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.044+0000 7fadd41ff700 1 -- 172.21.15.102:0/3147141798 wait complete. 2023-03-03T05:45:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.047+0000 7fadd41ff700 1 Processor -- start 2023-03-03T05:45:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.048+0000 7f67117fa700 1 -- 172.21.15.102:0/4064734514 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704004510 con 0x7f671406e360 2023-03-03T05:45:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.048+0000 7f67117fa700 1 -- 172.21.15.102:0/4064734514 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670401f070 con 0x7f671406e360 2023-03-03T05:45:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.048+0000 7f67117fa700 1 -- 172.21.15.102:0/4064734514 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67040038e0 con 0x7f671406e360 2023-03-03T05:45:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.048+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671415d9c0 con 0x7f671406e360 2023-03-03T05:45:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.048+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671415d6e0 con 0x7f671406e360 2023-03-03T05:45:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.049+0000 7fadd41ff700 1 -- start start 2023-03-03T05:45:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.051+0000 7fadd41ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadcc06eea0 0x7fadcc1532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.051+0000 7fadd41ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc0cb6d0 0x7fadcc150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.051+0000 7fadd41ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadcc0c0a60 0x7fadcc150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.051+0000 7fadd41ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc14d090 con 0x7fadcc06eea0 2023-03-03T05:45:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.051+0000 7fadd41ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc14d2a0 con 0x7fadcc0c0a60 2023-03-03T05:45:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.051+0000 7fadd41ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc14d580 con 0x7fadcc0cb6d0 2023-03-03T05:45:47.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.052+0000 7fadd279c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadcc0c0a60 0x7fadcc150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.052+0000 7fadd279c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadcc0c0a60 0x7fadcc150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50376/0 (socket says 172.21.15.102:50376) 2023-03-03T05:45:47.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.052+0000 7fadd279c700 1 -- 172.21.15.102:0/4208182404 learned_addr learned my addr 172.21.15.102:0/4208182404 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.053+0000 7fadd179a700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc0cb6d0 0x7fadcc150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.053+0000 7fadd279c700 1 -- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc0cb6d0 msgr2=0x7fadcc150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.053+0000 7fadd279c700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc0cb6d0 0x7fadcc150a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.053+0000 7fadd279c700 1 -- 172.21.15.102:0/4208182404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadcc06eea0 msgr2=0x7fadcc1532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.053+0000 7fadd279c700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadcc06eea0 0x7fadcc1532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.053+0000 7fadd279c700 1 -- 172.21.15.102:0/4208182404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc0094a0 con 0x7fadcc0c0a60 2023-03-03T05:45:47.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.054+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671415dca0 con 0x7f671406e360 2023-03-03T05:45:47.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.054+0000 7fadd279c700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadcc0c0a60 0x7fadcc150fa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c420 tx=0x7fadc800c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.055+0000 7fadc27fc700 1 -- 172.21.15.102:0/4208182404 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8005d70 con 0x7fadcc0c0a60 2023-03-03T05:45:47.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.055+0000 7fadc27fc700 1 -- 172.21.15.102:0/4208182404 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc80070d0 con 0x7fadcc0c0a60 2023-03-03T05:45:47.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.055+0000 7fadc27fc700 1 -- 172.21.15.102:0/4208182404 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8007480 con 0x7fadcc0c0a60 2023-03-03T05:45:47.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.055+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc0c5890 con 0x7fadcc0c0a60 2023-03-03T05:45:47.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.055+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc0c55b0 con 0x7fadcc0c0a60 2023-03-03T05:45:47.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.057+0000 7f67117fa700 1 -- 172.21.15.102:0/4064734514 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f670400f050 con 0x7f671406e360 2023-03-03T05:45:47.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.057+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb40052f0 con 0x7fadcc0c0a60 2023-03-03T05:45:47.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.059+0000 7f67117fa700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66f4068390 0x7f66f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.059+0000 7f67117fa700 1 -- 172.21.15.102:0/4064734514 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6704021070 con 0x7f671406e360 2023-03-03T05:45:47.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.060+0000 7f6713fff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66f4068390 0x7f66f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.061+0000 7f67117fa700 1 -- 172.21.15.102:0/4064734514 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6704053830 con 0x7f671406e360 2023-03-03T05:45:47.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.065+0000 7f6713fff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66f4068390 0x7f66f406a840 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6714140af0 tx=0x7f66fc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.064+0000 7fadc27fc700 1 -- 172.21.15.102:0/4208182404 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fadc8019030 con 0x7fadcc0c0a60 2023-03-03T05:45:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.068+0000 7fadc27fc700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadb00683f0 0x7fadb006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.068+0000 7fadc27fc700 1 -- 172.21.15.102:0/4208182404 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fadc8085510 con 0x7fadcc0c0a60 2023-03-03T05:45:47.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.068+0000 7fadc27fc700 1 -- 172.21.15.102:0/4208182404 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fadc80858f0 con 0x7fadcc0c0a60 2023-03-03T05:45:47.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.073+0000 7fadd1f9b700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadb00683f0 0x7fadb006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.098+0000 7fadd1f9b700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadb00683f0 0x7fadb006a8a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fadcc04eea0 tx=0x7fadbc010000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.114+0000 7efda85ff700 1 Processor -- start 2023-03-03T05:45:47.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.114+0000 7efda85ff700 1 -- start start 2023-03-03T05:45:47.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.116+0000 7efda85ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efda006e360 0x7efda0070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.117+0000 7efda85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0166a90 0x7efda0070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.117+0000 7efda85ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00c60e0 con 0x7efda006e730 2023-03-03T05:45:47.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.117+0000 7efda85ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00725c0 con 0x7efda006e360 2023-03-03T05:45:47.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.117+0000 7efda85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda006f0e0 con 0x7efda0166a90 2023-03-03T05:45:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.118+0000 7efda5b9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efda006e730 0x7efda0166380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42584/0 (socket says 172.21.15.102:42584) 2023-03-03T05:45:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.118+0000 7efda5b9a700 1 -- 172.21.15.102:0/2114731410 learned_addr learned my addr 172.21.15.102:0/2114731410 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.119+0000 7efda6b9c700 1 --2- 172.21.15.102:0/2114731410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0166a90 0x7efda0070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.119+0000 7efda639b700 1 --2- 172.21.15.102:0/2114731410 >> v2:172.21.15.172:3300/0 conn(0x7efda006e360 0x7efda0070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.120+0000 7efda6b9c700 1 -- 172.21.15.102:0/2114731410 >> v2:172.21.15.172:3300/0 conn(0x7efda006e360 msgr2=0x7efda0070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.120+0000 7efda6b9c700 1 --2- 172.21.15.102:0/2114731410 >> v2:172.21.15.172:3300/0 conn(0x7efda006e360 0x7efda0070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.120+0000 7efda6b9c700 1 -- 172.21.15.102:0/2114731410 >> v1:172.21.15.102:6789/0 conn(0x7efda006e730 legacy=0x7efda0166380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.120+0000 7efda6b9c700 1 -- 172.21.15.102:0/2114731410 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00c54d0 con 0x7efda0166a90 2023-03-03T05:45:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.120+0000 7efda6b9c700 1 --2- 172.21.15.102:0/2114731410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0166a90 0x7efda0070cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efd90009840 tx=0x7efd90009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=984877621a0c1013 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.121+0000 7efda5399700 1 -- 172.21.15.102:0/2114731410 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011830 con 0x7efda0166a90 2023-03-03T05:45:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.121+0000 7efda5399700 1 -- 172.21.15.102:0/2114731410 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90011990 con 0x7efda0166a90 2023-03-03T05:45:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.121+0000 7efda5399700 1 -- 172.21.15.102:0/2114731410 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011d20 con 0x7efda0166a90 2023-03-03T05:45:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.122+0000 7efda85ff700 1 -- 172.21.15.102:0/2114731410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0166a90 msgr2=0x7efda0070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.122+0000 7efda85ff700 1 --2- 172.21.15.102:0/2114731410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0166a90 0x7efda0070cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efd90009840 tx=0x7efd90009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.122+0000 7efda85ff700 1 -- 172.21.15.102:0/2114731410 shutdown_connections 2023-03-03T05:45:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.122+0000 7efda85ff700 1 --2- 172.21.15.102:0/2114731410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda0166a90 0x7efda0070cd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.122+0000 7efda85ff700 1 --2- 172.21.15.102:0/2114731410 >> v2:172.21.15.172:3300/0 conn(0x7efda006e360 0x7efda0070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.122+0000 7efda85ff700 1 -- 172.21.15.102:0/2114731410 >> 172.21.15.102:0/2114731410 conn(0x7efda006af90 msgr2=0x7efda00caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.124+0000 7efda85ff700 1 -- 172.21.15.102:0/2114731410 shutdown_connections 2023-03-03T05:45:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.124+0000 7efda85ff700 1 -- 172.21.15.102:0/2114731410 wait complete. 2023-03-03T05:45:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.126+0000 7ffa887ff700 1 Processor -- start 2023-03-03T05:45:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.127+0000 7ffa887ff700 1 -- start start 2023-03-03T05:45:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.129+0000 7ffa887ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffa8014f120 0x7ffa80070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.130+0000 7ffa887ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa800c64a0 con 0x7ffa80158470 2023-03-03T05:45:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.130+0000 7ffa887ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8006f110 con 0x7ffa8006ed00 2023-03-03T05:45:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.130+0000 7ffa887ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa800c2430 con 0x7ffa8014f120 2023-03-03T05:45:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.130+0000 7ffa85d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ffa8006ed00 0x7ffa800c5430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35926/0 (socket says 172.21.15.102:35926) 2023-03-03T05:45:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.130+0000 7ffa85d9a700 1 -- 172.21.15.102:0/1004143009 learned_addr learned my addr 172.21.15.102:0/1004143009 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.131+0000 7ffa8659b700 1 --2- 172.21.15.102:0/1004143009 >> v2:172.21.15.102:3300/0 conn(0x7ffa8014f120 0x7ffa80070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.132+0000 7ffa8659b700 1 -- 172.21.15.102:0/1004143009 >> v1:172.21.15.172:6789/0 conn(0x7ffa8006ed00 legacy=0x7ffa800c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.132+0000 7ffa8659b700 1 -- 172.21.15.102:0/1004143009 >> v1:172.21.15.102:6789/0 conn(0x7ffa80158470 legacy=0x7ffa801588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.132+0000 7ffa8659b700 1 -- 172.21.15.102:0/1004143009 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa800c5ee0 con 0x7ffa8014f120 2023-03-03T05:45:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.133+0000 7ffa8659b700 1 --2- 172.21.15.102:0/1004143009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa8014f120 0x7ffa80070790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009c80 tx=0x7ffa7c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19adce73dce36586 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.132+0000 7feb3b3ff700 1 Processor -- start 2023-03-03T05:45:47.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.132+0000 7feb3b3ff700 1 -- start start 2023-03-03T05:45:47.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.130+0000 7efda85ff700 1 Processor -- start 2023-03-03T05:45:47.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.131+0000 7efda85ff700 1 -- start start 2023-03-03T05:45:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7ffa85599700 1 -- 172.21.15.102:0/1004143009 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c00ca30 con 0x7ffa8014f120 2023-03-03T05:45:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.132+0000 7efda85ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efda006e360 0x7efda0203cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.133+0000 7efda639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efda006e360 0x7efda0203cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.133+0000 7efda639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efda006e360 0x7efda0203cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39484/0 (socket says 172.21.15.102:39484) 2023-03-03T05:45:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7ffa85599700 1 -- 172.21.15.102:0/1004143009 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c00cb90 con 0x7ffa8014f120 2023-03-03T05:45:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7ffa85599700 1 -- 172.21.15.102:0/1004143009 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c003a20 con 0x7ffa8014f120 2023-03-03T05:45:47.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.134+0000 7efda85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda006e730 0x7efda02046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7efda85ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efda0166a90 0x7efda0070900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7efda85ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda0204200 con 0x7efda006e360 2023-03-03T05:45:47.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7efda85ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda0204c10 con 0x7efda0166a90 2023-03-03T05:45:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7efda85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda00c3c00 con 0x7efda006e730 2023-03-03T05:45:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7efda639b700 1 -- 172.21.15.102:0/2356537690 learned_addr learned my addr 172.21.15.102:0/2356537690 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7efda5b9a700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda006e730 0x7efda02046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7efda6b9c700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efda0166a90 0x7efda0070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.134+0000 7feb3b3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feb34153000 0x7feb34159660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.135+0000 7feb3b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb340ce760 0x7feb3406cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7ffa887ff700 1 -- 172.21.15.102:0/1004143009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa8014f120 msgr2=0x7ffa80070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7ffa887ff700 1 --2- 172.21.15.102:0/1004143009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa8014f120 0x7ffa80070790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009c80 tx=0x7ffa7c009f90 comp rx=0 tx=0).stop 2023-03-03T05:45:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7ffa887ff700 1 -- 172.21.15.102:0/1004143009 shutdown_connections 2023-03-03T05:45:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7ffa887ff700 1 --2- 172.21.15.102:0/1004143009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa8014f120 0x7ffa80070790 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7ffa887ff700 1 -- 172.21.15.102:0/1004143009 >> 172.21.15.102:0/1004143009 conn(0x7ffa8006af90 msgr2=0x7ffa800caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7efda639b700 1 -- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda006e730 msgr2=0x7efda02046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7efda639b700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda006e730 0x7efda02046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7efda639b700 1 -- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efda0166a90 msgr2=0x7efda0070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7efda639b700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efda0166a90 0x7efda0070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7efda639b700 1 -- 172.21.15.102:0/2356537690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd900094a0 con 0x7efda006e360 2023-03-03T05:45:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7feb3999c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7feb3406d060 0x7feb340cd0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35940/0 (socket says 172.21.15.102:35940) 2023-03-03T05:45:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.136+0000 7feb3999c700 1 -- 172.21.15.102:0/281809311 learned_addr learned my addr 172.21.15.102:0/281809311 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7feb3919b700 1 --2- 172.21.15.102:0/281809311 >> v2:172.21.15.172:3300/0 conn(0x7feb34153000 0x7feb34159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7ffa887ff700 1 -- 172.21.15.102:0/1004143009 shutdown_connections 2023-03-03T05:45:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb340cbaa0 con 0x7feb3406d060 2023-03-03T05:45:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb341533d0 con 0x7feb34153000 2023-03-03T05:45:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb34159ba0 con 0x7feb340ce760 2023-03-03T05:45:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7feb3899a700 1 --2- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb340ce760 0x7feb3406cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7feb2bfff700 1 -- 172.21.15.102:0/281809311 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861153063 0 0) 0x7feb340cbaa0 con 0x7feb3406d060 2023-03-03T05:45:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.138+0000 7feb2bfff700 1 -- 172.21.15.102:0/281809311 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb200035b0 con 0x7feb3406d060 2023-03-03T05:45:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.138+0000 7efda639b700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efda006e360 0x7efda0203cc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd9800efc0 tx=0x7efd9800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.138+0000 7feb3919b700 1 -- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb340ce760 msgr2=0x7feb3406cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.138+0000 7feb3919b700 1 --2- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb340ce760 0x7feb3406cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.138+0000 7feb3919b700 1 -- 172.21.15.102:0/281809311 >> v1:172.21.15.172:6789/0 conn(0x7feb3406d060 legacy=0x7feb340cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.138+0000 7feb3919b700 1 -- 172.21.15.102:0/281809311 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb340cbd80 con 0x7feb34153000 2023-03-03T05:45:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.138+0000 7feb3919b700 1 --2- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb34153000 0x7feb34159660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009d90 tx=0x7feb2c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e400492e0d8dbc3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.139+0000 7feb2bfff700 1 -- 172.21.15.102:0/281809311 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00e070 con 0x7feb34153000 2023-03-03T05:45:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.139+0000 7feb2bfff700 1 -- 172.21.15.102:0/281809311 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c003a20 con 0x7feb34153000 2023-03-03T05:45:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.139+0000 7feb2bfff700 1 -- 172.21.15.102:0/281809311 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24001670 con 0x7feb34153000 2023-03-03T05:45:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.140+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb34153000 msgr2=0x7feb34159660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.140+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb34153000 0x7feb34159660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009d90 tx=0x7feb2c00b330 comp rx=0 tx=0).stop 2023-03-03T05:45:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.140+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 shutdown_connections 2023-03-03T05:45:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.140+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb34153000 0x7feb34159660 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009d90 tx=0x7feb2c00b330 comp rx=0 tx=0).stop 2023-03-03T05:45:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.140+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/281809311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb340ce760 0x7feb3406cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.140+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 >> 172.21.15.102:0/281809311 conn(0x7feb340ceef0 msgr2=0x7feb340cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.137+0000 7ffa887ff700 1 -- 172.21.15.102:0/1004143009 wait complete. 2023-03-03T05:45:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.141+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 shutdown_connections 2023-03-03T05:45:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.142+0000 7efd96ffd700 1 -- 172.21.15.102:0/2356537690 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9800cd10 con 0x7efda006e360 2023-03-03T05:45:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.142+0000 7efd96ffd700 1 -- 172.21.15.102:0/2356537690 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9800f040 con 0x7efda006e360 2023-03-03T05:45:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.142+0000 7efd96ffd700 1 -- 172.21.15.102:0/2356537690 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd980146a0 con 0x7efda006e360 2023-03-03T05:45:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.142+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda00c5a90 con 0x7efda006e360 2023-03-03T05:45:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.142+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda00c57b0 con 0x7efda006e360 2023-03-03T05:45:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.143+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda00c5d70 con 0x7efda006e360 2023-03-03T05:45:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.141+0000 7ffa887ff700 1 Processor -- start 2023-03-03T05:45:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.141+0000 7ffa887ff700 1 -- start start 2023-03-03T05:45:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.143+0000 7ffa887ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa8014f120 0x7ffa80155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.143+0000 7ffa887ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa80158470 0x7ffa80155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.143+0000 7ffa887ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa80152160 0x7ffa801505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.143+0000 7ffa887ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8014e4f0 con 0x7ffa80158470 2023-03-03T05:45:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.143+0000 7ffa887ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa801501a0 con 0x7ffa8014f120 2023-03-03T05:45:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.143+0000 7ffa887ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8014ec90 con 0x7ffa80152160 2023-03-03T05:45:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.144+0000 7ffa8659b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa8014f120 0x7ffa80155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.144+0000 7ffa8659b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa8014f120 0x7ffa80155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50402/0 (socket says 172.21.15.102:50402) 2023-03-03T05:45:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.144+0000 7ffa8659b700 1 -- 172.21.15.102:0/192693343 learned_addr learned my addr 172.21.15.102:0/192693343 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.144+0000 7ffa86d9c700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa80152160 0x7ffa801505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.142+0000 7fd2995ff700 1 Processor -- start 2023-03-03T05:45:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.144+0000 7fd2995ff700 1 -- start start 2023-03-03T05:45:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.145+0000 7ffa8659b700 1 -- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa80152160 msgr2=0x7ffa801505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.145+0000 7ffa8659b700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa80152160 0x7ffa801505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.145+0000 7ffa8659b700 1 -- 172.21.15.102:0/192693343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa80158470 msgr2=0x7ffa80155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.145+0000 7ffa8659b700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa80158470 0x7ffa80155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.145+0000 7ffa8659b700 1 -- 172.21.15.102:0/192693343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c00d040 con 0x7ffa8014f120 2023-03-03T05:45:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.146+0000 7fd2995ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2940c3b10 0x7fd294155b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.146+0000 7fd2995ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940c46e0 con 0x7fd2940c3ee0 2023-03-03T05:45:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.147+0000 7f8ab53ff700 1 Processor -- start 2023-03-03T05:45:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.147+0000 7ffa8659b700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa8014f120 0x7ffa80155400 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00c4f0 tx=0x7ffa7c004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.146+0000 7fd2995ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940c4ef0 con 0x7fd29406d460 2023-03-03T05:45:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.146+0000 7fd2995ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd294070f40 con 0x7fd2940c3b10 2023-03-03T05:45:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.147+0000 7fd28a7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd29406d460 0x7fd29414f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35946/0 (socket says 172.21.15.102:35946) 2023-03-03T05:45:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.147+0000 7fd28a7fc700 1 -- 172.21.15.102:0/1312449697 learned_addr learned my addr 172.21.15.102:0/1312449697 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.147+0000 7fd292ffd700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2940c3b10 0x7fd294155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.147+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844821097 0 0) 0x7fd2940c4ef0 con 0x7fd29406d460 2023-03-03T05:45:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.148+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd28000bdc0 con 0x7fd29406d460 2023-03-03T05:45:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.148+0000 7ffa76ffd700 1 -- 172.21.15.102:0/192693343 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c01f070 con 0x7ffa8014f120 2023-03-03T05:45:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.148+0000 7ffa76ffd700 1 -- 172.21.15.102:0/192693343 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c004650 con 0x7ffa8014f120 2023-03-03T05:45:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.148+0000 7ffa76ffd700 1 -- 172.21.15.102:0/192693343 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c004800 con 0x7ffa8014f120 2023-03-03T05:45:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.148+0000 7f8ab53ff700 1 -- start start 2023-03-03T05:45:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.149+0000 7efd96ffd700 1 -- 172.21.15.102:0/2356537690 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd98014800 con 0x7efda006e360 2023-03-03T05:45:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.149+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2774409223 0 0) 0x7fd28000bdc0 con 0x7fd29406d460 2023-03-03T05:45:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.149+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2940c3b10 msgr2=0x7fd294155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.149+0000 7fd2927fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2940c3b10 0x7fd294155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.149+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 >> v1:172.21.15.102:6789/0 conn(0x7fd2940c3ee0 legacy=0x7fd2941603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.149+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940c4930 con 0x7fd29406d460 2023-03-03T05:45:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.149+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1894587018 0 0) 0x7fd27c001670 con 0x7fd29406d460 2023-03-03T05:45:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 >> v1:172.21.15.172:6789/0 conn(0x7fd29406d460 legacy=0x7fd29414f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd2927fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000c210 0x7fd28000c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd2927fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd28000cc20 0x7fd28000d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd2927fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28000d930 0x7fd280014220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd294070f40 con 0x7fd28000c210 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd280014840 con 0x7fd28000d930 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd280014a00 con 0x7fd28000cc20 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.150+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd28000cc20 0x7fd28000d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7fd28a7fc700 1 -- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28000d930 msgr2=0x7fd280014220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28000d930 0x7fd280014220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7fd28a7fc700 1 -- 172.21.15.102:0/1312449697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000c210 msgr2=0x7fd28000c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000c210 0x7fd28000c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7fd28a7fc700 1 -- 172.21.15.102:0/1312449697 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940c4930 con 0x7fd28000cc20 2023-03-03T05:45:47.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.148+0000 7feb3b3ff700 1 -- 172.21.15.102:0/281809311 wait complete. 2023-03-03T05:45:47.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7efd96ffd700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efd840683c0 0x7efd8406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7efd96ffd700 1 -- 172.21.15.102:0/2356537690 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efd98086f40 con 0x7efda006e360 2023-03-03T05:45:47.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7feb3b3ff700 1 Processor -- start 2023-03-03T05:45:47.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7f8ab53ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8ab014f120 0x7f8ab0070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7f8ab53ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8ab006ed00 0x7f8ab00c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7f8ab53ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab00c64a0 con 0x7f8ab00c5970 2023-03-03T05:45:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7f8ab53ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab00c2430 con 0x7f8ab006ed00 2023-03-03T05:45:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.151+0000 7f8ab53ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab006f110 con 0x7f8ab014f120 2023-03-03T05:45:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.152+0000 7f8aa7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8ab006ed00 0x7f8ab00c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.152+0000 7f8aaf7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8ab00c5970 0x7f8ab0158470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42618/0 (socket says 172.21.15.102:42618) 2023-03-03T05:45:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.152+0000 7f8aaf7fe700 1 -- 172.21.15.102:0/3488035599 learned_addr learned my addr 172.21.15.102:0/3488035599 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.152+0000 7f8aaeffd700 1 --2- 172.21.15.102:0/3488035599 >> v2:172.21.15.172:3300/0 conn(0x7f8ab014f120 0x7f8ab0070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.152+0000 7efda5b9a700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efd840683c0 0x7efd8406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.152+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd28000cc20 0x7fd28000d0e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c370 tx=0x7fd28000e090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b42e308fb848f66 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8006e220 con 0x7ffa8014f120 2023-03-03T05:45:47.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa800c61c0 con 0x7ffa8014f120 2023-03-03T05:45:47.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7f8aa7fff700 1 -- 172.21.15.102:0/3488035599 >> v2:172.21.15.172:3300/0 conn(0x7f8ab014f120 msgr2=0x7f8ab0070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7f8aa7fff700 1 --2- 172.21.15.102:0/3488035599 >> v2:172.21.15.172:3300/0 conn(0x7f8ab014f120 0x7f8ab0070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7f8aa7fff700 1 -- 172.21.15.102:0/3488035599 >> v1:172.21.15.102:6789/0 conn(0x7f8ab00c5970 legacy=0x7f8ab0158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7f8aa7fff700 1 -- 172.21.15.102:0/3488035599 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab00c5ee0 con 0x7f8ab006ed00 2023-03-03T05:45:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7f8aa7fff700 1 --2- 172.21.15.102:0/3488035599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab006ed00 0x7f8ab00c5430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00c26a0 tx=0x7f8a9c009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88afb16fea97f220 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c005960 con 0x7fd28000cc20 2023-03-03T05:45:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c005ae0 con 0x7fd28000cc20 2023-03-03T05:45:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.153+0000 7fd2927fc700 1 -- 172.21.15.102:0/1312449697 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c005e90 con 0x7fd28000cc20 2023-03-03T05:45:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.154+0000 7fd2995ff700 1 -- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd28000cc20 msgr2=0x7fd28000d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.154+0000 7fd2995ff700 1 --2- 172.21.15.102:0/1312449697 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd28000cc20 0x7fd28000d0e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c370 tx=0x7fd28000e090 comp rx=0 tx=0).stop 2023-03-03T05:45:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.155+0000 7fd2937fe700 1 -- 172.21.15.102:0/1312449697 reap_dead start 2023-03-03T05:45:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.155+0000 7fd2995ff700 1 -- 172.21.15.102:0/1312449697 shutdown_connections 2023-03-03T05:45:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.155+0000 7fd2995ff700 1 -- 172.21.15.102:0/1312449697 >> 172.21.15.102:0/1312449697 conn(0x7fd29406af90 msgr2=0x7fd2940caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.155+0000 7fd2995ff700 1 -- 172.21.15.102:0/1312449697 shutdown_connections 2023-03-03T05:45:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.155+0000 7fd2995ff700 1 -- 172.21.15.102:0/1312449697 wait complete. 2023-03-03T05:45:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.156+0000 7f8aae7fc700 1 -- 172.21.15.102:0/3488035599 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c00f940 con 0x7f8ab006ed00 2023-03-03T05:45:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.156+0000 7f8aae7fc700 1 -- 172.21.15.102:0/3488035599 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c00faa0 con 0x7f8ab006ed00 2023-03-03T05:45:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.156+0000 7f8aae7fc700 1 -- 172.21.15.102:0/3488035599 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c00fe10 con 0x7f8ab006ed00 2023-03-03T05:45:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.156+0000 7efd96ffd700 1 -- 172.21.15.102:0/2356537690 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efd980536a0 con 0x7efda006e360 2023-03-03T05:45:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.156+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8006e500 con 0x7ffa8014f120 2023-03-03T05:45:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.157+0000 7efda5b9a700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efd840683c0 0x7efd8406a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efda0072180 tx=0x7efd9c007eb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.157+0000 7f8ab53ff700 1 -- 172.21.15.102:0/3488035599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab006ed00 msgr2=0x7f8ab00c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.157+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/3488035599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab006ed00 0x7f8ab00c5430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00c26a0 tx=0x7f8a9c009490 comp rx=0 tx=0).stop 2023-03-03T05:45:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.157+0000 7f8ab53ff700 1 -- 172.21.15.102:0/3488035599 shutdown_connections 2023-03-03T05:45:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.157+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/3488035599 >> v2:172.21.15.172:3300/0 conn(0x7f8ab014f120 0x7f8ab0070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.157+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/3488035599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab006ed00 0x7f8ab00c5430 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.157+0000 7f8ab53ff700 1 -- 172.21.15.102:0/3488035599 >> 172.21.15.102:0/3488035599 conn(0x7f8ab006af90 msgr2=0x7f8ab00caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.158+0000 7ffa76ffd700 1 -- 172.21.15.102:0/192693343 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffa7c017070 con 0x7ffa8014f120 2023-03-03T05:45:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.159+0000 7fd2995ff700 1 Processor -- start 2023-03-03T05:45:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7ffa76ffd700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffa640683c0 0x7ffa6406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7ffa76ffd700 1 -- 172.21.15.102:0/192693343 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ffa7c087310 con 0x7ffa8014f120 2023-03-03T05:45:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.158+0000 7feb3b3ff700 1 -- start start 2023-03-03T05:45:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7feb3b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3406d060 0x7feb341f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7feb3b3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb340c7540 0x7feb341e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7feb3b3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb341e5d20 0x7feb340c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7feb3b3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb341f6a00 con 0x7feb340c7540 2023-03-03T05:45:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7feb3b3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340b3610 con 0x7feb341e5d20 2023-03-03T05:45:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.160+0000 7feb3b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340b2760 con 0x7feb3406d060 2023-03-03T05:45:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7feb3999c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb341e5d20 0x7feb340c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7feb3999c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb341e5d20 0x7feb340c8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50430/0 (socket says 172.21.15.102:50430) 2023-03-03T05:45:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7feb3999c700 1 -- 172.21.15.102:0/3703345601 learned_addr learned my addr 172.21.15.102:0/3703345601 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7feb3919b700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3406d060 0x7feb341f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.159+0000 7fd2995ff700 1 -- start start 2023-03-03T05:45:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7ffa85d9a700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffa640683c0 0x7ffa6406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7feb3999c700 1 -- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3406d060 msgr2=0x7feb341f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7feb3999c700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3406d060 0x7feb341f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7feb3999c700 1 -- 172.21.15.102:0/3703345601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb340c7540 msgr2=0x7feb341e57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7feb3999c700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb340c7540 0x7feb341e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7feb3999c700 1 -- 172.21.15.102:0/3703345601 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c009a70 con 0x7feb341e5d20 2023-03-03T05:45:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7fd2995ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000cc20 0x7fd294145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7fd2995ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd294159490 0x7fd2941598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7fd2995ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd294157640 0x7fd294157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7fd2995ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940c15a0 con 0x7fd28000cc20 2023-03-03T05:45:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7fd2995ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd294158780 con 0x7fd294159490 2023-03-03T05:45:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.161+0000 7fd2995ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd294159000 con 0x7fd294157640 2023-03-03T05:45:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7fd2937fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd294157640 0x7fd294157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7fd2937fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd294157640 0x7fd294157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57932/0 (socket says 172.21.15.102:57932) 2023-03-03T05:45:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7fd2937fe700 1 -- 172.21.15.102:0/3935003733 learned_addr learned my addr 172.21.15.102:0/3935003733 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.162+0000 7fd292ffd700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000cc20 0x7fd294145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7feb3999c700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb341e5d20 0x7feb340c8b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feb2400c780 tx=0x7feb2400cb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7feb29ffb700 1 -- 172.21.15.102:0/3703345601 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24004c80 con 0x7feb341e5d20 2023-03-03T05:45:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7feb29ffb700 1 -- 172.21.15.102:0/3703345601 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24004de0 con 0x7feb341e5d20 2023-03-03T05:45:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7feb29ffb700 1 -- 172.21.15.102:0/3703345601 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb240072d0 con 0x7feb341e5d20 2023-03-03T05:45:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd294159490 0x7fd2941598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7fd292ffd700 1 -- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd294157640 msgr2=0x7fd294157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7fd292ffd700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd294157640 0x7fd294157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7fd292ffd700 1 -- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd294159490 msgr2=0x7fd2941598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7fd292ffd700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd294159490 0x7fd2941598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7fd292ffd700 1 -- 172.21.15.102:0/3935003733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940711e0 con 0x7fd28000cc20 2023-03-03T05:45:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd294159490 0x7fd2941598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T05:45:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.163+0000 7ffa85d9a700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffa640683c0 0x7ffa6406a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffa8006fa90 tx=0x7ffa70005cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.164+0000 7ffa76ffd700 1 -- 172.21.15.102:0/192693343 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa7c053aa0 con 0x7ffa8014f120 2023-03-03T05:45:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.164+0000 7fd292ffd700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000cc20 0x7fd294145510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd28400b9f0 tx=0x7fd28400bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.164+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb340cc340 con 0x7feb341e5d20 2023-03-03T05:45:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.164+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb340cc060 con 0x7feb341e5d20 2023-03-03T05:45:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.165+0000 7fd28bfff700 1 -- 172.21.15.102:0/3935003733 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284008720 con 0x7fd28000cc20 2023-03-03T05:45:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.165+0000 7fd28bfff700 1 -- 172.21.15.102:0/3935003733 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd284008880 con 0x7fd28000cc20 2023-03-03T05:45:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.165+0000 7fd28bfff700 1 -- 172.21.15.102:0/3935003733 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284008c10 con 0x7fd28000cc20 2023-03-03T05:45:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.165+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2940c4c10 con 0x7fd28000cc20 2023-03-03T05:45:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.165+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2940cb230 con 0x7fd28000cc20 2023-03-03T05:45:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.167+0000 7f8ab53ff700 1 -- 172.21.15.102:0/3488035599 shutdown_connections 2023-03-03T05:45:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.167+0000 7f8ab53ff700 1 -- 172.21.15.102:0/3488035599 wait complete. 2023-03-03T05:45:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.171+0000 7f8ab53ff700 1 Processor -- start 2023-03-03T05:45:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.167+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2941592e0 con 0x7fd28000cc20 2023-03-03T05:45:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.171+0000 7fd28bfff700 1 -- 172.21.15.102:0/3935003733 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd284008d70 con 0x7fd28000cc20 2023-03-03T05:45:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.166+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb340cc620 con 0x7feb341e5d20 2023-03-03T05:45:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.173+0000 7fd28bfff700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2740683c0 0x7fd27406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.173+0000 7fd28bfff700 1 -- 172.21.15.102:0/3935003733 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd2840857f0 con 0x7fd28000cc20 2023-03-03T05:45:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.174+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2740683c0 0x7fd27406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.173+0000 7feb29ffb700 1 -- 172.21.15.102:0/3703345601 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb2401a030 con 0x7feb341e5d20 2023-03-03T05:45:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.175+0000 7feb29ffb700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb180683c0 0x7feb1806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.175+0000 7feb29ffb700 1 -- 172.21.15.102:0/3703345601 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7feb240855a0 con 0x7feb341e5d20 2023-03-03T05:45:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.175+0000 7feb3919b700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb180683c0 0x7feb1806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.176+0000 7feb29ffb700 1 -- 172.21.15.102:0/3703345601 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb24085a20 con 0x7feb341e5d20 2023-03-03T05:45:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.176+0000 7f8ab53ff700 1 -- start start 2023-03-03T05:45:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.180+0000 7feb3919b700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb180683c0 0x7feb1806a870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb340cb8d0 tx=0x7feb2c010000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.179+0000 7fd28bfff700 1 -- 172.21.15.102:0/3935003733 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd284051fe0 con 0x7fd28000cc20 2023-03-03T05:45:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.180+0000 7fd28a7fc700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2740683c0 0x7fd27406a870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd29414ba80 tx=0x7fd27c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.178+0000 7f8ab53ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ab006ed00 0x7f8ab01515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.178+0000 7f8ab53ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab00c5970 0x7f8ab01ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.178+0000 7f8ab53ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ab014f120 0x7f8ab014ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.178+0000 7f8ab53ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0150c10 con 0x7f8ab00c5970 2023-03-03T05:45:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.178+0000 7f8ab53ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab01ef710 con 0x7f8ab014f120 2023-03-03T05:45:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.178+0000 7f8ab53ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0150e20 con 0x7f8ab006ed00 2023-03-03T05:45:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.179+0000 7f8aaeffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ab006ed00 0x7f8ab01515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.179+0000 7f8aaf7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ab014f120 0x7f8ab014ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.180+0000 7f8aaeffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ab006ed00 0x7f8ab01515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57936/0 (socket says 172.21.15.102:57936) 2023-03-03T05:45:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.180+0000 7f8aaeffd700 1 -- 172.21.15.102:0/1165819171 learned_addr learned my addr 172.21.15.102:0/1165819171 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.181+0000 7f8aaeffd700 1 -- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ab014f120 msgr2=0x7f8ab014ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.181+0000 7f8aaeffd700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ab014f120 0x7f8ab014ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.181+0000 7f8aaeffd700 1 -- 172.21.15.102:0/1165819171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab00c5970 msgr2=0x7f8ab01ee890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-03T05:45:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.181+0000 7f8aaeffd700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab00c5970 0x7f8ab01ee890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.181+0000 7f8aaeffd700 1 -- 172.21.15.102:0/1165819171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c009d30 con 0x7f8ab006ed00 2023-03-03T05:45:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.182+0000 7f8aaeffd700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ab006ed00 0x7f8ab01515e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000ce40 tx=0x7f8aa000a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.182+0000 7f8aa77fe700 1 -- 172.21.15.102:0/1165819171 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0003a20 con 0x7f8ab006ed00 2023-03-03T05:45:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.182+0000 7f8aa77fe700 1 -- 172.21.15.102:0/1165819171 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa0003b80 con 0x7f8ab006ed00 2023-03-03T05:45:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.182+0000 7f8aa77fe700 1 -- 172.21.15.102:0/1165819171 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0004020 con 0x7f8ab006ed00 2023-03-03T05:45:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.183+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab006e220 con 0x7f8ab006ed00 2023-03-03T05:45:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.183+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab00c61c0 con 0x7f8ab006ed00 2023-03-03T05:45:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.185+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab006e500 con 0x7f8ab006ed00 2023-03-03T05:45:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.188+0000 7f8aa77fe700 1 -- 172.21.15.102:0/1165819171 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8aa001c020 con 0x7f8ab006ed00 2023-03-03T05:45:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.190+0000 7f8aa77fe700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a980683f0 0x7f8a9806a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.191+0000 7f8aa77fe700 1 -- 172.21.15.102:0/1165819171 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8aa0012070 con 0x7f8ab006ed00 2023-03-03T05:45:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.192+0000 7f8aa7fff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a980683f0 0x7f8a9806a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.192+0000 7f8aa77fe700 1 -- 172.21.15.102:0/1165819171 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8aa0053110 con 0x7f8ab006ed00 2023-03-03T05:45:47.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.222+0000 7f8aa7fff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a980683f0 0x7f8a9806a8a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00c26a0 tx=0x7f8a9c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.400+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc938005160 con 0x7fc954070790 2023-03-03T05:45:47.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.402+0000 7fc942ffd700 1 -- 172.21.15.102:0/1522855042 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc94c018070 con 0x7fc954070790 2023-03-03T05:45:47.407 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2023-03-03T05:45:47.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.407+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc934068390 msgr2=0x7fc93406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.407+0000 7fc958fff700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc934068390 0x7fc93406a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc9480091b0 tx=0x7fc948008000 comp rx=0 tx=0).stop 2023-03-03T05:45:47.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.407+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc954070790 msgr2=0x7fc9541619c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.407+0000 7fc958fff700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc954070790 0x7fc9541619c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc94c002080 tx=0x7fc94c00a340 comp rx=0 tx=0).stop 2023-03-03T05:45:47.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 shutdown_connections 2023-03-03T05:45:47.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc954158470 0x7fc9541485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc954070790 0x7fc9541619c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc934068390 0x7fc93406a840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 --2- 172.21.15.102:0/1522855042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc95414f120 0x7fc954152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 >> 172.21.15.102:0/1522855042 conn(0x7fc95406af90 msgr2=0x7fc954071b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 shutdown_connections 2023-03-03T05:45:47.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.408+0000 7fc958fff700 1 -- 172.21.15.102:0/1522855042 wait complete. 2023-03-03T05:45:47.461 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.0 2023-03-03T05:45:47.461 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.490+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f671406fcb0 con 0x7f671406e360 2023-03-03T05:45:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.490+0000 7f67117fa700 1 -- 172.21.15.102:0/4064734514 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6704056e50 con 0x7f671406e360 2023-03-03T05:45:47.493 INFO:teuthology.orchestra.run.smithi102.stdout:55834574857 2023-03-03T05:45:47.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.494+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66f4068390 msgr2=0x7f66f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.494+0000 7f671b1ff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66f4068390 0x7f66f406a840 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6714140af0 tx=0x7f66fc00b040 comp rx=0 tx=0).stop 2023-03-03T05:45:47.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.494+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f671406e360 msgr2=0x7f671415ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.494+0000 7f671b1ff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f671406e360 0x7f671415ecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6704014040 tx=0x7f6704004750 comp rx=0 tx=0).stop 2023-03-03T05:45:47.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 shutdown_connections 2023-03-03T05:45:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66f4068390 0x7f66f406a840 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6714147b50 0x7f67141443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f671406e360 0x7f671415ecb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 --2- 172.21.15.102:0/4064734514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67140c4750 0x7f67141494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 >> 172.21.15.102:0/4064734514 conn(0x7f671406af90 msgr2=0x7f67140caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 shutdown_connections 2023-03-03T05:45:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.495+0000 7f671b1ff700 1 -- 172.21.15.102:0/4064734514 wait complete. 2023-03-03T05:45:47.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.501+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fadb4005870 con 0x7fadcc0c0a60 2023-03-03T05:45:47.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.502+0000 7fadc27fc700 1 -- 172.21.15.102:0/4208182404 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fadc8017020 con 0x7fadcc0c0a60 2023-03-03T05:45:47.504 INFO:teuthology.orchestra.run.smithi102.stdout:55834574857 2023-03-03T05:45:47.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.506+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadb00683f0 msgr2=0x7fadb006a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.506+0000 7fadd41ff700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadb00683f0 0x7fadb006a8a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fadcc04eea0 tx=0x7fadbc010000 comp rx=0 tx=0).stop 2023-03-03T05:45:47.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.506+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadcc0c0a60 msgr2=0x7fadcc150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.506+0000 7fadd41ff700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadcc0c0a60 0x7fadcc150fa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c420 tx=0x7fadc800c7e0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 shutdown_connections 2023-03-03T05:45:47.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadcc0c0a60 0x7fadcc150fa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadcc06eea0 0x7fadcc1532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadb00683f0 0x7fadb006a8a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 --2- 172.21.15.102:0/4208182404 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadcc0cb6d0 0x7fadcc150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 >> 172.21.15.102:0/4208182404 conn(0x7fadcc06af90 msgr2=0x7fadcc152190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 shutdown_connections 2023-03-03T05:45:47.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.507+0000 7fadd41ff700 1 -- 172.21.15.102:0/4208182404 wait complete. 2023-03-03T05:45:47.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.3 2023-03-03T05:45:47.550 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.550+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8ab004e6e0 con 0x7f8ab006ed00 2023-03-03T05:45:47.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.551+0000 7f8aa77fe700 1 -- 172.21.15.102:0/1165819171 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8aa0056730 con 0x7f8ab006ed00 2023-03-03T05:45:47.553 INFO:teuthology.orchestra.run.smithi102.stdout:55834574857 2023-03-03T05:45:47.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a980683f0 msgr2=0x7f8a9806a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a980683f0 0x7f8a9806a8a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00c26a0 tx=0x7f8a9c008030 comp rx=0 tx=0).stop 2023-03-03T05:45:47.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ab006ed00 msgr2=0x7f8ab01515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ab006ed00 0x7f8ab01515e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000ce40 tx=0x7f8aa000a570 comp rx=0 tx=0).stop 2023-03-03T05:45:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 shutdown_connections 2023-03-03T05:45:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a980683f0 0x7f8a9806a8a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ab006ed00 0x7f8ab01515e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ab014f120 0x7f8ab014ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 --2- 172.21.15.102:0/1165819171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ab00c5970 0x7f8ab01ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.554+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 >> 172.21.15.102:0/1165819171 conn(0x7f8ab006af90 msgr2=0x7f8ab00c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.555+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 shutdown_connections 2023-03-03T05:45:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.555+0000 7f8ab53ff700 1 -- 172.21.15.102:0/1165819171 wait complete. 2023-03-03T05:45:47.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.1 2023-03-03T05:45:47.559 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.579+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feb340b4e70 con 0x7feb341e5d20 2023-03-03T05:45:47.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.580+0000 7feb29ffb700 1 -- 172.21.15.102:0/3703345601 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb24051e40 con 0x7feb341e5d20 2023-03-03T05:45:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.580+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd29404e6e0 con 0x7fd28000cc20 2023-03-03T05:45:47.582 INFO:teuthology.orchestra.run.smithi102.stdout:55834574858 2023-03-03T05:45:47.583 INFO:teuthology.orchestra.run.smithi102.stdout:55834574858 2023-03-03T05:45:47.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.580+0000 7fd28bfff700 1 -- 172.21.15.102:0/3935003733 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd284055600 con 0x7fd28000cc20 2023-03-03T05:45:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb180683c0 msgr2=0x7feb1806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb180683c0 0x7feb1806a870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb340cb8d0 tx=0x7feb2c010000 comp rx=0 tx=0).stop 2023-03-03T05:45:47.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb341e5d20 msgr2=0x7feb340c8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb341e5d20 0x7feb340c8b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feb2400c780 tx=0x7feb2400cb40 comp rx=0 tx=0).stop 2023-03-03T05:45:47.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 shutdown_connections 2023-03-03T05:45:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb180683c0 0x7feb1806a870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb3406d060 0x7feb341f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb341e5d20 0x7feb340c8b20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 --2- 172.21.15.102:0/3703345601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb340c7540 0x7feb341e57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 >> 172.21.15.102:0/3703345601 conn(0x7feb340ceef0 msgr2=0x7feb34158040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2740683c0 msgr2=0x7fd27406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7fd2995ff700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2740683c0 0x7fd27406a870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd29414ba80 tx=0x7fd27c007000 comp rx=0 tx=0).stop 2023-03-03T05:45:47.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000cc20 msgr2=0x7fd294145510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7fd2995ff700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000cc20 0x7fd294145510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd28400b9f0 tx=0x7fd28400bd00 comp rx=0 tx=0).stop 2023-03-03T05:45:47.591 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.7 2023-03-03T05:45:47.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 shutdown_connections 2023-03-03T05:45:47.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7feb3b3ff700 1 -- 172.21.15.102:0/3703345601 wait complete. 2023-03-03T05:45:47.592 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.584+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 shutdown_connections 2023-03-03T05:45:47.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7fd2995ff700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2740683c0 0x7fd27406a870 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7fd2995ff700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd294157640 0x7fd294157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7fd2995ff700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28000cc20 0x7fd294145510 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7fd2995ff700 1 --2- 172.21.15.102:0/3935003733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd294159490 0x7fd2941598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 >> 172.21.15.102:0/3935003733 conn(0x7fd29406af90 msgr2=0x7fd294149380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 shutdown_connections 2023-03-03T05:45:47.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.585+0000 7fd2995ff700 1 -- 172.21.15.102:0/3935003733 wait complete. 2023-03-03T05:45:47.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.591+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7efda00c3980 con 0x7efda006e360 2023-03-03T05:45:47.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.592+0000 7efd96ffd700 1 -- 172.21.15.102:0/2356537690 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd98056cc0 con 0x7efda006e360 2023-03-03T05:45:47.596 INFO:teuthology.orchestra.run.smithi102.stdout:64424509449 2023-03-03T05:45:47.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efd840683c0 msgr2=0x7efd8406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efd840683c0 0x7efd8406a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efda0072180 tx=0x7efd9c007eb0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efda006e360 msgr2=0x7efda0203cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efda006e360 0x7efda0203cc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd9800efc0 tx=0x7efd9800c3a0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 shutdown_connections 2023-03-03T05:45:47.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efda0166a90 0x7efda0070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efda006e360 0x7efda0203cc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efd840683c0 0x7efd8406a870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 --2- 172.21.15.102:0/2356537690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efda006e730 0x7efda02046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.596+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 >> 172.21.15.102:0/2356537690 conn(0x7efda006af90 msgr2=0x7efda00c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.597+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 shutdown_connections 2023-03-03T05:45:47.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.597+0000 7efda85ff700 1 -- 172.21.15.102:0/2356537690 wait complete. 2023-03-03T05:45:47.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.598+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ffa80150440 con 0x7ffa8014f120 2023-03-03T05:45:47.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.599+0000 7ffa76ffd700 1 -- 172.21.15.102:0/192693343 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa7c0570c0 con 0x7ffa8014f120 2023-03-03T05:45:47.602 INFO:teuthology.orchestra.run.smithi102.stdout:55834574857 2023-03-03T05:45:47.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffa640683c0 msgr2=0x7ffa6406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffa640683c0 0x7ffa6406a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffa8006fa90 tx=0x7ffa70005cd0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa8014f120 msgr2=0x7ffa80155400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa8014f120 0x7ffa80155400 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00c4f0 tx=0x7ffa7c004b40 comp rx=0 tx=0).stop 2023-03-03T05:45:47.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 shutdown_connections 2023-03-03T05:45:47.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffa640683c0 0x7ffa6406a870 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffa80152160 0x7ffa801505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa8014f120 0x7ffa80155400 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 --2- 172.21.15.102:0/192693343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa80158470 0x7ffa80155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 >> 172.21.15.102:0/192693343 conn(0x7ffa8006af90 msgr2=0x7ffa800c92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 shutdown_connections 2023-03-03T05:45:47.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.603+0000 7ffa887ff700 1 -- 172.21.15.102:0/192693343 wait complete. 2023-03-03T05:45:47.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574858 for osd.6 2023-03-03T05:45:47.619 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574858 for osd.4 2023-03-03T05:45:47.620 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.2 2023-03-03T05:45:47.646 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.5 2023-03-03T05:45:47.650 DEBUG:teuthology.parallel:result is None 2023-03-03T05:45:47.650 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-03-03T05:45:47.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T05:45:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.930+0000 7fbce0fff700 1 Processor -- start 2023-03-03T05:45:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.930+0000 7fbce0fff700 1 -- start start 2023-03-03T05:45:47.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbce0fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbcdc150c80 0x7fbcdc06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbce0fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbcdc0c2020 0x7fbcdc0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbce0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc0c5d60 0x7fbcdc0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbce0fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc157ad0 con 0x7fbcdc0c2020 2023-03-03T05:45:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbce0fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc0c5390 con 0x7fbcdc150c80 2023-03-03T05:45:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbce0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc0c61e0 con 0x7fbcdc0c5d60 2023-03-03T05:45:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbcd3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbcdc0c2020 0x7fbcdc0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbcd3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbcdc0c2020 0x7fbcdc0c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39522/0 (socket says 172.21.15.102:39522) 2023-03-03T05:45:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbcd3fff700 1 -- 172.21.15.102:0/2607379462 learned_addr learned my addr 172.21.15.102:0/2607379462 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbcda59c700 1 --2- 172.21.15.102:0/2607379462 >> v2:172.21.15.172:3300/0 conn(0x7fbcdc150c80 0x7fbcdc06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.932+0000 7fbcdad9d700 1 --2- 172.21.15.102:0/2607379462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc0c5d60 0x7fbcdc0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd3fff700 1 -- 172.21.15.102:0/2607379462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc0c5d60 msgr2=0x7fbcdc0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd3fff700 1 --2- 172.21.15.102:0/2607379462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc0c5d60 0x7fbcdc0c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd3fff700 1 -- 172.21.15.102:0/2607379462 >> v2:172.21.15.172:3300/0 conn(0x7fbcdc150c80 msgr2=0x7fbcdc06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd3fff700 1 --2- 172.21.15.102:0/2607379462 >> v2:172.21.15.172:3300/0 conn(0x7fbcdc150c80 0x7fbcdc06d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd3fff700 1 -- 172.21.15.102:0/2607379462 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc0c4a20 con 0x7fbcdc0c2020 2023-03-03T05:45:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcdad9d700 1 --2- 172.21.15.102:0/2607379462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc0c5d60 0x7fbcdc0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd3fff700 1 --2- 172.21.15.102:0/2607379462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc0c5820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc072a90 tx=0x7fbcc4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec6a939ae0710fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd9d9b700 1 -- 172.21.15.102:0/2607379462 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4011830 con 0x7fbcdc0c2020 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd9d9b700 1 -- 172.21.15.102:0/2607379462 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4011990 con 0x7fbcdc0c2020 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.933+0000 7fbcd9d9b700 1 -- 172.21.15.102:0/2607379462 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4011d20 con 0x7fbcdc0c2020 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 -- 172.21.15.102:0/2607379462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c2020 msgr2=0x7fbcdc0c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 --2- 172.21.15.102:0/2607379462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc0c5820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc072a90 tx=0x7fbcc4009bd0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 -- 172.21.15.102:0/2607379462 shutdown_connections 2023-03-03T05:45:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 --2- 172.21.15.102:0/2607379462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc0c5d60 0x7fbcdc0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 --2- 172.21.15.102:0/2607379462 >> v2:172.21.15.172:3300/0 conn(0x7fbcdc150c80 0x7fbcdc06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 --2- 172.21.15.102:0/2607379462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc0c5820 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 -- 172.21.15.102:0/2607379462 >> 172.21.15.102:0/2607379462 conn(0x7fbcdc06af90 msgr2=0x7fbcdc06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 -- 172.21.15.102:0/2607379462 shutdown_connections 2023-03-03T05:45:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.934+0000 7fbce0fff700 1 -- 172.21.15.102:0/2607379462 wait complete. 2023-03-03T05:45:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.936+0000 7fbce0fff700 1 Processor -- start 2023-03-03T05:45:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.936+0000 7fbce0fff700 1 -- start start 2023-03-03T05:45:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.937+0000 7fbce0fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc1f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.937+0000 7fbce0fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c5d60 0x7fbcdc1f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.937+0000 7fbce0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc150c80 0x7fbcdc1f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.937+0000 7fbce0fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1f0570 con 0x7fbcdc0c5d60 2023-03-03T05:45:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.937+0000 7fbce0fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1ef960 con 0x7fbcdc0c2020 2023-03-03T05:45:47.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.937+0000 7fbce0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1efc40 con 0x7fbcdc150c80 2023-03-03T05:45:47.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc1f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc1f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50446/0 (socket says 172.21.15.102:50446) 2023-03-03T05:45:47.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 -- 172.21.15.102:0/911348727 learned_addr learned my addr 172.21.15.102:0/911348727 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:47.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcdad9d700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc150c80 0x7fbcdc1f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcd3fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c5d60 0x7fbcdc1f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 -- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc150c80 msgr2=0x7fbcdc1f0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc150c80 0x7fbcdc1f0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 -- 172.21.15.102:0/911348727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c5d60 msgr2=0x7fbcdc1f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c5d60 0x7fbcdc1f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 -- 172.21.15.102:0/911348727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4009550 con 0x7fbcdc0c2020 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.938+0000 7fbcda59c700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc1f1b30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e870 tx=0x7fbccc00eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.939+0000 7fbcd2ffd700 1 -- 172.21.15.102:0/911348727 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc009590 con 0x7fbcdc0c2020 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.939+0000 7fbcd2ffd700 1 -- 172.21.15.102:0/911348727 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc013680 con 0x7fbcdc0c2020 2023-03-03T05:45:47.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.939+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc157510 con 0x7fbcdc0c2020 2023-03-03T05:45:47.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.939+0000 7fbcd2ffd700 1 -- 172.21.15.102:0/911348727 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0139f0 con 0x7fbcdc0c2020 2023-03-03T05:45:47.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.939+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc06f010 con 0x7fbcdc0c2020 2023-03-03T05:45:47.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.940+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc1577f0 con 0x7fbcdc0c2020 2023-03-03T05:45:47.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.941+0000 7fbcd2ffd700 1 -- 172.21.15.102:0/911348727 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbccc019030 con 0x7fbcdc0c2020 2023-03-03T05:45:47.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.942+0000 7fbcd2ffd700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcbc0682f0 0x7fbcbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:47.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.942+0000 7fbcd2ffd700 1 -- 172.21.15.102:0/911348727 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbccc085bb0 con 0x7fbcdc0c2020 2023-03-03T05:45:47.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.942+0000 7fbcd3fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcbc0682f0 0x7fbcbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:47.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.943+0000 7fbcd3fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcbc0682f0 0x7fbcbc06a7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009c10 tx=0x7fbcc4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:47.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:47.946+0000 7fbcd2ffd700 1 -- 172.21.15.102:0/911348727 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbccc0521b0 con 0x7fbcdc0c2020 2023-03-03T05:45:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.136+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 --> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbcdc1efdb0 con 0x7fbcbc0682f0 2023-03-03T05:45:48.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.144+0000 7fbcd2ffd700 1 -- 172.21.15.102:0/911348727 <== mgr.4105 v2:172.21.15.172:6832/114058 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286600 (secure 0 0 0) 0x7fbcdc1efdb0 con 0x7fbcbc0682f0 2023-03-03T05:45:48.150 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:48.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcbc0682f0 msgr2=0x7fbcbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcbc0682f0 0x7fbcbc06a7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009c10 tx=0x7fbcc4008030 comp rx=0 tx=0).stop 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcdc0c2020 msgr2=0x7fbcdc1f1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc1f1b30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e870 tx=0x7fbccc00eb80 comp rx=0 tx=0).stop 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 shutdown_connections 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcbc0682f0 0x7fbcbc06a7a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcdc150c80 0x7fbcdc1f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcdc0c2020 0x7fbcdc1f1b30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 --2- 172.21.15.102:0/911348727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcdc0c5d60 0x7fbcdc1f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 >> 172.21.15.102:0/911348727 conn(0x7fbcdc06af90 msgr2=0x7fbcdc0caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:48.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 shutdown_connections 2023-03-03T05:45:48.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.149+0000 7fbce0fff700 1 -- 172.21.15.102:0/911348727 wait complete. 2023-03-03T05:45:48.153 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2023-03-03T05:45:48.178 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-03-03T05:45:47.945443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66244,"kb_used_data":3076,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":754908476,"statfs":{"total":773094113280,"available":773026279424,"internally_reserved":0,"allocated":3149824,"data_stored":1894175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.097105"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922943+0000","last_change":"2023-03-03T05:45:35.854816+0000","last_active":"2023-03-03T05:45:36.922943+0000","last_peered":"2023-03-03T05:45:36.922943+0000","last_clean":"2023-03-03T05:45:36.922943+0000","last_became_active":"2023-03-03T05:45:35.854447+0000","last_became_peered":"2023-03-03T05:45:35.854447+0000","last_unstale":"2023-03-03T05:45:36.922943+0000","last_undegraded":"2023-03-03T05:45:36.922943+0000","last_fullsized":"2023-03-03T05:45:36.922943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850859+0000","last_change":"2023-03-03T05:45:35.862354+0000","last_active":"2023-03-03T05:45:36.850859+0000","last_peered":"2023-03-03T05:45:36.850859+0000","last_clean":"2023-03-03T05:45:36.850859+0000","last_became_active":"2023-03-03T05:45:35.862016+0000","last_became_peered":"2023-03-03T05:45:35.862016+0000","last_unstale":"2023-03-03T05:45:36.850859+0000","last_undegraded":"2023-03-03T05:45:36.850859+0000","last_fullsized":"2023-03-03T05:45:36.850859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962731+0000","last_change":"2023-03-03T05:45:35.864429+0000","last_active":"2023-03-03T05:45:35.962731+0000","last_peered":"2023-03-03T05:45:35.962731+0000","last_clean":"2023-03-03T05:45:35.962731+0000","last_became_active":"2023-03-03T05:45:35.864091+0000","last_became_peered":"2023-03-03T05:45:35.864091+0000","last_unstale":"2023-03-03T05:45:35.962731+0000","last_undegraded":"2023-03-03T05:45:35.962731+0000","last_fullsized":"2023-03-03T05:45:35.962731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962683+0000","last_change":"2023-03-03T05:45:35.861374+0000","last_active":"2023-03-03T05:45:35.962683+0000","last_peered":"2023-03-03T05:45:35.962683+0000","last_clean":"2023-03-03T05:45:35.962683+0000","last_became_active":"2023-03-03T05:45:35.860993+0000","last_became_peered":"2023-03-03T05:45:35.860993+0000","last_unstale":"2023-03-03T05:45:35.962683+0000","last_undegraded":"2023-03-03T05:45:35.962683+0000","last_fullsized":"2023-03-03T05:45:35.962683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853383+0000","last_change":"2023-03-03T05:45:35.857703+0000","last_active":"2023-03-03T05:45:36.853383+0000","last_peered":"2023-03-03T05:45:36.853383+0000","last_clean":"2023-03-03T05:45:36.853383+0000","last_became_active":"2023-03-03T05:45:35.857421+0000","last_became_peered":"2023-03-03T05:45:35.857421+0000","last_unstale":"2023-03-03T05:45:36.853383+0000","last_undegraded":"2023-03-03T05:45:36.853383+0000","last_fullsized":"2023-03-03T05:45:36.853383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851248+0000","last_change":"2023-03-03T05:45:35.859892+0000","last_active":"2023-03-03T05:45:36.851248+0000","last_peered":"2023-03-03T05:45:36.851248+0000","last_clean":"2023-03-03T05:45:36.851248+0000","last_became_active":"2023-03-03T05:45:35.859617+0000","last_became_peered":"2023-03-03T05:45:35.859617+0000","last_unstale":"2023-03-03T05:45:36.851248+0000","last_undegraded":"2023-03-03T05:45:36.851248+0000","last_fullsized":"2023-03-03T05:45:36.851248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:23:53.405094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850434+0000","last_change":"2023-03-03T05:45:35.862939+0000","last_active":"2023-03-03T05:45:36.850434+0000","last_peered":"2023-03-03T05:45:36.850434+0000","last_clean":"2023-03-03T05:45:36.850434+0000","last_became_active":"2023-03-03T05:45:35.862681+0000","last_became_peered":"2023-03-03T05:45:35.862681+0000","last_unstale":"2023-03-03T05:45:36.850434+0000","last_undegraded":"2023-03-03T05:45:36.850434+0000","last_fullsized":"2023-03-03T05:45:36.850434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:58.535750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923559+0000","last_change":"2023-03-03T05:45:35.866302+0000","last_active":"2023-03-03T05:45:36.923559+0000","last_peered":"2023-03-03T05:45:36.923559+0000","last_clean":"2023-03-03T05:45:36.923559+0000","last_became_active":"2023-03-03T05:45:35.865627+0000","last_became_peered":"2023-03-03T05:45:35.865627+0000","last_unstale":"2023-03-03T05:45:36.923559+0000","last_undegraded":"2023-03-03T05:45:36.923559+0000","last_fullsized":"2023-03-03T05:45:36.923559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853588+0000","last_change":"2023-03-03T05:45:35.856762+0000","last_active":"2023-03-03T05:45:36.853588+0000","last_peered":"2023-03-03T05:45:36.853588+0000","last_clean":"2023-03-03T05:45:36.853588+0000","last_became_active":"2023-03-03T05:45:35.856458+0000","last_became_peered":"2023-03-03T05:45:35.856458+0000","last_unstale":"2023-03-03T05:45:36.853588+0000","last_undegraded":"2023-03-03T05:45:36.853588+0000","last_fullsized":"2023-03-03T05:45:36.853588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923093+0000","last_change":"2023-03-03T05:45:35.865258+0000","last_active":"2023-03-03T05:45:36.923093+0000","last_peered":"2023-03-03T05:45:36.923093+0000","last_clean":"2023-03-03T05:45:36.923093+0000","last_became_active":"2023-03-03T05:45:35.864889+0000","last_became_peered":"2023-03-03T05:45:35.864889+0000","last_unstale":"2023-03-03T05:45:36.923093+0000","last_undegraded":"2023-03-03T05:45:36.923093+0000","last_fullsized":"2023-03-03T05:45:36.923093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923368+0000","last_change":"2023-03-03T05:45:35.868970+0000","last_active":"2023-03-03T05:45:36.923368+0000","last_peered":"2023-03-03T05:45:36.923368+0000","last_clean":"2023-03-03T05:45:36.923368+0000","last_became_active":"2023-03-03T05:45:35.868555+0000","last_became_peered":"2023-03-03T05:45:35.868555+0000","last_unstale":"2023-03-03T05:45:36.923368+0000","last_undegraded":"2023-03-03T05:45:36.923368+0000","last_fullsized":"2023-03-03T05:45:36.923368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922618+0000","last_change":"2023-03-03T05:45:35.855291+0000","last_active":"2023-03-03T05:45:36.922618+0000","last_peered":"2023-03-03T05:45:36.922618+0000","last_clean":"2023-03-03T05:45:36.922618+0000","last_became_active":"2023-03-03T05:45:35.854925+0000","last_became_peered":"2023-03-03T05:45:35.854925+0000","last_unstale":"2023-03-03T05:45:36.922618+0000","last_undegraded":"2023-03-03T05:45:36.922618+0000","last_fullsized":"2023-03-03T05:45:36.922618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.924177+0000","last_change":"2023-03-03T05:45:35.853479+0000","last_active":"2023-03-03T05:45:36.924177+0000","last_peered":"2023-03-03T05:45:36.924177+0000","last_clean":"2023-03-03T05:45:36.924177+0000","last_became_active":"2023-03-03T05:45:35.853193+0000","last_became_peered":"2023-03-03T05:45:35.853193+0000","last_unstale":"2023-03-03T05:45:36.924177+0000","last_undegraded":"2023-03-03T05:45:36.924177+0000","last_fullsized":"2023-03-03T05:45:36.924177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923036+0000","last_change":"2023-03-03T05:45:35.865976+0000","last_active":"2023-03-03T05:45:36.923036+0000","last_peered":"2023-03-03T05:45:36.923036+0000","last_clean":"2023-03-03T05:45:36.923036+0000","last_became_active":"2023-03-03T05:45:35.865561+0000","last_became_peered":"2023-03-03T05:45:35.865561+0000","last_unstale":"2023-03-03T05:45:36.923036+0000","last_undegraded":"2023-03-03T05:45:36.923036+0000","last_fullsized":"2023-03-03T05:45:36.923036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962602+0000","last_change":"2023-03-03T05:45:35.862593+0000","last_active":"2023-03-03T05:45:35.962602+0000","last_peered":"2023-03-03T05:45:35.962602+0000","last_clean":"2023-03-03T05:45:35.962602+0000","last_became_active":"2023-03-03T05:45:35.862342+0000","last_became_peered":"2023-03-03T05:45:35.862342+0000","last_unstale":"2023-03-03T05:45:35.962602+0000","last_undegraded":"2023-03-03T05:45:35.962602+0000","last_fullsized":"2023-03-03T05:45:35.962602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850467+0000","last_change":"2023-03-03T05:45:35.857623+0000","last_active":"2023-03-03T05:45:36.850467+0000","last_peered":"2023-03-03T05:45:36.850467+0000","last_clean":"2023-03-03T05:45:36.850467+0000","last_became_active":"2023-03-03T05:45:35.857357+0000","last_became_peered":"2023-03-03T05:45:35.857357+0000","last_unstale":"2023-03-03T05:45:36.850467+0000","last_undegraded":"2023-03-03T05:45:36.850467+0000","last_fullsized":"2023-03-03T05:45:36.850467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:00.850961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923102+0000","last_change":"2023-03-03T05:45:35.859115+0000","last_active":"2023-03-03T05:45:36.923102+0000","last_peered":"2023-03-03T05:45:36.923102+0000","last_clean":"2023-03-03T05:45:36.923102+0000","last_became_active":"2023-03-03T05:45:35.858794+0000","last_became_peered":"2023-03-03T05:45:35.858794+0000","last_unstale":"2023-03-03T05:45:36.923102+0000","last_undegraded":"2023-03-03T05:45:36.923102+0000","last_fullsized":"2023-03-03T05:45:36.923102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962764+0000","last_change":"2023-03-03T05:45:35.861090+0000","last_active":"2023-03-03T05:45:35.962764+0000","last_peered":"2023-03-03T05:45:35.962764+0000","last_clean":"2023-03-03T05:45:35.962764+0000","last_became_active":"2023-03-03T05:45:35.860814+0000","last_became_peered":"2023-03-03T05:45:35.860814+0000","last_unstale":"2023-03-03T05:45:35.962764+0000","last_undegraded":"2023-03-03T05:45:35.962764+0000","last_fullsized":"2023-03-03T05:45:35.962764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923139+0000","last_change":"2023-03-03T05:45:35.855611+0000","last_active":"2023-03-03T05:45:36.923139+0000","last_peered":"2023-03-03T05:45:36.923139+0000","last_clean":"2023-03-03T05:45:36.923139+0000","last_became_active":"2023-03-03T05:45:35.855333+0000","last_became_peered":"2023-03-03T05:45:35.855333+0000","last_unstale":"2023-03-03T05:45:36.923139+0000","last_undegraded":"2023-03-03T05:45:36.923139+0000","last_fullsized":"2023-03-03T05:45:36.923139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923216+0000","last_change":"2023-03-03T05:45:35.865135+0000","last_active":"2023-03-03T05:45:36.923216+0000","last_peered":"2023-03-03T05:45:36.923216+0000","last_clean":"2023-03-03T05:45:36.923216+0000","last_became_active":"2023-03-03T05:45:35.864823+0000","last_became_peered":"2023-03-03T05:45:35.864823+0000","last_unstale":"2023-03-03T05:45:36.923216+0000","last_undegraded":"2023-03-03T05:45:36.923216+0000","last_fullsized":"2023-03-03T05:45:36.923216+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923121+0000","last_change":"2023-03-03T05:45:35.864573+0000","last_active":"2023-03-03T05:45:36.923121+0000","last_peered":"2023-03-03T05:45:36.923121+0000","last_clean":"2023-03-03T05:45:36.923121+0000","last_became_active":"2023-03-03T05:45:35.864297+0000","last_became_peered":"2023-03-03T05:45:35.864297+0000","last_unstale":"2023-03-03T05:45:36.923121+0000","last_undegraded":"2023-03-03T05:45:36.923121+0000","last_fullsized":"2023-03-03T05:45:36.923121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922982+0000","last_change":"2023-03-03T05:45:35.862720+0000","last_active":"2023-03-03T05:45:36.922982+0000","last_peered":"2023-03-03T05:45:36.922982+0000","last_clean":"2023-03-03T05:45:36.922982+0000","last_became_active":"2023-03-03T05:45:35.861282+0000","last_became_peered":"2023-03-03T05:45:35.861282+0000","last_unstale":"2023-03-03T05:45:36.922982+0000","last_undegraded":"2023-03-03T05:45:36.922982+0000","last_fullsized":"2023-03-03T05:45:36.922982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853407+0000","last_change":"2023-03-03T05:45:35.859828+0000","last_active":"2023-03-03T05:45:36.853407+0000","last_peered":"2023-03-03T05:45:36.853407+0000","last_clean":"2023-03-03T05:45:36.853407+0000","last_became_active":"2023-03-03T05:45:35.859568+0000","last_became_peered":"2023-03-03T05:45:35.859568+0000","last_unstale":"2023-03-03T05:45:36.853407+0000","last_undegraded":"2023-03-03T05:45:36.853407+0000","last_fullsized":"2023-03-03T05:45:36.853407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923461+0000","last_change":"2023-03-03T05:45:35.864435+0000","last_active":"2023-03-03T05:45:36.923461+0000","last_peered":"2023-03-03T05:45:36.923461+0000","last_clean":"2023-03-03T05:45:36.923461+0000","last_became_active":"2023-03-03T05:45:35.864129+0000","last_became_peered":"2023-03-03T05:45:35.864129+0000","last_unstale":"2023-03-03T05:45:36.923461+0000","last_undegraded":"2023-03-03T05:45:36.923461+0000","last_fullsized":"2023-03-03T05:45:36.923461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853682+0000","last_change":"2023-03-03T05:45:35.855909+0000","last_active":"2023-03-03T05:45:36.853682+0000","last_peered":"2023-03-03T05:45:36.853682+0000","last_clean":"2023-03-03T05:45:36.853682+0000","last_became_active":"2023-03-03T05:45:35.855388+0000","last_became_peered":"2023-03-03T05:45:35.855388+0000","last_unstale":"2023-03-03T05:45:36.853682+0000","last_undegraded":"2023-03-03T05:45:36.853682+0000","last_fullsized":"2023-03-03T05:45:36.853682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923513+0000","last_change":"2023-03-03T05:45:35.863167+0000","last_active":"2023-03-03T05:45:36.923513+0000","last_peered":"2023-03-03T05:45:36.923513+0000","last_clean":"2023-03-03T05:45:36.923513+0000","last_became_active":"2023-03-03T05:45:35.862519+0000","last_became_peered":"2023-03-03T05:45:35.862519+0000","last_unstale":"2023-03-03T05:45:36.923513+0000","last_undegraded":"2023-03-03T05:45:36.923513+0000","last_fullsized":"2023-03-03T05:45:36.923513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851022+0000","last_change":"2023-03-03T05:45:35.870150+0000","last_active":"2023-03-03T05:45:36.851022+0000","last_peered":"2023-03-03T05:45:36.851022+0000","last_clean":"2023-03-03T05:45:36.851022+0000","last_became_active":"2023-03-03T05:45:35.868786+0000","last_became_peered":"2023-03-03T05:45:35.868786+0000","last_unstale":"2023-03-03T05:45:36.851022+0000","last_undegraded":"2023-03-03T05:45:36.851022+0000","last_fullsized":"2023-03-03T05:45:36.851022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:52:23.931405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.310449+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.310197+0000","last_became_peered":"2023-03-03T05:45:35.310197+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923377+0000","last_change":"2023-03-03T05:45:35.861444+0000","last_active":"2023-03-03T05:45:36.923377+0000","last_peered":"2023-03-03T05:45:36.923377+0000","last_clean":"2023-03-03T05:45:36.923377+0000","last_became_active":"2023-03-03T05:45:35.861129+0000","last_became_peered":"2023-03-03T05:45:35.861129+0000","last_unstale":"2023-03-03T05:45:36.923377+0000","last_undegraded":"2023-03-03T05:45:36.923377+0000","last_fullsized":"2023-03-03T05:45:36.923377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843345+0000","last_change":"2023-03-03T05:45:35.308492+0000","last_active":"2023-03-03T05:45:35.843345+0000","last_peered":"2023-03-03T05:45:35.843345+0000","last_clean":"2023-03-03T05:45:35.843345+0000","last_became_active":"2023-03-03T05:45:35.308276+0000","last_became_peered":"2023-03-03T05:45:35.308276+0000","last_unstale":"2023-03-03T05:45:35.843345+0000","last_undegraded":"2023-03-03T05:45:35.843345+0000","last_fullsized":"2023-03-03T05:45:35.843345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:32.932001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853620+0000","last_change":"2023-03-03T05:45:35.866408+0000","last_active":"2023-03-03T05:45:36.853620+0000","last_peered":"2023-03-03T05:45:36.853620+0000","last_clean":"2023-03-03T05:45:36.853620+0000","last_became_active":"2023-03-03T05:45:35.866080+0000","last_became_peered":"2023-03-03T05:45:35.866080+0000","last_unstale":"2023-03-03T05:45:36.853620+0000","last_undegraded":"2023-03-03T05:45:36.853620+0000","last_fullsized":"2023-03-03T05:45:36.853620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:22.678556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848138+0000","last_change":"2023-03-03T05:45:35.293990+0000","last_active":"2023-03-03T05:45:35.848138+0000","last_peered":"2023-03-03T05:45:35.848138+0000","last_clean":"2023-03-03T05:45:35.848138+0000","last_became_active":"2023-03-03T05:45:35.293655+0000","last_became_peered":"2023-03-03T05:45:35.293655+0000","last_unstale":"2023-03-03T05:45:35.848138+0000","last_undegraded":"2023-03-03T05:45:35.848138+0000","last_fullsized":"2023-03-03T05:45:35.848138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850700+0000","last_change":"2023-03-03T05:45:35.852599+0000","last_active":"2023-03-03T05:45:36.850700+0000","last_peered":"2023-03-03T05:45:36.850700+0000","last_clean":"2023-03-03T05:45:36.850700+0000","last_became_active":"2023-03-03T05:45:35.852292+0000","last_became_peered":"2023-03-03T05:45:35.852292+0000","last_unstale":"2023-03-03T05:45:36.850700+0000","last_undegraded":"2023-03-03T05:45:36.850700+0000","last_fullsized":"2023-03-03T05:45:36.850700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:06.104140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848641+0000","last_change":"2023-03-03T05:45:35.297718+0000","last_active":"2023-03-03T05:45:35.848641+0000","last_peered":"2023-03-03T05:45:35.848641+0000","last_clean":"2023-03-03T05:45:35.848641+0000","last_became_active":"2023-03-03T05:45:35.297435+0000","last_became_peered":"2023-03-03T05:45:35.297435+0000","last_unstale":"2023-03-03T05:45:35.848641+0000","last_undegraded":"2023-03-03T05:45:35.848641+0000","last_fullsized":"2023-03-03T05:45:35.848641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:17:39.778447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923577+0000","last_change":"2023-03-03T05:45:35.871116+0000","last_active":"2023-03-03T05:45:36.923577+0000","last_peered":"2023-03-03T05:45:36.923577+0000","last_clean":"2023-03-03T05:45:36.923577+0000","last_became_active":"2023-03-03T05:45:35.870843+0000","last_became_peered":"2023-03-03T05:45:35.870843+0000","last_unstale":"2023-03-03T05:45:36.923577+0000","last_undegraded":"2023-03-03T05:45:36.923577+0000","last_fullsized":"2023-03-03T05:45:36.923577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851170+0000","last_change":"2023-03-03T05:45:35.304400+0000","last_active":"2023-03-03T05:45:36.851170+0000","last_peered":"2023-03-03T05:45:36.851170+0000","last_clean":"2023-03-03T05:45:36.851170+0000","last_became_active":"2023-03-03T05:45:35.304093+0000","last_became_peered":"2023-03-03T05:45:35.304093+0000","last_unstale":"2023-03-03T05:45:36.851170+0000","last_undegraded":"2023-03-03T05:45:36.851170+0000","last_fullsized":"2023-03-03T05:45:36.851170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853210+0000","last_change":"2023-03-03T05:45:35.862492+0000","last_active":"2023-03-03T05:45:36.853210+0000","last_peered":"2023-03-03T05:45:36.853210+0000","last_clean":"2023-03-03T05:45:36.853210+0000","last_became_active":"2023-03-03T05:45:35.862173+0000","last_became_peered":"2023-03-03T05:45:35.862173+0000","last_unstale":"2023-03-03T05:45:36.853210+0000","last_undegraded":"2023-03-03T05:45:36.853210+0000","last_fullsized":"2023-03-03T05:45:36.853210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.853604+0000","last_change":"2023-03-03T05:45:35.302282+0000","last_active":"2023-03-03T05:45:40.853604+0000","last_peered":"2023-03-03T05:45:40.853604+0000","last_clean":"2023-03-03T05:45:40.853604+0000","last_became_active":"2023-03-03T05:45:35.302002+0000","last_became_peered":"2023-03-03T05:45:35.302002+0000","last_unstale":"2023-03-03T05:45:40.853604+0000","last_undegraded":"2023-03-03T05:45:40.853604+0000","last_fullsized":"2023-03-03T05:45:40.853604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853414+0000","last_change":"2023-03-03T05:45:35.855928+0000","last_active":"2023-03-03T05:45:36.853414+0000","last_peered":"2023-03-03T05:45:36.853414+0000","last_clean":"2023-03-03T05:45:36.853414+0000","last_became_active":"2023-03-03T05:45:35.855612+0000","last_became_peered":"2023-03-03T05:45:35.855612+0000","last_unstale":"2023-03-03T05:45:36.853414+0000","last_undegraded":"2023-03-03T05:45:36.853414+0000","last_fullsized":"2023-03-03T05:45:36.853414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.860052+0000","last_change":"2023-03-03T05:45:35.304856+0000","last_active":"2023-03-03T05:45:40.860052+0000","last_peered":"2023-03-03T05:45:40.860052+0000","last_clean":"2023-03-03T05:45:40.860052+0000","last_became_active":"2023-03-03T05:45:35.304439+0000","last_became_peered":"2023-03-03T05:45:35.304439+0000","last_unstale":"2023-03-03T05:45:40.860052+0000","last_undegraded":"2023-03-03T05:45:40.860052+0000","last_fullsized":"2023-03-03T05:45:40.860052+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:30.692527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850547+0000","last_change":"2023-03-03T05:45:35.856887+0000","last_active":"2023-03-03T05:45:36.850547+0000","last_peered":"2023-03-03T05:45:36.850547+0000","last_clean":"2023-03-03T05:45:36.850547+0000","last_became_active":"2023-03-03T05:45:35.856498+0000","last_became_peered":"2023-03-03T05:45:35.856498+0000","last_unstale":"2023-03-03T05:45:36.850547+0000","last_undegraded":"2023-03-03T05:45:36.850547+0000","last_fullsized":"2023-03-03T05:45:36.850547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858066+0000","last_change":"2023-03-03T05:45:35.297635+0000","last_active":"2023-03-03T05:45:40.858066+0000","last_peered":"2023-03-03T05:45:40.858066+0000","last_clean":"2023-03-03T05:45:40.858066+0000","last_became_active":"2023-03-03T05:45:35.297373+0000","last_became_peered":"2023-03-03T05:45:35.297373+0000","last_unstale":"2023-03-03T05:45:40.858066+0000","last_undegraded":"2023-03-03T05:45:40.858066+0000","last_fullsized":"2023-03-03T05:45:40.858066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:48.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922855+0000","last_change":"2023-03-03T05:45:35.852113+0000","last_active":"2023-03-03T05:45:36.922855+0000","last_peered":"2023-03-03T05:45:36.922855+0000","last_clean":"2023-03-03T05:45:36.922855+0000","last_became_active":"2023-03-03T05:45:35.851799+0000","last_became_peered":"2023-03-03T05:45:35.851799+0000","last_unstale":"2023-03-03T05:45:36.922855+0000","last_undegraded":"2023-03-03T05:45:36.922855+0000","last_fullsized":"2023-03-03T05:45:36.922855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.845615+0000","last_change":"2023-03-03T05:45:35.292083+0000","last_active":"2023-03-03T05:45:35.845615+0000","last_peered":"2023-03-03T05:45:35.845615+0000","last_clean":"2023-03-03T05:45:35.845615+0000","last_became_active":"2023-03-03T05:45:35.291766+0000","last_became_peered":"2023-03-03T05:45:35.291766+0000","last_unstale":"2023-03-03T05:45:35.845615+0000","last_undegraded":"2023-03-03T05:45:35.845615+0000","last_fullsized":"2023-03-03T05:45:35.845615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851348+0000","last_change":"2023-03-03T05:45:35.870363+0000","last_active":"2023-03-03T05:45:36.851348+0000","last_peered":"2023-03-03T05:45:36.851348+0000","last_clean":"2023-03-03T05:45:36.851348+0000","last_became_active":"2023-03-03T05:45:35.868801+0000","last_became_peered":"2023-03-03T05:45:35.868801+0000","last_unstale":"2023-03-03T05:45:36.851348+0000","last_undegraded":"2023-03-03T05:45:36.851348+0000","last_fullsized":"2023-03-03T05:45:36.851348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:04.045341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844327+0000","last_change":"2023-03-03T05:45:35.304592+0000","last_active":"2023-03-03T05:45:35.844327+0000","last_peered":"2023-03-03T05:45:35.844327+0000","last_clean":"2023-03-03T05:45:35.844327+0000","last_became_active":"2023-03-03T05:45:35.304318+0000","last_became_peered":"2023-03-03T05:45:35.304318+0000","last_unstale":"2023-03-03T05:45:35.844327+0000","last_undegraded":"2023-03-03T05:45:35.844327+0000","last_fullsized":"2023-03-03T05:45:35.844327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:08.368039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922933+0000","last_change":"2023-03-03T05:45:35.857951+0000","last_active":"2023-03-03T05:45:36.922933+0000","last_peered":"2023-03-03T05:45:36.922933+0000","last_clean":"2023-03-03T05:45:36.922933+0000","last_became_active":"2023-03-03T05:45:35.857581+0000","last_became_peered":"2023-03-03T05:45:35.857581+0000","last_unstale":"2023-03-03T05:45:36.922933+0000","last_undegraded":"2023-03-03T05:45:36.922933+0000","last_fullsized":"2023-03-03T05:45:36.922933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858611+0000","last_change":"2023-03-03T05:45:35.309150+0000","last_active":"2023-03-03T05:45:40.858611+0000","last_peered":"2023-03-03T05:45:40.858611+0000","last_clean":"2023-03-03T05:45:40.858611+0000","last_became_active":"2023-03-03T05:45:35.308840+0000","last_became_peered":"2023-03-03T05:45:35.308840+0000","last_unstale":"2023-03-03T05:45:40.858611+0000","last_undegraded":"2023-03-03T05:45:40.858611+0000","last_fullsized":"2023-03-03T05:45:40.858611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923050+0000","last_change":"2023-03-03T05:45:35.864919+0000","last_active":"2023-03-03T05:45:36.923050+0000","last_peered":"2023-03-03T05:45:36.923050+0000","last_clean":"2023-03-03T05:45:36.923050+0000","last_became_active":"2023-03-03T05:45:35.864595+0000","last_became_peered":"2023-03-03T05:45:35.864595+0000","last_unstale":"2023-03-03T05:45:36.923050+0000","last_undegraded":"2023-03-03T05:45:36.923050+0000","last_fullsized":"2023-03-03T05:45:36.923050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.308976+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.308707+0000","last_became_peered":"2023-03-03T05:45:35.308707+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850640+0000","last_change":"2023-03-03T05:45:35.868861+0000","last_active":"2023-03-03T05:45:36.850640+0000","last_peered":"2023-03-03T05:45:36.850640+0000","last_clean":"2023-03-03T05:45:36.850640+0000","last_became_active":"2023-03-03T05:45:35.868438+0000","last_became_peered":"2023-03-03T05:45:35.868438+0000","last_unstale":"2023-03-03T05:45:36.850640+0000","last_undegraded":"2023-03-03T05:45:36.850640+0000","last_fullsized":"2023-03-03T05:45:36.850640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:59.810575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.854269+0000","last_change":"2023-03-03T05:45:35.303791+0000","last_active":"2023-03-03T05:45:40.854269+0000","last_peered":"2023-03-03T05:45:40.854269+0000","last_clean":"2023-03-03T05:45:40.854269+0000","last_became_active":"2023-03-03T05:45:35.303482+0000","last_became_peered":"2023-03-03T05:45:35.303482+0000","last_unstale":"2023-03-03T05:45:40.854269+0000","last_undegraded":"2023-03-03T05:45:40.854269+0000","last_fullsized":"2023-03-03T05:45:40.854269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:31.933113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923037+0000","last_change":"2023-03-03T05:45:35.864783+0000","last_active":"2023-03-03T05:45:36.923037+0000","last_peered":"2023-03-03T05:45:36.923037+0000","last_clean":"2023-03-03T05:45:36.923037+0000","last_became_active":"2023-03-03T05:45:35.864462+0000","last_became_peered":"2023-03-03T05:45:35.864462+0000","last_unstale":"2023-03-03T05:45:36.923037+0000","last_undegraded":"2023-03-03T05:45:36.923037+0000","last_fullsized":"2023-03-03T05:45:36.923037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851096+0000","last_change":"2023-03-03T05:45:35.310813+0000","last_active":"2023-03-03T05:45:36.851096+0000","last_peered":"2023-03-03T05:45:36.851096+0000","last_clean":"2023-03-03T05:45:36.851096+0000","last_became_active":"2023-03-03T05:45:35.310466+0000","last_became_peered":"2023-03-03T05:45:35.310466+0000","last_unstale":"2023-03-03T05:45:36.851096+0000","last_undegraded":"2023-03-03T05:45:36.851096+0000","last_fullsized":"2023-03-03T05:45:36.851096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:46.322139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850472+0000","last_change":"2023-03-03T05:45:35.856376+0000","last_active":"2023-03-03T05:45:36.850472+0000","last_peered":"2023-03-03T05:45:36.850472+0000","last_clean":"2023-03-03T05:45:36.850472+0000","last_became_active":"2023-03-03T05:45:35.856064+0000","last_became_peered":"2023-03-03T05:45:35.856064+0000","last_unstale":"2023-03-03T05:45:36.850472+0000","last_undegraded":"2023-03-03T05:45:36.850472+0000","last_fullsized":"2023-03-03T05:45:36.850472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962714+0000","last_change":"2023-03-03T05:45:35.295520+0000","last_active":"2023-03-03T05:45:35.962714+0000","last_peered":"2023-03-03T05:45:35.962714+0000","last_clean":"2023-03-03T05:45:35.962714+0000","last_became_active":"2023-03-03T05:45:35.295203+0000","last_became_peered":"2023-03-03T05:45:35.295203+0000","last_unstale":"2023-03-03T05:45:35.962714+0000","last_undegraded":"2023-03-03T05:45:35.962714+0000","last_fullsized":"2023-03-03T05:45:35.962714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923389+0000","last_change":"2023-03-03T05:45:35.852242+0000","last_active":"2023-03-03T05:45:36.923389+0000","last_peered":"2023-03-03T05:45:36.923389+0000","last_clean":"2023-03-03T05:45:36.923389+0000","last_became_active":"2023-03-03T05:45:35.851803+0000","last_became_peered":"2023-03-03T05:45:35.851803+0000","last_unstale":"2023-03-03T05:45:36.923389+0000","last_undegraded":"2023-03-03T05:45:36.923389+0000","last_fullsized":"2023-03-03T05:45:36.923389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:14.473334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.852939+0000","last_change":"2023-03-03T05:45:35.298542+0000","last_active":"2023-03-03T05:45:40.852939+0000","last_peered":"2023-03-03T05:45:40.852939+0000","last_clean":"2023-03-03T05:45:40.852939+0000","last_became_active":"2023-03-03T05:45:35.298219+0000","last_became_peered":"2023-03-03T05:45:35.298219+0000","last_unstale":"2023-03-03T05:45:40.852939+0000","last_undegraded":"2023-03-03T05:45:40.852939+0000","last_fullsized":"2023-03-03T05:45:40.852939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962699+0000","last_change":"2023-03-03T05:45:35.861231+0000","last_active":"2023-03-03T05:45:35.962699+0000","last_peered":"2023-03-03T05:45:35.962699+0000","last_clean":"2023-03-03T05:45:35.962699+0000","last_became_active":"2023-03-03T05:45:35.860675+0000","last_became_peered":"2023-03-03T05:45:35.860675+0000","last_unstale":"2023-03-03T05:45:35.962699+0000","last_undegraded":"2023-03-03T05:45:35.962699+0000","last_fullsized":"2023-03-03T05:45:35.962699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843762+0000","last_change":"2023-03-03T05:45:35.312143+0000","last_active":"2023-03-03T05:45:35.843762+0000","last_peered":"2023-03-03T05:45:35.843762+0000","last_clean":"2023-03-03T05:45:35.843762+0000","last_became_active":"2023-03-03T05:45:35.311848+0000","last_became_peered":"2023-03-03T05:45:35.311848+0000","last_unstale":"2023-03-03T05:45:35.843762+0000","last_undegraded":"2023-03-03T05:45:35.843762+0000","last_fullsized":"2023-03-03T05:45:35.843762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:22.294783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851303+0000","last_change":"2023-03-03T05:45:35.869075+0000","last_active":"2023-03-03T05:45:36.851303+0000","last_peered":"2023-03-03T05:45:36.851303+0000","last_clean":"2023-03-03T05:45:36.851303+0000","last_became_active":"2023-03-03T05:45:35.868413+0000","last_became_peered":"2023-03-03T05:45:35.868413+0000","last_unstale":"2023-03-03T05:45:36.851303+0000","last_undegraded":"2023-03-03T05:45:36.851303+0000","last_fullsized":"2023-03-03T05:45:36.851303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:34.743111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844226+0000","last_change":"2023-03-03T05:45:35.296611+0000","last_active":"2023-03-03T05:45:35.844226+0000","last_peered":"2023-03-03T05:45:35.844226+0000","last_clean":"2023-03-03T05:45:35.844226+0000","last_became_active":"2023-03-03T05:45:35.296294+0000","last_became_peered":"2023-03-03T05:45:35.296294+0000","last_unstale":"2023-03-03T05:45:35.844226+0000","last_undegraded":"2023-03-03T05:45:35.844226+0000","last_fullsized":"2023-03-03T05:45:35.844226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:16.655836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850964+0000","last_change":"2023-03-03T05:45:35.857451+0000","last_active":"2023-03-03T05:45:36.850964+0000","last_peered":"2023-03-03T05:45:36.850964+0000","last_clean":"2023-03-03T05:45:36.850964+0000","last_became_active":"2023-03-03T05:45:35.857087+0000","last_became_peered":"2023-03-03T05:45:35.857087+0000","last_unstale":"2023-03-03T05:45:36.850964+0000","last_undegraded":"2023-03-03T05:45:36.850964+0000","last_fullsized":"2023-03-03T05:45:36.850964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842797+0000","last_change":"2023-03-03T05:45:35.296114+0000","last_active":"2023-03-03T05:45:35.842797+0000","last_peered":"2023-03-03T05:45:35.842797+0000","last_clean":"2023-03-03T05:45:35.842797+0000","last_became_active":"2023-03-03T05:45:35.295855+0000","last_became_peered":"2023-03-03T05:45:35.295855+0000","last_unstale":"2023-03-03T05:45:35.842797+0000","last_undegraded":"2023-03-03T05:45:35.842797+0000","last_fullsized":"2023-03-03T05:45:35.842797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923246+0000","last_change":"2023-03-03T05:45:35.862249+0000","last_active":"2023-03-03T05:45:36.923246+0000","last_peered":"2023-03-03T05:45:36.923246+0000","last_clean":"2023-03-03T05:45:36.923246+0000","last_became_active":"2023-03-03T05:45:35.861773+0000","last_became_peered":"2023-03-03T05:45:35.861773+0000","last_unstale":"2023-03-03T05:45:36.923246+0000","last_undegraded":"2023-03-03T05:45:36.923246+0000","last_fullsized":"2023-03-03T05:45:36.923246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848485+0000","last_change":"2023-03-03T05:45:29.189033+0000","last_active":"2023-03-03T05:45:35.848485+0000","last_peered":"2023-03-03T05:45:35.848485+0000","last_clean":"2023-03-03T05:45:35.848485+0000","last_became_active":"2023-03-03T05:45:29.188719+0000","last_became_peered":"2023-03-03T05:45:29.188719+0000","last_unstale":"2023-03-03T05:45:35.848485+0000","last_undegraded":"2023-03-03T05:45:35.848485+0000","last_fullsized":"2023-03-03T05:45:35.848485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850356+0000","last_change":"2023-03-03T05:45:35.303756+0000","last_active":"2023-03-03T05:45:36.850356+0000","last_peered":"2023-03-03T05:45:36.850356+0000","last_clean":"2023-03-03T05:45:36.850356+0000","last_became_active":"2023-03-03T05:45:35.303448+0000","last_became_peered":"2023-03-03T05:45:35.303448+0000","last_unstale":"2023-03-03T05:45:36.850356+0000","last_undegraded":"2023-03-03T05:45:36.850356+0000","last_fullsized":"2023-03-03T05:45:36.850356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922899+0000","last_change":"2023-03-03T05:45:35.854166+0000","last_active":"2023-03-03T05:45:36.922899+0000","last_peered":"2023-03-03T05:45:36.922899+0000","last_clean":"2023-03-03T05:45:36.922899+0000","last_became_active":"2023-03-03T05:45:35.853873+0000","last_became_peered":"2023-03-03T05:45:35.853873+0000","last_unstale":"2023-03-03T05:45:36.922899+0000","last_undegraded":"2023-03-03T05:45:36.922899+0000","last_fullsized":"2023-03-03T05:45:36.922899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843699+0000","last_change":"2023-03-03T05:45:29.189075+0000","last_active":"2023-03-03T05:45:35.843699+0000","last_peered":"2023-03-03T05:45:35.843699+0000","last_clean":"2023-03-03T05:45:35.843699+0000","last_became_active":"2023-03-03T05:45:29.188778+0000","last_became_peered":"2023-03-03T05:45:29.188778+0000","last_unstale":"2023-03-03T05:45:35.843699+0000","last_undegraded":"2023-03-03T05:45:35.843699+0000","last_fullsized":"2023-03-03T05:45:35.843699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:54:35.536172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962645+0000","last_change":"2023-03-03T05:45:35.309781+0000","last_active":"2023-03-03T05:45:35.962645+0000","last_peered":"2023-03-03T05:45:35.962645+0000","last_clean":"2023-03-03T05:45:35.962645+0000","last_became_active":"2023-03-03T05:45:35.309497+0000","last_became_peered":"2023-03-03T05:45:35.309497+0000","last_unstale":"2023-03-03T05:45:35.962645+0000","last_undegraded":"2023-03-03T05:45:35.962645+0000","last_fullsized":"2023-03-03T05:45:35.962645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923571+0000","last_change":"2023-03-03T05:45:35.854721+0000","last_active":"2023-03-03T05:45:36.923571+0000","last_peered":"2023-03-03T05:45:36.923571+0000","last_clean":"2023-03-03T05:45:36.923571+0000","last_became_active":"2023-03-03T05:45:35.854423+0000","last_became_peered":"2023-03-03T05:45:35.854423+0000","last_unstale":"2023-03-03T05:45:36.923571+0000","last_undegraded":"2023-03-03T05:45:36.923571+0000","last_fullsized":"2023-03-03T05:45:36.923571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.852125+0000","last_change":"2023-03-03T05:45:29.057308+0000","last_active":"2023-03-03T05:45:35.852125+0000","last_peered":"2023-03-03T05:45:35.852125+0000","last_clean":"2023-03-03T05:45:35.852125+0000","last_became_active":"2023-03-03T05:45:29.056828+0000","last_became_peered":"2023-03-03T05:45:29.056828+0000","last_unstale":"2023-03-03T05:45:35.852125+0000","last_undegraded":"2023-03-03T05:45:35.852125+0000","last_fullsized":"2023-03-03T05:45:35.852125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:56.079893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848095+0000","last_change":"2023-03-03T05:45:35.295842+0000","last_active":"2023-03-03T05:45:35.848095+0000","last_peered":"2023-03-03T05:45:35.848095+0000","last_clean":"2023-03-03T05:45:35.848095+0000","last_became_active":"2023-03-03T05:45:35.295545+0000","last_became_peered":"2023-03-03T05:45:35.295545+0000","last_unstale":"2023-03-03T05:45:35.848095+0000","last_undegraded":"2023-03-03T05:45:35.848095+0000","last_fullsized":"2023-03-03T05:45:35.848095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923636+0000","last_change":"2023-03-03T05:45:35.862437+0000","last_active":"2023-03-03T05:45:36.923636+0000","last_peered":"2023-03-03T05:45:36.923636+0000","last_clean":"2023-03-03T05:45:36.923636+0000","last_became_active":"2023-03-03T05:45:35.862152+0000","last_became_peered":"2023-03-03T05:45:35.862152+0000","last_unstale":"2023-03-03T05:45:36.923636+0000","last_undegraded":"2023-03-03T05:45:36.923636+0000","last_fullsized":"2023-03-03T05:45:36.923636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842862+0000","last_change":"2023-03-03T05:45:29.057646+0000","last_active":"2023-03-03T05:45:35.842862+0000","last_peered":"2023-03-03T05:45:35.842862+0000","last_clean":"2023-03-03T05:45:35.842862+0000","last_became_active":"2023-03-03T05:45:29.057330+0000","last_became_peered":"2023-03-03T05:45:29.057330+0000","last_unstale":"2023-03-03T05:45:35.842862+0000","last_undegraded":"2023-03-03T05:45:35.842862+0000","last_fullsized":"2023-03-03T05:45:35.842862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:22:57.039574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.858102+0000","last_change":"2023-03-03T05:45:22.224435+0000","last_active":"2023-03-03T05:45:35.858102+0000","last_peered":"2023-03-03T05:45:35.858102+0000","last_clean":"2023-03-03T05:45:35.858102+0000","last_became_active":"2023-03-03T05:45:22.222958+0000","last_became_peered":"2023-03-03T05:45:22.222958+0000","last_unstale":"2023-03-03T05:45:35.858102+0000","last_undegraded":"2023-03-03T05:45:35.858102+0000","last_fullsized":"2023-03-03T05:45:35.858102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_clean_scrub_stamp":"2023-03-03T05:45:20.925942+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:10:20.426026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843695+0000","last_change":"2023-03-03T05:45:35.304483+0000","last_active":"2023-03-03T05:45:35.843695+0000","last_peered":"2023-03-03T05:45:35.843695+0000","last_clean":"2023-03-03T05:45:35.843695+0000","last_became_active":"2023-03-03T05:45:35.304202+0000","last_became_peered":"2023-03-03T05:45:35.304202+0000","last_unstale":"2023-03-03T05:45:35.843695+0000","last_undegraded":"2023-03-03T05:45:35.843695+0000","last_fullsized":"2023-03-03T05:45:35.843695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851295+0000","last_change":"2023-03-03T05:45:35.869049+0000","last_active":"2023-03-03T05:45:36.851295+0000","last_peered":"2023-03-03T05:45:36.851295+0000","last_clean":"2023-03-03T05:45:36.851295+0000","last_became_active":"2023-03-03T05:45:35.868376+0000","last_became_peered":"2023-03-03T05:45:35.868376+0000","last_unstale":"2023-03-03T05:45:36.851295+0000","last_undegraded":"2023-03-03T05:45:36.851295+0000","last_fullsized":"2023-03-03T05:45:36.851295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:42:55.207846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843590+0000","last_change":"2023-03-03T05:45:29.056874+0000","last_active":"2023-03-03T05:45:35.843590+0000","last_peered":"2023-03-03T05:45:35.843590+0000","last_clean":"2023-03-03T05:45:35.843590+0000","last_became_active":"2023-03-03T05:45:29.056438+0000","last_became_peered":"2023-03-03T05:45:29.056438+0000","last_unstale":"2023-03-03T05:45:35.843590+0000","last_undegraded":"2023-03-03T05:45:35.843590+0000","last_fullsized":"2023-03-03T05:45:35.843590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:08.113953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.846141+0000","last_change":"2023-03-03T05:45:35.304562+0000","last_active":"2023-03-03T05:45:35.846141+0000","last_peered":"2023-03-03T05:45:35.846141+0000","last_clean":"2023-03-03T05:45:35.846141+0000","last_became_active":"2023-03-03T05:45:35.304264+0000","last_became_peered":"2023-03-03T05:45:35.304264+0000","last_unstale":"2023-03-03T05:45:35.846141+0000","last_undegraded":"2023-03-03T05:45:35.846141+0000","last_fullsized":"2023-03-03T05:45:35.846141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923094+0000","last_change":"2023-03-03T05:45:35.859779+0000","last_active":"2023-03-03T05:45:36.923094+0000","last_peered":"2023-03-03T05:45:36.923094+0000","last_clean":"2023-03-03T05:45:36.923094+0000","last_became_active":"2023-03-03T05:45:35.857596+0000","last_became_peered":"2023-03-03T05:45:35.857596+0000","last_unstale":"2023-03-03T05:45:36.923094+0000","last_undegraded":"2023-03-03T05:45:36.923094+0000","last_fullsized":"2023-03-03T05:45:36.923094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844418+0000","last_change":"2023-03-03T05:45:29.056891+0000","last_active":"2023-03-03T05:45:35.844418+0000","last_peered":"2023-03-03T05:45:35.844418+0000","last_clean":"2023-03-03T05:45:35.844418+0000","last_became_active":"2023-03-03T05:45:29.056483+0000","last_became_peered":"2023-03-03T05:45:29.056483+0000","last_unstale":"2023-03-03T05:45:35.844418+0000","last_undegraded":"2023-03-03T05:45:35.844418+0000","last_fullsized":"2023-03-03T05:45:35.844418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:25.701585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.847358+0000","last_change":"2023-03-03T05:45:35.293821+0000","last_active":"2023-03-03T05:45:35.847358+0000","last_peered":"2023-03-03T05:45:35.847358+0000","last_clean":"2023-03-03T05:45:35.847358+0000","last_became_active":"2023-03-03T05:45:35.293533+0000","last_became_peered":"2023-03-03T05:45:35.293533+0000","last_unstale":"2023-03-03T05:45:35.847358+0000","last_undegraded":"2023-03-03T05:45:35.847358+0000","last_fullsized":"2023-03-03T05:45:35.847358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923917+0000","last_change":"2023-03-03T05:45:35.866830+0000","last_active":"2023-03-03T05:45:36.923917+0000","last_peered":"2023-03-03T05:45:36.923917+0000","last_clean":"2023-03-03T05:45:36.923917+0000","last_became_active":"2023-03-03T05:45:35.866509+0000","last_became_peered":"2023-03-03T05:45:35.866509+0000","last_unstale":"2023-03-03T05:45:36.923917+0000","last_undegraded":"2023-03-03T05:45:36.923917+0000","last_fullsized":"2023-03-03T05:45:36.923917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.850545+0000","last_change":"2023-03-03T05:45:35.300794+0000","last_active":"2023-03-03T05:45:40.850545+0000","last_peered":"2023-03-03T05:45:40.850545+0000","last_clean":"2023-03-03T05:45:40.850545+0000","last_became_active":"2023-03-03T05:45:35.300531+0000","last_became_peered":"2023-03-03T05:45:35.300531+0000","last_unstale":"2023-03-03T05:45:40.850545+0000","last_undegraded":"2023-03-03T05:45:40.850545+0000","last_fullsized":"2023-03-03T05:45:40.850545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843343+0000","last_change":"2023-03-03T05:45:29.193601+0000","last_active":"2023-03-03T05:45:35.843343+0000","last_peered":"2023-03-03T05:45:35.843343+0000","last_clean":"2023-03-03T05:45:35.843343+0000","last_became_active":"2023-03-03T05:45:29.193262+0000","last_became_peered":"2023-03-03T05:45:29.193262+0000","last_unstale":"2023-03-03T05:45:35.843343+0000","last_undegraded":"2023-03-03T05:45:35.843343+0000","last_fullsized":"2023-03-03T05:45:35.843343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853228+0000","last_change":"2023-03-03T05:45:35.864517+0000","last_active":"2023-03-03T05:45:36.853228+0000","last_peered":"2023-03-03T05:45:36.853228+0000","last_clean":"2023-03-03T05:45:36.853228+0000","last_became_active":"2023-03-03T05:45:35.864193+0000","last_became_peered":"2023-03-03T05:45:35.864193+0000","last_unstale":"2023-03-03T05:45:36.853228+0000","last_undegraded":"2023-03-03T05:45:36.853228+0000","last_fullsized":"2023-03-03T05:45:36.853228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:11:08.433881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844985+0000","last_change":"2023-03-03T05:45:29.193546+0000","last_active":"2023-03-03T05:45:35.844985+0000","last_peered":"2023-03-03T05:45:35.844985+0000","last_clean":"2023-03-03T05:45:35.844985+0000","last_became_active":"2023-03-03T05:45:29.193172+0000","last_became_peered":"2023-03-03T05:45:29.193172+0000","last_unstale":"2023-03-03T05:45:35.844985+0000","last_undegraded":"2023-03-03T05:45:35.844985+0000","last_fullsized":"2023-03-03T05:45:35.844985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:07:42.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843485+0000","last_change":"2023-03-03T05:45:35.296693+0000","last_active":"2023-03-03T05:45:35.843485+0000","last_peered":"2023-03-03T05:45:35.843485+0000","last_clean":"2023-03-03T05:45:35.843485+0000","last_became_active":"2023-03-03T05:45:35.296469+0000","last_became_peered":"2023-03-03T05:45:35.296469+0000","last_unstale":"2023-03-03T05:45:35.843485+0000","last_undegraded":"2023-03-03T05:45:35.843485+0000","last_fullsized":"2023-03-03T05:45:35.843485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850872+0000","last_change":"2023-03-03T05:45:35.856468+0000","last_active":"2023-03-03T05:45:36.850872+0000","last_peered":"2023-03-03T05:45:36.850872+0000","last_clean":"2023-03-03T05:45:36.850872+0000","last_became_active":"2023-03-03T05:45:35.855970+0000","last_became_peered":"2023-03-03T05:45:35.855970+0000","last_unstale":"2023-03-03T05:45:36.850872+0000","last_undegraded":"2023-03-03T05:45:36.850872+0000","last_fullsized":"2023-03-03T05:45:36.850872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853299+0000","last_change":"2023-03-03T05:45:35.301855+0000","last_active":"2023-03-03T05:45:36.853299+0000","last_peered":"2023-03-03T05:45:36.853299+0000","last_clean":"2023-03-03T05:45:36.853299+0000","last_became_active":"2023-03-03T05:45:35.301622+0000","last_became_peered":"2023-03-03T05:45:35.301622+0000","last_unstale":"2023-03-03T05:45:36.853299+0000","last_undegraded":"2023-03-03T05:45:36.853299+0000","last_fullsized":"2023-03-03T05:45:36.853299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:42.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853538+0000","last_change":"2023-03-03T05:45:35.851780+0000","last_active":"2023-03-03T05:45:36.853538+0000","last_peered":"2023-03-03T05:45:36.853538+0000","last_clean":"2023-03-03T05:45:36.853538+0000","last_became_active":"2023-03-03T05:45:35.850190+0000","last_became_peered":"2023-03-03T05:45:35.850190+0000","last_unstale":"2023-03-03T05:45:36.853538+0000","last_undegraded":"2023-03-03T05:45:36.853538+0000","last_fullsized":"2023-03-03T05:45:36.853538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856909+0000","last_change":"2023-03-03T05:45:35.309047+0000","last_active":"2023-03-03T05:45:40.856909+0000","last_peered":"2023-03-03T05:45:40.856909+0000","last_clean":"2023-03-03T05:45:40.856909+0000","last_became_active":"2023-03-03T05:45:35.308564+0000","last_became_peered":"2023-03-03T05:45:35.308564+0000","last_unstale":"2023-03-03T05:45:40.856909+0000","last_undegraded":"2023-03-03T05:45:40.856909+0000","last_fullsized":"2023-03-03T05:45:40.856909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850462+0000","last_change":"2023-03-03T05:45:35.861316+0000","last_active":"2023-03-03T05:45:36.850462+0000","last_peered":"2023-03-03T05:45:36.850462+0000","last_clean":"2023-03-03T05:45:36.850462+0000","last_became_active":"2023-03-03T05:45:35.861052+0000","last_became_peered":"2023-03-03T05:45:35.861052+0000","last_unstale":"2023-03-03T05:45:36.850462+0000","last_undegraded":"2023-03-03T05:45:36.850462+0000","last_fullsized":"2023-03-03T05:45:36.850462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858612+0000","last_change":"2023-03-03T05:45:35.303666+0000","last_active":"2023-03-03T05:45:40.858612+0000","last_peered":"2023-03-03T05:45:40.858612+0000","last_clean":"2023-03-03T05:45:40.858612+0000","last_became_active":"2023-03-03T05:45:35.303213+0000","last_became_peered":"2023-03-03T05:45:35.303213+0000","last_unstale":"2023-03-03T05:45:40.858612+0000","last_undegraded":"2023-03-03T05:45:40.858612+0000","last_fullsized":"2023-03-03T05:45:40.858612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:54.389717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850392+0000","last_change":"2023-03-03T05:45:35.857442+0000","last_active":"2023-03-03T05:45:36.850392+0000","last_peered":"2023-03-03T05:45:36.850392+0000","last_clean":"2023-03-03T05:45:36.850392+0000","last_became_active":"2023-03-03T05:45:35.857130+0000","last_became_peered":"2023-03-03T05:45:35.857130+0000","last_unstale":"2023-03-03T05:45:36.850392+0000","last_undegraded":"2023-03-03T05:45:36.850392+0000","last_fullsized":"2023-03-03T05:45:36.850392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923020+0000","last_change":"2023-03-03T05:45:35.305098+0000","last_active":"2023-03-03T05:45:36.923020+0000","last_peered":"2023-03-03T05:45:36.923020+0000","last_clean":"2023-03-03T05:45:36.923020+0000","last_became_active":"2023-03-03T05:45:35.304812+0000","last_became_peered":"2023-03-03T05:45:35.304812+0000","last_unstale":"2023-03-03T05:45:36.923020+0000","last_undegraded":"2023-03-03T05:45:36.923020+0000","last_fullsized":"2023-03-03T05:45:36.923020+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923341+0000","last_change":"2023-03-03T05:45:35.858806+0000","last_active":"2023-03-03T05:45:36.923341+0000","last_peered":"2023-03-03T05:45:36.923341+0000","last_clean":"2023-03-03T05:45:36.923341+0000","last_became_active":"2023-03-03T05:45:35.858471+0000","last_became_peered":"2023-03-03T05:45:35.858471+0000","last_unstale":"2023-03-03T05:45:36.923341+0000","last_undegraded":"2023-03-03T05:45:36.923341+0000","last_fullsized":"2023-03-03T05:45:36.923341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856244+0000","last_change":"2023-03-03T05:45:35.310654+0000","last_active":"2023-03-03T05:45:40.856244+0000","last_peered":"2023-03-03T05:45:40.856244+0000","last_clean":"2023-03-03T05:45:40.856244+0000","last_became_active":"2023-03-03T05:45:35.310377+0000","last_became_peered":"2023-03-03T05:45:35.310377+0000","last_unstale":"2023-03-03T05:45:40.856244+0000","last_undegraded":"2023-03-03T05:45:40.856244+0000","last_fullsized":"2023-03-03T05:45:40.856244+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850763+0000","last_change":"2023-03-03T05:45:35.850670+0000","last_active":"2023-03-03T05:45:36.850763+0000","last_peered":"2023-03-03T05:45:36.850763+0000","last_clean":"2023-03-03T05:45:36.850763+0000","last_became_active":"2023-03-03T05:45:35.850380+0000","last_became_peered":"2023-03-03T05:45:35.850380+0000","last_unstale":"2023-03-03T05:45:36.850763+0000","last_undegraded":"2023-03-03T05:45:36.850763+0000","last_fullsized":"2023-03-03T05:45:36.850763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923387+0000","last_change":"2023-03-03T05:45:35.863660+0000","last_active":"2023-03-03T05:45:36.923387+0000","last_peered":"2023-03-03T05:45:36.923387+0000","last_clean":"2023-03-03T05:45:36.923387+0000","last_became_active":"2023-03-03T05:45:35.863234+0000","last_became_peered":"2023-03-03T05:45:35.863234+0000","last_unstale":"2023-03-03T05:45:36.923387+0000","last_undegraded":"2023-03-03T05:45:36.923387+0000","last_fullsized":"2023-03-03T05:45:36.923387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922937+0000","last_change":"2023-03-03T05:45:35.851019+0000","last_active":"2023-03-03T05:45:36.922937+0000","last_peered":"2023-03-03T05:45:36.922937+0000","last_clean":"2023-03-03T05:45:36.922937+0000","last_became_active":"2023-03-03T05:45:35.850439+0000","last_became_peered":"2023-03-03T05:45:35.850439+0000","last_unstale":"2023-03-03T05:45:36.922937+0000","last_undegraded":"2023-03-03T05:45:36.922937+0000","last_fullsized":"2023-03-03T05:45:36.922937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:43.693100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962628+0000","last_change":"2023-03-03T05:45:35.860508+0000","last_active":"2023-03-03T05:45:35.962628+0000","last_peered":"2023-03-03T05:45:35.962628+0000","last_clean":"2023-03-03T05:45:35.962628+0000","last_became_active":"2023-03-03T05:45:35.860137+0000","last_became_peered":"2023-03-03T05:45:35.860137+0000","last_unstale":"2023-03-03T05:45:35.962628+0000","last_undegraded":"2023-03-03T05:45:35.962628+0000","last_fullsized":"2023-03-03T05:45:35.962628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923487+0000","last_change":"2023-03-03T05:45:35.864824+0000","last_active":"2023-03-03T05:45:36.923487+0000","last_peered":"2023-03-03T05:45:36.923487+0000","last_clean":"2023-03-03T05:45:36.923487+0000","last_became_active":"2023-03-03T05:45:35.864466+0000","last_became_peered":"2023-03-03T05:45:35.864466+0000","last_unstale":"2023-03-03T05:45:36.923487+0000","last_undegraded":"2023-03-03T05:45:36.923487+0000","last_fullsized":"2023-03-03T05:45:36.923487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922781+0000","last_change":"2023-03-03T05:45:35.859499+0000","last_active":"2023-03-03T05:45:36.922781+0000","last_peered":"2023-03-03T05:45:36.922781+0000","last_clean":"2023-03-03T05:45:36.922781+0000","last_became_active":"2023-03-03T05:45:35.859200+0000","last_became_peered":"2023-03-03T05:45:35.859200+0000","last_unstale":"2023-03-03T05:45:36.922781+0000","last_undegraded":"2023-03-03T05:45:36.922781+0000","last_fullsized":"2023-03-03T05:45:36.922781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":90307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.202}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0950000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":88905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.093}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0620000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6210000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4189999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7110000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4589999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8068,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363772,"statfs":{"total":96636764160,"available":96628502528,"internally_reserved":0,"allocated":258048,"data_stored":90411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.524}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.3719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1349999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2539999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9088,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362752,"statfs":{"total":96636764160,"available":96627458048,"internally_reserved":0,"allocated":843776,"data_stored":679785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9140,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362700,"statfs":{"total":96636764160,"available":96627404800,"internally_reserved":0,"allocated":831488,"data_stored":678400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7988,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363852,"statfs":{"total":96636764160,"available":96628584448,"internally_reserved":0,"allocated":241664,"data_stored":88849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T05:45:48.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T05:45:48.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.458+0000 7f6a3659d700 1 Processor -- start 2023-03-03T05:45:48.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.459+0000 7f6a3659d700 1 -- start start 2023-03-03T05:45:48.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a3659d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a301564b0 0x7f6a300cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a3659d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a300c9610 0x7f6a300cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a3659d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300c9ca0 con 0x7f6a300ca440 2023-03-03T05:45:48.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a3659d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a300b4920 con 0x7f6a301564b0 2023-03-03T05:45:48.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a3659d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300b4c00 con 0x7f6a300c9610 2023-03-03T05:45:48.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a34d9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a300c9610 0x7f6a300cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a3559b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a301564b0 0x7f6a300cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a34d9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a300c9610 0x7f6a300cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57954/0 (socket says 172.21.15.102:57954) 2023-03-03T05:45:48.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.460+0000 7f6a34d9a700 1 -- 172.21.15.102:0/4027756741 learned_addr learned my addr 172.21.15.102:0/4027756741 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:48.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.461+0000 7f6a3559b700 1 -- 172.21.15.102:0/4027756741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a300c9610 msgr2=0x7f6a300cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:48.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.461+0000 7f6a3559b700 1 --2- 172.21.15.102:0/4027756741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a300c9610 0x7f6a300cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.461+0000 7f6a3559b700 1 -- 172.21.15.102:0/4027756741 >> v1:172.21.15.102:6789/0 conn(0x7f6a300ca440 legacy=0x7f6a300ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.461+0000 7f6a3559b700 1 -- 172.21.15.102:0/4027756741 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a300b46f0 con 0x7f6a301564b0 2023-03-03T05:45:48.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.461+0000 7f6a27fff700 1 -- 172.21.15.102:0/4027756741 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974290422 0 0) 0x7f6a300c9ca0 con 0x7f6a300ca440 2023-03-03T05:45:48.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.461+0000 7f6a3559b700 1 --2- 172.21.15.102:0/4027756741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a301564b0 0x7f6a300cb660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009840 tx=0x7f6a2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899fe8181bffb7a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:48.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.461+0000 7f6a27fff700 1 -- 172.21.15.102:0/4027756741 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c011830 con 0x7f6a301564b0 2023-03-03T05:45:48.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a27fff700 1 -- 172.21.15.102:0/4027756741 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c011990 con 0x7f6a301564b0 2023-03-03T05:45:48.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a27fff700 1 -- 172.21.15.102:0/4027756741 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c011d20 con 0x7f6a301564b0 2023-03-03T05:45:48.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 -- 172.21.15.102:0/4027756741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a301564b0 msgr2=0x7f6a300cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 --2- 172.21.15.102:0/4027756741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a301564b0 0x7f6a300cb660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009840 tx=0x7f6a2c009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:48.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 -- 172.21.15.102:0/4027756741 shutdown_connections 2023-03-03T05:45:48.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 --2- 172.21.15.102:0/4027756741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a301564b0 0x7f6a300cb660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 --2- 172.21.15.102:0/4027756741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a300c9610 0x7f6a300cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 -- 172.21.15.102:0/4027756741 >> 172.21.15.102:0/4027756741 conn(0x7f6a30065fb0 msgr2=0x7f6a300663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:48.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 -- 172.21.15.102:0/4027756741 shutdown_connections 2023-03-03T05:45:48.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.462+0000 7f6a3659d700 1 -- 172.21.15.102:0/4027756741 wait complete. 2023-03-03T05:45:48.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.464+0000 7f6a3659d700 1 Processor -- start 2023-03-03T05:45:48.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.465+0000 7f6a3659d700 1 -- start start 2023-03-03T05:45:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.465+0000 7f6a3659d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a300c9610 0x7f6a300aa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.465+0000 7f6a3659d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a300ca440 0x7f6a301eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3659d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a301564b0 0x7f6a30158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3659d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30157d30 con 0x7f6a300c9610 2023-03-03T05:45:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3659d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301ef240 con 0x7f6a300ca440 2023-03-03T05:45:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3659d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301510a0 con 0x7f6a301564b0 2023-03-03T05:45:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a300c9610 0x7f6a300aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a300c9610 0x7f6a300aa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39552/0 (socket says 172.21.15.102:39552) 2023-03-03T05:45:48.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 -- 172.21.15.102:0/1444058563 learned_addr learned my addr 172.21.15.102:0/1444058563 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:48.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a34d9a700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a300ca440 0x7f6a301eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a35d9c700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a301564b0 0x7f6a30158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 -- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a301564b0 msgr2=0x7f6a30158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a301564b0 0x7f6a30158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 -- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a300ca440 msgr2=0x7f6a301eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a300ca440 0x7f6a301eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.466+0000 7f6a3559b700 1 -- 172.21.15.102:0/1444058563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c0094a0 con 0x7f6a300c9610 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.467+0000 7f6a3559b700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a300c9610 0x7f6a300aa710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009b90 tx=0x7f6a2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.467+0000 7f6a25ffb700 1 -- 172.21.15.102:0/1444058563 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c0036a0 con 0x7f6a300c9610 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.467+0000 7f6a25ffb700 1 -- 172.21.15.102:0/1444058563 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c01e070 con 0x7f6a300c9610 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.467+0000 7f6a25ffb700 1 -- 172.21.15.102:0/1444058563 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c003a60 con 0x7f6a300c9610 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.467+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a300b2210 con 0x7f6a300c9610 2023-03-03T05:45:48.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.467+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a300b1f30 con 0x7f6a300c9610 2023-03-03T05:45:48.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.468+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a30156880 con 0x7f6a300c9610 2023-03-03T05:45:48.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.469+0000 7f6a25ffb700 1 -- 172.21.15.102:0/1444058563 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a2c00f050 con 0x7f6a300c9610 2023-03-03T05:45:48.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.470+0000 7f6a25ffb700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a14068320 0x7f6a1406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.471+0000 7f6a25ffb700 1 -- 172.21.15.102:0/1444058563 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6a2c086c10 con 0x7f6a300c9610 2023-03-03T05:45:48.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.471+0000 7f6a34d9a700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a14068320 0x7f6a1406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.471+0000 7f6a34d9a700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a14068320 0x7f6a1406a7d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a300cb410 tx=0x7f6a200092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:48.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.475+0000 7f6a25ffb700 1 -- 172.21.15.102:0/1444058563 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a2c053380 con 0x7f6a300c9610 2023-03-03T05:45:48.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.663+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 --> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a300677f0 con 0x7f6a14068320 2023-03-03T05:45:48.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.672+0000 7f6a25ffb700 1 -- 172.21.15.102:0/1444058563 <== mgr.4105 v2:172.21.15.172:6832/114058 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286600 (secure 0 0 0) 0x7f6a300677f0 con 0x7f6a14068320 2023-03-03T05:45:48.679 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:48.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.676+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a14068320 msgr2=0x7f6a1406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.676+0000 7f6a3659d700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a14068320 0x7f6a1406a7d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a300cb410 tx=0x7f6a200092c0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.676+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a300c9610 msgr2=0x7f6a300aa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.676+0000 7f6a3659d700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a300c9610 0x7f6a300aa710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009b90 tx=0x7f6a2c004750 comp rx=0 tx=0).stop 2023-03-03T05:45:48.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 shutdown_connections 2023-03-03T05:45:48.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a14068320 0x7f6a1406a7d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a301564b0 0x7f6a30158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a300c9610 0x7f6a300aa710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 --2- 172.21.15.102:0/1444058563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a300ca440 0x7f6a301eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 >> 172.21.15.102:0/1444058563 conn(0x7f6a30065fb0 msgr2=0x7f6a301553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 shutdown_connections 2023-03-03T05:45:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.677+0000 7f6a3659d700 1 -- 172.21.15.102:0/1444058563 wait complete. 2023-03-03T05:45:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2023-03-03T05:45:48.706 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-03-03T05:45:47.945443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66244,"kb_used_data":3076,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":754908476,"statfs":{"total":773094113280,"available":773026279424,"internally_reserved":0,"allocated":3149824,"data_stored":1894175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.097105"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922943+0000","last_change":"2023-03-03T05:45:35.854816+0000","last_active":"2023-03-03T05:45:36.922943+0000","last_peered":"2023-03-03T05:45:36.922943+0000","last_clean":"2023-03-03T05:45:36.922943+0000","last_became_active":"2023-03-03T05:45:35.854447+0000","last_became_peered":"2023-03-03T05:45:35.854447+0000","last_unstale":"2023-03-03T05:45:36.922943+0000","last_undegraded":"2023-03-03T05:45:36.922943+0000","last_fullsized":"2023-03-03T05:45:36.922943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850859+0000","last_change":"2023-03-03T05:45:35.862354+0000","last_active":"2023-03-03T05:45:36.850859+0000","last_peered":"2023-03-03T05:45:36.850859+0000","last_clean":"2023-03-03T05:45:36.850859+0000","last_became_active":"2023-03-03T05:45:35.862016+0000","last_became_peered":"2023-03-03T05:45:35.862016+0000","last_unstale":"2023-03-03T05:45:36.850859+0000","last_undegraded":"2023-03-03T05:45:36.850859+0000","last_fullsized":"2023-03-03T05:45:36.850859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962731+0000","last_change":"2023-03-03T05:45:35.864429+0000","last_active":"2023-03-03T05:45:35.962731+0000","last_peered":"2023-03-03T05:45:35.962731+0000","last_clean":"2023-03-03T05:45:35.962731+0000","last_became_active":"2023-03-03T05:45:35.864091+0000","last_became_peered":"2023-03-03T05:45:35.864091+0000","last_unstale":"2023-03-03T05:45:35.962731+0000","last_undegraded":"2023-03-03T05:45:35.962731+0000","last_fullsized":"2023-03-03T05:45:35.962731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962683+0000","last_change":"2023-03-03T05:45:35.861374+0000","last_active":"2023-03-03T05:45:35.962683+0000","last_peered":"2023-03-03T05:45:35.962683+0000","last_clean":"2023-03-03T05:45:35.962683+0000","last_became_active":"2023-03-03T05:45:35.860993+0000","last_became_peered":"2023-03-03T05:45:35.860993+0000","last_unstale":"2023-03-03T05:45:35.962683+0000","last_undegraded":"2023-03-03T05:45:35.962683+0000","last_fullsized":"2023-03-03T05:45:35.962683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853383+0000","last_change":"2023-03-03T05:45:35.857703+0000","last_active":"2023-03-03T05:45:36.853383+0000","last_peered":"2023-03-03T05:45:36.853383+0000","last_clean":"2023-03-03T05:45:36.853383+0000","last_became_active":"2023-03-03T05:45:35.857421+0000","last_became_peered":"2023-03-03T05:45:35.857421+0000","last_unstale":"2023-03-03T05:45:36.853383+0000","last_undegraded":"2023-03-03T05:45:36.853383+0000","last_fullsized":"2023-03-03T05:45:36.853383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851248+0000","last_change":"2023-03-03T05:45:35.859892+0000","last_active":"2023-03-03T05:45:36.851248+0000","last_peered":"2023-03-03T05:45:36.851248+0000","last_clean":"2023-03-03T05:45:36.851248+0000","last_became_active":"2023-03-03T05:45:35.859617+0000","last_became_peered":"2023-03-03T05:45:35.859617+0000","last_unstale":"2023-03-03T05:45:36.851248+0000","last_undegraded":"2023-03-03T05:45:36.851248+0000","last_fullsized":"2023-03-03T05:45:36.851248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:23:53.405094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850434+0000","last_change":"2023-03-03T05:45:35.862939+0000","last_active":"2023-03-03T05:45:36.850434+0000","last_peered":"2023-03-03T05:45:36.850434+0000","last_clean":"2023-03-03T05:45:36.850434+0000","last_became_active":"2023-03-03T05:45:35.862681+0000","last_became_peered":"2023-03-03T05:45:35.862681+0000","last_unstale":"2023-03-03T05:45:36.850434+0000","last_undegraded":"2023-03-03T05:45:36.850434+0000","last_fullsized":"2023-03-03T05:45:36.850434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:58.535750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923559+0000","last_change":"2023-03-03T05:45:35.866302+0000","last_active":"2023-03-03T05:45:36.923559+0000","last_peered":"2023-03-03T05:45:36.923559+0000","last_clean":"2023-03-03T05:45:36.923559+0000","last_became_active":"2023-03-03T05:45:35.865627+0000","last_became_peered":"2023-03-03T05:45:35.865627+0000","last_unstale":"2023-03-03T05:45:36.923559+0000","last_undegraded":"2023-03-03T05:45:36.923559+0000","last_fullsized":"2023-03-03T05:45:36.923559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853588+0000","last_change":"2023-03-03T05:45:35.856762+0000","last_active":"2023-03-03T05:45:36.853588+0000","last_peered":"2023-03-03T05:45:36.853588+0000","last_clean":"2023-03-03T05:45:36.853588+0000","last_became_active":"2023-03-03T05:45:35.856458+0000","last_became_peered":"2023-03-03T05:45:35.856458+0000","last_unstale":"2023-03-03T05:45:36.853588+0000","last_undegraded":"2023-03-03T05:45:36.853588+0000","last_fullsized":"2023-03-03T05:45:36.853588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923093+0000","last_change":"2023-03-03T05:45:35.865258+0000","last_active":"2023-03-03T05:45:36.923093+0000","last_peered":"2023-03-03T05:45:36.923093+0000","last_clean":"2023-03-03T05:45:36.923093+0000","last_became_active":"2023-03-03T05:45:35.864889+0000","last_became_peered":"2023-03-03T05:45:35.864889+0000","last_unstale":"2023-03-03T05:45:36.923093+0000","last_undegraded":"2023-03-03T05:45:36.923093+0000","last_fullsized":"2023-03-03T05:45:36.923093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923368+0000","last_change":"2023-03-03T05:45:35.868970+0000","last_active":"2023-03-03T05:45:36.923368+0000","last_peered":"2023-03-03T05:45:36.923368+0000","last_clean":"2023-03-03T05:45:36.923368+0000","last_became_active":"2023-03-03T05:45:35.868555+0000","last_became_peered":"2023-03-03T05:45:35.868555+0000","last_unstale":"2023-03-03T05:45:36.923368+0000","last_undegraded":"2023-03-03T05:45:36.923368+0000","last_fullsized":"2023-03-03T05:45:36.923368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922618+0000","last_change":"2023-03-03T05:45:35.855291+0000","last_active":"2023-03-03T05:45:36.922618+0000","last_peered":"2023-03-03T05:45:36.922618+0000","last_clean":"2023-03-03T05:45:36.922618+0000","last_became_active":"2023-03-03T05:45:35.854925+0000","last_became_peered":"2023-03-03T05:45:35.854925+0000","last_unstale":"2023-03-03T05:45:36.922618+0000","last_undegraded":"2023-03-03T05:45:36.922618+0000","last_fullsized":"2023-03-03T05:45:36.922618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.924177+0000","last_change":"2023-03-03T05:45:35.853479+0000","last_active":"2023-03-03T05:45:36.924177+0000","last_peered":"2023-03-03T05:45:36.924177+0000","last_clean":"2023-03-03T05:45:36.924177+0000","last_became_active":"2023-03-03T05:45:35.853193+0000","last_became_peered":"2023-03-03T05:45:35.853193+0000","last_unstale":"2023-03-03T05:45:36.924177+0000","last_undegraded":"2023-03-03T05:45:36.924177+0000","last_fullsized":"2023-03-03T05:45:36.924177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923036+0000","last_change":"2023-03-03T05:45:35.865976+0000","last_active":"2023-03-03T05:45:36.923036+0000","last_peered":"2023-03-03T05:45:36.923036+0000","last_clean":"2023-03-03T05:45:36.923036+0000","last_became_active":"2023-03-03T05:45:35.865561+0000","last_became_peered":"2023-03-03T05:45:35.865561+0000","last_unstale":"2023-03-03T05:45:36.923036+0000","last_undegraded":"2023-03-03T05:45:36.923036+0000","last_fullsized":"2023-03-03T05:45:36.923036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962602+0000","last_change":"2023-03-03T05:45:35.862593+0000","last_active":"2023-03-03T05:45:35.962602+0000","last_peered":"2023-03-03T05:45:35.962602+0000","last_clean":"2023-03-03T05:45:35.962602+0000","last_became_active":"2023-03-03T05:45:35.862342+0000","last_became_peered":"2023-03-03T05:45:35.862342+0000","last_unstale":"2023-03-03T05:45:35.962602+0000","last_undegraded":"2023-03-03T05:45:35.962602+0000","last_fullsized":"2023-03-03T05:45:35.962602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850467+0000","last_change":"2023-03-03T05:45:35.857623+0000","last_active":"2023-03-03T05:45:36.850467+0000","last_peered":"2023-03-03T05:45:36.850467+0000","last_clean":"2023-03-03T05:45:36.850467+0000","last_became_active":"2023-03-03T05:45:35.857357+0000","last_became_peered":"2023-03-03T05:45:35.857357+0000","last_unstale":"2023-03-03T05:45:36.850467+0000","last_undegraded":"2023-03-03T05:45:36.850467+0000","last_fullsized":"2023-03-03T05:45:36.850467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:00.850961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923102+0000","last_change":"2023-03-03T05:45:35.859115+0000","last_active":"2023-03-03T05:45:36.923102+0000","last_peered":"2023-03-03T05:45:36.923102+0000","last_clean":"2023-03-03T05:45:36.923102+0000","last_became_active":"2023-03-03T05:45:35.858794+0000","last_became_peered":"2023-03-03T05:45:35.858794+0000","last_unstale":"2023-03-03T05:45:36.923102+0000","last_undegraded":"2023-03-03T05:45:36.923102+0000","last_fullsized":"2023-03-03T05:45:36.923102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962764+0000","last_change":"2023-03-03T05:45:35.861090+0000","last_active":"2023-03-03T05:45:35.962764+0000","last_peered":"2023-03-03T05:45:35.962764+0000","last_clean":"2023-03-03T05:45:35.962764+0000","last_became_active":"2023-03-03T05:45:35.860814+0000","last_became_peered":"2023-03-03T05:45:35.860814+0000","last_unstale":"2023-03-03T05:45:35.962764+0000","last_undegraded":"2023-03-03T05:45:35.962764+0000","last_fullsized":"2023-03-03T05:45:35.962764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923139+0000","last_change":"2023-03-03T05:45:35.855611+0000","last_active":"2023-03-03T05:45:36.923139+0000","last_peered":"2023-03-03T05:45:36.923139+0000","last_clean":"2023-03-03T05:45:36.923139+0000","last_became_active":"2023-03-03T05:45:35.855333+0000","last_became_peered":"2023-03-03T05:45:35.855333+0000","last_unstale":"2023-03-03T05:45:36.923139+0000","last_undegraded":"2023-03-03T05:45:36.923139+0000","last_fullsized":"2023-03-03T05:45:36.923139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923216+0000","last_change":"2023-03-03T05:45:35.865135+0000","last_active":"2023-03-03T05:45:36.923216+0000","last_peered":"2023-03-03T05:45:36.923216+0000","last_clean":"2023-03-03T05:45:36.923216+0000","last_became_active":"2023-03-03T05:45:35.864823+0000","last_became_peered":"2023-03-03T05:45:35.864823+0000","last_unstale":"2023-03-03T05:45:36.923216+0000","last_undegraded":"2023-03-03T05:45:36.923216+0000","last_fullsized":"2023-03-03T05:45:36.923216+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923121+0000","last_change":"2023-03-03T05:45:35.864573+0000","last_active":"2023-03-03T05:45:36.923121+0000","last_peered":"2023-03-03T05:45:36.923121+0000","last_clean":"2023-03-03T05:45:36.923121+0000","last_became_active":"2023-03-03T05:45:35.864297+0000","last_became_peered":"2023-03-03T05:45:35.864297+0000","last_unstale":"2023-03-03T05:45:36.923121+0000","last_undegraded":"2023-03-03T05:45:36.923121+0000","last_fullsized":"2023-03-03T05:45:36.923121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922982+0000","last_change":"2023-03-03T05:45:35.862720+0000","last_active":"2023-03-03T05:45:36.922982+0000","last_peered":"2023-03-03T05:45:36.922982+0000","last_clean":"2023-03-03T05:45:36.922982+0000","last_became_active":"2023-03-03T05:45:35.861282+0000","last_became_peered":"2023-03-03T05:45:35.861282+0000","last_unstale":"2023-03-03T05:45:36.922982+0000","last_undegraded":"2023-03-03T05:45:36.922982+0000","last_fullsized":"2023-03-03T05:45:36.922982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853407+0000","last_change":"2023-03-03T05:45:35.859828+0000","last_active":"2023-03-03T05:45:36.853407+0000","last_peered":"2023-03-03T05:45:36.853407+0000","last_clean":"2023-03-03T05:45:36.853407+0000","last_became_active":"2023-03-03T05:45:35.859568+0000","last_became_peered":"2023-03-03T05:45:35.859568+0000","last_unstale":"2023-03-03T05:45:36.853407+0000","last_undegraded":"2023-03-03T05:45:36.853407+0000","last_fullsized":"2023-03-03T05:45:36.853407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923461+0000","last_change":"2023-03-03T05:45:35.864435+0000","last_active":"2023-03-03T05:45:36.923461+0000","last_peered":"2023-03-03T05:45:36.923461+0000","last_clean":"2023-03-03T05:45:36.923461+0000","last_became_active":"2023-03-03T05:45:35.864129+0000","last_became_peered":"2023-03-03T05:45:35.864129+0000","last_unstale":"2023-03-03T05:45:36.923461+0000","last_undegraded":"2023-03-03T05:45:36.923461+0000","last_fullsized":"2023-03-03T05:45:36.923461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853682+0000","last_change":"2023-03-03T05:45:35.855909+0000","last_active":"2023-03-03T05:45:36.853682+0000","last_peered":"2023-03-03T05:45:36.853682+0000","last_clean":"2023-03-03T05:45:36.853682+0000","last_became_active":"2023-03-03T05:45:35.855388+0000","last_became_peered":"2023-03-03T05:45:35.855388+0000","last_unstale":"2023-03-03T05:45:36.853682+0000","last_undegraded":"2023-03-03T05:45:36.853682+0000","last_fullsized":"2023-03-03T05:45:36.853682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923513+0000","last_change":"2023-03-03T05:45:35.863167+0000","last_active":"2023-03-03T05:45:36.923513+0000","last_peered":"2023-03-03T05:45:36.923513+0000","last_clean":"2023-03-03T05:45:36.923513+0000","last_became_active":"2023-03-03T05:45:35.862519+0000","last_became_peered":"2023-03-03T05:45:35.862519+0000","last_unstale":"2023-03-03T05:45:36.923513+0000","last_undegraded":"2023-03-03T05:45:36.923513+0000","last_fullsized":"2023-03-03T05:45:36.923513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851022+0000","last_change":"2023-03-03T05:45:35.870150+0000","last_active":"2023-03-03T05:45:36.851022+0000","last_peered":"2023-03-03T05:45:36.851022+0000","last_clean":"2023-03-03T05:45:36.851022+0000","last_became_active":"2023-03-03T05:45:35.868786+0000","last_became_peered":"2023-03-03T05:45:35.868786+0000","last_unstale":"2023-03-03T05:45:36.851022+0000","last_undegraded":"2023-03-03T05:45:36.851022+0000","last_fullsized":"2023-03-03T05:45:36.851022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:52:23.931405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.310449+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.310197+0000","last_became_peered":"2023-03-03T05:45:35.310197+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923377+0000","last_change":"2023-03-03T05:45:35.861444+0000","last_active":"2023-03-03T05:45:36.923377+0000","last_peered":"2023-03-03T05:45:36.923377+0000","last_clean":"2023-03-03T05:45:36.923377+0000","last_became_active":"2023-03-03T05:45:35.861129+0000","last_became_peered":"2023-03-03T05:45:35.861129+0000","last_unstale":"2023-03-03T05:45:36.923377+0000","last_undegraded":"2023-03-03T05:45:36.923377+0000","last_fullsized":"2023-03-03T05:45:36.923377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843345+0000","last_change":"2023-03-03T05:45:35.308492+0000","last_active":"2023-03-03T05:45:35.843345+0000","last_peered":"2023-03-03T05:45:35.843345+0000","last_clean":"2023-03-03T05:45:35.843345+0000","last_became_active":"2023-03-03T05:45:35.308276+0000","last_became_peered":"2023-03-03T05:45:35.308276+0000","last_unstale":"2023-03-03T05:45:35.843345+0000","last_undegraded":"2023-03-03T05:45:35.843345+0000","last_fullsized":"2023-03-03T05:45:35.843345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:32.932001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853620+0000","last_change":"2023-03-03T05:45:35.866408+0000","last_active":"2023-03-03T05:45:36.853620+0000","last_peered":"2023-03-03T05:45:36.853620+0000","last_clean":"2023-03-03T05:45:36.853620+0000","last_became_active":"2023-03-03T05:45:35.866080+0000","last_became_peered":"2023-03-03T05:45:35.866080+0000","last_unstale":"2023-03-03T05:45:36.853620+0000","last_undegraded":"2023-03-03T05:45:36.853620+0000","last_fullsized":"2023-03-03T05:45:36.853620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:22.678556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848138+0000","last_change":"2023-03-03T05:45:35.293990+0000","last_active":"2023-03-03T05:45:35.848138+0000","last_peered":"2023-03-03T05:45:35.848138+0000","last_clean":"2023-03-03T05:45:35.848138+0000","last_became_active":"2023-03-03T05:45:35.293655+0000","last_became_peered":"2023-03-03T05:45:35.293655+0000","last_unstale":"2023-03-03T05:45:35.848138+0000","last_undegraded":"2023-03-03T05:45:35.848138+0000","last_fullsized":"2023-03-03T05:45:35.848138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850700+0000","last_change":"2023-03-03T05:45:35.852599+0000","last_active":"2023-03-03T05:45:36.850700+0000","last_peered":"2023-03-03T05:45:36.850700+0000","last_clean":"2023-03-03T05:45:36.850700+0000","last_became_active":"2023-03-03T05:45:35.852292+0000","last_became_peered":"2023-03-03T05:45:35.852292+0000","last_unstale":"2023-03-03T05:45:36.850700+0000","last_undegraded":"2023-03-03T05:45:36.850700+0000","last_fullsized":"2023-03-03T05:45:36.850700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:06.104140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848641+0000","last_change":"2023-03-03T05:45:35.297718+0000","last_active":"2023-03-03T05:45:35.848641+0000","last_peered":"2023-03-03T05:45:35.848641+0000","last_clean":"2023-03-03T05:45:35.848641+0000","last_became_active":"2023-03-03T05:45:35.297435+0000","last_became_peered":"2023-03-03T05:45:35.297435+0000","last_unstale":"2023-03-03T05:45:35.848641+0000","last_undegraded":"2023-03-03T05:45:35.848641+0000","last_fullsized":"2023-03-03T05:45:35.848641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:17:39.778447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923577+0000","last_change":"2023-03-03T05:45:35.871116+0000","last_active":"2023-03-03T05:45:36.923577+0000","last_peered":"2023-03-03T05:45:36.923577+0000","last_clean":"2023-03-03T05:45:36.923577+0000","last_became_active":"2023-03-03T05:45:35.870843+0000","last_became_peered":"2023-03-03T05:45:35.870843+0000","last_unstale":"2023-03-03T05:45:36.923577+0000","last_undegraded":"2023-03-03T05:45:36.923577+0000","last_fullsized":"2023-03-03T05:45:36.923577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851170+0000","last_change":"2023-03-03T05:45:35.304400+0000","last_active":"2023-03-03T05:45:36.851170+0000","last_peered":"2023-03-03T05:45:36.851170+0000","last_clean":"2023-03-03T05:45:36.851170+0000","last_became_active":"2023-03-03T05:45:35.304093+0000","last_became_peered":"2023-03-03T05:45:35.304093+0000","last_unstale":"2023-03-03T05:45:36.851170+0000","last_undegraded":"2023-03-03T05:45:36.851170+0000","last_fullsized":"2023-03-03T05:45:36.851170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853210+0000","last_change":"2023-03-03T05:45:35.862492+0000","last_active":"2023-03-03T05:45:36.853210+0000","last_peered":"2023-03-03T05:45:36.853210+0000","last_clean":"2023-03-03T05:45:36.853210+0000","last_became_active":"2023-03-03T05:45:35.862173+0000","last_became_peered":"2023-03-03T05:45:35.862173+0000","last_unstale":"2023-03-03T05:45:36.853210+0000","last_undegraded":"2023-03-03T05:45:36.853210+0000","last_fullsized":"2023-03-03T05:45:36.853210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.853604+0000","last_change":"2023-03-03T05:45:35.302282+0000","last_active":"2023-03-03T05:45:40.853604+0000","last_peered":"2023-03-03T05:45:40.853604+0000","last_clean":"2023-03-03T05:45:40.853604+0000","last_became_active":"2023-03-03T05:45:35.302002+0000","last_became_peered":"2023-03-03T05:45:35.302002+0000","last_unstale":"2023-03-03T05:45:40.853604+0000","last_undegraded":"2023-03-03T05:45:40.853604+0000","last_fullsized":"2023-03-03T05:45:40.853604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853414+0000","last_change":"2023-03-03T05:45:35.855928+0000","last_active":"2023-03-03T05:45:36.853414+0000","last_peered":"2023-03-03T05:45:36.853414+0000","last_clean":"2023-03-03T05:45:36.853414+0000","last_became_active":"2023-03-03T05:45:35.855612+0000","last_became_peered":"2023-03-03T05:45:35.855612+0000","last_unstale":"2023-03-03T05:45:36.853414+0000","last_undegraded":"2023-03-03T05:45:36.853414+0000","last_fullsized":"2023-03-03T05:45:36.853414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.860052+0000","last_change":"2023-03-03T05:45:35.304856+0000","last_active":"2023-03-03T05:45:40.860052+0000","last_peered":"2023-03-03T05:45:40.860052+0000","last_clean":"2023-03-03T05:45:40.860052+0000","last_became_active":"2023-03-03T05:45:35.304439+0000","last_became_peered":"2023-03-03T05:45:35.304439+0000","last_unstale":"2023-03-03T05:45:40.860052+0000","last_undegraded":"2023-03-03T05:45:40.860052+0000","last_fullsized":"2023-03-03T05:45:40.860052+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:30.692527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850547+0000","last_change":"2023-03-03T05:45:35.856887+0000","last_active":"2023-03-03T05:45:36.850547+0000","last_peered":"2023-03-03T05:45:36.850547+0000","last_clean":"2023-03-03T05:45:36.850547+0000","last_became_active":"2023-03-03T05:45:35.856498+0000","last_became_peered":"2023-03-03T05:45:35.856498+0000","last_unstale":"2023-03-03T05:45:36.850547+0000","last_undegraded":"2023-03-03T05:45:36.850547+0000","last_fullsized":"2023-03-03T05:45:36.850547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858066+0000","last_change":"2023-03-03T05:45:35.297635+0000","last_active":"2023-03-03T05:45:40.858066+0000","last_peered":"2023-03-03T05:45:40.858066+0000","last_clean":"2023-03-03T05:45:40.858066+0000","last_became_active":"2023-03-03T05:45:35.297373+0000","last_became_peered":"2023-03-03T05:45:35.297373+0000","last_unstale":"2023-03-03T05:45:40.858066+0000","last_undegraded":"2023-03-03T05:45:40.858066+0000","last_fullsized":"2023-03-03T05:45:40.858066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:48.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922855+0000","last_change":"2023-03-03T05:45:35.852113+0000","last_active":"2023-03-03T05:45:36.922855+0000","last_peered":"2023-03-03T05:45:36.922855+0000","last_clean":"2023-03-03T05:45:36.922855+0000","last_became_active":"2023-03-03T05:45:35.851799+0000","last_became_peered":"2023-03-03T05:45:35.851799+0000","last_unstale":"2023-03-03T05:45:36.922855+0000","last_undegraded":"2023-03-03T05:45:36.922855+0000","last_fullsized":"2023-03-03T05:45:36.922855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.845615+0000","last_change":"2023-03-03T05:45:35.292083+0000","last_active":"2023-03-03T05:45:35.845615+0000","last_peered":"2023-03-03T05:45:35.845615+0000","last_clean":"2023-03-03T05:45:35.845615+0000","last_became_active":"2023-03-03T05:45:35.291766+0000","last_became_peered":"2023-03-03T05:45:35.291766+0000","last_unstale":"2023-03-03T05:45:35.845615+0000","last_undegraded":"2023-03-03T05:45:35.845615+0000","last_fullsized":"2023-03-03T05:45:35.845615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851348+0000","last_change":"2023-03-03T05:45:35.870363+0000","last_active":"2023-03-03T05:45:36.851348+0000","last_peered":"2023-03-03T05:45:36.851348+0000","last_clean":"2023-03-03T05:45:36.851348+0000","last_became_active":"2023-03-03T05:45:35.868801+0000","last_became_peered":"2023-03-03T05:45:35.868801+0000","last_unstale":"2023-03-03T05:45:36.851348+0000","last_undegraded":"2023-03-03T05:45:36.851348+0000","last_fullsized":"2023-03-03T05:45:36.851348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:04.045341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844327+0000","last_change":"2023-03-03T05:45:35.304592+0000","last_active":"2023-03-03T05:45:35.844327+0000","last_peered":"2023-03-03T05:45:35.844327+0000","last_clean":"2023-03-03T05:45:35.844327+0000","last_became_active":"2023-03-03T05:45:35.304318+0000","last_became_peered":"2023-03-03T05:45:35.304318+0000","last_unstale":"2023-03-03T05:45:35.844327+0000","last_undegraded":"2023-03-03T05:45:35.844327+0000","last_fullsized":"2023-03-03T05:45:35.844327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:08.368039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922933+0000","last_change":"2023-03-03T05:45:35.857951+0000","last_active":"2023-03-03T05:45:36.922933+0000","last_peered":"2023-03-03T05:45:36.922933+0000","last_clean":"2023-03-03T05:45:36.922933+0000","last_became_active":"2023-03-03T05:45:35.857581+0000","last_became_peered":"2023-03-03T05:45:35.857581+0000","last_unstale":"2023-03-03T05:45:36.922933+0000","last_undegraded":"2023-03-03T05:45:36.922933+0000","last_fullsized":"2023-03-03T05:45:36.922933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858611+0000","last_change":"2023-03-03T05:45:35.309150+0000","last_active":"2023-03-03T05:45:40.858611+0000","last_peered":"2023-03-03T05:45:40.858611+0000","last_clean":"2023-03-03T05:45:40.858611+0000","last_became_active":"2023-03-03T05:45:35.308840+0000","last_became_peered":"2023-03-03T05:45:35.308840+0000","last_unstale":"2023-03-03T05:45:40.858611+0000","last_undegraded":"2023-03-03T05:45:40.858611+0000","last_fullsized":"2023-03-03T05:45:40.858611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923050+0000","last_change":"2023-03-03T05:45:35.864919+0000","last_active":"2023-03-03T05:45:36.923050+0000","last_peered":"2023-03-03T05:45:36.923050+0000","last_clean":"2023-03-03T05:45:36.923050+0000","last_became_active":"2023-03-03T05:45:35.864595+0000","last_became_peered":"2023-03-03T05:45:35.864595+0000","last_unstale":"2023-03-03T05:45:36.923050+0000","last_undegraded":"2023-03-03T05:45:36.923050+0000","last_fullsized":"2023-03-03T05:45:36.923050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.308976+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.308707+0000","last_became_peered":"2023-03-03T05:45:35.308707+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850640+0000","last_change":"2023-03-03T05:45:35.868861+0000","last_active":"2023-03-03T05:45:36.850640+0000","last_peered":"2023-03-03T05:45:36.850640+0000","last_clean":"2023-03-03T05:45:36.850640+0000","last_became_active":"2023-03-03T05:45:35.868438+0000","last_became_peered":"2023-03-03T05:45:35.868438+0000","last_unstale":"2023-03-03T05:45:36.850640+0000","last_undegraded":"2023-03-03T05:45:36.850640+0000","last_fullsized":"2023-03-03T05:45:36.850640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:59.810575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.854269+0000","last_change":"2023-03-03T05:45:35.303791+0000","last_active":"2023-03-03T05:45:40.854269+0000","last_peered":"2023-03-03T05:45:40.854269+0000","last_clean":"2023-03-03T05:45:40.854269+0000","last_became_active":"2023-03-03T05:45:35.303482+0000","last_became_peered":"2023-03-03T05:45:35.303482+0000","last_unstale":"2023-03-03T05:45:40.854269+0000","last_undegraded":"2023-03-03T05:45:40.854269+0000","last_fullsized":"2023-03-03T05:45:40.854269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:31.933113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923037+0000","last_change":"2023-03-03T05:45:35.864783+0000","last_active":"2023-03-03T05:45:36.923037+0000","last_peered":"2023-03-03T05:45:36.923037+0000","last_clean":"2023-03-03T05:45:36.923037+0000","last_became_active":"2023-03-03T05:45:35.864462+0000","last_became_peered":"2023-03-03T05:45:35.864462+0000","last_unstale":"2023-03-03T05:45:36.923037+0000","last_undegraded":"2023-03-03T05:45:36.923037+0000","last_fullsized":"2023-03-03T05:45:36.923037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851096+0000","last_change":"2023-03-03T05:45:35.310813+0000","last_active":"2023-03-03T05:45:36.851096+0000","last_peered":"2023-03-03T05:45:36.851096+0000","last_clean":"2023-03-03T05:45:36.851096+0000","last_became_active":"2023-03-03T05:45:35.310466+0000","last_became_peered":"2023-03-03T05:45:35.310466+0000","last_unstale":"2023-03-03T05:45:36.851096+0000","last_undegraded":"2023-03-03T05:45:36.851096+0000","last_fullsized":"2023-03-03T05:45:36.851096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:46.322139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850472+0000","last_change":"2023-03-03T05:45:35.856376+0000","last_active":"2023-03-03T05:45:36.850472+0000","last_peered":"2023-03-03T05:45:36.850472+0000","last_clean":"2023-03-03T05:45:36.850472+0000","last_became_active":"2023-03-03T05:45:35.856064+0000","last_became_peered":"2023-03-03T05:45:35.856064+0000","last_unstale":"2023-03-03T05:45:36.850472+0000","last_undegraded":"2023-03-03T05:45:36.850472+0000","last_fullsized":"2023-03-03T05:45:36.850472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962714+0000","last_change":"2023-03-03T05:45:35.295520+0000","last_active":"2023-03-03T05:45:35.962714+0000","last_peered":"2023-03-03T05:45:35.962714+0000","last_clean":"2023-03-03T05:45:35.962714+0000","last_became_active":"2023-03-03T05:45:35.295203+0000","last_became_peered":"2023-03-03T05:45:35.295203+0000","last_unstale":"2023-03-03T05:45:35.962714+0000","last_undegraded":"2023-03-03T05:45:35.962714+0000","last_fullsized":"2023-03-03T05:45:35.962714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923389+0000","last_change":"2023-03-03T05:45:35.852242+0000","last_active":"2023-03-03T05:45:36.923389+0000","last_peered":"2023-03-03T05:45:36.923389+0000","last_clean":"2023-03-03T05:45:36.923389+0000","last_became_active":"2023-03-03T05:45:35.851803+0000","last_became_peered":"2023-03-03T05:45:35.851803+0000","last_unstale":"2023-03-03T05:45:36.923389+0000","last_undegraded":"2023-03-03T05:45:36.923389+0000","last_fullsized":"2023-03-03T05:45:36.923389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:14.473334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.852939+0000","last_change":"2023-03-03T05:45:35.298542+0000","last_active":"2023-03-03T05:45:40.852939+0000","last_peered":"2023-03-03T05:45:40.852939+0000","last_clean":"2023-03-03T05:45:40.852939+0000","last_became_active":"2023-03-03T05:45:35.298219+0000","last_became_peered":"2023-03-03T05:45:35.298219+0000","last_unstale":"2023-03-03T05:45:40.852939+0000","last_undegraded":"2023-03-03T05:45:40.852939+0000","last_fullsized":"2023-03-03T05:45:40.852939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962699+0000","last_change":"2023-03-03T05:45:35.861231+0000","last_active":"2023-03-03T05:45:35.962699+0000","last_peered":"2023-03-03T05:45:35.962699+0000","last_clean":"2023-03-03T05:45:35.962699+0000","last_became_active":"2023-03-03T05:45:35.860675+0000","last_became_peered":"2023-03-03T05:45:35.860675+0000","last_unstale":"2023-03-03T05:45:35.962699+0000","last_undegraded":"2023-03-03T05:45:35.962699+0000","last_fullsized":"2023-03-03T05:45:35.962699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843762+0000","last_change":"2023-03-03T05:45:35.312143+0000","last_active":"2023-03-03T05:45:35.843762+0000","last_peered":"2023-03-03T05:45:35.843762+0000","last_clean":"2023-03-03T05:45:35.843762+0000","last_became_active":"2023-03-03T05:45:35.311848+0000","last_became_peered":"2023-03-03T05:45:35.311848+0000","last_unstale":"2023-03-03T05:45:35.843762+0000","last_undegraded":"2023-03-03T05:45:35.843762+0000","last_fullsized":"2023-03-03T05:45:35.843762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:22.294783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851303+0000","last_change":"2023-03-03T05:45:35.869075+0000","last_active":"2023-03-03T05:45:36.851303+0000","last_peered":"2023-03-03T05:45:36.851303+0000","last_clean":"2023-03-03T05:45:36.851303+0000","last_became_active":"2023-03-03T05:45:35.868413+0000","last_became_peered":"2023-03-03T05:45:35.868413+0000","last_unstale":"2023-03-03T05:45:36.851303+0000","last_undegraded":"2023-03-03T05:45:36.851303+0000","last_fullsized":"2023-03-03T05:45:36.851303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:34.743111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844226+0000","last_change":"2023-03-03T05:45:35.296611+0000","last_active":"2023-03-03T05:45:35.844226+0000","last_peered":"2023-03-03T05:45:35.844226+0000","last_clean":"2023-03-03T05:45:35.844226+0000","last_became_active":"2023-03-03T05:45:35.296294+0000","last_became_peered":"2023-03-03T05:45:35.296294+0000","last_unstale":"2023-03-03T05:45:35.844226+0000","last_undegraded":"2023-03-03T05:45:35.844226+0000","last_fullsized":"2023-03-03T05:45:35.844226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:16.655836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850964+0000","last_change":"2023-03-03T05:45:35.857451+0000","last_active":"2023-03-03T05:45:36.850964+0000","last_peered":"2023-03-03T05:45:36.850964+0000","last_clean":"2023-03-03T05:45:36.850964+0000","last_became_active":"2023-03-03T05:45:35.857087+0000","last_became_peered":"2023-03-03T05:45:35.857087+0000","last_unstale":"2023-03-03T05:45:36.850964+0000","last_undegraded":"2023-03-03T05:45:36.850964+0000","last_fullsized":"2023-03-03T05:45:36.850964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842797+0000","last_change":"2023-03-03T05:45:35.296114+0000","last_active":"2023-03-03T05:45:35.842797+0000","last_peered":"2023-03-03T05:45:35.842797+0000","last_clean":"2023-03-03T05:45:35.842797+0000","last_became_active":"2023-03-03T05:45:35.295855+0000","last_became_peered":"2023-03-03T05:45:35.295855+0000","last_unstale":"2023-03-03T05:45:35.842797+0000","last_undegraded":"2023-03-03T05:45:35.842797+0000","last_fullsized":"2023-03-03T05:45:35.842797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923246+0000","last_change":"2023-03-03T05:45:35.862249+0000","last_active":"2023-03-03T05:45:36.923246+0000","last_peered":"2023-03-03T05:45:36.923246+0000","last_clean":"2023-03-03T05:45:36.923246+0000","last_became_active":"2023-03-03T05:45:35.861773+0000","last_became_peered":"2023-03-03T05:45:35.861773+0000","last_unstale":"2023-03-03T05:45:36.923246+0000","last_undegraded":"2023-03-03T05:45:36.923246+0000","last_fullsized":"2023-03-03T05:45:36.923246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848485+0000","last_change":"2023-03-03T05:45:29.189033+0000","last_active":"2023-03-03T05:45:35.848485+0000","last_peered":"2023-03-03T05:45:35.848485+0000","last_clean":"2023-03-03T05:45:35.848485+0000","last_became_active":"2023-03-03T05:45:29.188719+0000","last_became_peered":"2023-03-03T05:45:29.188719+0000","last_unstale":"2023-03-03T05:45:35.848485+0000","last_undegraded":"2023-03-03T05:45:35.848485+0000","last_fullsized":"2023-03-03T05:45:35.848485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850356+0000","last_change":"2023-03-03T05:45:35.303756+0000","last_active":"2023-03-03T05:45:36.850356+0000","last_peered":"2023-03-03T05:45:36.850356+0000","last_clean":"2023-03-03T05:45:36.850356+0000","last_became_active":"2023-03-03T05:45:35.303448+0000","last_became_peered":"2023-03-03T05:45:35.303448+0000","last_unstale":"2023-03-03T05:45:36.850356+0000","last_undegraded":"2023-03-03T05:45:36.850356+0000","last_fullsized":"2023-03-03T05:45:36.850356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922899+0000","last_change":"2023-03-03T05:45:35.854166+0000","last_active":"2023-03-03T05:45:36.922899+0000","last_peered":"2023-03-03T05:45:36.922899+0000","last_clean":"2023-03-03T05:45:36.922899+0000","last_became_active":"2023-03-03T05:45:35.853873+0000","last_became_peered":"2023-03-03T05:45:35.853873+0000","last_unstale":"2023-03-03T05:45:36.922899+0000","last_undegraded":"2023-03-03T05:45:36.922899+0000","last_fullsized":"2023-03-03T05:45:36.922899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843699+0000","last_change":"2023-03-03T05:45:29.189075+0000","last_active":"2023-03-03T05:45:35.843699+0000","last_peered":"2023-03-03T05:45:35.843699+0000","last_clean":"2023-03-03T05:45:35.843699+0000","last_became_active":"2023-03-03T05:45:29.188778+0000","last_became_peered":"2023-03-03T05:45:29.188778+0000","last_unstale":"2023-03-03T05:45:35.843699+0000","last_undegraded":"2023-03-03T05:45:35.843699+0000","last_fullsized":"2023-03-03T05:45:35.843699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:54:35.536172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962645+0000","last_change":"2023-03-03T05:45:35.309781+0000","last_active":"2023-03-03T05:45:35.962645+0000","last_peered":"2023-03-03T05:45:35.962645+0000","last_clean":"2023-03-03T05:45:35.962645+0000","last_became_active":"2023-03-03T05:45:35.309497+0000","last_became_peered":"2023-03-03T05:45:35.309497+0000","last_unstale":"2023-03-03T05:45:35.962645+0000","last_undegraded":"2023-03-03T05:45:35.962645+0000","last_fullsized":"2023-03-03T05:45:35.962645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923571+0000","last_change":"2023-03-03T05:45:35.854721+0000","last_active":"2023-03-03T05:45:36.923571+0000","last_peered":"2023-03-03T05:45:36.923571+0000","last_clean":"2023-03-03T05:45:36.923571+0000","last_became_active":"2023-03-03T05:45:35.854423+0000","last_became_peered":"2023-03-03T05:45:35.854423+0000","last_unstale":"2023-03-03T05:45:36.923571+0000","last_undegraded":"2023-03-03T05:45:36.923571+0000","last_fullsized":"2023-03-03T05:45:36.923571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.852125+0000","last_change":"2023-03-03T05:45:29.057308+0000","last_active":"2023-03-03T05:45:35.852125+0000","last_peered":"2023-03-03T05:45:35.852125+0000","last_clean":"2023-03-03T05:45:35.852125+0000","last_became_active":"2023-03-03T05:45:29.056828+0000","last_became_peered":"2023-03-03T05:45:29.056828+0000","last_unstale":"2023-03-03T05:45:35.852125+0000","last_undegraded":"2023-03-03T05:45:35.852125+0000","last_fullsized":"2023-03-03T05:45:35.852125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:56.079893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848095+0000","last_change":"2023-03-03T05:45:35.295842+0000","last_active":"2023-03-03T05:45:35.848095+0000","last_peered":"2023-03-03T05:45:35.848095+0000","last_clean":"2023-03-03T05:45:35.848095+0000","last_became_active":"2023-03-03T05:45:35.295545+0000","last_became_peered":"2023-03-03T05:45:35.295545+0000","last_unstale":"2023-03-03T05:45:35.848095+0000","last_undegraded":"2023-03-03T05:45:35.848095+0000","last_fullsized":"2023-03-03T05:45:35.848095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923636+0000","last_change":"2023-03-03T05:45:35.862437+0000","last_active":"2023-03-03T05:45:36.923636+0000","last_peered":"2023-03-03T05:45:36.923636+0000","last_clean":"2023-03-03T05:45:36.923636+0000","last_became_active":"2023-03-03T05:45:35.862152+0000","last_became_peered":"2023-03-03T05:45:35.862152+0000","last_unstale":"2023-03-03T05:45:36.923636+0000","last_undegraded":"2023-03-03T05:45:36.923636+0000","last_fullsized":"2023-03-03T05:45:36.923636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842862+0000","last_change":"2023-03-03T05:45:29.057646+0000","last_active":"2023-03-03T05:45:35.842862+0000","last_peered":"2023-03-03T05:45:35.842862+0000","last_clean":"2023-03-03T05:45:35.842862+0000","last_became_active":"2023-03-03T05:45:29.057330+0000","last_became_peered":"2023-03-03T05:45:29.057330+0000","last_unstale":"2023-03-03T05:45:35.842862+0000","last_undegraded":"2023-03-03T05:45:35.842862+0000","last_fullsized":"2023-03-03T05:45:35.842862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:22:57.039574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.858102+0000","last_change":"2023-03-03T05:45:22.224435+0000","last_active":"2023-03-03T05:45:35.858102+0000","last_peered":"2023-03-03T05:45:35.858102+0000","last_clean":"2023-03-03T05:45:35.858102+0000","last_became_active":"2023-03-03T05:45:22.222958+0000","last_became_peered":"2023-03-03T05:45:22.222958+0000","last_unstale":"2023-03-03T05:45:35.858102+0000","last_undegraded":"2023-03-03T05:45:35.858102+0000","last_fullsized":"2023-03-03T05:45:35.858102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_clean_scrub_stamp":"2023-03-03T05:45:20.925942+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:10:20.426026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843695+0000","last_change":"2023-03-03T05:45:35.304483+0000","last_active":"2023-03-03T05:45:35.843695+0000","last_peered":"2023-03-03T05:45:35.843695+0000","last_clean":"2023-03-03T05:45:35.843695+0000","last_became_active":"2023-03-03T05:45:35.304202+0000","last_became_peered":"2023-03-03T05:45:35.304202+0000","last_unstale":"2023-03-03T05:45:35.843695+0000","last_undegraded":"2023-03-03T05:45:35.843695+0000","last_fullsized":"2023-03-03T05:45:35.843695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851295+0000","last_change":"2023-03-03T05:45:35.869049+0000","last_active":"2023-03-03T05:45:36.851295+0000","last_peered":"2023-03-03T05:45:36.851295+0000","last_clean":"2023-03-03T05:45:36.851295+0000","last_became_active":"2023-03-03T05:45:35.868376+0000","last_became_peered":"2023-03-03T05:45:35.868376+0000","last_unstale":"2023-03-03T05:45:36.851295+0000","last_undegraded":"2023-03-03T05:45:36.851295+0000","last_fullsized":"2023-03-03T05:45:36.851295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:42:55.207846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843590+0000","last_change":"2023-03-03T05:45:29.056874+0000","last_active":"2023-03-03T05:45:35.843590+0000","last_peered":"2023-03-03T05:45:35.843590+0000","last_clean":"2023-03-03T05:45:35.843590+0000","last_became_active":"2023-03-03T05:45:29.056438+0000","last_became_peered":"2023-03-03T05:45:29.056438+0000","last_unstale":"2023-03-03T05:45:35.843590+0000","last_undegraded":"2023-03-03T05:45:35.843590+0000","last_fullsized":"2023-03-03T05:45:35.843590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:08.113953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.846141+0000","last_change":"2023-03-03T05:45:35.304562+0000","last_active":"2023-03-03T05:45:35.846141+0000","last_peered":"2023-03-03T05:45:35.846141+0000","last_clean":"2023-03-03T05:45:35.846141+0000","last_became_active":"2023-03-03T05:45:35.304264+0000","last_became_peered":"2023-03-03T05:45:35.304264+0000","last_unstale":"2023-03-03T05:45:35.846141+0000","last_undegraded":"2023-03-03T05:45:35.846141+0000","last_fullsized":"2023-03-03T05:45:35.846141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923094+0000","last_change":"2023-03-03T05:45:35.859779+0000","last_active":"2023-03-03T05:45:36.923094+0000","last_peered":"2023-03-03T05:45:36.923094+0000","last_clean":"2023-03-03T05:45:36.923094+0000","last_became_active":"2023-03-03T05:45:35.857596+0000","last_became_peered":"2023-03-03T05:45:35.857596+0000","last_unstale":"2023-03-03T05:45:36.923094+0000","last_undegraded":"2023-03-03T05:45:36.923094+0000","last_fullsized":"2023-03-03T05:45:36.923094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844418+0000","last_change":"2023-03-03T05:45:29.056891+0000","last_active":"2023-03-03T05:45:35.844418+0000","last_peered":"2023-03-03T05:45:35.844418+0000","last_clean":"2023-03-03T05:45:35.844418+0000","last_became_active":"2023-03-03T05:45:29.056483+0000","last_became_peered":"2023-03-03T05:45:29.056483+0000","last_unstale":"2023-03-03T05:45:35.844418+0000","last_undegraded":"2023-03-03T05:45:35.844418+0000","last_fullsized":"2023-03-03T05:45:35.844418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:25.701585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.847358+0000","last_change":"2023-03-03T05:45:35.293821+0000","last_active":"2023-03-03T05:45:35.847358+0000","last_peered":"2023-03-03T05:45:35.847358+0000","last_clean":"2023-03-03T05:45:35.847358+0000","last_became_active":"2023-03-03T05:45:35.293533+0000","last_became_peered":"2023-03-03T05:45:35.293533+0000","last_unstale":"2023-03-03T05:45:35.847358+0000","last_undegraded":"2023-03-03T05:45:35.847358+0000","last_fullsized":"2023-03-03T05:45:35.847358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923917+0000","last_change":"2023-03-03T05:45:35.866830+0000","last_active":"2023-03-03T05:45:36.923917+0000","last_peered":"2023-03-03T05:45:36.923917+0000","last_clean":"2023-03-03T05:45:36.923917+0000","last_became_active":"2023-03-03T05:45:35.866509+0000","last_became_peered":"2023-03-03T05:45:35.866509+0000","last_unstale":"2023-03-03T05:45:36.923917+0000","last_undegraded":"2023-03-03T05:45:36.923917+0000","last_fullsized":"2023-03-03T05:45:36.923917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.850545+0000","last_change":"2023-03-03T05:45:35.300794+0000","last_active":"2023-03-03T05:45:40.850545+0000","last_peered":"2023-03-03T05:45:40.850545+0000","last_clean":"2023-03-03T05:45:40.850545+0000","last_became_active":"2023-03-03T05:45:35.300531+0000","last_became_peered":"2023-03-03T05:45:35.300531+0000","last_unstale":"2023-03-03T05:45:40.850545+0000","last_undegraded":"2023-03-03T05:45:40.850545+0000","last_fullsized":"2023-03-03T05:45:40.850545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843343+0000","last_change":"2023-03-03T05:45:29.193601+0000","last_active":"2023-03-03T05:45:35.843343+0000","last_peered":"2023-03-03T05:45:35.843343+0000","last_clean":"2023-03-03T05:45:35.843343+0000","last_became_active":"2023-03-03T05:45:29.193262+0000","last_became_peered":"2023-03-03T05:45:29.193262+0000","last_unstale":"2023-03-03T05:45:35.843343+0000","last_undegraded":"2023-03-03T05:45:35.843343+0000","last_fullsized":"2023-03-03T05:45:35.843343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853228+0000","last_change":"2023-03-03T05:45:35.864517+0000","last_active":"2023-03-03T05:45:36.853228+0000","last_peered":"2023-03-03T05:45:36.853228+0000","last_clean":"2023-03-03T05:45:36.853228+0000","last_became_active":"2023-03-03T05:45:35.864193+0000","last_became_peered":"2023-03-03T05:45:35.864193+0000","last_unstale":"2023-03-03T05:45:36.853228+0000","last_undegraded":"2023-03-03T05:45:36.853228+0000","last_fullsized":"2023-03-03T05:45:36.853228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:11:08.433881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844985+0000","last_change":"2023-03-03T05:45:29.193546+0000","last_active":"2023-03-03T05:45:35.844985+0000","last_peered":"2023-03-03T05:45:35.844985+0000","last_clean":"2023-03-03T05:45:35.844985+0000","last_became_active":"2023-03-03T05:45:29.193172+0000","last_became_peered":"2023-03-03T05:45:29.193172+0000","last_unstale":"2023-03-03T05:45:35.844985+0000","last_undegraded":"2023-03-03T05:45:35.844985+0000","last_fullsized":"2023-03-03T05:45:35.844985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:07:42.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843485+0000","last_change":"2023-03-03T05:45:35.296693+0000","last_active":"2023-03-03T05:45:35.843485+0000","last_peered":"2023-03-03T05:45:35.843485+0000","last_clean":"2023-03-03T05:45:35.843485+0000","last_became_active":"2023-03-03T05:45:35.296469+0000","last_became_peered":"2023-03-03T05:45:35.296469+0000","last_unstale":"2023-03-03T05:45:35.843485+0000","last_undegraded":"2023-03-03T05:45:35.843485+0000","last_fullsized":"2023-03-03T05:45:35.843485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850872+0000","last_change":"2023-03-03T05:45:35.856468+0000","last_active":"2023-03-03T05:45:36.850872+0000","last_peered":"2023-03-03T05:45:36.850872+0000","last_clean":"2023-03-03T05:45:36.850872+0000","last_became_active":"2023-03-03T05:45:35.855970+0000","last_became_peered":"2023-03-03T05:45:35.855970+0000","last_unstale":"2023-03-03T05:45:36.850872+0000","last_undegraded":"2023-03-03T05:45:36.850872+0000","last_fullsized":"2023-03-03T05:45:36.850872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853299+0000","last_change":"2023-03-03T05:45:35.301855+0000","last_active":"2023-03-03T05:45:36.853299+0000","last_peered":"2023-03-03T05:45:36.853299+0000","last_clean":"2023-03-03T05:45:36.853299+0000","last_became_active":"2023-03-03T05:45:35.301622+0000","last_became_peered":"2023-03-03T05:45:35.301622+0000","last_unstale":"2023-03-03T05:45:36.853299+0000","last_undegraded":"2023-03-03T05:45:36.853299+0000","last_fullsized":"2023-03-03T05:45:36.853299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:42.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853538+0000","last_change":"2023-03-03T05:45:35.851780+0000","last_active":"2023-03-03T05:45:36.853538+0000","last_peered":"2023-03-03T05:45:36.853538+0000","last_clean":"2023-03-03T05:45:36.853538+0000","last_became_active":"2023-03-03T05:45:35.850190+0000","last_became_peered":"2023-03-03T05:45:35.850190+0000","last_unstale":"2023-03-03T05:45:36.853538+0000","last_undegraded":"2023-03-03T05:45:36.853538+0000","last_fullsized":"2023-03-03T05:45:36.853538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856909+0000","last_change":"2023-03-03T05:45:35.309047+0000","last_active":"2023-03-03T05:45:40.856909+0000","last_peered":"2023-03-03T05:45:40.856909+0000","last_clean":"2023-03-03T05:45:40.856909+0000","last_became_active":"2023-03-03T05:45:35.308564+0000","last_became_peered":"2023-03-03T05:45:35.308564+0000","last_unstale":"2023-03-03T05:45:40.856909+0000","last_undegraded":"2023-03-03T05:45:40.856909+0000","last_fullsized":"2023-03-03T05:45:40.856909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850462+0000","last_change":"2023-03-03T05:45:35.861316+0000","last_active":"2023-03-03T05:45:36.850462+0000","last_peered":"2023-03-03T05:45:36.850462+0000","last_clean":"2023-03-03T05:45:36.850462+0000","last_became_active":"2023-03-03T05:45:35.861052+0000","last_became_peered":"2023-03-03T05:45:35.861052+0000","last_unstale":"2023-03-03T05:45:36.850462+0000","last_undegraded":"2023-03-03T05:45:36.850462+0000","last_fullsized":"2023-03-03T05:45:36.850462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858612+0000","last_change":"2023-03-03T05:45:35.303666+0000","last_active":"2023-03-03T05:45:40.858612+0000","last_peered":"2023-03-03T05:45:40.858612+0000","last_clean":"2023-03-03T05:45:40.858612+0000","last_became_active":"2023-03-03T05:45:35.303213+0000","last_became_peered":"2023-03-03T05:45:35.303213+0000","last_unstale":"2023-03-03T05:45:40.858612+0000","last_undegraded":"2023-03-03T05:45:40.858612+0000","last_fullsized":"2023-03-03T05:45:40.858612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:54.389717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850392+0000","last_change":"2023-03-03T05:45:35.857442+0000","last_active":"2023-03-03T05:45:36.850392+0000","last_peered":"2023-03-03T05:45:36.850392+0000","last_clean":"2023-03-03T05:45:36.850392+0000","last_became_active":"2023-03-03T05:45:35.857130+0000","last_became_peered":"2023-03-03T05:45:35.857130+0000","last_unstale":"2023-03-03T05:45:36.850392+0000","last_undegraded":"2023-03-03T05:45:36.850392+0000","last_fullsized":"2023-03-03T05:45:36.850392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923020+0000","last_change":"2023-03-03T05:45:35.305098+0000","last_active":"2023-03-03T05:45:36.923020+0000","last_peered":"2023-03-03T05:45:36.923020+0000","last_clean":"2023-03-03T05:45:36.923020+0000","last_became_active":"2023-03-03T05:45:35.304812+0000","last_became_peered":"2023-03-03T05:45:35.304812+0000","last_unstale":"2023-03-03T05:45:36.923020+0000","last_undegraded":"2023-03-03T05:45:36.923020+0000","last_fullsized":"2023-03-03T05:45:36.923020+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923341+0000","last_change":"2023-03-03T05:45:35.858806+0000","last_active":"2023-03-03T05:45:36.923341+0000","last_peered":"2023-03-03T05:45:36.923341+0000","last_clean":"2023-03-03T05:45:36.923341+0000","last_became_active":"2023-03-03T05:45:35.858471+0000","last_became_peered":"2023-03-03T05:45:35.858471+0000","last_unstale":"2023-03-03T05:45:36.923341+0000","last_undegraded":"2023-03-03T05:45:36.923341+0000","last_fullsized":"2023-03-03T05:45:36.923341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856244+0000","last_change":"2023-03-03T05:45:35.310654+0000","last_active":"2023-03-03T05:45:40.856244+0000","last_peered":"2023-03-03T05:45:40.856244+0000","last_clean":"2023-03-03T05:45:40.856244+0000","last_became_active":"2023-03-03T05:45:35.310377+0000","last_became_peered":"2023-03-03T05:45:35.310377+0000","last_unstale":"2023-03-03T05:45:40.856244+0000","last_undegraded":"2023-03-03T05:45:40.856244+0000","last_fullsized":"2023-03-03T05:45:40.856244+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850763+0000","last_change":"2023-03-03T05:45:35.850670+0000","last_active":"2023-03-03T05:45:36.850763+0000","last_peered":"2023-03-03T05:45:36.850763+0000","last_clean":"2023-03-03T05:45:36.850763+0000","last_became_active":"2023-03-03T05:45:35.850380+0000","last_became_peered":"2023-03-03T05:45:35.850380+0000","last_unstale":"2023-03-03T05:45:36.850763+0000","last_undegraded":"2023-03-03T05:45:36.850763+0000","last_fullsized":"2023-03-03T05:45:36.850763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923387+0000","last_change":"2023-03-03T05:45:35.863660+0000","last_active":"2023-03-03T05:45:36.923387+0000","last_peered":"2023-03-03T05:45:36.923387+0000","last_clean":"2023-03-03T05:45:36.923387+0000","last_became_active":"2023-03-03T05:45:35.863234+0000","last_became_peered":"2023-03-03T05:45:35.863234+0000","last_unstale":"2023-03-03T05:45:36.923387+0000","last_undegraded":"2023-03-03T05:45:36.923387+0000","last_fullsized":"2023-03-03T05:45:36.923387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922937+0000","last_change":"2023-03-03T05:45:35.851019+0000","last_active":"2023-03-03T05:45:36.922937+0000","last_peered":"2023-03-03T05:45:36.922937+0000","last_clean":"2023-03-03T05:45:36.922937+0000","last_became_active":"2023-03-03T05:45:35.850439+0000","last_became_peered":"2023-03-03T05:45:35.850439+0000","last_unstale":"2023-03-03T05:45:36.922937+0000","last_undegraded":"2023-03-03T05:45:36.922937+0000","last_fullsized":"2023-03-03T05:45:36.922937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:43.693100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962628+0000","last_change":"2023-03-03T05:45:35.860508+0000","last_active":"2023-03-03T05:45:35.962628+0000","last_peered":"2023-03-03T05:45:35.962628+0000","last_clean":"2023-03-03T05:45:35.962628+0000","last_became_active":"2023-03-03T05:45:35.860137+0000","last_became_peered":"2023-03-03T05:45:35.860137+0000","last_unstale":"2023-03-03T05:45:35.962628+0000","last_undegraded":"2023-03-03T05:45:35.962628+0000","last_fullsized":"2023-03-03T05:45:35.962628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923487+0000","last_change":"2023-03-03T05:45:35.864824+0000","last_active":"2023-03-03T05:45:36.923487+0000","last_peered":"2023-03-03T05:45:36.923487+0000","last_clean":"2023-03-03T05:45:36.923487+0000","last_became_active":"2023-03-03T05:45:35.864466+0000","last_became_peered":"2023-03-03T05:45:35.864466+0000","last_unstale":"2023-03-03T05:45:36.923487+0000","last_undegraded":"2023-03-03T05:45:36.923487+0000","last_fullsized":"2023-03-03T05:45:36.923487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922781+0000","last_change":"2023-03-03T05:45:35.859499+0000","last_active":"2023-03-03T05:45:36.922781+0000","last_peered":"2023-03-03T05:45:36.922781+0000","last_clean":"2023-03-03T05:45:36.922781+0000","last_became_active":"2023-03-03T05:45:35.859200+0000","last_became_peered":"2023-03-03T05:45:35.859200+0000","last_unstale":"2023-03-03T05:45:36.922781+0000","last_undegraded":"2023-03-03T05:45:36.922781+0000","last_fullsized":"2023-03-03T05:45:36.922781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":90307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.202}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0950000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":88905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.093}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0620000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6210000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4189999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7110000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4589999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8068,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363772,"statfs":{"total":96636764160,"available":96628502528,"internally_reserved":0,"allocated":258048,"data_stored":90411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.524}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.3719999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1349999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2539999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9088,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362752,"statfs":{"total":96636764160,"available":96627458048,"internally_reserved":0,"allocated":843776,"data_stored":679785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9140,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362700,"statfs":{"total":96636764160,"available":96627404800,"internally_reserved":0,"allocated":831488,"data_stored":678400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7988,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363852,"statfs":{"total":96636764160,"available":96628584448,"internally_reserved":0,"allocated":241664,"data_stored":88849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T05:45:48.713 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-03-03T05:45:48.713 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-03-03T05:45:48.713 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-03-03T05:45:48.714 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-03-03T05:45:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.985+0000 7fa1723ff700 1 Processor -- start 2023-03-03T05:45:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.986+0000 7fa1723ff700 1 -- start start 2023-03-03T05:45:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa1723ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa16c1597f0 0x7fa16c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa1723ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c0cf9a0 0x7fa16c163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa1723ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c0cb9f0 con 0x7fa16c0ced80 2023-03-03T05:45:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa1723ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16c0ca750 con 0x7fa16c1597f0 2023-03-03T05:45:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa1723ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c15a230 con 0x7fa16c0cf9a0 2023-03-03T05:45:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa163fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa16c1597f0 0x7fa16c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa17099c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c0cf9a0 0x7fa16c163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa16bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa16c0ced80 0x7fa16c0cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42646/0 (socket says 172.21.15.102:42646) 2023-03-03T05:45:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa17099c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c0cf9a0 0x7fa16c163660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57982/0 (socket says 172.21.15.102:57982) 2023-03-03T05:45:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa163fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa16c1597f0 0x7fa16c0cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50472/0 (socket says 172.21.15.102:50472) 2023-03-03T05:45:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.987+0000 7fa17099c700 1 -- 172.21.15.102:0/3077459747 learned_addr learned my addr 172.21.15.102:0/3077459747 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.988+0000 7fa17099c700 1 -- 172.21.15.102:0/3077459747 >> v2:172.21.15.172:3300/0 conn(0x7fa16c1597f0 msgr2=0x7fa16c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.988+0000 7fa17099c700 1 --2- 172.21.15.102:0/3077459747 >> v2:172.21.15.172:3300/0 conn(0x7fa16c1597f0 0x7fa16c0cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.988+0000 7fa17099c700 1 -- 172.21.15.102:0/3077459747 >> v1:172.21.15.102:6789/0 conn(0x7fa16c0ced80 legacy=0x7fa16c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.988+0000 7fa17099c700 1 -- 172.21.15.102:0/3077459747 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c159c80 con 0x7fa16c0cf9a0 2023-03-03T05:45:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.988+0000 7fa163fff700 1 --2- 172.21.15.102:0/3077459747 >> v2:172.21.15.172:3300/0 conn(0x7fa16c1597f0 0x7fa16c0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.988+0000 7fa17099c700 1 --2- 172.21.15.102:0/3077459747 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c0cf9a0 0x7fa16c163660 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa164009e50 tx=0x7fa16400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f719f459b58e7f9e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.988+0000 7fa16b7fe700 1 -- 172.21.15.102:0/3077459747 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16400e070 con 0x7fa16c0cf9a0 2023-03-03T05:45:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa16b7fe700 1 -- 172.21.15.102:0/3077459747 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16400b910 con 0x7fa16c0cf9a0 2023-03-03T05:45:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa16b7fe700 1 -- 172.21.15.102:0/3077459747 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16400bcc0 con 0x7fa16c0cf9a0 2023-03-03T05:45:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 -- 172.21.15.102:0/3077459747 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c0cf9a0 msgr2=0x7fa16c163660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3077459747 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c0cf9a0 0x7fa16c163660 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa164009e50 tx=0x7fa16400b330 comp rx=0 tx=0).stop 2023-03-03T05:45:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 -- 172.21.15.102:0/3077459747 shutdown_connections 2023-03-03T05:45:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3077459747 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c0cf9a0 0x7fa16c163660 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3077459747 >> v2:172.21.15.172:3300/0 conn(0x7fa16c1597f0 0x7fa16c0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 -- 172.21.15.102:0/3077459747 >> 172.21.15.102:0/3077459747 conn(0x7fa16c065fb0 msgr2=0x7fa16c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 -- 172.21.15.102:0/3077459747 shutdown_connections 2023-03-03T05:45:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.989+0000 7fa1723ff700 1 -- 172.21.15.102:0/3077459747 wait complete. 2023-03-03T05:45:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.991+0000 7fa1723ff700 1 Processor -- start 2023-03-03T05:45:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.992+0000 7fa1723ff700 1 -- start start 2023-03-03T05:45:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa1723ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c0ced80 0x7fa16c1e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa1723ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa16c0cf9a0 0x7fa16c1e3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa1723ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c1597f0 0x7fa16c152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa1723ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c156b50 con 0x7fa16c0ced80 2023-03-03T05:45:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa1723ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c156870 con 0x7fa16c0cf9a0 2023-03-03T05:45:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa1723ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c1ef4e0 con 0x7fa16c1597f0 2023-03-03T05:45:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c0ced80 0x7fa16c1e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c0ced80 0x7fa16c1e5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39566/0 (socket says 172.21.15.102:39566) 2023-03-03T05:45:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 -- 172.21.15.102:0/3856673655 learned_addr learned my addr 172.21.15.102:0/3856673655 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa17099c700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c1597f0 0x7fa16c152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa163fff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa16c0cf9a0 0x7fa16c1e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 -- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c1597f0 msgr2=0x7fa16c152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c1597f0 0x7fa16c152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 -- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa16c0cf9a0 msgr2=0x7fa16c1e3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa16c0cf9a0 0x7fa16c1e3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.993+0000 7fa16bfff700 1 -- 172.21.15.102:0/3856673655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1540094a0 con 0x7fa16c0ced80 2023-03-03T05:45:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.994+0000 7fa16bfff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c0ced80 0x7fa16c1e5310 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00c500 tx=0x7fa15c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.994+0000 7fa1697fa700 1 -- 172.21.15.102:0/3856673655 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15c005cb0 con 0x7fa16c0ced80 2023-03-03T05:45:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.994+0000 7fa1697fa700 1 -- 172.21.15.102:0/3856673655 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15c005e10 con 0x7fa16c0ced80 2023-03-03T05:45:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.994+0000 7fa1697fa700 1 -- 172.21.15.102:0/3856673655 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15c0072d0 con 0x7fa16c0ced80 2023-03-03T05:45:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.994+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa164009b30 con 0x7fa16c0ced80 2023-03-03T05:45:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.994+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16c0cb0c0 con 0x7fa16c0ced80 2023-03-03T05:45:49.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.995+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa16c0cade0 con 0x7fa16c0ced80 2023-03-03T05:45:49.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.996+0000 7fa1697fa700 1 -- 172.21.15.102:0/3856673655 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa15c019030 con 0x7fa16c0ced80 2023-03-03T05:45:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.998+0000 7fa1697fa700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa14c0682f0 0x7fa14c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.998+0000 7fa1697fa700 1 -- 172.21.15.102:0/3856673655 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa15c084eb0 con 0x7fa16c0ced80 2023-03-03T05:45:49.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.999+0000 7fa163fff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa14c0682f0 0x7fa14c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:48.999+0000 7fa163fff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa14c0682f0 0x7fa14c06a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa16c157260 tx=0x7fa154009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:49.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.002+0000 7fa1697fa700 1 -- 172.21.15.102:0/3856673655 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa15c04b5e0 con 0x7fa16c0ced80 2023-03-03T05:45:49.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.238+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa16c0cb3a0 con 0x7fa16c0ced80 2023-03-03T05:45:49.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.238+0000 7fa1697fa700 1 -- 172.21.15.102:0/3856673655 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa15c017020 con 0x7fa16c0ced80 2023-03-03T05:45:49.241 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:49.241 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-03-03T05:45:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa14c0682f0 msgr2=0x7fa14c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa14c0682f0 0x7fa14c06a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa16c157260 tx=0x7fa154009210 comp rx=0 tx=0).stop 2023-03-03T05:45:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c0ced80 msgr2=0x7fa16c1e5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c0ced80 0x7fa16c1e5310 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00c500 tx=0x7fa15c00c810 comp rx=0 tx=0).stop 2023-03-03T05:45:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 shutdown_connections 2023-03-03T05:45:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa14c0682f0 0x7fa14c06a7a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa16c1597f0 0x7fa16c152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c0ced80 0x7fa16c1e5310 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 --2- 172.21.15.102:0/3856673655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa16c0cf9a0 0x7fa16c1e3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 >> 172.21.15.102:0/3856673655 conn(0x7fa16c065fb0 msgr2=0x7fa16c0c5410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 shutdown_connections 2023-03-03T05:45:49.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.241+0000 7fa1723ff700 1 -- 172.21.15.102:0/3856673655 wait complete. 2023-03-03T05:45:49.270 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-03-03T05:45:49.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.512+0000 7f0d23dff700 1 Processor -- start 2023-03-03T05:45:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.512+0000 7f0d23dff700 1 -- start start 2023-03-03T05:45:49.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.513+0000 7f0d23dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d1c0ced80 0x7f0d1c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:49.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.513+0000 7f0d23dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d1c0cf9a0 0x7f0d1c163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.513+0000 7f0d23dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0cb9f0 con 0x7f0d1c1597f0 2023-03-03T05:45:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d23dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c0ca750 con 0x7f0d1c0cf9a0 2023-03-03T05:45:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d23dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c15a230 con 0x7f0d1c0ced80 2023-03-03T05:45:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d21b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d1c0ced80 0x7f0d1c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d2139a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0d1c1597f0 0x7f0d1c0cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35952/0 (socket says 172.21.15.102:35952) 2023-03-03T05:45:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d2139a700 1 -- 172.21.15.102:0/2441370363 learned_addr learned my addr 172.21.15.102:0/2441370363 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d2239c700 1 --2- 172.21.15.102:0/2441370363 >> v2:172.21.15.102:3300/0 conn(0x7f0d1c0cf9a0 0x7f0d1c163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d20b99700 1 -- 172.21.15.102:0/2441370363 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547284656 0 0) 0x7f0d1c0cb9f0 con 0x7f0d1c1597f0 2023-03-03T05:45:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d20b99700 1 -- 172.21.15.102:0/2441370363 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d080035b0 con 0x7f0d1c1597f0 2023-03-03T05:45:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d21b9b700 1 -- 172.21.15.102:0/2441370363 >> v2:172.21.15.102:3300/0 conn(0x7f0d1c0cf9a0 msgr2=0x7f0d1c163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d21b9b700 1 --2- 172.21.15.102:0/2441370363 >> v2:172.21.15.102:3300/0 conn(0x7f0d1c0cf9a0 0x7f0d1c163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d21b9b700 1 -- 172.21.15.102:0/2441370363 >> v1:172.21.15.172:6789/0 conn(0x7f0d1c1597f0 legacy=0x7f0d1c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.514+0000 7f0d21b9b700 1 -- 172.21.15.102:0/2441370363 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c159c80 con 0x7f0d1c0ced80 2023-03-03T05:45:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.515+0000 7f0d21b9b700 1 --2- 172.21.15.102:0/2441370363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c0cf150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0699d0 tx=0x7f0d18009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6efc2c0b1f715e9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.515+0000 7f0d20b99700 1 -- 172.21.15.102:0/2441370363 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18011830 con 0x7f0d1c0ced80 2023-03-03T05:45:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.515+0000 7f0d20b99700 1 -- 172.21.15.102:0/2441370363 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d18011990 con 0x7f0d1c0ced80 2023-03-03T05:45:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.515+0000 7f0d20b99700 1 -- 172.21.15.102:0/2441370363 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18011d20 con 0x7f0d1c0ced80 2023-03-03T05:45:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.515+0000 7f0d23dff700 1 -- 172.21.15.102:0/2441370363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 msgr2=0x7f0d1c0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.515+0000 7f0d23dff700 1 --2- 172.21.15.102:0/2441370363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c0cf150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0699d0 tx=0x7f0d18009b20 comp rx=0 tx=0).stop 2023-03-03T05:45:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.516+0000 7f0d23dff700 1 -- 172.21.15.102:0/2441370363 shutdown_connections 2023-03-03T05:45:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.516+0000 7f0d23dff700 1 --2- 172.21.15.102:0/2441370363 >> v2:172.21.15.102:3300/0 conn(0x7f0d1c0cf9a0 0x7f0d1c163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.516+0000 7f0d23dff700 1 --2- 172.21.15.102:0/2441370363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c0cf150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.516+0000 7f0d23dff700 1 -- 172.21.15.102:0/2441370363 >> 172.21.15.102:0/2441370363 conn(0x7f0d1c065fb0 msgr2=0x7f0d1c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.516+0000 7f0d23dff700 1 -- 172.21.15.102:0/2441370363 shutdown_connections 2023-03-03T05:45:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.516+0000 7f0d23dff700 1 -- 172.21.15.102:0/2441370363 wait complete. 2023-03-03T05:45:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.518+0000 7f0d23dff700 1 Processor -- start 2023-03-03T05:45:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.518+0000 7f0d23dff700 1 -- start start 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d23dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c1f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d23dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0cf9a0 0x7f0d1c14f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d23dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c1597f0 0x7f0d1c1eea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d23dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c150c00 con 0x7f0d1c1597f0 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d23dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c1587b0 con 0x7f0d1c0ced80 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d23dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0c7500 con 0x7f0d1c0cf9a0 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d2239c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c1597f0 0x7f0d1c1eea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d21b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c1f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d2139a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0cf9a0 0x7f0d1c14f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d21b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c1f2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50514/0 (socket says 172.21.15.102:50514) 2023-03-03T05:45:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.519+0000 7f0d21b9b700 1 -- 172.21.15.102:0/3146502195 learned_addr learned my addr 172.21.15.102:0/3146502195 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d21b9b700 1 -- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0cf9a0 msgr2=0x7f0d1c14f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d21b9b700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0cf9a0 0x7f0d1c14f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d21b9b700 1 -- 172.21.15.102:0/3146502195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c1597f0 msgr2=0x7f0d1c1eea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d21b9b700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c1597f0 0x7f0d1c1eea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d21b9b700 1 -- 172.21.15.102:0/3146502195 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d180094a0 con 0x7f0d1c0ced80 2023-03-03T05:45:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d2139a700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0cf9a0 0x7f0d1c14f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d2239c700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c1597f0 0x7f0d1c1eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d21b9b700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c1f2ae0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0d18000c00 tx=0x7f0d18004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d127fc700 1 -- 172.21.15.102:0/3146502195 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18004590 con 0x7f0d1c0ced80 2023-03-03T05:45:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c0cb0c0 con 0x7f0d1c0ced80 2023-03-03T05:45:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d127fc700 1 -- 172.21.15.102:0/3146502195 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1801e070 con 0x7f0d1c0ced80 2023-03-03T05:45:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.520+0000 7f0d127fc700 1 -- 172.21.15.102:0/3146502195 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d180038e0 con 0x7f0d1c0ced80 2023-03-03T05:45:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.521+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c0cade0 con 0x7f0d1c0ced80 2023-03-03T05:45:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.521+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c0cb3a0 con 0x7f0d1c0ced80 2023-03-03T05:45:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.523+0000 7f0d127fc700 1 -- 172.21.15.102:0/3146502195 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d1800f050 con 0x7f0d1c0ced80 2023-03-03T05:45:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.524+0000 7f0d127fc700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d000682f0 0x7f0d0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.524+0000 7f0d127fc700 1 -- 172.21.15.102:0/3146502195 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0d18087530 con 0x7f0d1c0ced80 2023-03-03T05:45:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.524+0000 7f0d2139a700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d000682f0 0x7f0d0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.525+0000 7f0d2139a700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d000682f0 0x7f0d0006a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0b4d20 tx=0x7f0d0c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.527+0000 7f0d127fc700 1 -- 172.21.15.102:0/3146502195 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d18053c20 con 0x7f0d1c0ced80 2023-03-03T05:45:49.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.722+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d1c0677f0 con 0x7f0d1c0ced80 2023-03-03T05:45:49.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.724+0000 7f0d127fc700 1 -- 172.21.15.102:0/3146502195 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f0d1807fe60 con 0x7f0d1c0ced80 2023-03-03T05:45:49.727 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:49.727 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:49.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.726+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d000682f0 msgr2=0x7f0d0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.726+0000 7f0d23dff700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d000682f0 0x7f0d0006a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0b4d20 tx=0x7f0d0c007dc0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 msgr2=0x7f0d1c1f2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:49.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c1f2ae0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0d18000c00 tx=0x7f0d18004750 comp rx=0 tx=0).stop 2023-03-03T05:45:49.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 shutdown_connections 2023-03-03T05:45:49.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c1597f0 0x7f0d1c1eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0ced80 0x7f0d1c1f2ae0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d000682f0 0x7f0d0006a7a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 --2- 172.21.15.102:0/3146502195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0cf9a0 0x7f0d1c14f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 >> 172.21.15.102:0/3146502195 conn(0x7f0d1c065fb0 msgr2=0x7f0d1c1e51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 shutdown_connections 2023-03-03T05:45:49.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:49.727+0000 7f0d23dff700 1 -- 172.21.15.102:0/3146502195 wait complete. 2023-03-03T05:45:49.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T05:45:20.032740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T05:45:27.930463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T05:45:33.613145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T05:45:34.842860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T05:45:49.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T05:45:50.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.001+0000 7fc48279d700 1 Processor -- start 2023-03-03T05:45:50.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.001+0000 7fc48279d700 1 -- start start 2023-03-03T05:45:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc48279d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c1507d0 0x7fc47c06fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc48279d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc47c0c2770 0x7fc47c0c7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc48279d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c0c22e0 con 0x7fc47c150ba0 2023-03-03T05:45:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc48279d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47c0c3450 con 0x7fc47c0c2770 2023-03-03T05:45:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc48279d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c157440 con 0x7fc47c1507d0 2023-03-03T05:45:50.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc48179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c1507d0 0x7fc47c06fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc480f9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc47c150ba0 0x7fc47c0c8a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35958/0 (socket says 172.21.15.102:35958) 2023-03-03T05:45:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc480f9a700 1 -- 172.21.15.102:0/2608143568 learned_addr learned my addr 172.21.15.102:0/2608143568 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc481f9c700 1 --2- 172.21.15.102:0/2608143568 >> v2:172.21.15.172:3300/0 conn(0x7fc47c0c2770 0x7fc47c0c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.003+0000 7fc473fff700 1 -- 172.21.15.102:0/2608143568 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097241878 0 0) 0x7fc47c0c22e0 con 0x7fc47c150ba0 2023-03-03T05:45:50.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc473fff700 1 -- 172.21.15.102:0/2608143568 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4680035b0 con 0x7fc47c150ba0 2023-03-03T05:45:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc481f9c700 1 -- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c1507d0 msgr2=0x7fc47c06fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc481f9c700 1 --2- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c1507d0 0x7fc47c06fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc481f9c700 1 -- 172.21.15.102:0/2608143568 >> v1:172.21.15.172:6789/0 conn(0x7fc47c150ba0 legacy=0x7fc47c0c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc481f9c700 1 -- 172.21.15.102:0/2608143568 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c157720 con 0x7fc47c0c2770 2023-03-03T05:45:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc48179b700 1 --2- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c1507d0 0x7fc47c06fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc481f9c700 1 --2- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0c2770 0x7fc47c0c7450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc474009d90 tx=0x7fc47400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3905182a18cc4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc473fff700 1 -- 172.21.15.102:0/2608143568 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47400e070 con 0x7fc47c0c2770 2023-03-03T05:45:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc473fff700 1 -- 172.21.15.102:0/2608143568 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47400bb20 con 0x7fc47c0c2770 2023-03-03T05:45:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.004+0000 7fc473fff700 1 -- 172.21.15.102:0/2608143568 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc474012430 con 0x7fc47c0c2770 2023-03-03T05:45:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 -- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0c2770 msgr2=0x7fc47c0c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 --2- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0c2770 0x7fc47c0c7450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc474009d90 tx=0x7fc47400b330 comp rx=0 tx=0).stop 2023-03-03T05:45:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 -- 172.21.15.102:0/2608143568 shutdown_connections 2023-03-03T05:45:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 --2- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0c2770 0x7fc47c0c7450 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 --2- 172.21.15.102:0/2608143568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c1507d0 0x7fc47c06fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 -- 172.21.15.102:0/2608143568 >> 172.21.15.102:0/2608143568 conn(0x7fc47c066190 msgr2=0x7fc47c066560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 -- 172.21.15.102:0/2608143568 shutdown_connections 2023-03-03T05:45:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.005+0000 7fc48279d700 1 -- 172.21.15.102:0/2608143568 wait complete. 2023-03-03T05:45:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.007+0000 7fc48279d700 1 Processor -- start 2023-03-03T05:45:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.007+0000 7fc48279d700 1 -- start start 2023-03-03T05:45:50.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.008+0000 7fc48279d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c0c2770 0x7fc47c1f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.008+0000 7fc48279d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c1507d0 0x7fc47c0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.008+0000 7fc48279d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc47c150ba0 0x7fc47c0c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.008+0000 7fc48279d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c0c6d90 con 0x7fc47c150ba0 2023-03-03T05:45:50.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.008+0000 7fc48279d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c159d30 con 0x7fc47c1507d0 2023-03-03T05:45:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.008+0000 7fc48279d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c0c4000 con 0x7fc47c0c2770 2023-03-03T05:45:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c0c2770 0x7fc47c1f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc480f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c1507d0 0x7fc47c0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c0c2770 0x7fc47c1f16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58018/0 (socket says 172.21.15.102:58018) 2023-03-03T05:45:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 -- 172.21.15.102:0/3737279326 learned_addr learned my addr 172.21.15.102:0/3737279326 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc481f9c700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc47c150ba0 0x7fc47c0c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 -- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c1507d0 msgr2=0x7fc47c0c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c1507d0 0x7fc47c0c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 -- 172.21.15.102:0/3737279326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc47c150ba0 msgr2=0x7fc47c0c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc47c150ba0 0x7fc47c0c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 -- 172.21.15.102:0/3737279326 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780094a0 con 0x7fc47c0c2770 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc481f9c700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc47c150ba0 0x7fc47c0c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T05:45:50.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc48179b700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c0c2770 0x7fc47c1f16a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc47800ec10 tx=0x7fc47800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.009+0000 7fc471ffb700 1 -- 172.21.15.102:0/3737279326 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800c910 con 0x7fc47c0c2770 2023-03-03T05:45:50.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.010+0000 7fc471ffb700 1 -- 172.21.15.102:0/3737279326 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47800ca70 con 0x7fc47c0c2770 2023-03-03T05:45:50.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.010+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc474009a70 con 0x7fc47c0c2770 2023-03-03T05:45:50.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.010+0000 7fc471ffb700 1 -- 172.21.15.102:0/3737279326 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800cde0 con 0x7fc47c0c2770 2023-03-03T05:45:50.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.010+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c0c1c90 con 0x7fc47c0c2770 2023-03-03T05:45:50.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.011+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c0c19b0 con 0x7fc47c0c2770 2023-03-03T05:45:50.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.012+0000 7fc471ffb700 1 -- 172.21.15.102:0/3737279326 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc478019030 con 0x7fc47c0c2770 2023-03-03T05:45:50.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.013+0000 7fc471ffb700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4600682f0 0x7fc46006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.013+0000 7fc471ffb700 1 -- 172.21.15.102:0/3737279326 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc478085e10 con 0x7fc47c0c2770 2023-03-03T05:45:50.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.013+0000 7fc480f9a700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4600682f0 0x7fc46006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.014+0000 7fc480f9a700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4600682f0 0x7fc46006a7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0c7ce0 tx=0x7fc46c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.017+0000 7fc471ffb700 1 -- 172.21.15.102:0/3737279326 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc478089030 con 0x7fc47c0c2770 2023-03-03T05:45:50.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.208+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc47c0c1f70 con 0x7fc47c0c2770 2023-03-03T05:45:50.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.209+0000 7fc471ffb700 1 -- 172.21.15.102:0/3737279326 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4780210a0 con 0x7fc47c0c2770 2023-03-03T05:45:50.211 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2023-03-03T05:45:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4600682f0 msgr2=0x7fc46006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4600682f0 0x7fc46006a7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0c7ce0 tx=0x7fc46c007dc0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c0c2770 msgr2=0x7fc47c1f16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c0c2770 0x7fc47c1f16a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc47800ec10 tx=0x7fc47800ef20 comp rx=0 tx=0).stop 2023-03-03T05:45:50.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 shutdown_connections 2023-03-03T05:45:50.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4600682f0 0x7fc46006a7a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc47c0c2770 0x7fc47c1f16a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc47c150ba0 0x7fc47c0c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 --2- 172.21.15.102:0/3737279326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c1507d0 0x7fc47c0c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 >> 172.21.15.102:0/3737279326 conn(0x7fc47c066190 msgr2=0x7fc47c0c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:50.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 shutdown_connections 2023-03-03T05:45:50.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.212+0000 7fc48279d700 1 -- 172.21.15.102:0/3737279326 wait complete. 2023-03-03T05:45:50.220 INFO:tasks.ceph.mds.e.smithi102.stderr:2023-03-03T05:45:50.219+0000 7f68973f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T05:45:50.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T05:45:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.486+0000 7f5d8afff700 1 Processor -- start 2023-03-03T05:45:50.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.486+0000 7f5d8afff700 1 -- start start 2023-03-03T05:45:50.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.487+0000 7f5d8afff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5d84153450 0x7f5d840cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.487+0000 7f5d8afff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d8415a380 0x7f5d84159940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.487+0000 7f5d8afff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d84159e80 con 0x7f5d840cf9a0 2023-03-03T05:45:50.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d8afff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d840cae70 con 0x7f5d84153450 2023-03-03T05:45:50.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d8afff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840cb110 con 0x7f5d8415a380 2023-03-03T05:45:50.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d88d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5d84153450 0x7f5d840cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d8959c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5d840cf9a0 0x7f5d840ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42660/0 (socket says 172.21.15.102:42660) 2023-03-03T05:45:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d83fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d8415a380 0x7f5d84159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d8959c700 1 -- 172.21.15.102:0/1194607524 learned_addr learned my addr 172.21.15.102:0/1194607524 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d837fe700 1 -- 172.21.15.102:0/1194607524 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425768733 0 0) 0x7f5d84159e80 con 0x7f5d840cf9a0 2023-03-03T05:45:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d88d9b700 1 -- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d8415a380 msgr2=0x7f5d84159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d88d9b700 1 --2- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d8415a380 0x7f5d84159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d88d9b700 1 -- 172.21.15.102:0/1194607524 >> v1:172.21.15.102:6789/0 conn(0x7f5d840cf9a0 legacy=0x7f5d840ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d88d9b700 1 -- 172.21.15.102:0/1194607524 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d840cb3f0 con 0x7f5d84153450 2023-03-03T05:45:50.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.488+0000 7f5d83fff700 1 --2- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d8415a380 0x7f5d84159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:50.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.489+0000 7f5d88d9b700 1 --2- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d84153450 0x7f5d840cc1b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009840 tx=0x7f5d78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69ab16f11833a94c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.489+0000 7f5d837fe700 1 -- 172.21.15.102:0/1194607524 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78011830 con 0x7f5d84153450 2023-03-03T05:45:50.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.489+0000 7f5d837fe700 1 -- 172.21.15.102:0/1194607524 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78011990 con 0x7f5d84153450 2023-03-03T05:45:50.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.489+0000 7f5d837fe700 1 -- 172.21.15.102:0/1194607524 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78011d20 con 0x7f5d84153450 2023-03-03T05:45:50.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.489+0000 7f5d8afff700 1 -- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d84153450 msgr2=0x7f5d840cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.489+0000 7f5d8afff700 1 --2- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d84153450 0x7f5d840cc1b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009840 tx=0x7f5d78009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:50.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.490+0000 7f5d8afff700 1 -- 172.21.15.102:0/1194607524 shutdown_connections 2023-03-03T05:45:50.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.490+0000 7f5d8afff700 1 --2- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d84153450 0x7f5d840cc1b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.490+0000 7f5d8afff700 1 --2- 172.21.15.102:0/1194607524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d8415a380 0x7f5d84159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.490+0000 7f5d8afff700 1 -- 172.21.15.102:0/1194607524 >> 172.21.15.102:0/1194607524 conn(0x7f5d84065fb0 msgr2=0x7f5d840663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:50.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.490+0000 7f5d8afff700 1 -- 172.21.15.102:0/1194607524 shutdown_connections 2023-03-03T05:45:50.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.490+0000 7f5d8afff700 1 -- 172.21.15.102:0/1194607524 wait complete. 2023-03-03T05:45:50.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.492+0000 7f5d8afff700 1 Processor -- start 2023-03-03T05:45:50.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.492+0000 7f5d8afff700 1 -- start start 2023-03-03T05:45:50.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.493+0000 7f5d8afff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cf9a0 0x7f5d84150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.493+0000 7f5d8afff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d84153450 0x7f5d840af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.493+0000 7f5d8afff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d8415a380 0x7f5d841e4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.493+0000 7f5d8afff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840afd40 con 0x7f5d84153450 2023-03-03T05:45:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.493+0000 7f5d8afff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d84150a30 con 0x7f5d8415a380 2023-03-03T05:45:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.493+0000 7f5d8afff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d841ef7d0 con 0x7f5d840cf9a0 2023-03-03T05:45:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d83fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d84153450 0x7f5d840af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d88d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cf9a0 0x7f5d84150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d8959c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d8415a380 0x7f5d841e4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d83fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d84153450 0x7f5d840af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39598/0 (socket says 172.21.15.102:39598) 2023-03-03T05:45:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d83fff700 1 -- 172.21.15.102:0/3106276922 learned_addr learned my addr 172.21.15.102:0/3106276922 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:50.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d88d9b700 1 -- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d8415a380 msgr2=0x7f5d841e4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d88d9b700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d8415a380 0x7f5d841e4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d88d9b700 1 -- 172.21.15.102:0/3106276922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d84153450 msgr2=0x7f5d840af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d88d9b700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d84153450 0x7f5d840af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d88d9b700 1 -- 172.21.15.102:0/3106276922 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700094a0 con 0x7f5d840cf9a0 2023-03-03T05:45:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d8959c700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d8415a380 0x7f5d841e4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d83fff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d84153450 0x7f5d840af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.494+0000 7f5d88d9b700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cf9a0 0x7f5d84150ec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d78016000 tx=0x7f5d78004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.495+0000 7f5d817fa700 1 -- 172.21.15.102:0/3106276922 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78004550 con 0x7f5d840cf9a0 2023-03-03T05:45:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.495+0000 7f5d817fa700 1 -- 172.21.15.102:0/3106276922 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7801e070 con 0x7f5d840cf9a0 2023-03-03T05:45:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.495+0000 7f5d817fa700 1 -- 172.21.15.102:0/3106276922 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d780038e0 con 0x7f5d840cf9a0 2023-03-03T05:45:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.495+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d780094a0 con 0x7f5d840cf9a0 2023-03-03T05:45:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.495+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d840cb9b0 con 0x7f5d840cf9a0 2023-03-03T05:45:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.496+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d840cb6d0 con 0x7f5d840cf9a0 2023-03-03T05:45:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.497+0000 7f5d817fa700 1 -- 172.21.15.102:0/3106276922 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d7800f050 con 0x7f5d840cf9a0 2023-03-03T05:45:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.498+0000 7f5d817fa700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d680682f0 0x7f5d6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.498+0000 7f5d817fa700 1 -- 172.21.15.102:0/3106276922 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5d78086c70 con 0x7f5d840cf9a0 2023-03-03T05:45:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.499+0000 7f5d83fff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d680682f0 0x7f5d6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.499+0000 7f5d83fff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d680682f0 0x7f5d6806a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5d841ec9b0 tx=0x7f5d70009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.503+0000 7f5d817fa700 1 -- 172.21.15.102:0/3106276922 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d780533e0 con 0x7f5d840cf9a0 2023-03-03T05:45:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.695+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5d840cbc90 con 0x7f5d840cf9a0 2023-03-03T05:45:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.696+0000 7f5d817fa700 1 -- 172.21.15.102:0/3106276922 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5d78056a00 con 0x7f5d840cf9a0 2023-03-03T05:45:50.697 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2023-03-03T05:45:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d680682f0 msgr2=0x7f5d6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d680682f0 0x7f5d6806a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5d841ec9b0 tx=0x7f5d70009210 comp rx=0 tx=0).stop 2023-03-03T05:45:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cf9a0 msgr2=0x7f5d84150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cf9a0 0x7f5d84150ec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d78016000 tx=0x7f5d78004750 comp rx=0 tx=0).stop 2023-03-03T05:45:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 shutdown_connections 2023-03-03T05:45:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d680682f0 0x7f5d6806a7a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d840cf9a0 0x7f5d84150ec0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d8415a380 0x7f5d841e4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 --2- 172.21.15.102:0/3106276922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d84153450 0x7f5d840af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 >> 172.21.15.102:0/3106276922 conn(0x7f5d84065fb0 msgr2=0x7f5d840ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 shutdown_connections 2023-03-03T05:45:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.698+0000 7f5d8afff700 1 -- 172.21.15.102:0/3106276922 wait complete. 2023-03-03T05:45:50.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T05:45:50.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.972+0000 7f81e35ff700 1 Processor -- start 2023-03-03T05:45:50.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.972+0000 7f81e35ff700 1 -- start start 2023-03-03T05:45:50.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.973+0000 7f81e35ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f81dc0ce410 0x7f81dc0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.973+0000 7f81e35ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81dc0cd760 0x7f81dc163720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.973+0000 7f81e35ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc153630 con 0x7f81dc0ccc40 2023-03-03T05:45:50.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.973+0000 7f81e35ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0ca7a0 con 0x7f81dc0cd760 2023-03-03T05:45:50.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e35ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0cd550 con 0x7f81dc0ce410 2023-03-03T05:45:50.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e1b9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81dc0cd760 0x7f81dc163720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e139b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f81dc0ccc40 0x7f81dc0cdd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42666/0 (socket says 172.21.15.102:42666) 2023-03-03T05:45:50.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e139b700 1 -- 172.21.15.102:0/1890698169 learned_addr learned my addr 172.21.15.102:0/1890698169 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:50.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e0b9a700 1 --2- 172.21.15.102:0/1890698169 >> v2:172.21.15.172:3300/0 conn(0x7f81dc0ce410 0x7f81dc0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e1b9c700 1 -- 172.21.15.102:0/1890698169 >> v2:172.21.15.172:3300/0 conn(0x7f81dc0ce410 msgr2=0x7f81dc0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:50.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e1b9c700 1 --2- 172.21.15.102:0/1890698169 >> v2:172.21.15.172:3300/0 conn(0x7f81dc0ce410 0x7f81dc0cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e1b9c700 1 -- 172.21.15.102:0/1890698169 >> v1:172.21.15.102:6789/0 conn(0x7f81dc0ccc40 legacy=0x7f81dc0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.974+0000 7f81e1b9c700 1 -- 172.21.15.102:0/1890698169 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc15d1d0 con 0x7f81dc0cd760 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81e1b9c700 1 --2- 172.21.15.102:0/1890698169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0cd760 0x7f81dc163720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f81dc156ab0 tx=0x7f81d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fe2529147d7eab server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81d3fff700 1 -- 172.21.15.102:0/1890698169 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d400e070 con 0x7f81dc0cd760 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81d3fff700 1 -- 172.21.15.102:0/1890698169 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d400bb20 con 0x7f81dc0cd760 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81d3fff700 1 -- 172.21.15.102:0/1890698169 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4012430 con 0x7f81dc0cd760 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81e35ff700 1 -- 172.21.15.102:0/1890698169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0cd760 msgr2=0x7f81dc163720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81e35ff700 1 --2- 172.21.15.102:0/1890698169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0cd760 0x7f81dc163720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f81dc156ab0 tx=0x7f81d400b330 comp rx=0 tx=0).stop 2023-03-03T05:45:50.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81e35ff700 1 -- 172.21.15.102:0/1890698169 shutdown_connections 2023-03-03T05:45:50.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81e35ff700 1 --2- 172.21.15.102:0/1890698169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0cd760 0x7f81dc163720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81e35ff700 1 --2- 172.21.15.102:0/1890698169 >> v2:172.21.15.172:3300/0 conn(0x7f81dc0ce410 0x7f81dc0cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.975+0000 7f81e35ff700 1 -- 172.21.15.102:0/1890698169 >> 172.21.15.102:0/1890698169 conn(0x7f81dc065fb0 msgr2=0x7f81dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:50.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.976+0000 7f81e35ff700 1 -- 172.21.15.102:0/1890698169 shutdown_connections 2023-03-03T05:45:50.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.976+0000 7f81e35ff700 1 -- 172.21.15.102:0/1890698169 wait complete. 2023-03-03T05:45:50.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.978+0000 7f81e35ff700 1 Processor -- start 2023-03-03T05:45:50.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.978+0000 7f81e35ff700 1 -- start start 2023-03-03T05:45:50.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e35ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0ccc40 0x7f81dc0b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e35ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc0cd760 0x7f81dc1f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e35ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc0ce410 0x7f81dc1f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e35ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0af1e0 con 0x7f81dc0ccc40 2023-03-03T05:45:50.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e35ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0afcf0 con 0x7f81dc0cd760 2023-03-03T05:45:50.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e35ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0af6a0 con 0x7f81dc0ce410 2023-03-03T05:45:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e139b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0ccc40 0x7f81dc0b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e139b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0ccc40 0x7f81dc0b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39620/0 (socket says 172.21.15.102:39620) 2023-03-03T05:45:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e139b700 1 -- 172.21.15.102:0/4067309157 learned_addr learned my addr 172.21.15.102:0/4067309157 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e1b9c700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc0ce410 0x7f81dc1f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e0b9a700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc0cd760 0x7f81dc1f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e139b700 1 -- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc0ce410 msgr2=0x7f81dc1f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.979+0000 7f81e139b700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc0ce410 0x7f81dc1f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81e139b700 1 -- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc0cd760 msgr2=0x7f81dc1f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81e139b700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc0cd760 0x7f81dc1f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81e139b700 1 -- 172.21.15.102:0/4067309157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d4009a70 con 0x7f81dc0ccc40 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81e139b700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0ccc40 0x7f81dc0b35a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81d800c520 tx=0x7f81d800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81d1ffb700 1 -- 172.21.15.102:0/4067309157 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8005d70 con 0x7f81dc0ccc40 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81d1ffb700 1 -- 172.21.15.102:0/4067309157 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d80070d0 con 0x7f81dc0ccc40 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81d1ffb700 1 -- 172.21.15.102:0/4067309157 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8007440 con 0x7f81dc0ccc40 2023-03-03T05:45:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc1594b0 con 0x7f81dc0ccc40 2023-03-03T05:45:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.980+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc1591d0 con 0x7f81dc0ccc40 2023-03-03T05:45:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.981+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc159790 con 0x7f81dc0ccc40 2023-03-03T05:45:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.982+0000 7f81d1ffb700 1 -- 172.21.15.102:0/4067309157 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81d8019030 con 0x7f81dc0ccc40 2023-03-03T05:45:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.984+0000 7f81d1ffb700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f81c00682f0 0x7f81c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.984+0000 7f81d1ffb700 1 -- 172.21.15.102:0/4067309157 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f81d8084ed0 con 0x7f81dc0ccc40 2023-03-03T05:45:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.984+0000 7f81e0b9a700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f81c00682f0 0x7f81c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:50.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.985+0000 7f81e0b9a700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f81c00682f0 0x7f81c006a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0b2fc0 tx=0x7f81cc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:50.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:50.988+0000 7f81d1ffb700 1 -- 172.21.15.102:0/4067309157 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81d804b5e0 con 0x7f81dc0ccc40 2023-03-03T05:45:51.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.181+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81dc0affd0 con 0x7f81dc0ccc40 2023-03-03T05:45:51.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.181+0000 7f81d1ffb700 1 -- 172.21.15.102:0/4067309157 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f81d8017020 con 0x7f81dc0ccc40 2023-03-03T05:45:51.184 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2023-03-03T05:45:51.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f81c00682f0 msgr2=0x7f81c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f81c00682f0 0x7f81c006a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0b2fc0 tx=0x7f81cc009210 comp rx=0 tx=0).stop 2023-03-03T05:45:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0ccc40 msgr2=0x7f81dc0b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0ccc40 0x7f81dc0b35a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81d800c520 tx=0x7f81d800c830 comp rx=0 tx=0).stop 2023-03-03T05:45:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 shutdown_connections 2023-03-03T05:45:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f81c00682f0 0x7f81c006a7a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f81dc0ce410 0x7f81dc1f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81dc0ccc40 0x7f81dc0b35a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 --2- 172.21.15.102:0/4067309157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81dc0cd760 0x7f81dc1f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 >> 172.21.15.102:0/4067309157 conn(0x7f81dc065fb0 msgr2=0x7f81dc0cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 shutdown_connections 2023-03-03T05:45:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.184+0000 7f81e35ff700 1 -- 172.21.15.102:0/4067309157 wait complete. 2023-03-03T05:45:51.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T05:45:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.462+0000 7f7cdefff700 1 Processor -- start 2023-03-03T05:45:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.462+0000 7f7cdefff700 1 -- start start 2023-03-03T05:45:51.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdefff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd815a380 0x7f7cd80cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdefff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7cd80cf9a0 0x7f7cd80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdefff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80cae70 con 0x7f7cd8153450 2023-03-03T05:45:51.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdefff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd80cb110 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdefff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd80cb3f0 con 0x7f7cd815a380 2023-03-03T05:45:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cd7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd815a380 0x7f7cd80cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdcd9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7cd8153450 0x7f7cd8159940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35966/0 (socket says 172.21.15.102:35966) 2023-03-03T05:45:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdcd9b700 1 -- 172.21.15.102:0/1480229555 learned_addr learned my addr 172.21.15.102:0/1480229555 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.464+0000 7f7cdd59c700 1 --2- 172.21.15.102:0/1480229555 >> v2:172.21.15.102:3300/0 conn(0x7f7cd80cf9a0 0x7f7cd80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd77fe700 1 -- 172.21.15.102:0/1480229555 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101405122 0 0) 0x7f7cd80cae70 con 0x7f7cd8153450 2023-03-03T05:45:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd7fff700 1 -- 172.21.15.102:0/1480229555 >> v2:172.21.15.102:3300/0 conn(0x7f7cd80cf9a0 msgr2=0x7f7cd80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd7fff700 1 --2- 172.21.15.102:0/1480229555 >> v2:172.21.15.102:3300/0 conn(0x7f7cd80cf9a0 0x7f7cd80ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd7fff700 1 -- 172.21.15.102:0/1480229555 >> v1:172.21.15.172:6789/0 conn(0x7f7cd8153450 legacy=0x7f7cd8159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd7fff700 1 -- 172.21.15.102:0/1480229555 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd80cb6d0 con 0x7f7cd815a380 2023-03-03T05:45:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd7fff700 1 --2- 172.21.15.102:0/1480229555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd815a380 0x7f7cd80cc1b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009840 tx=0x7f7cc4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67d10d117c1f6e4b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd77fe700 1 -- 172.21.15.102:0/1480229555 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4011830 con 0x7f7cd815a380 2023-03-03T05:45:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd77fe700 1 -- 172.21.15.102:0/1480229555 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4011990 con 0x7f7cd815a380 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.465+0000 7f7cd77fe700 1 -- 172.21.15.102:0/1480229555 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4011d20 con 0x7f7cd815a380 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 -- 172.21.15.102:0/1480229555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd815a380 msgr2=0x7f7cd80cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 --2- 172.21.15.102:0/1480229555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd815a380 0x7f7cd80cc1b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009840 tx=0x7f7cc4009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 -- 172.21.15.102:0/1480229555 shutdown_connections 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 --2- 172.21.15.102:0/1480229555 >> v2:172.21.15.102:3300/0 conn(0x7f7cd80cf9a0 0x7f7cd80ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 --2- 172.21.15.102:0/1480229555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd815a380 0x7f7cd80cc1b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 -- 172.21.15.102:0/1480229555 >> 172.21.15.102:0/1480229555 conn(0x7f7cd8065fb0 msgr2=0x7f7cd80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 -- 172.21.15.102:0/1480229555 shutdown_connections 2023-03-03T05:45:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.466+0000 7f7cdefff700 1 -- 172.21.15.102:0/1480229555 wait complete. 2023-03-03T05:45:51.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.468+0000 7f7cdefff700 1 Processor -- start 2023-03-03T05:45:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.468+0000 7f7cdefff700 1 -- start start 2023-03-03T05:45:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdefff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cd80cf9a0 0x7f7cd81f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdefff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd8153450 0x7f7cd814ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdefff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd815a380 0x7f7cd81f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdefff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd814e8b0 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdefff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd814cda0 con 0x7f7cd815a380 2023-03-03T05:45:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdefff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd80b47a0 con 0x7f7cd8153450 2023-03-03T05:45:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdcd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cd80cf9a0 0x7f7cd81f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cd7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd8153450 0x7f7cd814ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.469+0000 7f7cdcd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cd80cf9a0 0x7f7cd81f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39644/0 (socket says 172.21.15.102:39644) 2023-03-03T05:45:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdcd9b700 1 -- 172.21.15.102:0/2684163751 learned_addr learned my addr 172.21.15.102:0/2684163751 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cd7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd8153450 0x7f7cd814ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58054/0 (socket says 172.21.15.102:58054) 2023-03-03T05:45:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdd59c700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd815a380 0x7f7cd81f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdcd9b700 1 -- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd8153450 msgr2=0x7f7cd814ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdcd9b700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd8153450 0x7f7cd814ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdcd9b700 1 -- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd815a380 msgr2=0x7f7cd81f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdcd9b700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd815a380 0x7f7cd81f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdcd9b700 1 -- 172.21.15.102:0/2684163751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc40094a0 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cd7fff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd8153450 0x7f7cd814ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdd59c700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd815a380 0x7f7cd81f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.470+0000 7f7cdcd9b700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cd80cf9a0 0x7f7cd81f2800 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c550 tx=0x7f7ccc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.471+0000 7f7cd57fa700 1 -- 172.21.15.102:0/2684163751 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc004c80 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.471+0000 7f7cd57fa700 1 -- 172.21.15.102:0/2684163751 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc004e00 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.471+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd80cbc90 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.471+0000 7f7cd57fa700 1 -- 172.21.15.102:0/2684163751 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc007310 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.471+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd80cb9b0 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.472+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd8066ac0 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.473+0000 7f7cd57fa700 1 -- 172.21.15.102:0/2684163751 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ccc01a020 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.474+0000 7f7cd57fa700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7cbc068320 0x7f7cbc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.474+0000 7f7cd57fa700 1 -- 172.21.15.102:0/2684163751 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7ccc085790 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.475+0000 7f7cd7fff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7cbc068320 0x7f7cbc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.475+0000 7f7cd7fff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7cbc068320 0x7f7cbc06a7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40090b0 tx=0x7f7cc401d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:51.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.478+0000 7f7cd57fa700 1 -- 172.21.15.102:0/2684163751 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ccc051f00 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.672+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7cd80677f0 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.673+0000 7f7cd57fa700 1 -- 172.21.15.102:0/2684163751 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ccc055520 con 0x7f7cd80cf9a0 2023-03-03T05:45:51.675 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2023-03-03T05:45:51.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7cbc068320 msgr2=0x7f7cbc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7cbc068320 0x7f7cbc06a7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40090b0 tx=0x7f7cc401d040 comp rx=0 tx=0).stop 2023-03-03T05:45:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cd80cf9a0 msgr2=0x7f7cd81f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cd80cf9a0 0x7f7cd81f2800 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c550 tx=0x7f7ccc00c860 comp rx=0 tx=0).stop 2023-03-03T05:45:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 shutdown_connections 2023-03-03T05:45:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd815a380 0x7f7cd81f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cd80cf9a0 0x7f7cd81f2800 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7cbc068320 0x7f7cbc06a7d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 --2- 172.21.15.102:0/2684163751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7cd8153450 0x7f7cd814ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 >> 172.21.15.102:0/2684163751 conn(0x7f7cd8065fb0 msgr2=0x7f7cd80ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 shutdown_connections 2023-03-03T05:45:51.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.676+0000 7f7cdefff700 1 -- 172.21.15.102:0/2684163751 wait complete. 2023-03-03T05:45:51.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:45:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.952+0000 7f71e1b9d700 1 Processor -- start 2023-03-03T05:45:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.952+0000 7f71e1b9d700 1 -- start start 2023-03-03T05:45:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e1b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f71dc1564b0 0x7f71dc0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e1b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc0c9610 0x7f71dc0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e1b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc0c9ca0 con 0x7f71dc0ca440 2023-03-03T05:45:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e1b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71dc0b4920 con 0x7f71dc1564b0 2023-03-03T05:45:51.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e1b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc0b4c00 con 0x7f71dc0c9610 2023-03-03T05:45:51.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e0b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f71dc1564b0 0x7f71dc0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e139c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f71dc0ca440 0x7f71dc0ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35974/0 (socket says 172.21.15.102:35974) 2023-03-03T05:45:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71e139c700 1 -- 172.21.15.102:0/2095727377 learned_addr learned my addr 172.21.15.102:0/2095727377 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71dbfff700 1 --2- 172.21.15.102:0/2095727377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc0c9610 0x7f71dc0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.954+0000 7f71db7fe700 1 -- 172.21.15.102:0/2095727377 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367093073 0 0) 0x7f71dc0c9ca0 con 0x7f71dc0ca440 2023-03-03T05:45:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71db7fe700 1 -- 172.21.15.102:0/2095727377 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71cc0035b0 con 0x7f71dc0ca440 2023-03-03T05:45:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71dbfff700 1 -- 172.21.15.102:0/2095727377 >> v2:172.21.15.172:3300/0 conn(0x7f71dc1564b0 msgr2=0x7f71dc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71dbfff700 1 --2- 172.21.15.102:0/2095727377 >> v2:172.21.15.172:3300/0 conn(0x7f71dc1564b0 0x7f71dc0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71dbfff700 1 -- 172.21.15.102:0/2095727377 >> v1:172.21.15.172:6789/0 conn(0x7f71dc0ca440 legacy=0x7f71dc0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71dbfff700 1 -- 172.21.15.102:0/2095727377 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc0b46f0 con 0x7f71dc0c9610 2023-03-03T05:45:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71e0b9b700 1 --2- 172.21.15.102:0/2095727377 >> v2:172.21.15.172:3300/0 conn(0x7f71dc1564b0 0x7f71dc0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71dbfff700 1 --2- 172.21.15.102:0/2095727377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc0c9610 0x7f71dc0cbba0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009840 tx=0x7f71c8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=322dcfaf72858e00 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71db7fe700 1 -- 172.21.15.102:0/2095727377 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8011830 con 0x7f71dc0c9610 2023-03-03T05:45:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71db7fe700 1 -- 172.21.15.102:0/2095727377 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8011990 con 0x7f71dc0c9610 2023-03-03T05:45:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.955+0000 7f71db7fe700 1 -- 172.21.15.102:0/2095727377 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8011d20 con 0x7f71dc0c9610 2023-03-03T05:45:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 -- 172.21.15.102:0/2095727377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc0c9610 msgr2=0x7f71dc0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/2095727377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc0c9610 0x7f71dc0cbba0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009840 tx=0x7f71c8009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:51.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 -- 172.21.15.102:0/2095727377 shutdown_connections 2023-03-03T05:45:51.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/2095727377 >> v2:172.21.15.172:3300/0 conn(0x7f71dc1564b0 0x7f71dc0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/2095727377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc0c9610 0x7f71dc0cbba0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 -- 172.21.15.102:0/2095727377 >> 172.21.15.102:0/2095727377 conn(0x7f71dc065fb0 msgr2=0x7f71dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:51.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 -- 172.21.15.102:0/2095727377 shutdown_connections 2023-03-03T05:45:51.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.956+0000 7f71e1b9d700 1 -- 172.21.15.102:0/2095727377 wait complete. 2023-03-03T05:45:51.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.958+0000 7f71e1b9d700 1 Processor -- start 2023-03-03T05:45:51.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.958+0000 7f71e1b9d700 1 -- start start 2023-03-03T05:45:51.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e1b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71dc0c9610 0x7f71dc0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e1b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71dc0ca440 0x7f71dc0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e1b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc1564b0 0x7f71dc0cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e1b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc157520 con 0x7f71dc0ca440 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e1b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc1eeec0 con 0x7f71dc0c9610 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e1b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc0ce8a0 con 0x7f71dc1564b0 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e0b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71dc0c9610 0x7f71dc0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.959+0000 7f71e0b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71dc0c9610 0x7f71dc0cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50590/0 (socket says 172.21.15.102:50590) 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e0b9b700 1 -- 172.21.15.102:0/3000904260 learned_addr learned my addr 172.21.15.102:0/3000904260 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:51.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71dbfff700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71dc0ca440 0x7f71dc0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e139c700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc1564b0 0x7f71dc0cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e0b9b700 1 -- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc1564b0 msgr2=0x7f71dc0cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e0b9b700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc1564b0 0x7f71dc0cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e0b9b700 1 -- 172.21.15.102:0/3000904260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71dc0ca440 msgr2=0x7f71dc0ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e0b9b700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71dc0ca440 0x7f71dc0ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e0b9b700 1 -- 172.21.15.102:0/3000904260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00094a0 con 0x7f71dc0c9610 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71dbfff700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71dc0ca440 0x7f71dc0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.960+0000 7f71e0b9b700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71dc0c9610 0x7f71dc0cdbb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71d000ec10 tx=0x7f71d000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:51.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.961+0000 7f71d97fa700 1 -- 172.21.15.102:0/3000904260 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0009760 con 0x7f71dc0c9610 2023-03-03T05:45:51.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.961+0000 7f71d97fa700 1 -- 172.21.15.102:0/3000904260 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d000c850 con 0x7f71dc0c9610 2023-03-03T05:45:51.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.961+0000 7f71d97fa700 1 -- 172.21.15.102:0/3000904260 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d000cbe0 con 0x7f71dc0c9610 2023-03-03T05:45:51.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.961+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c80094a0 con 0x7f71dc0c9610 2023-03-03T05:45:51.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.961+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71dc0b2210 con 0x7f71dc0c9610 2023-03-03T05:45:51.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.962+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc0b1f30 con 0x7f71dc0c9610 2023-03-03T05:45:51.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.963+0000 7f71d97fa700 1 -- 172.21.15.102:0/3000904260 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f71d0019030 con 0x7f71dc0c9610 2023-03-03T05:45:51.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.965+0000 7f71d97fa700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f71c00682f0 0x7f71c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:51.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.965+0000 7f71d97fa700 1 -- 172.21.15.102:0/3000904260 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f71d0085c00 con 0x7f71dc0c9610 2023-03-03T05:45:51.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.965+0000 7f71dbfff700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f71c00682f0 0x7f71c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:51.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.966+0000 7f71dbfff700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f71c00682f0 0x7f71c006a7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:51.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:51.969+0000 7f71d97fa700 1 -- 172.21.15.102:0/3000904260 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f71d0052310 con 0x7f71dc0c9610 2023-03-03T05:45:52.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.201+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71dc156880 con 0x7f71dc0c9610 2023-03-03T05:45:52.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.202+0000 7f71d97fa700 1 -- 172.21.15.102:0/3000904260 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6173 (secure 0 0 0) 0x7f71d0055930 con 0x7f71dc0c9610 2023-03-03T05:45:52.204 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:52.204 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:45:41.326836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4315},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4315":{"gid":4315,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6837/1262584332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":1262584332},{"type":"v1","addr":"172.21.15.102:6837","nonce":1262584332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:45:52.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f71c00682f0 msgr2=0x7f71c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:52.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f71c00682f0 0x7f71c006a7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c8008030 comp rx=0 tx=0).stop 2023-03-03T05:45:52.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71dc0c9610 msgr2=0x7f71dc0cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:52.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71dc0c9610 0x7f71dc0cdbb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71d000ec10 tx=0x7f71d000ef20 comp rx=0 tx=0).stop 2023-03-03T05:45:52.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 shutdown_connections 2023-03-03T05:45:52.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f71c00682f0 0x7f71c006a7a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:52.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f71dc1564b0 0x7f71dc0cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:52.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71dc0c9610 0x7f71dc0cdbb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:52.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 --2- 172.21.15.102:0/3000904260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71dc0ca440 0x7f71dc0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:52.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 >> 172.21.15.102:0/3000904260 conn(0x7f71dc065fb0 msgr2=0x7f71dc0c6830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:52.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 shutdown_connections 2023-03-03T05:45:52.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:52.205+0000 7f71e1b9d700 1 -- 172.21.15.102:0/3000904260 wait complete. 2023-03-03T05:45:52.208 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2023-03-03T05:45:52.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T05:45:35.837662+0000', 'modified': '2023-03-03T05:45:41.326836+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4315}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4315': {'gid': 4315, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.102:6837/1262584332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 1262584332}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 1262584332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T05:45:52.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T05:45:52.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-03-03T05:45:53.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.084+0000 7fa979fff700 1 Processor -- start 2023-03-03T05:45:53.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.084+0000 7fa979fff700 1 -- start start 2023-03-03T05:45:53.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa979fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9740ce760 0x7fa974159660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa979fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9740cd0b0 0x7fa9740cd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa979fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9740cda70 con 0x7fa974153000 2023-03-03T05:45:53.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa979fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9741533d0 con 0x7fa9740cd0b0 2023-03-03T05:45:53.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa979fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa974159ba0 con 0x7fa9740ce760 2023-03-03T05:45:53.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa973fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9740cd0b0 0x7fa9740cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa972ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9740ce760 0x7fa974159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa972ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9740ce760 0x7fa974159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50606/0 (socket says 172.21.15.102:50606) 2023-03-03T05:45:53.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa9737fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa974153000 0x7fa97406cb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35976/0 (socket says 172.21.15.102:35976) 2023-03-03T05:45:53.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa972ffd700 1 -- 172.21.15.102:0/1576783398 learned_addr learned my addr 172.21.15.102:0/1576783398 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.086+0000 7fa9727fc700 1 -- 172.21.15.102:0/1576783398 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161419392 0 0) 0x7fa9740cda70 con 0x7fa974153000 2023-03-03T05:45:53.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa973fff700 1 -- 172.21.15.102:0/1576783398 >> v2:172.21.15.172:3300/0 conn(0x7fa9740ce760 msgr2=0x7fa974159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa973fff700 1 --2- 172.21.15.102:0/1576783398 >> v2:172.21.15.172:3300/0 conn(0x7fa9740ce760 0x7fa974159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa973fff700 1 -- 172.21.15.102:0/1576783398 >> v1:172.21.15.172:6789/0 conn(0x7fa974153000 legacy=0x7fa97406cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa973fff700 1 -- 172.21.15.102:0/1576783398 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9740cbaa0 con 0x7fa9740cd0b0 2023-03-03T05:45:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa973fff700 1 --2- 172.21.15.102:0/1576783398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9740cd0b0 0x7fa9740cd530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa97408c720 tx=0x7fa96c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699d69749478cdc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa9727fc700 1 -- 172.21.15.102:0/1576783398 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c00e070 con 0x7fa9740cd0b0 2023-03-03T05:45:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa9727fc700 1 -- 172.21.15.102:0/1576783398 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c00bb20 con 0x7fa9740cd0b0 2023-03-03T05:45:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.087+0000 7fa9727fc700 1 -- 172.21.15.102:0/1576783398 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c012430 con 0x7fa9740cd0b0 2023-03-03T05:45:53.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 -- 172.21.15.102:0/1576783398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9740cd0b0 msgr2=0x7fa9740cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 --2- 172.21.15.102:0/1576783398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9740cd0b0 0x7fa9740cd530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa97408c720 tx=0x7fa96c00b330 comp rx=0 tx=0).stop 2023-03-03T05:45:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 -- 172.21.15.102:0/1576783398 shutdown_connections 2023-03-03T05:45:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 --2- 172.21.15.102:0/1576783398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9740cd0b0 0x7fa9740cd530 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 --2- 172.21.15.102:0/1576783398 >> v2:172.21.15.172:3300/0 conn(0x7fa9740ce760 0x7fa974159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 -- 172.21.15.102:0/1576783398 >> 172.21.15.102:0/1576783398 conn(0x7fa9740ceef0 msgr2=0x7fa9740cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 -- 172.21.15.102:0/1576783398 shutdown_connections 2023-03-03T05:45:53.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.088+0000 7fa979fff700 1 -- 172.21.15.102:0/1576783398 wait complete. 2023-03-03T05:45:53.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.090+0000 7fa979fff700 1 Processor -- start 2023-03-03T05:45:53.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.090+0000 7fa979fff700 1 -- start start 2023-03-03T05:45:53.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa979fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9740cd0b0 0x7fa9740afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa979fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9740ce760 0x7fa9740c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa979fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa974153000 0x7fa9740c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa979fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9741ef1c0 con 0x7fa974153000 2023-03-03T05:45:53.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa979fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9741ee8d0 con 0x7fa9740ce760 2023-03-03T05:45:53.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa979fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9741f28c0 con 0x7fa9740cd0b0 2023-03-03T05:45:53.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa973fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa974153000 0x7fa9740c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa973fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa974153000 0x7fa9740c8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39684/0 (socket says 172.21.15.102:39684) 2023-03-03T05:45:53.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa973fff700 1 -- 172.21.15.102:0/2455624957 learned_addr learned my addr 172.21.15.102:0/2455624957 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa972ffd700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9740ce760 0x7fa9740c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.091+0000 7fa9737fe700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9740cd0b0 0x7fa9740afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa973fff700 1 -- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9740cd0b0 msgr2=0x7fa9740afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa973fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9740cd0b0 0x7fa9740afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa973fff700 1 -- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9740ce760 msgr2=0x7fa9740c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa973fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9740ce760 0x7fa9740c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa973fff700 1 -- 172.21.15.102:0/2455624957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c009a70 con 0x7fa974153000 2023-03-03T05:45:53.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa973fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa974153000 0x7fa9740c8b20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa974074280 tx=0x7fa96c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa95bfff700 1 -- 172.21.15.102:0/2455624957 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c012430 con 0x7fa974153000 2023-03-03T05:45:53.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa95bfff700 1 -- 172.21.15.102:0/2455624957 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c020070 con 0x7fa974153000 2023-03-03T05:45:53.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa95bfff700 1 -- 172.21.15.102:0/2455624957 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c004280 con 0x7fa974153000 2023-03-03T05:45:53.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.092+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9740cc060 con 0x7fa974153000 2023-03-03T05:45:53.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.093+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9740cbd80 con 0x7fa974153000 2023-03-03T05:45:53.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.095+0000 7fa95bfff700 1 -- 172.21.15.102:0/2455624957 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa96c02e070 con 0x7fa974153000 2023-03-03T05:45:53.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.096+0000 7fa95bfff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa960068320 0x7fa96006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.096+0000 7fa95bfff700 1 -- 172.21.15.102:0/2455624957 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa96c022070 con 0x7fa974153000 2023-03-03T05:45:53.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.096+0000 7fa9737fe700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa960068320 0x7fa96006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.097+0000 7fa9737fe700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa960068320 0x7fa96006a7d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa964000f60 tx=0x7fa964007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.098+0000 7fa98d975b80 1 Processor -- start 2023-03-03T05:45:53.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.098+0000 7fa98d975b80 1 -- start start 2023-03-03T05:45:53.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.099+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570494520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.099+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x55557049c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.099+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x55557048cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.099+0000 7fa98d975b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555703c84f0 con 0x5555704928a0 2023-03-03T05:45:53.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.099+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55557033db60 con 0x55557049cac0 2023-03-03T05:45:53.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.099+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x5555703acad0 con 0x555570492c70 2023-03-03T05:45:53.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x55557049c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570494520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x55557049c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58086/0 (socket says 172.21.15.102:58086) 2023-03-03T05:45:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 -- 172.21.15.102:0/1225079381 learned_addr learned my addr 172.21.15.102:0/1225079381 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa973fff700 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x55557048cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 -- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 msgr2=0x55557048cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x55557048cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 -- 172.21.15.102:0/1225079381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 msgr2=0x555570494520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570494520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 -- 172.21.15.102:0/1225079381 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55557019a540 con 0x555570492c70 2023-03-03T05:45:53.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.100+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x55557049c580 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa968009c10 tx=0x7fa968009f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d971d25a998e8d31 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa959ffb700 1 -- 172.21.15.102:0/1225079381 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968011a80 con 0x555570492c70 2023-03-03T05:45:53.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa959ffb700 1 -- 172.21.15.102:0/1225079381 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa968011be0 con 0x555570492c70 2023-03-03T05:45:53.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa959ffb700 1 -- 172.21.15.102:0/1225079381 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968014440 con 0x555570492c70 2023-03-03T05:45:53.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 -- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 msgr2=0x55557049c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x55557049c580 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa968009c10 tx=0x7fa968009f20 comp rx=0 tx=0).stop 2023-03-03T05:45:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 -- 172.21.15.102:0/1225079381 shutdown_connections 2023-03-03T05:45:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x55557048cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570494520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1225079381 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x55557049c580 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 -- 172.21.15.102:0/1225079381 >> 172.21.15.102:0/1225079381 conn(0x555570452480 msgr2=0x555570450bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.101+0000 7fa98d975b80 1 -- 172.21.15.102:0/1225079381 shutdown_connections 2023-03-03T05:45:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.102+0000 7fa98d975b80 1 -- 172.21.15.102:0/1225079381 wait complete. 2023-03-03T05:45:53.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.102+0000 7fa98d975b80 1 -- start start 2023-03-03T05:45:53.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.103+0000 7fa98d975b80 1 Processor -- start 2023-03-03T05:45:53.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5555704928a0 0x5555704855f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55557049cac0 0x55557047e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa98d975b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555703a5e40 con 0x55557049cac0 2023-03-03T05:45:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55557019b140 con 0x555570492c70 2023-03-03T05:45:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x555570490c90 con 0x5555704928a0 2023-03-03T05:45:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa98d975b80 10 client.0 _ll_register_callbacks cb 0x5555703441c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T05:45:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5555704928a0 0x5555704855f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa973fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55557049cac0 0x55557047e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa973fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55557049cac0 0x55557047e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39708/0 (socket says 172.21.15.102:39708) 2023-03-03T05:45:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5555704928a0 0x5555704855f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58102/0 (socket says 172.21.15.102:58102) 2023-03-03T05:45:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa973fff700 1 -- 172.21.15.102:0/1753468415 learned_addr learned my addr 172.21.15.102:0/1753468415 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.104+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9737fe700 1 -- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 msgr2=0x5555704a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9737fe700 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9737fe700 1 -- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55557049cac0 msgr2=0x55557047e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9737fe700 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55557049cac0 0x55557047e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9737fe700 1 -- 172.21.15.102:0/1753468415 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa968009840 con 0x5555704928a0 2023-03-03T05:45:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa973fff700 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55557049cac0 0x55557047e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9737fe700 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5555704928a0 0x5555704855f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa9640076d0 tx=0x7fa964000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f357f8835b9cec42 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9527fc700 10 client.0 ms_handle_connect on v2:172.21.15.172:3301/0 2023-03-03T05:45:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9527fc700 1 -- 172.21.15.102:0/1753468415 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9640019b0 con 0x5555704928a0 2023-03-03T05:45:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa9527fc700 1 -- 172.21.15.102:0/1753468415 <== mon.2 v2:172.21.15.172:3301/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa96401b070 con 0x5555704928a0 2023-03-03T05:45:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.105+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555570445c50 con 0x5555704928a0 2023-03-03T05:45:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa9527fc700 1 -- 172.21.15.102:0/1753468415 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa964001b10 con 0x5555704928a0 2023-03-03T05:45:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 10 client.4508 fetch_fsmap learned FSMap version 8 2023-03-03T05:45:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x5555701d4b10 con 0x5555704928a0 2023-03-03T05:45:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa9527fc700 1 -- 172.21.15.102:0/1753468415 <== mon.2 v2:172.21.15.172:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555570445c50 con 0x5555704928a0 2023-03-03T05:45:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa9527fc700 1 -- 172.21.15.102:0/1753468415 <== mon.2 v2:172.21.15.172:3301/0 5 ==== fsmap(e 8) v1 ==== 4196+0+0 (secure 0 0 0) 0x7fa964021440 con 0x5555704928a0 2023-03-03T05:45:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 10 client.4508 fetch_fsmap finished waiting for FSMap version 8 2023-03-03T05:45:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 10 client.4508 resolve_mds: resolved name 'e' to daemon mds.e 2023-03-03T05:45:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 20 client.4508 populate_metadata read hostname 'smithi102' 2023-03-03T05:45:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x5555704a6880 0x55557047ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 4 client.4508 mds_command: new command op to 4315 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T05:45:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 --> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55557039f020 con 0x5555704a6880 2023-03-03T05:45:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.106+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x5555704a6880 0x55557047ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.107+0000 7fa9527fc700 10 client.4508 ms_handle_connect on v2:172.21.15.102:6836/1262584332 2023-03-03T05:45:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.107+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x5555704a6880 0x55557047ca20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9deb389c03f89e9f server_cookie=5dcde9397bdf3df0 in_seq=0 out_seq=0 2023-03-03T05:45:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa9527fc700 1 -- 172.21.15.102:0/1753468415 <== mds.0 v2:172.21.15.102:6836/1262584332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55557039f020 con 0x5555704a6880 2023-03-03T05:45:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa9527fc700 10 client.4508 handle_command_reply: tid=0 2023-03-03T05:45:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 client.4508 shutdown 2023-03-03T05:45:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5555704928a0 msgr2=0x5555704855f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5555704928a0 0x5555704855f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa9640076d0 tx=0x7fa964000ed0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 shutdown_connections 2023-03-03T05:45:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x5555704a6880 0x55557047ca20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x5555704928a0 0x5555704855f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55557049cac0 0x55557047e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa9527fc700 0 client.4508 ms_handle_reset on v2:172.21.15.102:6836/1262584332 2023-03-03T05:45:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1753468415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 >> 172.21.15.102:0/1753468415 conn(0x555570452480 msgr2=0x555570493810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 shutdown_connections 2023-03-03T05:45:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.108+0000 7fa98d975b80 1 -- 172.21.15.102:0/1753468415 wait complete. 2023-03-03T05:45:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.109+0000 7fa98d975b80 20 client.4508 trim_cache size 0 max 16384 2023-03-03T05:45:53.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.127+0000 7fa98d975b80 1 Processor -- start 2023-03-03T05:45:53.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.127+0000 7fa98d975b80 1 -- start start 2023-03-03T05:45:53.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x55557044e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x5555704871d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x555570487710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa98d975b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x555570449c50 con 0x5555704928a0 2023-03-03T05:45:53.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x555570454d70 con 0x55557049cac0 2023-03-03T05:45:53.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x555570490b10 con 0x555570492c70 2023-03-03T05:45:53.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x55557044e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x55557044e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39724/0 (socket says 172.21.15.102:39724) 2023-03-03T05:45:53.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa9737fe700 1 -- 172.21.15.102:0/2975664979 learned_addr learned my addr 172.21.15.102:0/2975664979 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa972ffd700 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x5555704871d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.128+0000 7fa973fff700 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x555570487710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa9737fe700 1 -- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 msgr2=0x5555704871d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa9737fe700 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x5555704871d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa9737fe700 1 -- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 msgr2=0x555570487710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa9737fe700 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x555570487710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa9737fe700 1 -- 172.21.15.102:0/2975664979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55557036d230 con 0x5555704928a0 2023-03-03T05:45:53.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa9737fe700 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x55557044e6b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa964001120 tx=0x7fa964020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8f7f22e58d1ad2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa95a7fc700 1 -- 172.21.15.102:0/2975664979 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9640219a0 con 0x5555704928a0 2023-03-03T05:45:53.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa95a7fc700 1 -- 172.21.15.102:0/2975664979 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96400b070 con 0x5555704928a0 2023-03-03T05:45:53.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.129+0000 7fa95a7fc700 1 -- 172.21.15.102:0/2975664979 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964013d30 con 0x5555704928a0 2023-03-03T05:45:53.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 -- 172.21.15.102:0/2975664979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 msgr2=0x55557044e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x55557044e6b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa964001120 tx=0x7fa964020000 comp rx=0 tx=0).stop 2023-03-03T05:45:53.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 -- 172.21.15.102:0/2975664979 shutdown_connections 2023-03-03T05:45:53.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55557049cac0 0x555570487710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x55557044e6b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 --2- 172.21.15.102:0/2975664979 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x555570492c70 0x5555704871d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 -- 172.21.15.102:0/2975664979 >> 172.21.15.102:0/2975664979 conn(0x555570452480 msgr2=0x5555704b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 -- 172.21.15.102:0/2975664979 shutdown_connections 2023-03-03T05:45:53.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.130+0000 7fa98d975b80 1 -- 172.21.15.102:0/2975664979 wait complete. 2023-03-03T05:45:53.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.131+0000 7fa98d975b80 1 -- start start 2023-03-03T05:45:53.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.131+0000 7fa98d975b80 1 Processor -- start 2023-03-03T05:45:53.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570493540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa98d975b80 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55557049cac0 0x5555704f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa98d975b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555704ff070 con 0x5555704928a0 2023-03-03T05:45:53.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55557048fe70 con 0x555570492c70 2023-03-03T05:45:53.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa98d975b80 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x555570486180 con 0x55557049cac0 2023-03-03T05:45:53.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa98d975b80 10 client.0 _ll_register_callbacks cb 0x5555703441c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T05:45:53.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570493540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570493540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39732/0 (socket says 172.21.15.102:39732) 2023-03-03T05:45:53.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa9737fe700 1 -- 172.21.15.102:0/1863354284 learned_addr learned my addr 172.21.15.102:0/1863354284 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.132+0000 7fa973fff700 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55557049cac0 0x5555704f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9737fe700 1 -- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55557049cac0 msgr2=0x5555704f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9737fe700 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55557049cac0 0x5555704f56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9737fe700 1 -- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 msgr2=0x5555704f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9737fe700 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704f51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9737fe700 1 -- 172.21.15.102:0/1863354284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555701d4b10 con 0x5555704928a0 2023-03-03T05:45:53.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9737fe700 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570493540 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa964000ff0 tx=0x7fa96400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67addaf9ef50ecdc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9527fc700 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2023-03-03T05:45:53.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9527fc700 1 -- 172.21.15.102:0/1863354284 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964021be0 con 0x5555704928a0 2023-03-03T05:45:53.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9527fc700 1 -- 172.21.15.102:0/1863354284 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa964021d40 con 0x5555704928a0 2023-03-03T05:45:53.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa968011be0 con 0x5555704928a0 2023-03-03T05:45:53.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa9527fc700 1 -- 172.21.15.102:0/1863354284 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9640019b0 con 0x5555704928a0 2023-03-03T05:45:53.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa98d975b80 10 client.4545 fetch_fsmap learned FSMap version 8 2023-03-03T05:45:53.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.133+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x5555701bd990 con 0x5555704928a0 2023-03-03T05:45:53.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa9527fc700 1 -- 172.21.15.102:0/1863354284 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa968011be0 con 0x5555704928a0 2023-03-03T05:45:53.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa9527fc700 1 -- 172.21.15.102:0/1863354284 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 8) v1 ==== 4196+0+0 (secure 0 0 0) 0x7fa964042020 con 0x5555704928a0 2023-03-03T05:45:53.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa98d975b80 10 client.4545 fetch_fsmap finished waiting for FSMap version 8 2023-03-03T05:45:53.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa98d975b80 10 client.4545 resolve_mds: resolved name 'e' to daemon mds.e 2023-03-03T05:45:53.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa98d975b80 20 client.4545 populate_metadata read hostname 'smithi102' 2023-03-03T05:45:53.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x55557044d300 0x55557044d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa98d975b80 4 client.4545 mds_command: new command op to 4315 tid=0[{"prefix": "status"}] 2023-03-03T05:45:53.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 --> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5555701f5ca0 con 0x55557044d300 2023-03-03T05:45:53.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.134+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x55557044d300 0x55557044d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa9527fc700 10 client.4545 ms_handle_connect on v2:172.21.15.102:6836/1262584332 2023-03-03T05:45:53.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa972ffd700 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x55557044d300 0x55557044d6d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83a1ab22c7507209 server_cookie=eca837647cfc3570 in_seq=0 out_seq=0 2023-03-03T05:45:53.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa9527fc700 1 -- 172.21.15.102:0/1863354284 <== mds.0 v2:172.21.15.102:6836/1262584332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5555701f5ca0 con 0x55557044d300 2023-03-03T05:45:53.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa9527fc700 10 client.4545 handle_command_reply: tid=0 2023-03-03T05:45:53.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 client.4545 shutdown 2023-03-03T05:45:53.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 msgr2=0x555570493540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570493540 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa964000ff0 tx=0x7fa96400a000 comp rx=0 tx=0).stop 2023-03-03T05:45:53.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 shutdown_connections 2023-03-03T05:45:53.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:6836/1262584332,v1:172.21.15.102:6837/1262584332] conn(0x55557044d300 0x55557044d6d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x55557049cac0 0x5555704f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5555704928a0 0x555570493540 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa9527fc700 0 client.4545 ms_handle_reset on v2:172.21.15.102:6836/1262584332 2023-03-03T05:45:53.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 --2- 172.21.15.102:0/1863354284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555570492c70 0x5555704f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.135+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 >> 172.21.15.102:0/1863354284 conn(0x555570452480 msgr2=0x555570480f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 shutdown_connections 2023-03-03T05:45:53.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa98d975b80 1 -- 172.21.15.102:0/1863354284 wait complete. 2023-03-03T05:45:53.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa98d975b80 20 client.4545 trim_cache size 0 max 16384 2023-03-03T05:45:53.147 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2023-03-03T05:45:53.147 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "56ef27ff-a10a-47b1-b557-af27e3b0b6e2", 2023-03-03T05:45:53.147 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2023-03-03T05:45:53.147 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4315, 2023-03-03T05:45:53.147 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2023-03-03T05:45:53.147 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2023-03-03T05:45:53.147 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2023-03-03T05:45:53.148 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 17.295206739000001, 2023-03-03T05:45:53.148 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 8, 2023-03-03T05:45:53.148 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 23, 2023-03-03T05:45:53.148 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 23, 2023-03-03T05:45:53.148 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 23.803065476 2023-03-03T05:45:53.148 INFO:teuthology.orchestra.run.smithi102.stdout:} 2023-03-03T05:45:53.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa960068320 msgr2=0x7fa96006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa979fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa960068320 0x7fa96006a7d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa964000f60 tx=0x7fa964007e70 comp rx=0 tx=0).stop 2023-03-03T05:45:53.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa974153000 msgr2=0x7fa9740c8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa979fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa974153000 0x7fa9740c8b20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa974074280 tx=0x7fa96c004ad0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.136+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 shutdown_connections 2023-03-03T05:45:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.137+0000 7fa979fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa960068320 0x7fa96006a7d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.137+0000 7fa979fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9740cd0b0 0x7fa9740afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.137+0000 7fa979fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa974153000 0x7fa9740c8b20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.137+0000 7fa979fff700 1 --2- 172.21.15.102:0/2455624957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9740ce760 0x7fa9740c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.137+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 >> 172.21.15.102:0/2455624957 conn(0x7fa9740ceef0 msgr2=0x7fa974157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.137+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 shutdown_connections 2023-03-03T05:45:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.137+0000 7fa979fff700 1 -- 172.21.15.102:0/2455624957 wait complete. 2023-03-03T05:45:53.164 INFO:teuthology.run_tasks:Running task kclient... 2023-03-03T05:45:53.213 INFO:tasks.kclient:Mounting kernel clients... 2023-03-03T05:45:53.213 INFO:tasks.kclient:config is {'client.0': {}} 2023-03-03T05:45:53.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:53.214 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2023-03-03T05:45:53.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:53.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2023-03-03T05:45:53.388 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2023-03-03T05:45:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T05:45:53.393 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-03-03T05:45:53.393 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T05:45:53.394 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T05:45:53.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T05:45:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.636+0000 7fa2ef3ff700 1 Processor -- start 2023-03-03T05:45:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.636+0000 7fa2ef3ff700 1 -- start start 2023-03-03T05:45:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.637+0000 7fa2ef3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa2e80ced80 0x7fa2e80cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ef3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2e81597f0 0x7fa2e80cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ef3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80cf9a0 0x7fa2e815c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ef3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e80ca750 con 0x7fa2e81597f0 2023-03-03T05:45:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ef3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e815a230 con 0x7fa2e80ced80 2023-03-03T05:45:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ef3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80cf690 con 0x7fa2e80cf9a0 2023-03-03T05:45:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2e81597f0 0x7fa2e80cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2e81597f0 0x7fa2e80cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39734/0 (socket says 172.21.15.102:39734) 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 -- 172.21.15.102:0/2679160043 learned_addr learned my addr 172.21.15.102:0/2679160043 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ed99c700 1 --2- 172.21.15.102:0/2679160043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80cf9a0 0x7fa2e815c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ed19b700 1 --2- 172.21.15.102:0/2679160043 >> v2:172.21.15.172:3300/0 conn(0x7fa2e80ced80 0x7fa2e80cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 -- 172.21.15.102:0/2679160043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80cf9a0 msgr2=0x7fa2e815c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 --2- 172.21.15.102:0/2679160043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80cf9a0 0x7fa2e815c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 -- 172.21.15.102:0/2679160043 >> v2:172.21.15.172:3300/0 conn(0x7fa2e80ced80 msgr2=0x7fa2e80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 --2- 172.21.15.102:0/2679160043 >> v2:172.21.15.172:3300/0 conn(0x7fa2e80ced80 0x7fa2e80cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.638+0000 7fa2ec99a700 1 -- 172.21.15.102:0/2679160043 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80cb9f0 con 0x7fa2e81597f0 2023-03-03T05:45:53.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.639+0000 7fa2ec99a700 1 --2- 172.21.15.102:0/2679160043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e81597f0 0x7fa2e80cbe80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80699d0 tx=0x7fa2d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2df20252221bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.639+0000 7fa2dffff700 1 -- 172.21.15.102:0/2679160043 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8011830 con 0x7fa2e81597f0 2023-03-03T05:45:53.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.639+0000 7fa2dffff700 1 -- 172.21.15.102:0/2679160043 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d8011990 con 0x7fa2e81597f0 2023-03-03T05:45:53.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.639+0000 7fa2dffff700 1 -- 172.21.15.102:0/2679160043 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8011d20 con 0x7fa2e81597f0 2023-03-03T05:45:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.639+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/2679160043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e81597f0 msgr2=0x7fa2e80cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.639+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/2679160043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e81597f0 0x7fa2e80cbe80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80699d0 tx=0x7fa2d8009c00 comp rx=0 tx=0).stop 2023-03-03T05:45:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.640+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/2679160043 shutdown_connections 2023-03-03T05:45:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.640+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/2679160043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80cf9a0 0x7fa2e815c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.640+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/2679160043 >> v2:172.21.15.172:3300/0 conn(0x7fa2e80ced80 0x7fa2e80cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.640+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/2679160043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e81597f0 0x7fa2e80cbe80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.640+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/2679160043 >> 172.21.15.102:0/2679160043 conn(0x7fa2e8065fb0 msgr2=0x7fa2e80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.640+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/2679160043 shutdown_connections 2023-03-03T05:45:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.640+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/2679160043 wait complete. 2023-03-03T05:45:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.642+0000 7fa2ef3ff700 1 Processor -- start 2023-03-03T05:45:53.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.642+0000 7fa2ef3ff700 1 -- start start 2023-03-03T05:45:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ef3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80ced80 0x7fa2e814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ef3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e80cf9a0 0x7fa2e81509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ef3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2e81597f0 0x7fa2e80c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ef3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80c68b0 con 0x7fa2e80cf9a0 2023-03-03T05:45:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ef3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8150f00 con 0x7fa2e81597f0 2023-03-03T05:45:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ef3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8150530 con 0x7fa2e80ced80 2023-03-03T05:45:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ec99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e80cf9a0 0x7fa2e81509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ed19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80ced80 0x7fa2e814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ed19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80ced80 0x7fa2e814d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58126/0 (socket says 172.21.15.102:58126) 2023-03-03T05:45:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ed19b700 1 -- 172.21.15.102:0/1032194004 learned_addr learned my addr 172.21.15.102:0/1032194004 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:53.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.643+0000 7fa2ed99c700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2e81597f0 0x7fa2e80c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ed19b700 1 -- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2e81597f0 msgr2=0x7fa2e80c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ed19b700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2e81597f0 0x7fa2e80c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ed19b700 1 -- 172.21.15.102:0/1032194004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e80cf9a0 msgr2=0x7fa2e81509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ed19b700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e80cf9a0 0x7fa2e81509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ed19b700 1 -- 172.21.15.102:0/1032194004 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8009550 con 0x7fa2e80ced80 2023-03-03T05:45:53.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ec99a700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e80cf9a0 0x7fa2e81509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ed19b700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80ced80 0x7fa2e814d580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400e8a0 tx=0x7fa2e400ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ddffb700 1 -- 172.21.15.102:0/1032194004 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4013680 con 0x7fa2e80ced80 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ddffb700 1 -- 172.21.15.102:0/1032194004 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e40137e0 con 0x7fa2e80ced80 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e80cade0 con 0x7fa2e80ced80 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.644+0000 7fa2ddffb700 1 -- 172.21.15.102:0/1032194004 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4013b90 con 0x7fa2e80ced80 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.645+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e8159c80 con 0x7fa2e80ced80 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.646+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e80cb0c0 con 0x7fa2e80ced80 2023-03-03T05:45:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.647+0000 7fa2ddffb700 1 -- 172.21.15.102:0/1032194004 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa2e4023070 con 0x7fa2e80ced80 2023-03-03T05:45:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.648+0000 7fa2ddffb700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2cc0682f0 0x7fa2cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.648+0000 7fa2ddffb700 1 -- 172.21.15.102:0/1032194004 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa2e4085e30 con 0x7fa2e80ced80 2023-03-03T05:45:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.648+0000 7fa2ec99a700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2cc0682f0 0x7fa2cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.649+0000 7fa2ec99a700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2cc0682f0 0x7fa2cc06a7a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8014040 tx=0x7fa2d8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:53.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.652+0000 7fa2ddffb700 1 -- 172.21.15.102:0/1032194004 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa2e40524b0 con 0x7fa2e80ced80 2023-03-03T05:45:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.885+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa2e80c70c0 con 0x7fa2e80ced80 2023-03-03T05:45:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.886+0000 7fa2ddffb700 1 -- 172.21.15.102:0/1032194004 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa2e80c70c0 con 0x7fa2e80ced80 2023-03-03T05:45:53.888 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T05:45:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2cc0682f0 msgr2=0x7fa2cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2cc0682f0 0x7fa2cc06a7a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8014040 tx=0x7fa2d8008030 comp rx=0 tx=0).stop 2023-03-03T05:45:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80ced80 msgr2=0x7fa2e814d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80ced80 0x7fa2e814d580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400e8a0 tx=0x7fa2e400ebb0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 shutdown_connections 2023-03-03T05:45:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2cc0682f0 0x7fa2cc06a7a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2e80ced80 0x7fa2e814d580 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2e81597f0 0x7fa2e80c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 --2- 172.21.15.102:0/1032194004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e80cf9a0 0x7fa2e81509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.888+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 >> 172.21.15.102:0/1032194004 conn(0x7fa2e8065fb0 msgr2=0x7fa2e81516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.889+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 shutdown_connections 2023-03-03T05:45:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:53.889+0000 7fa2ef3ff700 1 -- 172.21.15.102:0/1032194004 wait complete. 2023-03-03T05:45:53.917 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T05:45:53.918 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T05:45:53.918 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2023-03-03T05:45:53.918 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T05:45:53.918 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T05:45:53.919 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T05:45:53.919 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T05:45:53.919 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T05:45:53.919 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T05:45:53.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:53.920 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2023-03-03T05:45:53.988 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T05:45:53.989 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T05:45:53.989 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T05:45:53.989 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2023-03-03T05:45:53.989 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2023-03-03T05:45:53.989 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2023-03-03T05:45:53.989 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T05:45:53.990 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2023-03-03T05:45:53.990 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2023-03-03T05:45:53.990 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T05:45:53.990 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2023-03-03T05:45:53.990 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2023-03-03T05:45:53.990 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T05:45:53.990 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2023-03-03T05:45:53.991 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T05:45:53.991 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2023-03-03T05:45:53.991 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T05:45:53.991 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42204sec preferred_lft 42204sec 2023-03-03T05:45:53.991 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2023-03-03T05:45:53.991 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2023-03-03T05:45:53.992 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-03T05:45:53.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:53.993 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2023-03-03T05:45:53.993 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2023-03-03T05:45:53.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2023-03-03T05:45:53.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2023-03-03T05:45:53.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2023-03-03T05:45:53.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-03T05:45:53.994 DEBUG:teuthology.orchestra.run.smithi102:> ') 2023-03-03T05:45:54.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:54.510 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T05:45:54.659 INFO:teuthology.orchestra.run.smithi102.stdout:1 2023-03-03T05:45:54.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:54.664 DEBUG:teuthology.orchestra.run.smithi102:> route 2023-03-03T05:45:54.752 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2023-03-03T05:45:54.753 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T05:45:54.753 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T05:45:54.753 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T05:45:54.753 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T05:45:54.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:54.755 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2023-03-03T05:45:54.755 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2023-03-03T05:45:54.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T05:45:54.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T05:45:54.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T05:45:54.756 DEBUG:teuthology.orchestra.run.smithi102:> ') 2023-03-03T05:45:55.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:55.507 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2023-03-03T05:45:55.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:55.576 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2023-03-03T05:45:55.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:55.646 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2023-03-03T05:45:55.646 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2023-03-03T05:45:55.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T05:45:55.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-03-03T05:45:55.647 DEBUG:teuthology.orchestra.run.smithi102:> ') 2023-03-03T05:45:55.850 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-03-03T05:45:55.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:55.851 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2023-03-03T05:45:55.851 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2023-03-03T05:45:55.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-03-03T05:45:55.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-03T05:45:55.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-03-03T05:45:55.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-03-03T05:45:55.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-03-03T05:45:55.853 DEBUG:teuthology.orchestra.run.smithi102:> ') 2023-03-03T05:45:56.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:56.223 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2023-03-03T05:45:56.223 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2023-03-03T05:45:56.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2023-03-03T05:45:56.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2023-03-03T05:45:56.224 DEBUG:teuthology.orchestra.run.smithi102:> ') 2023-03-03T05:45:56.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:45:56.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:45:56.463 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T05:45:56.464 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T05:45:56.533 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T05:45:56.534 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T05:45:56.534 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T05:45:56.641 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-03-03T05:45:56.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:56.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-03-03T05:45:56.787 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-03-03T05:45:56.787 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-03-03T05:45:56.787 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2023-03-03T05:45:57.288 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-03-03T05:45:57.288 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-03-03T05:45:57.289 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2023-03-03T05:45:57.289 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-03-03T05:45:57.289 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T05:45:57.318 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-03-03T05:45:57.318 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-03-03T05:45:57.318 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2023-03-03T05:45:57.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.172:3300,172.21.15.172:3301" 2023-03-03T05:45:57.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T05:45:57.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.172:6789,172.21.15.172:6790" 2023-03-03T05:45:57.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.172:6789,172.21.15.172:6790:/ 2023-03-03T05:45:57.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=56ef27ff-a10a-47b1-b557-af27e3b0b6e2" will pass to kernel 2023-03-03T05:45:57.330 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-03-03T05:45:57.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:57.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T05:45:57.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:57.432 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T05:45:57.580 INFO:teuthology.orchestra.run.smithi102.stdout:client4523 2023-03-03T05:45:57.581 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:45:57.581 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T05:45:57.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:57.649 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T05:45:57.792 INFO:teuthology.orchestra.run.smithi102.stdout:client4523 2023-03-03T05:45:57.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:57.793 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T05:45:57.935 INFO:teuthology.orchestra.run.smithi102.stdout:client4523 2023-03-03T05:45:57.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T05:45:57.937 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/56ef27ff-a10a-47b1-b557-af27e3b0b6e2.client4523/status') 2023-03-03T05:45:58.107 INFO:teuthology.orchestra.run.smithi102.stdout:instance: client.4523 (3)172.21.15.102:0/1525500283 2023-03-03T05:45:58.107 INFO:teuthology.orchestra.run.smithi102.stdout:blocklisted: false 2023-03-03T05:45:58.107 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records in 2023-03-03T05:45:58.108 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records out 2023-03-03T05:45:58.108 INFO:teuthology.orchestra.run.smithi102.stderr:71 bytes copied, 9.6503e-05 s, 736 kB/s 2023-03-03T05:45:58.112 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-03-03T05:45:58.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:45:58.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.384+0000 7fb971bff700 1 Processor -- start 2023-03-03T05:45:58.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.384+0000 7fb971bff700 1 -- start start 2023-03-03T05:45:58.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.385+0000 7fb971bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0ced80 0x7fb96c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.385+0000 7fb971bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb96c1597f0 0x7fb96c0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb971bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0cb9f0 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb971bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96c0ca750 con 0x7fb96c1597f0 2023-03-03T05:45:58.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb971bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c15a230 con 0x7fb96c0ced80 2023-03-03T05:45:58.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb96c1597f0 0x7fb96c0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb96c1597f0 0x7fb96c0cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42644/0 (socket says 172.21.15.102:42644) 2023-03-03T05:45:58.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb96c0cf9a0 0x7fb96c15c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38776/0 (socket says 172.21.15.102:38776) 2023-03-03T05:45:58.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96affd700 1 -- 172.21.15.102:0/1494131523 learned_addr learned my addr 172.21.15.102:0/1494131523 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:58.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1494131523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0ced80 0x7fb96c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96affd700 1 -- 172.21.15.102:0/1494131523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0ced80 msgr2=0x7fb96c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:58.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96affd700 1 --2- 172.21.15.102:0/1494131523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0ced80 0x7fb96c0cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96affd700 1 -- 172.21.15.102:0/1494131523 >> v1:172.21.15.102:6789/0 conn(0x7fb96c0cf9a0 legacy=0x7fb96c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.386+0000 7fb96affd700 1 -- 172.21.15.102:0/1494131523 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c159c80 con 0x7fb96c1597f0 2023-03-03T05:45:58.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.387+0000 7fb96affd700 1 --2- 172.21.15.102:0/1494131523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c0cbe80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb9600098f0 tx=0x7fb960009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4291bef9d3067b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:58.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.387+0000 7fb96a7fc700 1 -- 172.21.15.102:0/1494131523 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960011830 con 0x7fb96c1597f0 2023-03-03T05:45:58.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.387+0000 7fb96a7fc700 1 -- 172.21.15.102:0/1494131523 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb960011990 con 0x7fb96c1597f0 2023-03-03T05:45:58.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.387+0000 7fb96a7fc700 1 -- 172.21.15.102:0/1494131523 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960011d20 con 0x7fb96c1597f0 2023-03-03T05:45:58.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 -- 172.21.15.102:0/1494131523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 msgr2=0x7fb96c0cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 --2- 172.21.15.102:0/1494131523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c0cbe80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb9600098f0 tx=0x7fb960009c00 comp rx=0 tx=0).stop 2023-03-03T05:45:58.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 -- 172.21.15.102:0/1494131523 shutdown_connections 2023-03-03T05:45:58.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 --2- 172.21.15.102:0/1494131523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0ced80 0x7fb96c0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 --2- 172.21.15.102:0/1494131523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c0cbe80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 -- 172.21.15.102:0/1494131523 >> 172.21.15.102:0/1494131523 conn(0x7fb96c065fb0 msgr2=0x7fb96c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:58.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 -- 172.21.15.102:0/1494131523 shutdown_connections 2023-03-03T05:45:58.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.388+0000 7fb971bff700 1 -- 172.21.15.102:0/1494131523 wait complete. 2023-03-03T05:45:58.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.390+0000 7fb971bff700 1 Processor -- start 2023-03-03T05:45:58.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.390+0000 7fb971bff700 1 -- start start 2023-03-03T05:45:58.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.391+0000 7fb971bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb96c0ced80 0x7fb96c14a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.391+0000 7fb971bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0cf9a0 0x7fb96c151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.391+0000 7fb971bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c1517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.391+0000 7fb971bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c14ae50 con 0x7fb96c1597f0 2023-03-03T05:45:58.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.391+0000 7fb971bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c158610 con 0x7fb96c0ced80 2023-03-03T05:45:58.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.391+0000 7fb971bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c158bd0 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.391+0000 7fb96affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0cf9a0 0x7fb96c151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb96c0ced80 0x7fb96c14a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c1517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0cf9a0 0x7fb96c151290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37670/0 (socket says 172.21.15.102:37670) 2023-03-03T05:45:58.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 -- 172.21.15.102:0/2801475668 learned_addr learned my addr 172.21.15.102:0/2801475668 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:58.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 -- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb96c0ced80 msgr2=0x7fb96c14a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb96c0ced80 0x7fb96c14a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 -- 172.21.15.102:0/2801475668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 msgr2=0x7fb96c1517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c1517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 -- 172.21.15.102:0/2801475668 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb960009550 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96bfff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:58.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.392+0000 7fb96affd700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0cf9a0 0x7fb96c151290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb960014040 tx=0x7fb960004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:58.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.393+0000 7fb953fff700 1 -- 172.21.15.102:0/2801475668 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960004510 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.393+0000 7fb953fff700 1 -- 172.21.15.102:0/2801475668 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96001e070 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.393+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c0cb0c0 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.393+0000 7fb953fff700 1 -- 172.21.15.102:0/2801475668 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9600038e0 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.393+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c0cade0 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.394+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c0cb3a0 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.395+0000 7fb953fff700 1 -- 172.21.15.102:0/2801475668 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb96000f050 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.396+0000 7fb953fff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb958068320 0x7fb95806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.396+0000 7fb953fff700 1 -- 172.21.15.102:0/2801475668 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb960086b70 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.396+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb958068320 0x7fb95806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.397+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb958068320 0x7fb95806a7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb95c009b40 tx=0x7fb95c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:58.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.401+0000 7fb953fff700 1 -- 172.21.15.102:0/2801475668 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb960053260 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.601+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb96c0677f0 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.603+0000 7fb953fff700 1 -- 172.21.15.102:0/2801475668 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7fb960020090 con 0x7fb96c0cf9a0 2023-03-03T05:45:58.605 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:45:58.605 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:45:58.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.605+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb958068320 msgr2=0x7fb95806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.605+0000 7fb971bff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb958068320 0x7fb95806a7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb95c009b40 tx=0x7fb95c009210 comp rx=0 tx=0).stop 2023-03-03T05:45:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.605+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0cf9a0 msgr2=0x7fb96c151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.605+0000 7fb971bff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0cf9a0 0x7fb96c151290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb960014040 tx=0x7fb960004750 comp rx=0 tx=0).stop 2023-03-03T05:45:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 shutdown_connections 2023-03-03T05:45:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c1597f0 0x7fb96c1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb96c0ced80 0x7fb96c14a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb958068320 0x7fb95806a7d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 --2- 172.21.15.102:0/2801475668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb96c0cf9a0 0x7fb96c151290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 >> 172.21.15.102:0/2801475668 conn(0x7fb96c065fb0 msgr2=0x7fb96c0cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:58.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 shutdown_connections 2023-03-03T05:45:58.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.606+0000 7fb971bff700 1 -- 172.21.15.102:0/2801475668 wait complete. 2023-03-03T05:45:58.637 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T05:45:20.032740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T05:45:27.930463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T05:45:33.613145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T05:45:34.842860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T05:45:58.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T05:45:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.885+0000 7f6dfa7ff700 1 Processor -- start 2023-03-03T05:45:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.886+0000 7f6dfa7ff700 1 -- start start 2023-03-03T05:45:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6dfa7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df4153450 0x7f6df40cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6dfa7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6df415a380 0x7f6df4159940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6dfa7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df4159e80 con 0x7f6df40cf9a0 2023-03-03T05:45:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6dfa7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df40cae70 con 0x7f6df415a380 2023-03-03T05:45:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6dfa7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40cb110 con 0x7f6df4153450 2023-03-03T05:45:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6debfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6df415a380 0x7f6df4159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6df8d9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6df40cf9a0 0x7f6df40ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33826/0 (socket says 172.21.15.102:33826) 2023-03-03T05:45:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6df3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df4153450 0x7f6df40cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.887+0000 7f6df8d9c700 1 -- 172.21.15.102:0/4077557070 learned_addr learned my addr 172.21.15.102:0/4077557070 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:58.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6df37fe700 1 -- 172.21.15.102:0/4077557070 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523479980 0 0) 0x7f6df4159e80 con 0x7f6df40cf9a0 2023-03-03T05:45:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6df37fe700 1 -- 172.21.15.102:0/4077557070 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de00035b0 con 0x7f6df40cf9a0 2023-03-03T05:45:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6debfff700 1 -- 172.21.15.102:0/4077557070 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df4153450 msgr2=0x7f6df40cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:45:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6debfff700 1 --2- 172.21.15.102:0/4077557070 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df4153450 0x7f6df40cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6debfff700 1 -- 172.21.15.102:0/4077557070 >> v1:172.21.15.172:6789/0 conn(0x7f6df40cf9a0 legacy=0x7f6df40ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6debfff700 1 -- 172.21.15.102:0/4077557070 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df40cb3f0 con 0x7f6df415a380 2023-03-03T05:45:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6debfff700 1 --2- 172.21.15.102:0/4077557070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df415a380 0x7f6df4159940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009840 tx=0x7f6ddc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02b146af7e969d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.888+0000 7f6df37fe700 1 -- 172.21.15.102:0/4077557070 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc011830 con 0x7f6df415a380 2023-03-03T05:45:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6df37fe700 1 -- 172.21.15.102:0/4077557070 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ddc011990 con 0x7f6df415a380 2023-03-03T05:45:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6df37fe700 1 -- 172.21.15.102:0/4077557070 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc011d20 con 0x7f6df415a380 2023-03-03T05:45:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/4077557070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df415a380 msgr2=0x7f6df4159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/4077557070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df415a380 0x7f6df4159940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009840 tx=0x7f6ddc009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/4077557070 shutdown_connections 2023-03-03T05:45:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/4077557070 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df4153450 0x7f6df40cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/4077557070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df415a380 0x7f6df4159940 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/4077557070 >> 172.21.15.102:0/4077557070 conn(0x7f6df4065fb0 msgr2=0x7f6df40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:58.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/4077557070 shutdown_connections 2023-03-03T05:45:58.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.889+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/4077557070 wait complete. 2023-03-03T05:45:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.891+0000 7f6dfa7ff700 1 Processor -- start 2023-03-03T05:45:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.892+0000 7f6dfa7ff700 1 -- start start 2023-03-03T05:45:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6dfa7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df40cf9a0 0x7f6df414df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6dfa7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df4153450 0x7f6df41e2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6dfa7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6df415a380 0x7f6df41f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6dfa7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40b4b10 con 0x7f6df4153450 2023-03-03T05:45:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6dfa7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40cd8c0 con 0x7f6df415a380 2023-03-03T05:45:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6dfa7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40b41b0 con 0x7f6df40cf9a0 2023-03-03T05:45:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df40cf9a0 0x7f6df414df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df40cf9a0 0x7f6df414df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37688/0 (socket says 172.21.15.102:37688) 2023-03-03T05:45:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 -- 172.21.15.102:0/2865445652 learned_addr learned my addr 172.21.15.102:0/2865445652 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6debfff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df4153450 0x7f6df41e2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df8d9c700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6df415a380 0x7f6df41f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 -- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6df415a380 msgr2=0x7f6df41f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6df415a380 0x7f6df41f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 -- 172.21.15.102:0/2865445652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df4153450 msgr2=0x7f6df41e2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df4153450 0x7f6df41e2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df3fff700 1 -- 172.21.15.102:0/2865445652 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc0094a0 con 0x7f6df40cf9a0 2023-03-03T05:45:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.893+0000 7f6df8d9c700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6df415a380 0x7f6df41f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T05:45:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.894+0000 7f6df3fff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df40cf9a0 0x7f6df414df20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6de400e8a0 tx=0x7f6de400ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.894+0000 7f6df17fa700 1 -- 172.21.15.102:0/2865445652 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4009590 con 0x7f6df40cf9a0 2023-03-03T05:45:58.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.894+0000 7f6df17fa700 1 -- 172.21.15.102:0/2865445652 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de4013680 con 0x7f6df40cf9a0 2023-03-03T05:45:58.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.894+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df40cb9b0 con 0x7f6df40cf9a0 2023-03-03T05:45:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.894+0000 7f6df17fa700 1 -- 172.21.15.102:0/2865445652 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de40139f0 con 0x7f6df40cf9a0 2023-03-03T05:45:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.894+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df40cb6d0 con 0x7f6df40cf9a0 2023-03-03T05:45:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.895+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df40cbc90 con 0x7f6df40cf9a0 2023-03-03T05:45:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.897+0000 7f6df17fa700 1 -- 172.21.15.102:0/2865445652 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6de4023070 con 0x7f6df40cf9a0 2023-03-03T05:45:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.898+0000 7f6df17fa700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6dd40682f0 0x7f6dd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.898+0000 7f6df17fa700 1 -- 172.21.15.102:0/2865445652 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6de4085eb0 con 0x7f6df40cf9a0 2023-03-03T05:45:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.898+0000 7f6debfff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6dd40682f0 0x7f6dd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.899+0000 7f6debfff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6dd40682f0 0x7f6dd406a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc000c00 tx=0x7f6ddc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:58.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:58.902+0000 7f6df17fa700 1 -- 172.21.15.102:0/2865445652 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6de40525c0 con 0x7f6df40cf9a0 2023-03-03T05:45:59.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.093+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6df40cc6c0 con 0x7f6df40cf9a0 2023-03-03T05:45:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.094+0000 7f6df17fa700 1 -- 172.21.15.102:0/2865445652 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6de4055be0 con 0x7f6df40cf9a0 2023-03-03T05:45:59.096 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2023-03-03T05:45:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6dd40682f0 msgr2=0x7f6dd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6dd40682f0 0x7f6dd406a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc000c00 tx=0x7f6ddc008030 comp rx=0 tx=0).stop 2023-03-03T05:45:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df40cf9a0 msgr2=0x7f6df414df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df40cf9a0 0x7f6df414df20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6de400e8a0 tx=0x7f6de400ebb0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 shutdown_connections 2023-03-03T05:45:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6dd40682f0 0x7f6dd406a7a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6df40cf9a0 0x7f6df414df20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6df415a380 0x7f6df41f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 --2- 172.21.15.102:0/2865445652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6df4153450 0x7f6df41e2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 >> 172.21.15.102:0/2865445652 conn(0x7f6df4065fb0 msgr2=0x7f6df4159360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 shutdown_connections 2023-03-03T05:45:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.097+0000 7f6dfa7ff700 1 -- 172.21.15.102:0/2865445652 wait complete. 2023-03-03T05:45:59.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T05:45:59.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.375+0000 7f8e435ff700 1 Processor -- start 2023-03-03T05:45:59.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.375+0000 7f8e435ff700 1 -- start start 2023-03-03T05:45:59.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.377+0000 7f8e435ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8e3c153450 0x7f8e3c0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.377+0000 7f8e435ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c0cae70 con 0x7f8e3c15a380 2023-03-03T05:45:59.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.377+0000 7f8e435ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c0cb110 con 0x7f8e3c0cf9a0 2023-03-03T05:45:59.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.377+0000 7f8e435ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3c0cb3f0 con 0x7f8e3c153450 2023-03-03T05:45:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.377+0000 7f8e4139b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8e3c153450 0x7f8e3c0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.377+0000 7f8e41b9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8e3c0cf9a0 0x7f8e3c0ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33840/0 (socket says 172.21.15.102:33840) 2023-03-03T05:45:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.377+0000 7f8e41b9c700 1 -- 172.21.15.102:0/1314299009 learned_addr learned my addr 172.21.15.102:0/1314299009 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:59.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.378+0000 7f8e33fff700 1 -- 172.21.15.102:0/1314299009 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961763425 0 0) 0x7f8e3c0cb110 con 0x7f8e3c0cf9a0 2023-03-03T05:45:59.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.378+0000 7f8e33fff700 1 -- 172.21.15.102:0/1314299009 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e280035b0 con 0x7f8e3c0cf9a0 2023-03-03T05:45:59.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.378+0000 7f8e4139b700 1 -- 172.21.15.102:0/1314299009 >> v1:172.21.15.172:6789/0 conn(0x7f8e3c0cf9a0 legacy=0x7f8e3c0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.378+0000 7f8e4139b700 1 -- 172.21.15.102:0/1314299009 >> v1:172.21.15.102:6789/0 conn(0x7f8e3c15a380 legacy=0x7f8e3c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.378+0000 7f8e33fff700 1 -- 172.21.15.102:0/1314299009 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555898879 0 0) 0x7f8e3c0cae70 con 0x7f8e3c15a380 2023-03-03T05:45:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.378+0000 7f8e4139b700 1 -- 172.21.15.102:0/1314299009 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c0cb6d0 con 0x7f8e3c153450 2023-03-03T05:45:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.378+0000 7f8e4139b700 1 --2- 172.21.15.102:0/1314299009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0cc1b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009840 tx=0x7f8e38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a6a042d22ead4e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e33fff700 1 -- 172.21.15.102:0/1314299009 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38011830 con 0x7f8e3c153450 2023-03-03T05:45:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e33fff700 1 -- 172.21.15.102:0/1314299009 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38011990 con 0x7f8e3c153450 2023-03-03T05:45:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e33fff700 1 -- 172.21.15.102:0/1314299009 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38011d20 con 0x7f8e3c153450 2023-03-03T05:45:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e435ff700 1 -- 172.21.15.102:0/1314299009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 msgr2=0x7f8e3c0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e435ff700 1 --2- 172.21.15.102:0/1314299009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0cc1b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009840 tx=0x7f8e38009b50 comp rx=0 tx=0).stop 2023-03-03T05:45:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e435ff700 1 -- 172.21.15.102:0/1314299009 shutdown_connections 2023-03-03T05:45:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e435ff700 1 --2- 172.21.15.102:0/1314299009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0cc1b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e435ff700 1 -- 172.21.15.102:0/1314299009 >> 172.21.15.102:0/1314299009 conn(0x7f8e3c065fb0 msgr2=0x7f8e3c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e435ff700 1 -- 172.21.15.102:0/1314299009 shutdown_connections 2023-03-03T05:45:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.379+0000 7f8e435ff700 1 -- 172.21.15.102:0/1314299009 wait complete. 2023-03-03T05:45:59.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.381+0000 7f8e435ff700 1 Processor -- start 2023-03-03T05:45:59.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.382+0000 7f8e435ff700 1 -- start start 2023-03-03T05:45:59.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.382+0000 7f8e435ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e3c0cf9a0 0x7f8e3c0c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e435ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e435ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e3c15a380 0x7f8e3c1e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e435ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c1e5270 con 0x7f8e3c153450 2023-03-03T05:45:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e435ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c0c7660 con 0x7f8e3c15a380 2023-03-03T05:45:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e435ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c1e4a60 con 0x7f8e3c0cf9a0 2023-03-03T05:45:59.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e4139b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e3c0cf9a0 0x7f8e3c0c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42694/0 (socket says 172.21.15.102:42694) 2023-03-03T05:45:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 -- 172.21.15.102:0/4008966765 learned_addr learned my addr 172.21.15.102:0/4008966765 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e41b9c700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e3c15a380 0x7f8e3c1e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 -- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e3c0cf9a0 msgr2=0x7f8e3c0c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e3c0cf9a0 0x7f8e3c0c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 -- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e3c15a380 msgr2=0x7f8e3c1e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e3c15a380 0x7f8e3c1e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e40b9a700 1 -- 172.21.15.102:0/4008966765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e380094a0 con 0x7f8e3c153450 2023-03-03T05:45:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.383+0000 7f8e41b9c700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e3c15a380 0x7f8e3c1e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:45:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.384+0000 7f8e40b9a700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0c7120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00c750 tx=0x7f8e2c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.384+0000 7f8e31ffb700 1 -- 172.21.15.102:0/4008966765 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0054e0 con 0x7f8e3c153450 2023-03-03T05:45:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.384+0000 7f8e31ffb700 1 -- 172.21.15.102:0/4008966765 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c003680 con 0x7f8e3c153450 2023-03-03T05:45:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.384+0000 7f8e31ffb700 1 -- 172.21.15.102:0/4008966765 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c003a30 con 0x7f8e3c153450 2023-03-03T05:45:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.384+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3c0cbc90 con 0x7f8e3c153450 2023-03-03T05:45:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.384+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3c0cb9b0 con 0x7f8e3c153450 2023-03-03T05:45:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.385+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3c0cf490 con 0x7f8e3c153450 2023-03-03T05:45:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.386+0000 7f8e31ffb700 1 -- 172.21.15.102:0/4008966765 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e2c019030 con 0x7f8e3c153450 2023-03-03T05:45:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.388+0000 7f8e31ffb700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e20068320 0x7f8e2006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.388+0000 7f8e31ffb700 1 -- 172.21.15.102:0/4008966765 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8e2c085060 con 0x7f8e3c153450 2023-03-03T05:45:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.388+0000 7f8e4139b700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e20068320 0x7f8e2006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.389+0000 7f8e4139b700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e20068320 0x7f8e2006a7d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009fd0 tx=0x7f8e3801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:59.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.392+0000 7f8e31ffb700 1 -- 172.21.15.102:0/4008966765 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e2c051890 con 0x7f8e3c153450 2023-03-03T05:45:59.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.583+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8e3c0677f0 con 0x7f8e3c153450 2023-03-03T05:45:59.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.583+0000 7f8e31ffb700 1 -- 172.21.15.102:0/4008966765 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8e2c0170a0 con 0x7f8e3c153450 2023-03-03T05:45:59.586 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2023-03-03T05:45:59.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e20068320 msgr2=0x7f8e2006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e20068320 0x7f8e2006a7d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009fd0 tx=0x7f8e3801c040 comp rx=0 tx=0).stop 2023-03-03T05:45:59.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 msgr2=0x7f8e3c0c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0c7120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00c750 tx=0x7f8e2c00ca60 comp rx=0 tx=0).stop 2023-03-03T05:45:59.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 shutdown_connections 2023-03-03T05:45:59.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e20068320 0x7f8e2006a7d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e3c0cf9a0 0x7f8e3c0c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e3c15a380 0x7f8e3c1e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 --2- 172.21.15.102:0/4008966765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e3c153450 0x7f8e3c0c7120 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 >> 172.21.15.102:0/4008966765 conn(0x7f8e3c065fb0 msgr2=0x7f8e3c066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:59.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 shutdown_connections 2023-03-03T05:45:59.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.586+0000 7f8e435ff700 1 -- 172.21.15.102:0/4008966765 wait complete. 2023-03-03T05:45:59.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T05:45:59.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.861+0000 7fc4b7fff700 1 Processor -- start 2023-03-03T05:45:59.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.861+0000 7fc4b7fff700 1 -- start start 2023-03-03T05:45:59.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.863+0000 7fc4b7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc4b80c4390 0x7fc4b80c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.863+0000 7fc4b7fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b8159140 con 0x7fc4b8158cb0 2023-03-03T05:45:59.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.863+0000 7fc4b7fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b8159420 con 0x7fc4b8153180 2023-03-03T05:45:59.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.863+0000 7fc4b7fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b8159cc0 con 0x7fc4b80c4390 2023-03-03T05:45:59.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.863+0000 7fc4b6ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc4b8153180 0x7fc4b80c9640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33846/0 (socket says 172.21.15.102:33846) 2023-03-03T05:45:59.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.863+0000 7fc4b6ffd700 1 -- 172.21.15.102:0/3676305855 learned_addr learned my addr 172.21.15.102:0/3676305855 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:45:59.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.864+0000 7fc4b77fe700 1 --2- 172.21.15.102:0/3676305855 >> v2:172.21.15.102:3300/0 conn(0x7fc4b80c4390 0x7fc4b80c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.864+0000 7fc4b77fe700 1 -- 172.21.15.102:0/3676305855 >> v1:172.21.15.172:6789/0 conn(0x7fc4b8153180 legacy=0x7fc4b80c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.864+0000 7fc4b77fe700 1 -- 172.21.15.102:0/3676305855 >> v1:172.21.15.102:6789/0 conn(0x7fc4b8158cb0 legacy=0x7fc4b815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.864+0000 7fc4b5ffb700 1 -- 172.21.15.102:0/3676305855 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025511868 0 0) 0x7fc4b8159420 con 0x7fc4b8153180 2023-03-03T05:45:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.864+0000 7fc4b77fe700 1 -- 172.21.15.102:0/3676305855 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b8159700 con 0x7fc4b80c4390 2023-03-03T05:45:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.865+0000 7fc4b77fe700 1 --2- 172.21.15.102:0/3676305855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80c47d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009d90 tx=0x7fc4b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45654ff3e73404aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:59.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.865+0000 7fc4b5ffb700 1 -- 172.21.15.102:0/3676305855 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000e070 con 0x7fc4b80c4390 2023-03-03T05:45:59.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.865+0000 7fc4b5ffb700 1 -- 172.21.15.102:0/3676305855 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b000bad0 con 0x7fc4b80c4390 2023-03-03T05:45:59.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.865+0000 7fc4b5ffb700 1 -- 172.21.15.102:0/3676305855 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000be80 con 0x7fc4b80c4390 2023-03-03T05:45:59.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.866+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3676305855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 msgr2=0x7fc4b80c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.866+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3676305855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80c47d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009d90 tx=0x7fc4b000b330 comp rx=0 tx=0).stop 2023-03-03T05:45:59.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.866+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3676305855 shutdown_connections 2023-03-03T05:45:59.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.866+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3676305855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80c47d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.866+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3676305855 >> 172.21.15.102:0/3676305855 conn(0x7fc4b8071a30 msgr2=0x7fc4b8065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:45:59.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.866+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3676305855 shutdown_connections 2023-03-03T05:45:59.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.866+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3676305855 wait complete. 2023-03-03T05:45:59.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.868+0000 7fc4b7fff700 1 Processor -- start 2023-03-03T05:45:59.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.869+0000 7fc4b7fff700 1 -- start start 2023-03-03T05:45:59.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4b8153180 0x7fc4b814ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4b8158cb0 0x7fc4b814b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b7fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b814ea80 con 0x7fc4b80c4390 2023-03-03T05:45:59.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b81516a0 con 0x7fc4b8158cb0 2023-03-03T05:45:59.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b814f450 con 0x7fc4b8153180 2023-03-03T05:45:59.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42714/0 (socket says 172.21.15.102:42714) 2023-03-03T05:45:59.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b6ffd700 1 -- 172.21.15.102:0/3266670719 learned_addr learned my addr 172.21.15.102:0/3266670719 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:45:59.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b67fc700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4b8153180 0x7fc4b814ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.870+0000 7fc4b77fe700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4b8158cb0 0x7fc4b814b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b6ffd700 1 -- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4b8153180 msgr2=0x7fc4b814ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b6ffd700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4b8153180 0x7fc4b814ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b6ffd700 1 -- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4b8158cb0 msgr2=0x7fc4b814b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:45:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b6ffd700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4b8158cb0 0x7fc4b814b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:45:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b6ffd700 1 -- 172.21.15.102:0/3266670719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b0009a70 con 0x7fc4b80c4390 2023-03-03T05:45:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b6ffd700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80b42e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80011e0 tx=0x7fc4a800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc49f7fe700 1 -- 172.21.15.102:0/3266670719 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004c80 con 0x7fc4b80c4390 2023-03-03T05:45:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc49f7fe700 1 -- 172.21.15.102:0/3266670719 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a8004e00 con 0x7fc4b80c4390 2023-03-03T05:45:59.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc49f7fe700 1 -- 172.21.15.102:0/3266670719 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8007310 con 0x7fc4b80c4390 2023-03-03T05:45:59.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b815c3c0 con 0x7fc4b80c4390 2023-03-03T05:45:59.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.871+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b81599e0 con 0x7fc4b80c4390 2023-03-03T05:45:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.872+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b80c5050 con 0x7fc4b80c4390 2023-03-03T05:45:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.874+0000 7fc49f7fe700 1 -- 172.21.15.102:0/3266670719 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc4a801a020 con 0x7fc4b80c4390 2023-03-03T05:45:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.875+0000 7fc49f7fe700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4a40682f0 0x7fc4a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:45:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.875+0000 7fc49f7fe700 1 -- 172.21.15.102:0/3266670719 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc4a8085810 con 0x7fc4b80c4390 2023-03-03T05:45:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.875+0000 7fc4b67fc700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4a40682f0 0x7fc4a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:45:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.876+0000 7fc4b67fc700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4a40682f0 0x7fc4a406a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc4b814c5e0 tx=0x7fc4ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:45:59.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:45:59.878+0000 7fc49f7fe700 1 -- 172.21.15.102:0/3266670719 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4a8051f80 con 0x7fc4b80c4390 2023-03-03T05:46:00.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.071+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4b80c9b70 con 0x7fc4b80c4390 2023-03-03T05:46:00.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.071+0000 7fc49f7fe700 1 -- 172.21.15.102:0/3266670719 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4b80c9b70 con 0x7fc4b80c4390 2023-03-03T05:46:00.074 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2023-03-03T05:46:00.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4a40682f0 msgr2=0x7fc4a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4a40682f0 0x7fc4a406a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc4b814c5e0 tx=0x7fc4ac009040 comp rx=0 tx=0).stop 2023-03-03T05:46:00.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 msgr2=0x7fc4b80b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80b42e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80011e0 tx=0x7fc4a800ca30 comp rx=0 tx=0).stop 2023-03-03T05:46:00.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 shutdown_connections 2023-03-03T05:46:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc4b8158cb0 0x7fc4b814b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4b80c4390 0x7fc4b80b42e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc4a40682f0 0x7fc4a406a7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 --2- 172.21.15.102:0/3266670719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc4b8153180 0x7fc4b814ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 >> 172.21.15.102:0/3266670719 conn(0x7fc4b8071a30 msgr2=0x7fc4b806bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:00.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 shutdown_connections 2023-03-03T05:46:00.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.074+0000 7fc4b7fff700 1 -- 172.21.15.102:0/3266670719 wait complete. 2023-03-03T05:46:00.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T05:46:00.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.349+0000 7feac27ff700 1 Processor -- start 2023-03-03T05:46:00.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.349+0000 7feac27ff700 1 -- start start 2023-03-03T05:46:00.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feac27ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feabc070960 0x7feabc155230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feac27ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feabc070d30 0x7feabc071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feac27ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0c72a0 con 0x7feabc0c7cf0 2023-03-03T05:46:00.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feac27ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feabc0c2710 con 0x7feabc070960 2023-03-03T05:46:00.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feac27ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feabc0c7860 con 0x7feabc070d30 2023-03-03T05:46:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feabbfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feabc070960 0x7feabc155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feac0d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7feabc0c7cf0 0x7feabc14f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38802/0 (socket says 172.21.15.102:38802) 2023-03-03T05:46:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feac0d9c700 1 -- 172.21.15.102:0/2029544331 learned_addr learned my addr 172.21.15.102:0/2029544331 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feabbfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feabc070960 0x7feabc155230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42726/0 (socket says 172.21.15.102:42726) 2023-03-03T05:46:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.351+0000 7feabb7fe700 1 --2- 172.21.15.102:0/2029544331 >> v2:172.21.15.172:3300/0 conn(0x7feabc070d30 0x7feabc071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.352+0000 7feabbfff700 1 -- 172.21.15.102:0/2029544331 >> v2:172.21.15.172:3300/0 conn(0x7feabc070d30 msgr2=0x7feabc071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.352+0000 7feabbfff700 1 --2- 172.21.15.102:0/2029544331 >> v2:172.21.15.172:3300/0 conn(0x7feabc070d30 0x7feabc071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.352+0000 7feabbfff700 1 -- 172.21.15.102:0/2029544331 >> v1:172.21.15.102:6789/0 conn(0x7feabc0c7cf0 legacy=0x7feabc14f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.352+0000 7feabbfff700 1 -- 172.21.15.102:0/2029544331 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc071f00 con 0x7feabc070960 2023-03-03T05:46:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.352+0000 7feabbfff700 1 --2- 172.21.15.102:0/2029544331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc155230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feabc06d830 tx=0x7feaac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcada72666bed9f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.352+0000 7feabaffd700 1 -- 172.21.15.102:0/2029544331 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac011830 con 0x7feabc070960 2023-03-03T05:46:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feabaffd700 1 -- 172.21.15.102:0/2029544331 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac011990 con 0x7feabc070960 2023-03-03T05:46:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feabaffd700 1 -- 172.21.15.102:0/2029544331 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac011d20 con 0x7feabc070960 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 -- 172.21.15.102:0/2029544331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 msgr2=0x7feabc155230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 --2- 172.21.15.102:0/2029544331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc155230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feabc06d830 tx=0x7feaac009b20 comp rx=0 tx=0).stop 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 -- 172.21.15.102:0/2029544331 shutdown_connections 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 --2- 172.21.15.102:0/2029544331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc155230 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 --2- 172.21.15.102:0/2029544331 >> v2:172.21.15.172:3300/0 conn(0x7feabc070d30 0x7feabc071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 -- 172.21.15.102:0/2029544331 >> 172.21.15.102:0/2029544331 conn(0x7feabc06af90 msgr2=0x7feabc06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 -- 172.21.15.102:0/2029544331 shutdown_connections 2023-03-03T05:46:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.353+0000 7feac27ff700 1 -- 172.21.15.102:0/2029544331 wait complete. 2023-03-03T05:46:00.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.355+0000 7feac27ff700 1 Processor -- start 2023-03-03T05:46:00.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.355+0000 7feac27ff700 1 -- start start 2023-03-03T05:46:00.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.356+0000 7feac27ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc1deaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.356+0000 7feac27ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feabc070d30 0x7feabc1481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.356+0000 7feac27ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feabc0c7cf0 0x7feabc1466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.356+0000 7feac27ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc14aa30 con 0x7feabc070960 2023-03-03T05:46:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.356+0000 7feac27ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc148ab0 con 0x7feabc070d30 2023-03-03T05:46:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.356+0000 7feac27ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc149cd0 con 0x7feabc0c7cf0 2023-03-03T05:46:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc1deaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc1deaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42734/0 (socket says 172.21.15.102:42734) 2023-03-03T05:46:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 -- 172.21.15.102:0/2248736 learned_addr learned my addr 172.21.15.102:0/2248736 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabb7fe700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feabc070d30 0x7feabc1481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feac0d9c700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feabc0c7cf0 0x7feabc1466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 -- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feabc0c7cf0 msgr2=0x7feabc1466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feabc0c7cf0 0x7feabc1466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 -- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feabc070d30 msgr2=0x7feabc1481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feabc070d30 0x7feabc1481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feabbfff700 1 -- 172.21.15.102:0/2248736 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0094a0 con 0x7feabc070960 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.357+0000 7feac0d9c700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feabc0c7cf0 0x7feabc1466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T05:46:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.358+0000 7feabbfff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc1deaf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feaac009b60 tx=0x7feaac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:00.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.358+0000 7feab8ff9700 1 -- 172.21.15.102:0/2248736 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac0036a0 con 0x7feabc070960 2023-03-03T05:46:00.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.358+0000 7feab8ff9700 1 -- 172.21.15.102:0/2248736 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac01e070 con 0x7feabc070960 2023-03-03T05:46:00.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.358+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc0724c0 con 0x7feabc070960 2023-03-03T05:46:00.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.358+0000 7feab8ff9700 1 -- 172.21.15.102:0/2248736 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac003a60 con 0x7feabc070960 2023-03-03T05:46:00.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.358+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc0721e0 con 0x7feabc070960 2023-03-03T05:46:00.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.359+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc0c74f0 con 0x7feabc070960 2023-03-03T05:46:00.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.360+0000 7feab8ff9700 1 -- 172.21.15.102:0/2248736 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feaac00f050 con 0x7feabc070960 2023-03-03T05:46:00.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.361+0000 7feab8ff9700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feaa80682f0 0x7feaa806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.361+0000 7feab8ff9700 1 -- 172.21.15.102:0/2248736 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7feaac086c90 con 0x7feabc070960 2023-03-03T05:46:00.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.362+0000 7feabb7fe700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feaa80682f0 0x7feaa806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.362+0000 7feabb7fe700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feaa80682f0 0x7feaa806a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feabc0c1620 tx=0x7feab000a320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:00.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.365+0000 7feab8ff9700 1 -- 172.21.15.102:0/2248736 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feaac053400 con 0x7feabc070960 2023-03-03T05:46:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.556+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feabc04e6e0 con 0x7feabc070960 2023-03-03T05:46:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.556+0000 7feab8ff9700 1 -- 172.21.15.102:0/2248736 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7feaac0150a0 con 0x7feabc070960 2023-03-03T05:46:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2023-03-03T05:46:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feaa80682f0 msgr2=0x7feaa806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feaa80682f0 0x7feaa806a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feabc0c1620 tx=0x7feab000a320 comp rx=0 tx=0).stop 2023-03-03T05:46:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 msgr2=0x7feabc1deaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc1deaf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feaac009b60 tx=0x7feaac004750 comp rx=0 tx=0).stop 2023-03-03T05:46:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 shutdown_connections 2023-03-03T05:46:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feaa80682f0 0x7feaa806a7a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feabc0c7cf0 0x7feabc1466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feabc070960 0x7feabc1deaf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 --2- 172.21.15.102:0/2248736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feabc070d30 0x7feabc1481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 >> 172.21.15.102:0/2248736 conn(0x7feabc06af90 msgr2=0x7feabc0c4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 shutdown_connections 2023-03-03T05:46:00.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.559+0000 7feac27ff700 1 -- 172.21.15.102:0/2248736 wait complete. 2023-03-03T05:46:00.588 INFO:tasks.mds_thrash:mds thrasher using random seed: 1677822360 2023-03-03T05:46:00.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:46:00.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.830+0000 7fd4707ff700 1 Processor -- start 2023-03-03T05:46:00.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.830+0000 7fd4707ff700 1 -- start start 2023-03-03T05:46:00.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.831+0000 7fd4707ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd4680ced80 0x7fd4680cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.831+0000 7fd4707ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd4680cf9a0 0x7fd468163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.831+0000 7fd4707ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4680cb9f0 con 0x7fd4681597f0 2023-03-03T05:46:00.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.831+0000 7fd4707ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4680ca750 con 0x7fd4680cf9a0 2023-03-03T05:46:00.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.831+0000 7fd4707ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46815a230 con 0x7fd4680ced80 2023-03-03T05:46:00.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46ed9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd4680cf9a0 0x7fd468163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46e59b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd4680ced80 0x7fd4680cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46dd9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd4681597f0 0x7fd4680cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38812/0 (socket says 172.21.15.102:38812) 2023-03-03T05:46:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46ed9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd4680cf9a0 0x7fd468163660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42744/0 (socket says 172.21.15.102:42744) 2023-03-03T05:46:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46dd9a700 1 -- 172.21.15.102:0/3966499020 learned_addr learned my addr 172.21.15.102:0/3966499020 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46ed9c700 1 -- 172.21.15.102:0/3966499020 >> v2:172.21.15.172:3300/0 conn(0x7fd4680ced80 msgr2=0x7fd4680cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46ed9c700 1 --2- 172.21.15.102:0/3966499020 >> v2:172.21.15.172:3300/0 conn(0x7fd4680ced80 0x7fd4680cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46ed9c700 1 -- 172.21.15.102:0/3966499020 >> v1:172.21.15.102:6789/0 conn(0x7fd4681597f0 legacy=0x7fd4680cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.832+0000 7fd46ed9c700 1 -- 172.21.15.102:0/3966499020 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd468159c80 con 0x7fd4680cf9a0 2023-03-03T05:46:00.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd46ed9c700 1 --2- 172.21.15.102:0/3966499020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 0x7fd468163660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4680699d0 tx=0x7fd46000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28cebb211584a7b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:00.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd46d599700 1 -- 172.21.15.102:0/3966499020 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46000e070 con 0x7fd4680cf9a0 2023-03-03T05:46:00.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd46d599700 1 -- 172.21.15.102:0/3966499020 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46000bb20 con 0x7fd4680cf9a0 2023-03-03T05:46:00.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd46d599700 1 -- 172.21.15.102:0/3966499020 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460012430 con 0x7fd4680cf9a0 2023-03-03T05:46:00.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd4707ff700 1 -- 172.21.15.102:0/3966499020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 msgr2=0x7fd468163660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd4707ff700 1 --2- 172.21.15.102:0/3966499020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 0x7fd468163660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4680699d0 tx=0x7fd46000b330 comp rx=0 tx=0).stop 2023-03-03T05:46:00.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd4707ff700 1 -- 172.21.15.102:0/3966499020 shutdown_connections 2023-03-03T05:46:00.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd4707ff700 1 --2- 172.21.15.102:0/3966499020 >> v2:172.21.15.172:3300/0 conn(0x7fd4680ced80 0x7fd4680cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd4707ff700 1 --2- 172.21.15.102:0/3966499020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 0x7fd468163660 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.833+0000 7fd4707ff700 1 -- 172.21.15.102:0/3966499020 >> 172.21.15.102:0/3966499020 conn(0x7fd468065fb0 msgr2=0x7fd4680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:00.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.834+0000 7fd4707ff700 1 -- 172.21.15.102:0/3966499020 shutdown_connections 2023-03-03T05:46:00.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.834+0000 7fd4707ff700 1 -- 172.21.15.102:0/3966499020 wait complete. 2023-03-03T05:46:00.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.836+0000 7fd4707ff700 1 Processor -- start 2023-03-03T05:46:00.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.836+0000 7fd4707ff700 1 -- start start 2023-03-03T05:46:00.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd4707ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4680ced80 0x7fd4681e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd4707ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 0x7fd4680b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd4707ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd4681597f0 0x7fd4681ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd4707ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680b26d0 con 0x7fd4680cf9a0 2023-03-03T05:46:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd4707ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd468066470 con 0x7fd4681597f0 2023-03-03T05:46:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd4707ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680b2970 con 0x7fd4680ced80 2023-03-03T05:46:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd46e59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4680ced80 0x7fd4681e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd46e59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4680ced80 0x7fd4681e4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37712/0 (socket says 172.21.15.102:37712) 2023-03-03T05:46:00.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd46e59b700 1 -- 172.21.15.102:0/1415843950 learned_addr learned my addr 172.21.15.102:0/1415843950 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:00.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd46dd9a700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 0x7fd4680b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.837+0000 7fd46ed9c700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd4681597f0 0x7fd4681ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd46e59b700 1 -- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd4681597f0 msgr2=0x7fd4681ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd46e59b700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd4681597f0 0x7fd4681ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd46e59b700 1 -- 172.21.15.102:0/1415843950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 msgr2=0x7fd4680b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:00.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd46e59b700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 0x7fd4680b0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd46e59b700 1 -- 172.21.15.102:0/1415843950 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd460009a70 con 0x7fd4680ced80 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd46e59b700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4680ced80 0x7fd4681e4a70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd46400e870 tx=0x7fd46400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd45effd700 1 -- 172.21.15.102:0/1415843950 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464009590 con 0x7fd4680ced80 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd45effd700 1 -- 172.21.15.102:0/1415843950 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464013680 con 0x7fd4680ced80 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd45effd700 1 -- 172.21.15.102:0/1415843950 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640139f0 con 0x7fd4680ced80 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4680cb0c0 con 0x7fd4680ced80 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.838+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4680cade0 con 0x7fd4680ced80 2023-03-03T05:46:00.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.839+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4680cb3a0 con 0x7fd4680ced80 2023-03-03T05:46:00.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.841+0000 7fd45effd700 1 -- 172.21.15.102:0/1415843950 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd464019030 con 0x7fd4680ced80 2023-03-03T05:46:00.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.842+0000 7fd45effd700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd44c0682f0 0x7fd44c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:00.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.842+0000 7fd45effd700 1 -- 172.21.15.102:0/1415843950 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd464085a40 con 0x7fd4680ced80 2023-03-03T05:46:00.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.842+0000 7fd46dd9a700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd44c0682f0 0x7fd44c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:00.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.843+0000 7fd46dd9a700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd44c0682f0 0x7fd44c06a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd46806c770 tx=0x7fd458009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:00.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:00.846+0000 7fd45effd700 1 -- 172.21.15.102:0/1415843950 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd4640521b0 con 0x7fd4680ced80 2023-03-03T05:46:01.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.039+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4680677f0 con 0x7fd4680ced80 2023-03-03T05:46:01.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.041+0000 7fd45effd700 1 -- 172.21.15.102:0/1415843950 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7fd4640557d0 con 0x7fd4680ced80 2023-03-03T05:46:01.043 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:01.043 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:46:01.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd44c0682f0 msgr2=0x7fd44c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd44c0682f0 0x7fd44c06a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd46806c770 tx=0x7fd458009040 comp rx=0 tx=0).stop 2023-03-03T05:46:01.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4680ced80 msgr2=0x7fd4681e4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4680ced80 0x7fd4681e4a70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd46400e870 tx=0x7fd46400eb80 comp rx=0 tx=0).stop 2023-03-03T05:46:01.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 shutdown_connections 2023-03-03T05:46:01.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd44c0682f0 0x7fd44c06a7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4680ced80 0x7fd4681e4a70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd4681597f0 0x7fd4681ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 --2- 172.21.15.102:0/1415843950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4680cf9a0 0x7fd4680b0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.043+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 >> 172.21.15.102:0/1415843950 conn(0x7fd468065fb0 msgr2=0x7fd4680ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:01.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.044+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 shutdown_connections 2023-03-03T05:46:01.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.044+0000 7fd4707ff700 1 -- 172.21.15.102:0/1415843950 wait complete. 2023-03-03T05:46:01.073 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T05:45:20.032740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T05:45:27.930463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T05:45:33.613145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T05:45:34.842860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T05:46:01.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T05:46:01.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.322+0000 7f6f6bbff700 1 Processor -- start 2023-03-03T05:46:01.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.322+0000 7f6f6bbff700 1 -- start start 2023-03-03T05:46:01.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.323+0000 7f6f6bbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f64153450 0x7f6f640cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.323+0000 7f6f6bbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6f6415a380 0x7f6f64159940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6bbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f64159e80 con 0x7f6f640cf9a0 2023-03-03T05:46:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6bbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f640cae70 con 0x7f6f64153450 2023-03-03T05:46:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6bbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f640cb110 con 0x7f6f6415a380 2023-03-03T05:46:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6999b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f64153450 0x7f6f640cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6919a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6f6415a380 0x7f6f64159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6999b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f64153450 0x7f6f640cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42760/0 (socket says 172.21.15.102:42760) 2023-03-03T05:46:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6999b700 1 -- 172.21.15.102:0/1680939387 learned_addr learned my addr 172.21.15.102:0/1680939387 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6999b700 1 -- 172.21.15.102:0/1680939387 >> v2:172.21.15.172:3300/0 conn(0x7f6f6415a380 msgr2=0x7f6f64159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6999b700 1 --2- 172.21.15.102:0/1680939387 >> v2:172.21.15.172:3300/0 conn(0x7f6f6415a380 0x7f6f64159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6999b700 1 -- 172.21.15.102:0/1680939387 >> v1:172.21.15.172:6789/0 conn(0x7f6f640cf9a0 legacy=0x7f6f640ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f6999b700 1 -- 172.21.15.102:0/1680939387 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640cb3f0 con 0x7f6f64153450 2023-03-03T05:46:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.324+0000 7f6f68999700 1 -- 172.21.15.102:0/1680939387 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940217084 0 0) 0x7f6f64159e80 con 0x7f6f640cf9a0 2023-03-03T05:46:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.325+0000 7f6f6999b700 1 --2- 172.21.15.102:0/1680939387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f64153450 0x7f6f640cc1b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f6406d4d0 tx=0x7f6f60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da9e0898102aaeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.325+0000 7f6f68999700 1 -- 172.21.15.102:0/1680939387 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60011830 con 0x7f6f64153450 2023-03-03T05:46:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.325+0000 7f6f68999700 1 -- 172.21.15.102:0/1680939387 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f60011990 con 0x7f6f64153450 2023-03-03T05:46:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.325+0000 7f6f68999700 1 -- 172.21.15.102:0/1680939387 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60011d20 con 0x7f6f64153450 2023-03-03T05:46:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 -- 172.21.15.102:0/1680939387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f64153450 msgr2=0x7f6f640cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/1680939387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f64153450 0x7f6f640cc1b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f6406d4d0 tx=0x7f6f60009bd0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 -- 172.21.15.102:0/1680939387 shutdown_connections 2023-03-03T05:46:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/1680939387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f64153450 0x7f6f640cc1b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/1680939387 >> v2:172.21.15.172:3300/0 conn(0x7f6f6415a380 0x7f6f64159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 -- 172.21.15.102:0/1680939387 >> 172.21.15.102:0/1680939387 conn(0x7f6f64065fb0 msgr2=0x7f6f640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 -- 172.21.15.102:0/1680939387 shutdown_connections 2023-03-03T05:46:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.326+0000 7f6f6bbff700 1 -- 172.21.15.102:0/1680939387 wait complete. 2023-03-03T05:46:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.328+0000 7f6f6bbff700 1 Processor -- start 2023-03-03T05:46:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.328+0000 7f6f6bbff700 1 -- start start 2023-03-03T05:46:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6bbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f640cf9a0 0x7f6f640b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6bbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f64153450 0x7f6f640aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6bbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f6415a380 0x7f6f64153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6bbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641562f0 con 0x7f6f6415a380 2023-03-03T05:46:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6bbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640af260 con 0x7f6f64153450 2023-03-03T05:46:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6bbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641539b0 con 0x7f6f640cf9a0 2023-03-03T05:46:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6999b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f640cf9a0 0x7f6f640b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6919a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f64153450 0x7f6f640aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6999b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f640cf9a0 0x7f6f640b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37724/0 (socket says 172.21.15.102:37724) 2023-03-03T05:46:01.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6999b700 1 -- 172.21.15.102:0/2170147929 learned_addr learned my addr 172.21.15.102:0/2170147929 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.329+0000 7f6f6a19c700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f6415a380 0x7f6f64153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6999b700 1 -- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f64153450 msgr2=0x7f6f640aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6999b700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f64153450 0x7f6f640aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6999b700 1 -- 172.21.15.102:0/2170147929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f6415a380 msgr2=0x7f6f64153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6999b700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f6415a380 0x7f6f64153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6999b700 1 -- 172.21.15.102:0/2170147929 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60009550 con 0x7f6f640cf9a0 2023-03-03T05:46:01.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6919a700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f64153450 0x7f6f640aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6999b700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f640cf9a0 0x7f6f640b09e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009c10 tx=0x7f6f60004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f5a7fc700 1 -- 172.21.15.102:0/2170147929 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60004510 con 0x7f6f640cf9a0 2023-03-03T05:46:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f5a7fc700 1 -- 172.21.15.102:0/2170147929 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f600036a0 con 0x7f6f640cf9a0 2023-03-03T05:46:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f5a7fc700 1 -- 172.21.15.102:0/2170147929 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60003a10 con 0x7f6f640cf9a0 2023-03-03T05:46:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.330+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f640cb9b0 con 0x7f6f640cf9a0 2023-03-03T05:46:01.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.331+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f640cb6d0 con 0x7f6f640cf9a0 2023-03-03T05:46:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.332+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f640cbc90 con 0x7f6f640cf9a0 2023-03-03T05:46:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.333+0000 7f6f5a7fc700 1 -- 172.21.15.102:0/2170147929 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f6002d070 con 0x7f6f640cf9a0 2023-03-03T05:46:01.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.334+0000 7f6f5a7fc700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f480682f0 0x7f6f4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.334+0000 7f6f5a7fc700 1 -- 172.21.15.102:0/2170147929 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6f60021030 con 0x7f6f640cf9a0 2023-03-03T05:46:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.334+0000 7f6f6919a700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f480682f0 0x7f6f4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.335+0000 7f6f6919a700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f480682f0 0x7f6f4806a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f640c2680 tx=0x7f6f54009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:01.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.338+0000 7f6f5a7fc700 1 -- 172.21.15.102:0/2170147929 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f60053af0 con 0x7f6f640cf9a0 2023-03-03T05:46:01.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.529+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f64153c90 con 0x7f6f640cf9a0 2023-03-03T05:46:01.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.530+0000 7f6f5a7fc700 1 -- 172.21.15.102:0/2170147929 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f600150a0 con 0x7f6f640cf9a0 2023-03-03T05:46:01.532 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2023-03-03T05:46:01.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f480682f0 msgr2=0x7f6f4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f480682f0 0x7f6f4806a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f640c2680 tx=0x7f6f54009210 comp rx=0 tx=0).stop 2023-03-03T05:46:01.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f640cf9a0 msgr2=0x7f6f640b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f640cf9a0 0x7f6f640b09e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009c10 tx=0x7f6f60004750 comp rx=0 tx=0).stop 2023-03-03T05:46:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 shutdown_connections 2023-03-03T05:46:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f480682f0 0x7f6f4806a7a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f640cf9a0 0x7f6f640b09e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f6415a380 0x7f6f64153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 --2- 172.21.15.102:0/2170147929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f64153450 0x7f6f640aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 >> 172.21.15.102:0/2170147929 conn(0x7f6f64065fb0 msgr2=0x7f6f6415b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:01.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 shutdown_connections 2023-03-03T05:46:01.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.533+0000 7f6f6bbff700 1 -- 172.21.15.102:0/2170147929 wait complete. 2023-03-03T05:46:01.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T05:46:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.806+0000 7f0a1ffff700 1 Processor -- start 2023-03-03T05:46:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.806+0000 7f0a1ffff700 1 -- start start 2023-03-03T05:46:01.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.807+0000 7f0a1ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a180ce410 0x7f0a180cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.807+0000 7f0a1ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0a180cd760 0x7f0a18163720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a18153630 con 0x7f0a180ccc40 2023-03-03T05:46:01.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1ffff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a180ca7a0 con 0x7f0a180cd760 2023-03-03T05:46:01.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1ffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a180cd550 con 0x7f0a180ce410 2023-03-03T05:46:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0a180cd760 0x7f0a18163720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1d59a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a180ce410 0x7f0a180cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1dd9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0a180ccc40 0x7f0a180cdd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33872/0 (socket says 172.21.15.102:33872) 2023-03-03T05:46:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1d59a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a180ce410 0x7f0a180cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56956/0 (socket says 172.21.15.102:56956) 2023-03-03T05:46:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1dd9b700 1 -- 172.21.15.102:0/1527547613 learned_addr learned my addr 172.21.15.102:0/1527547613 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1e59c700 1 -- 172.21.15.102:0/1527547613 >> v2:172.21.15.172:3300/0 conn(0x7f0a180ce410 msgr2=0x7f0a180cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1e59c700 1 --2- 172.21.15.102:0/1527547613 >> v2:172.21.15.172:3300/0 conn(0x7f0a180ce410 0x7f0a180cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1cd99700 1 -- 172.21.15.102:0/1527547613 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626881093 0 0) 0x7f0a18153630 con 0x7f0a180ccc40 2023-03-03T05:46:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1e59c700 1 -- 172.21.15.102:0/1527547613 >> v1:172.21.15.172:6789/0 conn(0x7f0a180ccc40 legacy=0x7f0a180cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.808+0000 7f0a1e59c700 1 -- 172.21.15.102:0/1527547613 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1815d1d0 con 0x7f0a180cd760 2023-03-03T05:46:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.809+0000 7f0a1e59c700 1 --2- 172.21.15.102:0/1527547613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18163720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a18156ab0 tx=0x7f0a1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77390faca986134 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.809+0000 7f0a1cd99700 1 -- 172.21.15.102:0/1527547613 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1000e070 con 0x7f0a180cd760 2023-03-03T05:46:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.809+0000 7f0a1cd99700 1 -- 172.21.15.102:0/1527547613 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1000bb20 con 0x7f0a180cd760 2023-03-03T05:46:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.809+0000 7f0a1cd99700 1 -- 172.21.15.102:0/1527547613 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10012430 con 0x7f0a180cd760 2023-03-03T05:46:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 -- 172.21.15.102:0/1527547613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 msgr2=0x7f0a18163720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/1527547613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18163720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a18156ab0 tx=0x7f0a1000b330 comp rx=0 tx=0).stop 2023-03-03T05:46:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 -- 172.21.15.102:0/1527547613 shutdown_connections 2023-03-03T05:46:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/1527547613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18163720 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/1527547613 >> v2:172.21.15.172:3300/0 conn(0x7f0a180ce410 0x7f0a180cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 -- 172.21.15.102:0/1527547613 >> 172.21.15.102:0/1527547613 conn(0x7f0a18065fb0 msgr2=0x7f0a180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 -- 172.21.15.102:0/1527547613 shutdown_connections 2023-03-03T05:46:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.810+0000 7f0a1ffff700 1 -- 172.21.15.102:0/1527547613 wait complete. 2023-03-03T05:46:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.812+0000 7f0a1ffff700 1 Processor -- start 2023-03-03T05:46:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.812+0000 7f0a1ffff700 1 -- start start 2023-03-03T05:46:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a180ccc40 0x7f0a180b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a180ce410 0x7f0a181562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1815bfd0 con 0x7f0a180cd760 2023-03-03T05:46:01.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a181555d0 con 0x7f0a180ce410 2023-03-03T05:46:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a180aee70 con 0x7f0a180ccc40 2023-03-03T05:46:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1d59a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.813+0000 7f0a1dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a180ccc40 0x7f0a180b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a180ce410 0x7f0a181562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1d59a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42794/0 (socket says 172.21.15.102:42794) 2023-03-03T05:46:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1d59a700 1 -- 172.21.15.102:0/2195378895 learned_addr learned my addr 172.21.15.102:0/2195378895 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1dd9b700 1 -- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a180ce410 msgr2=0x7f0a181562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1dd9b700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a180ce410 0x7f0a181562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1dd9b700 1 -- 172.21.15.102:0/2195378895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 msgr2=0x7f0a18155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1dd9b700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1dd9b700 1 -- 172.21.15.102:0/2195378895 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10009a70 con 0x7f0a180ccc40 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1d59a700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a1dd9b700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a180ccc40 0x7f0a180b10a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400c4b0 tx=0x7f0a1400c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.814+0000 7f0a0e7fc700 1 -- 172.21.15.102:0/2195378895 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14004c80 con 0x7f0a180ccc40 2023-03-03T05:46:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.815+0000 7f0a0e7fc700 1 -- 172.21.15.102:0/2195378895 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a14004e00 con 0x7f0a180ccc40 2023-03-03T05:46:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.815+0000 7f0a0e7fc700 1 -- 172.21.15.102:0/2195378895 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14007310 con 0x7f0a180ccc40 2023-03-03T05:46:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.815+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a181594b0 con 0x7f0a180ccc40 2023-03-03T05:46:01.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.815+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a181591d0 con 0x7f0a180ccc40 2023-03-03T05:46:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.816+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a18159790 con 0x7f0a180ccc40 2023-03-03T05:46:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.817+0000 7f0a0e7fc700 1 -- 172.21.15.102:0/2195378895 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0a1401a020 con 0x7f0a180ccc40 2023-03-03T05:46:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.818+0000 7f0a0e7fc700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09fc068320 0x7f09fc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.818+0000 7f0a0e7fc700 1 -- 172.21.15.102:0/2195378895 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0a140857a0 con 0x7f0a180ccc40 2023-03-03T05:46:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.818+0000 7f0a1d59a700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09fc068320 0x7f09fc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.819+0000 7f0a1d59a700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09fc068320 0x7f09fc06a7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0a18158470 tx=0x7f0a08009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:01.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:01.822+0000 7f0a0e7fc700 1 -- 172.21.15.102:0/2195378895 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a14052080 con 0x7f0a180ccc40 2023-03-03T05:46:02.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.015+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0a1804e6e0 con 0x7f0a180ccc40 2023-03-03T05:46:02.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.016+0000 7f0a0e7fc700 1 -- 172.21.15.102:0/2195378895 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0a140556a0 con 0x7f0a180ccc40 2023-03-03T05:46:02.018 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2023-03-03T05:46:02.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09fc068320 msgr2=0x7f09fc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09fc068320 0x7f09fc06a7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0a18158470 tx=0x7f0a08009210 comp rx=0 tx=0).stop 2023-03-03T05:46:02.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a180ccc40 msgr2=0x7f0a180b10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a180ccc40 0x7f0a180b10a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400c4b0 tx=0x7f0a1400c7c0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 shutdown_connections 2023-03-03T05:46:02.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09fc068320 0x7f09fc06a7d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a180ccc40 0x7f0a180b10a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a180ce410 0x7f0a181562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 --2- 172.21.15.102:0/2195378895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a180cd760 0x7f0a18155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.018+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 >> 172.21.15.102:0/2195378895 conn(0x7f0a18065fb0 msgr2=0x7f0a18152470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:02.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.019+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 shutdown_connections 2023-03-03T05:46:02.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.019+0000 7f0a1ffff700 1 -- 172.21.15.102:0/2195378895 wait complete. 2023-03-03T05:46:02.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T05:46:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.293+0000 7f5ce1f9d700 1 Processor -- start 2023-03-03T05:46:02.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.294+0000 7f5ce1f9d700 1 -- start start 2023-03-03T05:46:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.295+0000 7f5ce1f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5cdc158cb0 0x7f5cdc15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.295+0000 7f5ce1f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5cdc15ce20 0x7f5cdc0c4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.295+0000 7f5ce1f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc159140 con 0x7f5cdc153180 2023-03-03T05:46:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.295+0000 7f5ce1f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc159cc0 con 0x7f5cdc15ce20 2023-03-03T05:46:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.295+0000 7f5ce1f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc159420 con 0x7f5cdc158cb0 2023-03-03T05:46:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.295+0000 7f5cdbfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5cdc158cb0 0x7f5cdc15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5ce0f9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5cdc153180 0x7f5cdc0c9640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33874/0 (socket says 172.21.15.102:33874) 2023-03-03T05:46:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5ce179c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5cdc15ce20 0x7f5cdc0c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5ce0f9b700 1 -- 172.21.15.102:0/366191258 learned_addr learned my addr 172.21.15.102:0/366191258 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5cdbfff700 1 -- 172.21.15.102:0/366191258 >> v2:172.21.15.102:3300/0 conn(0x7f5cdc15ce20 msgr2=0x7f5cdc0c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5cdbfff700 1 --2- 172.21.15.102:0/366191258 >> v2:172.21.15.102:3300/0 conn(0x7f5cdc15ce20 0x7f5cdc0c4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5cdbfff700 1 -- 172.21.15.102:0/366191258 >> v1:172.21.15.172:6789/0 conn(0x7f5cdc153180 legacy=0x7f5cdc0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5cdbfff700 1 -- 172.21.15.102:0/366191258 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc159700 con 0x7f5cdc158cb0 2023-03-03T05:46:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5ce179c700 1 --2- 172.21.15.102:0/366191258 >> v2:172.21.15.102:3300/0 conn(0x7f5cdc15ce20 0x7f5cdc0c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.296+0000 7f5cdbfff700 1 --2- 172.21.15.102:0/366191258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc15c8e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc0b4ef0 tx=0x7f5cc8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b99bc6b85ddd450 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.297+0000 7f5cdb7fe700 1 -- 172.21.15.102:0/366191258 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8011830 con 0x7f5cdc158cb0 2023-03-03T05:46:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.297+0000 7f5cdb7fe700 1 -- 172.21.15.102:0/366191258 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc8011990 con 0x7f5cdc158cb0 2023-03-03T05:46:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.297+0000 7f5cdb7fe700 1 -- 172.21.15.102:0/366191258 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8011d20 con 0x7f5cdc158cb0 2023-03-03T05:46:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/366191258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 msgr2=0x7f5cdc15c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/366191258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc15c8e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc0b4ef0 tx=0x7f5cc8009b20 comp rx=0 tx=0).stop 2023-03-03T05:46:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/366191258 shutdown_connections 2023-03-03T05:46:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/366191258 >> v2:172.21.15.102:3300/0 conn(0x7f5cdc15ce20 0x7f5cdc0c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/366191258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc15c8e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/366191258 >> 172.21.15.102:0/366191258 conn(0x7f5cdc071a30 msgr2=0x7f5cdc065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/366191258 shutdown_connections 2023-03-03T05:46:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.298+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/366191258 wait complete. 2023-03-03T05:46:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.301+0000 7f5ce1f9d700 1 Processor -- start 2023-03-03T05:46:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.301+0000 7f5ce1f9d700 1 -- start start 2023-03-03T05:46:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.302+0000 7f5ce1f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cdc153180 0x7f5cdc0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.302+0000 7f5ce1f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc0cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.302+0000 7f5ce1f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc15ce20 0x7f5cdc1f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.302+0000 7f5ce1f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0b4cf0 con 0x7f5cdc15ce20 2023-03-03T05:46:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.302+0000 7f5ce1f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc14e870 con 0x7f5cdc158cb0 2023-03-03T05:46:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.302+0000 7f5ce1f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc14bf50 con 0x7f5cdc153180 2023-03-03T05:46:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cdc153180 0x7f5cdc0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cdc153180 0x7f5cdc0cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37754/0 (socket says 172.21.15.102:37754) 2023-03-03T05:46:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 -- 172.21.15.102:0/3571749257 learned_addr learned my addr 172.21.15.102:0/3571749257 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5cdbfff700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc0cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce179c700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc15ce20 0x7f5cdc1f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 -- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 msgr2=0x7f5cdc0cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc0cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 -- 172.21.15.102:0/3571749257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc15ce20 msgr2=0x7f5cdc1f1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc15ce20 0x7f5cdc1f1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce0f9b700 1 -- 172.21.15.102:0/3571749257 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4009a70 con 0x7f5cdc153180 2023-03-03T05:46:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5cdbfff700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc0cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.303+0000 7f5ce179c700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc15ce20 0x7f5cdc1f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.304+0000 7f5ce0f9b700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cdc153180 0x7f5cdc0cad90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000c520 tx=0x7f5cd000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.304+0000 7f5cd97fa700 1 -- 172.21.15.102:0/3571749257 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0005d70 con 0x7f5cdc153180 2023-03-03T05:46:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.304+0000 7f5cd97fa700 1 -- 172.21.15.102:0/3571749257 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd00070d0 con 0x7f5cdc153180 2023-03-03T05:46:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.304+0000 7f5cd97fa700 1 -- 172.21.15.102:0/3571749257 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0007440 con 0x7f5cdc153180 2023-03-03T05:46:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.304+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc80094a0 con 0x7f5cdc153180 2023-03-03T05:46:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.304+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc15c3c0 con 0x7f5cdc153180 2023-03-03T05:46:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.305+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc1599e0 con 0x7f5cdc153180 2023-03-03T05:46:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.306+0000 7f5cd97fa700 1 -- 172.21.15.102:0/3571749257 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5cd0019030 con 0x7f5cdc153180 2023-03-03T05:46:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.307+0000 7f5cd97fa700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5cc00682f0 0x7f5cc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.308+0000 7f5cd97fa700 1 -- 172.21.15.102:0/3571749257 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5cd0084ed0 con 0x7f5cdc153180 2023-03-03T05:46:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.308+0000 7f5cdbfff700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5cc00682f0 0x7f5cc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.308+0000 7f5cdbfff700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5cc00682f0 0x7f5cc006a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80090b0 tx=0x7f5cc8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:02.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.312+0000 7f5cd97fa700 1 -- 172.21.15.102:0/3571749257 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5cd004b5e0 con 0x7f5cdc153180 2023-03-03T05:46:02.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.502+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5cdc0c5050 con 0x7f5cdc153180 2023-03-03T05:46:02.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.503+0000 7f5cd97fa700 1 -- 172.21.15.102:0/3571749257 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5cd0017020 con 0x7f5cdc153180 2023-03-03T05:46:02.505 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2023-03-03T05:46:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5cc00682f0 msgr2=0x7f5cc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5cc00682f0 0x7f5cc006a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80090b0 tx=0x7f5cc8008030 comp rx=0 tx=0).stop 2023-03-03T05:46:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cdc153180 msgr2=0x7f5cdc0cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cdc153180 0x7f5cdc0cad90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000c520 tx=0x7f5cd000c830 comp rx=0 tx=0).stop 2023-03-03T05:46:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 shutdown_connections 2023-03-03T05:46:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5cc00682f0 0x7f5cc006a7a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5cdc153180 0x7f5cdc0cad90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc15ce20 0x7f5cdc1f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 --2- 172.21.15.102:0/3571749257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cdc158cb0 0x7f5cdc0cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.505+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 >> 172.21.15.102:0/3571749257 conn(0x7f5cdc071a30 msgr2=0x7f5cdc14d420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.506+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 shutdown_connections 2023-03-03T05:46:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.506+0000 7f5ce1f9d700 1 -- 172.21.15.102:0/3571749257 wait complete. 2023-03-03T05:46:02.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T05:46:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.780+0000 7f995f39d700 1 Processor -- start 2023-03-03T05:46:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.780+0000 7f995f39d700 1 -- start start 2023-03-03T05:46:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995f39d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9958153180 0x7f99580c9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995f39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f99580c4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995f39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958159140 con 0x7f9958158cb0 2023-03-03T05:46:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995f39d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9958159cc0 con 0x7f9958153180 2023-03-03T05:46:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995f39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9958159420 con 0x7f99580c9b80 2023-03-03T05:46:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995e39b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9958153180 0x7f99580c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995e39b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9958153180 0x7f99580c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42824/0 (socket says 172.21.15.102:42824) 2023-03-03T05:46:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995e39b700 1 -- 172.21.15.102:0/1415129009 learned_addr learned my addr 172.21.15.102:0/1415129009 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.782+0000 7f995eb9c700 1 --2- 172.21.15.102:0/1415129009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f99580c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995e39b700 1 -- 172.21.15.102:0/1415129009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 msgr2=0x7f99580c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995e39b700 1 --2- 172.21.15.102:0/1415129009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f99580c4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995e39b700 1 -- 172.21.15.102:0/1415129009 >> v1:172.21.15.102:6789/0 conn(0x7f9958158cb0 legacy=0x7f995815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995e39b700 1 -- 172.21.15.102:0/1415129009 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958159700 con 0x7f9958153180 2023-03-03T05:46:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995d399700 1 -- 172.21.15.102:0/1415129009 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890181973 0 0) 0x7f9958159140 con 0x7f9958158cb0 2023-03-03T05:46:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995e39b700 1 --2- 172.21.15.102:0/1415129009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f99580c9640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f99540098f0 tx=0x7f9954009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7745ea36411159d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995d399700 1 -- 172.21.15.102:0/1415129009 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954011830 con 0x7f9958153180 2023-03-03T05:46:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995d399700 1 -- 172.21.15.102:0/1415129009 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9954011990 con 0x7f9958153180 2023-03-03T05:46:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.783+0000 7f995d399700 1 -- 172.21.15.102:0/1415129009 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954011d20 con 0x7f9958153180 2023-03-03T05:46:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 -- 172.21.15.102:0/1415129009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 msgr2=0x7f99580c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 --2- 172.21.15.102:0/1415129009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f99580c9640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f99540098f0 tx=0x7f9954009c00 comp rx=0 tx=0).stop 2023-03-03T05:46:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 -- 172.21.15.102:0/1415129009 shutdown_connections 2023-03-03T05:46:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 --2- 172.21.15.102:0/1415129009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f99580c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 --2- 172.21.15.102:0/1415129009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f99580c9640 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 -- 172.21.15.102:0/1415129009 >> 172.21.15.102:0/1415129009 conn(0x7f9958071a30 msgr2=0x7f9958065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 -- 172.21.15.102:0/1415129009 shutdown_connections 2023-03-03T05:46:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.784+0000 7f995f39d700 1 -- 172.21.15.102:0/1415129009 wait complete. 2023-03-03T05:46:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.786+0000 7f995f39d700 1 Processor -- start 2023-03-03T05:46:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.786+0000 7f995f39d700 1 -- start start 2023-03-03T05:46:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.787+0000 7f995f39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f995815b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.787+0000 7f995f39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f995814d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.787+0000 7f995f39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9958158cb0 0x7f995814deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.787+0000 7f995f39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99581eeaa0 con 0x7f9958153180 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.787+0000 7f995f39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9958151580 con 0x7f9958158cb0 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.787+0000 7f995f39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99581ef8c0 con 0x7f99580c9b80 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f995815b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995eb9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9958158cb0 0x7f995814deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995db9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f995814d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f995815b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37760/0 (socket says 172.21.15.102:37760) 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 -- 172.21.15.102:0/2098907774 learned_addr learned my addr 172.21.15.102:0/2098907774 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 -- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9958158cb0 msgr2=0x7f995814deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9958158cb0 0x7f995814deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 -- 172.21.15.102:0/2098907774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 msgr2=0x7f995814d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f995814d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 -- 172.21.15.102:0/2098907774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9954009550 con 0x7f99580c9b80 2023-03-03T05:46:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995db9a700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f995814d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.788+0000 7f995e39b700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f995815b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9954009c40 tx=0x7f9954004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.789+0000 7f994effd700 1 -- 172.21.15.102:0/2098907774 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954004300 con 0x7f99580c9b80 2023-03-03T05:46:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.789+0000 7f994effd700 1 -- 172.21.15.102:0/2098907774 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995401f070 con 0x7f99580c9b80 2023-03-03T05:46:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.789+0000 7f994effd700 1 -- 172.21.15.102:0/2098907774 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99540036a0 con 0x7f99580c9b80 2023-03-03T05:46:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.789+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995815c3c0 con 0x7f99580c9b80 2023-03-03T05:46:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.789+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99581599e0 con 0x7f99580c9b80 2023-03-03T05:46:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.790+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99580c5050 con 0x7f99580c9b80 2023-03-03T05:46:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.791+0000 7f994effd700 1 -- 172.21.15.102:0/2098907774 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f995402a020 con 0x7f99580c9b80 2023-03-03T05:46:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.792+0000 7f994effd700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f993c0682f0 0x7f993c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.792+0000 7f994effd700 1 -- 172.21.15.102:0/2098907774 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9954087340 con 0x7f99580c9b80 2023-03-03T05:46:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.793+0000 7f995db9a700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f993c0682f0 0x7f993c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.793+0000 7f995db9a700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f993c0682f0 0x7f993c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f99581562c0 tx=0x7f9948009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:02.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.797+0000 7f994effd700 1 -- 172.21.15.102:0/2098907774 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f99540539d0 con 0x7f99580c9b80 2023-03-03T05:46:02.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.985+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f995814d690 con 0x7f99580c9b80 2023-03-03T05:46:02.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.986+0000 7f994effd700 1 -- 172.21.15.102:0/2098907774 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9954056ff0 con 0x7f99580c9b80 2023-03-03T05:46:02.988 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2023-03-03T05:46:02.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f993c0682f0 msgr2=0x7f993c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f993c0682f0 0x7f993c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f99581562c0 tx=0x7f9948009040 comp rx=0 tx=0).stop 2023-03-03T05:46:02.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 msgr2=0x7f995815b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:02.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f995815b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9954009c40 tx=0x7f9954004750 comp rx=0 tx=0).stop 2023-03-03T05:46:02.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 shutdown_connections 2023-03-03T05:46:02.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f993c0682f0 0x7f993c06a7a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99580c9b80 0x7f995815b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.988+0000 7f995f39d700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9958158cb0 0x7f995814deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.989+0000 7f995f39d700 1 --2- 172.21.15.102:0/2098907774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9958153180 0x7f995814d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:02.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.989+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 >> 172.21.15.102:0/2098907774 conn(0x7f9958071a30 msgr2=0x7f99581578f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:02.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.989+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 shutdown_connections 2023-03-03T05:46:02.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:02.989+0000 7f995f39d700 1 -- 172.21.15.102:0/2098907774 wait complete. 2023-03-03T05:46:03.017 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-03-03T05:46:03.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:46:03.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.260+0000 7fc27bdff700 1 Processor -- start 2023-03-03T05:46:03.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.260+0000 7fc27bdff700 1 -- start start 2023-03-03T05:46:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.261+0000 7fc27bdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2740ced80 0x7fc2740cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.261+0000 7fc27bdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2741597f0 0x7fc2740cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.261+0000 7fc27bdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2740cb9f0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.261+0000 7fc27bdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2740ca750 con 0x7fc2741597f0 2023-03-03T05:46:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.261+0000 7fc27bdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27415a230 con 0x7fc2740ced80 2023-03-03T05:46:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc279b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2740ced80 0x7fc2740cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc27939a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2741597f0 0x7fc2740cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc27a39c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc2740cf9a0 0x7fc27415c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33882/0 (socket says 172.21.15.102:33882) 2023-03-03T05:46:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc27a39c700 1 -- 172.21.15.102:0/1703606078 learned_addr learned my addr 172.21.15.102:0/1703606078 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc279b9b700 1 -- 172.21.15.102:0/1703606078 >> v2:172.21.15.102:3300/0 conn(0x7fc2741597f0 msgr2=0x7fc2740cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc279b9b700 1 --2- 172.21.15.102:0/1703606078 >> v2:172.21.15.102:3300/0 conn(0x7fc2741597f0 0x7fc2740cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc279b9b700 1 -- 172.21.15.102:0/1703606078 >> v1:172.21.15.172:6789/0 conn(0x7fc2740cf9a0 legacy=0x7fc27415c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc279b9b700 1 -- 172.21.15.102:0/1703606078 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc274159c80 con 0x7fc2740ced80 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc27939a700 1 --2- 172.21.15.102:0/1703606078 >> v2:172.21.15.102:3300/0 conn(0x7fc2741597f0 0x7fc2740cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc278b99700 1 -- 172.21.15.102:0/1703606078 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830739298 0 0) 0x7fc2740cb9f0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.262+0000 7fc279b9b700 1 --2- 172.21.15.102:0/1703606078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2740ced80 0x7fc2740cf150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc270009840 tx=0x7fc270009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f642e2bde96093a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc278b99700 1 -- 172.21.15.102:0/1703606078 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270011830 con 0x7fc2740ced80 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc278b99700 1 -- 172.21.15.102:0/1703606078 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc270011990 con 0x7fc2740ced80 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc278b99700 1 -- 172.21.15.102:0/1703606078 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270011d20 con 0x7fc2740ced80 2023-03-03T05:46:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 -- 172.21.15.102:0/1703606078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2740ced80 msgr2=0x7fc2740cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1703606078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2740ced80 0x7fc2740cf150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc270009840 tx=0x7fc270009b50 comp rx=0 tx=0).stop 2023-03-03T05:46:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 -- 172.21.15.102:0/1703606078 shutdown_connections 2023-03-03T05:46:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1703606078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2740ced80 0x7fc2740cf150 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1703606078 >> v2:172.21.15.102:3300/0 conn(0x7fc2741597f0 0x7fc2740cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 -- 172.21.15.102:0/1703606078 >> 172.21.15.102:0/1703606078 conn(0x7fc274065fb0 msgr2=0x7fc2740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 -- 172.21.15.102:0/1703606078 shutdown_connections 2023-03-03T05:46:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.263+0000 7fc27bdff700 1 -- 172.21.15.102:0/1703606078 wait complete. 2023-03-03T05:46:03.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.265+0000 7fc27bdff700 1 Processor -- start 2023-03-03T05:46:03.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.266+0000 7fc27bdff700 1 -- start start 2023-03-03T05:46:03.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27bdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2740ced80 0x7fc2741f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27bdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2740cf9a0 0x7fc2740c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27bdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2741597f0 0x7fc2740c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27bdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27414aaa0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27bdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741f2540 con 0x7fc2740ced80 2023-03-03T05:46:03.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27bdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741f22a0 con 0x7fc2741597f0 2023-03-03T05:46:03.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2740cf9a0 0x7fc2740c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2740cf9a0 0x7fc2740c3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42842/0 (socket says 172.21.15.102:42842) 2023-03-03T05:46:03.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 -- 172.21.15.102:0/1045756040 learned_addr learned my addr 172.21.15.102:0/1045756040 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:03.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27a39c700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2741597f0 0x7fc2740c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc279b9b700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2740ced80 0x7fc2741f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 -- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2741597f0 msgr2=0x7fc2740c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2741597f0 0x7fc2740c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 -- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2740ced80 msgr2=0x7fc2741f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2740ced80 0x7fc2741f19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.267+0000 7fc27939a700 1 -- 172.21.15.102:0/1045756040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640094a0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.268+0000 7fc27939a700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2740cf9a0 0x7fc2740c3d60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc26400ec10 tx=0x7fc26400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.268+0000 7fc26a7fc700 1 -- 172.21.15.102:0/1045756040 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264009760 con 0x7fc2740cf9a0 2023-03-03T05:46:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.268+0000 7fc26a7fc700 1 -- 172.21.15.102:0/1045756040 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26400c850 con 0x7fc2740cf9a0 2023-03-03T05:46:03.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.268+0000 7fc26a7fc700 1 -- 172.21.15.102:0/1045756040 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26400cbe0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.268+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2700094a0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.268+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2740cb0c0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.269+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2740cade0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.270+0000 7fc26a7fc700 1 -- 172.21.15.102:0/1045756040 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc264019030 con 0x7fc2740cf9a0 2023-03-03T05:46:03.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.272+0000 7fc26a7fc700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc258068320 0x7fc25806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.272+0000 7fc26a7fc700 1 -- 172.21.15.102:0/1045756040 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc264085ba0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.272+0000 7fc279b9b700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc258068320 0x7fc25806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.273+0000 7fc279b9b700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc258068320 0x7fc25806a7d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc270009fd0 tx=0x7fc27001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:03.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.276+0000 7fc26a7fc700 1 -- 172.21.15.102:0/1045756040 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc264052310 con 0x7fc2740cf9a0 2023-03-03T05:46:03.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.514+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2740cb3a0 con 0x7fc2740cf9a0 2023-03-03T05:46:03.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.515+0000 7fc26a7fc700 1 -- 172.21.15.102:0/1045756040 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6173 (secure 0 0 0) 0x7fc264055930 con 0x7fc2740cf9a0 2023-03-03T05:46:03.517 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:03.518 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:45:41.326836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4315},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4315":{"gid":4315,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6837/1262584332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":1262584332},{"type":"v1","addr":"172.21.15.102:6837","nonce":1262584332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:46:03.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc258068320 msgr2=0x7fc25806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc258068320 0x7fc25806a7d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc270009fd0 tx=0x7fc27001c040 comp rx=0 tx=0).stop 2023-03-03T05:46:03.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2740cf9a0 msgr2=0x7fc2740c3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2740cf9a0 0x7fc2740c3d60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc26400ec10 tx=0x7fc26400ef20 comp rx=0 tx=0).stop 2023-03-03T05:46:03.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 shutdown_connections 2023-03-03T05:46:03.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc258068320 0x7fc25806a7d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2741597f0 0x7fc2740c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2740ced80 0x7fc2741f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 --2- 172.21.15.102:0/1045756040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2740cf9a0 0x7fc2740c3d60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 >> 172.21.15.102:0/1045756040 conn(0x7fc274065fb0 msgr2=0x7fc274156a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 shutdown_connections 2023-03-03T05:46:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.518+0000 7fc27bdff700 1 -- 172.21.15.102:0/1045756040 wait complete. 2023-03-03T05:46:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2023-03-03T05:46:03.547 INFO:tasks.mds_thrash:Ready to start thrashing 2023-03-03T05:46:03.547 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-03-03T05:46:03.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T05:46:03.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.790+0000 7f50093ff700 1 Processor -- start 2023-03-03T05:46:03.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.790+0000 7f50093ff700 1 -- start start 2023-03-03T05:46:03.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.791+0000 7f50093ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50041597f0 0x7f50040cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.791+0000 7f50093ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50040cb9f0 con 0x7f50040cf9a0 2023-03-03T05:46:03.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f50093ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5004159c80 con 0x7f50040ced80 2023-03-03T05:46:03.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f50093ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040ca750 con 0x7f50041597f0 2023-03-03T05:46:03.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f4ffbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50041597f0 0x7f50040cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f5002ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f50040ced80 0x7f50040cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33884/0 (socket says 172.21.15.102:33884) 2023-03-03T05:46:03.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f5002ffd700 1 -- 172.21.15.102:0/957898127 learned_addr learned my addr 172.21.15.102:0/957898127 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:03.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f4ffbfff700 1 -- 172.21.15.102:0/957898127 >> v1:172.21.15.172:6789/0 conn(0x7f50040ced80 legacy=0x7f50040cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f4ffbfff700 1 -- 172.21.15.102:0/957898127 >> v1:172.21.15.102:6789/0 conn(0x7f50040cf9a0 legacy=0x7f5004163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f4ffbfff700 1 -- 172.21.15.102:0/957898127 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040cade0 con 0x7f50041597f0 2023-03-03T05:46:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.792+0000 7f50027fc700 1 -- 172.21.15.102:0/957898127 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299086102 0 0) 0x7f5004159c80 con 0x7f50040ced80 2023-03-03T05:46:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f4ffbfff700 1 --2- 172.21.15.102:0/957898127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50041597f0 0x7f50040cf150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009870 tx=0x7f4fec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4ae21a6d590f5cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50027fc700 1 -- 172.21.15.102:0/957898127 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec011830 con 0x7f50041597f0 2023-03-03T05:46:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50027fc700 1 -- 172.21.15.102:0/957898127 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec011990 con 0x7f50041597f0 2023-03-03T05:46:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50027fc700 1 -- 172.21.15.102:0/957898127 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec011d20 con 0x7f50041597f0 2023-03-03T05:46:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50093ff700 1 -- 172.21.15.102:0/957898127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50041597f0 msgr2=0x7f50040cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50093ff700 1 --2- 172.21.15.102:0/957898127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50041597f0 0x7f50040cf150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009870 tx=0x7f4fec009b80 comp rx=0 tx=0).stop 2023-03-03T05:46:03.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50093ff700 1 -- 172.21.15.102:0/957898127 shutdown_connections 2023-03-03T05:46:03.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50093ff700 1 --2- 172.21.15.102:0/957898127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50041597f0 0x7f50040cf150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.793+0000 7f50093ff700 1 -- 172.21.15.102:0/957898127 >> 172.21.15.102:0/957898127 conn(0x7f5004065fb0 msgr2=0x7f50040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:03.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.794+0000 7f50093ff700 1 -- 172.21.15.102:0/957898127 shutdown_connections 2023-03-03T05:46:03.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.794+0000 7f50093ff700 1 -- 172.21.15.102:0/957898127 wait complete. 2023-03-03T05:46:03.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.795+0000 7f50093ff700 1 Processor -- start 2023-03-03T05:46:03.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.796+0000 7f50093ff700 1 -- start start 2023-03-03T05:46:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f50093ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040ced80 0x7f50041ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f50093ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50040cf9a0 0x7f500414f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f50093ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50041597f0 0x7f500414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f50093ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50041f2660 con 0x7f50040cf9a0 2023-03-03T05:46:03.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f50093ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5004150bd0 con 0x7f50041597f0 2023-03-03T05:46:03.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f50093ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5004150eb0 con 0x7f50040ced80 2023-03-03T05:46:03.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f5002ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040ced80 0x7f50041ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f50037fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50041597f0 0x7f500414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f5002ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040ced80 0x7f50041ee7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37796/0 (socket says 172.21.15.102:37796) 2023-03-03T05:46:03.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f5002ffd700 1 -- 172.21.15.102:0/1272634463 learned_addr learned my addr 172.21.15.102:0/1272634463 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:03.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f4ffbfff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50040cf9a0 0x7f500414f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f5002ffd700 1 -- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50041597f0 msgr2=0x7f500414fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f5002ffd700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50041597f0 0x7f500414fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f5002ffd700 1 -- 172.21.15.102:0/1272634463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50040cf9a0 msgr2=0x7f500414f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.797+0000 7f5002ffd700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50040cf9a0 0x7f500414f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f5002ffd700 1 -- 172.21.15.102:0/1272634463 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0094a0 con 0x7f50040ced80 2023-03-03T05:46:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f50037fe700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50041597f0 0x7f500414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f5002ffd700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040ced80 0x7f50041ee7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400c4d0 tx=0x7f4ff400c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f4ffb7fe700 1 -- 172.21.15.102:0/1272634463 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4004c80 con 0x7f50040ced80 2023-03-03T05:46:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f4ffb7fe700 1 -- 172.21.15.102:0/1272634463 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff4004de0 con 0x7f50040ced80 2023-03-03T05:46:03.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50040cb3a0 con 0x7f50040ced80 2023-03-03T05:46:03.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f4ffb7fe700 1 -- 172.21.15.102:0/1272634463 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4007310 con 0x7f50040ced80 2023-03-03T05:46:03.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.798+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50040cb0c0 con 0x7f50040ced80 2023-03-03T05:46:03.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.799+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50040cb680 con 0x7f50040ced80 2023-03-03T05:46:03.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.801+0000 7f4ffb7fe700 1 -- 172.21.15.102:0/1272634463 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ff401a020 con 0x7f50040ced80 2023-03-03T05:46:03.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.802+0000 7f4ffb7fe700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fe40682f0 0x7f4fe406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:03.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.802+0000 7f4ffb7fe700 1 -- 172.21.15.102:0/1272634463 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4ff4085840 con 0x7f50040ced80 2023-03-03T05:46:03.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.802+0000 7f4ffbfff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fe40682f0 0x7f4fe406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:03.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.803+0000 7f4ffbfff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fe40682f0 0x7f4fe406a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097c0 tx=0x7f4fec01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:03.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.806+0000 7f4ffb7fe700 1 -- 172.21.15.102:0/1272634463 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ff4051f20 con 0x7f50040ced80 2023-03-03T05:46:03.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:03.993+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 --> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50040677f0 con 0x7f4fe40682f0 2023-03-03T05:46:04.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.002+0000 7f4ffb7fe700 1 -- 172.21.15.102:0/1272634463 <== mgr.4105 v2:172.21.15.172:6832/114058 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286578 (secure 0 0 0) 0x7f50040677f0 con 0x7f4fe40682f0 2023-03-03T05:46:04.008 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:04.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fe40682f0 msgr2=0x7f4fe406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fe40682f0 0x7f4fe406a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097c0 tx=0x7f4fec01c040 comp rx=0 tx=0).stop 2023-03-03T05:46:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040ced80 msgr2=0x7f50041ee7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040ced80 0x7f50041ee7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400c4d0 tx=0x7f4ff400c7e0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 shutdown_connections 2023-03-03T05:46:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fe40682f0 0x7f4fe406a7a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50040ced80 0x7f50041ee7c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50041597f0 0x7f500414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 --2- 172.21.15.102:0/1272634463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50040cf9a0 0x7f500414f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 >> 172.21.15.102:0/1272634463 conn(0x7f5004065fb0 msgr2=0x7f5004066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:04.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 shutdown_connections 2023-03-03T05:46:04.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.007+0000 7f50093ff700 1 -- 172.21.15.102:0/1272634463 wait complete. 2023-03-03T05:46:04.011 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2023-03-03T05:46:04.036 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-03-03T05:46:03.950051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592957,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66244,"kb_used_data":3076,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":754908476,"statfs":{"total":773094113280,"available":773026279424,"internally_reserved":0,"allocated":3149824,"data_stored":1894693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":259,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003582"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922943+0000","last_change":"2023-03-03T05:45:35.854816+0000","last_active":"2023-03-03T05:45:36.922943+0000","last_peered":"2023-03-03T05:45:36.922943+0000","last_clean":"2023-03-03T05:45:36.922943+0000","last_became_active":"2023-03-03T05:45:35.854447+0000","last_became_peered":"2023-03-03T05:45:35.854447+0000","last_unstale":"2023-03-03T05:45:36.922943+0000","last_undegraded":"2023-03-03T05:45:36.922943+0000","last_fullsized":"2023-03-03T05:45:36.922943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850859+0000","last_change":"2023-03-03T05:45:35.862354+0000","last_active":"2023-03-03T05:45:36.850859+0000","last_peered":"2023-03-03T05:45:36.850859+0000","last_clean":"2023-03-03T05:45:36.850859+0000","last_became_active":"2023-03-03T05:45:35.862016+0000","last_became_peered":"2023-03-03T05:45:35.862016+0000","last_unstale":"2023-03-03T05:45:36.850859+0000","last_undegraded":"2023-03-03T05:45:36.850859+0000","last_fullsized":"2023-03-03T05:45:36.850859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962731+0000","last_change":"2023-03-03T05:45:35.864429+0000","last_active":"2023-03-03T05:45:35.962731+0000","last_peered":"2023-03-03T05:45:35.962731+0000","last_clean":"2023-03-03T05:45:35.962731+0000","last_became_active":"2023-03-03T05:45:35.864091+0000","last_became_peered":"2023-03-03T05:45:35.864091+0000","last_unstale":"2023-03-03T05:45:35.962731+0000","last_undegraded":"2023-03-03T05:45:35.962731+0000","last_fullsized":"2023-03-03T05:45:35.962731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962683+0000","last_change":"2023-03-03T05:45:35.861374+0000","last_active":"2023-03-03T05:45:35.962683+0000","last_peered":"2023-03-03T05:45:35.962683+0000","last_clean":"2023-03-03T05:45:35.962683+0000","last_became_active":"2023-03-03T05:45:35.860993+0000","last_became_peered":"2023-03-03T05:45:35.860993+0000","last_unstale":"2023-03-03T05:45:35.962683+0000","last_undegraded":"2023-03-03T05:45:35.962683+0000","last_fullsized":"2023-03-03T05:45:35.962683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853383+0000","last_change":"2023-03-03T05:45:35.857703+0000","last_active":"2023-03-03T05:45:36.853383+0000","last_peered":"2023-03-03T05:45:36.853383+0000","last_clean":"2023-03-03T05:45:36.853383+0000","last_became_active":"2023-03-03T05:45:35.857421+0000","last_became_peered":"2023-03-03T05:45:35.857421+0000","last_unstale":"2023-03-03T05:45:36.853383+0000","last_undegraded":"2023-03-03T05:45:36.853383+0000","last_fullsized":"2023-03-03T05:45:36.853383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851248+0000","last_change":"2023-03-03T05:45:35.859892+0000","last_active":"2023-03-03T05:45:36.851248+0000","last_peered":"2023-03-03T05:45:36.851248+0000","last_clean":"2023-03-03T05:45:36.851248+0000","last_became_active":"2023-03-03T05:45:35.859617+0000","last_became_peered":"2023-03-03T05:45:35.859617+0000","last_unstale":"2023-03-03T05:45:36.851248+0000","last_undegraded":"2023-03-03T05:45:36.851248+0000","last_fullsized":"2023-03-03T05:45:36.851248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:23:53.405094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850434+0000","last_change":"2023-03-03T05:45:35.862939+0000","last_active":"2023-03-03T05:45:36.850434+0000","last_peered":"2023-03-03T05:45:36.850434+0000","last_clean":"2023-03-03T05:45:36.850434+0000","last_became_active":"2023-03-03T05:45:35.862681+0000","last_became_peered":"2023-03-03T05:45:35.862681+0000","last_unstale":"2023-03-03T05:45:36.850434+0000","last_undegraded":"2023-03-03T05:45:36.850434+0000","last_fullsized":"2023-03-03T05:45:36.850434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:58.535750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923559+0000","last_change":"2023-03-03T05:45:35.866302+0000","last_active":"2023-03-03T05:45:36.923559+0000","last_peered":"2023-03-03T05:45:36.923559+0000","last_clean":"2023-03-03T05:45:36.923559+0000","last_became_active":"2023-03-03T05:45:35.865627+0000","last_became_peered":"2023-03-03T05:45:35.865627+0000","last_unstale":"2023-03-03T05:45:36.923559+0000","last_undegraded":"2023-03-03T05:45:36.923559+0000","last_fullsized":"2023-03-03T05:45:36.923559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853588+0000","last_change":"2023-03-03T05:45:35.856762+0000","last_active":"2023-03-03T05:45:36.853588+0000","last_peered":"2023-03-03T05:45:36.853588+0000","last_clean":"2023-03-03T05:45:36.853588+0000","last_became_active":"2023-03-03T05:45:35.856458+0000","last_became_peered":"2023-03-03T05:45:35.856458+0000","last_unstale":"2023-03-03T05:45:36.853588+0000","last_undegraded":"2023-03-03T05:45:36.853588+0000","last_fullsized":"2023-03-03T05:45:36.853588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923093+0000","last_change":"2023-03-03T05:45:35.865258+0000","last_active":"2023-03-03T05:45:36.923093+0000","last_peered":"2023-03-03T05:45:36.923093+0000","last_clean":"2023-03-03T05:45:36.923093+0000","last_became_active":"2023-03-03T05:45:35.864889+0000","last_became_peered":"2023-03-03T05:45:35.864889+0000","last_unstale":"2023-03-03T05:45:36.923093+0000","last_undegraded":"2023-03-03T05:45:36.923093+0000","last_fullsized":"2023-03-03T05:45:36.923093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923368+0000","last_change":"2023-03-03T05:45:35.868970+0000","last_active":"2023-03-03T05:45:36.923368+0000","last_peered":"2023-03-03T05:45:36.923368+0000","last_clean":"2023-03-03T05:45:36.923368+0000","last_became_active":"2023-03-03T05:45:35.868555+0000","last_became_peered":"2023-03-03T05:45:35.868555+0000","last_unstale":"2023-03-03T05:45:36.923368+0000","last_undegraded":"2023-03-03T05:45:36.923368+0000","last_fullsized":"2023-03-03T05:45:36.923368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922618+0000","last_change":"2023-03-03T05:45:35.855291+0000","last_active":"2023-03-03T05:45:36.922618+0000","last_peered":"2023-03-03T05:45:36.922618+0000","last_clean":"2023-03-03T05:45:36.922618+0000","last_became_active":"2023-03-03T05:45:35.854925+0000","last_became_peered":"2023-03-03T05:45:35.854925+0000","last_unstale":"2023-03-03T05:45:36.922618+0000","last_undegraded":"2023-03-03T05:45:36.922618+0000","last_fullsized":"2023-03-03T05:45:36.922618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.924177+0000","last_change":"2023-03-03T05:45:35.853479+0000","last_active":"2023-03-03T05:45:36.924177+0000","last_peered":"2023-03-03T05:45:36.924177+0000","last_clean":"2023-03-03T05:45:36.924177+0000","last_became_active":"2023-03-03T05:45:35.853193+0000","last_became_peered":"2023-03-03T05:45:35.853193+0000","last_unstale":"2023-03-03T05:45:36.924177+0000","last_undegraded":"2023-03-03T05:45:36.924177+0000","last_fullsized":"2023-03-03T05:45:36.924177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923036+0000","last_change":"2023-03-03T05:45:35.865976+0000","last_active":"2023-03-03T05:45:36.923036+0000","last_peered":"2023-03-03T05:45:36.923036+0000","last_clean":"2023-03-03T05:45:36.923036+0000","last_became_active":"2023-03-03T05:45:35.865561+0000","last_became_peered":"2023-03-03T05:45:35.865561+0000","last_unstale":"2023-03-03T05:45:36.923036+0000","last_undegraded":"2023-03-03T05:45:36.923036+0000","last_fullsized":"2023-03-03T05:45:36.923036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962602+0000","last_change":"2023-03-03T05:45:35.862593+0000","last_active":"2023-03-03T05:45:35.962602+0000","last_peered":"2023-03-03T05:45:35.962602+0000","last_clean":"2023-03-03T05:45:35.962602+0000","last_became_active":"2023-03-03T05:45:35.862342+0000","last_became_peered":"2023-03-03T05:45:35.862342+0000","last_unstale":"2023-03-03T05:45:35.962602+0000","last_undegraded":"2023-03-03T05:45:35.962602+0000","last_fullsized":"2023-03-03T05:45:35.962602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850467+0000","last_change":"2023-03-03T05:45:35.857623+0000","last_active":"2023-03-03T05:45:36.850467+0000","last_peered":"2023-03-03T05:45:36.850467+0000","last_clean":"2023-03-03T05:45:36.850467+0000","last_became_active":"2023-03-03T05:45:35.857357+0000","last_became_peered":"2023-03-03T05:45:35.857357+0000","last_unstale":"2023-03-03T05:45:36.850467+0000","last_undegraded":"2023-03-03T05:45:36.850467+0000","last_fullsized":"2023-03-03T05:45:36.850467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:00.850961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923102+0000","last_change":"2023-03-03T05:45:35.859115+0000","last_active":"2023-03-03T05:45:36.923102+0000","last_peered":"2023-03-03T05:45:36.923102+0000","last_clean":"2023-03-03T05:45:36.923102+0000","last_became_active":"2023-03-03T05:45:35.858794+0000","last_became_peered":"2023-03-03T05:45:35.858794+0000","last_unstale":"2023-03-03T05:45:36.923102+0000","last_undegraded":"2023-03-03T05:45:36.923102+0000","last_fullsized":"2023-03-03T05:45:36.923102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962764+0000","last_change":"2023-03-03T05:45:35.861090+0000","last_active":"2023-03-03T05:45:35.962764+0000","last_peered":"2023-03-03T05:45:35.962764+0000","last_clean":"2023-03-03T05:45:35.962764+0000","last_became_active":"2023-03-03T05:45:35.860814+0000","last_became_peered":"2023-03-03T05:45:35.860814+0000","last_unstale":"2023-03-03T05:45:35.962764+0000","last_undegraded":"2023-03-03T05:45:35.962764+0000","last_fullsized":"2023-03-03T05:45:35.962764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923139+0000","last_change":"2023-03-03T05:45:35.855611+0000","last_active":"2023-03-03T05:45:36.923139+0000","last_peered":"2023-03-03T05:45:36.923139+0000","last_clean":"2023-03-03T05:45:36.923139+0000","last_became_active":"2023-03-03T05:45:35.855333+0000","last_became_peered":"2023-03-03T05:45:35.855333+0000","last_unstale":"2023-03-03T05:45:36.923139+0000","last_undegraded":"2023-03-03T05:45:36.923139+0000","last_fullsized":"2023-03-03T05:45:36.923139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923216+0000","last_change":"2023-03-03T05:45:35.865135+0000","last_active":"2023-03-03T05:45:36.923216+0000","last_peered":"2023-03-03T05:45:36.923216+0000","last_clean":"2023-03-03T05:45:36.923216+0000","last_became_active":"2023-03-03T05:45:35.864823+0000","last_became_peered":"2023-03-03T05:45:35.864823+0000","last_unstale":"2023-03-03T05:45:36.923216+0000","last_undegraded":"2023-03-03T05:45:36.923216+0000","last_fullsized":"2023-03-03T05:45:36.923216+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923121+0000","last_change":"2023-03-03T05:45:35.864573+0000","last_active":"2023-03-03T05:45:36.923121+0000","last_peered":"2023-03-03T05:45:36.923121+0000","last_clean":"2023-03-03T05:45:36.923121+0000","last_became_active":"2023-03-03T05:45:35.864297+0000","last_became_peered":"2023-03-03T05:45:35.864297+0000","last_unstale":"2023-03-03T05:45:36.923121+0000","last_undegraded":"2023-03-03T05:45:36.923121+0000","last_fullsized":"2023-03-03T05:45:36.923121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922982+0000","last_change":"2023-03-03T05:45:35.862720+0000","last_active":"2023-03-03T05:45:36.922982+0000","last_peered":"2023-03-03T05:45:36.922982+0000","last_clean":"2023-03-03T05:45:36.922982+0000","last_became_active":"2023-03-03T05:45:35.861282+0000","last_became_peered":"2023-03-03T05:45:35.861282+0000","last_unstale":"2023-03-03T05:45:36.922982+0000","last_undegraded":"2023-03-03T05:45:36.922982+0000","last_fullsized":"2023-03-03T05:45:36.922982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853407+0000","last_change":"2023-03-03T05:45:35.859828+0000","last_active":"2023-03-03T05:45:36.853407+0000","last_peered":"2023-03-03T05:45:36.853407+0000","last_clean":"2023-03-03T05:45:36.853407+0000","last_became_active":"2023-03-03T05:45:35.859568+0000","last_became_peered":"2023-03-03T05:45:35.859568+0000","last_unstale":"2023-03-03T05:45:36.853407+0000","last_undegraded":"2023-03-03T05:45:36.853407+0000","last_fullsized":"2023-03-03T05:45:36.853407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923461+0000","last_change":"2023-03-03T05:45:35.864435+0000","last_active":"2023-03-03T05:45:36.923461+0000","last_peered":"2023-03-03T05:45:36.923461+0000","last_clean":"2023-03-03T05:45:36.923461+0000","last_became_active":"2023-03-03T05:45:35.864129+0000","last_became_peered":"2023-03-03T05:45:35.864129+0000","last_unstale":"2023-03-03T05:45:36.923461+0000","last_undegraded":"2023-03-03T05:45:36.923461+0000","last_fullsized":"2023-03-03T05:45:36.923461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853682+0000","last_change":"2023-03-03T05:45:35.855909+0000","last_active":"2023-03-03T05:45:36.853682+0000","last_peered":"2023-03-03T05:45:36.853682+0000","last_clean":"2023-03-03T05:45:36.853682+0000","last_became_active":"2023-03-03T05:45:35.855388+0000","last_became_peered":"2023-03-03T05:45:35.855388+0000","last_unstale":"2023-03-03T05:45:36.853682+0000","last_undegraded":"2023-03-03T05:45:36.853682+0000","last_fullsized":"2023-03-03T05:45:36.853682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923513+0000","last_change":"2023-03-03T05:45:35.863167+0000","last_active":"2023-03-03T05:45:36.923513+0000","last_peered":"2023-03-03T05:45:36.923513+0000","last_clean":"2023-03-03T05:45:36.923513+0000","last_became_active":"2023-03-03T05:45:35.862519+0000","last_became_peered":"2023-03-03T05:45:35.862519+0000","last_unstale":"2023-03-03T05:45:36.923513+0000","last_undegraded":"2023-03-03T05:45:36.923513+0000","last_fullsized":"2023-03-03T05:45:36.923513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851022+0000","last_change":"2023-03-03T05:45:35.870150+0000","last_active":"2023-03-03T05:45:36.851022+0000","last_peered":"2023-03-03T05:45:36.851022+0000","last_clean":"2023-03-03T05:45:36.851022+0000","last_became_active":"2023-03-03T05:45:35.868786+0000","last_became_peered":"2023-03-03T05:45:35.868786+0000","last_unstale":"2023-03-03T05:45:36.851022+0000","last_undegraded":"2023-03-03T05:45:36.851022+0000","last_fullsized":"2023-03-03T05:45:36.851022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:52:23.931405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.310449+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.310197+0000","last_became_peered":"2023-03-03T05:45:35.310197+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923377+0000","last_change":"2023-03-03T05:45:35.861444+0000","last_active":"2023-03-03T05:45:36.923377+0000","last_peered":"2023-03-03T05:45:36.923377+0000","last_clean":"2023-03-03T05:45:36.923377+0000","last_became_active":"2023-03-03T05:45:35.861129+0000","last_became_peered":"2023-03-03T05:45:35.861129+0000","last_unstale":"2023-03-03T05:45:36.923377+0000","last_undegraded":"2023-03-03T05:45:36.923377+0000","last_fullsized":"2023-03-03T05:45:36.923377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843345+0000","last_change":"2023-03-03T05:45:35.308492+0000","last_active":"2023-03-03T05:45:35.843345+0000","last_peered":"2023-03-03T05:45:35.843345+0000","last_clean":"2023-03-03T05:45:35.843345+0000","last_became_active":"2023-03-03T05:45:35.308276+0000","last_became_peered":"2023-03-03T05:45:35.308276+0000","last_unstale":"2023-03-03T05:45:35.843345+0000","last_undegraded":"2023-03-03T05:45:35.843345+0000","last_fullsized":"2023-03-03T05:45:35.843345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:32.932001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853620+0000","last_change":"2023-03-03T05:45:35.866408+0000","last_active":"2023-03-03T05:45:36.853620+0000","last_peered":"2023-03-03T05:45:36.853620+0000","last_clean":"2023-03-03T05:45:36.853620+0000","last_became_active":"2023-03-03T05:45:35.866080+0000","last_became_peered":"2023-03-03T05:45:35.866080+0000","last_unstale":"2023-03-03T05:45:36.853620+0000","last_undegraded":"2023-03-03T05:45:36.853620+0000","last_fullsized":"2023-03-03T05:45:36.853620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:22.678556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848138+0000","last_change":"2023-03-03T05:45:35.293990+0000","last_active":"2023-03-03T05:45:35.848138+0000","last_peered":"2023-03-03T05:45:35.848138+0000","last_clean":"2023-03-03T05:45:35.848138+0000","last_became_active":"2023-03-03T05:45:35.293655+0000","last_became_peered":"2023-03-03T05:45:35.293655+0000","last_unstale":"2023-03-03T05:45:35.848138+0000","last_undegraded":"2023-03-03T05:45:35.848138+0000","last_fullsized":"2023-03-03T05:45:35.848138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850700+0000","last_change":"2023-03-03T05:45:35.852599+0000","last_active":"2023-03-03T05:45:36.850700+0000","last_peered":"2023-03-03T05:45:36.850700+0000","last_clean":"2023-03-03T05:45:36.850700+0000","last_became_active":"2023-03-03T05:45:35.852292+0000","last_became_peered":"2023-03-03T05:45:35.852292+0000","last_unstale":"2023-03-03T05:45:36.850700+0000","last_undegraded":"2023-03-03T05:45:36.850700+0000","last_fullsized":"2023-03-03T05:45:36.850700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:06.104140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848641+0000","last_change":"2023-03-03T05:45:35.297718+0000","last_active":"2023-03-03T05:45:35.848641+0000","last_peered":"2023-03-03T05:45:35.848641+0000","last_clean":"2023-03-03T05:45:35.848641+0000","last_became_active":"2023-03-03T05:45:35.297435+0000","last_became_peered":"2023-03-03T05:45:35.297435+0000","last_unstale":"2023-03-03T05:45:35.848641+0000","last_undegraded":"2023-03-03T05:45:35.848641+0000","last_fullsized":"2023-03-03T05:45:35.848641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:17:39.778447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923577+0000","last_change":"2023-03-03T05:45:35.871116+0000","last_active":"2023-03-03T05:45:36.923577+0000","last_peered":"2023-03-03T05:45:36.923577+0000","last_clean":"2023-03-03T05:45:36.923577+0000","last_became_active":"2023-03-03T05:45:35.870843+0000","last_became_peered":"2023-03-03T05:45:35.870843+0000","last_unstale":"2023-03-03T05:45:36.923577+0000","last_undegraded":"2023-03-03T05:45:36.923577+0000","last_fullsized":"2023-03-03T05:45:36.923577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851170+0000","last_change":"2023-03-03T05:45:35.304400+0000","last_active":"2023-03-03T05:45:36.851170+0000","last_peered":"2023-03-03T05:45:36.851170+0000","last_clean":"2023-03-03T05:45:36.851170+0000","last_became_active":"2023-03-03T05:45:35.304093+0000","last_became_peered":"2023-03-03T05:45:35.304093+0000","last_unstale":"2023-03-03T05:45:36.851170+0000","last_undegraded":"2023-03-03T05:45:36.851170+0000","last_fullsized":"2023-03-03T05:45:36.851170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853210+0000","last_change":"2023-03-03T05:45:35.862492+0000","last_active":"2023-03-03T05:45:36.853210+0000","last_peered":"2023-03-03T05:45:36.853210+0000","last_clean":"2023-03-03T05:45:36.853210+0000","last_became_active":"2023-03-03T05:45:35.862173+0000","last_became_peered":"2023-03-03T05:45:35.862173+0000","last_unstale":"2023-03-03T05:45:36.853210+0000","last_undegraded":"2023-03-03T05:45:36.853210+0000","last_fullsized":"2023-03-03T05:45:36.853210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.853604+0000","last_change":"2023-03-03T05:45:35.302282+0000","last_active":"2023-03-03T05:45:40.853604+0000","last_peered":"2023-03-03T05:45:40.853604+0000","last_clean":"2023-03-03T05:45:40.853604+0000","last_became_active":"2023-03-03T05:45:35.302002+0000","last_became_peered":"2023-03-03T05:45:35.302002+0000","last_unstale":"2023-03-03T05:45:40.853604+0000","last_undegraded":"2023-03-03T05:45:40.853604+0000","last_fullsized":"2023-03-03T05:45:40.853604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853414+0000","last_change":"2023-03-03T05:45:35.855928+0000","last_active":"2023-03-03T05:45:36.853414+0000","last_peered":"2023-03-03T05:45:36.853414+0000","last_clean":"2023-03-03T05:45:36.853414+0000","last_became_active":"2023-03-03T05:45:35.855612+0000","last_became_peered":"2023-03-03T05:45:35.855612+0000","last_unstale":"2023-03-03T05:45:36.853414+0000","last_undegraded":"2023-03-03T05:45:36.853414+0000","last_fullsized":"2023-03-03T05:45:36.853414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:57.313631+0000","last_change":"2023-03-03T05:45:35.304856+0000","last_active":"2023-03-03T05:45:57.313631+0000","last_peered":"2023-03-03T05:45:57.313631+0000","last_clean":"2023-03-03T05:45:57.313631+0000","last_became_active":"2023-03-03T05:45:35.304439+0000","last_became_peered":"2023-03-03T05:45:35.304439+0000","last_unstale":"2023-03-03T05:45:57.313631+0000","last_undegraded":"2023-03-03T05:45:57.313631+0000","last_fullsized":"2023-03-03T05:45:57.313631+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:30.692527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1807,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850547+0000","last_change":"2023-03-03T05:45:35.856887+0000","last_active":"2023-03-03T05:45:36.850547+0000","last_peered":"2023-03-03T05:45:36.850547+0000","last_clean":"2023-03-03T05:45:36.850547+0000","last_became_active":"2023-03-03T05:45:35.856498+0000","last_became_peered":"2023-03-03T05:45:35.856498+0000","last_unstale":"2023-03-03T05:45:36.850547+0000","last_undegraded":"2023-03-03T05:45:36.850547+0000","last_fullsized":"2023-03-03T05:45:36.850547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858066+0000","last_change":"2023-03-03T05:45:35.297635+0000","last_active":"2023-03-03T05:45:40.858066+0000","last_peered":"2023-03-03T05:45:40.858066+0000","last_clean":"2023-03-03T05:45:40.858066+0000","last_became_active":"2023-03-03T05:45:35.297373+0000","last_became_peered":"2023-03-03T05:45:35.297373+0000","last_unstale":"2023-03-03T05:45:40.858066+0000","last_undegraded":"2023-03-03T05:45:40.858066+0000","last_fullsized":"2023-03-03T05:45:40.858066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:48.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922855+0000","last_change":"2023-03-03T05:45:35.852113+0000","last_active":"2023-03-03T05:45:36.922855+0000","last_peered":"2023-03-03T05:45:36.922855+0000","last_clean":"2023-03-03T05:45:36.922855+0000","last_became_active":"2023-03-03T05:45:35.851799+0000","last_became_peered":"2023-03-03T05:45:35.851799+0000","last_unstale":"2023-03-03T05:45:36.922855+0000","last_undegraded":"2023-03-03T05:45:36.922855+0000","last_fullsized":"2023-03-03T05:45:36.922855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.845615+0000","last_change":"2023-03-03T05:45:35.292083+0000","last_active":"2023-03-03T05:45:35.845615+0000","last_peered":"2023-03-03T05:45:35.845615+0000","last_clean":"2023-03-03T05:45:35.845615+0000","last_became_active":"2023-03-03T05:45:35.291766+0000","last_became_peered":"2023-03-03T05:45:35.291766+0000","last_unstale":"2023-03-03T05:45:35.845615+0000","last_undegraded":"2023-03-03T05:45:35.845615+0000","last_fullsized":"2023-03-03T05:45:35.845615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851348+0000","last_change":"2023-03-03T05:45:35.870363+0000","last_active":"2023-03-03T05:45:36.851348+0000","last_peered":"2023-03-03T05:45:36.851348+0000","last_clean":"2023-03-03T05:45:36.851348+0000","last_became_active":"2023-03-03T05:45:35.868801+0000","last_became_peered":"2023-03-03T05:45:35.868801+0000","last_unstale":"2023-03-03T05:45:36.851348+0000","last_undegraded":"2023-03-03T05:45:36.851348+0000","last_fullsized":"2023-03-03T05:45:36.851348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:04.045341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844327+0000","last_change":"2023-03-03T05:45:35.304592+0000","last_active":"2023-03-03T05:45:35.844327+0000","last_peered":"2023-03-03T05:45:35.844327+0000","last_clean":"2023-03-03T05:45:35.844327+0000","last_became_active":"2023-03-03T05:45:35.304318+0000","last_became_peered":"2023-03-03T05:45:35.304318+0000","last_unstale":"2023-03-03T05:45:35.844327+0000","last_undegraded":"2023-03-03T05:45:35.844327+0000","last_fullsized":"2023-03-03T05:45:35.844327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:08.368039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922933+0000","last_change":"2023-03-03T05:45:35.857951+0000","last_active":"2023-03-03T05:45:36.922933+0000","last_peered":"2023-03-03T05:45:36.922933+0000","last_clean":"2023-03-03T05:45:36.922933+0000","last_became_active":"2023-03-03T05:45:35.857581+0000","last_became_peered":"2023-03-03T05:45:35.857581+0000","last_unstale":"2023-03-03T05:45:36.922933+0000","last_undegraded":"2023-03-03T05:45:36.922933+0000","last_fullsized":"2023-03-03T05:45:36.922933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858611+0000","last_change":"2023-03-03T05:45:35.309150+0000","last_active":"2023-03-03T05:45:40.858611+0000","last_peered":"2023-03-03T05:45:40.858611+0000","last_clean":"2023-03-03T05:45:40.858611+0000","last_became_active":"2023-03-03T05:45:35.308840+0000","last_became_peered":"2023-03-03T05:45:35.308840+0000","last_unstale":"2023-03-03T05:45:40.858611+0000","last_undegraded":"2023-03-03T05:45:40.858611+0000","last_fullsized":"2023-03-03T05:45:40.858611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923050+0000","last_change":"2023-03-03T05:45:35.864919+0000","last_active":"2023-03-03T05:45:36.923050+0000","last_peered":"2023-03-03T05:45:36.923050+0000","last_clean":"2023-03-03T05:45:36.923050+0000","last_became_active":"2023-03-03T05:45:35.864595+0000","last_became_peered":"2023-03-03T05:45:35.864595+0000","last_unstale":"2023-03-03T05:45:36.923050+0000","last_undegraded":"2023-03-03T05:45:36.923050+0000","last_fullsized":"2023-03-03T05:45:36.923050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.308976+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.308707+0000","last_became_peered":"2023-03-03T05:45:35.308707+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850640+0000","last_change":"2023-03-03T05:45:35.868861+0000","last_active":"2023-03-03T05:45:36.850640+0000","last_peered":"2023-03-03T05:45:36.850640+0000","last_clean":"2023-03-03T05:45:36.850640+0000","last_became_active":"2023-03-03T05:45:35.868438+0000","last_became_peered":"2023-03-03T05:45:35.868438+0000","last_unstale":"2023-03-03T05:45:36.850640+0000","last_undegraded":"2023-03-03T05:45:36.850640+0000","last_fullsized":"2023-03-03T05:45:36.850640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:59.810575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.854269+0000","last_change":"2023-03-03T05:45:35.303791+0000","last_active":"2023-03-03T05:45:40.854269+0000","last_peered":"2023-03-03T05:45:40.854269+0000","last_clean":"2023-03-03T05:45:40.854269+0000","last_became_active":"2023-03-03T05:45:35.303482+0000","last_became_peered":"2023-03-03T05:45:35.303482+0000","last_unstale":"2023-03-03T05:45:40.854269+0000","last_undegraded":"2023-03-03T05:45:40.854269+0000","last_fullsized":"2023-03-03T05:45:40.854269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:31.933113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923037+0000","last_change":"2023-03-03T05:45:35.864783+0000","last_active":"2023-03-03T05:45:36.923037+0000","last_peered":"2023-03-03T05:45:36.923037+0000","last_clean":"2023-03-03T05:45:36.923037+0000","last_became_active":"2023-03-03T05:45:35.864462+0000","last_became_peered":"2023-03-03T05:45:35.864462+0000","last_unstale":"2023-03-03T05:45:36.923037+0000","last_undegraded":"2023-03-03T05:45:36.923037+0000","last_fullsized":"2023-03-03T05:45:36.923037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851096+0000","last_change":"2023-03-03T05:45:35.310813+0000","last_active":"2023-03-03T05:45:36.851096+0000","last_peered":"2023-03-03T05:45:36.851096+0000","last_clean":"2023-03-03T05:45:36.851096+0000","last_became_active":"2023-03-03T05:45:35.310466+0000","last_became_peered":"2023-03-03T05:45:35.310466+0000","last_unstale":"2023-03-03T05:45:36.851096+0000","last_undegraded":"2023-03-03T05:45:36.851096+0000","last_fullsized":"2023-03-03T05:45:36.851096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:46.322139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850472+0000","last_change":"2023-03-03T05:45:35.856376+0000","last_active":"2023-03-03T05:45:36.850472+0000","last_peered":"2023-03-03T05:45:36.850472+0000","last_clean":"2023-03-03T05:45:36.850472+0000","last_became_active":"2023-03-03T05:45:35.856064+0000","last_became_peered":"2023-03-03T05:45:35.856064+0000","last_unstale":"2023-03-03T05:45:36.850472+0000","last_undegraded":"2023-03-03T05:45:36.850472+0000","last_fullsized":"2023-03-03T05:45:36.850472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962714+0000","last_change":"2023-03-03T05:45:35.295520+0000","last_active":"2023-03-03T05:45:35.962714+0000","last_peered":"2023-03-03T05:45:35.962714+0000","last_clean":"2023-03-03T05:45:35.962714+0000","last_became_active":"2023-03-03T05:45:35.295203+0000","last_became_peered":"2023-03-03T05:45:35.295203+0000","last_unstale":"2023-03-03T05:45:35.962714+0000","last_undegraded":"2023-03-03T05:45:35.962714+0000","last_fullsized":"2023-03-03T05:45:35.962714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923389+0000","last_change":"2023-03-03T05:45:35.852242+0000","last_active":"2023-03-03T05:45:36.923389+0000","last_peered":"2023-03-03T05:45:36.923389+0000","last_clean":"2023-03-03T05:45:36.923389+0000","last_became_active":"2023-03-03T05:45:35.851803+0000","last_became_peered":"2023-03-03T05:45:35.851803+0000","last_unstale":"2023-03-03T05:45:36.923389+0000","last_undegraded":"2023-03-03T05:45:36.923389+0000","last_fullsized":"2023-03-03T05:45:36.923389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:14.473334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.852939+0000","last_change":"2023-03-03T05:45:35.298542+0000","last_active":"2023-03-03T05:45:40.852939+0000","last_peered":"2023-03-03T05:45:40.852939+0000","last_clean":"2023-03-03T05:45:40.852939+0000","last_became_active":"2023-03-03T05:45:35.298219+0000","last_became_peered":"2023-03-03T05:45:35.298219+0000","last_unstale":"2023-03-03T05:45:40.852939+0000","last_undegraded":"2023-03-03T05:45:40.852939+0000","last_fullsized":"2023-03-03T05:45:40.852939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962699+0000","last_change":"2023-03-03T05:45:35.861231+0000","last_active":"2023-03-03T05:45:35.962699+0000","last_peered":"2023-03-03T05:45:35.962699+0000","last_clean":"2023-03-03T05:45:35.962699+0000","last_became_active":"2023-03-03T05:45:35.860675+0000","last_became_peered":"2023-03-03T05:45:35.860675+0000","last_unstale":"2023-03-03T05:45:35.962699+0000","last_undegraded":"2023-03-03T05:45:35.962699+0000","last_fullsized":"2023-03-03T05:45:35.962699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843762+0000","last_change":"2023-03-03T05:45:35.312143+0000","last_active":"2023-03-03T05:45:35.843762+0000","last_peered":"2023-03-03T05:45:35.843762+0000","last_clean":"2023-03-03T05:45:35.843762+0000","last_became_active":"2023-03-03T05:45:35.311848+0000","last_became_peered":"2023-03-03T05:45:35.311848+0000","last_unstale":"2023-03-03T05:45:35.843762+0000","last_undegraded":"2023-03-03T05:45:35.843762+0000","last_fullsized":"2023-03-03T05:45:35.843762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:22.294783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851303+0000","last_change":"2023-03-03T05:45:35.869075+0000","last_active":"2023-03-03T05:45:36.851303+0000","last_peered":"2023-03-03T05:45:36.851303+0000","last_clean":"2023-03-03T05:45:36.851303+0000","last_became_active":"2023-03-03T05:45:35.868413+0000","last_became_peered":"2023-03-03T05:45:35.868413+0000","last_unstale":"2023-03-03T05:45:36.851303+0000","last_undegraded":"2023-03-03T05:45:36.851303+0000","last_fullsized":"2023-03-03T05:45:36.851303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:34.743111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844226+0000","last_change":"2023-03-03T05:45:35.296611+0000","last_active":"2023-03-03T05:45:35.844226+0000","last_peered":"2023-03-03T05:45:35.844226+0000","last_clean":"2023-03-03T05:45:35.844226+0000","last_became_active":"2023-03-03T05:45:35.296294+0000","last_became_peered":"2023-03-03T05:45:35.296294+0000","last_unstale":"2023-03-03T05:45:35.844226+0000","last_undegraded":"2023-03-03T05:45:35.844226+0000","last_fullsized":"2023-03-03T05:45:35.844226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:16.655836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850964+0000","last_change":"2023-03-03T05:45:35.857451+0000","last_active":"2023-03-03T05:45:36.850964+0000","last_peered":"2023-03-03T05:45:36.850964+0000","last_clean":"2023-03-03T05:45:36.850964+0000","last_became_active":"2023-03-03T05:45:35.857087+0000","last_became_peered":"2023-03-03T05:45:35.857087+0000","last_unstale":"2023-03-03T05:45:36.850964+0000","last_undegraded":"2023-03-03T05:45:36.850964+0000","last_fullsized":"2023-03-03T05:45:36.850964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842797+0000","last_change":"2023-03-03T05:45:35.296114+0000","last_active":"2023-03-03T05:45:35.842797+0000","last_peered":"2023-03-03T05:45:35.842797+0000","last_clean":"2023-03-03T05:45:35.842797+0000","last_became_active":"2023-03-03T05:45:35.295855+0000","last_became_peered":"2023-03-03T05:45:35.295855+0000","last_unstale":"2023-03-03T05:45:35.842797+0000","last_undegraded":"2023-03-03T05:45:35.842797+0000","last_fullsized":"2023-03-03T05:45:35.842797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923246+0000","last_change":"2023-03-03T05:45:35.862249+0000","last_active":"2023-03-03T05:45:36.923246+0000","last_peered":"2023-03-03T05:45:36.923246+0000","last_clean":"2023-03-03T05:45:36.923246+0000","last_became_active":"2023-03-03T05:45:35.861773+0000","last_became_peered":"2023-03-03T05:45:35.861773+0000","last_unstale":"2023-03-03T05:45:36.923246+0000","last_undegraded":"2023-03-03T05:45:36.923246+0000","last_fullsized":"2023-03-03T05:45:36.923246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848485+0000","last_change":"2023-03-03T05:45:29.189033+0000","last_active":"2023-03-03T05:45:35.848485+0000","last_peered":"2023-03-03T05:45:35.848485+0000","last_clean":"2023-03-03T05:45:35.848485+0000","last_became_active":"2023-03-03T05:45:29.188719+0000","last_became_peered":"2023-03-03T05:45:29.188719+0000","last_unstale":"2023-03-03T05:45:35.848485+0000","last_undegraded":"2023-03-03T05:45:35.848485+0000","last_fullsized":"2023-03-03T05:45:35.848485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850356+0000","last_change":"2023-03-03T05:45:35.303756+0000","last_active":"2023-03-03T05:45:36.850356+0000","last_peered":"2023-03-03T05:45:36.850356+0000","last_clean":"2023-03-03T05:45:36.850356+0000","last_became_active":"2023-03-03T05:45:35.303448+0000","last_became_peered":"2023-03-03T05:45:35.303448+0000","last_unstale":"2023-03-03T05:45:36.850356+0000","last_undegraded":"2023-03-03T05:45:36.850356+0000","last_fullsized":"2023-03-03T05:45:36.850356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922899+0000","last_change":"2023-03-03T05:45:35.854166+0000","last_active":"2023-03-03T05:45:36.922899+0000","last_peered":"2023-03-03T05:45:36.922899+0000","last_clean":"2023-03-03T05:45:36.922899+0000","last_became_active":"2023-03-03T05:45:35.853873+0000","last_became_peered":"2023-03-03T05:45:35.853873+0000","last_unstale":"2023-03-03T05:45:36.922899+0000","last_undegraded":"2023-03-03T05:45:36.922899+0000","last_fullsized":"2023-03-03T05:45:36.922899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843699+0000","last_change":"2023-03-03T05:45:29.189075+0000","last_active":"2023-03-03T05:45:35.843699+0000","last_peered":"2023-03-03T05:45:35.843699+0000","last_clean":"2023-03-03T05:45:35.843699+0000","last_became_active":"2023-03-03T05:45:29.188778+0000","last_became_peered":"2023-03-03T05:45:29.188778+0000","last_unstale":"2023-03-03T05:45:35.843699+0000","last_undegraded":"2023-03-03T05:45:35.843699+0000","last_fullsized":"2023-03-03T05:45:35.843699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:54:35.536172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962645+0000","last_change":"2023-03-03T05:45:35.309781+0000","last_active":"2023-03-03T05:45:35.962645+0000","last_peered":"2023-03-03T05:45:35.962645+0000","last_clean":"2023-03-03T05:45:35.962645+0000","last_became_active":"2023-03-03T05:45:35.309497+0000","last_became_peered":"2023-03-03T05:45:35.309497+0000","last_unstale":"2023-03-03T05:45:35.962645+0000","last_undegraded":"2023-03-03T05:45:35.962645+0000","last_fullsized":"2023-03-03T05:45:35.962645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923571+0000","last_change":"2023-03-03T05:45:35.854721+0000","last_active":"2023-03-03T05:45:36.923571+0000","last_peered":"2023-03-03T05:45:36.923571+0000","last_clean":"2023-03-03T05:45:36.923571+0000","last_became_active":"2023-03-03T05:45:35.854423+0000","last_became_peered":"2023-03-03T05:45:35.854423+0000","last_unstale":"2023-03-03T05:45:36.923571+0000","last_undegraded":"2023-03-03T05:45:36.923571+0000","last_fullsized":"2023-03-03T05:45:36.923571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.852125+0000","last_change":"2023-03-03T05:45:29.057308+0000","last_active":"2023-03-03T05:45:35.852125+0000","last_peered":"2023-03-03T05:45:35.852125+0000","last_clean":"2023-03-03T05:45:35.852125+0000","last_became_active":"2023-03-03T05:45:29.056828+0000","last_became_peered":"2023-03-03T05:45:29.056828+0000","last_unstale":"2023-03-03T05:45:35.852125+0000","last_undegraded":"2023-03-03T05:45:35.852125+0000","last_fullsized":"2023-03-03T05:45:35.852125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:56.079893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848095+0000","last_change":"2023-03-03T05:45:35.295842+0000","last_active":"2023-03-03T05:45:35.848095+0000","last_peered":"2023-03-03T05:45:35.848095+0000","last_clean":"2023-03-03T05:45:35.848095+0000","last_became_active":"2023-03-03T05:45:35.295545+0000","last_became_peered":"2023-03-03T05:45:35.295545+0000","last_unstale":"2023-03-03T05:45:35.848095+0000","last_undegraded":"2023-03-03T05:45:35.848095+0000","last_fullsized":"2023-03-03T05:45:35.848095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923636+0000","last_change":"2023-03-03T05:45:35.862437+0000","last_active":"2023-03-03T05:45:36.923636+0000","last_peered":"2023-03-03T05:45:36.923636+0000","last_clean":"2023-03-03T05:45:36.923636+0000","last_became_active":"2023-03-03T05:45:35.862152+0000","last_became_peered":"2023-03-03T05:45:35.862152+0000","last_unstale":"2023-03-03T05:45:36.923636+0000","last_undegraded":"2023-03-03T05:45:36.923636+0000","last_fullsized":"2023-03-03T05:45:36.923636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842862+0000","last_change":"2023-03-03T05:45:29.057646+0000","last_active":"2023-03-03T05:45:35.842862+0000","last_peered":"2023-03-03T05:45:35.842862+0000","last_clean":"2023-03-03T05:45:35.842862+0000","last_became_active":"2023-03-03T05:45:29.057330+0000","last_became_peered":"2023-03-03T05:45:29.057330+0000","last_unstale":"2023-03-03T05:45:35.842862+0000","last_undegraded":"2023-03-03T05:45:35.842862+0000","last_fullsized":"2023-03-03T05:45:35.842862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:22:57.039574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.858102+0000","last_change":"2023-03-03T05:45:22.224435+0000","last_active":"2023-03-03T05:45:35.858102+0000","last_peered":"2023-03-03T05:45:35.858102+0000","last_clean":"2023-03-03T05:45:35.858102+0000","last_became_active":"2023-03-03T05:45:22.222958+0000","last_became_peered":"2023-03-03T05:45:22.222958+0000","last_unstale":"2023-03-03T05:45:35.858102+0000","last_undegraded":"2023-03-03T05:45:35.858102+0000","last_fullsized":"2023-03-03T05:45:35.858102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_clean_scrub_stamp":"2023-03-03T05:45:20.925942+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:10:20.426026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843695+0000","last_change":"2023-03-03T05:45:35.304483+0000","last_active":"2023-03-03T05:45:35.843695+0000","last_peered":"2023-03-03T05:45:35.843695+0000","last_clean":"2023-03-03T05:45:35.843695+0000","last_became_active":"2023-03-03T05:45:35.304202+0000","last_became_peered":"2023-03-03T05:45:35.304202+0000","last_unstale":"2023-03-03T05:45:35.843695+0000","last_undegraded":"2023-03-03T05:45:35.843695+0000","last_fullsized":"2023-03-03T05:45:35.843695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851295+0000","last_change":"2023-03-03T05:45:35.869049+0000","last_active":"2023-03-03T05:45:36.851295+0000","last_peered":"2023-03-03T05:45:36.851295+0000","last_clean":"2023-03-03T05:45:36.851295+0000","last_became_active":"2023-03-03T05:45:35.868376+0000","last_became_peered":"2023-03-03T05:45:35.868376+0000","last_unstale":"2023-03-03T05:45:36.851295+0000","last_undegraded":"2023-03-03T05:45:36.851295+0000","last_fullsized":"2023-03-03T05:45:36.851295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:42:55.207846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843590+0000","last_change":"2023-03-03T05:45:29.056874+0000","last_active":"2023-03-03T05:45:35.843590+0000","last_peered":"2023-03-03T05:45:35.843590+0000","last_clean":"2023-03-03T05:45:35.843590+0000","last_became_active":"2023-03-03T05:45:29.056438+0000","last_became_peered":"2023-03-03T05:45:29.056438+0000","last_unstale":"2023-03-03T05:45:35.843590+0000","last_undegraded":"2023-03-03T05:45:35.843590+0000","last_fullsized":"2023-03-03T05:45:35.843590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:08.113953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.846141+0000","last_change":"2023-03-03T05:45:35.304562+0000","last_active":"2023-03-03T05:45:35.846141+0000","last_peered":"2023-03-03T05:45:35.846141+0000","last_clean":"2023-03-03T05:45:35.846141+0000","last_became_active":"2023-03-03T05:45:35.304264+0000","last_became_peered":"2023-03-03T05:45:35.304264+0000","last_unstale":"2023-03-03T05:45:35.846141+0000","last_undegraded":"2023-03-03T05:45:35.846141+0000","last_fullsized":"2023-03-03T05:45:35.846141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923094+0000","last_change":"2023-03-03T05:45:35.859779+0000","last_active":"2023-03-03T05:45:36.923094+0000","last_peered":"2023-03-03T05:45:36.923094+0000","last_clean":"2023-03-03T05:45:36.923094+0000","last_became_active":"2023-03-03T05:45:35.857596+0000","last_became_peered":"2023-03-03T05:45:35.857596+0000","last_unstale":"2023-03-03T05:45:36.923094+0000","last_undegraded":"2023-03-03T05:45:36.923094+0000","last_fullsized":"2023-03-03T05:45:36.923094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844418+0000","last_change":"2023-03-03T05:45:29.056891+0000","last_active":"2023-03-03T05:45:35.844418+0000","last_peered":"2023-03-03T05:45:35.844418+0000","last_clean":"2023-03-03T05:45:35.844418+0000","last_became_active":"2023-03-03T05:45:29.056483+0000","last_became_peered":"2023-03-03T05:45:29.056483+0000","last_unstale":"2023-03-03T05:45:35.844418+0000","last_undegraded":"2023-03-03T05:45:35.844418+0000","last_fullsized":"2023-03-03T05:45:35.844418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:25.701585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.847358+0000","last_change":"2023-03-03T05:45:35.293821+0000","last_active":"2023-03-03T05:45:35.847358+0000","last_peered":"2023-03-03T05:45:35.847358+0000","last_clean":"2023-03-03T05:45:35.847358+0000","last_became_active":"2023-03-03T05:45:35.293533+0000","last_became_peered":"2023-03-03T05:45:35.293533+0000","last_unstale":"2023-03-03T05:45:35.847358+0000","last_undegraded":"2023-03-03T05:45:35.847358+0000","last_fullsized":"2023-03-03T05:45:35.847358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923917+0000","last_change":"2023-03-03T05:45:35.866830+0000","last_active":"2023-03-03T05:45:36.923917+0000","last_peered":"2023-03-03T05:45:36.923917+0000","last_clean":"2023-03-03T05:45:36.923917+0000","last_became_active":"2023-03-03T05:45:35.866509+0000","last_became_peered":"2023-03-03T05:45:35.866509+0000","last_unstale":"2023-03-03T05:45:36.923917+0000","last_undegraded":"2023-03-03T05:45:36.923917+0000","last_fullsized":"2023-03-03T05:45:36.923917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.850545+0000","last_change":"2023-03-03T05:45:35.300794+0000","last_active":"2023-03-03T05:45:40.850545+0000","last_peered":"2023-03-03T05:45:40.850545+0000","last_clean":"2023-03-03T05:45:40.850545+0000","last_became_active":"2023-03-03T05:45:35.300531+0000","last_became_peered":"2023-03-03T05:45:35.300531+0000","last_unstale":"2023-03-03T05:45:40.850545+0000","last_undegraded":"2023-03-03T05:45:40.850545+0000","last_fullsized":"2023-03-03T05:45:40.850545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843343+0000","last_change":"2023-03-03T05:45:29.193601+0000","last_active":"2023-03-03T05:45:35.843343+0000","last_peered":"2023-03-03T05:45:35.843343+0000","last_clean":"2023-03-03T05:45:35.843343+0000","last_became_active":"2023-03-03T05:45:29.193262+0000","last_became_peered":"2023-03-03T05:45:29.193262+0000","last_unstale":"2023-03-03T05:45:35.843343+0000","last_undegraded":"2023-03-03T05:45:35.843343+0000","last_fullsized":"2023-03-03T05:45:35.843343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853228+0000","last_change":"2023-03-03T05:45:35.864517+0000","last_active":"2023-03-03T05:45:36.853228+0000","last_peered":"2023-03-03T05:45:36.853228+0000","last_clean":"2023-03-03T05:45:36.853228+0000","last_became_active":"2023-03-03T05:45:35.864193+0000","last_became_peered":"2023-03-03T05:45:35.864193+0000","last_unstale":"2023-03-03T05:45:36.853228+0000","last_undegraded":"2023-03-03T05:45:36.853228+0000","last_fullsized":"2023-03-03T05:45:36.853228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:11:08.433881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844985+0000","last_change":"2023-03-03T05:45:29.193546+0000","last_active":"2023-03-03T05:45:35.844985+0000","last_peered":"2023-03-03T05:45:35.844985+0000","last_clean":"2023-03-03T05:45:35.844985+0000","last_became_active":"2023-03-03T05:45:29.193172+0000","last_became_peered":"2023-03-03T05:45:29.193172+0000","last_unstale":"2023-03-03T05:45:35.844985+0000","last_undegraded":"2023-03-03T05:45:35.844985+0000","last_fullsized":"2023-03-03T05:45:35.844985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:07:42.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853612+0000","last_change":"2023-03-03T05:45:35.296693+0000","last_active":"2023-03-03T05:45:36.853612+0000","last_peered":"2023-03-03T05:45:36.853612+0000","last_clean":"2023-03-03T05:45:36.853612+0000","last_became_active":"2023-03-03T05:45:35.296469+0000","last_became_peered":"2023-03-03T05:45:35.296469+0000","last_unstale":"2023-03-03T05:45:36.853612+0000","last_undegraded":"2023-03-03T05:45:36.853612+0000","last_fullsized":"2023-03-03T05:45:36.853612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850872+0000","last_change":"2023-03-03T05:45:35.856468+0000","last_active":"2023-03-03T05:45:36.850872+0000","last_peered":"2023-03-03T05:45:36.850872+0000","last_clean":"2023-03-03T05:45:36.850872+0000","last_became_active":"2023-03-03T05:45:35.855970+0000","last_became_peered":"2023-03-03T05:45:35.855970+0000","last_unstale":"2023-03-03T05:45:36.850872+0000","last_undegraded":"2023-03-03T05:45:36.850872+0000","last_fullsized":"2023-03-03T05:45:36.850872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853299+0000","last_change":"2023-03-03T05:45:35.301855+0000","last_active":"2023-03-03T05:45:36.853299+0000","last_peered":"2023-03-03T05:45:36.853299+0000","last_clean":"2023-03-03T05:45:36.853299+0000","last_became_active":"2023-03-03T05:45:35.301622+0000","last_became_peered":"2023-03-03T05:45:35.301622+0000","last_unstale":"2023-03-03T05:45:36.853299+0000","last_undegraded":"2023-03-03T05:45:36.853299+0000","last_fullsized":"2023-03-03T05:45:36.853299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:42.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853538+0000","last_change":"2023-03-03T05:45:35.851780+0000","last_active":"2023-03-03T05:45:36.853538+0000","last_peered":"2023-03-03T05:45:36.853538+0000","last_clean":"2023-03-03T05:45:36.853538+0000","last_became_active":"2023-03-03T05:45:35.850190+0000","last_became_peered":"2023-03-03T05:45:35.850190+0000","last_unstale":"2023-03-03T05:45:36.853538+0000","last_undegraded":"2023-03-03T05:45:36.853538+0000","last_fullsized":"2023-03-03T05:45:36.853538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856909+0000","last_change":"2023-03-03T05:45:35.309047+0000","last_active":"2023-03-03T05:45:40.856909+0000","last_peered":"2023-03-03T05:45:40.856909+0000","last_clean":"2023-03-03T05:45:40.856909+0000","last_became_active":"2023-03-03T05:45:35.308564+0000","last_became_peered":"2023-03-03T05:45:35.308564+0000","last_unstale":"2023-03-03T05:45:40.856909+0000","last_undegraded":"2023-03-03T05:45:40.856909+0000","last_fullsized":"2023-03-03T05:45:40.856909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850462+0000","last_change":"2023-03-03T05:45:35.861316+0000","last_active":"2023-03-03T05:45:36.850462+0000","last_peered":"2023-03-03T05:45:36.850462+0000","last_clean":"2023-03-03T05:45:36.850462+0000","last_became_active":"2023-03-03T05:45:35.861052+0000","last_became_peered":"2023-03-03T05:45:35.861052+0000","last_unstale":"2023-03-03T05:45:36.850462+0000","last_undegraded":"2023-03-03T05:45:36.850462+0000","last_fullsized":"2023-03-03T05:45:36.850462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858612+0000","last_change":"2023-03-03T05:45:35.303666+0000","last_active":"2023-03-03T05:45:40.858612+0000","last_peered":"2023-03-03T05:45:40.858612+0000","last_clean":"2023-03-03T05:45:40.858612+0000","last_became_active":"2023-03-03T05:45:35.303213+0000","last_became_peered":"2023-03-03T05:45:35.303213+0000","last_unstale":"2023-03-03T05:45:40.858612+0000","last_undegraded":"2023-03-03T05:45:40.858612+0000","last_fullsized":"2023-03-03T05:45:40.858612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:54.389717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850392+0000","last_change":"2023-03-03T05:45:35.857442+0000","last_active":"2023-03-03T05:45:36.850392+0000","last_peered":"2023-03-03T05:45:36.850392+0000","last_clean":"2023-03-03T05:45:36.850392+0000","last_became_active":"2023-03-03T05:45:35.857130+0000","last_became_peered":"2023-03-03T05:45:35.857130+0000","last_unstale":"2023-03-03T05:45:36.850392+0000","last_undegraded":"2023-03-03T05:45:36.850392+0000","last_fullsized":"2023-03-03T05:45:36.850392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923020+0000","last_change":"2023-03-03T05:45:35.305098+0000","last_active":"2023-03-03T05:45:36.923020+0000","last_peered":"2023-03-03T05:45:36.923020+0000","last_clean":"2023-03-03T05:45:36.923020+0000","last_became_active":"2023-03-03T05:45:35.304812+0000","last_became_peered":"2023-03-03T05:45:35.304812+0000","last_unstale":"2023-03-03T05:45:36.923020+0000","last_undegraded":"2023-03-03T05:45:36.923020+0000","last_fullsized":"2023-03-03T05:45:36.923020+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923341+0000","last_change":"2023-03-03T05:45:35.858806+0000","last_active":"2023-03-03T05:45:36.923341+0000","last_peered":"2023-03-03T05:45:36.923341+0000","last_clean":"2023-03-03T05:45:36.923341+0000","last_became_active":"2023-03-03T05:45:35.858471+0000","last_became_peered":"2023-03-03T05:45:35.858471+0000","last_unstale":"2023-03-03T05:45:36.923341+0000","last_undegraded":"2023-03-03T05:45:36.923341+0000","last_fullsized":"2023-03-03T05:45:36.923341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856244+0000","last_change":"2023-03-03T05:45:35.310654+0000","last_active":"2023-03-03T05:45:40.856244+0000","last_peered":"2023-03-03T05:45:40.856244+0000","last_clean":"2023-03-03T05:45:40.856244+0000","last_became_active":"2023-03-03T05:45:35.310377+0000","last_became_peered":"2023-03-03T05:45:35.310377+0000","last_unstale":"2023-03-03T05:45:40.856244+0000","last_undegraded":"2023-03-03T05:45:40.856244+0000","last_fullsized":"2023-03-03T05:45:40.856244+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850763+0000","last_change":"2023-03-03T05:45:35.850670+0000","last_active":"2023-03-03T05:45:36.850763+0000","last_peered":"2023-03-03T05:45:36.850763+0000","last_clean":"2023-03-03T05:45:36.850763+0000","last_became_active":"2023-03-03T05:45:35.850380+0000","last_became_peered":"2023-03-03T05:45:35.850380+0000","last_unstale":"2023-03-03T05:45:36.850763+0000","last_undegraded":"2023-03-03T05:45:36.850763+0000","last_fullsized":"2023-03-03T05:45:36.850763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923387+0000","last_change":"2023-03-03T05:45:35.863660+0000","last_active":"2023-03-03T05:45:36.923387+0000","last_peered":"2023-03-03T05:45:36.923387+0000","last_clean":"2023-03-03T05:45:36.923387+0000","last_became_active":"2023-03-03T05:45:35.863234+0000","last_became_peered":"2023-03-03T05:45:35.863234+0000","last_unstale":"2023-03-03T05:45:36.923387+0000","last_undegraded":"2023-03-03T05:45:36.923387+0000","last_fullsized":"2023-03-03T05:45:36.923387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922937+0000","last_change":"2023-03-03T05:45:35.851019+0000","last_active":"2023-03-03T05:45:36.922937+0000","last_peered":"2023-03-03T05:45:36.922937+0000","last_clean":"2023-03-03T05:45:36.922937+0000","last_became_active":"2023-03-03T05:45:35.850439+0000","last_became_peered":"2023-03-03T05:45:35.850439+0000","last_unstale":"2023-03-03T05:45:36.922937+0000","last_undegraded":"2023-03-03T05:45:36.922937+0000","last_fullsized":"2023-03-03T05:45:36.922937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:43.693100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962628+0000","last_change":"2023-03-03T05:45:35.860508+0000","last_active":"2023-03-03T05:45:35.962628+0000","last_peered":"2023-03-03T05:45:35.962628+0000","last_clean":"2023-03-03T05:45:35.962628+0000","last_became_active":"2023-03-03T05:45:35.860137+0000","last_became_peered":"2023-03-03T05:45:35.860137+0000","last_unstale":"2023-03-03T05:45:35.962628+0000","last_undegraded":"2023-03-03T05:45:35.962628+0000","last_fullsized":"2023-03-03T05:45:35.962628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923487+0000","last_change":"2023-03-03T05:45:35.864824+0000","last_active":"2023-03-03T05:45:36.923487+0000","last_peered":"2023-03-03T05:45:36.923487+0000","last_clean":"2023-03-03T05:45:36.923487+0000","last_became_active":"2023-03-03T05:45:35.864466+0000","last_became_peered":"2023-03-03T05:45:35.864466+0000","last_unstale":"2023-03-03T05:45:36.923487+0000","last_undegraded":"2023-03-03T05:45:36.923487+0000","last_fullsized":"2023-03-03T05:45:36.923487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922781+0000","last_change":"2023-03-03T05:45:35.859499+0000","last_active":"2023-03-03T05:45:36.922781+0000","last_peered":"2023-03-03T05:45:36.922781+0000","last_clean":"2023-03-03T05:45:36.922781+0000","last_became_active":"2023-03-03T05:45:35.859200+0000","last_became_peered":"2023-03-03T05:45:35.859200+0000","last_unstale":"2023-03-03T05:45:36.922781+0000","last_undegraded":"2023-03-03T05:45:36.922781+0000","last_fullsized":"2023-03-03T05:45:36.922781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2589,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":6149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":90566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":88905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8068,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363772,"statfs":{"total":96636764160,"available":96628502528,"internally_reserved":0,"allocated":258048,"data_stored":90670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9088,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362752,"statfs":{"total":96636764160,"available":96627458048,"internally_reserved":0,"allocated":843776,"data_stored":679785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9140,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362700,"statfs":{"total":96636764160,"available":96627404800,"internally_reserved":0,"allocated":831488,"data_stored":678400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7988,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363852,"statfs":{"total":96636764160,"available":96628584448,"internally_reserved":0,"allocated":241664,"data_stored":88849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T05:46:04.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T05:46:04.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.317+0000 7f8e22dff700 1 Processor -- start 2023-03-03T05:46:04.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.317+0000 7f8e22dff700 1 -- start start 2023-03-03T05:46:04.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.318+0000 7f8e22dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.318+0000 7f8e22dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8e1c0cf9a0 0x7f8e1c163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.318+0000 7f8e22dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c0cb9f0 con 0x7f8e1c1597f0 2023-03-03T05:46:04.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.318+0000 7f8e22dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1c0ca750 con 0x7f8e1c0cf9a0 2023-03-03T05:46:04.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.318+0000 7f8e22dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c15a230 con 0x7f8e1c0ced80 2023-03-03T05:46:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e2139c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8e1c0cf9a0 0x7f8e1c163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37804/0 (socket says 172.21.15.102:37804) 2023-03-03T05:46:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 -- 172.21.15.102:0/187692936 learned_addr learned my addr 172.21.15.102:0/187692936 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e1bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8e1c1597f0 0x7f8e1c0cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38844/0 (socket says 172.21.15.102:38844) 2023-03-03T05:46:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 -- 172.21.15.102:0/187692936 >> v2:172.21.15.172:3300/0 conn(0x7f8e1c0cf9a0 msgr2=0x7f8e1c163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 --2- 172.21.15.102:0/187692936 >> v2:172.21.15.172:3300/0 conn(0x7f8e1c0cf9a0 0x7f8e1c163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 -- 172.21.15.102:0/187692936 >> v1:172.21.15.102:6789/0 conn(0x7f8e1c1597f0 legacy=0x7f8e1c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 -- 172.21.15.102:0/187692936 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c159c80 con 0x7f8e1c0ced80 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.319+0000 7f8e20b9b700 1 --2- 172.21.15.102:0/187692936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0cf150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8e100098f0 tx=0x7f8e10009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90216432bfef6f21 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e1b7fe700 1 -- 172.21.15.102:0/187692936 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011830 con 0x7f8e1c0ced80 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e1b7fe700 1 -- 172.21.15.102:0/187692936 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e10011990 con 0x7f8e1c0ced80 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e1b7fe700 1 -- 172.21.15.102:0/187692936 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011d20 con 0x7f8e1c0ced80 2023-03-03T05:46:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e22dff700 1 -- 172.21.15.102:0/187692936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 msgr2=0x7f8e1c0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e22dff700 1 --2- 172.21.15.102:0/187692936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0cf150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8e100098f0 tx=0x7f8e10009c00 comp rx=0 tx=0).stop 2023-03-03T05:46:04.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e22dff700 1 -- 172.21.15.102:0/187692936 shutdown_connections 2023-03-03T05:46:04.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e22dff700 1 --2- 172.21.15.102:0/187692936 >> v2:172.21.15.172:3300/0 conn(0x7f8e1c0cf9a0 0x7f8e1c163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e22dff700 1 --2- 172.21.15.102:0/187692936 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0cf150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.320+0000 7f8e22dff700 1 -- 172.21.15.102:0/187692936 >> 172.21.15.102:0/187692936 conn(0x7f8e1c065fb0 msgr2=0x7f8e1c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.321+0000 7f8e22dff700 1 -- 172.21.15.102:0/187692936 shutdown_connections 2023-03-03T05:46:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.321+0000 7f8e22dff700 1 -- 172.21.15.102:0/187692936 wait complete. 2023-03-03T05:46:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.322+0000 7f8e22dff700 1 Processor -- start 2023-03-03T05:46:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.323+0000 7f8e22dff700 1 -- start start 2023-03-03T05:46:04.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e22dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e22dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e1c0cf9a0 0x7f8e1c0b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e22dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e1c1597f0 0x7f8e1c0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e22dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0c5090 con 0x7f8e1c1597f0 2023-03-03T05:46:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e22dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0c8370 con 0x7f8e1c0cf9a0 2023-03-03T05:46:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e22dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0c77c0 con 0x7f8e1c0ced80 2023-03-03T05:46:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e1bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e1c0cf9a0 0x7f8e1c0b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e2139c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e1c1597f0 0x7f8e1c0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37812/0 (socket says 172.21.15.102:37812) 2023-03-03T05:46:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 -- 172.21.15.102:0/1865736175 learned_addr learned my addr 172.21.15.102:0/1865736175 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 -- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e1c0cf9a0 msgr2=0x7f8e1c0b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e1c0cf9a0 0x7f8e1c0b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 -- 172.21.15.102:0/1865736175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e1c1597f0 msgr2=0x7f8e1c0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e1c1597f0 0x7f8e1c0c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e20b9b700 1 -- 172.21.15.102:0/1865736175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e10009550 con 0x7f8e1c0ced80 2023-03-03T05:46:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.324+0000 7f8e1bfff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e1c0cf9a0 0x7f8e1c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.325+0000 7f8e20b9b700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0c2920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8e10014040 tx=0x7f8e10004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.325+0000 7f8e197fa700 1 -- 172.21.15.102:0/1865736175 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10004510 con 0x7f8e1c0ced80 2023-03-03T05:46:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.325+0000 7f8e197fa700 1 -- 172.21.15.102:0/1865736175 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1001e070 con 0x7f8e1c0ced80 2023-03-03T05:46:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.325+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c0cb0c0 con 0x7f8e1c0ced80 2023-03-03T05:46:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.325+0000 7f8e197fa700 1 -- 172.21.15.102:0/1865736175 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e100038e0 con 0x7f8e1c0ced80 2023-03-03T05:46:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.325+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c0cade0 con 0x7f8e1c0ced80 2023-03-03T05:46:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.326+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c0cb3a0 con 0x7f8e1c0ced80 2023-03-03T05:46:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.328+0000 7f8e197fa700 1 -- 172.21.15.102:0/1865736175 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e1000f050 con 0x7f8e1c0ced80 2023-03-03T05:46:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.329+0000 7f8e197fa700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e000682f0 0x7f8e0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.329+0000 7f8e197fa700 1 -- 172.21.15.102:0/1865736175 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8e10086c50 con 0x7f8e1c0ced80 2023-03-03T05:46:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.329+0000 7f8e1bfff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e000682f0 0x7f8e0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.330+0000 7f8e1bfff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e000682f0 0x7f8e0006a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c08cce0 tx=0x7f8e08007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:04.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.333+0000 7f8e197fa700 1 -- 172.21.15.102:0/1865736175 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e10053340 con 0x7f8e1c0ced80 2023-03-03T05:46:04.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.520+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 --> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e1c0677f0 con 0x7f8e000682f0 2023-03-03T05:46:04.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.529+0000 7f8e197fa700 1 -- 172.21.15.102:0/1865736175 <== mgr.4105 v2:172.21.15.172:6832/114058 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286578 (secure 0 0 0) 0x7f8e1c0677f0 con 0x7f8e000682f0 2023-03-03T05:46:04.534 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:04.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.533+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e000682f0 msgr2=0x7f8e0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.533+0000 7f8e22dff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e000682f0 0x7f8e0006a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c08cce0 tx=0x7f8e08007ed0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.533+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 msgr2=0x7f8e1c0c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.533+0000 7f8e22dff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0c2920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8e10014040 tx=0x7f8e10004750 comp rx=0 tx=0).stop 2023-03-03T05:46:04.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.533+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 shutdown_connections 2023-03-03T05:46:04.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.534+0000 7f8e22dff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e000682f0 0x7f8e0006a7a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.534+0000 7f8e22dff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e1c0ced80 0x7f8e1c0c2920 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.534+0000 7f8e22dff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e1c1597f0 0x7f8e1c0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.534+0000 7f8e22dff700 1 --2- 172.21.15.102:0/1865736175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e1c0cf9a0 0x7f8e1c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.534+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 >> 172.21.15.102:0/1865736175 conn(0x7f8e1c065fb0 msgr2=0x7f8e1c158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:04.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.534+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 shutdown_connections 2023-03-03T05:46:04.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.534+0000 7f8e22dff700 1 -- 172.21.15.102:0/1865736175 wait complete. 2023-03-03T05:46:04.537 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2023-03-03T05:46:04.563 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-03-03T05:46:03.950051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592957,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66244,"kb_used_data":3076,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":754908476,"statfs":{"total":773094113280,"available":773026279424,"internally_reserved":0,"allocated":3149824,"data_stored":1894693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":259,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003582"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922943+0000","last_change":"2023-03-03T05:45:35.854816+0000","last_active":"2023-03-03T05:45:36.922943+0000","last_peered":"2023-03-03T05:45:36.922943+0000","last_clean":"2023-03-03T05:45:36.922943+0000","last_became_active":"2023-03-03T05:45:35.854447+0000","last_became_peered":"2023-03-03T05:45:35.854447+0000","last_unstale":"2023-03-03T05:45:36.922943+0000","last_undegraded":"2023-03-03T05:45:36.922943+0000","last_fullsized":"2023-03-03T05:45:36.922943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850859+0000","last_change":"2023-03-03T05:45:35.862354+0000","last_active":"2023-03-03T05:45:36.850859+0000","last_peered":"2023-03-03T05:45:36.850859+0000","last_clean":"2023-03-03T05:45:36.850859+0000","last_became_active":"2023-03-03T05:45:35.862016+0000","last_became_peered":"2023-03-03T05:45:35.862016+0000","last_unstale":"2023-03-03T05:45:36.850859+0000","last_undegraded":"2023-03-03T05:45:36.850859+0000","last_fullsized":"2023-03-03T05:45:36.850859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962731+0000","last_change":"2023-03-03T05:45:35.864429+0000","last_active":"2023-03-03T05:45:35.962731+0000","last_peered":"2023-03-03T05:45:35.962731+0000","last_clean":"2023-03-03T05:45:35.962731+0000","last_became_active":"2023-03-03T05:45:35.864091+0000","last_became_peered":"2023-03-03T05:45:35.864091+0000","last_unstale":"2023-03-03T05:45:35.962731+0000","last_undegraded":"2023-03-03T05:45:35.962731+0000","last_fullsized":"2023-03-03T05:45:35.962731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962683+0000","last_change":"2023-03-03T05:45:35.861374+0000","last_active":"2023-03-03T05:45:35.962683+0000","last_peered":"2023-03-03T05:45:35.962683+0000","last_clean":"2023-03-03T05:45:35.962683+0000","last_became_active":"2023-03-03T05:45:35.860993+0000","last_became_peered":"2023-03-03T05:45:35.860993+0000","last_unstale":"2023-03-03T05:45:35.962683+0000","last_undegraded":"2023-03-03T05:45:35.962683+0000","last_fullsized":"2023-03-03T05:45:35.962683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853383+0000","last_change":"2023-03-03T05:45:35.857703+0000","last_active":"2023-03-03T05:45:36.853383+0000","last_peered":"2023-03-03T05:45:36.853383+0000","last_clean":"2023-03-03T05:45:36.853383+0000","last_became_active":"2023-03-03T05:45:35.857421+0000","last_became_peered":"2023-03-03T05:45:35.857421+0000","last_unstale":"2023-03-03T05:45:36.853383+0000","last_undegraded":"2023-03-03T05:45:36.853383+0000","last_fullsized":"2023-03-03T05:45:36.853383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851248+0000","last_change":"2023-03-03T05:45:35.859892+0000","last_active":"2023-03-03T05:45:36.851248+0000","last_peered":"2023-03-03T05:45:36.851248+0000","last_clean":"2023-03-03T05:45:36.851248+0000","last_became_active":"2023-03-03T05:45:35.859617+0000","last_became_peered":"2023-03-03T05:45:35.859617+0000","last_unstale":"2023-03-03T05:45:36.851248+0000","last_undegraded":"2023-03-03T05:45:36.851248+0000","last_fullsized":"2023-03-03T05:45:36.851248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:23:53.405094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850434+0000","last_change":"2023-03-03T05:45:35.862939+0000","last_active":"2023-03-03T05:45:36.850434+0000","last_peered":"2023-03-03T05:45:36.850434+0000","last_clean":"2023-03-03T05:45:36.850434+0000","last_became_active":"2023-03-03T05:45:35.862681+0000","last_became_peered":"2023-03-03T05:45:35.862681+0000","last_unstale":"2023-03-03T05:45:36.850434+0000","last_undegraded":"2023-03-03T05:45:36.850434+0000","last_fullsized":"2023-03-03T05:45:36.850434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:58.535750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923559+0000","last_change":"2023-03-03T05:45:35.866302+0000","last_active":"2023-03-03T05:45:36.923559+0000","last_peered":"2023-03-03T05:45:36.923559+0000","last_clean":"2023-03-03T05:45:36.923559+0000","last_became_active":"2023-03-03T05:45:35.865627+0000","last_became_peered":"2023-03-03T05:45:35.865627+0000","last_unstale":"2023-03-03T05:45:36.923559+0000","last_undegraded":"2023-03-03T05:45:36.923559+0000","last_fullsized":"2023-03-03T05:45:36.923559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853588+0000","last_change":"2023-03-03T05:45:35.856762+0000","last_active":"2023-03-03T05:45:36.853588+0000","last_peered":"2023-03-03T05:45:36.853588+0000","last_clean":"2023-03-03T05:45:36.853588+0000","last_became_active":"2023-03-03T05:45:35.856458+0000","last_became_peered":"2023-03-03T05:45:35.856458+0000","last_unstale":"2023-03-03T05:45:36.853588+0000","last_undegraded":"2023-03-03T05:45:36.853588+0000","last_fullsized":"2023-03-03T05:45:36.853588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923093+0000","last_change":"2023-03-03T05:45:35.865258+0000","last_active":"2023-03-03T05:45:36.923093+0000","last_peered":"2023-03-03T05:45:36.923093+0000","last_clean":"2023-03-03T05:45:36.923093+0000","last_became_active":"2023-03-03T05:45:35.864889+0000","last_became_peered":"2023-03-03T05:45:35.864889+0000","last_unstale":"2023-03-03T05:45:36.923093+0000","last_undegraded":"2023-03-03T05:45:36.923093+0000","last_fullsized":"2023-03-03T05:45:36.923093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923368+0000","last_change":"2023-03-03T05:45:35.868970+0000","last_active":"2023-03-03T05:45:36.923368+0000","last_peered":"2023-03-03T05:45:36.923368+0000","last_clean":"2023-03-03T05:45:36.923368+0000","last_became_active":"2023-03-03T05:45:35.868555+0000","last_became_peered":"2023-03-03T05:45:35.868555+0000","last_unstale":"2023-03-03T05:45:36.923368+0000","last_undegraded":"2023-03-03T05:45:36.923368+0000","last_fullsized":"2023-03-03T05:45:36.923368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922618+0000","last_change":"2023-03-03T05:45:35.855291+0000","last_active":"2023-03-03T05:45:36.922618+0000","last_peered":"2023-03-03T05:45:36.922618+0000","last_clean":"2023-03-03T05:45:36.922618+0000","last_became_active":"2023-03-03T05:45:35.854925+0000","last_became_peered":"2023-03-03T05:45:35.854925+0000","last_unstale":"2023-03-03T05:45:36.922618+0000","last_undegraded":"2023-03-03T05:45:36.922618+0000","last_fullsized":"2023-03-03T05:45:36.922618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.924177+0000","last_change":"2023-03-03T05:45:35.853479+0000","last_active":"2023-03-03T05:45:36.924177+0000","last_peered":"2023-03-03T05:45:36.924177+0000","last_clean":"2023-03-03T05:45:36.924177+0000","last_became_active":"2023-03-03T05:45:35.853193+0000","last_became_peered":"2023-03-03T05:45:35.853193+0000","last_unstale":"2023-03-03T05:45:36.924177+0000","last_undegraded":"2023-03-03T05:45:36.924177+0000","last_fullsized":"2023-03-03T05:45:36.924177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923036+0000","last_change":"2023-03-03T05:45:35.865976+0000","last_active":"2023-03-03T05:45:36.923036+0000","last_peered":"2023-03-03T05:45:36.923036+0000","last_clean":"2023-03-03T05:45:36.923036+0000","last_became_active":"2023-03-03T05:45:35.865561+0000","last_became_peered":"2023-03-03T05:45:35.865561+0000","last_unstale":"2023-03-03T05:45:36.923036+0000","last_undegraded":"2023-03-03T05:45:36.923036+0000","last_fullsized":"2023-03-03T05:45:36.923036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962602+0000","last_change":"2023-03-03T05:45:35.862593+0000","last_active":"2023-03-03T05:45:35.962602+0000","last_peered":"2023-03-03T05:45:35.962602+0000","last_clean":"2023-03-03T05:45:35.962602+0000","last_became_active":"2023-03-03T05:45:35.862342+0000","last_became_peered":"2023-03-03T05:45:35.862342+0000","last_unstale":"2023-03-03T05:45:35.962602+0000","last_undegraded":"2023-03-03T05:45:35.962602+0000","last_fullsized":"2023-03-03T05:45:35.962602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850467+0000","last_change":"2023-03-03T05:45:35.857623+0000","last_active":"2023-03-03T05:45:36.850467+0000","last_peered":"2023-03-03T05:45:36.850467+0000","last_clean":"2023-03-03T05:45:36.850467+0000","last_became_active":"2023-03-03T05:45:35.857357+0000","last_became_peered":"2023-03-03T05:45:35.857357+0000","last_unstale":"2023-03-03T05:45:36.850467+0000","last_undegraded":"2023-03-03T05:45:36.850467+0000","last_fullsized":"2023-03-03T05:45:36.850467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:00.850961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923102+0000","last_change":"2023-03-03T05:45:35.859115+0000","last_active":"2023-03-03T05:45:36.923102+0000","last_peered":"2023-03-03T05:45:36.923102+0000","last_clean":"2023-03-03T05:45:36.923102+0000","last_became_active":"2023-03-03T05:45:35.858794+0000","last_became_peered":"2023-03-03T05:45:35.858794+0000","last_unstale":"2023-03-03T05:45:36.923102+0000","last_undegraded":"2023-03-03T05:45:36.923102+0000","last_fullsized":"2023-03-03T05:45:36.923102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962764+0000","last_change":"2023-03-03T05:45:35.861090+0000","last_active":"2023-03-03T05:45:35.962764+0000","last_peered":"2023-03-03T05:45:35.962764+0000","last_clean":"2023-03-03T05:45:35.962764+0000","last_became_active":"2023-03-03T05:45:35.860814+0000","last_became_peered":"2023-03-03T05:45:35.860814+0000","last_unstale":"2023-03-03T05:45:35.962764+0000","last_undegraded":"2023-03-03T05:45:35.962764+0000","last_fullsized":"2023-03-03T05:45:35.962764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923139+0000","last_change":"2023-03-03T05:45:35.855611+0000","last_active":"2023-03-03T05:45:36.923139+0000","last_peered":"2023-03-03T05:45:36.923139+0000","last_clean":"2023-03-03T05:45:36.923139+0000","last_became_active":"2023-03-03T05:45:35.855333+0000","last_became_peered":"2023-03-03T05:45:35.855333+0000","last_unstale":"2023-03-03T05:45:36.923139+0000","last_undegraded":"2023-03-03T05:45:36.923139+0000","last_fullsized":"2023-03-03T05:45:36.923139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923216+0000","last_change":"2023-03-03T05:45:35.865135+0000","last_active":"2023-03-03T05:45:36.923216+0000","last_peered":"2023-03-03T05:45:36.923216+0000","last_clean":"2023-03-03T05:45:36.923216+0000","last_became_active":"2023-03-03T05:45:35.864823+0000","last_became_peered":"2023-03-03T05:45:35.864823+0000","last_unstale":"2023-03-03T05:45:36.923216+0000","last_undegraded":"2023-03-03T05:45:36.923216+0000","last_fullsized":"2023-03-03T05:45:36.923216+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923121+0000","last_change":"2023-03-03T05:45:35.864573+0000","last_active":"2023-03-03T05:45:36.923121+0000","last_peered":"2023-03-03T05:45:36.923121+0000","last_clean":"2023-03-03T05:45:36.923121+0000","last_became_active":"2023-03-03T05:45:35.864297+0000","last_became_peered":"2023-03-03T05:45:35.864297+0000","last_unstale":"2023-03-03T05:45:36.923121+0000","last_undegraded":"2023-03-03T05:45:36.923121+0000","last_fullsized":"2023-03-03T05:45:36.923121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922982+0000","last_change":"2023-03-03T05:45:35.862720+0000","last_active":"2023-03-03T05:45:36.922982+0000","last_peered":"2023-03-03T05:45:36.922982+0000","last_clean":"2023-03-03T05:45:36.922982+0000","last_became_active":"2023-03-03T05:45:35.861282+0000","last_became_peered":"2023-03-03T05:45:35.861282+0000","last_unstale":"2023-03-03T05:45:36.922982+0000","last_undegraded":"2023-03-03T05:45:36.922982+0000","last_fullsized":"2023-03-03T05:45:36.922982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853407+0000","last_change":"2023-03-03T05:45:35.859828+0000","last_active":"2023-03-03T05:45:36.853407+0000","last_peered":"2023-03-03T05:45:36.853407+0000","last_clean":"2023-03-03T05:45:36.853407+0000","last_became_active":"2023-03-03T05:45:35.859568+0000","last_became_peered":"2023-03-03T05:45:35.859568+0000","last_unstale":"2023-03-03T05:45:36.853407+0000","last_undegraded":"2023-03-03T05:45:36.853407+0000","last_fullsized":"2023-03-03T05:45:36.853407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923461+0000","last_change":"2023-03-03T05:45:35.864435+0000","last_active":"2023-03-03T05:45:36.923461+0000","last_peered":"2023-03-03T05:45:36.923461+0000","last_clean":"2023-03-03T05:45:36.923461+0000","last_became_active":"2023-03-03T05:45:35.864129+0000","last_became_peered":"2023-03-03T05:45:35.864129+0000","last_unstale":"2023-03-03T05:45:36.923461+0000","last_undegraded":"2023-03-03T05:45:36.923461+0000","last_fullsized":"2023-03-03T05:45:36.923461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853682+0000","last_change":"2023-03-03T05:45:35.855909+0000","last_active":"2023-03-03T05:45:36.853682+0000","last_peered":"2023-03-03T05:45:36.853682+0000","last_clean":"2023-03-03T05:45:36.853682+0000","last_became_active":"2023-03-03T05:45:35.855388+0000","last_became_peered":"2023-03-03T05:45:35.855388+0000","last_unstale":"2023-03-03T05:45:36.853682+0000","last_undegraded":"2023-03-03T05:45:36.853682+0000","last_fullsized":"2023-03-03T05:45:36.853682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923513+0000","last_change":"2023-03-03T05:45:35.863167+0000","last_active":"2023-03-03T05:45:36.923513+0000","last_peered":"2023-03-03T05:45:36.923513+0000","last_clean":"2023-03-03T05:45:36.923513+0000","last_became_active":"2023-03-03T05:45:35.862519+0000","last_became_peered":"2023-03-03T05:45:35.862519+0000","last_unstale":"2023-03-03T05:45:36.923513+0000","last_undegraded":"2023-03-03T05:45:36.923513+0000","last_fullsized":"2023-03-03T05:45:36.923513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851022+0000","last_change":"2023-03-03T05:45:35.870150+0000","last_active":"2023-03-03T05:45:36.851022+0000","last_peered":"2023-03-03T05:45:36.851022+0000","last_clean":"2023-03-03T05:45:36.851022+0000","last_became_active":"2023-03-03T05:45:35.868786+0000","last_became_peered":"2023-03-03T05:45:35.868786+0000","last_unstale":"2023-03-03T05:45:36.851022+0000","last_undegraded":"2023-03-03T05:45:36.851022+0000","last_fullsized":"2023-03-03T05:45:36.851022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:52:23.931405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.310449+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.310197+0000","last_became_peered":"2023-03-03T05:45:35.310197+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923377+0000","last_change":"2023-03-03T05:45:35.861444+0000","last_active":"2023-03-03T05:45:36.923377+0000","last_peered":"2023-03-03T05:45:36.923377+0000","last_clean":"2023-03-03T05:45:36.923377+0000","last_became_active":"2023-03-03T05:45:35.861129+0000","last_became_peered":"2023-03-03T05:45:35.861129+0000","last_unstale":"2023-03-03T05:45:36.923377+0000","last_undegraded":"2023-03-03T05:45:36.923377+0000","last_fullsized":"2023-03-03T05:45:36.923377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843345+0000","last_change":"2023-03-03T05:45:35.308492+0000","last_active":"2023-03-03T05:45:35.843345+0000","last_peered":"2023-03-03T05:45:35.843345+0000","last_clean":"2023-03-03T05:45:35.843345+0000","last_became_active":"2023-03-03T05:45:35.308276+0000","last_became_peered":"2023-03-03T05:45:35.308276+0000","last_unstale":"2023-03-03T05:45:35.843345+0000","last_undegraded":"2023-03-03T05:45:35.843345+0000","last_fullsized":"2023-03-03T05:45:35.843345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:32.932001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853620+0000","last_change":"2023-03-03T05:45:35.866408+0000","last_active":"2023-03-03T05:45:36.853620+0000","last_peered":"2023-03-03T05:45:36.853620+0000","last_clean":"2023-03-03T05:45:36.853620+0000","last_became_active":"2023-03-03T05:45:35.866080+0000","last_became_peered":"2023-03-03T05:45:35.866080+0000","last_unstale":"2023-03-03T05:45:36.853620+0000","last_undegraded":"2023-03-03T05:45:36.853620+0000","last_fullsized":"2023-03-03T05:45:36.853620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:22.678556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848138+0000","last_change":"2023-03-03T05:45:35.293990+0000","last_active":"2023-03-03T05:45:35.848138+0000","last_peered":"2023-03-03T05:45:35.848138+0000","last_clean":"2023-03-03T05:45:35.848138+0000","last_became_active":"2023-03-03T05:45:35.293655+0000","last_became_peered":"2023-03-03T05:45:35.293655+0000","last_unstale":"2023-03-03T05:45:35.848138+0000","last_undegraded":"2023-03-03T05:45:35.848138+0000","last_fullsized":"2023-03-03T05:45:35.848138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850700+0000","last_change":"2023-03-03T05:45:35.852599+0000","last_active":"2023-03-03T05:45:36.850700+0000","last_peered":"2023-03-03T05:45:36.850700+0000","last_clean":"2023-03-03T05:45:36.850700+0000","last_became_active":"2023-03-03T05:45:35.852292+0000","last_became_peered":"2023-03-03T05:45:35.852292+0000","last_unstale":"2023-03-03T05:45:36.850700+0000","last_undegraded":"2023-03-03T05:45:36.850700+0000","last_fullsized":"2023-03-03T05:45:36.850700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:06.104140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848641+0000","last_change":"2023-03-03T05:45:35.297718+0000","last_active":"2023-03-03T05:45:35.848641+0000","last_peered":"2023-03-03T05:45:35.848641+0000","last_clean":"2023-03-03T05:45:35.848641+0000","last_became_active":"2023-03-03T05:45:35.297435+0000","last_became_peered":"2023-03-03T05:45:35.297435+0000","last_unstale":"2023-03-03T05:45:35.848641+0000","last_undegraded":"2023-03-03T05:45:35.848641+0000","last_fullsized":"2023-03-03T05:45:35.848641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:17:39.778447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923577+0000","last_change":"2023-03-03T05:45:35.871116+0000","last_active":"2023-03-03T05:45:36.923577+0000","last_peered":"2023-03-03T05:45:36.923577+0000","last_clean":"2023-03-03T05:45:36.923577+0000","last_became_active":"2023-03-03T05:45:35.870843+0000","last_became_peered":"2023-03-03T05:45:35.870843+0000","last_unstale":"2023-03-03T05:45:36.923577+0000","last_undegraded":"2023-03-03T05:45:36.923577+0000","last_fullsized":"2023-03-03T05:45:36.923577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851170+0000","last_change":"2023-03-03T05:45:35.304400+0000","last_active":"2023-03-03T05:45:36.851170+0000","last_peered":"2023-03-03T05:45:36.851170+0000","last_clean":"2023-03-03T05:45:36.851170+0000","last_became_active":"2023-03-03T05:45:35.304093+0000","last_became_peered":"2023-03-03T05:45:35.304093+0000","last_unstale":"2023-03-03T05:45:36.851170+0000","last_undegraded":"2023-03-03T05:45:36.851170+0000","last_fullsized":"2023-03-03T05:45:36.851170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853210+0000","last_change":"2023-03-03T05:45:35.862492+0000","last_active":"2023-03-03T05:45:36.853210+0000","last_peered":"2023-03-03T05:45:36.853210+0000","last_clean":"2023-03-03T05:45:36.853210+0000","last_became_active":"2023-03-03T05:45:35.862173+0000","last_became_peered":"2023-03-03T05:45:35.862173+0000","last_unstale":"2023-03-03T05:45:36.853210+0000","last_undegraded":"2023-03-03T05:45:36.853210+0000","last_fullsized":"2023-03-03T05:45:36.853210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.853604+0000","last_change":"2023-03-03T05:45:35.302282+0000","last_active":"2023-03-03T05:45:40.853604+0000","last_peered":"2023-03-03T05:45:40.853604+0000","last_clean":"2023-03-03T05:45:40.853604+0000","last_became_active":"2023-03-03T05:45:35.302002+0000","last_became_peered":"2023-03-03T05:45:35.302002+0000","last_unstale":"2023-03-03T05:45:40.853604+0000","last_undegraded":"2023-03-03T05:45:40.853604+0000","last_fullsized":"2023-03-03T05:45:40.853604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853414+0000","last_change":"2023-03-03T05:45:35.855928+0000","last_active":"2023-03-03T05:45:36.853414+0000","last_peered":"2023-03-03T05:45:36.853414+0000","last_clean":"2023-03-03T05:45:36.853414+0000","last_became_active":"2023-03-03T05:45:35.855612+0000","last_became_peered":"2023-03-03T05:45:35.855612+0000","last_unstale":"2023-03-03T05:45:36.853414+0000","last_undegraded":"2023-03-03T05:45:36.853414+0000","last_fullsized":"2023-03-03T05:45:36.853414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:57.313631+0000","last_change":"2023-03-03T05:45:35.304856+0000","last_active":"2023-03-03T05:45:57.313631+0000","last_peered":"2023-03-03T05:45:57.313631+0000","last_clean":"2023-03-03T05:45:57.313631+0000","last_became_active":"2023-03-03T05:45:35.304439+0000","last_became_peered":"2023-03-03T05:45:35.304439+0000","last_unstale":"2023-03-03T05:45:57.313631+0000","last_undegraded":"2023-03-03T05:45:57.313631+0000","last_fullsized":"2023-03-03T05:45:57.313631+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:30.692527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1807,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850547+0000","last_change":"2023-03-03T05:45:35.856887+0000","last_active":"2023-03-03T05:45:36.850547+0000","last_peered":"2023-03-03T05:45:36.850547+0000","last_clean":"2023-03-03T05:45:36.850547+0000","last_became_active":"2023-03-03T05:45:35.856498+0000","last_became_peered":"2023-03-03T05:45:35.856498+0000","last_unstale":"2023-03-03T05:45:36.850547+0000","last_undegraded":"2023-03-03T05:45:36.850547+0000","last_fullsized":"2023-03-03T05:45:36.850547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858066+0000","last_change":"2023-03-03T05:45:35.297635+0000","last_active":"2023-03-03T05:45:40.858066+0000","last_peered":"2023-03-03T05:45:40.858066+0000","last_clean":"2023-03-03T05:45:40.858066+0000","last_became_active":"2023-03-03T05:45:35.297373+0000","last_became_peered":"2023-03-03T05:45:35.297373+0000","last_unstale":"2023-03-03T05:45:40.858066+0000","last_undegraded":"2023-03-03T05:45:40.858066+0000","last_fullsized":"2023-03-03T05:45:40.858066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:48.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922855+0000","last_change":"2023-03-03T05:45:35.852113+0000","last_active":"2023-03-03T05:45:36.922855+0000","last_peered":"2023-03-03T05:45:36.922855+0000","last_clean":"2023-03-03T05:45:36.922855+0000","last_became_active":"2023-03-03T05:45:35.851799+0000","last_became_peered":"2023-03-03T05:45:35.851799+0000","last_unstale":"2023-03-03T05:45:36.922855+0000","last_undegraded":"2023-03-03T05:45:36.922855+0000","last_fullsized":"2023-03-03T05:45:36.922855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.845615+0000","last_change":"2023-03-03T05:45:35.292083+0000","last_active":"2023-03-03T05:45:35.845615+0000","last_peered":"2023-03-03T05:45:35.845615+0000","last_clean":"2023-03-03T05:45:35.845615+0000","last_became_active":"2023-03-03T05:45:35.291766+0000","last_became_peered":"2023-03-03T05:45:35.291766+0000","last_unstale":"2023-03-03T05:45:35.845615+0000","last_undegraded":"2023-03-03T05:45:35.845615+0000","last_fullsized":"2023-03-03T05:45:35.845615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851348+0000","last_change":"2023-03-03T05:45:35.870363+0000","last_active":"2023-03-03T05:45:36.851348+0000","last_peered":"2023-03-03T05:45:36.851348+0000","last_clean":"2023-03-03T05:45:36.851348+0000","last_became_active":"2023-03-03T05:45:35.868801+0000","last_became_peered":"2023-03-03T05:45:35.868801+0000","last_unstale":"2023-03-03T05:45:36.851348+0000","last_undegraded":"2023-03-03T05:45:36.851348+0000","last_fullsized":"2023-03-03T05:45:36.851348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:04.045341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844327+0000","last_change":"2023-03-03T05:45:35.304592+0000","last_active":"2023-03-03T05:45:35.844327+0000","last_peered":"2023-03-03T05:45:35.844327+0000","last_clean":"2023-03-03T05:45:35.844327+0000","last_became_active":"2023-03-03T05:45:35.304318+0000","last_became_peered":"2023-03-03T05:45:35.304318+0000","last_unstale":"2023-03-03T05:45:35.844327+0000","last_undegraded":"2023-03-03T05:45:35.844327+0000","last_fullsized":"2023-03-03T05:45:35.844327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:08.368039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922933+0000","last_change":"2023-03-03T05:45:35.857951+0000","last_active":"2023-03-03T05:45:36.922933+0000","last_peered":"2023-03-03T05:45:36.922933+0000","last_clean":"2023-03-03T05:45:36.922933+0000","last_became_active":"2023-03-03T05:45:35.857581+0000","last_became_peered":"2023-03-03T05:45:35.857581+0000","last_unstale":"2023-03-03T05:45:36.922933+0000","last_undegraded":"2023-03-03T05:45:36.922933+0000","last_fullsized":"2023-03-03T05:45:36.922933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858611+0000","last_change":"2023-03-03T05:45:35.309150+0000","last_active":"2023-03-03T05:45:40.858611+0000","last_peered":"2023-03-03T05:45:40.858611+0000","last_clean":"2023-03-03T05:45:40.858611+0000","last_became_active":"2023-03-03T05:45:35.308840+0000","last_became_peered":"2023-03-03T05:45:35.308840+0000","last_unstale":"2023-03-03T05:45:40.858611+0000","last_undegraded":"2023-03-03T05:45:40.858611+0000","last_fullsized":"2023-03-03T05:45:40.858611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923050+0000","last_change":"2023-03-03T05:45:35.864919+0000","last_active":"2023-03-03T05:45:36.923050+0000","last_peered":"2023-03-03T05:45:36.923050+0000","last_clean":"2023-03-03T05:45:36.923050+0000","last_became_active":"2023-03-03T05:45:35.864595+0000","last_became_peered":"2023-03-03T05:45:35.864595+0000","last_unstale":"2023-03-03T05:45:36.923050+0000","last_undegraded":"2023-03-03T05:45:36.923050+0000","last_fullsized":"2023-03-03T05:45:36.923050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.308976+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.308707+0000","last_became_peered":"2023-03-03T05:45:35.308707+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850640+0000","last_change":"2023-03-03T05:45:35.868861+0000","last_active":"2023-03-03T05:45:36.850640+0000","last_peered":"2023-03-03T05:45:36.850640+0000","last_clean":"2023-03-03T05:45:36.850640+0000","last_became_active":"2023-03-03T05:45:35.868438+0000","last_became_peered":"2023-03-03T05:45:35.868438+0000","last_unstale":"2023-03-03T05:45:36.850640+0000","last_undegraded":"2023-03-03T05:45:36.850640+0000","last_fullsized":"2023-03-03T05:45:36.850640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:59.810575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.854269+0000","last_change":"2023-03-03T05:45:35.303791+0000","last_active":"2023-03-03T05:45:40.854269+0000","last_peered":"2023-03-03T05:45:40.854269+0000","last_clean":"2023-03-03T05:45:40.854269+0000","last_became_active":"2023-03-03T05:45:35.303482+0000","last_became_peered":"2023-03-03T05:45:35.303482+0000","last_unstale":"2023-03-03T05:45:40.854269+0000","last_undegraded":"2023-03-03T05:45:40.854269+0000","last_fullsized":"2023-03-03T05:45:40.854269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:31.933113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923037+0000","last_change":"2023-03-03T05:45:35.864783+0000","last_active":"2023-03-03T05:45:36.923037+0000","last_peered":"2023-03-03T05:45:36.923037+0000","last_clean":"2023-03-03T05:45:36.923037+0000","last_became_active":"2023-03-03T05:45:35.864462+0000","last_became_peered":"2023-03-03T05:45:35.864462+0000","last_unstale":"2023-03-03T05:45:36.923037+0000","last_undegraded":"2023-03-03T05:45:36.923037+0000","last_fullsized":"2023-03-03T05:45:36.923037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851096+0000","last_change":"2023-03-03T05:45:35.310813+0000","last_active":"2023-03-03T05:45:36.851096+0000","last_peered":"2023-03-03T05:45:36.851096+0000","last_clean":"2023-03-03T05:45:36.851096+0000","last_became_active":"2023-03-03T05:45:35.310466+0000","last_became_peered":"2023-03-03T05:45:35.310466+0000","last_unstale":"2023-03-03T05:45:36.851096+0000","last_undegraded":"2023-03-03T05:45:36.851096+0000","last_fullsized":"2023-03-03T05:45:36.851096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:46.322139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850472+0000","last_change":"2023-03-03T05:45:35.856376+0000","last_active":"2023-03-03T05:45:36.850472+0000","last_peered":"2023-03-03T05:45:36.850472+0000","last_clean":"2023-03-03T05:45:36.850472+0000","last_became_active":"2023-03-03T05:45:35.856064+0000","last_became_peered":"2023-03-03T05:45:35.856064+0000","last_unstale":"2023-03-03T05:45:36.850472+0000","last_undegraded":"2023-03-03T05:45:36.850472+0000","last_fullsized":"2023-03-03T05:45:36.850472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962714+0000","last_change":"2023-03-03T05:45:35.295520+0000","last_active":"2023-03-03T05:45:35.962714+0000","last_peered":"2023-03-03T05:45:35.962714+0000","last_clean":"2023-03-03T05:45:35.962714+0000","last_became_active":"2023-03-03T05:45:35.295203+0000","last_became_peered":"2023-03-03T05:45:35.295203+0000","last_unstale":"2023-03-03T05:45:35.962714+0000","last_undegraded":"2023-03-03T05:45:35.962714+0000","last_fullsized":"2023-03-03T05:45:35.962714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923389+0000","last_change":"2023-03-03T05:45:35.852242+0000","last_active":"2023-03-03T05:45:36.923389+0000","last_peered":"2023-03-03T05:45:36.923389+0000","last_clean":"2023-03-03T05:45:36.923389+0000","last_became_active":"2023-03-03T05:45:35.851803+0000","last_became_peered":"2023-03-03T05:45:35.851803+0000","last_unstale":"2023-03-03T05:45:36.923389+0000","last_undegraded":"2023-03-03T05:45:36.923389+0000","last_fullsized":"2023-03-03T05:45:36.923389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:14.473334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.852939+0000","last_change":"2023-03-03T05:45:35.298542+0000","last_active":"2023-03-03T05:45:40.852939+0000","last_peered":"2023-03-03T05:45:40.852939+0000","last_clean":"2023-03-03T05:45:40.852939+0000","last_became_active":"2023-03-03T05:45:35.298219+0000","last_became_peered":"2023-03-03T05:45:35.298219+0000","last_unstale":"2023-03-03T05:45:40.852939+0000","last_undegraded":"2023-03-03T05:45:40.852939+0000","last_fullsized":"2023-03-03T05:45:40.852939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962699+0000","last_change":"2023-03-03T05:45:35.861231+0000","last_active":"2023-03-03T05:45:35.962699+0000","last_peered":"2023-03-03T05:45:35.962699+0000","last_clean":"2023-03-03T05:45:35.962699+0000","last_became_active":"2023-03-03T05:45:35.860675+0000","last_became_peered":"2023-03-03T05:45:35.860675+0000","last_unstale":"2023-03-03T05:45:35.962699+0000","last_undegraded":"2023-03-03T05:45:35.962699+0000","last_fullsized":"2023-03-03T05:45:35.962699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843762+0000","last_change":"2023-03-03T05:45:35.312143+0000","last_active":"2023-03-03T05:45:35.843762+0000","last_peered":"2023-03-03T05:45:35.843762+0000","last_clean":"2023-03-03T05:45:35.843762+0000","last_became_active":"2023-03-03T05:45:35.311848+0000","last_became_peered":"2023-03-03T05:45:35.311848+0000","last_unstale":"2023-03-03T05:45:35.843762+0000","last_undegraded":"2023-03-03T05:45:35.843762+0000","last_fullsized":"2023-03-03T05:45:35.843762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:22.294783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851303+0000","last_change":"2023-03-03T05:45:35.869075+0000","last_active":"2023-03-03T05:45:36.851303+0000","last_peered":"2023-03-03T05:45:36.851303+0000","last_clean":"2023-03-03T05:45:36.851303+0000","last_became_active":"2023-03-03T05:45:35.868413+0000","last_became_peered":"2023-03-03T05:45:35.868413+0000","last_unstale":"2023-03-03T05:45:36.851303+0000","last_undegraded":"2023-03-03T05:45:36.851303+0000","last_fullsized":"2023-03-03T05:45:36.851303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:34.743111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844226+0000","last_change":"2023-03-03T05:45:35.296611+0000","last_active":"2023-03-03T05:45:35.844226+0000","last_peered":"2023-03-03T05:45:35.844226+0000","last_clean":"2023-03-03T05:45:35.844226+0000","last_became_active":"2023-03-03T05:45:35.296294+0000","last_became_peered":"2023-03-03T05:45:35.296294+0000","last_unstale":"2023-03-03T05:45:35.844226+0000","last_undegraded":"2023-03-03T05:45:35.844226+0000","last_fullsized":"2023-03-03T05:45:35.844226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:16.655836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850964+0000","last_change":"2023-03-03T05:45:35.857451+0000","last_active":"2023-03-03T05:45:36.850964+0000","last_peered":"2023-03-03T05:45:36.850964+0000","last_clean":"2023-03-03T05:45:36.850964+0000","last_became_active":"2023-03-03T05:45:35.857087+0000","last_became_peered":"2023-03-03T05:45:35.857087+0000","last_unstale":"2023-03-03T05:45:36.850964+0000","last_undegraded":"2023-03-03T05:45:36.850964+0000","last_fullsized":"2023-03-03T05:45:36.850964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842797+0000","last_change":"2023-03-03T05:45:35.296114+0000","last_active":"2023-03-03T05:45:35.842797+0000","last_peered":"2023-03-03T05:45:35.842797+0000","last_clean":"2023-03-03T05:45:35.842797+0000","last_became_active":"2023-03-03T05:45:35.295855+0000","last_became_peered":"2023-03-03T05:45:35.295855+0000","last_unstale":"2023-03-03T05:45:35.842797+0000","last_undegraded":"2023-03-03T05:45:35.842797+0000","last_fullsized":"2023-03-03T05:45:35.842797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923246+0000","last_change":"2023-03-03T05:45:35.862249+0000","last_active":"2023-03-03T05:45:36.923246+0000","last_peered":"2023-03-03T05:45:36.923246+0000","last_clean":"2023-03-03T05:45:36.923246+0000","last_became_active":"2023-03-03T05:45:35.861773+0000","last_became_peered":"2023-03-03T05:45:35.861773+0000","last_unstale":"2023-03-03T05:45:36.923246+0000","last_undegraded":"2023-03-03T05:45:36.923246+0000","last_fullsized":"2023-03-03T05:45:36.923246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848485+0000","last_change":"2023-03-03T05:45:29.189033+0000","last_active":"2023-03-03T05:45:35.848485+0000","last_peered":"2023-03-03T05:45:35.848485+0000","last_clean":"2023-03-03T05:45:35.848485+0000","last_became_active":"2023-03-03T05:45:29.188719+0000","last_became_peered":"2023-03-03T05:45:29.188719+0000","last_unstale":"2023-03-03T05:45:35.848485+0000","last_undegraded":"2023-03-03T05:45:35.848485+0000","last_fullsized":"2023-03-03T05:45:35.848485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850356+0000","last_change":"2023-03-03T05:45:35.303756+0000","last_active":"2023-03-03T05:45:36.850356+0000","last_peered":"2023-03-03T05:45:36.850356+0000","last_clean":"2023-03-03T05:45:36.850356+0000","last_became_active":"2023-03-03T05:45:35.303448+0000","last_became_peered":"2023-03-03T05:45:35.303448+0000","last_unstale":"2023-03-03T05:45:36.850356+0000","last_undegraded":"2023-03-03T05:45:36.850356+0000","last_fullsized":"2023-03-03T05:45:36.850356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922899+0000","last_change":"2023-03-03T05:45:35.854166+0000","last_active":"2023-03-03T05:45:36.922899+0000","last_peered":"2023-03-03T05:45:36.922899+0000","last_clean":"2023-03-03T05:45:36.922899+0000","last_became_active":"2023-03-03T05:45:35.853873+0000","last_became_peered":"2023-03-03T05:45:35.853873+0000","last_unstale":"2023-03-03T05:45:36.922899+0000","last_undegraded":"2023-03-03T05:45:36.922899+0000","last_fullsized":"2023-03-03T05:45:36.922899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843699+0000","last_change":"2023-03-03T05:45:29.189075+0000","last_active":"2023-03-03T05:45:35.843699+0000","last_peered":"2023-03-03T05:45:35.843699+0000","last_clean":"2023-03-03T05:45:35.843699+0000","last_became_active":"2023-03-03T05:45:29.188778+0000","last_became_peered":"2023-03-03T05:45:29.188778+0000","last_unstale":"2023-03-03T05:45:35.843699+0000","last_undegraded":"2023-03-03T05:45:35.843699+0000","last_fullsized":"2023-03-03T05:45:35.843699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:54:35.536172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962645+0000","last_change":"2023-03-03T05:45:35.309781+0000","last_active":"2023-03-03T05:45:35.962645+0000","last_peered":"2023-03-03T05:45:35.962645+0000","last_clean":"2023-03-03T05:45:35.962645+0000","last_became_active":"2023-03-03T05:45:35.309497+0000","last_became_peered":"2023-03-03T05:45:35.309497+0000","last_unstale":"2023-03-03T05:45:35.962645+0000","last_undegraded":"2023-03-03T05:45:35.962645+0000","last_fullsized":"2023-03-03T05:45:35.962645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923571+0000","last_change":"2023-03-03T05:45:35.854721+0000","last_active":"2023-03-03T05:45:36.923571+0000","last_peered":"2023-03-03T05:45:36.923571+0000","last_clean":"2023-03-03T05:45:36.923571+0000","last_became_active":"2023-03-03T05:45:35.854423+0000","last_became_peered":"2023-03-03T05:45:35.854423+0000","last_unstale":"2023-03-03T05:45:36.923571+0000","last_undegraded":"2023-03-03T05:45:36.923571+0000","last_fullsized":"2023-03-03T05:45:36.923571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.852125+0000","last_change":"2023-03-03T05:45:29.057308+0000","last_active":"2023-03-03T05:45:35.852125+0000","last_peered":"2023-03-03T05:45:35.852125+0000","last_clean":"2023-03-03T05:45:35.852125+0000","last_became_active":"2023-03-03T05:45:29.056828+0000","last_became_peered":"2023-03-03T05:45:29.056828+0000","last_unstale":"2023-03-03T05:45:35.852125+0000","last_undegraded":"2023-03-03T05:45:35.852125+0000","last_fullsized":"2023-03-03T05:45:35.852125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:56.079893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848095+0000","last_change":"2023-03-03T05:45:35.295842+0000","last_active":"2023-03-03T05:45:35.848095+0000","last_peered":"2023-03-03T05:45:35.848095+0000","last_clean":"2023-03-03T05:45:35.848095+0000","last_became_active":"2023-03-03T05:45:35.295545+0000","last_became_peered":"2023-03-03T05:45:35.295545+0000","last_unstale":"2023-03-03T05:45:35.848095+0000","last_undegraded":"2023-03-03T05:45:35.848095+0000","last_fullsized":"2023-03-03T05:45:35.848095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923636+0000","last_change":"2023-03-03T05:45:35.862437+0000","last_active":"2023-03-03T05:45:36.923636+0000","last_peered":"2023-03-03T05:45:36.923636+0000","last_clean":"2023-03-03T05:45:36.923636+0000","last_became_active":"2023-03-03T05:45:35.862152+0000","last_became_peered":"2023-03-03T05:45:35.862152+0000","last_unstale":"2023-03-03T05:45:36.923636+0000","last_undegraded":"2023-03-03T05:45:36.923636+0000","last_fullsized":"2023-03-03T05:45:36.923636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842862+0000","last_change":"2023-03-03T05:45:29.057646+0000","last_active":"2023-03-03T05:45:35.842862+0000","last_peered":"2023-03-03T05:45:35.842862+0000","last_clean":"2023-03-03T05:45:35.842862+0000","last_became_active":"2023-03-03T05:45:29.057330+0000","last_became_peered":"2023-03-03T05:45:29.057330+0000","last_unstale":"2023-03-03T05:45:35.842862+0000","last_undegraded":"2023-03-03T05:45:35.842862+0000","last_fullsized":"2023-03-03T05:45:35.842862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:22:57.039574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.858102+0000","last_change":"2023-03-03T05:45:22.224435+0000","last_active":"2023-03-03T05:45:35.858102+0000","last_peered":"2023-03-03T05:45:35.858102+0000","last_clean":"2023-03-03T05:45:35.858102+0000","last_became_active":"2023-03-03T05:45:22.222958+0000","last_became_peered":"2023-03-03T05:45:22.222958+0000","last_unstale":"2023-03-03T05:45:35.858102+0000","last_undegraded":"2023-03-03T05:45:35.858102+0000","last_fullsized":"2023-03-03T05:45:35.858102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_clean_scrub_stamp":"2023-03-03T05:45:20.925942+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:10:20.426026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843695+0000","last_change":"2023-03-03T05:45:35.304483+0000","last_active":"2023-03-03T05:45:35.843695+0000","last_peered":"2023-03-03T05:45:35.843695+0000","last_clean":"2023-03-03T05:45:35.843695+0000","last_became_active":"2023-03-03T05:45:35.304202+0000","last_became_peered":"2023-03-03T05:45:35.304202+0000","last_unstale":"2023-03-03T05:45:35.843695+0000","last_undegraded":"2023-03-03T05:45:35.843695+0000","last_fullsized":"2023-03-03T05:45:35.843695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851295+0000","last_change":"2023-03-03T05:45:35.869049+0000","last_active":"2023-03-03T05:45:36.851295+0000","last_peered":"2023-03-03T05:45:36.851295+0000","last_clean":"2023-03-03T05:45:36.851295+0000","last_became_active":"2023-03-03T05:45:35.868376+0000","last_became_peered":"2023-03-03T05:45:35.868376+0000","last_unstale":"2023-03-03T05:45:36.851295+0000","last_undegraded":"2023-03-03T05:45:36.851295+0000","last_fullsized":"2023-03-03T05:45:36.851295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:42:55.207846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843590+0000","last_change":"2023-03-03T05:45:29.056874+0000","last_active":"2023-03-03T05:45:35.843590+0000","last_peered":"2023-03-03T05:45:35.843590+0000","last_clean":"2023-03-03T05:45:35.843590+0000","last_became_active":"2023-03-03T05:45:29.056438+0000","last_became_peered":"2023-03-03T05:45:29.056438+0000","last_unstale":"2023-03-03T05:45:35.843590+0000","last_undegraded":"2023-03-03T05:45:35.843590+0000","last_fullsized":"2023-03-03T05:45:35.843590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:08.113953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.846141+0000","last_change":"2023-03-03T05:45:35.304562+0000","last_active":"2023-03-03T05:45:35.846141+0000","last_peered":"2023-03-03T05:45:35.846141+0000","last_clean":"2023-03-03T05:45:35.846141+0000","last_became_active":"2023-03-03T05:45:35.304264+0000","last_became_peered":"2023-03-03T05:45:35.304264+0000","last_unstale":"2023-03-03T05:45:35.846141+0000","last_undegraded":"2023-03-03T05:45:35.846141+0000","last_fullsized":"2023-03-03T05:45:35.846141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923094+0000","last_change":"2023-03-03T05:45:35.859779+0000","last_active":"2023-03-03T05:45:36.923094+0000","last_peered":"2023-03-03T05:45:36.923094+0000","last_clean":"2023-03-03T05:45:36.923094+0000","last_became_active":"2023-03-03T05:45:35.857596+0000","last_became_peered":"2023-03-03T05:45:35.857596+0000","last_unstale":"2023-03-03T05:45:36.923094+0000","last_undegraded":"2023-03-03T05:45:36.923094+0000","last_fullsized":"2023-03-03T05:45:36.923094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844418+0000","last_change":"2023-03-03T05:45:29.056891+0000","last_active":"2023-03-03T05:45:35.844418+0000","last_peered":"2023-03-03T05:45:35.844418+0000","last_clean":"2023-03-03T05:45:35.844418+0000","last_became_active":"2023-03-03T05:45:29.056483+0000","last_became_peered":"2023-03-03T05:45:29.056483+0000","last_unstale":"2023-03-03T05:45:35.844418+0000","last_undegraded":"2023-03-03T05:45:35.844418+0000","last_fullsized":"2023-03-03T05:45:35.844418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:25.701585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.847358+0000","last_change":"2023-03-03T05:45:35.293821+0000","last_active":"2023-03-03T05:45:35.847358+0000","last_peered":"2023-03-03T05:45:35.847358+0000","last_clean":"2023-03-03T05:45:35.847358+0000","last_became_active":"2023-03-03T05:45:35.293533+0000","last_became_peered":"2023-03-03T05:45:35.293533+0000","last_unstale":"2023-03-03T05:45:35.847358+0000","last_undegraded":"2023-03-03T05:45:35.847358+0000","last_fullsized":"2023-03-03T05:45:35.847358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923917+0000","last_change":"2023-03-03T05:45:35.866830+0000","last_active":"2023-03-03T05:45:36.923917+0000","last_peered":"2023-03-03T05:45:36.923917+0000","last_clean":"2023-03-03T05:45:36.923917+0000","last_became_active":"2023-03-03T05:45:35.866509+0000","last_became_peered":"2023-03-03T05:45:35.866509+0000","last_unstale":"2023-03-03T05:45:36.923917+0000","last_undegraded":"2023-03-03T05:45:36.923917+0000","last_fullsized":"2023-03-03T05:45:36.923917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.850545+0000","last_change":"2023-03-03T05:45:35.300794+0000","last_active":"2023-03-03T05:45:40.850545+0000","last_peered":"2023-03-03T05:45:40.850545+0000","last_clean":"2023-03-03T05:45:40.850545+0000","last_became_active":"2023-03-03T05:45:35.300531+0000","last_became_peered":"2023-03-03T05:45:35.300531+0000","last_unstale":"2023-03-03T05:45:40.850545+0000","last_undegraded":"2023-03-03T05:45:40.850545+0000","last_fullsized":"2023-03-03T05:45:40.850545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843343+0000","last_change":"2023-03-03T05:45:29.193601+0000","last_active":"2023-03-03T05:45:35.843343+0000","last_peered":"2023-03-03T05:45:35.843343+0000","last_clean":"2023-03-03T05:45:35.843343+0000","last_became_active":"2023-03-03T05:45:29.193262+0000","last_became_peered":"2023-03-03T05:45:29.193262+0000","last_unstale":"2023-03-03T05:45:35.843343+0000","last_undegraded":"2023-03-03T05:45:35.843343+0000","last_fullsized":"2023-03-03T05:45:35.843343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853228+0000","last_change":"2023-03-03T05:45:35.864517+0000","last_active":"2023-03-03T05:45:36.853228+0000","last_peered":"2023-03-03T05:45:36.853228+0000","last_clean":"2023-03-03T05:45:36.853228+0000","last_became_active":"2023-03-03T05:45:35.864193+0000","last_became_peered":"2023-03-03T05:45:35.864193+0000","last_unstale":"2023-03-03T05:45:36.853228+0000","last_undegraded":"2023-03-03T05:45:36.853228+0000","last_fullsized":"2023-03-03T05:45:36.853228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:11:08.433881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844985+0000","last_change":"2023-03-03T05:45:29.193546+0000","last_active":"2023-03-03T05:45:35.844985+0000","last_peered":"2023-03-03T05:45:35.844985+0000","last_clean":"2023-03-03T05:45:35.844985+0000","last_became_active":"2023-03-03T05:45:29.193172+0000","last_became_peered":"2023-03-03T05:45:29.193172+0000","last_unstale":"2023-03-03T05:45:35.844985+0000","last_undegraded":"2023-03-03T05:45:35.844985+0000","last_fullsized":"2023-03-03T05:45:35.844985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:07:42.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853612+0000","last_change":"2023-03-03T05:45:35.296693+0000","last_active":"2023-03-03T05:45:36.853612+0000","last_peered":"2023-03-03T05:45:36.853612+0000","last_clean":"2023-03-03T05:45:36.853612+0000","last_became_active":"2023-03-03T05:45:35.296469+0000","last_became_peered":"2023-03-03T05:45:35.296469+0000","last_unstale":"2023-03-03T05:45:36.853612+0000","last_undegraded":"2023-03-03T05:45:36.853612+0000","last_fullsized":"2023-03-03T05:45:36.853612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850872+0000","last_change":"2023-03-03T05:45:35.856468+0000","last_active":"2023-03-03T05:45:36.850872+0000","last_peered":"2023-03-03T05:45:36.850872+0000","last_clean":"2023-03-03T05:45:36.850872+0000","last_became_active":"2023-03-03T05:45:35.855970+0000","last_became_peered":"2023-03-03T05:45:35.855970+0000","last_unstale":"2023-03-03T05:45:36.850872+0000","last_undegraded":"2023-03-03T05:45:36.850872+0000","last_fullsized":"2023-03-03T05:45:36.850872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853299+0000","last_change":"2023-03-03T05:45:35.301855+0000","last_active":"2023-03-03T05:45:36.853299+0000","last_peered":"2023-03-03T05:45:36.853299+0000","last_clean":"2023-03-03T05:45:36.853299+0000","last_became_active":"2023-03-03T05:45:35.301622+0000","last_became_peered":"2023-03-03T05:45:35.301622+0000","last_unstale":"2023-03-03T05:45:36.853299+0000","last_undegraded":"2023-03-03T05:45:36.853299+0000","last_fullsized":"2023-03-03T05:45:36.853299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:42.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853538+0000","last_change":"2023-03-03T05:45:35.851780+0000","last_active":"2023-03-03T05:45:36.853538+0000","last_peered":"2023-03-03T05:45:36.853538+0000","last_clean":"2023-03-03T05:45:36.853538+0000","last_became_active":"2023-03-03T05:45:35.850190+0000","last_became_peered":"2023-03-03T05:45:35.850190+0000","last_unstale":"2023-03-03T05:45:36.853538+0000","last_undegraded":"2023-03-03T05:45:36.853538+0000","last_fullsized":"2023-03-03T05:45:36.853538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856909+0000","last_change":"2023-03-03T05:45:35.309047+0000","last_active":"2023-03-03T05:45:40.856909+0000","last_peered":"2023-03-03T05:45:40.856909+0000","last_clean":"2023-03-03T05:45:40.856909+0000","last_became_active":"2023-03-03T05:45:35.308564+0000","last_became_peered":"2023-03-03T05:45:35.308564+0000","last_unstale":"2023-03-03T05:45:40.856909+0000","last_undegraded":"2023-03-03T05:45:40.856909+0000","last_fullsized":"2023-03-03T05:45:40.856909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850462+0000","last_change":"2023-03-03T05:45:35.861316+0000","last_active":"2023-03-03T05:45:36.850462+0000","last_peered":"2023-03-03T05:45:36.850462+0000","last_clean":"2023-03-03T05:45:36.850462+0000","last_became_active":"2023-03-03T05:45:35.861052+0000","last_became_peered":"2023-03-03T05:45:35.861052+0000","last_unstale":"2023-03-03T05:45:36.850462+0000","last_undegraded":"2023-03-03T05:45:36.850462+0000","last_fullsized":"2023-03-03T05:45:36.850462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858612+0000","last_change":"2023-03-03T05:45:35.303666+0000","last_active":"2023-03-03T05:45:40.858612+0000","last_peered":"2023-03-03T05:45:40.858612+0000","last_clean":"2023-03-03T05:45:40.858612+0000","last_became_active":"2023-03-03T05:45:35.303213+0000","last_became_peered":"2023-03-03T05:45:35.303213+0000","last_unstale":"2023-03-03T05:45:40.858612+0000","last_undegraded":"2023-03-03T05:45:40.858612+0000","last_fullsized":"2023-03-03T05:45:40.858612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:54.389717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850392+0000","last_change":"2023-03-03T05:45:35.857442+0000","last_active":"2023-03-03T05:45:36.850392+0000","last_peered":"2023-03-03T05:45:36.850392+0000","last_clean":"2023-03-03T05:45:36.850392+0000","last_became_active":"2023-03-03T05:45:35.857130+0000","last_became_peered":"2023-03-03T05:45:35.857130+0000","last_unstale":"2023-03-03T05:45:36.850392+0000","last_undegraded":"2023-03-03T05:45:36.850392+0000","last_fullsized":"2023-03-03T05:45:36.850392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923020+0000","last_change":"2023-03-03T05:45:35.305098+0000","last_active":"2023-03-03T05:45:36.923020+0000","last_peered":"2023-03-03T05:45:36.923020+0000","last_clean":"2023-03-03T05:45:36.923020+0000","last_became_active":"2023-03-03T05:45:35.304812+0000","last_became_peered":"2023-03-03T05:45:35.304812+0000","last_unstale":"2023-03-03T05:45:36.923020+0000","last_undegraded":"2023-03-03T05:45:36.923020+0000","last_fullsized":"2023-03-03T05:45:36.923020+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923341+0000","last_change":"2023-03-03T05:45:35.858806+0000","last_active":"2023-03-03T05:45:36.923341+0000","last_peered":"2023-03-03T05:45:36.923341+0000","last_clean":"2023-03-03T05:45:36.923341+0000","last_became_active":"2023-03-03T05:45:35.858471+0000","last_became_peered":"2023-03-03T05:45:35.858471+0000","last_unstale":"2023-03-03T05:45:36.923341+0000","last_undegraded":"2023-03-03T05:45:36.923341+0000","last_fullsized":"2023-03-03T05:45:36.923341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856244+0000","last_change":"2023-03-03T05:45:35.310654+0000","last_active":"2023-03-03T05:45:40.856244+0000","last_peered":"2023-03-03T05:45:40.856244+0000","last_clean":"2023-03-03T05:45:40.856244+0000","last_became_active":"2023-03-03T05:45:35.310377+0000","last_became_peered":"2023-03-03T05:45:35.310377+0000","last_unstale":"2023-03-03T05:45:40.856244+0000","last_undegraded":"2023-03-03T05:45:40.856244+0000","last_fullsized":"2023-03-03T05:45:40.856244+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850763+0000","last_change":"2023-03-03T05:45:35.850670+0000","last_active":"2023-03-03T05:45:36.850763+0000","last_peered":"2023-03-03T05:45:36.850763+0000","last_clean":"2023-03-03T05:45:36.850763+0000","last_became_active":"2023-03-03T05:45:35.850380+0000","last_became_peered":"2023-03-03T05:45:35.850380+0000","last_unstale":"2023-03-03T05:45:36.850763+0000","last_undegraded":"2023-03-03T05:45:36.850763+0000","last_fullsized":"2023-03-03T05:45:36.850763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923387+0000","last_change":"2023-03-03T05:45:35.863660+0000","last_active":"2023-03-03T05:45:36.923387+0000","last_peered":"2023-03-03T05:45:36.923387+0000","last_clean":"2023-03-03T05:45:36.923387+0000","last_became_active":"2023-03-03T05:45:35.863234+0000","last_became_peered":"2023-03-03T05:45:35.863234+0000","last_unstale":"2023-03-03T05:45:36.923387+0000","last_undegraded":"2023-03-03T05:45:36.923387+0000","last_fullsized":"2023-03-03T05:45:36.923387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922937+0000","last_change":"2023-03-03T05:45:35.851019+0000","last_active":"2023-03-03T05:45:36.922937+0000","last_peered":"2023-03-03T05:45:36.922937+0000","last_clean":"2023-03-03T05:45:36.922937+0000","last_became_active":"2023-03-03T05:45:35.850439+0000","last_became_peered":"2023-03-03T05:45:35.850439+0000","last_unstale":"2023-03-03T05:45:36.922937+0000","last_undegraded":"2023-03-03T05:45:36.922937+0000","last_fullsized":"2023-03-03T05:45:36.922937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:43.693100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962628+0000","last_change":"2023-03-03T05:45:35.860508+0000","last_active":"2023-03-03T05:45:35.962628+0000","last_peered":"2023-03-03T05:45:35.962628+0000","last_clean":"2023-03-03T05:45:35.962628+0000","last_became_active":"2023-03-03T05:45:35.860137+0000","last_became_peered":"2023-03-03T05:45:35.860137+0000","last_unstale":"2023-03-03T05:45:35.962628+0000","last_undegraded":"2023-03-03T05:45:35.962628+0000","last_fullsized":"2023-03-03T05:45:35.962628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923487+0000","last_change":"2023-03-03T05:45:35.864824+0000","last_active":"2023-03-03T05:45:36.923487+0000","last_peered":"2023-03-03T05:45:36.923487+0000","last_clean":"2023-03-03T05:45:36.923487+0000","last_became_active":"2023-03-03T05:45:35.864466+0000","last_became_peered":"2023-03-03T05:45:35.864466+0000","last_unstale":"2023-03-03T05:45:36.923487+0000","last_undegraded":"2023-03-03T05:45:36.923487+0000","last_fullsized":"2023-03-03T05:45:36.923487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922781+0000","last_change":"2023-03-03T05:45:35.859499+0000","last_active":"2023-03-03T05:45:36.922781+0000","last_peered":"2023-03-03T05:45:36.922781+0000","last_clean":"2023-03-03T05:45:36.922781+0000","last_became_active":"2023-03-03T05:45:35.859200+0000","last_became_peered":"2023-03-03T05:45:35.859200+0000","last_unstale":"2023-03-03T05:45:36.922781+0000","last_undegraded":"2023-03-03T05:45:36.922781+0000","last_fullsized":"2023-03-03T05:45:36.922781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2589,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":6149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":90566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":88905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8068,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363772,"statfs":{"total":96636764160,"available":96628502528,"internally_reserved":0,"allocated":258048,"data_stored":90670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9088,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362752,"statfs":{"total":96636764160,"available":96627458048,"internally_reserved":0,"allocated":843776,"data_stored":679785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9140,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362700,"statfs":{"total":96636764160,"available":96627404800,"internally_reserved":0,"allocated":831488,"data_stored":678400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7988,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363852,"statfs":{"total":96636764160,"available":96628584448,"internally_reserved":0,"allocated":241664,"data_stored":88849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T05:46:04.569 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-03-03T05:46:04.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T05:46:04.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.846+0000 7f2b23bff700 1 Processor -- start 2023-03-03T05:46:04.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.847+0000 7f2b23bff700 1 -- start start 2023-03-03T05:46:04.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b23bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c0ced80 0x7f2b1c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b23bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2b1c1597f0 0x7f2b1c0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b23bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b1c0cf9a0 0x7f2b1c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b23bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c0ca750 con 0x7f2b1c1597f0 2023-03-03T05:46:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b23bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c15a230 con 0x7f2b1c0cf9a0 2023-03-03T05:46:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b23bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0cf690 con 0x7f2b1c0ced80 2023-03-03T05:46:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b2119a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2b1c1597f0 0x7f2b1c0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b2199b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c0ced80 0x7f2b1c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b2119a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2b1c1597f0 0x7f2b1c0cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42870/0 (socket says 172.21.15.102:42870) 2023-03-03T05:46:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b2119a700 1 -- 172.21.15.102:0/3991717230 learned_addr learned my addr 172.21.15.102:0/3991717230 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:04.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b2199b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c0ced80 0x7f2b1c0cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37814/0 (socket says 172.21.15.102:37814) 2023-03-03T05:46:04.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.848+0000 7f2b2219c700 1 --2- 172.21.15.102:0/3991717230 >> v2:172.21.15.172:3300/0 conn(0x7f2b1c0cf9a0 0x7f2b1c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b2119a700 1 -- 172.21.15.102:0/3991717230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c0ced80 msgr2=0x7f2b1c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:04.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b2119a700 1 --2- 172.21.15.102:0/3991717230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c0ced80 0x7f2b1c0cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b2119a700 1 -- 172.21.15.102:0/3991717230 >> v2:172.21.15.172:3300/0 conn(0x7f2b1c0cf9a0 msgr2=0x7f2b1c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:04.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b2119a700 1 --2- 172.21.15.102:0/3991717230 >> v2:172.21.15.172:3300/0 conn(0x7f2b1c0cf9a0 0x7f2b1c15c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b2119a700 1 -- 172.21.15.102:0/3991717230 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0cb9f0 con 0x7f2b1c1597f0 2023-03-03T05:46:04.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b2119a700 1 --2- 172.21.15.102:0/3991717230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c1597f0 0x7f2b1c0cbe80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c1569f0 tx=0x7f2b0c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea875ff5118fa0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:04.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b20999700 1 -- 172.21.15.102:0/3991717230 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c011830 con 0x7f2b1c1597f0 2023-03-03T05:46:04.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b20999700 1 -- 172.21.15.102:0/3991717230 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c011990 con 0x7f2b1c1597f0 2023-03-03T05:46:04.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.849+0000 7f2b20999700 1 -- 172.21.15.102:0/3991717230 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c011d20 con 0x7f2b1c1597f0 2023-03-03T05:46:04.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 -- 172.21.15.102:0/3991717230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c1597f0 msgr2=0x7f2b1c0cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 --2- 172.21.15.102:0/3991717230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c1597f0 0x7f2b1c0cbe80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c1569f0 tx=0x7f2b0c009c00 comp rx=0 tx=0).stop 2023-03-03T05:46:04.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 -- 172.21.15.102:0/3991717230 shutdown_connections 2023-03-03T05:46:04.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 --2- 172.21.15.102:0/3991717230 >> v2:172.21.15.172:3300/0 conn(0x7f2b1c0cf9a0 0x7f2b1c15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 --2- 172.21.15.102:0/3991717230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c0ced80 0x7f2b1c0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 --2- 172.21.15.102:0/3991717230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c1597f0 0x7f2b1c0cbe80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 -- 172.21.15.102:0/3991717230 >> 172.21.15.102:0/3991717230 conn(0x7f2b1c065fb0 msgr2=0x7f2b1c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:04.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 -- 172.21.15.102:0/3991717230 shutdown_connections 2023-03-03T05:46:04.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.850+0000 7f2b23bff700 1 -- 172.21.15.102:0/3991717230 wait complete. 2023-03-03T05:46:04.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.852+0000 7f2b23bff700 1 Processor -- start 2023-03-03T05:46:04.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.852+0000 7f2b23bff700 1 -- start start 2023-03-03T05:46:04.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.853+0000 7f2b23bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c0ced80 0x7f2b1c0b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.853+0000 7f2b23bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b1c0cf9a0 0x7f2b1c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.853+0000 7f2b23bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c1597f0 0x7f2b1c156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b23bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0c61d0 con 0x7f2b1c0ced80 2023-03-03T05:46:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b23bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c14fc90 con 0x7f2b1c0cf9a0 2023-03-03T05:46:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b23bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0b49f0 con 0x7f2b1c1597f0 2023-03-03T05:46:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c0ced80 0x7f2b1c0b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c0ced80 0x7f2b1c0b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42886/0 (socket says 172.21.15.102:42886) 2023-03-03T05:46:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 -- 172.21.15.102:0/1327900142 learned_addr learned my addr 172.21.15.102:0/1327900142 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2219c700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c1597f0 0x7f2b1c156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2119a700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b1c0cf9a0 0x7f2b1c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 -- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c1597f0 msgr2=0x7f2b1c156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c1597f0 0x7f2b1c156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 -- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b1c0cf9a0 msgr2=0x7f2b1c1578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b1c0cf9a0 0x7f2b1c1578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.854+0000 7f2b2199b700 1 -- 172.21.15.102:0/1327900142 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c009550 con 0x7f2b1c0ced80 2023-03-03T05:46:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.855+0000 7f2b2199b700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c0ced80 0x7f2b1c0b44b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800e8a0 tx=0x7f2b1800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.855+0000 7f2b127fc700 1 -- 172.21.15.102:0/1327900142 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18013680 con 0x7f2b1c0ced80 2023-03-03T05:46:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.855+0000 7f2b127fc700 1 -- 172.21.15.102:0/1327900142 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b180137e0 con 0x7f2b1c0ced80 2023-03-03T05:46:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.855+0000 7f2b127fc700 1 -- 172.21.15.102:0/1327900142 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18013b90 con 0x7f2b1c0ced80 2023-03-03T05:46:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.855+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c0cade0 con 0x7f2b1c0ced80 2023-03-03T05:46:04.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.855+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c159c80 con 0x7f2b1c0ced80 2023-03-03T05:46:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.856+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c0cb0c0 con 0x7f2b1c0ced80 2023-03-03T05:46:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.857+0000 7f2b127fc700 1 -- 172.21.15.102:0/1327900142 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2b18019030 con 0x7f2b1c0ced80 2023-03-03T05:46:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.859+0000 7f2b127fc700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b000682f0 0x7f2b0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.859+0000 7f2b127fc700 1 -- 172.21.15.102:0/1327900142 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2b18085a40 con 0x7f2b1c0ced80 2023-03-03T05:46:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.859+0000 7f2b2119a700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b000682f0 0x7f2b0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.860+0000 7f2b2119a700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b000682f0 0x7f2b0006a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c014040 tx=0x7f2b0c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:04.863+0000 7f2b127fc700 1 -- 172.21.15.102:0/1327900142 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2b18052150 con 0x7f2b1c0ced80 2023-03-03T05:46:05.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.053+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 --> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b1c14fdd0 con 0x7f2b000682f0 2023-03-03T05:46:05.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.062+0000 7f2b127fc700 1 -- 172.21.15.102:0/1327900142 <== mgr.4105 v2:172.21.15.172:6832/114058 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286578 (secure 0 0 0) 0x7f2b0c011810 con 0x7f2b000682f0 2023-03-03T05:46:05.067 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b000682f0 msgr2=0x7f2b0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b000682f0 0x7f2b0006a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c014040 tx=0x7f2b0c008030 comp rx=0 tx=0).stop 2023-03-03T05:46:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c0ced80 msgr2=0x7f2b1c0b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c0ced80 0x7f2b1c0b44b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800e8a0 tx=0x7f2b1800ebb0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 shutdown_connections 2023-03-03T05:46:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b000682f0 0x7f2b0006a7a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b1c1597f0 0x7f2b1c156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b1c0ced80 0x7f2b1c0b44b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 --2- 172.21.15.102:0/1327900142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b1c0cf9a0 0x7f2b1c1578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 >> 172.21.15.102:0/1327900142 conn(0x7f2b1c065fb0 msgr2=0x7f2b1c158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 shutdown_connections 2023-03-03T05:46:05.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.067+0000 7f2b23bff700 1 -- 172.21.15.102:0/1327900142 wait complete. 2023-03-03T05:46:05.071 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2023-03-03T05:46:05.096 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-03-03T05:46:03.950051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592957,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66244,"kb_used_data":3076,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":754908476,"statfs":{"total":773094113280,"available":773026279424,"internally_reserved":0,"allocated":3149824,"data_stored":1894693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":259,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003582"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922943+0000","last_change":"2023-03-03T05:45:35.854816+0000","last_active":"2023-03-03T05:45:36.922943+0000","last_peered":"2023-03-03T05:45:36.922943+0000","last_clean":"2023-03-03T05:45:36.922943+0000","last_became_active":"2023-03-03T05:45:35.854447+0000","last_became_peered":"2023-03-03T05:45:35.854447+0000","last_unstale":"2023-03-03T05:45:36.922943+0000","last_undegraded":"2023-03-03T05:45:36.922943+0000","last_fullsized":"2023-03-03T05:45:36.922943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850859+0000","last_change":"2023-03-03T05:45:35.862354+0000","last_active":"2023-03-03T05:45:36.850859+0000","last_peered":"2023-03-03T05:45:36.850859+0000","last_clean":"2023-03-03T05:45:36.850859+0000","last_became_active":"2023-03-03T05:45:35.862016+0000","last_became_peered":"2023-03-03T05:45:35.862016+0000","last_unstale":"2023-03-03T05:45:36.850859+0000","last_undegraded":"2023-03-03T05:45:36.850859+0000","last_fullsized":"2023-03-03T05:45:36.850859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962731+0000","last_change":"2023-03-03T05:45:35.864429+0000","last_active":"2023-03-03T05:45:35.962731+0000","last_peered":"2023-03-03T05:45:35.962731+0000","last_clean":"2023-03-03T05:45:35.962731+0000","last_became_active":"2023-03-03T05:45:35.864091+0000","last_became_peered":"2023-03-03T05:45:35.864091+0000","last_unstale":"2023-03-03T05:45:35.962731+0000","last_undegraded":"2023-03-03T05:45:35.962731+0000","last_fullsized":"2023-03-03T05:45:35.962731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962683+0000","last_change":"2023-03-03T05:45:35.861374+0000","last_active":"2023-03-03T05:45:35.962683+0000","last_peered":"2023-03-03T05:45:35.962683+0000","last_clean":"2023-03-03T05:45:35.962683+0000","last_became_active":"2023-03-03T05:45:35.860993+0000","last_became_peered":"2023-03-03T05:45:35.860993+0000","last_unstale":"2023-03-03T05:45:35.962683+0000","last_undegraded":"2023-03-03T05:45:35.962683+0000","last_fullsized":"2023-03-03T05:45:35.962683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853383+0000","last_change":"2023-03-03T05:45:35.857703+0000","last_active":"2023-03-03T05:45:36.853383+0000","last_peered":"2023-03-03T05:45:36.853383+0000","last_clean":"2023-03-03T05:45:36.853383+0000","last_became_active":"2023-03-03T05:45:35.857421+0000","last_became_peered":"2023-03-03T05:45:35.857421+0000","last_unstale":"2023-03-03T05:45:36.853383+0000","last_undegraded":"2023-03-03T05:45:36.853383+0000","last_fullsized":"2023-03-03T05:45:36.853383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851248+0000","last_change":"2023-03-03T05:45:35.859892+0000","last_active":"2023-03-03T05:45:36.851248+0000","last_peered":"2023-03-03T05:45:36.851248+0000","last_clean":"2023-03-03T05:45:36.851248+0000","last_became_active":"2023-03-03T05:45:35.859617+0000","last_became_peered":"2023-03-03T05:45:35.859617+0000","last_unstale":"2023-03-03T05:45:36.851248+0000","last_undegraded":"2023-03-03T05:45:36.851248+0000","last_fullsized":"2023-03-03T05:45:36.851248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:23:53.405094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850434+0000","last_change":"2023-03-03T05:45:35.862939+0000","last_active":"2023-03-03T05:45:36.850434+0000","last_peered":"2023-03-03T05:45:36.850434+0000","last_clean":"2023-03-03T05:45:36.850434+0000","last_became_active":"2023-03-03T05:45:35.862681+0000","last_became_peered":"2023-03-03T05:45:35.862681+0000","last_unstale":"2023-03-03T05:45:36.850434+0000","last_undegraded":"2023-03-03T05:45:36.850434+0000","last_fullsized":"2023-03-03T05:45:36.850434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:00:58.535750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923559+0000","last_change":"2023-03-03T05:45:35.866302+0000","last_active":"2023-03-03T05:45:36.923559+0000","last_peered":"2023-03-03T05:45:36.923559+0000","last_clean":"2023-03-03T05:45:36.923559+0000","last_became_active":"2023-03-03T05:45:35.865627+0000","last_became_peered":"2023-03-03T05:45:35.865627+0000","last_unstale":"2023-03-03T05:45:36.923559+0000","last_undegraded":"2023-03-03T05:45:36.923559+0000","last_fullsized":"2023-03-03T05:45:36.923559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853588+0000","last_change":"2023-03-03T05:45:35.856762+0000","last_active":"2023-03-03T05:45:36.853588+0000","last_peered":"2023-03-03T05:45:36.853588+0000","last_clean":"2023-03-03T05:45:36.853588+0000","last_became_active":"2023-03-03T05:45:35.856458+0000","last_became_peered":"2023-03-03T05:45:35.856458+0000","last_unstale":"2023-03-03T05:45:36.853588+0000","last_undegraded":"2023-03-03T05:45:36.853588+0000","last_fullsized":"2023-03-03T05:45:36.853588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923093+0000","last_change":"2023-03-03T05:45:35.865258+0000","last_active":"2023-03-03T05:45:36.923093+0000","last_peered":"2023-03-03T05:45:36.923093+0000","last_clean":"2023-03-03T05:45:36.923093+0000","last_became_active":"2023-03-03T05:45:35.864889+0000","last_became_peered":"2023-03-03T05:45:35.864889+0000","last_unstale":"2023-03-03T05:45:36.923093+0000","last_undegraded":"2023-03-03T05:45:36.923093+0000","last_fullsized":"2023-03-03T05:45:36.923093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923368+0000","last_change":"2023-03-03T05:45:35.868970+0000","last_active":"2023-03-03T05:45:36.923368+0000","last_peered":"2023-03-03T05:45:36.923368+0000","last_clean":"2023-03-03T05:45:36.923368+0000","last_became_active":"2023-03-03T05:45:35.868555+0000","last_became_peered":"2023-03-03T05:45:35.868555+0000","last_unstale":"2023-03-03T05:45:36.923368+0000","last_undegraded":"2023-03-03T05:45:36.923368+0000","last_fullsized":"2023-03-03T05:45:36.923368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922618+0000","last_change":"2023-03-03T05:45:35.855291+0000","last_active":"2023-03-03T05:45:36.922618+0000","last_peered":"2023-03-03T05:45:36.922618+0000","last_clean":"2023-03-03T05:45:36.922618+0000","last_became_active":"2023-03-03T05:45:35.854925+0000","last_became_peered":"2023-03-03T05:45:35.854925+0000","last_unstale":"2023-03-03T05:45:36.922618+0000","last_undegraded":"2023-03-03T05:45:36.922618+0000","last_fullsized":"2023-03-03T05:45:36.922618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.924177+0000","last_change":"2023-03-03T05:45:35.853479+0000","last_active":"2023-03-03T05:45:36.924177+0000","last_peered":"2023-03-03T05:45:36.924177+0000","last_clean":"2023-03-03T05:45:36.924177+0000","last_became_active":"2023-03-03T05:45:35.853193+0000","last_became_peered":"2023-03-03T05:45:35.853193+0000","last_unstale":"2023-03-03T05:45:36.924177+0000","last_undegraded":"2023-03-03T05:45:36.924177+0000","last_fullsized":"2023-03-03T05:45:36.924177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923036+0000","last_change":"2023-03-03T05:45:35.865976+0000","last_active":"2023-03-03T05:45:36.923036+0000","last_peered":"2023-03-03T05:45:36.923036+0000","last_clean":"2023-03-03T05:45:36.923036+0000","last_became_active":"2023-03-03T05:45:35.865561+0000","last_became_peered":"2023-03-03T05:45:35.865561+0000","last_unstale":"2023-03-03T05:45:36.923036+0000","last_undegraded":"2023-03-03T05:45:36.923036+0000","last_fullsized":"2023-03-03T05:45:36.923036+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962602+0000","last_change":"2023-03-03T05:45:35.862593+0000","last_active":"2023-03-03T05:45:35.962602+0000","last_peered":"2023-03-03T05:45:35.962602+0000","last_clean":"2023-03-03T05:45:35.962602+0000","last_became_active":"2023-03-03T05:45:35.862342+0000","last_became_peered":"2023-03-03T05:45:35.862342+0000","last_unstale":"2023-03-03T05:45:35.962602+0000","last_undegraded":"2023-03-03T05:45:35.962602+0000","last_fullsized":"2023-03-03T05:45:35.962602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850467+0000","last_change":"2023-03-03T05:45:35.857623+0000","last_active":"2023-03-03T05:45:36.850467+0000","last_peered":"2023-03-03T05:45:36.850467+0000","last_clean":"2023-03-03T05:45:36.850467+0000","last_became_active":"2023-03-03T05:45:35.857357+0000","last_became_peered":"2023-03-03T05:45:35.857357+0000","last_unstale":"2023-03-03T05:45:36.850467+0000","last_undegraded":"2023-03-03T05:45:36.850467+0000","last_fullsized":"2023-03-03T05:45:36.850467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:00.850961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923102+0000","last_change":"2023-03-03T05:45:35.859115+0000","last_active":"2023-03-03T05:45:36.923102+0000","last_peered":"2023-03-03T05:45:36.923102+0000","last_clean":"2023-03-03T05:45:36.923102+0000","last_became_active":"2023-03-03T05:45:35.858794+0000","last_became_peered":"2023-03-03T05:45:35.858794+0000","last_unstale":"2023-03-03T05:45:36.923102+0000","last_undegraded":"2023-03-03T05:45:36.923102+0000","last_fullsized":"2023-03-03T05:45:36.923102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962764+0000","last_change":"2023-03-03T05:45:35.861090+0000","last_active":"2023-03-03T05:45:35.962764+0000","last_peered":"2023-03-03T05:45:35.962764+0000","last_clean":"2023-03-03T05:45:35.962764+0000","last_became_active":"2023-03-03T05:45:35.860814+0000","last_became_peered":"2023-03-03T05:45:35.860814+0000","last_unstale":"2023-03-03T05:45:35.962764+0000","last_undegraded":"2023-03-03T05:45:35.962764+0000","last_fullsized":"2023-03-03T05:45:35.962764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923139+0000","last_change":"2023-03-03T05:45:35.855611+0000","last_active":"2023-03-03T05:45:36.923139+0000","last_peered":"2023-03-03T05:45:36.923139+0000","last_clean":"2023-03-03T05:45:36.923139+0000","last_became_active":"2023-03-03T05:45:35.855333+0000","last_became_peered":"2023-03-03T05:45:35.855333+0000","last_unstale":"2023-03-03T05:45:36.923139+0000","last_undegraded":"2023-03-03T05:45:36.923139+0000","last_fullsized":"2023-03-03T05:45:36.923139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:09.376421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923216+0000","last_change":"2023-03-03T05:45:35.865135+0000","last_active":"2023-03-03T05:45:36.923216+0000","last_peered":"2023-03-03T05:45:36.923216+0000","last_clean":"2023-03-03T05:45:36.923216+0000","last_became_active":"2023-03-03T05:45:35.864823+0000","last_became_peered":"2023-03-03T05:45:35.864823+0000","last_unstale":"2023-03-03T05:45:36.923216+0000","last_undegraded":"2023-03-03T05:45:36.923216+0000","last_fullsized":"2023-03-03T05:45:36.923216+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923121+0000","last_change":"2023-03-03T05:45:35.864573+0000","last_active":"2023-03-03T05:45:36.923121+0000","last_peered":"2023-03-03T05:45:36.923121+0000","last_clean":"2023-03-03T05:45:36.923121+0000","last_became_active":"2023-03-03T05:45:35.864297+0000","last_became_peered":"2023-03-03T05:45:35.864297+0000","last_unstale":"2023-03-03T05:45:36.923121+0000","last_undegraded":"2023-03-03T05:45:36.923121+0000","last_fullsized":"2023-03-03T05:45:36.923121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922982+0000","last_change":"2023-03-03T05:45:35.862720+0000","last_active":"2023-03-03T05:45:36.922982+0000","last_peered":"2023-03-03T05:45:36.922982+0000","last_clean":"2023-03-03T05:45:36.922982+0000","last_became_active":"2023-03-03T05:45:35.861282+0000","last_became_peered":"2023-03-03T05:45:35.861282+0000","last_unstale":"2023-03-03T05:45:36.922982+0000","last_undegraded":"2023-03-03T05:45:36.922982+0000","last_fullsized":"2023-03-03T05:45:36.922982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853407+0000","last_change":"2023-03-03T05:45:35.859828+0000","last_active":"2023-03-03T05:45:36.853407+0000","last_peered":"2023-03-03T05:45:36.853407+0000","last_clean":"2023-03-03T05:45:36.853407+0000","last_became_active":"2023-03-03T05:45:35.859568+0000","last_became_peered":"2023-03-03T05:45:35.859568+0000","last_unstale":"2023-03-03T05:45:36.853407+0000","last_undegraded":"2023-03-03T05:45:36.853407+0000","last_fullsized":"2023-03-03T05:45:36.853407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923461+0000","last_change":"2023-03-03T05:45:35.864435+0000","last_active":"2023-03-03T05:45:36.923461+0000","last_peered":"2023-03-03T05:45:36.923461+0000","last_clean":"2023-03-03T05:45:36.923461+0000","last_became_active":"2023-03-03T05:45:35.864129+0000","last_became_peered":"2023-03-03T05:45:35.864129+0000","last_unstale":"2023-03-03T05:45:36.923461+0000","last_undegraded":"2023-03-03T05:45:36.923461+0000","last_fullsized":"2023-03-03T05:45:36.923461+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853682+0000","last_change":"2023-03-03T05:45:35.855909+0000","last_active":"2023-03-03T05:45:36.853682+0000","last_peered":"2023-03-03T05:45:36.853682+0000","last_clean":"2023-03-03T05:45:36.853682+0000","last_became_active":"2023-03-03T05:45:35.855388+0000","last_became_peered":"2023-03-03T05:45:35.855388+0000","last_unstale":"2023-03-03T05:45:36.853682+0000","last_undegraded":"2023-03-03T05:45:36.853682+0000","last_fullsized":"2023-03-03T05:45:36.853682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923513+0000","last_change":"2023-03-03T05:45:35.863167+0000","last_active":"2023-03-03T05:45:36.923513+0000","last_peered":"2023-03-03T05:45:36.923513+0000","last_clean":"2023-03-03T05:45:36.923513+0000","last_became_active":"2023-03-03T05:45:35.862519+0000","last_became_peered":"2023-03-03T05:45:35.862519+0000","last_unstale":"2023-03-03T05:45:36.923513+0000","last_undegraded":"2023-03-03T05:45:36.923513+0000","last_fullsized":"2023-03-03T05:45:36.923513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851022+0000","last_change":"2023-03-03T05:45:35.870150+0000","last_active":"2023-03-03T05:45:36.851022+0000","last_peered":"2023-03-03T05:45:36.851022+0000","last_clean":"2023-03-03T05:45:36.851022+0000","last_became_active":"2023-03-03T05:45:35.868786+0000","last_became_peered":"2023-03-03T05:45:35.868786+0000","last_unstale":"2023-03-03T05:45:36.851022+0000","last_undegraded":"2023-03-03T05:45:36.851022+0000","last_fullsized":"2023-03-03T05:45:36.851022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:52:23.931405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.310449+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.310197+0000","last_became_peered":"2023-03-03T05:45:35.310197+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923377+0000","last_change":"2023-03-03T05:45:35.861444+0000","last_active":"2023-03-03T05:45:36.923377+0000","last_peered":"2023-03-03T05:45:36.923377+0000","last_clean":"2023-03-03T05:45:36.923377+0000","last_became_active":"2023-03-03T05:45:35.861129+0000","last_became_peered":"2023-03-03T05:45:35.861129+0000","last_unstale":"2023-03-03T05:45:36.923377+0000","last_undegraded":"2023-03-03T05:45:36.923377+0000","last_fullsized":"2023-03-03T05:45:36.923377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843345+0000","last_change":"2023-03-03T05:45:35.308492+0000","last_active":"2023-03-03T05:45:35.843345+0000","last_peered":"2023-03-03T05:45:35.843345+0000","last_clean":"2023-03-03T05:45:35.843345+0000","last_became_active":"2023-03-03T05:45:35.308276+0000","last_became_peered":"2023-03-03T05:45:35.308276+0000","last_unstale":"2023-03-03T05:45:35.843345+0000","last_undegraded":"2023-03-03T05:45:35.843345+0000","last_fullsized":"2023-03-03T05:45:35.843345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:32.932001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853620+0000","last_change":"2023-03-03T05:45:35.866408+0000","last_active":"2023-03-03T05:45:36.853620+0000","last_peered":"2023-03-03T05:45:36.853620+0000","last_clean":"2023-03-03T05:45:36.853620+0000","last_became_active":"2023-03-03T05:45:35.866080+0000","last_became_peered":"2023-03-03T05:45:35.866080+0000","last_unstale":"2023-03-03T05:45:36.853620+0000","last_undegraded":"2023-03-03T05:45:36.853620+0000","last_fullsized":"2023-03-03T05:45:36.853620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:23:22.678556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848138+0000","last_change":"2023-03-03T05:45:35.293990+0000","last_active":"2023-03-03T05:45:35.848138+0000","last_peered":"2023-03-03T05:45:35.848138+0000","last_clean":"2023-03-03T05:45:35.848138+0000","last_became_active":"2023-03-03T05:45:35.293655+0000","last_became_peered":"2023-03-03T05:45:35.293655+0000","last_unstale":"2023-03-03T05:45:35.848138+0000","last_undegraded":"2023-03-03T05:45:35.848138+0000","last_fullsized":"2023-03-03T05:45:35.848138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850700+0000","last_change":"2023-03-03T05:45:35.852599+0000","last_active":"2023-03-03T05:45:36.850700+0000","last_peered":"2023-03-03T05:45:36.850700+0000","last_clean":"2023-03-03T05:45:36.850700+0000","last_became_active":"2023-03-03T05:45:35.852292+0000","last_became_peered":"2023-03-03T05:45:35.852292+0000","last_unstale":"2023-03-03T05:45:36.850700+0000","last_undegraded":"2023-03-03T05:45:36.850700+0000","last_fullsized":"2023-03-03T05:45:36.850700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:55:06.104140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848641+0000","last_change":"2023-03-03T05:45:35.297718+0000","last_active":"2023-03-03T05:45:35.848641+0000","last_peered":"2023-03-03T05:45:35.848641+0000","last_clean":"2023-03-03T05:45:35.848641+0000","last_became_active":"2023-03-03T05:45:35.297435+0000","last_became_peered":"2023-03-03T05:45:35.297435+0000","last_unstale":"2023-03-03T05:45:35.848641+0000","last_undegraded":"2023-03-03T05:45:35.848641+0000","last_fullsized":"2023-03-03T05:45:35.848641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:17:39.778447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923577+0000","last_change":"2023-03-03T05:45:35.871116+0000","last_active":"2023-03-03T05:45:36.923577+0000","last_peered":"2023-03-03T05:45:36.923577+0000","last_clean":"2023-03-03T05:45:36.923577+0000","last_became_active":"2023-03-03T05:45:35.870843+0000","last_became_peered":"2023-03-03T05:45:35.870843+0000","last_unstale":"2023-03-03T05:45:36.923577+0000","last_undegraded":"2023-03-03T05:45:36.923577+0000","last_fullsized":"2023-03-03T05:45:36.923577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851170+0000","last_change":"2023-03-03T05:45:35.304400+0000","last_active":"2023-03-03T05:45:36.851170+0000","last_peered":"2023-03-03T05:45:36.851170+0000","last_clean":"2023-03-03T05:45:36.851170+0000","last_became_active":"2023-03-03T05:45:35.304093+0000","last_became_peered":"2023-03-03T05:45:35.304093+0000","last_unstale":"2023-03-03T05:45:36.851170+0000","last_undegraded":"2023-03-03T05:45:36.851170+0000","last_fullsized":"2023-03-03T05:45:36.851170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853210+0000","last_change":"2023-03-03T05:45:35.862492+0000","last_active":"2023-03-03T05:45:36.853210+0000","last_peered":"2023-03-03T05:45:36.853210+0000","last_clean":"2023-03-03T05:45:36.853210+0000","last_became_active":"2023-03-03T05:45:35.862173+0000","last_became_peered":"2023-03-03T05:45:35.862173+0000","last_unstale":"2023-03-03T05:45:36.853210+0000","last_undegraded":"2023-03-03T05:45:36.853210+0000","last_fullsized":"2023-03-03T05:45:36.853210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:10:30.692997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.853604+0000","last_change":"2023-03-03T05:45:35.302282+0000","last_active":"2023-03-03T05:45:40.853604+0000","last_peered":"2023-03-03T05:45:40.853604+0000","last_clean":"2023-03-03T05:45:40.853604+0000","last_became_active":"2023-03-03T05:45:35.302002+0000","last_became_peered":"2023-03-03T05:45:35.302002+0000","last_unstale":"2023-03-03T05:45:40.853604+0000","last_undegraded":"2023-03-03T05:45:40.853604+0000","last_fullsized":"2023-03-03T05:45:40.853604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853414+0000","last_change":"2023-03-03T05:45:35.855928+0000","last_active":"2023-03-03T05:45:36.853414+0000","last_peered":"2023-03-03T05:45:36.853414+0000","last_clean":"2023-03-03T05:45:36.853414+0000","last_became_active":"2023-03-03T05:45:35.855612+0000","last_became_peered":"2023-03-03T05:45:35.855612+0000","last_unstale":"2023-03-03T05:45:36.853414+0000","last_undegraded":"2023-03-03T05:45:36.853414+0000","last_fullsized":"2023-03-03T05:45:36.853414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:17:27.360294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:57.313631+0000","last_change":"2023-03-03T05:45:35.304856+0000","last_active":"2023-03-03T05:45:57.313631+0000","last_peered":"2023-03-03T05:45:57.313631+0000","last_clean":"2023-03-03T05:45:57.313631+0000","last_became_active":"2023-03-03T05:45:35.304439+0000","last_became_peered":"2023-03-03T05:45:35.304439+0000","last_unstale":"2023-03-03T05:45:57.313631+0000","last_undegraded":"2023-03-03T05:45:57.313631+0000","last_fullsized":"2023-03-03T05:45:57.313631+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:30.692527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1807,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850547+0000","last_change":"2023-03-03T05:45:35.856887+0000","last_active":"2023-03-03T05:45:36.850547+0000","last_peered":"2023-03-03T05:45:36.850547+0000","last_clean":"2023-03-03T05:45:36.850547+0000","last_became_active":"2023-03-03T05:45:35.856498+0000","last_became_peered":"2023-03-03T05:45:35.856498+0000","last_unstale":"2023-03-03T05:45:36.850547+0000","last_undegraded":"2023-03-03T05:45:36.850547+0000","last_fullsized":"2023-03-03T05:45:36.850547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858066+0000","last_change":"2023-03-03T05:45:35.297635+0000","last_active":"2023-03-03T05:45:40.858066+0000","last_peered":"2023-03-03T05:45:40.858066+0000","last_clean":"2023-03-03T05:45:40.858066+0000","last_became_active":"2023-03-03T05:45:35.297373+0000","last_became_peered":"2023-03-03T05:45:35.297373+0000","last_unstale":"2023-03-03T05:45:40.858066+0000","last_undegraded":"2023-03-03T05:45:40.858066+0000","last_fullsized":"2023-03-03T05:45:40.858066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:48.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922855+0000","last_change":"2023-03-03T05:45:35.852113+0000","last_active":"2023-03-03T05:45:36.922855+0000","last_peered":"2023-03-03T05:45:36.922855+0000","last_clean":"2023-03-03T05:45:36.922855+0000","last_became_active":"2023-03-03T05:45:35.851799+0000","last_became_peered":"2023-03-03T05:45:35.851799+0000","last_unstale":"2023-03-03T05:45:36.922855+0000","last_undegraded":"2023-03-03T05:45:36.922855+0000","last_fullsized":"2023-03-03T05:45:36.922855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.845615+0000","last_change":"2023-03-03T05:45:35.292083+0000","last_active":"2023-03-03T05:45:35.845615+0000","last_peered":"2023-03-03T05:45:35.845615+0000","last_clean":"2023-03-03T05:45:35.845615+0000","last_became_active":"2023-03-03T05:45:35.291766+0000","last_became_peered":"2023-03-03T05:45:35.291766+0000","last_unstale":"2023-03-03T05:45:35.845615+0000","last_undegraded":"2023-03-03T05:45:35.845615+0000","last_fullsized":"2023-03-03T05:45:35.845615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851348+0000","last_change":"2023-03-03T05:45:35.870363+0000","last_active":"2023-03-03T05:45:36.851348+0000","last_peered":"2023-03-03T05:45:36.851348+0000","last_clean":"2023-03-03T05:45:36.851348+0000","last_became_active":"2023-03-03T05:45:35.868801+0000","last_became_peered":"2023-03-03T05:45:35.868801+0000","last_unstale":"2023-03-03T05:45:36.851348+0000","last_undegraded":"2023-03-03T05:45:36.851348+0000","last_fullsized":"2023-03-03T05:45:36.851348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:04.045341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844327+0000","last_change":"2023-03-03T05:45:35.304592+0000","last_active":"2023-03-03T05:45:35.844327+0000","last_peered":"2023-03-03T05:45:35.844327+0000","last_clean":"2023-03-03T05:45:35.844327+0000","last_became_active":"2023-03-03T05:45:35.304318+0000","last_became_peered":"2023-03-03T05:45:35.304318+0000","last_unstale":"2023-03-03T05:45:35.844327+0000","last_undegraded":"2023-03-03T05:45:35.844327+0000","last_fullsized":"2023-03-03T05:45:35.844327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:26:08.368039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922933+0000","last_change":"2023-03-03T05:45:35.857951+0000","last_active":"2023-03-03T05:45:36.922933+0000","last_peered":"2023-03-03T05:45:36.922933+0000","last_clean":"2023-03-03T05:45:36.922933+0000","last_became_active":"2023-03-03T05:45:35.857581+0000","last_became_peered":"2023-03-03T05:45:35.857581+0000","last_unstale":"2023-03-03T05:45:36.922933+0000","last_undegraded":"2023-03-03T05:45:36.922933+0000","last_fullsized":"2023-03-03T05:45:36.922933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:36:46.177425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858611+0000","last_change":"2023-03-03T05:45:35.309150+0000","last_active":"2023-03-03T05:45:40.858611+0000","last_peered":"2023-03-03T05:45:40.858611+0000","last_clean":"2023-03-03T05:45:40.858611+0000","last_became_active":"2023-03-03T05:45:35.308840+0000","last_became_peered":"2023-03-03T05:45:35.308840+0000","last_unstale":"2023-03-03T05:45:40.858611+0000","last_undegraded":"2023-03-03T05:45:40.858611+0000","last_fullsized":"2023-03-03T05:45:40.858611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923050+0000","last_change":"2023-03-03T05:45:35.864919+0000","last_active":"2023-03-03T05:45:36.923050+0000","last_peered":"2023-03-03T05:45:36.923050+0000","last_clean":"2023-03-03T05:45:36.923050+0000","last_became_active":"2023-03-03T05:45:35.864595+0000","last_became_peered":"2023-03-03T05:45:35.864595+0000","last_unstale":"2023-03-03T05:45:36.923050+0000","last_undegraded":"2023-03-03T05:45:36.923050+0000","last_fullsized":"2023-03-03T05:45:36.923050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:10.769623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923125+0000","last_change":"2023-03-03T05:45:35.308976+0000","last_active":"2023-03-03T05:45:36.923125+0000","last_peered":"2023-03-03T05:45:36.923125+0000","last_clean":"2023-03-03T05:45:36.923125+0000","last_became_active":"2023-03-03T05:45:35.308707+0000","last_became_peered":"2023-03-03T05:45:35.308707+0000","last_unstale":"2023-03-03T05:45:36.923125+0000","last_undegraded":"2023-03-03T05:45:36.923125+0000","last_fullsized":"2023-03-03T05:45:36.923125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850640+0000","last_change":"2023-03-03T05:45:35.868861+0000","last_active":"2023-03-03T05:45:36.850640+0000","last_peered":"2023-03-03T05:45:36.850640+0000","last_clean":"2023-03-03T05:45:36.850640+0000","last_became_active":"2023-03-03T05:45:35.868438+0000","last_became_peered":"2023-03-03T05:45:35.868438+0000","last_unstale":"2023-03-03T05:45:36.850640+0000","last_undegraded":"2023-03-03T05:45:36.850640+0000","last_fullsized":"2023-03-03T05:45:36.850640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:59.810575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.854269+0000","last_change":"2023-03-03T05:45:35.303791+0000","last_active":"2023-03-03T05:45:40.854269+0000","last_peered":"2023-03-03T05:45:40.854269+0000","last_clean":"2023-03-03T05:45:40.854269+0000","last_became_active":"2023-03-03T05:45:35.303482+0000","last_became_peered":"2023-03-03T05:45:35.303482+0000","last_unstale":"2023-03-03T05:45:40.854269+0000","last_undegraded":"2023-03-03T05:45:40.854269+0000","last_fullsized":"2023-03-03T05:45:40.854269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:31.933113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923037+0000","last_change":"2023-03-03T05:45:35.864783+0000","last_active":"2023-03-03T05:45:36.923037+0000","last_peered":"2023-03-03T05:45:36.923037+0000","last_clean":"2023-03-03T05:45:36.923037+0000","last_became_active":"2023-03-03T05:45:35.864462+0000","last_became_peered":"2023-03-03T05:45:35.864462+0000","last_unstale":"2023-03-03T05:45:36.923037+0000","last_undegraded":"2023-03-03T05:45:36.923037+0000","last_fullsized":"2023-03-03T05:45:36.923037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851096+0000","last_change":"2023-03-03T05:45:35.310813+0000","last_active":"2023-03-03T05:45:36.851096+0000","last_peered":"2023-03-03T05:45:36.851096+0000","last_clean":"2023-03-03T05:45:36.851096+0000","last_became_active":"2023-03-03T05:45:35.310466+0000","last_became_peered":"2023-03-03T05:45:35.310466+0000","last_unstale":"2023-03-03T05:45:36.851096+0000","last_undegraded":"2023-03-03T05:45:36.851096+0000","last_fullsized":"2023-03-03T05:45:36.851096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:46.322139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850472+0000","last_change":"2023-03-03T05:45:35.856376+0000","last_active":"2023-03-03T05:45:36.850472+0000","last_peered":"2023-03-03T05:45:36.850472+0000","last_clean":"2023-03-03T05:45:36.850472+0000","last_became_active":"2023-03-03T05:45:35.856064+0000","last_became_peered":"2023-03-03T05:45:35.856064+0000","last_unstale":"2023-03-03T05:45:36.850472+0000","last_undegraded":"2023-03-03T05:45:36.850472+0000","last_fullsized":"2023-03-03T05:45:36.850472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962714+0000","last_change":"2023-03-03T05:45:35.295520+0000","last_active":"2023-03-03T05:45:35.962714+0000","last_peered":"2023-03-03T05:45:35.962714+0000","last_clean":"2023-03-03T05:45:35.962714+0000","last_became_active":"2023-03-03T05:45:35.295203+0000","last_became_peered":"2023-03-03T05:45:35.295203+0000","last_unstale":"2023-03-03T05:45:35.962714+0000","last_undegraded":"2023-03-03T05:45:35.962714+0000","last_fullsized":"2023-03-03T05:45:35.962714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923389+0000","last_change":"2023-03-03T05:45:35.852242+0000","last_active":"2023-03-03T05:45:36.923389+0000","last_peered":"2023-03-03T05:45:36.923389+0000","last_clean":"2023-03-03T05:45:36.923389+0000","last_became_active":"2023-03-03T05:45:35.851803+0000","last_became_peered":"2023-03-03T05:45:35.851803+0000","last_unstale":"2023-03-03T05:45:36.923389+0000","last_undegraded":"2023-03-03T05:45:36.923389+0000","last_fullsized":"2023-03-03T05:45:36.923389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:28:14.473334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.852939+0000","last_change":"2023-03-03T05:45:35.298542+0000","last_active":"2023-03-03T05:45:40.852939+0000","last_peered":"2023-03-03T05:45:40.852939+0000","last_clean":"2023-03-03T05:45:40.852939+0000","last_became_active":"2023-03-03T05:45:35.298219+0000","last_became_peered":"2023-03-03T05:45:35.298219+0000","last_unstale":"2023-03-03T05:45:40.852939+0000","last_undegraded":"2023-03-03T05:45:40.852939+0000","last_fullsized":"2023-03-03T05:45:40.852939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:17:17.694420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962699+0000","last_change":"2023-03-03T05:45:35.861231+0000","last_active":"2023-03-03T05:45:35.962699+0000","last_peered":"2023-03-03T05:45:35.962699+0000","last_clean":"2023-03-03T05:45:35.962699+0000","last_became_active":"2023-03-03T05:45:35.860675+0000","last_became_peered":"2023-03-03T05:45:35.860675+0000","last_unstale":"2023-03-03T05:45:35.962699+0000","last_undegraded":"2023-03-03T05:45:35.962699+0000","last_fullsized":"2023-03-03T05:45:35.962699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843762+0000","last_change":"2023-03-03T05:45:35.312143+0000","last_active":"2023-03-03T05:45:35.843762+0000","last_peered":"2023-03-03T05:45:35.843762+0000","last_clean":"2023-03-03T05:45:35.843762+0000","last_became_active":"2023-03-03T05:45:35.311848+0000","last_became_peered":"2023-03-03T05:45:35.311848+0000","last_unstale":"2023-03-03T05:45:35.843762+0000","last_undegraded":"2023-03-03T05:45:35.843762+0000","last_fullsized":"2023-03-03T05:45:35.843762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:22.294783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851303+0000","last_change":"2023-03-03T05:45:35.869075+0000","last_active":"2023-03-03T05:45:36.851303+0000","last_peered":"2023-03-03T05:45:36.851303+0000","last_clean":"2023-03-03T05:45:36.851303+0000","last_became_active":"2023-03-03T05:45:35.868413+0000","last_became_peered":"2023-03-03T05:45:35.868413+0000","last_unstale":"2023-03-03T05:45:36.851303+0000","last_undegraded":"2023-03-03T05:45:36.851303+0000","last_fullsized":"2023-03-03T05:45:36.851303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:34.743111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844226+0000","last_change":"2023-03-03T05:45:35.296611+0000","last_active":"2023-03-03T05:45:35.844226+0000","last_peered":"2023-03-03T05:45:35.844226+0000","last_clean":"2023-03-03T05:45:35.844226+0000","last_became_active":"2023-03-03T05:45:35.296294+0000","last_became_peered":"2023-03-03T05:45:35.296294+0000","last_unstale":"2023-03-03T05:45:35.844226+0000","last_undegraded":"2023-03-03T05:45:35.844226+0000","last_fullsized":"2023-03-03T05:45:35.844226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:16.655836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850964+0000","last_change":"2023-03-03T05:45:35.857451+0000","last_active":"2023-03-03T05:45:36.850964+0000","last_peered":"2023-03-03T05:45:36.850964+0000","last_clean":"2023-03-03T05:45:36.850964+0000","last_became_active":"2023-03-03T05:45:35.857087+0000","last_became_peered":"2023-03-03T05:45:35.857087+0000","last_unstale":"2023-03-03T05:45:36.850964+0000","last_undegraded":"2023-03-03T05:45:36.850964+0000","last_fullsized":"2023-03-03T05:45:36.850964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842797+0000","last_change":"2023-03-03T05:45:35.296114+0000","last_active":"2023-03-03T05:45:35.842797+0000","last_peered":"2023-03-03T05:45:35.842797+0000","last_clean":"2023-03-03T05:45:35.842797+0000","last_became_active":"2023-03-03T05:45:35.295855+0000","last_became_peered":"2023-03-03T05:45:35.295855+0000","last_unstale":"2023-03-03T05:45:35.842797+0000","last_undegraded":"2023-03-03T05:45:35.842797+0000","last_fullsized":"2023-03-03T05:45:35.842797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:55:25.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923246+0000","last_change":"2023-03-03T05:45:35.862249+0000","last_active":"2023-03-03T05:45:36.923246+0000","last_peered":"2023-03-03T05:45:36.923246+0000","last_clean":"2023-03-03T05:45:36.923246+0000","last_became_active":"2023-03-03T05:45:35.861773+0000","last_became_peered":"2023-03-03T05:45:35.861773+0000","last_unstale":"2023-03-03T05:45:36.923246+0000","last_undegraded":"2023-03-03T05:45:36.923246+0000","last_fullsized":"2023-03-03T05:45:36.923246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848485+0000","last_change":"2023-03-03T05:45:29.189033+0000","last_active":"2023-03-03T05:45:35.848485+0000","last_peered":"2023-03-03T05:45:35.848485+0000","last_clean":"2023-03-03T05:45:35.848485+0000","last_became_active":"2023-03-03T05:45:29.188719+0000","last_became_peered":"2023-03-03T05:45:29.188719+0000","last_unstale":"2023-03-03T05:45:35.848485+0000","last_undegraded":"2023-03-03T05:45:35.848485+0000","last_fullsized":"2023-03-03T05:45:35.848485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850356+0000","last_change":"2023-03-03T05:45:35.303756+0000","last_active":"2023-03-03T05:45:36.850356+0000","last_peered":"2023-03-03T05:45:36.850356+0000","last_clean":"2023-03-03T05:45:36.850356+0000","last_became_active":"2023-03-03T05:45:35.303448+0000","last_became_peered":"2023-03-03T05:45:35.303448+0000","last_unstale":"2023-03-03T05:45:36.850356+0000","last_undegraded":"2023-03-03T05:45:36.850356+0000","last_fullsized":"2023-03-03T05:45:36.850356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922899+0000","last_change":"2023-03-03T05:45:35.854166+0000","last_active":"2023-03-03T05:45:36.922899+0000","last_peered":"2023-03-03T05:45:36.922899+0000","last_clean":"2023-03-03T05:45:36.922899+0000","last_became_active":"2023-03-03T05:45:35.853873+0000","last_became_peered":"2023-03-03T05:45:35.853873+0000","last_unstale":"2023-03-03T05:45:36.922899+0000","last_undegraded":"2023-03-03T05:45:36.922899+0000","last_fullsized":"2023-03-03T05:45:36.922899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:09:53.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843699+0000","last_change":"2023-03-03T05:45:29.189075+0000","last_active":"2023-03-03T05:45:35.843699+0000","last_peered":"2023-03-03T05:45:35.843699+0000","last_clean":"2023-03-03T05:45:35.843699+0000","last_became_active":"2023-03-03T05:45:29.188778+0000","last_became_peered":"2023-03-03T05:45:29.188778+0000","last_unstale":"2023-03-03T05:45:35.843699+0000","last_undegraded":"2023-03-03T05:45:35.843699+0000","last_fullsized":"2023-03-03T05:45:35.843699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:54:35.536172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962645+0000","last_change":"2023-03-03T05:45:35.309781+0000","last_active":"2023-03-03T05:45:35.962645+0000","last_peered":"2023-03-03T05:45:35.962645+0000","last_clean":"2023-03-03T05:45:35.962645+0000","last_became_active":"2023-03-03T05:45:35.309497+0000","last_became_peered":"2023-03-03T05:45:35.309497+0000","last_unstale":"2023-03-03T05:45:35.962645+0000","last_undegraded":"2023-03-03T05:45:35.962645+0000","last_fullsized":"2023-03-03T05:45:35.962645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923571+0000","last_change":"2023-03-03T05:45:35.854721+0000","last_active":"2023-03-03T05:45:36.923571+0000","last_peered":"2023-03-03T05:45:36.923571+0000","last_clean":"2023-03-03T05:45:36.923571+0000","last_became_active":"2023-03-03T05:45:35.854423+0000","last_became_peered":"2023-03-03T05:45:35.854423+0000","last_unstale":"2023-03-03T05:45:36.923571+0000","last_undegraded":"2023-03-03T05:45:36.923571+0000","last_fullsized":"2023-03-03T05:45:36.923571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.852125+0000","last_change":"2023-03-03T05:45:29.057308+0000","last_active":"2023-03-03T05:45:35.852125+0000","last_peered":"2023-03-03T05:45:35.852125+0000","last_clean":"2023-03-03T05:45:35.852125+0000","last_became_active":"2023-03-03T05:45:29.056828+0000","last_became_peered":"2023-03-03T05:45:29.056828+0000","last_unstale":"2023-03-03T05:45:35.852125+0000","last_undegraded":"2023-03-03T05:45:35.852125+0000","last_fullsized":"2023-03-03T05:45:35.852125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:18:56.079893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.848095+0000","last_change":"2023-03-03T05:45:35.295842+0000","last_active":"2023-03-03T05:45:35.848095+0000","last_peered":"2023-03-03T05:45:35.848095+0000","last_clean":"2023-03-03T05:45:35.848095+0000","last_became_active":"2023-03-03T05:45:35.295545+0000","last_became_peered":"2023-03-03T05:45:35.295545+0000","last_unstale":"2023-03-03T05:45:35.848095+0000","last_undegraded":"2023-03-03T05:45:35.848095+0000","last_fullsized":"2023-03-03T05:45:35.848095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923636+0000","last_change":"2023-03-03T05:45:35.862437+0000","last_active":"2023-03-03T05:45:36.923636+0000","last_peered":"2023-03-03T05:45:36.923636+0000","last_clean":"2023-03-03T05:45:36.923636+0000","last_became_active":"2023-03-03T05:45:35.862152+0000","last_became_peered":"2023-03-03T05:45:35.862152+0000","last_unstale":"2023-03-03T05:45:36.923636+0000","last_undegraded":"2023-03-03T05:45:36.923636+0000","last_fullsized":"2023-03-03T05:45:36.923636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:19:45.589027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.842862+0000","last_change":"2023-03-03T05:45:29.057646+0000","last_active":"2023-03-03T05:45:35.842862+0000","last_peered":"2023-03-03T05:45:35.842862+0000","last_clean":"2023-03-03T05:45:35.842862+0000","last_became_active":"2023-03-03T05:45:29.057330+0000","last_became_peered":"2023-03-03T05:45:29.057330+0000","last_unstale":"2023-03-03T05:45:35.842862+0000","last_undegraded":"2023-03-03T05:45:35.842862+0000","last_fullsized":"2023-03-03T05:45:35.842862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:22:57.039574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.858102+0000","last_change":"2023-03-03T05:45:22.224435+0000","last_active":"2023-03-03T05:45:35.858102+0000","last_peered":"2023-03-03T05:45:35.858102+0000","last_clean":"2023-03-03T05:45:35.858102+0000","last_became_active":"2023-03-03T05:45:22.222958+0000","last_became_peered":"2023-03-03T05:45:22.222958+0000","last_unstale":"2023-03-03T05:45:35.858102+0000","last_undegraded":"2023-03-03T05:45:35.858102+0000","last_fullsized":"2023-03-03T05:45:35.858102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:20.925942+0000","last_clean_scrub_stamp":"2023-03-03T05:45:20.925942+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:10:20.426026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843695+0000","last_change":"2023-03-03T05:45:35.304483+0000","last_active":"2023-03-03T05:45:35.843695+0000","last_peered":"2023-03-03T05:45:35.843695+0000","last_clean":"2023-03-03T05:45:35.843695+0000","last_became_active":"2023-03-03T05:45:35.304202+0000","last_became_peered":"2023-03-03T05:45:35.304202+0000","last_unstale":"2023-03-03T05:45:35.843695+0000","last_undegraded":"2023-03-03T05:45:35.843695+0000","last_fullsized":"2023-03-03T05:45:35.843695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.851295+0000","last_change":"2023-03-03T05:45:35.869049+0000","last_active":"2023-03-03T05:45:36.851295+0000","last_peered":"2023-03-03T05:45:36.851295+0000","last_clean":"2023-03-03T05:45:36.851295+0000","last_became_active":"2023-03-03T05:45:35.868376+0000","last_became_peered":"2023-03-03T05:45:35.868376+0000","last_unstale":"2023-03-03T05:45:36.851295+0000","last_undegraded":"2023-03-03T05:45:36.851295+0000","last_fullsized":"2023-03-03T05:45:36.851295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:42:55.207846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843590+0000","last_change":"2023-03-03T05:45:29.056874+0000","last_active":"2023-03-03T05:45:35.843590+0000","last_peered":"2023-03-03T05:45:35.843590+0000","last_clean":"2023-03-03T05:45:35.843590+0000","last_became_active":"2023-03-03T05:45:29.056438+0000","last_became_peered":"2023-03-03T05:45:29.056438+0000","last_unstale":"2023-03-03T05:45:35.843590+0000","last_undegraded":"2023-03-03T05:45:35.843590+0000","last_fullsized":"2023-03-03T05:45:35.843590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:07:08.113953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.846141+0000","last_change":"2023-03-03T05:45:35.304562+0000","last_active":"2023-03-03T05:45:35.846141+0000","last_peered":"2023-03-03T05:45:35.846141+0000","last_clean":"2023-03-03T05:45:35.846141+0000","last_became_active":"2023-03-03T05:45:35.304264+0000","last_became_peered":"2023-03-03T05:45:35.304264+0000","last_unstale":"2023-03-03T05:45:35.846141+0000","last_undegraded":"2023-03-03T05:45:35.846141+0000","last_fullsized":"2023-03-03T05:45:35.846141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:24.370705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923094+0000","last_change":"2023-03-03T05:45:35.859779+0000","last_active":"2023-03-03T05:45:36.923094+0000","last_peered":"2023-03-03T05:45:36.923094+0000","last_clean":"2023-03-03T05:45:36.923094+0000","last_became_active":"2023-03-03T05:45:35.857596+0000","last_became_peered":"2023-03-03T05:45:35.857596+0000","last_unstale":"2023-03-03T05:45:36.923094+0000","last_undegraded":"2023-03-03T05:45:36.923094+0000","last_fullsized":"2023-03-03T05:45:36.923094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844418+0000","last_change":"2023-03-03T05:45:29.056891+0000","last_active":"2023-03-03T05:45:35.844418+0000","last_peered":"2023-03-03T05:45:35.844418+0000","last_clean":"2023-03-03T05:45:35.844418+0000","last_became_active":"2023-03-03T05:45:29.056483+0000","last_became_peered":"2023-03-03T05:45:29.056483+0000","last_unstale":"2023-03-03T05:45:35.844418+0000","last_undegraded":"2023-03-03T05:45:35.844418+0000","last_fullsized":"2023-03-03T05:45:35.844418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:25.701585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.847358+0000","last_change":"2023-03-03T05:45:35.293821+0000","last_active":"2023-03-03T05:45:35.847358+0000","last_peered":"2023-03-03T05:45:35.847358+0000","last_clean":"2023-03-03T05:45:35.847358+0000","last_became_active":"2023-03-03T05:45:35.293533+0000","last_became_peered":"2023-03-03T05:45:35.293533+0000","last_unstale":"2023-03-03T05:45:35.847358+0000","last_undegraded":"2023-03-03T05:45:35.847358+0000","last_fullsized":"2023-03-03T05:45:35.847358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:02:59.445259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923917+0000","last_change":"2023-03-03T05:45:35.866830+0000","last_active":"2023-03-03T05:45:36.923917+0000","last_peered":"2023-03-03T05:45:36.923917+0000","last_clean":"2023-03-03T05:45:36.923917+0000","last_became_active":"2023-03-03T05:45:35.866509+0000","last_became_peered":"2023-03-03T05:45:35.866509+0000","last_unstale":"2023-03-03T05:45:36.923917+0000","last_undegraded":"2023-03-03T05:45:36.923917+0000","last_fullsized":"2023-03-03T05:45:36.923917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:23.303165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.850545+0000","last_change":"2023-03-03T05:45:35.300794+0000","last_active":"2023-03-03T05:45:40.850545+0000","last_peered":"2023-03-03T05:45:40.850545+0000","last_clean":"2023-03-03T05:45:40.850545+0000","last_became_active":"2023-03-03T05:45:35.300531+0000","last_became_peered":"2023-03-03T05:45:35.300531+0000","last_unstale":"2023-03-03T05:45:40.850545+0000","last_undegraded":"2023-03-03T05:45:40.850545+0000","last_fullsized":"2023-03-03T05:45:40.850545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:19:57.349299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.843343+0000","last_change":"2023-03-03T05:45:29.193601+0000","last_active":"2023-03-03T05:45:35.843343+0000","last_peered":"2023-03-03T05:45:35.843343+0000","last_clean":"2023-03-03T05:45:35.843343+0000","last_became_active":"2023-03-03T05:45:29.193262+0000","last_became_peered":"2023-03-03T05:45:29.193262+0000","last_unstale":"2023-03-03T05:45:35.843343+0000","last_undegraded":"2023-03-03T05:45:35.843343+0000","last_fullsized":"2023-03-03T05:45:35.843343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:11:59.999523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853228+0000","last_change":"2023-03-03T05:45:35.864517+0000","last_active":"2023-03-03T05:45:36.853228+0000","last_peered":"2023-03-03T05:45:36.853228+0000","last_clean":"2023-03-03T05:45:36.853228+0000","last_became_active":"2023-03-03T05:45:35.864193+0000","last_became_peered":"2023-03-03T05:45:35.864193+0000","last_unstale":"2023-03-03T05:45:36.853228+0000","last_undegraded":"2023-03-03T05:45:36.853228+0000","last_fullsized":"2023-03-03T05:45:36.853228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:11:08.433881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.844985+0000","last_change":"2023-03-03T05:45:29.193546+0000","last_active":"2023-03-03T05:45:35.844985+0000","last_peered":"2023-03-03T05:45:35.844985+0000","last_clean":"2023-03-03T05:45:35.844985+0000","last_became_active":"2023-03-03T05:45:29.193172+0000","last_became_peered":"2023-03-03T05:45:29.193172+0000","last_unstale":"2023-03-03T05:45:35.844985+0000","last_undegraded":"2023-03-03T05:45:35.844985+0000","last_fullsized":"2023-03-03T05:45:35.844985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:28.037307+0000","last_clean_scrub_stamp":"2023-03-03T05:45:28.037307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:07:42.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853612+0000","last_change":"2023-03-03T05:45:35.296693+0000","last_active":"2023-03-03T05:45:36.853612+0000","last_peered":"2023-03-03T05:45:36.853612+0000","last_clean":"2023-03-03T05:45:36.853612+0000","last_became_active":"2023-03-03T05:45:35.296469+0000","last_became_peered":"2023-03-03T05:45:35.296469+0000","last_unstale":"2023-03-03T05:45:36.853612+0000","last_undegraded":"2023-03-03T05:45:36.853612+0000","last_fullsized":"2023-03-03T05:45:36.853612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850872+0000","last_change":"2023-03-03T05:45:35.856468+0000","last_active":"2023-03-03T05:45:36.850872+0000","last_peered":"2023-03-03T05:45:36.850872+0000","last_clean":"2023-03-03T05:45:36.850872+0000","last_became_active":"2023-03-03T05:45:35.855970+0000","last_became_peered":"2023-03-03T05:45:35.855970+0000","last_unstale":"2023-03-03T05:45:36.850872+0000","last_undegraded":"2023-03-03T05:45:36.850872+0000","last_fullsized":"2023-03-03T05:45:36.850872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:34:31.700909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853299+0000","last_change":"2023-03-03T05:45:35.301855+0000","last_active":"2023-03-03T05:45:36.853299+0000","last_peered":"2023-03-03T05:45:36.853299+0000","last_clean":"2023-03-03T05:45:36.853299+0000","last_became_active":"2023-03-03T05:45:35.301622+0000","last_became_peered":"2023-03-03T05:45:35.301622+0000","last_unstale":"2023-03-03T05:45:36.853299+0000","last_undegraded":"2023-03-03T05:45:36.853299+0000","last_fullsized":"2023-03-03T05:45:36.853299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:42.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.853538+0000","last_change":"2023-03-03T05:45:35.851780+0000","last_active":"2023-03-03T05:45:36.853538+0000","last_peered":"2023-03-03T05:45:36.853538+0000","last_clean":"2023-03-03T05:45:36.853538+0000","last_became_active":"2023-03-03T05:45:35.850190+0000","last_became_peered":"2023-03-03T05:45:35.850190+0000","last_unstale":"2023-03-03T05:45:36.853538+0000","last_undegraded":"2023-03-03T05:45:36.853538+0000","last_fullsized":"2023-03-03T05:45:36.853538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:13:17.664218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856909+0000","last_change":"2023-03-03T05:45:35.309047+0000","last_active":"2023-03-03T05:45:40.856909+0000","last_peered":"2023-03-03T05:45:40.856909+0000","last_clean":"2023-03-03T05:45:40.856909+0000","last_became_active":"2023-03-03T05:45:35.308564+0000","last_became_peered":"2023-03-03T05:45:35.308564+0000","last_unstale":"2023-03-03T05:45:40.856909+0000","last_undegraded":"2023-03-03T05:45:40.856909+0000","last_fullsized":"2023-03-03T05:45:40.856909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:55:40.540163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850462+0000","last_change":"2023-03-03T05:45:35.861316+0000","last_active":"2023-03-03T05:45:36.850462+0000","last_peered":"2023-03-03T05:45:36.850462+0000","last_clean":"2023-03-03T05:45:36.850462+0000","last_became_active":"2023-03-03T05:45:35.861052+0000","last_became_peered":"2023-03-03T05:45:35.861052+0000","last_unstale":"2023-03-03T05:45:36.850462+0000","last_undegraded":"2023-03-03T05:45:36.850462+0000","last_fullsized":"2023-03-03T05:45:36.850462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.858612+0000","last_change":"2023-03-03T05:45:35.303666+0000","last_active":"2023-03-03T05:45:40.858612+0000","last_peered":"2023-03-03T05:45:40.858612+0000","last_clean":"2023-03-03T05:45:40.858612+0000","last_became_active":"2023-03-03T05:45:35.303213+0000","last_became_peered":"2023-03-03T05:45:35.303213+0000","last_unstale":"2023-03-03T05:45:40.858612+0000","last_undegraded":"2023-03-03T05:45:40.858612+0000","last_fullsized":"2023-03-03T05:45:40.858612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:54.389717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850392+0000","last_change":"2023-03-03T05:45:35.857442+0000","last_active":"2023-03-03T05:45:36.850392+0000","last_peered":"2023-03-03T05:45:36.850392+0000","last_clean":"2023-03-03T05:45:36.850392+0000","last_became_active":"2023-03-03T05:45:35.857130+0000","last_became_peered":"2023-03-03T05:45:35.857130+0000","last_unstale":"2023-03-03T05:45:36.850392+0000","last_undegraded":"2023-03-03T05:45:36.850392+0000","last_fullsized":"2023-03-03T05:45:36.850392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923020+0000","last_change":"2023-03-03T05:45:35.305098+0000","last_active":"2023-03-03T05:45:36.923020+0000","last_peered":"2023-03-03T05:45:36.923020+0000","last_clean":"2023-03-03T05:45:36.923020+0000","last_became_active":"2023-03-03T05:45:35.304812+0000","last_became_peered":"2023-03-03T05:45:35.304812+0000","last_unstale":"2023-03-03T05:45:36.923020+0000","last_undegraded":"2023-03-03T05:45:36.923020+0000","last_fullsized":"2023-03-03T05:45:36.923020+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:54:45.080541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923341+0000","last_change":"2023-03-03T05:45:35.858806+0000","last_active":"2023-03-03T05:45:36.923341+0000","last_peered":"2023-03-03T05:45:36.923341+0000","last_clean":"2023-03-03T05:45:36.923341+0000","last_became_active":"2023-03-03T05:45:35.858471+0000","last_became_peered":"2023-03-03T05:45:35.858471+0000","last_unstale":"2023-03-03T05:45:36.923341+0000","last_undegraded":"2023-03-03T05:45:36.923341+0000","last_fullsized":"2023-03-03T05:45:36.923341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:07:33.940383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:40.856244+0000","last_change":"2023-03-03T05:45:35.310654+0000","last_active":"2023-03-03T05:45:40.856244+0000","last_peered":"2023-03-03T05:45:40.856244+0000","last_clean":"2023-03-03T05:45:40.856244+0000","last_became_active":"2023-03-03T05:45:35.310377+0000","last_became_peered":"2023-03-03T05:45:35.310377+0000","last_unstale":"2023-03-03T05:45:40.856244+0000","last_undegraded":"2023-03-03T05:45:40.856244+0000","last_fullsized":"2023-03-03T05:45:40.856244+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:34.268835+0000","last_clean_scrub_stamp":"2023-03-03T05:45:34.268835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:16:40.674337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.850763+0000","last_change":"2023-03-03T05:45:35.850670+0000","last_active":"2023-03-03T05:45:36.850763+0000","last_peered":"2023-03-03T05:45:36.850763+0000","last_clean":"2023-03-03T05:45:36.850763+0000","last_became_active":"2023-03-03T05:45:35.850380+0000","last_became_peered":"2023-03-03T05:45:35.850380+0000","last_unstale":"2023-03-03T05:45:36.850763+0000","last_undegraded":"2023-03-03T05:45:36.850763+0000","last_fullsized":"2023-03-03T05:45:36.850763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:47.330521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923387+0000","last_change":"2023-03-03T05:45:35.863660+0000","last_active":"2023-03-03T05:45:36.923387+0000","last_peered":"2023-03-03T05:45:36.923387+0000","last_clean":"2023-03-03T05:45:36.923387+0000","last_became_active":"2023-03-03T05:45:35.863234+0000","last_became_peered":"2023-03-03T05:45:35.863234+0000","last_unstale":"2023-03-03T05:45:36.923387+0000","last_undegraded":"2023-03-03T05:45:36.923387+0000","last_fullsized":"2023-03-03T05:45:36.923387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:39:49.621652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922937+0000","last_change":"2023-03-03T05:45:35.851019+0000","last_active":"2023-03-03T05:45:36.922937+0000","last_peered":"2023-03-03T05:45:36.922937+0000","last_clean":"2023-03-03T05:45:36.922937+0000","last_became_active":"2023-03-03T05:45:35.850439+0000","last_became_peered":"2023-03-03T05:45:35.850439+0000","last_unstale":"2023-03-03T05:45:36.922937+0000","last_undegraded":"2023-03-03T05:45:36.922937+0000","last_fullsized":"2023-03-03T05:45:36.922937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:13:43.693100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T05:45:35.962628+0000","last_change":"2023-03-03T05:45:35.860508+0000","last_active":"2023-03-03T05:45:35.962628+0000","last_peered":"2023-03-03T05:45:35.962628+0000","last_clean":"2023-03-03T05:45:35.962628+0000","last_became_active":"2023-03-03T05:45:35.860137+0000","last_became_peered":"2023-03-03T05:45:35.860137+0000","last_unstale":"2023-03-03T05:45:35.962628+0000","last_undegraded":"2023-03-03T05:45:35.962628+0000","last_fullsized":"2023-03-03T05:45:35.962628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:16:24.862378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.923487+0000","last_change":"2023-03-03T05:45:35.864824+0000","last_active":"2023-03-03T05:45:36.923487+0000","last_peered":"2023-03-03T05:45:36.923487+0000","last_clean":"2023-03-03T05:45:36.923487+0000","last_became_active":"2023-03-03T05:45:35.864466+0000","last_became_peered":"2023-03-03T05:45:35.864466+0000","last_unstale":"2023-03-03T05:45:36.923487+0000","last_undegraded":"2023-03-03T05:45:36.923487+0000","last_fullsized":"2023-03-03T05:45:36.923487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:50:29.403209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T05:45:36.922781+0000","last_change":"2023-03-03T05:45:35.859499+0000","last_active":"2023-03-03T05:45:36.922781+0000","last_peered":"2023-03-03T05:45:36.922781+0000","last_clean":"2023-03-03T05:45:36.922781+0000","last_became_active":"2023-03-03T05:45:35.859200+0000","last_became_peered":"2023-03-03T05:45:35.859200+0000","last_unstale":"2023-03-03T05:45:36.922781+0000","last_undegraded":"2023-03-03T05:45:36.922781+0000","last_fullsized":"2023-03-03T05:45:36.922781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T05:45:35.277217+0000","last_clean_scrub_stamp":"2023-03-03T05:45:35.277217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:14:15.179691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2589,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":6149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":90566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":88905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8068,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363772,"statfs":{"total":96636764160,"available":96628502528,"internally_reserved":0,"allocated":258048,"data_stored":90670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9088,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362752,"statfs":{"total":96636764160,"available":96627458048,"internally_reserved":0,"allocated":843776,"data_stored":679785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9140,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362700,"statfs":{"total":96636764160,"available":96627404800,"internally_reserved":0,"allocated":831488,"data_stored":678400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7988,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363852,"statfs":{"total":96636764160,"available":96628584448,"internally_reserved":0,"allocated":241664,"data_stored":88849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":88759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T05:46:05.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:46:05.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.380+0000 7f96e29ff700 1 Processor -- start 2023-03-03T05:46:05.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.380+0000 7f96e29ff700 1 -- start start 2023-03-03T05:46:05.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.381+0000 7f96e29ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f96dc0ced80 0x7f96dc0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.381+0000 7f96e29ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.381+0000 7f96e29ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc0cb9f0 con 0x7f96dc0cf9a0 2023-03-03T05:46:05.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.381+0000 7f96e29ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc0ca750 con 0x7f96dc0ced80 2023-03-03T05:46:05.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.381+0000 7f96e29ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc15a230 con 0x7f96dc1597f0 2023-03-03T05:46:05.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96dbfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f96dc0ced80 0x7f96dc0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96e0f9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f96dc0cf9a0 0x7f96dc15c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38860/0 (socket says 172.21.15.102:38860) 2023-03-03T05:46:05.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96e0f9c700 1 -- 172.21.15.102:0/4132163173 learned_addr learned my addr 172.21.15.102:0/4132163173 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:05.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96db7fe700 1 --2- 172.21.15.102:0/4132163173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96dbfff700 1 -- 172.21.15.102:0/4132163173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 msgr2=0x7f96dc0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96dbfff700 1 --2- 172.21.15.102:0/4132163173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96dbfff700 1 -- 172.21.15.102:0/4132163173 >> v1:172.21.15.102:6789/0 conn(0x7f96dc0cf9a0 legacy=0x7f96dc15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.382+0000 7f96dbfff700 1 -- 172.21.15.102:0/4132163173 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc159c80 con 0x7f96dc0ced80 2023-03-03T05:46:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96dbfff700 1 --2- 172.21.15.102:0/4132163173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc0cf150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009840 tx=0x7f96cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb959b2707db905 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96daffd700 1 -- 172.21.15.102:0/4132163173 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc011830 con 0x7f96dc0ced80 2023-03-03T05:46:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96daffd700 1 -- 172.21.15.102:0/4132163173 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc011990 con 0x7f96dc0ced80 2023-03-03T05:46:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96daffd700 1 -- 172.21.15.102:0/4132163173 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc011d20 con 0x7f96dc0ced80 2023-03-03T05:46:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96e29ff700 1 -- 172.21.15.102:0/4132163173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 msgr2=0x7f96dc0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96e29ff700 1 --2- 172.21.15.102:0/4132163173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc0cf150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009840 tx=0x7f96cc009b50 comp rx=0 tx=0).stop 2023-03-03T05:46:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96e29ff700 1 -- 172.21.15.102:0/4132163173 shutdown_connections 2023-03-03T05:46:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96e29ff700 1 --2- 172.21.15.102:0/4132163173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc0cf150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.383+0000 7f96e29ff700 1 --2- 172.21.15.102:0/4132163173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.384+0000 7f96e29ff700 1 -- 172.21.15.102:0/4132163173 >> 172.21.15.102:0/4132163173 conn(0x7f96dc065fb0 msgr2=0x7f96dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.384+0000 7f96e29ff700 1 -- 172.21.15.102:0/4132163173 shutdown_connections 2023-03-03T05:46:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.384+0000 7f96e29ff700 1 -- 172.21.15.102:0/4132163173 wait complete. 2023-03-03T05:46:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.386+0000 7f96e29ff700 1 Processor -- start 2023-03-03T05:46:05.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.386+0000 7f96e29ff700 1 -- start start 2023-03-03T05:46:05.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96e29ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96e29ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96dc0cf9a0 0x7f96dc14f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96e29ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96e29ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc14df10 con 0x7f96dc0ced80 2023-03-03T05:46:05.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96e29ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc150cc0 con 0x7f96dc0cf9a0 2023-03-03T05:46:05.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96e29ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc150830 con 0x7f96dc1597f0 2023-03-03T05:46:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96dbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96dbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc15a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42912/0 (socket says 172.21.15.102:42912) 2023-03-03T05:46:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96dbfff700 1 -- 172.21.15.102:0/875912559 learned_addr learned my addr 172.21.15.102:0/875912559 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96db7fe700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96dc0cf9a0 0x7f96dc14f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.387+0000 7f96e0f9c700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96dbfff700 1 -- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 msgr2=0x7f96dc14f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96dbfff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc14f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96dbfff700 1 -- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96dc0cf9a0 msgr2=0x7f96dc14f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96dbfff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96dc0cf9a0 0x7f96dc14f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96dbfff700 1 -- 172.21.15.102:0/875912559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0094a0 con 0x7f96dc0ced80 2023-03-03T05:46:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96db7fe700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96dc0cf9a0 0x7f96dc14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96dbfff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc15a960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0090b0 tx=0x7f96cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96d8ff9700 1 -- 172.21.15.102:0/875912559 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc0036a0 con 0x7f96dc0ced80 2023-03-03T05:46:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96d8ff9700 1 -- 172.21.15.102:0/875912559 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc01e070 con 0x7f96dc0ced80 2023-03-03T05:46:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96d8ff9700 1 -- 172.21.15.102:0/875912559 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc003a60 con 0x7f96dc0ced80 2023-03-03T05:46:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.388+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc0cb0c0 con 0x7f96dc0ced80 2023-03-03T05:46:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.389+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc0cade0 con 0x7f96dc0ced80 2023-03-03T05:46:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.390+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc0cb3a0 con 0x7f96dc0ced80 2023-03-03T05:46:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.391+0000 7f96d8ff9700 1 -- 172.21.15.102:0/875912559 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f96cc00f050 con 0x7f96dc0ced80 2023-03-03T05:46:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.392+0000 7f96d8ff9700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96c80682f0 0x7f96c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.392+0000 7f96d8ff9700 1 -- 172.21.15.102:0/875912559 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f96cc086c60 con 0x7f96dc0ced80 2023-03-03T05:46:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.393+0000 7f96db7fe700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96c80682f0 0x7f96c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.393+0000 7f96db7fe700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96c80682f0 0x7f96c806a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96dc14ed80 tx=0x7f96d0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:05.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.396+0000 7f96d8ff9700 1 -- 172.21.15.102:0/875912559 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f96cc0533d0 con 0x7f96dc0ced80 2023-03-03T05:46:05.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.591+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96dc0677f0 con 0x7f96dc0ced80 2023-03-03T05:46:05.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.592+0000 7f96d8ff9700 1 -- 172.21.15.102:0/875912559 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f96cc020020 con 0x7f96dc0ced80 2023-03-03T05:46:05.594 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:05.594 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:46:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96c80682f0 msgr2=0x7f96c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96c80682f0 0x7f96c806a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96dc14ed80 tx=0x7f96d0009210 comp rx=0 tx=0).stop 2023-03-03T05:46:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 msgr2=0x7f96dc15a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc15a960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0090b0 tx=0x7f96cc004750 comp rx=0 tx=0).stop 2023-03-03T05:46:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 shutdown_connections 2023-03-03T05:46:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96c80682f0 0x7f96c806a7a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96dc1597f0 0x7f96dc14f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96dc0ced80 0x7f96dc15a960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 --2- 172.21.15.102:0/875912559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96dc0cf9a0 0x7f96dc14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 >> 172.21.15.102:0/875912559 conn(0x7f96dc065fb0 msgr2=0x7f96dc0cbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 shutdown_connections 2023-03-03T05:46:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.595+0000 7f96e29ff700 1 -- 172.21.15.102:0/875912559 wait complete. 2023-03-03T05:46:05.624 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T05:45:20.032740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T05:45:27.930463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T05:45:33.613145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T05:45:34.842860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T05:46:05.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T05:46:05.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.870+0000 7f9eed9ff700 1 Processor -- start 2023-03-03T05:46:05.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.870+0000 7f9eed9ff700 1 -- start start 2023-03-03T05:46:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.871+0000 7f9eed9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ee80ce410 0x7f9ee80cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.871+0000 7f9eed9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80cd760 0x7f9ee8163720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.871+0000 7f9eed9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee8153630 con 0x7f9ee80ccc40 2023-03-03T05:46:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.871+0000 7f9eed9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee80ca7a0 con 0x7f9ee80ce410 2023-03-03T05:46:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.871+0000 7f9eed9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80cd550 con 0x7f9ee80cd760 2023-03-03T05:46:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80cd760 0x7f9ee8163720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80cd760 0x7f9ee8163720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37842/0 (socket says 172.21.15.102:37842) 2023-03-03T05:46:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 -- 172.21.15.102:0/2344590071 learned_addr learned my addr 172.21.15.102:0/2344590071 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee67fc700 1 --2- 172.21.15.102:0/2344590071 >> v2:172.21.15.102:3300/0 conn(0x7f9ee80ce410 0x7f9ee80cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 -- 172.21.15.102:0/2344590071 >> v2:172.21.15.102:3300/0 conn(0x7f9ee80ce410 msgr2=0x7f9ee80cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 --2- 172.21.15.102:0/2344590071 >> v2:172.21.15.102:3300/0 conn(0x7f9ee80ce410 0x7f9ee80cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 -- 172.21.15.102:0/2344590071 >> v1:172.21.15.172:6789/0 conn(0x7f9ee80ccc40 legacy=0x7f9ee80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 -- 172.21.15.102:0/2344590071 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee815d1d0 con 0x7f9ee80cd760 2023-03-03T05:46:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.872+0000 7f9ee77fe700 1 --2- 172.21.15.102:0/2344590071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80cd760 0x7f9ee8163720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009e50 tx=0x7f9ee000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237bdc16236d880e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/2344590071 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee000e070 con 0x7f9ee80cd760 2023-03-03T05:46:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/2344590071 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee000b9c0 con 0x7f9ee80cd760 2023-03-03T05:46:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9ee5ffb700 1 -- 172.21.15.102:0/2344590071 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee000bd70 con 0x7f9ee80cd760 2023-03-03T05:46:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 -- 172.21.15.102:0/2344590071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80cd760 msgr2=0x7f9ee8163720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/2344590071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80cd760 0x7f9ee8163720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009e50 tx=0x7f9ee000b330 comp rx=0 tx=0).stop 2023-03-03T05:46:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 -- 172.21.15.102:0/2344590071 shutdown_connections 2023-03-03T05:46:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/2344590071 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80cd760 0x7f9ee8163720 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/2344590071 >> v2:172.21.15.102:3300/0 conn(0x7f9ee80ce410 0x7f9ee80cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 -- 172.21.15.102:0/2344590071 >> 172.21.15.102:0/2344590071 conn(0x7f9ee8065fb0 msgr2=0x7f9ee80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 -- 172.21.15.102:0/2344590071 shutdown_connections 2023-03-03T05:46:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.873+0000 7f9eed9ff700 1 -- 172.21.15.102:0/2344590071 wait complete. 2023-03-03T05:46:05.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.875+0000 7f9eed9ff700 1 Processor -- start 2023-03-03T05:46:05.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.876+0000 7f9eed9ff700 1 -- start start 2023-03-03T05:46:05.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.876+0000 7f9eed9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ee80ccc40 0x7f9ee8151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9eed9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ee80cd760 0x7f9ee8150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9eed9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80ce410 0x7f9ee8155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9eed9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8155380 con 0x7f9ee80ccc40 2023-03-03T05:46:05.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9eed9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8154da0 con 0x7f9ee80cd760 2023-03-03T05:46:05.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9eed9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee814ef00 con 0x7f9ee80ce410 2023-03-03T05:46:05.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ee80ccc40 0x7f9ee8151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ee80ccc40 0x7f9ee8151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42932/0 (socket says 172.21.15.102:42932) 2023-03-03T05:46:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee6ffd700 1 -- 172.21.15.102:0/3654098665 learned_addr learned my addr 172.21.15.102:0/3654098665 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee67fc700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ee80cd760 0x7f9ee8150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee77fe700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80ce410 0x7f9ee8155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee77fe700 1 -- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ee80cd760 msgr2=0x7f9ee8150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee77fe700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ee80cd760 0x7f9ee8150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee77fe700 1 -- 172.21.15.102:0/3654098665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ee80ccc40 msgr2=0x7f9ee8151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee77fe700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ee80ccc40 0x7f9ee8151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:05.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee77fe700 1 -- 172.21.15.102:0/3654098665 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0009b30 con 0x7f9ee80ce410 2023-03-03T05:46:05.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.877+0000 7f9ee6ffd700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ee80ccc40 0x7f9ee8151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:05.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.878+0000 7f9ee77fe700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80ce410 0x7f9ee8155810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8067730 tx=0x7f9ee0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.878+0000 7f9ecffff700 1 -- 172.21.15.102:0/3654098665 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee00099b0 con 0x7f9ee80ce410 2023-03-03T05:46:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.878+0000 7f9ecffff700 1 -- 172.21.15.102:0/3654098665 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee0004250 con 0x7f9ee80ce410 2023-03-03T05:46:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.878+0000 7f9ecffff700 1 -- 172.21.15.102:0/3654098665 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee00045e0 con 0x7f9ee80ce410 2023-03-03T05:46:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.878+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee81594b0 con 0x7f9ee80ce410 2023-03-03T05:46:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.878+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee81591d0 con 0x7f9ee80ce410 2023-03-03T05:46:05.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.879+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee8159790 con 0x7f9ee80ce410 2023-03-03T05:46:05.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.880+0000 7f9ecffff700 1 -- 172.21.15.102:0/3654098665 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ee0004740 con 0x7f9ee80ce410 2023-03-03T05:46:05.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.882+0000 7f9ecffff700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ed40682f0 0x7f9ed406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:05.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.882+0000 7f9ecffff700 1 -- 172.21.15.102:0/3654098665 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9ee0086f60 con 0x7f9ee80ce410 2023-03-03T05:46:05.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.882+0000 7f9ee6ffd700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ed40682f0 0x7f9ed406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:05.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.883+0000 7f9ee6ffd700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ed40682f0 0x7f9ed406a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8001d10 tx=0x7f9ed8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:05.886+0000 7f9ecffff700 1 -- 172.21.15.102:0/3654098665 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ee0053640 con 0x7f9ee80ce410 2023-03-03T05:46:06.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.077+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ee8155530 con 0x7f9ee80ce410 2023-03-03T05:46:06.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.078+0000 7f9ecffff700 1 -- 172.21.15.102:0/3654098665 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ee8155530 con 0x7f9ee80ce410 2023-03-03T05:46:06.080 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2023-03-03T05:46:06.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ed40682f0 msgr2=0x7f9ed406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ed40682f0 0x7f9ed406a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8001d10 tx=0x7f9ed8009040 comp rx=0 tx=0).stop 2023-03-03T05:46:06.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80ce410 msgr2=0x7f9ee8155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80ce410 0x7f9ee8155810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8067730 tx=0x7f9ee0004ad0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 shutdown_connections 2023-03-03T05:46:06.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ed40682f0 0x7f9ed406a7a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ee80ce410 0x7f9ee8155810 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ee80ccc40 0x7f9ee8151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 --2- 172.21.15.102:0/3654098665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ee80cd760 0x7f9ee8150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 >> 172.21.15.102:0/3654098665 conn(0x7f9ee8065fb0 msgr2=0x7f9ee8152900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:06.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 shutdown_connections 2023-03-03T05:46:06.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.081+0000 7f9eed9ff700 1 -- 172.21.15.102:0/3654098665 wait complete. 2023-03-03T05:46:06.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T05:46:06.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.354+0000 7f37d319d700 1 Processor -- start 2023-03-03T05:46:06.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.355+0000 7f37d319d700 1 -- start start 2023-03-03T05:46:06.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.356+0000 7f37d319d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc153180 0x7f37cc0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.356+0000 7f37d319d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37cc0c9b80 0x7f37cc0c4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.356+0000 7f37d319d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc159140 con 0x7f37cc158cb0 2023-03-03T05:46:06.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.356+0000 7f37d319d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc159cc0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.356+0000 7f37d319d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc159420 con 0x7f37cc153180 2023-03-03T05:46:06.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.356+0000 7f37d299c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37cc0c9b80 0x7f37cc0c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d219b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc153180 0x7f37cc0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d299c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37cc0c9b80 0x7f37cc0c4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42948/0 (socket says 172.21.15.102:42948) 2023-03-03T05:46:06.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d299c700 1 -- 172.21.15.102:0/3087536219 learned_addr learned my addr 172.21.15.102:0/3087536219 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:06.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d299c700 1 -- 172.21.15.102:0/3087536219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc153180 msgr2=0x7f37cc0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:06.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d299c700 1 --2- 172.21.15.102:0/3087536219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc153180 0x7f37cc0c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d299c700 1 -- 172.21.15.102:0/3087536219 >> v1:172.21.15.172:6789/0 conn(0x7f37cc158cb0 legacy=0x7f37cc15c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d299c700 1 -- 172.21.15.102:0/3087536219 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc159700 con 0x7f37cc0c9b80 2023-03-03T05:46:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d1199700 1 -- 172.21.15.102:0/3087536219 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751114760 0 0) 0x7f37cc159140 con 0x7f37cc158cb0 2023-03-03T05:46:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.357+0000 7f37d299c700 1 --2- 172.21.15.102:0/3087536219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0c9b80 0x7f37cc0c4390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009e50 tx=0x7f37c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97499b96661bb5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d1199700 1 -- 172.21.15.102:0/3087536219 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c400e070 con 0x7f37cc0c9b80 2023-03-03T05:46:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d1199700 1 -- 172.21.15.102:0/3087536219 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c400b910 con 0x7f37cc0c9b80 2023-03-03T05:46:06.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d1199700 1 -- 172.21.15.102:0/3087536219 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c400bcc0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d319d700 1 -- 172.21.15.102:0/3087536219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0c9b80 msgr2=0x7f37cc0c4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d319d700 1 --2- 172.21.15.102:0/3087536219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0c9b80 0x7f37cc0c4390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009e50 tx=0x7f37c400b330 comp rx=0 tx=0).stop 2023-03-03T05:46:06.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d319d700 1 -- 172.21.15.102:0/3087536219 shutdown_connections 2023-03-03T05:46:06.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d319d700 1 --2- 172.21.15.102:0/3087536219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0c9b80 0x7f37cc0c4390 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d319d700 1 --2- 172.21.15.102:0/3087536219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc153180 0x7f37cc0c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d319d700 1 -- 172.21.15.102:0/3087536219 >> 172.21.15.102:0/3087536219 conn(0x7f37cc071a30 msgr2=0x7f37cc065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:06.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.358+0000 7f37d319d700 1 -- 172.21.15.102:0/3087536219 shutdown_connections 2023-03-03T05:46:06.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.359+0000 7f37d319d700 1 -- 172.21.15.102:0/3087536219 wait complete. 2023-03-03T05:46:06.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.361+0000 7f37d319d700 1 Processor -- start 2023-03-03T05:46:06.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.362+0000 7f37d319d700 1 -- start start 2023-03-03T05:46:06.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d319d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0c9b80 0x7f37cc159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d319d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc153180 0x7f37cc14a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d319d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc158cb0 0x7f37cc1f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d319d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc14f2a0 con 0x7f37cc153180 2023-03-03T05:46:06.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d319d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc1f1bf0 con 0x7f37cc158cb0 2023-03-03T05:46:06.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d319d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc14e7b0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d199a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc153180 0x7f37cc14a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d219b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0c9b80 0x7f37cc159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d199a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc153180 0x7f37cc14a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42950/0 (socket says 172.21.15.102:42950) 2023-03-03T05:46:06.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d199a700 1 -- 172.21.15.102:0/3128120292 learned_addr learned my addr 172.21.15.102:0/3128120292 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:06.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.363+0000 7f37d299c700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc158cb0 0x7f37cc1f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d219b700 1 -- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc158cb0 msgr2=0x7f37cc1f2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d219b700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc158cb0 0x7f37cc1f2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d219b700 1 -- 172.21.15.102:0/3128120292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc153180 msgr2=0x7f37cc14a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d219b700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc153180 0x7f37cc14a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d219b700 1 -- 172.21.15.102:0/3128120292 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c4009b30 con 0x7f37cc0c9b80 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d199a700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc153180 0x7f37cc14a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d219b700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0c9b80 0x7f37cc159e10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37c800e900 tx=0x7f37c800ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37c2ffd700 1 -- 172.21.15.102:0/3128120292 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8013680 con 0x7f37cc0c9b80 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37c2ffd700 1 -- 172.21.15.102:0/3128120292 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c80137e0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37c2ffd700 1 -- 172.21.15.102:0/3128120292 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8013b50 con 0x7f37cc0c9b80 2023-03-03T05:46:06.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.364+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc15c3c0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.365+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc1599e0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.366+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc1527f0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.367+0000 7f37c2ffd700 1 -- 172.21.15.102:0/3128120292 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37c8019030 con 0x7f37cc0c9b80 2023-03-03T05:46:06.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.368+0000 7f37c2ffd700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37b00682f0 0x7f37b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.368+0000 7f37c2ffd700 1 -- 172.21.15.102:0/3128120292 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f37c8085e40 con 0x7f37cc0c9b80 2023-03-03T05:46:06.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.368+0000 7f37d199a700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37b00682f0 0x7f37b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.369+0000 7f37d199a700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37b00682f0 0x7f37b006a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37cc14eba0 tx=0x7f37bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:06.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.372+0000 7f37c2ffd700 1 -- 172.21.15.102:0/3128120292 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37c80525b0 con 0x7f37cc0c9b80 2023-03-03T05:46:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.560+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f37cc0c5050 con 0x7f37cc0c9b80 2023-03-03T05:46:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.561+0000 7f37c2ffd700 1 -- 172.21.15.102:0/3128120292 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f37c8017020 con 0x7f37cc0c9b80 2023-03-03T05:46:06.563 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 8 2023-03-03T05:46:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37b00682f0 msgr2=0x7f37b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37b00682f0 0x7f37b006a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37cc14eba0 tx=0x7f37bc009040 comp rx=0 tx=0).stop 2023-03-03T05:46:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0c9b80 msgr2=0x7f37cc159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0c9b80 0x7f37cc159e10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37c800e900 tx=0x7f37c800ec10 comp rx=0 tx=0).stop 2023-03-03T05:46:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 shutdown_connections 2023-03-03T05:46:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37b00682f0 0x7f37b006a7a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0c9b80 0x7f37cc159e10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc158cb0 0x7f37cc1f2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 --2- 172.21.15.102:0/3128120292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc153180 0x7f37cc14a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 >> 172.21.15.102:0/3128120292 conn(0x7f37cc071a30 msgr2=0x7f37cc157520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 shutdown_connections 2023-03-03T05:46:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.564+0000 7f37d319d700 1 -- 172.21.15.102:0/3128120292 wait complete. 2023-03-03T05:46:06.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T05:46:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.836+0000 7f2aadf9d700 1 Processor -- start 2023-03-03T05:46:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.836+0000 7f2aadf9d700 1 -- start start 2023-03-03T05:46:06.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.838+0000 7f2aadf9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa80c9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.838+0000 7f2aadf9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2aa8158cb0 0x7f2aa815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aadf9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa8159140 con 0x7f2aa80c9b80 2023-03-03T05:46:06.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aadf9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa8159cc0 con 0x7f2aa8158cb0 2023-03-03T05:46:06.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aadf9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa8159420 con 0x7f2aa8153180 2023-03-03T05:46:06.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aacf9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa80c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aa7fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2aa8158cb0 0x7f2aa815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aad79c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2aa80c9b80 0x7f2aa80c4390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37504/0 (socket says 172.21.15.102:37504) 2023-03-03T05:46:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aad79c700 1 -- 172.21.15.102:0/3968676460 learned_addr learned my addr 172.21.15.102:0/3968676460 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aacf9b700 1 -- 172.21.15.102:0/3968676460 >> v2:172.21.15.172:3300/0 conn(0x7f2aa8158cb0 msgr2=0x7f2aa815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aacf9b700 1 --2- 172.21.15.102:0/3968676460 >> v2:172.21.15.172:3300/0 conn(0x7f2aa8158cb0 0x7f2aa815c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aacf9b700 1 -- 172.21.15.102:0/3968676460 >> v1:172.21.15.172:6789/0 conn(0x7f2aa80c9b80 legacy=0x7f2aa80c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.839+0000 7f2aacf9b700 1 -- 172.21.15.102:0/3968676460 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8159700 con 0x7f2aa8153180 2023-03-03T05:46:06.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.840+0000 7f2aacf9b700 1 --2- 172.21.15.102:0/3968676460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa80c9640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009840 tx=0x7f2a9c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=328883ebf6f89d5a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:06.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.840+0000 7f2aa77fe700 1 -- 172.21.15.102:0/3968676460 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c011830 con 0x7f2aa8153180 2023-03-03T05:46:06.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.840+0000 7f2aa77fe700 1 -- 172.21.15.102:0/3968676460 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c011990 con 0x7f2aa8153180 2023-03-03T05:46:06.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.840+0000 7f2aa77fe700 1 -- 172.21.15.102:0/3968676460 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c011d20 con 0x7f2aa8153180 2023-03-03T05:46:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.840+0000 7f2aadf9d700 1 -- 172.21.15.102:0/3968676460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 msgr2=0x7f2aa80c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.840+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/3968676460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa80c9640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009840 tx=0x7f2a9c009b50 comp rx=0 tx=0).stop 2023-03-03T05:46:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.841+0000 7f2aadf9d700 1 -- 172.21.15.102:0/3968676460 shutdown_connections 2023-03-03T05:46:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.841+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/3968676460 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa80c9640 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.841+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/3968676460 >> v2:172.21.15.172:3300/0 conn(0x7f2aa8158cb0 0x7f2aa815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.841+0000 7f2aadf9d700 1 -- 172.21.15.102:0/3968676460 >> 172.21.15.102:0/3968676460 conn(0x7f2aa8071a30 msgr2=0x7f2aa8065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.841+0000 7f2aadf9d700 1 -- 172.21.15.102:0/3968676460 shutdown_connections 2023-03-03T05:46:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.841+0000 7f2aadf9d700 1 -- 172.21.15.102:0/3968676460 wait complete. 2023-03-03T05:46:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.843+0000 7f2aadf9d700 1 Processor -- start 2023-03-03T05:46:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.843+0000 7f2aadf9d700 1 -- start start 2023-03-03T05:46:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aadf9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa80c9b80 0x7f2aa815b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aadf9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa814d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aadf9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2aa8158cb0 0x7f2aa81f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aadf9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80cbc00 con 0x7f2aa8158cb0 2023-03-03T05:46:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aadf9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa814c0b0 con 0x7f2aa80c9b80 2023-03-03T05:46:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aadf9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa81f2770 con 0x7f2aa8153180 2023-03-03T05:46:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aa7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa814d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aad79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2aa8158cb0 0x7f2aa81f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aa7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa814d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39016/0 (socket says 172.21.15.102:39016) 2023-03-03T05:46:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aad79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2aa8158cb0 0x7f2aa81f1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54876/0 (socket says 172.21.15.102:54876) 2023-03-03T05:46:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aa7fff700 1 -- 172.21.15.102:0/2103090723 learned_addr learned my addr 172.21.15.102:0/2103090723 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.844+0000 7f2aacf9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa80c9b80 0x7f2aa815b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa7fff700 1 -- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa80c9b80 msgr2=0x7f2aa815b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa7fff700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa80c9b80 0x7f2aa815b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa7fff700 1 -- 172.21.15.102:0/2103090723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2aa8158cb0 msgr2=0x7f2aa81f1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa7fff700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2aa8158cb0 0x7f2aa81f1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:06.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa7fff700 1 -- 172.21.15.102:0/2103090723 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0094a0 con 0x7f2aa8153180 2023-03-03T05:46:06.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa7fff700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa814d7e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400e8a0 tx=0x7f2a9400ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:06.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa57fa700 1 -- 172.21.15.102:0/2103090723 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94009590 con 0x7f2aa8153180 2023-03-03T05:46:06.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa57fa700 1 -- 172.21.15.102:0/2103090723 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94013680 con 0x7f2aa8153180 2023-03-03T05:46:06.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aa57fa700 1 -- 172.21.15.102:0/2103090723 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a940139f0 con 0x7f2aa8153180 2023-03-03T05:46:06.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.845+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa815c3c0 con 0x7f2aa8153180 2023-03-03T05:46:06.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.846+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa81599e0 con 0x7f2aa8153180 2023-03-03T05:46:06.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.847+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa80c5050 con 0x7f2aa8153180 2023-03-03T05:46:06.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.848+0000 7f2aa57fa700 1 -- 172.21.15.102:0/2103090723 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a94019030 con 0x7f2aa8153180 2023-03-03T05:46:06.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.849+0000 7f2aa57fa700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2a8c0682f0 0x7f2a8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:06.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.849+0000 7f2aa57fa700 1 -- 172.21.15.102:0/2103090723 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2a94085a30 con 0x7f2aa8153180 2023-03-03T05:46:06.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.850+0000 7f2aacf9b700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2a8c0682f0 0x7f2a8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:06.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.850+0000 7f2aacf9b700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2a8c0682f0 0x7f2a8c06a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c000c00 tx=0x7f2a9c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:06.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:06.853+0000 7f2aa57fa700 1 -- 172.21.15.102:0/2103090723 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a940521a0 con 0x7f2aa8153180 2023-03-03T05:46:07.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.048+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2aa814c1f0 con 0x7f2aa8153180 2023-03-03T05:46:07.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.048+0000 7f2aa57fa700 1 -- 172.21.15.102:0/2103090723 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2aa814c1f0 con 0x7f2aa8153180 2023-03-03T05:46:07.051 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2023-03-03T05:46:07.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2a8c0682f0 msgr2=0x7f2a8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2a8c0682f0 0x7f2a8c06a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c000c00 tx=0x7f2a9c008030 comp rx=0 tx=0).stop 2023-03-03T05:46:07.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 msgr2=0x7f2aa814d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa814d7e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400e8a0 tx=0x7f2a9400ebb0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 shutdown_connections 2023-03-03T05:46:07.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2aa8158cb0 0x7f2aa81f1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa80c9b80 0x7f2aa815b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2a8c0682f0 0x7f2a8c06a7a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 --2- 172.21.15.102:0/2103090723 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2aa8153180 0x7f2aa814d7e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 >> 172.21.15.102:0/2103090723 conn(0x7f2aa8071a30 msgr2=0x7f2aa806bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:07.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 shutdown_connections 2023-03-03T05:46:07.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.051+0000 7f2aadf9d700 1 -- 172.21.15.102:0/2103090723 wait complete. 2023-03-03T05:46:07.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T05:46:07.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.333+0000 7f433df9d700 1 Processor -- start 2023-03-03T05:46:07.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.333+0000 7f433df9d700 1 -- start start 2023-03-03T05:46:07.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.335+0000 7f433df9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4338158cb0 0x7f433815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.335+0000 7f433df9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4338159140 con 0x7f4338153180 2023-03-03T05:46:07.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.335+0000 7f433df9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4338159420 con 0x7f433815ce20 2023-03-03T05:46:07.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.335+0000 7f433df9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4338159cc0 con 0x7f4338158cb0 2023-03-03T05:46:07.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.335+0000 7f433cf9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4338153180 0x7f43380c9640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50702/0 (socket says 172.21.15.102:50702) 2023-03-03T05:46:07.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.335+0000 7f433cf9b700 1 -- 172.21.15.102:0/4040237402 learned_addr learned my addr 172.21.15.102:0/4040237402 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:07.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.335+0000 7f4337fff700 1 --2- 172.21.15.102:0/4040237402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4338158cb0 0x7f433815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f43377fe700 1 -- 172.21.15.102:0/4040237402 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547520220 0 0) 0x7f4338159140 con 0x7f4338153180 2023-03-03T05:46:07.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f43377fe700 1 -- 172.21.15.102:0/4040237402 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43280035b0 con 0x7f4338153180 2023-03-03T05:46:07.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f43377fe700 1 -- 172.21.15.102:0/4040237402 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546225472 0 0) 0x7f4338159420 con 0x7f433815ce20 2023-03-03T05:46:07.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f43377fe700 1 -- 172.21.15.102:0/4040237402 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4338159140 con 0x7f433815ce20 2023-03-03T05:46:07.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f4337fff700 1 -- 172.21.15.102:0/4040237402 >> v1:172.21.15.172:6789/0 conn(0x7f433815ce20 legacy=0x7f43380c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f4337fff700 1 -- 172.21.15.102:0/4040237402 >> v1:172.21.15.102:6789/0 conn(0x7f4338153180 legacy=0x7f43380c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f4337fff700 1 -- 172.21.15.102:0/4040237402 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338159700 con 0x7f4338158cb0 2023-03-03T05:46:07.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f4337fff700 1 --2- 172.21.15.102:0/4040237402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4338158cb0 0x7f433815c8e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4324009870 tx=0x7f4324009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c45bb7b3b1870cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:07.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f43377fe700 1 -- 172.21.15.102:0/4040237402 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324011830 con 0x7f4338158cb0 2023-03-03T05:46:07.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f43377fe700 1 -- 172.21.15.102:0/4040237402 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324011990 con 0x7f4338158cb0 2023-03-03T05:46:07.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.336+0000 7f43377fe700 1 -- 172.21.15.102:0/4040237402 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324011d20 con 0x7f4338158cb0 2023-03-03T05:46:07.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.337+0000 7f433df9d700 1 -- 172.21.15.102:0/4040237402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4338158cb0 msgr2=0x7f433815c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.337+0000 7f433df9d700 1 --2- 172.21.15.102:0/4040237402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4338158cb0 0x7f433815c8e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4324009870 tx=0x7f4324009b80 comp rx=0 tx=0).stop 2023-03-03T05:46:07.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.337+0000 7f433df9d700 1 -- 172.21.15.102:0/4040237402 shutdown_connections 2023-03-03T05:46:07.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.337+0000 7f433df9d700 1 --2- 172.21.15.102:0/4040237402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4338158cb0 0x7f433815c8e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.337+0000 7f433df9d700 1 -- 172.21.15.102:0/4040237402 >> 172.21.15.102:0/4040237402 conn(0x7f4338071a30 msgr2=0x7f4338065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:07.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.337+0000 7f433df9d700 1 -- 172.21.15.102:0/4040237402 shutdown_connections 2023-03-03T05:46:07.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.337+0000 7f433df9d700 1 -- 172.21.15.102:0/4040237402 wait complete. 2023-03-03T05:46:07.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.339+0000 7f433df9d700 1 Processor -- start 2023-03-03T05:46:07.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.339+0000 7f433df9d700 1 -- start start 2023-03-03T05:46:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.340+0000 7f433df9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4338153180 0x7f43381eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.340+0000 7f433df9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4338158cb0 0x7f43381f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.340+0000 7f433df9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f433815ce20 0x7f4338159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.340+0000 7f433df9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43381ee410 con 0x7f4338153180 2023-03-03T05:46:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.340+0000 7f433df9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380b2780 con 0x7f4338158cb0 2023-03-03T05:46:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.340+0000 7f433df9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380b2a60 con 0x7f433815ce20 2023-03-03T05:46:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433cf9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4338153180 0x7f43381eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f433815ce20 0x7f4338159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f4337fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4338158cb0 0x7f43381f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f433815ce20 0x7f4338159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39036/0 (socket says 172.21.15.102:39036) 2023-03-03T05:46:07.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 -- 172.21.15.102:0/2194279091 learned_addr learned my addr 172.21.15.102:0/2194279091 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:07.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 -- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4338158cb0 msgr2=0x7f43381f2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4338158cb0 0x7f43381f2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 -- 172.21.15.102:0/2194279091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4338153180 msgr2=0x7f43381eeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4338153180 0x7f43381eeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 -- 172.21.15.102:0/2194279091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43240094a0 con 0x7f433815ce20 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433cf9b700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4338153180 0x7f43381eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f4337fff700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4338158cb0 0x7f43381f2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.341+0000 7f433d79c700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f433815ce20 0x7f4338159e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f433000cbc0 tx=0x7f433000ced0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.342+0000 7f43357fa700 1 -- 172.21.15.102:0/2194279091 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330003c00 con 0x7f433815ce20 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.342+0000 7f43357fa700 1 -- 172.21.15.102:0/2194279091 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330003d60 con 0x7f433815ce20 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.342+0000 7f43357fa700 1 -- 172.21.15.102:0/2194279091 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000aaa0 con 0x7f433815ce20 2023-03-03T05:46:07.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.342+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433815c3c0 con 0x7f433815ce20 2023-03-03T05:46:07.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.342+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43381599e0 con 0x7f433815ce20 2023-03-03T05:46:07.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.343+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43380c5050 con 0x7f433815ce20 2023-03-03T05:46:07.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.345+0000 7f43357fa700 1 -- 172.21.15.102:0/2194279091 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4330010050 con 0x7f433815ce20 2023-03-03T05:46:07.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.346+0000 7f43357fa700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f431c068320 0x7f431c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.346+0000 7f43357fa700 1 -- 172.21.15.102:0/2194279091 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4330085b90 con 0x7f433815ce20 2023-03-03T05:46:07.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.346+0000 7f433cf9b700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f431c068320 0x7f431c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.347+0000 7f433cf9b700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f431c068320 0x7f431c06a7d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f432c007fd0 tx=0x7f432c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:07.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.350+0000 7f43357fa700 1 -- 172.21.15.102:0/2194279091 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4330052300 con 0x7f433815ce20 2023-03-03T05:46:07.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.541+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43380ca030 con 0x7f433815ce20 2023-03-03T05:46:07.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.541+0000 7f43357fa700 1 -- 172.21.15.102:0/2194279091 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4330055920 con 0x7f433815ce20 2023-03-03T05:46:07.545 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2023-03-03T05:46:07.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f431c068320 msgr2=0x7f431c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f431c068320 0x7f431c06a7d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f432c007fd0 tx=0x7f432c007de0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f433815ce20 msgr2=0x7f4338159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f433815ce20 0x7f4338159e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f433000cbc0 tx=0x7f433000ced0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 shutdown_connections 2023-03-03T05:46:07.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f431c068320 0x7f431c06a7d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f433815ce20 0x7f4338159e10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4338153180 0x7f43381eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 --2- 172.21.15.102:0/2194279091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4338158cb0 0x7f43381f2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 >> 172.21.15.102:0/2194279091 conn(0x7f4338071a30 msgr2=0x7f43380c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:07.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 shutdown_connections 2023-03-03T05:46:07.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.544+0000 7f433df9d700 1 -- 172.21.15.102:0/2194279091 wait complete. 2023-03-03T05:46:07.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:46:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.814+0000 7f1843fff700 1 Processor -- start 2023-03-03T05:46:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.815+0000 7f1843fff700 1 -- start start 2023-03-03T05:46:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1843fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f183c153290 0x7f183c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1843fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f183c1593c0 0x7f183c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1843fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1843fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c153660 con 0x7f183c1593c0 2023-03-03T05:46:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1843fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c06c360 con 0x7f183c153290 2023-03-03T05:46:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1843fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c15cea0 con 0x7f183c0cd8e0 2023-03-03T05:46:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1841d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f183c153290 0x7f183c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1841d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f183c153290 0x7f183c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40258/0 (socket says 172.21.15.102:40258) 2023-03-03T05:46:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f1841d9b700 1 -- 172.21.15.102:0/2040609665 learned_addr learned my addr 172.21.15.102:0/2040609665 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f184259c700 1 --2- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.816+0000 7f184159a700 1 --2- 172.21.15.102:0/2040609665 >> v2:172.21.15.102:3300/0 conn(0x7f183c1593c0 0x7f183c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1841d9b700 1 -- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 msgr2=0x7f183c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1841d9b700 1 --2- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1841d9b700 1 -- 172.21.15.102:0/2040609665 >> v2:172.21.15.102:3300/0 conn(0x7f183c1593c0 msgr2=0x7f183c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1841d9b700 1 --2- 172.21.15.102:0/2040609665 >> v2:172.21.15.102:3300/0 conn(0x7f183c1593c0 0x7f183c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1841d9b700 1 -- 172.21.15.102:0/2040609665 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c06c7f0 con 0x7f183c153290 2023-03-03T05:46:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1841d9b700 1 --2- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c153290 0x7f183c158e80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f183c08cc70 tx=0x7f1838009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb3ac251306ead4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1840d99700 1 -- 172.21.15.102:0/2040609665 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838011830 con 0x7f183c153290 2023-03-03T05:46:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1840d99700 1 -- 172.21.15.102:0/2040609665 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1838011990 con 0x7f183c153290 2023-03-03T05:46:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.817+0000 7f1840d99700 1 -- 172.21.15.102:0/2040609665 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838011d20 con 0x7f183c153290 2023-03-03T05:46:07.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 -- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c153290 msgr2=0x7f183c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 --2- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c153290 0x7f183c158e80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f183c08cc70 tx=0x7f1838009bd0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 -- 172.21.15.102:0/2040609665 shutdown_connections 2023-03-03T05:46:07.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 --2- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 --2- 172.21.15.102:0/2040609665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c153290 0x7f183c158e80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 --2- 172.21.15.102:0/2040609665 >> v2:172.21.15.102:3300/0 conn(0x7f183c1593c0 0x7f183c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 -- 172.21.15.102:0/2040609665 >> 172.21.15.102:0/2040609665 conn(0x7f183c065fb0 msgr2=0x7f183c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:07.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 -- 172.21.15.102:0/2040609665 shutdown_connections 2023-03-03T05:46:07.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.818+0000 7f1843fff700 1 -- 172.21.15.102:0/2040609665 wait complete. 2023-03-03T05:46:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.820+0000 7f1843fff700 1 Processor -- start 2023-03-03T05:46:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.820+0000 7f1843fff700 1 -- start start 2023-03-03T05:46:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.821+0000 7f1843fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.821+0000 7f1843fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f183c153290 0x7f183c151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.821+0000 7f1843fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c1593c0 0x7f183c0b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.821+0000 7f1843fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0b2ac0 con 0x7f183c153290 2023-03-03T05:46:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.821+0000 7f1843fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0b4560 con 0x7f183c1593c0 2023-03-03T05:46:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.821+0000 7f1843fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c150a10 con 0x7f183c0cd8e0 2023-03-03T05:46:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f183c153290 0x7f183c151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f183c153290 0x7f183c151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54920/0 (socket says 172.21.15.102:54920) 2023-03-03T05:46:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 -- 172.21.15.102:0/1805111623 learned_addr learned my addr 172.21.15.102:0/1805111623 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:07.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f1841d9b700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184259c700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c1593c0 0x7f183c0b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 -- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 msgr2=0x7f183c0b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 -- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c1593c0 msgr2=0x7f183c0b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:07.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c1593c0 0x7f183c0b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 -- 172.21.15.102:0/1805111623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838009550 con 0x7f183c153290 2023-03-03T05:46:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.822+0000 7f184159a700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f183c153290 0x7f183c151860 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f182c00d800 tx=0x7f182c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.823+0000 7f18327fc700 1 -- 172.21.15.102:0/1805111623 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c0125f0 con 0x7f183c153290 2023-03-03T05:46:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.823+0000 7f18327fc700 1 -- 172.21.15.102:0/1805111623 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c012750 con 0x7f183c153290 2023-03-03T05:46:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.823+0000 7f18327fc700 1 -- 172.21.15.102:0/1805111623 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c012b00 con 0x7f183c153290 2023-03-03T05:46:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.823+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c0ccde0 con 0x7f183c153290 2023-03-03T05:46:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.823+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c15d160 con 0x7f183c153290 2023-03-03T05:46:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.824+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c0cd0c0 con 0x7f183c153290 2023-03-03T05:46:07.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.825+0000 7f18327fc700 1 -- 172.21.15.102:0/1805111623 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f182c012c60 con 0x7f183c153290 2023-03-03T05:46:07.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.826+0000 7f18327fc700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18200682f0 0x7f182006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:07.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.826+0000 7f18327fc700 1 -- 172.21.15.102:0/1805111623 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f182c085e00 con 0x7f183c153290 2023-03-03T05:46:07.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.826+0000 7f1841d9b700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18200682f0 0x7f182006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:07.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.827+0000 7f1841d9b700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18200682f0 0x7f182006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1838014040 tx=0x7f1838008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:07.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:07.830+0000 7f18327fc700 1 -- 172.21.15.102:0/1805111623 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f182c052570 con 0x7f183c153290 2023-03-03T05:46:08.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.070+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f183c04e6e0 con 0x7f183c153290 2023-03-03T05:46:08.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.071+0000 7f18327fc700 1 -- 172.21.15.102:0/1805111623 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6173 (secure 0 0 0) 0x7f182c055b90 con 0x7f183c153290 2023-03-03T05:46:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:08.074 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:45:41.326836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4315},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4315":{"gid":4315,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6837/1262584332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":1262584332},{"type":"v1","addr":"172.21.15.102:6837","nonce":1262584332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:46:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18200682f0 msgr2=0x7f182006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18200682f0 0x7f182006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1838014040 tx=0x7f1838008030 comp rx=0 tx=0).stop 2023-03-03T05:46:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f183c153290 msgr2=0x7f183c151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f183c153290 0x7f183c151860 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f182c00d800 tx=0x7f182c00db10 comp rx=0 tx=0).stop 2023-03-03T05:46:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 shutdown_connections 2023-03-03T05:46:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18200682f0 0x7f182006a7a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f183c0cd8e0 0x7f183c0b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f183c1593c0 0x7f183c0b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 --2- 172.21.15.102:0/1805111623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f183c153290 0x7f183c151860 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 >> 172.21.15.102:0/1805111623 conn(0x7f183c065fb0 msgr2=0x7f183c15ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 shutdown_connections 2023-03-03T05:46:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.074+0000 7f1843fff700 1 -- 172.21.15.102:0/1805111623 wait complete. 2023-03-03T05:46:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2023-03-03T05:46:08.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T05:46:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.350+0000 7f586e7ff700 1 Processor -- start 2023-03-03T05:46:08.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.350+0000 7f586e7ff700 1 -- start start 2023-03-03T05:46:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.351+0000 7f586e7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5868153290 0x7f5868158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.351+0000 7f586e7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58680cd8e0 0x7f58680ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.351+0000 7f586e7ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5868153660 con 0x7f58681593c0 2023-03-03T05:46:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.351+0000 7f586e7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f586806c360 con 0x7f58680cd8e0 2023-03-03T05:46:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.351+0000 7f586e7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586815cea0 con 0x7f5868153290 2023-03-03T05:46:08.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f58677fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f58681593c0 0x7f58680cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50704/0 (socket says 172.21.15.102:50704) 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f5867fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5868153290 0x7f5868158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f58677fe700 1 -- 172.21.15.102:0/1251705909 learned_addr learned my addr 172.21.15.102:0/1251705909 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f586cd9c700 1 --2- 172.21.15.102:0/1251705909 >> v2:172.21.15.172:3300/0 conn(0x7f58680cd8e0 0x7f58680ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f5867fff700 1 -- 172.21.15.102:0/1251705909 >> v2:172.21.15.172:3300/0 conn(0x7f58680cd8e0 msgr2=0x7f58680ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f5867fff700 1 --2- 172.21.15.102:0/1251705909 >> v2:172.21.15.172:3300/0 conn(0x7f58680cd8e0 0x7f58680ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f5867fff700 1 -- 172.21.15.102:0/1251705909 >> v1:172.21.15.102:6789/0 conn(0x7f58681593c0 legacy=0x7f58680cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f5867fff700 1 -- 172.21.15.102:0/1251705909 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586806c7f0 con 0x7f5868153290 2023-03-03T05:46:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.352+0000 7f5867fff700 1 --2- 172.21.15.102:0/1251705909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5868153290 0x7f5868158e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5858009840 tx=0x7f5858009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fccb4a1d19d7be0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f5866ffd700 1 -- 172.21.15.102:0/1251705909 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858011830 con 0x7f5868153290 2023-03-03T05:46:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f5866ffd700 1 -- 172.21.15.102:0/1251705909 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5858011990 con 0x7f5868153290 2023-03-03T05:46:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f5866ffd700 1 -- 172.21.15.102:0/1251705909 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858011d20 con 0x7f5868153290 2023-03-03T05:46:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 -- 172.21.15.102:0/1251705909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5868153290 msgr2=0x7f5868158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 --2- 172.21.15.102:0/1251705909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5868153290 0x7f5868158e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5858009840 tx=0x7f5858009b50 comp rx=0 tx=0).stop 2023-03-03T05:46:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 -- 172.21.15.102:0/1251705909 shutdown_connections 2023-03-03T05:46:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 --2- 172.21.15.102:0/1251705909 >> v2:172.21.15.172:3300/0 conn(0x7f58680cd8e0 0x7f58680ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 --2- 172.21.15.102:0/1251705909 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5868153290 0x7f5868158e80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 -- 172.21.15.102:0/1251705909 >> 172.21.15.102:0/1251705909 conn(0x7f5868065fb0 msgr2=0x7f58680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 -- 172.21.15.102:0/1251705909 shutdown_connections 2023-03-03T05:46:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.353+0000 7f586e7ff700 1 -- 172.21.15.102:0/1251705909 wait complete. 2023-03-03T05:46:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.356+0000 7f586e7ff700 1 Processor -- start 2023-03-03T05:46:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.356+0000 7f586e7ff700 1 -- start start 2023-03-03T05:46:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f586e7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58680cd8e0 0x7f58681e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f586e7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5868153290 0x7f58681e4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f586e7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58681593c0 0x7f58680c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f586e7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681ef690 con 0x7f5868153290 2023-03-03T05:46:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f586e7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681ee7c0 con 0x7f58680cd8e0 2023-03-03T05:46:08.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f586e7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58681eea80 con 0x7f58681593c0 2023-03-03T05:46:08.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f586cd9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58681593c0 0x7f58680c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:08.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f5867fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58680cd8e0 0x7f58681e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:08.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f58677fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5868153290 0x7f58681e4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:08.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f5867fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58680cd8e0 0x7f58681e3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40286/0 (socket says 172.21.15.102:40286) 2023-03-03T05:46:08.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.357+0000 7f5867fff700 1 -- 172.21.15.102:0/278614550 learned_addr learned my addr 172.21.15.102:0/278614550 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:46:08.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f586cd9c700 1 -- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58680cd8e0 msgr2=0x7f58681e3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f586cd9c700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58680cd8e0 0x7f58681e3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f586cd9c700 1 -- 172.21.15.102:0/278614550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5868153290 msgr2=0x7f58681e4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f586cd9c700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5868153290 0x7f58681e4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f586cd9c700 1 -- 172.21.15.102:0/278614550 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580094a0 con 0x7f58681593c0 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f5867fff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58680cd8e0 0x7f58681e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f586cd9c700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58681593c0 0x7f58680c7550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f586000eba0 tx=0x7f586000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f5864ff9700 1 -- 172.21.15.102:0/278614550 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860009a80 con 0x7f58681593c0 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f5864ff9700 1 -- 172.21.15.102:0/278614550 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586000f040 con 0x7f58681593c0 2023-03-03T05:46:08.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f5864ff9700 1 -- 172.21.15.102:0/278614550 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586000cc40 con 0x7f58681593c0 2023-03-03T05:46:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.358+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58680ccde0 con 0x7f58681593c0 2023-03-03T05:46:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.359+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586815d160 con 0x7f58681593c0 2023-03-03T05:46:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.359+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58680cd0c0 con 0x7f58681593c0 2023-03-03T05:46:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.361+0000 7f5864ff9700 1 -- 172.21.15.102:0/278614550 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5860014480 con 0x7f58681593c0 2023-03-03T05:46:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.362+0000 7f5864ff9700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58540682f0 0x7f585406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:46:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.362+0000 7f5864ff9700 1 -- 172.21.15.102:0/278614550 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f58600868c0 con 0x7f58681593c0 2023-03-03T05:46:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.362+0000 7f5867fff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58540682f0 0x7f585406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:46:08.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.363+0000 7f5867fff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58540682f0 0x7f585406a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5858009b90 tx=0x7f5858008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:46:08.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.366+0000 7f5864ff9700 1 -- 172.21.15.102:0/278614550 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5860052fb0 con 0x7f58681593c0 2023-03-03T05:46:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.558+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58680c7350 con 0x7f58681593c0 2023-03-03T05:46:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.560+0000 7f5864ff9700 1 -- 172.21.15.102:0/278614550 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f5860018090 con 0x7f58681593c0 2023-03-03T05:46:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:46:08.564 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"fsid":"56ef27ff-a10a-47b1-b557-af27e3b0b6e2","created":"2023-03-03T05:45:08.291544+0000","modified":"2023-03-03T05:45:36.839870+0000","last_up_change":"2023-03-03T05:45:21.933331+0000","last_in_change":"2023-03-03T05:45:11.075336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T05:45:20.032740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T05:45:27.930463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T05:45:33.613145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T05:45:34.842860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d23900ad-c97a-46fb-b261-8cf9f0f56dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":109067},{"type":"v1","addr":"172.21.15.102:6817","nonce":109067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6819","nonce":109067},{"type":"v1","addr":"172.21.15.102:6820","nonce":109067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6825","nonce":109067},{"type":"v1","addr":"172.21.15.102:6827","nonce":109067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":109067},{"type":"v1","addr":"172.21.15.102:6823","nonce":109067}]},"public_addr":"172.21.15.102:6817/109067","cluster_addr":"172.21.15.102:6820/109067","heartbeat_back_addr":"172.21.15.102:6827/109067","heartbeat_front_addr":"172.21.15.102:6823/109067","state":["exists","up"]},{"osd":1,"uuid":"ad558fcb-2250-48d2-8a46-7bce17eeb6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":109064},{"type":"v1","addr":"172.21.15.102:6809","nonce":109064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":109064},{"type":"v1","addr":"172.21.15.102:6811","nonce":109064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":109064},{"type":"v1","addr":"172.21.15.102:6815","nonce":109064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":109064},{"type":"v1","addr":"172.21.15.102:6813","nonce":109064}]},"public_addr":"172.21.15.102:6809/109064","cluster_addr":"172.21.15.102:6811/109064","heartbeat_back_addr":"172.21.15.102:6815/109064","heartbeat_front_addr":"172.21.15.102:6813/109064","state":["exists","up"]},{"osd":2,"uuid":"086ff50f-40e5-4583-83eb-bc1cded78708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":109068},{"type":"v1","addr":"172.21.15.102:6821","nonce":109068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":109068},{"type":"v1","addr":"172.21.15.102:6826","nonce":109068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":109068},{"type":"v1","addr":"172.21.15.102:6831","nonce":109068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":109068},{"type":"v1","addr":"172.21.15.102:6829","nonce":109068}]},"public_addr":"172.21.15.102:6821/109068","cluster_addr":"172.21.15.102:6826/109068","heartbeat_back_addr":"172.21.15.102:6831/109068","heartbeat_front_addr":"172.21.15.102:6829/109068","state":["exists","up"]},{"osd":3,"uuid":"35415ce3-4ea3-4de5-921b-cf70bacdc9f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":109066},{"type":"v1","addr":"172.21.15.102:6801","nonce":109066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":109066},{"type":"v1","addr":"172.21.15.102:6803","nonce":109066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":109066},{"type":"v1","addr":"172.21.15.102:6807","nonce":109066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":109066},{"type":"v1","addr":"172.21.15.102:6805","nonce":109066}]},"public_addr":"172.21.15.102:6801/109066","cluster_addr":"172.21.15.102:6803/109066","heartbeat_back_addr":"172.21.15.102:6807/109066","heartbeat_front_addr":"172.21.15.102:6805/109066","state":["exists","up"]},{"osd":4,"uuid":"873b86f0-b407-479c-9074-12767a5419cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114772},{"type":"v1","addr":"172.21.15.172:6817","nonce":114772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114772},{"type":"v1","addr":"172.21.15.172:6819","nonce":114772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114772},{"type":"v1","addr":"172.21.15.172:6823","nonce":114772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114772},{"type":"v1","addr":"172.21.15.172:6821","nonce":114772}]},"public_addr":"172.21.15.172:6817/114772","cluster_addr":"172.21.15.172:6819/114772","heartbeat_back_addr":"172.21.15.172:6823/114772","heartbeat_front_addr":"172.21.15.172:6821/114772","state":["exists","up"]},{"osd":5,"uuid":"312da3e9-4358-4b98-bb71-8ef9d455a191","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":114774},{"type":"v1","addr":"172.21.15.172:6825","nonce":114774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":114774},{"type":"v1","addr":"172.21.15.172:6827","nonce":114774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":114774},{"type":"v1","addr":"172.21.15.172:6831","nonce":114774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":114774},{"type":"v1","addr":"172.21.15.172:6829","nonce":114774}]},"public_addr":"172.21.15.172:6825/114774","cluster_addr":"172.21.15.172:6827/114774","heartbeat_back_addr":"172.21.15.172:6831/114774","heartbeat_front_addr":"172.21.15.172:6829/114774","state":["exists","up"]},{"osd":6,"uuid":"c9e44237-2654-4425-95bd-31ee7706f4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114763},{"type":"v1","addr":"172.21.15.172:6802","nonce":114763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114763},{"type":"v1","addr":"172.21.15.172:6806","nonce":114763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114763},{"type":"v1","addr":"172.21.15.172:6814","nonce":114763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114763},{"type":"v1","addr":"172.21.15.172:6810","nonce":114763}]},"public_addr":"172.21.15.172:6802/114763","cluster_addr":"172.21.15.172:6806/114763","heartbeat_back_addr":"172.21.15.172:6814/114763","heartbeat_front_addr":"172.21.15.172:6810/114763","state":["exists","up"]},{"osd":7,"uuid":"c2898a5d-7f10-454d-b3b7-22034f852296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":114764},{"type":"v1","addr":"172.21.15.172:6803","nonce":114764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":114764},{"type":"v1","addr":"172.21.15.172:6807","nonce":114764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":114764},{"type":"v1","addr":"172.21.15.172:6815","nonce":114764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":114764},{"type":"v1","addr":"172.21.15.172:6811","nonce":114764}]},"public_addr":"172.21.15.172:6803/114764","cluster_addr":"172.21.15.172:6807/114764","heartbeat_back_addr":"172.21.15.172:6815/114764","heartbeat_front_addr":"172.21.15.172:6811/114764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.897025+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.831715+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.901466+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.885857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.298731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.261041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.267519+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T05:45:16.265555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T05:46:08.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.562+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58540682f0 msgr2=0x7f585406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.562+0000 7f586e7ff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58540682f0 0x7f585406a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5858009b90 tx=0x7f5858008030 comp rx=0 tx=0).stop 2023-03-03T05:46:08.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.562+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58681593c0 msgr2=0x7f58680c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:46:08.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.562+0000 7f586e7ff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58681593c0 0x7f58680c7550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f586000eba0 tx=0x7f586000eeb0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 shutdown_connections 2023-03-03T05:46:08.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58540682f0 0x7f585406a7a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58681593c0 0x7f58680c7550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58680cd8e0 0x7f58681e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 --2- 172.21.15.102:0/278614550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5868153290 0x7f58681e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:46:08.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 >> 172.21.15.102:0/278614550 conn(0x7f5868065fb0 msgr2=0x7f586815be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:46:08.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 shutdown_connections 2023-03-03T05:46:08.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:46:08.563+0000 7f586e7ff700 1 -- 172.21.15.102:0/278614550 wait complete. 2023-03-03T05:46:08.594 DEBUG:tasks.mds_thrash:Yielding 2023-03-03T05:46:08.594 INFO:teuthology.run_tasks:Running task workunit... 2023-03-03T05:46:08.627 INFO:tasks.workunit:Pulling workunits from ref 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T05:46:08.628 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-03-03T05:46:08.628 INFO:tasks.workunit:timeout=6h 2023-03-03T05:46:08.629 INFO:tasks.workunit:cleanup=True 2023-03-03T05:46:08.629 DEBUG:teuthology.orchestra.run.smithi102:> stat -- /home/ubuntu/cephtest/mnt.0 2023-03-03T05:46:08.630 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-03-03T05:46:08.630 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-03-03T05:46:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-03-03T05:46:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-03-03T05:46:08.699 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 2ah/42d Inode: 1 Links: 2 2023-03-03T05:46:08.699 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T05:46:08.699 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:cephfs_t:s0 2023-03-03T05:46:08.700 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-03-03T05:46:08.700 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-03-03 05:45:40.842883416 +0000 2023-03-03T05:46:08.700 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-03-03 05:45:57.425191432 +0000 2023-03-03T05:46:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2023-03-03 05:45:40.842883416 +0000 2023-03-03T05:46:08.700 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-03-03T05:46:08.700 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-03-03T05:46:08.806 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T05:46:08.878 INFO:tasks.workunit.client.0.smithi102.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-03-03T05:46:26.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:46:26.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:46:56.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:46:56.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:47:01.300 INFO:tasks.workunit.client.0.smithi102.stderr:Updating files: 55% (6401/11620) Updating files: 56% (6508/11620) Updating files: 57% (6624/11620) Updating files: 58% (6740/11620) Updating files: 59% (6856/11620) Updating files: 60% (6972/11620) Updating files: 61% (7089/11620) Updating files: 62% (7205/11620) Updating files: 63% (7321/11620) Updating files: 64% (7437/11620) Updating files: 65% (7553/11620) Updating files: 66% (7670/11620) Updating files: 67% (7786/11620) Updating files: 68% (7902/11620) Updating files: 69% (8018/11620) Updating files: 70% (8134/11620) Updating files: 71% (8251/11620) Updating files: 72% (8367/11620) Updating files: 73% (8483/11620) Updating files: 74% (8599/11620) Updating files: 75% (8715/11620) Updating files: 76% (8832/11620) Updating files: 77% (8948/11620) Updating files: 78% (9064/11620) Updating files: 79% (9180/11620) Updating files: 80% (9296/11620) Updating files: 81% (9413/11620) Updating files: 82% (9529/11620) Updating files: 82% (9573/11620) Updating files: 83% (9645/11620) Updating files: 84% (9761/11620) Updating files: 85% (9877/11620) Updating files: 86% (9994/11620) Updating files: 87% (10110/11620) Updating files: 88% (10226/11620) Updating files: 89% (10342/11620) Updating files: 90% (10458/11620) Updating files: 91% (10575/11620) Updating files: 92% (10691/11620) Updating files: 93% (10807/11620) Updating files: 94% (10923/11620) Updating files: 95% (11039/11620) Updating files: 96% (11156/11620) Updating files: 97% (11272/11620) Updating files: 98% (11388/11620) Updating files: 99% (11504/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-03-03T05:47:01.733 INFO:tasks.workunit.client.0.smithi102.stderr:Note: switching to '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'. 2023-03-03T05:47:01.733 INFO:tasks.workunit.client.0.smithi102.stderr: 2023-03-03T05:47:01.734 INFO:tasks.workunit.client.0.smithi102.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-03-03T05:47:01.734 INFO:tasks.workunit.client.0.smithi102.stderr:changes and commit them, and you can discard any commits you make in this 2023-03-03T05:47:01.734 INFO:tasks.workunit.client.0.smithi102.stderr:state without impacting any branches by switching back to a branch. 2023-03-03T05:47:01.735 INFO:tasks.workunit.client.0.smithi102.stderr: 2023-03-03T05:47:01.735 INFO:tasks.workunit.client.0.smithi102.stderr:If you want to create a new branch to retain commits you create, you may 2023-03-03T05:47:01.735 INFO:tasks.workunit.client.0.smithi102.stderr:do so (now or later) by using -c with the switch command. Example: 2023-03-03T05:47:01.735 INFO:tasks.workunit.client.0.smithi102.stderr: 2023-03-03T05:47:01.736 INFO:tasks.workunit.client.0.smithi102.stderr: git switch -c 2023-03-03T05:47:01.736 INFO:tasks.workunit.client.0.smithi102.stderr: 2023-03-03T05:47:01.736 INFO:tasks.workunit.client.0.smithi102.stderr:Or undo this operation with: 2023-03-03T05:47:01.736 INFO:tasks.workunit.client.0.smithi102.stderr: 2023-03-03T05:47:01.736 INFO:tasks.workunit.client.0.smithi102.stderr: git switch - 2023-03-03T05:47:01.737 INFO:tasks.workunit.client.0.smithi102.stderr: 2023-03-03T05:47:01.737 INFO:tasks.workunit.client.0.smithi102.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-03-03T05:47:01.737 INFO:tasks.workunit.client.0.smithi102.stderr: 2023-03-03T05:47:01.737 INFO:tasks.workunit.client.0.smithi102.stderr:HEAD is now at 472aaef4ec3 Merge PR #45224 into wip-vshankar-testing-20230303.023823 2023-03-03T05:47:01.754 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-03-03T05:47:01.831 INFO:tasks.workunit.client.0.smithi102.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-03-03T05:47:01.838 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-03T05:47:01.838 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-03-03T05:47:02.470 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-03-03T05:47:02.588 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-03-03T05:47:02.673 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-03T05:47:02.679 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-03T05:47:02.679 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-03-03T05:47:02.763 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-03T05:47:02.775 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-03-03T05:47:02.775 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-03-03T05:47:02.844 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-03-03T05:47:02.847 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-03-03T05:47:02.847 DEBUG:teuthology.orchestra.run.smithi102:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-03-03T05:47:02.935 INFO:tasks.workunit.client.0.smithi102.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-03-03T05:47:05.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ mkdir foo 2023-03-03T05:47:05.286 INFO:tasks.workunit.client.0.smithi102.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-03-03T05:47:05.424 INFO:tasks.workunit.client.0.smithi102.stderr:+ mkdir foo/.snap/barsnap 2023-03-03T05:47:05.436 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-03-03T05:47:05.445 INFO:tasks.workunit.client.0.smithi102.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-03-03T05:47:05.800 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'passed: files are identical' 2023-03-03T05:47:05.801 INFO:tasks.workunit.client.0.smithi102.stdout:passed: files are identical 2023-03-03T05:47:05.801 INFO:tasks.workunit.client.0.smithi102.stderr:+ rmdir foo/.snap/barsnap 2023-03-03T05:47:05.918 INFO:tasks.workunit.client.0.smithi102.stdout:OK 2023-03-03T05:47:05.919 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo OK 2023-03-03T05:47:05.920 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-03-03T05:47:05.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-03-03T05:47:06.018 INFO:tasks.ceph.mds.e.smithi102.stderr:2023-03-03T05:47:06.015+0000 7f6899bf6700 -1 mds.0.cache.den(0x601 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray1/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x5580decacc00) (dversion lock w=1 last_client=4523) pv=21 v=20 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x5580decf3180] 2023-03-03T05:47:06.018 INFO:tasks.ceph.mds.e.smithi102.stderr:2023-03-03T05:47:06.015+0000 7f6899bf6700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x601is fatal because it is a system directory for this rank 2023-03-03T05:47:06.303 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T05:47:06.301+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T05:47:06.304 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T05:47:06.301+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2023-03-03T05:47:06.347 INFO:tasks.ceph.mds.e.smithi102.stderr: -14> 2023-03-03T05:47:06.015+0000 7f6899bf6700 -1 mds.0.cache.den(0x601 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray1/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x5580decacc00) (dversion lock w=1 last_client=4523) pv=21 v=20 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x5580decf3180] 2023-03-03T05:47:06.347 INFO:tasks.ceph.mds.e.smithi102.stderr: -12> 2023-03-03T05:47:06.015+0000 7f6899bf6700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x601is fatal because it is a system directory for this rank 2023-03-03T05:47:06.407 INFO:tasks.ceph.mds.e.smithi102.stdout:starting mds.e at 2023-03-03T05:47:26.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:47:26.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:47:52.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:47:52.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.895+0000 7f1ad7fff700 1 Processor -- start 2023-03-03T05:47:52.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.895+0000 7f1ad7fff700 1 -- start start 2023-03-03T05:47:52.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.896+0000 7f1ad7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:47:52.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.896+0000 7f1ad7fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad0153660 con 0x7f1ad00cd390 2023-03-03T05:47:52.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.896+0000 7f1ad7fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad006c360 con 0x7f1ad0153290 2023-03-03T05:47:52.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.896+0000 7f1ad7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad015cea0 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad5d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1ad0153290 0x7f1ad0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:43820/0 (socket says 172.21.15.102:43820) 2023-03-03T05:47:52.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad5d9b700 1 -- 172.21.15.102:0/1322177793 learned_addr learned my addr 172.21.15.102:0/1322177793 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:47:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad559a700 1 --2- 172.21.15.102:0/1322177793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:47:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad4d99700 1 -- 172.21.15.102:0/1322177793 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789433164 0 0) 0x7f1ad006c360 con 0x7f1ad0153290 2023-03-03T05:47:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad4d99700 1 -- 172.21.15.102:0/1322177793 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1abc0035b0 con 0x7f1ad0153290 2023-03-03T05:47:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad4d99700 1 -- 172.21.15.102:0/1322177793 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060718739 0 0) 0x7f1ad0153660 con 0x7f1ad00cd390 2023-03-03T05:47:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad4d99700 1 -- 172.21.15.102:0/1322177793 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad006c360 con 0x7f1ad00cd390 2023-03-03T05:47:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad559a700 1 -- 172.21.15.102:0/1322177793 >> v1:172.21.15.172:6789/0 conn(0x7f1ad0153290 legacy=0x7f1ad0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:47:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad559a700 1 -- 172.21.15.102:0/1322177793 >> v1:172.21.15.102:6789/0 conn(0x7f1ad00cd390 legacy=0x7f1ad00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:47:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.897+0000 7f1ad559a700 1 -- 172.21.15.102:0/1322177793 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad006c7f0 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.898+0000 7f1ad559a700 1 --2- 172.21.15.102:0/1322177793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad00cde20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009870 tx=0x7f1ac0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66b18fde0bf3e1c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:47:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.898+0000 7f1ad4d99700 1 -- 172.21.15.102:0/1322177793 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0011830 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.898+0000 7f1ad4d99700 1 -- 172.21.15.102:0/1322177793 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac0011990 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.898+0000 7f1ad4d99700 1 -- 172.21.15.102:0/1322177793 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0011d20 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.898+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1322177793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 msgr2=0x7f1ad00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:47:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.898+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1322177793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad00cde20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009870 tx=0x7f1ac0009b80 comp rx=0 tx=0).stop 2023-03-03T05:47:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.898+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1322177793 shutdown_connections 2023-03-03T05:47:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.899+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1322177793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad00cde20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:47:52.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.899+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1322177793 >> 172.21.15.102:0/1322177793 conn(0x7f1ad0065fb0 msgr2=0x7f1ad00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:47:52.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.899+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1322177793 shutdown_connections 2023-03-03T05:47:52.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.899+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1322177793 wait complete. 2023-03-03T05:47:52.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.901+0000 7f1ad7fff700 1 Processor -- start 2023-03-03T05:47:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.901+0000 7f1ad7fff700 1 -- start start 2023-03-03T05:47:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad00cd390 0x7f1ad01e2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:47:52.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad01e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:47:52.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad0153290 0x7f1ad01f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:47:52.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad7fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad01e3d90 con 0x7f1ad00cd390 2023-03-03T05:47:52.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad01e5070 con 0x7f1ad0153290 2023-03-03T05:47:52.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad00c6a30 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad5d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad00cd390 0x7f1ad01e2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:47:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad559a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad01e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:47:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad0153290 0x7f1ad01f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:47:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad5d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad00cd390 0x7f1ad01e2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43678/0 (socket says 172.21.15.102:43678) 2023-03-03T05:47:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.902+0000 7f1ad5d9b700 1 -- 172.21.15.102:0/1600414372 learned_addr learned my addr 172.21.15.102:0/1600414372 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:47:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad559a700 1 -- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad0153290 msgr2=0x7f1ad01f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:47:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad559a700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad0153290 0x7f1ad01f20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:47:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad559a700 1 -- 172.21.15.102:0/1600414372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad00cd390 msgr2=0x7f1ad01e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:47:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad559a700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad00cd390 0x7f1ad01e2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:47:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad559a700 1 -- 172.21.15.102:0/1600414372 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00094a0 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad659c700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad0153290 0x7f1ad01f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:47:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad5d9b700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad00cd390 0x7f1ad01e2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:47:52.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad559a700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad01e28b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0016000 tx=0x7f1ac0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:47:52.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ac67fc700 1 -- 172.21.15.102:0/1600414372 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac001a6e0 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ac67fc700 1 -- 172.21.15.102:0/1600414372 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac001a860 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ac67fc700 1 -- 172.21.15.102:0/1600414372 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac001ac10 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.903+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad00ccde0 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.904+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad015d160 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.905+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad00cd0c0 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.906+0000 7f1ac67fc700 1 -- 172.21.15.102:0/1600414372 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1ac000f050 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.907+0000 7f1ac67fc700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab40682f0 0x7f1ab406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:47:52.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.907+0000 7f1ac67fc700 1 -- 172.21.15.102:0/1600414372 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1ac0086b30 con 0x7f1ad00cd8e0 2023-03-03T05:47:52.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.908+0000 7f1ad5d9b700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab40682f0 0x7f1ab406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:47:52.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.908+0000 7f1ad5d9b700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab40682f0 0x7f1ab406a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1acc000f60 tx=0x7f1acc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:47:52.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:52.912+0000 7f1ac67fc700 1 -- 172.21.15.102:0/1600414372 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ac0053240 con 0x7f1ad00cd8e0 2023-03-03T05:47:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.147+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ad004e6e0 con 0x7f1ad00cd8e0 2023-03-03T05:47:53.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.148+0000 7f1ac67fc700 1 -- 172.21.15.102:0/1600414372 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1ac0015090 con 0x7f1ad00cd8e0 2023-03-03T05:47:53.151 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:47:53.152 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:47:53.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab40682f0 msgr2=0x7f1ab406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:47:53.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab40682f0 0x7f1ab406a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1acc000f60 tx=0x7f1acc009040 comp rx=0 tx=0).stop 2023-03-03T05:47:53.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 msgr2=0x7f1ad01e28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:47:53.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad01e28b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0016000 tx=0x7f1ac0004750 comp rx=0 tx=0).stop 2023-03-03T05:47:53.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 shutdown_connections 2023-03-03T05:47:53.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad0153290 0x7f1ad01f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:47:53.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad00cd390 0x7f1ad01e2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:47:53.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab40682f0 0x7f1ab406a7a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:47:53.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 --2- 172.21.15.102:0/1600414372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad00cd8e0 0x7f1ad01e28b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:47:53.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 >> 172.21.15.102:0/1600414372 conn(0x7f1ad0065fb0 msgr2=0x7f1ad00cb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:47:53.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 shutdown_connections 2023-03-03T05:47:53.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:47:53.151+0000 7f1ad7fff700 1 -- 172.21.15.102:0/1600414372 wait complete. 2023-03-03T05:47:53.156 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:47:53.182 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-03-03T05:47:57.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:47:57.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:48:27.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:48:27.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:48:57.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:48:57.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:49:27.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:49:27.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:49:46.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:49:46.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.435+0000 7feb621ff700 1 Processor -- start 2023-03-03T05:49:46.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.435+0000 7feb621ff700 1 -- start start 2023-03-03T05:49:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.436+0000 7feb621ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb5c0cd8e0 0x7feb5c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:49:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.436+0000 7feb621ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c153660 con 0x7feb5c153290 2023-03-03T05:49:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.436+0000 7feb621ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c06c360 con 0x7feb5c0cd390 2023-03-03T05:49:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.436+0000 7feb621ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5c15cea0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb5c0cd8e0 0x7feb5c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:49:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7feb5c153290 0x7feb5c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40486/0 (socket says 172.21.15.102:40486) 2023-03-03T05:49:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5b7fe700 1 -- 172.21.15.102:0/2327408658 learned_addr learned my addr 172.21.15.102:0/2327408658 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:49:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5affd700 1 -- 172.21.15.102:0/2327408658 >> v1:172.21.15.172:6789/0 conn(0x7feb5c0cd390 legacy=0x7feb5c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:49:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5affd700 1 -- 172.21.15.102:0/2327408658 >> v1:172.21.15.102:6789/0 conn(0x7feb5c153290 legacy=0x7feb5c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:49:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5a7fc700 1 -- 172.21.15.102:0/2327408658 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726181734 0 0) 0x7feb5c153660 con 0x7feb5c153290 2023-03-03T05:49:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5affd700 1 -- 172.21.15.102:0/2327408658 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c06c7f0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.437+0000 7feb5affd700 1 --2- 172.21.15.102:0/2327408658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c0cd8e0 0x7feb5c0cde20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb50009840 tx=0x7feb50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b32fe215504e91 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:49:46.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb5a7fc700 1 -- 172.21.15.102:0/2327408658 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50011830 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb5a7fc700 1 -- 172.21.15.102:0/2327408658 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb50011990 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb5a7fc700 1 -- 172.21.15.102:0/2327408658 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50011d20 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb621ff700 1 -- 172.21.15.102:0/2327408658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c0cd8e0 msgr2=0x7feb5c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:49:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb621ff700 1 --2- 172.21.15.102:0/2327408658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c0cd8e0 0x7feb5c0cde20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb50009840 tx=0x7feb50009b50 comp rx=0 tx=0).stop 2023-03-03T05:49:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb621ff700 1 -- 172.21.15.102:0/2327408658 shutdown_connections 2023-03-03T05:49:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb621ff700 1 --2- 172.21.15.102:0/2327408658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c0cd8e0 0x7feb5c0cde20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:49:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb621ff700 1 -- 172.21.15.102:0/2327408658 >> 172.21.15.102:0/2327408658 conn(0x7feb5c065fb0 msgr2=0x7feb5c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:49:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb621ff700 1 -- 172.21.15.102:0/2327408658 shutdown_connections 2023-03-03T05:49:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.438+0000 7feb621ff700 1 -- 172.21.15.102:0/2327408658 wait complete. 2023-03-03T05:49:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.440+0000 7feb621ff700 1 Processor -- start 2023-03-03T05:49:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.440+0000 7feb621ff700 1 -- start start 2023-03-03T05:49:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.441+0000 7feb621ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb5c0cd390 0x7feb5c1e2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:49:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.441+0000 7feb621ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb5c0cd8e0 0x7feb5c1efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:49:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.441+0000 7feb621ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c153290 0x7feb5c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:49:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.441+0000 7feb621ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c150920 con 0x7feb5c153290 2023-03-03T05:49:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.441+0000 7feb621ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c150310 con 0x7feb5c0cd390 2023-03-03T05:49:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.441+0000 7feb621ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c150db0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb5c0cd390 0x7feb5c1e2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:49:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb5c0cd8e0 0x7feb5c1efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:49:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c153290 0x7feb5c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:49:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb5c0cd8e0 0x7feb5c1efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48794/0 (socket says 172.21.15.102:48794) 2023-03-03T05:49:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 -- 172.21.15.102:0/803090097 learned_addr learned my addr 172.21.15.102:0/803090097 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:49:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb5c0cd390 0x7feb5c1e2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57294/0 (socket says 172.21.15.102:57294) 2023-03-03T05:49:46.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 -- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb5c0cd390 msgr2=0x7feb5c1e2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:49:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb5c0cd390 0x7feb5c1e2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:49:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 -- 172.21.15.102:0/803090097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c153290 msgr2=0x7feb5c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:49:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c153290 0x7feb5c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:49:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5affd700 1 -- 172.21.15.102:0/803090097 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500094a0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5bfff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c153290 0x7feb5c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:49:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.442+0000 7feb5b7fe700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb5c0cd390 0x7feb5c1e2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:49:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.443+0000 7feb5affd700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb5c0cd8e0 0x7feb5c1efac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb500090b0 tx=0x7feb50004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:49:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.443+0000 7feb43fff700 1 -- 172.21.15.102:0/803090097 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5001a7e0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.443+0000 7feb43fff700 1 -- 172.21.15.102:0/803090097 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5001a940 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.443+0000 7feb43fff700 1 -- 172.21.15.102:0/803090097 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5001acb0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.443+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c0ccde0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.444+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c15d160 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.445+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c0cd0c0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.448+0000 7feb43fff700 1 -- 172.21.15.102:0/803090097 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb5002b030 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.449+0000 7feb43fff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb48068320 0x7feb4806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:49:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.449+0000 7feb43fff700 1 -- 172.21.15.102:0/803090097 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7feb50087490 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.449+0000 7feb5b7fe700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb48068320 0x7feb4806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:49:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.450+0000 7feb5b7fe700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb48068320 0x7feb4806a7d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feb4c007c40 tx=0x7feb4c007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:49:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.451+0000 7feb43fff700 1 -- 172.21.15.102:0/803090097 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb50053cd0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.687+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb5c150a90 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.689+0000 7feb43fff700 1 -- 172.21.15.102:0/803090097 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7feb500572f0 con 0x7feb5c0cd8e0 2023-03-03T05:49:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:49:46.692 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:49:46.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb48068320 msgr2=0x7feb4806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:49:46.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb48068320 0x7feb4806a7d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feb4c007c40 tx=0x7feb4c007b90 comp rx=0 tx=0).stop 2023-03-03T05:49:46.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb5c0cd8e0 msgr2=0x7feb5c1efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:49:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb5c0cd8e0 0x7feb5c1efac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb500090b0 tx=0x7feb50004750 comp rx=0 tx=0).stop 2023-03-03T05:49:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 shutdown_connections 2023-03-03T05:49:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb5c153290 0x7feb5c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:49:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feb5c0cd390 0x7feb5c1e2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:49:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feb48068320 0x7feb4806a7d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:49:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 --2- 172.21.15.102:0/803090097 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feb5c0cd8e0 0x7feb5c1efac0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:49:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 >> 172.21.15.102:0/803090097 conn(0x7feb5c065fb0 msgr2=0x7feb5c156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:49:46.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.691+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 shutdown_connections 2023-03-03T05:49:46.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:49:46.692+0000 7feb621ff700 1 -- 172.21.15.102:0/803090097 wait complete. 2023-03-03T05:49:46.695 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:49:46.720 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-03-03T05:49:58.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:49:58.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:50:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T05:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T05:50:28.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:50:28.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:50:58.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:50:58.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:51:09.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:51:09.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.970+0000 7fda7fdff700 1 Processor -- start 2023-03-03T05:51:09.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.970+0000 7fda7fdff700 1 -- start start 2023-03-03T05:51:09.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.971+0000 7fda7fdff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda780cd390 0x7fda780ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:09.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.971+0000 7fda7fdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda78153660 con 0x7fda78153290 2023-03-03T05:51:09.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7fdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7806c360 con 0x7fda780cd8e0 2023-03-03T05:51:09.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7fdff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7815cea0 con 0x7fda780cd390 2023-03-03T05:51:09.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7db9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fda78153290 0x7fda78158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35840/0 (socket says 172.21.15.102:35840) 2023-03-03T05:51:09.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7db9b700 1 -- 172.21.15.102:0/4069944217 learned_addr learned my addr 172.21.15.102:0/4069944217 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:51:09.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7e39c700 1 --2- 172.21.15.102:0/4069944217 >> v2:172.21.15.172:3300/0 conn(0x7fda780cd390 0x7fda780ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:09.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7cb99700 1 -- 172.21.15.102:0/4069944217 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212355152 0 0) 0x7fda78153660 con 0x7fda78153290 2023-03-03T05:51:09.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7cb99700 1 -- 172.21.15.102:0/4069944217 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda640035b0 con 0x7fda78153290 2023-03-03T05:51:09.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7cb99700 1 -- 172.21.15.102:0/4069944217 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296096308 0 0) 0x7fda7806c360 con 0x7fda780cd8e0 2023-03-03T05:51:09.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7cb99700 1 -- 172.21.15.102:0/4069944217 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda78153660 con 0x7fda780cd8e0 2023-03-03T05:51:09.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7e39c700 1 -- 172.21.15.102:0/4069944217 >> v1:172.21.15.172:6789/0 conn(0x7fda780cd8e0 legacy=0x7fda780cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:09.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7e39c700 1 -- 172.21.15.102:0/4069944217 >> v1:172.21.15.102:6789/0 conn(0x7fda78153290 legacy=0x7fda78158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:09.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.972+0000 7fda7e39c700 1 -- 172.21.15.102:0/4069944217 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7806c7f0 con 0x7fda780cd390 2023-03-03T05:51:09.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.973+0000 7fda7e39c700 1 --2- 172.21.15.102:0/4069944217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 0x7fda780ce530 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fda70009d90 tx=0x7fda7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c694f6caa15c5238 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:51:09.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.973+0000 7fda7cb99700 1 -- 172.21.15.102:0/4069944217 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000e070 con 0x7fda780cd390 2023-03-03T05:51:09.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.973+0000 7fda7cb99700 1 -- 172.21.15.102:0/4069944217 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000bb20 con 0x7fda780cd390 2023-03-03T05:51:09.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.973+0000 7fda7cb99700 1 -- 172.21.15.102:0/4069944217 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70012430 con 0x7fda780cd390 2023-03-03T05:51:09.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.974+0000 7fda7fdff700 1 -- 172.21.15.102:0/4069944217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 msgr2=0x7fda780ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:09.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.974+0000 7fda7fdff700 1 --2- 172.21.15.102:0/4069944217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 0x7fda780ce530 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fda70009d90 tx=0x7fda7000b330 comp rx=0 tx=0).stop 2023-03-03T05:51:09.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.974+0000 7fda7fdff700 1 -- 172.21.15.102:0/4069944217 shutdown_connections 2023-03-03T05:51:09.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.974+0000 7fda7fdff700 1 --2- 172.21.15.102:0/4069944217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 0x7fda780ce530 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:09.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.974+0000 7fda7fdff700 1 -- 172.21.15.102:0/4069944217 >> 172.21.15.102:0/4069944217 conn(0x7fda78065fb0 msgr2=0x7fda780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:51:09.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.974+0000 7fda7fdff700 1 -- 172.21.15.102:0/4069944217 shutdown_connections 2023-03-03T05:51:09.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.974+0000 7fda7fdff700 1 -- 172.21.15.102:0/4069944217 wait complete. 2023-03-03T05:51:09.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.976+0000 7fda7fdff700 1 Processor -- start 2023-03-03T05:51:09.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.976+0000 7fda7fdff700 1 -- start start 2023-03-03T05:51:09.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7fdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 0x7fda781f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:09.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7fdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda780cd8e0 0x7fda78157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:09.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7fdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda78153290 0x7fda7814e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:09.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7fdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda78153bb0 con 0x7fda780cd8e0 2023-03-03T05:51:09.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7fdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7814fa40 con 0x7fda780cd390 2023-03-03T05:51:09.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7fdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda781f2770 con 0x7fda78153290 2023-03-03T05:51:09.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7d39a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda780cd8e0 0x7fda78157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:09.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7db9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 0x7fda781f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:09.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7d39a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda780cd8e0 0x7fda78157850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34054/0 (socket says 172.21.15.102:34054) 2023-03-03T05:51:09.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.977+0000 7fda7d39a700 1 -- 172.21.15.102:0/290514309 learned_addr learned my addr 172.21.15.102:0/290514309 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:51:09.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda7e39c700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda78153290 0x7fda7814e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:09.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda7d39a700 1 -- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda78153290 msgr2=0x7fda7814e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:09.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda7d39a700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda78153290 0x7fda7814e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:09.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda7d39a700 1 -- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 msgr2=0x7fda781f2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:09.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda7d39a700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 0x7fda781f2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:09.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda7d39a700 1 -- 172.21.15.102:0/290514309 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70009a70 con 0x7fda780cd8e0 2023-03-03T05:51:09.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda7d39a700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda780cd8e0 0x7fda78157850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda6800c550 tx=0x7fda6800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:51:09.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda6e7fc700 1 -- 172.21.15.102:0/290514309 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda680052d0 con 0x7fda780cd8e0 2023-03-03T05:51:09.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.978+0000 7fda6e7fc700 1 -- 172.21.15.102:0/290514309 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda68005430 con 0x7fda780cd8e0 2023-03-03T05:51:09.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.979+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda780ccde0 con 0x7fda780cd8e0 2023-03-03T05:51:09.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.979+0000 7fda6e7fc700 1 -- 172.21.15.102:0/290514309 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda680038c0 con 0x7fda780cd8e0 2023-03-03T05:51:09.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.979+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7815d160 con 0x7fda780cd8e0 2023-03-03T05:51:09.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.980+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda780cd0c0 con 0x7fda780cd8e0 2023-03-03T05:51:09.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.981+0000 7fda6e7fc700 1 -- 172.21.15.102:0/290514309 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda68019030 con 0x7fda780cd8e0 2023-03-03T05:51:09.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.982+0000 7fda6e7fc700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda5c0682f0 0x7fda5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:09.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.982+0000 7fda6e7fc700 1 -- 172.21.15.102:0/290514309 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fda68085150 con 0x7fda780cd8e0 2023-03-03T05:51:09.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.982+0000 7fda7db9b700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda5c0682f0 0x7fda5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:09.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.983+0000 7fda7db9b700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda5c0682f0 0x7fda5c06a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda74007fd0 tx=0x7fda74007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:51:09.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:09.987+0000 7fda6e7fc700 1 -- 172.21.15.102:0/290514309 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda680517d0 con 0x7fda780cd8e0 2023-03-03T05:51:10.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.221+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda7804e6e0 con 0x7fda780cd8e0 2023-03-03T05:51:10.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.222+0000 7fda6e7fc700 1 -- 172.21.15.102:0/290514309 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fda68017090 con 0x7fda780cd8e0 2023-03-03T05:51:10.225 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:51:10.225 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:51:10.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda5c0682f0 msgr2=0x7fda5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:10.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda5c0682f0 0x7fda5c06a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda74007fd0 tx=0x7fda74007de0 comp rx=0 tx=0).stop 2023-03-03T05:51:10.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda780cd8e0 msgr2=0x7fda78157850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:10.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda780cd8e0 0x7fda78157850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda6800c550 tx=0x7fda6800c860 comp rx=0 tx=0).stop 2023-03-03T05:51:10.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 shutdown_connections 2023-03-03T05:51:10.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda5c0682f0 0x7fda5c06a7a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:10.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda78153290 0x7fda7814e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:10.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda780cd390 0x7fda781f2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:10.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 --2- 172.21.15.102:0/290514309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda780cd8e0 0x7fda78157850 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:10.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 >> 172.21.15.102:0/290514309 conn(0x7fda78065fb0 msgr2=0x7fda78151050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:51:10.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 shutdown_connections 2023-03-03T05:51:10.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:10.225+0000 7fda7fdff700 1 -- 172.21.15.102:0/290514309 wait complete. 2023-03-03T05:51:10.230 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:51:10.255 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T05:51:28.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:51:28.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:51:54.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:51:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.516+0000 7fdf0359e700 1 Processor -- start 2023-03-03T05:51:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.517+0000 7fdf0359e700 1 -- start start 2023-03-03T05:51:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.518+0000 7fdf0359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdf040c9610 0x7fdf040ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.518+0000 7fdf0359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040ca980 0x7fdf041636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.518+0000 7fdf0359e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040c9ca0 con 0x7fdf041564b0 2023-03-03T05:51:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.518+0000 7fdf0359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040b4920 con 0x7fdf040c9610 2023-03-03T05:51:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.518+0000 7fdf0359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040b4c00 con 0x7fdf040ca980 2023-03-03T05:51:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdf040c9610 0x7fdf040ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdf040c9610 0x7fdf040ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52890/0 (socket says 172.21.15.102:52890) 2023-03-03T05:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 -- 172.21.15.102:0/1404377860 learned_addr learned my addr 172.21.15.102:0/1404377860 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf02d9d700 1 --2- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040ca980 0x7fdf041636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf0259c700 1 --1- 172.21.15.102:0/1404377860 >> v1:172.21.15.172:6789/0 conn(0x7fdf041564b0 0x7fdf040cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49388/0 (socket says 172.21.15.102:49388) 2023-03-03T05:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 -- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040ca980 msgr2=0x7fdf041636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 --2- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040ca980 0x7fdf041636a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 -- 172.21.15.102:0/1404377860 >> v1:172.21.15.172:6789/0 conn(0x7fdf041564b0 legacy=0x7fdf040cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 -- 172.21.15.102:0/1404377860 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040b46f0 con 0x7fdf040c9610 2023-03-03T05:51:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf0159a700 1 -- 172.21.15.102:0/1404377860 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42527868 0 0) 0x7fdf040c9ca0 con 0x7fdf041564b0 2023-03-03T05:51:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf01d9b700 1 --2- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040c9610 0x7fdf040ca440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdef80098f0 tx=0x7fdef8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3e7a38869a692a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:51:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.519+0000 7fdf0159a700 1 -- 172.21.15.102:0/1404377860 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8011830 con 0x7fdf040c9610 2023-03-03T05:51:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0159a700 1 -- 172.21.15.102:0/1404377860 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef8011990 con 0x7fdf040c9610 2023-03-03T05:51:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0159a700 1 -- 172.21.15.102:0/1404377860 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8011d20 con 0x7fdf040c9610 2023-03-03T05:51:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 -- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040c9610 msgr2=0x7fdf040ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 --2- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040c9610 0x7fdf040ca440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdef80098f0 tx=0x7fdef8009c00 comp rx=0 tx=0).stop 2023-03-03T05:51:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 -- 172.21.15.102:0/1404377860 shutdown_connections 2023-03-03T05:51:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 --2- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040ca980 0x7fdf041636a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 --2- 172.21.15.102:0/1404377860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040c9610 0x7fdf040ca440 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 -- 172.21.15.102:0/1404377860 >> 172.21.15.102:0/1404377860 conn(0x7fdf04065fb0 msgr2=0x7fdf040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:51:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 -- 172.21.15.102:0/1404377860 shutdown_connections 2023-03-03T05:51:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.520+0000 7fdf0359e700 1 -- 172.21.15.102:0/1404377860 wait complete. 2023-03-03T05:51:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.522+0000 7fdf0359e700 1 Processor -- start 2023-03-03T05:51:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.522+0000 7fdf0359e700 1 -- start start 2023-03-03T05:51:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.523+0000 7fdf0359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040c9610 0x7fdf041f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.523+0000 7fdf0359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040ca980 0x7fdf0415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.523+0000 7fdf0359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf041564b0 0x7fdf0415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.523+0000 7fdf0359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040c62b0 con 0x7fdf041564b0 2023-03-03T05:51:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.523+0000 7fdf0359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415cad0 con 0x7fdf040ca980 2023-03-03T05:51:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.523+0000 7fdf0359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415cd70 con 0x7fdf040c9610 2023-03-03T05:51:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf02d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf041564b0 0x7fdf0415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf0259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040c9610 0x7fdf041f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040ca980 0x7fdf0415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040ca980 0x7fdf0415a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52904/0 (socket says 172.21.15.102:52904) 2023-03-03T05:51:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf0259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040c9610 0x7fdf041f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41646/0 (socket says 172.21.15.102:41646) 2023-03-03T05:51:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 -- 172.21.15.102:0/3411113207 learned_addr learned my addr 172.21.15.102:0/3411113207 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:51:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 -- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040c9610 msgr2=0x7fdf041f1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040c9610 0x7fdf041f1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 -- 172.21.15.102:0/3411113207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf041564b0 msgr2=0x7fdf0415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf041564b0 0x7fdf0415a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf01d9b700 1 -- 172.21.15.102:0/3411113207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef8009550 con 0x7fdf040ca980 2023-03-03T05:51:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.524+0000 7fdf0259c700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040c9610 0x7fdf041f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:51:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.525+0000 7fdf01d9b700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040ca980 0x7fdf0415a1d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009c40 tx=0x7fdef8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:51:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.525+0000 7fdef2ffd700 1 -- 172.21.15.102:0/3411113207 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8004300 con 0x7fdf040ca980 2023-03-03T05:51:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.525+0000 7fdef2ffd700 1 -- 172.21.15.102:0/3411113207 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef801f070 con 0x7fdf040ca980 2023-03-03T05:51:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.525+0000 7fdef2ffd700 1 -- 172.21.15.102:0/3411113207 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef80036a0 con 0x7fdf040ca980 2023-03-03T05:51:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.525+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040b2210 con 0x7fdf040ca980 2023-03-03T05:51:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.525+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf040b1f30 con 0x7fdf040ca980 2023-03-03T05:51:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.526+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf04156880 con 0x7fdf040ca980 2023-03-03T05:51:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.527+0000 7fdef2ffd700 1 -- 172.21.15.102:0/3411113207 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdef802a020 con 0x7fdf040ca980 2023-03-03T05:51:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.528+0000 7fdef2ffd700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdee4068320 0x7fdee406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:51:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.528+0000 7fdef2ffd700 1 -- 172.21.15.102:0/3411113207 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdef8087330 con 0x7fdf040ca980 2023-03-03T05:51:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.529+0000 7fdf0259c700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdee4068320 0x7fdee406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:51:54.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.529+0000 7fdf0259c700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdee4068320 0x7fdee406a7d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdef4000f60 tx=0x7fdef4007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:51:54.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.533+0000 7fdef2ffd700 1 -- 172.21.15.102:0/3411113207 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdef80539c0 con 0x7fdf040ca980 2023-03-03T05:51:54.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.764+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf0404e6e0 con 0x7fdf040ca980 2023-03-03T05:51:54.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.765+0000 7fdef2ffd700 1 -- 172.21.15.102:0/3411113207 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdef8056fe0 con 0x7fdf040ca980 2023-03-03T05:51:54.768 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:51:54.768 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:51:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdee4068320 msgr2=0x7fdee406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdee4068320 0x7fdee406a7d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdef4000f60 tx=0x7fdef4007e70 comp rx=0 tx=0).stop 2023-03-03T05:51:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040ca980 msgr2=0x7fdf0415a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:51:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040ca980 0x7fdf0415a1d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009c40 tx=0x7fdef8004750 comp rx=0 tx=0).stop 2023-03-03T05:51:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 shutdown_connections 2023-03-03T05:51:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdee4068320 0x7fdee406a7d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf040c9610 0x7fdf041f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf041564b0 0x7fdf0415a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 --2- 172.21.15.102:0/3411113207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf040ca980 0x7fdf0415a1d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:51:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 >> 172.21.15.102:0/3411113207 conn(0x7fdf04065fb0 msgr2=0x7fdf04156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:51:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 shutdown_connections 2023-03-03T05:51:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:51:54.768+0000 7fdf0359e700 1 -- 172.21.15.102:0/3411113207 wait complete. 2023-03-03T05:51:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:51:54.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T05:51:58.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:51:58.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:52:29.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:52:29.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:52:33.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:52:34.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.047+0000 7f27023ff700 1 Processor -- start 2023-03-03T05:52:34.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.048+0000 7f27023ff700 1 -- start start 2023-03-03T05:52:34.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f27023ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc153290 0x7f26fc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:52:34.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f27023ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f26fc1593c0 0x7f26fc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:52:34.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f27023ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f26fc0cd8e0 0x7f26fc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:52:34.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f27023ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc153660 con 0x7f26fc1593c0 2023-03-03T05:52:34.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f27023ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc06c360 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f27023ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc15cea0 con 0x7f26fc153290 2023-03-03T05:52:34.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f26fb7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f26fc1593c0 0x7f26fc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:52:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.049+0000 7f26fbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc153290 0x7f26fc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:52:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f26fc1593c0 0x7f26fc0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41506/0 (socket says 172.21.15.102:41506) 2023-03-03T05:52:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 -- 172.21.15.102:0/4003184840 learned_addr learned my addr 172.21.15.102:0/4003184840 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:52:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f270099c700 1 --2- 172.21.15.102:0/4003184840 >> v2:172.21.15.172:3300/0 conn(0x7f26fc0cd8e0 0x7f26fc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:52:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 -- 172.21.15.102:0/4003184840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc153290 msgr2=0x7f26fc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:52:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 --2- 172.21.15.102:0/4003184840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc153290 0x7f26fc158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 -- 172.21.15.102:0/4003184840 >> v2:172.21.15.172:3300/0 conn(0x7f26fc0cd8e0 msgr2=0x7f26fc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:52:34.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 --2- 172.21.15.102:0/4003184840 >> v2:172.21.15.172:3300/0 conn(0x7f26fc0cd8e0 0x7f26fc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 -- 172.21.15.102:0/4003184840 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc06c7f0 con 0x7f26fc1593c0 2023-03-03T05:52:34.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.050+0000 7f26fb7fe700 1 --2- 172.21.15.102:0/4003184840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 0x7f26fc0cde20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f26fc06d0e0 tx=0x7f26f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b8dac9eb6502af server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:52:34.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f26faffd700 1 -- 172.21.15.102:0/4003184840 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0011830 con 0x7f26fc1593c0 2023-03-03T05:52:34.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f26faffd700 1 -- 172.21.15.102:0/4003184840 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f0011990 con 0x7f26fc1593c0 2023-03-03T05:52:34.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f26faffd700 1 -- 172.21.15.102:0/4003184840 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0011d20 con 0x7f26fc1593c0 2023-03-03T05:52:34.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 -- 172.21.15.102:0/4003184840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 msgr2=0x7f26fc0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:52:34.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 --2- 172.21.15.102:0/4003184840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 0x7f26fc0cde20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f26fc06d0e0 tx=0x7f26f0009bd0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 -- 172.21.15.102:0/4003184840 shutdown_connections 2023-03-03T05:52:34.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 --2- 172.21.15.102:0/4003184840 >> v2:172.21.15.172:3300/0 conn(0x7f26fc0cd8e0 0x7f26fc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 --2- 172.21.15.102:0/4003184840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc153290 0x7f26fc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 --2- 172.21.15.102:0/4003184840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 0x7f26fc0cde20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 -- 172.21.15.102:0/4003184840 >> 172.21.15.102:0/4003184840 conn(0x7f26fc065fb0 msgr2=0x7f26fc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:52:34.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.051+0000 7f27023ff700 1 -- 172.21.15.102:0/4003184840 shutdown_connections 2023-03-03T05:52:34.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.052+0000 7f27023ff700 1 -- 172.21.15.102:0/4003184840 wait complete. 2023-03-03T05:52:34.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.053+0000 7f27023ff700 1 Processor -- start 2023-03-03T05:52:34.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.054+0000 7f27023ff700 1 -- start start 2023-03-03T05:52:34.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f27023ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc0cd8e0 0x7f26fc0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:52:34.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f27023ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26fc153290 0x7f26fc155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:52:34.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f27023ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 0x7f26fc155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:52:34.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f27023ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0c7060 con 0x7f26fc1593c0 2023-03-03T05:52:34.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f27023ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc1e50c0 con 0x7f26fc153290 2023-03-03T05:52:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f27023ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc153970 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc0cd8e0 0x7f26fc0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:52:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26fc153290 0x7f26fc155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:52:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc0cd8e0 0x7f26fc0c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42594/0 (socket says 172.21.15.102:42594) 2023-03-03T05:52:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 -- 172.21.15.102:0/1074585167 learned_addr learned my addr 172.21.15.102:0/1074585167 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:52:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f270099c700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 0x7f26fc155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:52:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 -- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26fc153290 msgr2=0x7f26fc155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:52:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26fc153290 0x7f26fc155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 -- 172.21.15.102:0/1074585167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 msgr2=0x7f26fc155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:52:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 0x7f26fc155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.055+0000 7f26fbfff700 1 -- 172.21.15.102:0/1074585167 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f0009550 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.056+0000 7f26fbfff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc0cd8e0 0x7f26fc0c8a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00e870 tx=0x7f26ec00eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:52:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.056+0000 7f26f8ff9700 1 -- 172.21.15.102:0/1074585167 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec009590 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.056+0000 7f26f8ff9700 1 -- 172.21.15.102:0/1074585167 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec013680 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.056+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc0ccde0 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.056+0000 7f26f8ff9700 1 -- 172.21.15.102:0/1074585167 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0139f0 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.056+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc15d160 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.057+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc0cd0c0 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.059+0000 7f26f8ff9700 1 -- 172.21.15.102:0/1074585167 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f26ec019030 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.060+0000 7f26f8ff9700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f26e80682f0 0x7f26e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:52:34.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.060+0000 7f26f8ff9700 1 -- 172.21.15.102:0/1074585167 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f26ec085b10 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.060+0000 7f26fb7fe700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f26e80682f0 0x7f26e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:52:34.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.061+0000 7f26fb7fe700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f26e80682f0 0x7f26e806a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009c10 tx=0x7f26f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:52:34.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.064+0000 7f26f8ff9700 1 -- 172.21.15.102:0/1074585167 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f26ec052130 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.300+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26fc04e6e0 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.300+0000 7f26f8ff9700 1 -- 172.21.15.102:0/1074585167 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f26ec021090 con 0x7f26fc0cd8e0 2023-03-03T05:52:34.303 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:52:34.304 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:52:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f26e80682f0 msgr2=0x7f26e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:52:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f26e80682f0 0x7f26e806a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009c10 tx=0x7f26f0008030 comp rx=0 tx=0).stop 2023-03-03T05:52:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc0cd8e0 msgr2=0x7f26fc0c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:52:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc0cd8e0 0x7f26fc0c8a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00e870 tx=0x7f26ec00eb80 comp rx=0 tx=0).stop 2023-03-03T05:52:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 shutdown_connections 2023-03-03T05:52:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f26e80682f0 0x7f26e806a7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f26fc0cd8e0 0x7f26fc0c8a20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26fc1593c0 0x7f26fc155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 --2- 172.21.15.102:0/1074585167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26fc153290 0x7f26fc155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:52:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 >> 172.21.15.102:0/1074585167 conn(0x7f26fc065fb0 msgr2=0x7f26fc1e52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:52:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 shutdown_connections 2023-03-03T05:52:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:52:34.303+0000 7f27023ff700 1 -- 172.21.15.102:0/1074585167 wait complete. 2023-03-03T05:52:34.308 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:52:34.333 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-03-03T05:52:59.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:52:59.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:53:29.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:53:29.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:53:57.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:53:57.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.600+0000 7f5af29ff700 1 Processor -- start 2023-03-03T05:53:57.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.600+0000 7f5af29ff700 1 -- start start 2023-03-03T05:53:57.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.601+0000 7f5af29ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5aec14f4d0 0x7f5aec155150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:53:57.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.601+0000 7f5af29ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0c7500 con 0x7f5aec0706f0 2023-03-03T05:53:57.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5af29ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec071a00 con 0x7f5aec0715a0 2023-03-03T05:53:57.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5af29ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec0728f0 con 0x7f5aec14f4d0 2023-03-03T05:53:57.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5af0f9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5aec0706f0 0x7f5aec070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43852/0 (socket says 172.21.15.102:43852) 2023-03-03T05:53:57.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aeb7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5aec0715a0 0x7f5aec0c7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37114/0 (socket says 172.21.15.102:37114) 2023-03-03T05:53:57.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5af0f9c700 1 -- 172.21.15.102:0/2545522755 learned_addr learned my addr 172.21.15.102:0/2545522755 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:53:57.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aebfff700 1 --2- 172.21.15.102:0/2545522755 >> v2:172.21.15.172:3300/0 conn(0x7f5aec14f4d0 0x7f5aec155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:53:57.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aeaffd700 1 -- 172.21.15.102:0/2545522755 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262024999 0 0) 0x7f5aec071a00 con 0x7f5aec0715a0 2023-03-03T05:53:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aeaffd700 1 -- 172.21.15.102:0/2545522755 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad40035b0 con 0x7f5aec0715a0 2023-03-03T05:53:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aeaffd700 1 -- 172.21.15.102:0/2545522755 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296972945 0 0) 0x7f5aec0c7500 con 0x7f5aec0706f0 2023-03-03T05:53:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aebfff700 1 -- 172.21.15.102:0/2545522755 >> v1:172.21.15.172:6789/0 conn(0x7f5aec0715a0 legacy=0x7f5aec0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:53:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aebfff700 1 -- 172.21.15.102:0/2545522755 >> v1:172.21.15.102:6789/0 conn(0x7f5aec0706f0 legacy=0x7f5aec070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:53:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.602+0000 7f5aebfff700 1 -- 172.21.15.102:0/2545522755 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0c71d0 con 0x7f5aec14f4d0 2023-03-03T05:53:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.603+0000 7f5aebfff700 1 --2- 172.21.15.102:0/2545522755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec14f4d0 0x7f5aec155150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009840 tx=0x7f5adc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=638399a09ba93837 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:53:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.603+0000 7f5aeaffd700 1 -- 172.21.15.102:0/2545522755 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc011830 con 0x7f5aec14f4d0 2023-03-03T05:53:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.603+0000 7f5aeaffd700 1 -- 172.21.15.102:0/2545522755 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc011990 con 0x7f5aec14f4d0 2023-03-03T05:53:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.603+0000 7f5aeaffd700 1 -- 172.21.15.102:0/2545522755 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc011d20 con 0x7f5aec14f4d0 2023-03-03T05:53:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.603+0000 7f5af29ff700 1 -- 172.21.15.102:0/2545522755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec14f4d0 msgr2=0x7f5aec155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:53:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.603+0000 7f5af29ff700 1 --2- 172.21.15.102:0/2545522755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec14f4d0 0x7f5aec155150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009840 tx=0x7f5adc009b50 comp rx=0 tx=0).stop 2023-03-03T05:53:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.604+0000 7f5af29ff700 1 -- 172.21.15.102:0/2545522755 shutdown_connections 2023-03-03T05:53:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.604+0000 7f5af29ff700 1 --2- 172.21.15.102:0/2545522755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec14f4d0 0x7f5aec155150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:53:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.604+0000 7f5af29ff700 1 -- 172.21.15.102:0/2545522755 >> 172.21.15.102:0/2545522755 conn(0x7f5aec06af90 msgr2=0x7f5aec06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:53:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.604+0000 7f5af29ff700 1 -- 172.21.15.102:0/2545522755 shutdown_connections 2023-03-03T05:53:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.604+0000 7f5af29ff700 1 -- 172.21.15.102:0/2545522755 wait complete. 2023-03-03T05:53:57.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.606+0000 7f5af29ff700 1 Processor -- start 2023-03-03T05:53:57.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.607+0000 7f5af29ff700 1 -- start start 2023-03-03T05:53:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5af29ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec0706f0 0x7f5aec13de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:53:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5af29ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aec0715a0 0x7f5aec13e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:53:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5af29ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aec14f4d0 0x7f5aec13bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:53:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5af29ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec153220 con 0x7f5aec0715a0 2023-03-03T05:53:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5af29ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec149090 con 0x7f5aec0706f0 2023-03-03T05:53:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5af29ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec1f3990 con 0x7f5aec14f4d0 2023-03-03T05:53:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5aebfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec0706f0 0x7f5aec13de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:53:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5aebfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec0706f0 0x7f5aec13de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38466/0 (socket says 172.21.15.102:38466) 2023-03-03T05:53:57.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5aebfff700 1 -- 172.21.15.102:0/289706475 learned_addr learned my addr 172.21.15.102:0/289706475 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:53:57.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.608+0000 7f5aeb7fe700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aec0715a0 0x7f5aec13e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:53:57.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.609+0000 7f5af0f9c700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aec14f4d0 0x7f5aec13bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:53:57.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.609+0000 7f5aebfff700 1 -- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aec14f4d0 msgr2=0x7f5aec13bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:53:57.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.609+0000 7f5aebfff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aec14f4d0 0x7f5aec13bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:53:57.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.609+0000 7f5aebfff700 1 -- 172.21.15.102:0/289706475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aec0715a0 msgr2=0x7f5aec13e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:53:57.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.609+0000 7f5aebfff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aec0715a0 0x7f5aec13e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:53:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.609+0000 7f5aebfff700 1 -- 172.21.15.102:0/289706475 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc0094a0 con 0x7f5aec0706f0 2023-03-03T05:53:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.609+0000 7f5aebfff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec0706f0 0x7f5aec13de70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5adc014040 tx=0x7f5adc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:53:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.610+0000 7f5ae8ff9700 1 -- 172.21.15.102:0/289706475 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc01a7e0 con 0x7f5aec0706f0 2023-03-03T05:53:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.610+0000 7f5ae8ff9700 1 -- 172.21.15.102:0/289706475 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc01a940 con 0x7f5aec0706f0 2023-03-03T05:53:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.610+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec0c5390 con 0x7f5aec0706f0 2023-03-03T05:53:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.610+0000 7f5ae8ff9700 1 -- 172.21.15.102:0/289706475 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc01acd0 con 0x7f5aec0706f0 2023-03-03T05:53:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.610+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec14f8a0 con 0x7f5aec0706f0 2023-03-03T05:53:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.611+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec071c50 con 0x7f5aec0706f0 2023-03-03T05:53:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.612+0000 7f5ae8ff9700 1 -- 172.21.15.102:0/289706475 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5adc00f050 con 0x7f5aec0706f0 2023-03-03T05:53:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.613+0000 7f5ae8ff9700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ad80682f0 0x7f5ad806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:53:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.613+0000 7f5ae8ff9700 1 -- 172.21.15.102:0/289706475 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5adc0873e0 con 0x7f5aec0706f0 2023-03-03T05:53:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.614+0000 7f5aeb7fe700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ad80682f0 0x7f5ad806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:53:57.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.614+0000 7f5aeb7fe700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ad80682f0 0x7f5ad806a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5aec165ab0 tx=0x7f5ae0007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:53:57.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.617+0000 7f5ae8ff9700 1 -- 172.21.15.102:0/289706475 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5adc053ae0 con 0x7f5aec0706f0 2023-03-03T05:53:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.848+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5aec04e6e0 con 0x7f5aec0706f0 2023-03-03T05:53:57.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.849+0000 7f5ae8ff9700 1 -- 172.21.15.102:0/289706475 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5adc029020 con 0x7f5aec0706f0 2023-03-03T05:53:57.851 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:53:57.851 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:53:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ad80682f0 msgr2=0x7f5ad806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:53:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ad80682f0 0x7f5ad806a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5aec165ab0 tx=0x7f5ae0007d80 comp rx=0 tx=0).stop 2023-03-03T05:53:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec0706f0 msgr2=0x7f5aec13de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:53:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec0706f0 0x7f5aec13de70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5adc014040 tx=0x7f5adc004750 comp rx=0 tx=0).stop 2023-03-03T05:53:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 shutdown_connections 2023-03-03T05:53:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ad80682f0 0x7f5ad806a7a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:53:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aec14f4d0 0x7f5aec13bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:53:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aec0706f0 0x7f5aec13de70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:53:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 --2- 172.21.15.102:0/289706475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aec0715a0 0x7f5aec13e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:53:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 >> 172.21.15.102:0/289706475 conn(0x7f5aec06af90 msgr2=0x7f5aec06cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:53:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 shutdown_connections 2023-03-03T05:53:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:53:57.852+0000 7f5af29ff700 1 -- 172.21.15.102:0/289706475 wait complete. 2023-03-03T05:53:57.856 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:53:57.882 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2023-03-03T05:53:59.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:53:59.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:54:29.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:54:29.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:55:00.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:55:00.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:55:17.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:55:18.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.129+0000 7f053559d700 1 Processor -- start 2023-03-03T05:55:18.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.129+0000 7f053559d700 1 -- start start 2023-03-03T05:55:18.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f053559d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05300c9610 0x7f05300ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:55:18.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f053559d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300c9ca0 con 0x7f05300ca980 2023-03-03T05:55:18.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f053559d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300b46f0 con 0x7f05301564b0 2023-03-03T05:55:18.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f053559d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300b4920 con 0x7f05300c9610 2023-03-03T05:55:18.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f052f7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05300c9610 0x7f05300ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:55:18.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f0534d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f05300ca980 0x7f05301636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33694/0 (socket says 172.21.15.102:33694) 2023-03-03T05:55:18.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f0534d9c700 1 -- 172.21.15.102:0/2746179101 learned_addr learned my addr 172.21.15.102:0/2746179101 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T05:55:18.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.131+0000 7f052ffff700 1 --1- 172.21.15.102:0/2746179101 >> v1:172.21.15.172:6789/0 conn(0x7f05301564b0 0x7f05300cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46440/0 (socket says 172.21.15.102:46440) 2023-03-03T05:55:18.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.132+0000 7f052f7fe700 1 -- 172.21.15.102:0/2746179101 >> v1:172.21.15.172:6789/0 conn(0x7f05301564b0 legacy=0x7f05300cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:55:18.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.132+0000 7f052f7fe700 1 -- 172.21.15.102:0/2746179101 >> v1:172.21.15.102:6789/0 conn(0x7f05300ca980 legacy=0x7f05301636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:55:18.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.132+0000 7f052effd700 1 -- 172.21.15.102:0/2746179101 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448257781 0 0) 0x7f05300b46f0 con 0x7f05301564b0 2023-03-03T05:55:18.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.132+0000 7f052f7fe700 1 -- 172.21.15.102:0/2746179101 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05300b4c00 con 0x7f05300c9610 2023-03-03T05:55:18.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.132+0000 7f052f7fe700 1 --2- 172.21.15.102:0/2746179101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05300c9610 0x7f05300ca440 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0524009840 tx=0x7f0524009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8dbd24a8c2a478e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:55:18.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.132+0000 7f052effd700 1 -- 172.21.15.102:0/2746179101 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524004230 con 0x7f05300c9610 2023-03-03T05:55:18.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f052effd700 1 -- 172.21.15.102:0/2746179101 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0524004390 con 0x7f05300c9610 2023-03-03T05:55:18.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f052effd700 1 -- 172.21.15.102:0/2746179101 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05240036a0 con 0x7f05300c9610 2023-03-03T05:55:18.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f053559d700 1 -- 172.21.15.102:0/2746179101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05300c9610 msgr2=0x7f05300ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:55:18.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f053559d700 1 --2- 172.21.15.102:0/2746179101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05300c9610 0x7f05300ca440 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0524009840 tx=0x7f0524009b50 comp rx=0 tx=0).stop 2023-03-03T05:55:18.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f053559d700 1 -- 172.21.15.102:0/2746179101 shutdown_connections 2023-03-03T05:55:18.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f053559d700 1 --2- 172.21.15.102:0/2746179101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05300c9610 0x7f05300ca440 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:55:18.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f053559d700 1 -- 172.21.15.102:0/2746179101 >> 172.21.15.102:0/2746179101 conn(0x7f0530065fb0 msgr2=0x7f05300663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:55:18.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f053559d700 1 -- 172.21.15.102:0/2746179101 shutdown_connections 2023-03-03T05:55:18.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.133+0000 7f053559d700 1 -- 172.21.15.102:0/2746179101 wait complete. 2023-03-03T05:55:18.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.135+0000 7f053559d700 1 Processor -- start 2023-03-03T05:55:18.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.135+0000 7f053559d700 1 -- start start 2023-03-03T05:55:18.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.136+0000 7f053559d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05300c9610 0x7f05301ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:55:18.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.136+0000 7f053559d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05300ca980 0x7f053015cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:55:18.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.136+0000 7f053559d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05301564b0 0x7f0530159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:55:18.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.136+0000 7f053559d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300c6f80 con 0x7f05300ca980 2023-03-03T05:55:18.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.136+0000 7f053559d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05301ef5c0 con 0x7f05301564b0 2023-03-03T05:55:18.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.136+0000 7f053559d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300c71b0 con 0x7f05300c9610 2023-03-03T05:55:18.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05300ca980 0x7f053015cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:55:18.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05300ca980 0x7f053015cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35430/0 (socket says 172.21.15.102:35430) 2023-03-03T05:55:18.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 -- 172.21.15.102:0/1799370509 learned_addr learned my addr 172.21.15.102:0/1799370509 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:55:18.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052ffff700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05300c9610 0x7f05301ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:55:18.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f0534d9c700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05301564b0 0x7f0530159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:55:18.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 -- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05300c9610 msgr2=0x7f05301ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:55:18.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05300c9610 0x7f05301ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:55:18.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 -- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05301564b0 msgr2=0x7f0530159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:55:18.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05301564b0 0x7f0530159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:55:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052f7fe700 1 -- 172.21.15.102:0/1799370509 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05240094a0 con 0x7f05300ca980 2023-03-03T05:55:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.137+0000 7f052ffff700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05300c9610 0x7f05301ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:55:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.138+0000 7f052f7fe700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05300ca980 0x7f053015cb20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f05240090b0 tx=0x7f0524004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:55:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.138+0000 7f052cff9700 1 -- 172.21.15.102:0/1799370509 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524003cd0 con 0x7f05300ca980 2023-03-03T05:55:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.138+0000 7f052cff9700 1 -- 172.21.15.102:0/1799370509 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052401f070 con 0x7f05300ca980 2023-03-03T05:55:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.138+0000 7f052cff9700 1 -- 172.21.15.102:0/1799370509 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524011870 con 0x7f05300ca980 2023-03-03T05:55:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.138+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05300b2210 con 0x7f05300ca980 2023-03-03T05:55:18.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.138+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05300b1f30 con 0x7f05300ca980 2023-03-03T05:55:18.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.139+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0530156880 con 0x7f05300ca980 2023-03-03T05:55:18.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.141+0000 7f052cff9700 1 -- 172.21.15.102:0/1799370509 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f052402a020 con 0x7f05300ca980 2023-03-03T05:55:18.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.142+0000 7f052cff9700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f051c0682f0 0x7f051c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:55:18.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.142+0000 7f052cff9700 1 -- 172.21.15.102:0/1799370509 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0524087390 con 0x7f05300ca980 2023-03-03T05:55:18.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.142+0000 7f052ffff700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f051c0682f0 0x7f051c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:55:18.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.143+0000 7f052ffff700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f051c0682f0 0x7f051c06a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0520007b10 tx=0x7f0520007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:55:18.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.147+0000 7f052cff9700 1 -- 172.21.15.102:0/1799370509 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0524053aa0 con 0x7f05300ca980 2023-03-03T05:55:18.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.377+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0530156880 con 0x7f05300ca980 2023-03-03T05:55:18.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.378+0000 7f052cff9700 1 -- 172.21.15.102:0/1799370509 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f05240570c0 con 0x7f05300ca980 2023-03-03T05:55:18.381 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:55:18.381 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:55:18.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f051c0682f0 msgr2=0x7f051c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:55:18.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f051c0682f0 0x7f051c06a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0520007b10 tx=0x7f0520007a60 comp rx=0 tx=0).stop 2023-03-03T05:55:18.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05300ca980 msgr2=0x7f053015cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:55:18.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05300ca980 0x7f053015cb20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f05240090b0 tx=0x7f0524004750 comp rx=0 tx=0).stop 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 shutdown_connections 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f051c0682f0 0x7f051c06a7a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05300c9610 0x7f05301ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05301564b0 0x7f0530159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 --2- 172.21.15.102:0/1799370509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05300ca980 0x7f053015cb20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.380+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 >> 172.21.15.102:0/1799370509 conn(0x7f0530065fb0 msgr2=0x7f0530066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.381+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 shutdown_connections 2023-03-03T05:55:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:55:18.381+0000 7f053559d700 1 -- 172.21.15.102:0/1799370509 wait complete. 2023-03-03T05:55:18.385 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:55:18.410 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2023-03-03T05:55:30.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:55:30.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:56:00.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:56:00.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:56:30.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:56:30.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:56:57.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:56:57.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.665+0000 7fbb799ff700 1 Processor -- start 2023-03-03T05:56:57.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.665+0000 7fbb799ff700 1 -- start start 2023-03-03T05:56:57.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.666+0000 7fbb799ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb74153290 0x7fbb74158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:56:57.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.666+0000 7fbb799ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb741593c0 0x7fbb740cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:56:57.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.666+0000 7fbb799ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb740cd8e0 0x7fbb740ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:56:57.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.666+0000 7fbb799ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb74153660 con 0x7fbb740cd8e0 2023-03-03T05:56:57.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.666+0000 7fbb799ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7406c360 con 0x7fbb741593c0 2023-03-03T05:56:57.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.666+0000 7fbb799ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7415cea0 con 0x7fbb74153290 2023-03-03T05:56:57.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb727fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb741593c0 0x7fbb740cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:56:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb72ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb74153290 0x7fbb74158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:56:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb737fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb740cd8e0 0x7fbb740ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:56:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb72ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb74153290 0x7fbb74158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:52786/0 (socket says 172.21.15.102:52786) 2023-03-03T05:56:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb72ffd700 1 -- 172.21.15.102:0/2126896339 learned_addr learned my addr 172.21.15.102:0/2126896339 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:56:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb727fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb741593c0 0x7fbb740cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52280/0 (socket says 172.21.15.102:52280) 2023-03-03T05:56:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb737fe700 1 -- 172.21.15.102:0/2126896339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb74153290 msgr2=0x7fbb74158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:56:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb737fe700 1 --2- 172.21.15.102:0/2126896339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb74153290 0x7fbb74158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb737fe700 1 -- 172.21.15.102:0/2126896339 >> v2:172.21.15.172:3300/0 conn(0x7fbb741593c0 msgr2=0x7fbb740cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:56:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb737fe700 1 --2- 172.21.15.102:0/2126896339 >> v2:172.21.15.172:3300/0 conn(0x7fbb741593c0 0x7fbb740cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb737fe700 1 -- 172.21.15.102:0/2126896339 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7406c7f0 con 0x7fbb740cd8e0 2023-03-03T05:56:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.667+0000 7fbb727fc700 1 --2- 172.21.15.102:0/2126896339 >> v2:172.21.15.172:3300/0 conn(0x7fbb741593c0 0x7fbb740cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:56:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb737fe700 1 --2- 172.21.15.102:0/2126896339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb740cd8e0 0x7fbb740ce360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb740b1970 tx=0x7fbb6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bea55b6fbd690e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:56:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb71ffb700 1 -- 172.21.15.102:0/2126896339 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00e070 con 0x7fbb740cd8e0 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb71ffb700 1 -- 172.21.15.102:0/2126896339 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c00bb20 con 0x7fbb740cd8e0 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb71ffb700 1 -- 172.21.15.102:0/2126896339 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c012430 con 0x7fbb740cd8e0 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb799ff700 1 -- 172.21.15.102:0/2126896339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb740cd8e0 msgr2=0x7fbb740ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb799ff700 1 --2- 172.21.15.102:0/2126896339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb740cd8e0 0x7fbb740ce360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb740b1970 tx=0x7fbb6c00b330 comp rx=0 tx=0).stop 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb799ff700 1 -- 172.21.15.102:0/2126896339 shutdown_connections 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb799ff700 1 --2- 172.21.15.102:0/2126896339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb740cd8e0 0x7fbb740ce360 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.668+0000 7fbb799ff700 1 --2- 172.21.15.102:0/2126896339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb74153290 0x7fbb74158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.669+0000 7fbb799ff700 1 --2- 172.21.15.102:0/2126896339 >> v2:172.21.15.172:3300/0 conn(0x7fbb741593c0 0x7fbb740cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.669+0000 7fbb799ff700 1 -- 172.21.15.102:0/2126896339 >> 172.21.15.102:0/2126896339 conn(0x7fbb74065fb0 msgr2=0x7fbb740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:56:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.669+0000 7fbb799ff700 1 -- 172.21.15.102:0/2126896339 shutdown_connections 2023-03-03T05:56:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.669+0000 7fbb799ff700 1 -- 172.21.15.102:0/2126896339 wait complete. 2023-03-03T05:56:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.671+0000 7fbb799ff700 1 Processor -- start 2023-03-03T05:56:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.671+0000 7fbb799ff700 1 -- start start 2023-03-03T05:56:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb799ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cd8e0 0x7fbb74156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:56:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb799ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb74153290 0x7fbb740c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:56:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb799ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb741593c0 0x7fbb741f0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:56:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb799ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74150240 con 0x7fbb74153290 2023-03-03T05:56:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb799ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740c5990 con 0x7fbb741593c0 2023-03-03T05:56:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb799ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74155440 con 0x7fbb740cd8e0 2023-03-03T05:56:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb727fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb74153290 0x7fbb740c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:56:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb727fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb74153290 0x7fbb740c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46130/0 (socket says 172.21.15.102:46130) 2023-03-03T05:56:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb727fc700 1 -- 172.21.15.102:0/1558111218 learned_addr learned my addr 172.21.15.102:0/1558111218 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:56:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb72ffd700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cd8e0 0x7fbb74156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:56:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.672+0000 7fbb737fe700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb741593c0 0x7fbb741f0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:56:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb727fc700 1 -- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cd8e0 msgr2=0x7fbb74156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:56:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb727fc700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cd8e0 0x7fbb74156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb727fc700 1 -- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb741593c0 msgr2=0x7fbb741f0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:56:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb727fc700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb741593c0 0x7fbb741f0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb727fc700 1 -- 172.21.15.102:0/1558111218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb680094a0 con 0x7fbb74153290 2023-03-03T05:56:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb727fc700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb74153290 0x7fbb740c8640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800ebe0 tx=0x7fbb6800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:56:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb5bfff700 1 -- 172.21.15.102:0/1558111218 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6800c850 con 0x7fbb74153290 2023-03-03T05:56:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb5bfff700 1 -- 172.21.15.102:0/1558111218 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6800c9b0 con 0x7fbb74153290 2023-03-03T05:56:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c009a70 con 0x7fbb74153290 2023-03-03T05:56:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb5bfff700 1 -- 172.21.15.102:0/1558111218 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6800cd40 con 0x7fbb74153290 2023-03-03T05:56:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.673+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb740ccde0 con 0x7fbb74153290 2023-03-03T05:56:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.675+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7415d160 con 0x7fbb74153290 2023-03-03T05:56:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.676+0000 7fbb5bfff700 1 -- 172.21.15.102:0/1558111218 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb68019030 con 0x7fbb74153290 2023-03-03T05:56:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.677+0000 7fbb5bfff700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb60068320 0x7fbb6006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:56:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.677+0000 7fbb5bfff700 1 -- 172.21.15.102:0/1558111218 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbb68085c70 con 0x7fbb74153290 2023-03-03T05:56:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.678+0000 7fbb72ffd700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb60068320 0x7fbb6006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:56:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.678+0000 7fbb72ffd700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb60068320 0x7fbb6006a7d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009b30 tx=0x7fbb640092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:56:57.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.682+0000 7fbb5bfff700 1 -- 172.21.15.102:0/1558111218 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb68052380 con 0x7fbb74153290 2023-03-03T05:56:57.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.916+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb740cd0c0 con 0x7fbb74153290 2023-03-03T05:56:57.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.917+0000 7fbb5bfff700 1 -- 172.21.15.102:0/1558111218 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbb6800cea0 con 0x7fbb74153290 2023-03-03T05:56:57.919 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:56:57.919 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:56:57.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb60068320 msgr2=0x7fbb6006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:56:57.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb60068320 0x7fbb6006a7d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009b30 tx=0x7fbb640092c0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb74153290 msgr2=0x7fbb740c8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:56:57.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb74153290 0x7fbb740c8640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800ebe0 tx=0x7fbb6800eef0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 shutdown_connections 2023-03-03T05:56:57.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb60068320 0x7fbb6006a7d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb740cd8e0 0x7fbb74156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb741593c0 0x7fbb741f0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 --2- 172.21.15.102:0/1558111218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb74153290 0x7fbb740c8640 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:56:57.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 >> 172.21.15.102:0/1558111218 conn(0x7fbb74065fb0 msgr2=0x7fbb74150d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:56:57.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 shutdown_connections 2023-03-03T05:56:57.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:56:57.920+0000 7fbb799ff700 1 -- 172.21.15.102:0/1558111218 wait complete. 2023-03-03T05:56:57.923 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:56:57.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-03-03T05:57:00.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:57:00.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:57:16.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:57:17.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.207+0000 7f75329ff700 1 Processor -- start 2023-03-03T05:57:17.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.207+0000 7f75329ff700 1 -- start start 2023-03-03T05:57:17.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.208+0000 7f75329ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f752c0ced80 0x7f752c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:57:17.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.208+0000 7f75329ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f752c0cf9a0 0x7f752c163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:57:17.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.208+0000 7f75329ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0cb9f0 con 0x7f752c1597f0 2023-03-03T05:57:17.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.208+0000 7f75329ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752c0ca750 con 0x7f752c0cf9a0 2023-03-03T05:57:17.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.208+0000 7f75329ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752c15a230 con 0x7f752c0ced80 2023-03-03T05:57:17.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7530f9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f752c0cf9a0 0x7f752c163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:57:17.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f752bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f752c0ced80 0x7f752c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:57:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7530f9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f752c0cf9a0 0x7f752c163660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48410/0 (socket says 172.21.15.102:48410) 2023-03-03T05:57:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7530f9c700 1 -- 172.21.15.102:0/795474513 learned_addr learned my addr 172.21.15.102:0/795474513 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:57:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7523fff700 1 --1- 172.21.15.102:0/795474513 >> v1:172.21.15.102:6789/0 conn(0x7f752c1597f0 0x7f752c0cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40030/0 (socket says 172.21.15.102:40030) 2023-03-03T05:57:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7530f9c700 1 -- 172.21.15.102:0/795474513 >> v2:172.21.15.172:3300/0 conn(0x7f752c0ced80 msgr2=0x7f752c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:57:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7530f9c700 1 --2- 172.21.15.102:0/795474513 >> v2:172.21.15.172:3300/0 conn(0x7f752c0ced80 0x7f752c0cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7530f9c700 1 -- 172.21.15.102:0/795474513 >> v1:172.21.15.102:6789/0 conn(0x7f752c1597f0 legacy=0x7f752c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:57:17.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f7530f9c700 1 -- 172.21.15.102:0/795474513 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c159c80 con 0x7f752c0cf9a0 2023-03-03T05:57:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.209+0000 7f752b7fe700 1 -- 172.21.15.102:0/795474513 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693679358 0 0) 0x7f752c0cb9f0 con 0x7f752c1597f0 2023-03-03T05:57:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.210+0000 7f7530f9c700 1 --2- 172.21.15.102:0/795474513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0cf9a0 0x7f752c163660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f752c0699d0 tx=0x7f752400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab0df52996b25bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:57:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.210+0000 7f752b7fe700 1 -- 172.21.15.102:0/795474513 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752400e070 con 0x7f752c0cf9a0 2023-03-03T05:57:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.210+0000 7f752b7fe700 1 -- 172.21.15.102:0/795474513 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752400b910 con 0x7f752c0cf9a0 2023-03-03T05:57:17.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.210+0000 7f752b7fe700 1 -- 172.21.15.102:0/795474513 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752400bcc0 con 0x7f752c0cf9a0 2023-03-03T05:57:17.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.210+0000 7f75329ff700 1 -- 172.21.15.102:0/795474513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0cf9a0 msgr2=0x7f752c163660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:57:17.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.210+0000 7f75329ff700 1 --2- 172.21.15.102:0/795474513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0cf9a0 0x7f752c163660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f752c0699d0 tx=0x7f752400b330 comp rx=0 tx=0).stop 2023-03-03T05:57:17.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.211+0000 7f75329ff700 1 -- 172.21.15.102:0/795474513 shutdown_connections 2023-03-03T05:57:17.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.211+0000 7f75329ff700 1 --2- 172.21.15.102:0/795474513 >> v2:172.21.15.172:3300/0 conn(0x7f752c0ced80 0x7f752c0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.211+0000 7f75329ff700 1 --2- 172.21.15.102:0/795474513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0cf9a0 0x7f752c163660 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.211+0000 7f75329ff700 1 -- 172.21.15.102:0/795474513 >> 172.21.15.102:0/795474513 conn(0x7f752c065fb0 msgr2=0x7f752c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:57:17.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.211+0000 7f75329ff700 1 -- 172.21.15.102:0/795474513 shutdown_connections 2023-03-03T05:57:17.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.211+0000 7f75329ff700 1 -- 172.21.15.102:0/795474513 wait complete. 2023-03-03T05:57:17.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.213+0000 7f75329ff700 1 Processor -- start 2023-03-03T05:57:17.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.213+0000 7f75329ff700 1 -- start start 2023-03-03T05:57:17.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f75329ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0ced80 0x7f752c14be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:57:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f75329ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f752c0cf9a0 0x7f752c0b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:57:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f75329ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f752c1597f0 0x7f752c0b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:57:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f75329ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c14c820 con 0x7f752c0ced80 2023-03-03T05:57:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f75329ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c14af40 con 0x7f752c1597f0 2023-03-03T05:57:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f75329ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752c14c3d0 con 0x7f752c0cf9a0 2023-03-03T05:57:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f752bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0ced80 0x7f752c14be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:57:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f752bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0ced80 0x7f752c14be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48414/0 (socket says 172.21.15.102:48414) 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f752bfff700 1 -- 172.21.15.102:0/2013688604 learned_addr learned my addr 172.21.15.102:0/2013688604 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f7523fff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f752c0cf9a0 0x7f752c0b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.214+0000 7f7530f9c700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f752c1597f0 0x7f752c0b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f752bfff700 1 -- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f752c0cf9a0 msgr2=0x7f752c0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f752bfff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f752c0cf9a0 0x7f752c0b2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f752bfff700 1 -- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f752c1597f0 msgr2=0x7f752c0b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f752bfff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f752c1597f0 0x7f752c0b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f752bfff700 1 -- 172.21.15.102:0/2013688604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7524009b30 con 0x7f752c0ced80 2023-03-03T05:57:17.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f752bfff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0ced80 0x7f752c14be90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f751c00e870 tx=0x7f751c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:57:17.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f75297fa700 1 -- 172.21.15.102:0/2013688604 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c009590 con 0x7f752c0ced80 2023-03-03T05:57:17.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f75297fa700 1 -- 172.21.15.102:0/2013688604 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751c013680 con 0x7f752c0ced80 2023-03-03T05:57:17.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c0cb0c0 con 0x7f752c0ced80 2023-03-03T05:57:17.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.215+0000 7f75297fa700 1 -- 172.21.15.102:0/2013688604 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c0139f0 con 0x7f752c0ced80 2023-03-03T05:57:17.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.216+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c0cade0 con 0x7f752c0ced80 2023-03-03T05:57:17.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.217+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752c0cb3a0 con 0x7f752c0ced80 2023-03-03T05:57:17.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.218+0000 7f75297fa700 1 -- 172.21.15.102:0/2013688604 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f751c019030 con 0x7f752c0ced80 2023-03-03T05:57:17.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.219+0000 7f75297fa700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f750c0682f0 0x7f750c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:57:17.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.219+0000 7f75297fa700 1 -- 172.21.15.102:0/2013688604 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f751c085b00 con 0x7f752c0ced80 2023-03-03T05:57:17.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.219+0000 7f7523fff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f750c0682f0 0x7f750c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:57:17.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.220+0000 7f7523fff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f750c0682f0 0x7f750c06a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f752c0cde40 tx=0x7f7514007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:57:17.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.223+0000 7f75297fa700 1 -- 172.21.15.102:0/2013688604 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f751c087e60 con 0x7f752c0ced80 2023-03-03T05:57:17.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.459+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f752c0b4a20 con 0x7f752c0ced80 2023-03-03T05:57:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.460+0000 7f75297fa700 1 -- 172.21.15.102:0/2013688604 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f751c0521b0 con 0x7f752c0ced80 2023-03-03T05:57:17.462 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:57:17.462 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:57:17.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f750c0682f0 msgr2=0x7f750c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:57:17.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f750c0682f0 0x7f750c06a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f752c0cde40 tx=0x7f7514007e70 comp rx=0 tx=0).stop 2023-03-03T05:57:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0ced80 msgr2=0x7f752c14be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:57:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0ced80 0x7f752c14be90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f751c00e870 tx=0x7f751c00eb80 comp rx=0 tx=0).stop 2023-03-03T05:57:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 shutdown_connections 2023-03-03T05:57:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f752c1597f0 0x7f752c0b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f752c0ced80 0x7f752c14be90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f750c0682f0 0x7f750c06a7a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 --2- 172.21.15.102:0/2013688604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f752c0cf9a0 0x7f752c0b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:57:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 >> 172.21.15.102:0/2013688604 conn(0x7f752c065fb0 msgr2=0x7f752c1515c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:57:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 shutdown_connections 2023-03-03T05:57:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:57:17.463+0000 7f75329ff700 1 -- 172.21.15.102:0/2013688604 wait complete. 2023-03-03T05:57:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:57:17.492 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-03-03T05:57:30.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:57:30.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:58:01.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:58:01.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:58:31.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:58:31.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:58:53.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T05:58:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.740+0000 7f7f8abff700 1 Processor -- start 2023-03-03T05:58:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.741+0000 7f7f8abff700 1 -- start start 2023-03-03T05:58:53.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8abff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f84153290 0x7f7f84158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:58:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8abff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f841593c0 0x7f7f840cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:58:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8abff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f84153660 con 0x7f7f840cd8e0 2023-03-03T05:58:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8abff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8406c360 con 0x7f7f84153290 2023-03-03T05:58:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8abff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8415cea0 con 0x7f7f841593c0 2023-03-03T05:58:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8899b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f84153290 0x7f7f84158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:58:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8899b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f84153290 0x7f7f84158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55628/0 (socket says 172.21.15.102:55628) 2023-03-03T05:58:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8899b700 1 -- 172.21.15.102:0/4148519620 learned_addr learned my addr 172.21.15.102:0/4148519620 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:58:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f8919c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7f840cd8e0 0x7f7f840ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56296/0 (socket says 172.21.15.102:56296) 2023-03-03T05:58:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.742+0000 7f7f83fff700 1 --2- 172.21.15.102:0/4148519620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f841593c0 0x7f7f840cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:58:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.743+0000 7f7f8899b700 1 -- 172.21.15.102:0/4148519620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f841593c0 msgr2=0x7f7f840cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T05:58:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.743+0000 7f7f8899b700 1 --2- 172.21.15.102:0/4148519620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f841593c0 0x7f7f840cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.743+0000 7f7f8899b700 1 -- 172.21.15.102:0/4148519620 >> v1:172.21.15.102:6789/0 conn(0x7f7f840cd8e0 legacy=0x7f7f840ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:58:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.743+0000 7f7f8899b700 1 -- 172.21.15.102:0/4148519620 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8406c7f0 con 0x7f7f84153290 2023-03-03T05:58:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.743+0000 7f7f837fe700 1 -- 172.21.15.102:0/4148519620 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560733699 0 0) 0x7f7f84153660 con 0x7f7f840cd8e0 2023-03-03T05:58:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.743+0000 7f7f8899b700 1 --2- 172.21.15.102:0/4148519620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f84153290 0x7f7f84158e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f780098f0 tx=0x7f7f78009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83583e3a1f8960d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:58:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f837fe700 1 -- 172.21.15.102:0/4148519620 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f78011830 con 0x7f7f84153290 2023-03-03T05:58:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f837fe700 1 -- 172.21.15.102:0/4148519620 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f78011990 con 0x7f7f84153290 2023-03-03T05:58:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f837fe700 1 -- 172.21.15.102:0/4148519620 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f78011d20 con 0x7f7f84153290 2023-03-03T05:58:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 -- 172.21.15.102:0/4148519620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f84153290 msgr2=0x7f7f84158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:58:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 --2- 172.21.15.102:0/4148519620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f84153290 0x7f7f84158e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f780098f0 tx=0x7f7f78009c00 comp rx=0 tx=0).stop 2023-03-03T05:58:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 -- 172.21.15.102:0/4148519620 shutdown_connections 2023-03-03T05:58:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 --2- 172.21.15.102:0/4148519620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f84153290 0x7f7f84158e80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 --2- 172.21.15.102:0/4148519620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f841593c0 0x7f7f840cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 -- 172.21.15.102:0/4148519620 >> 172.21.15.102:0/4148519620 conn(0x7f7f84065fb0 msgr2=0x7f7f840663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:58:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 -- 172.21.15.102:0/4148519620 shutdown_connections 2023-03-03T05:58:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.744+0000 7f7f8abff700 1 -- 172.21.15.102:0/4148519620 wait complete. 2023-03-03T05:58:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.746+0000 7f7f8abff700 1 Processor -- start 2023-03-03T05:58:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.747+0000 7f7f8abff700 1 -- start start 2023-03-03T05:58:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8abff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f840cd8e0 0x7f7f840c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:58:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8abff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f84153290 0x7f7f8414e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:58:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8abff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f841593c0 0x7f7f8414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:58:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8abff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8414dad0 con 0x7f7f840cd8e0 2023-03-03T05:58:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8abff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8414c100 con 0x7f7f841593c0 2023-03-03T05:58:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8abff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8414c3a0 con 0x7f7f84153290 2023-03-03T05:58:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8899b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f840cd8e0 0x7f7f840c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:58:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8899b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f840cd8e0 0x7f7f840c88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55642/0 (socket says 172.21.15.102:55642) 2023-03-03T05:58:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8899b700 1 -- 172.21.15.102:0/123854480 learned_addr learned my addr 172.21.15.102:0/123854480 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T05:58:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f83fff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f84153290 0x7f7f8414e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:58:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.748+0000 7f7f8919c700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f841593c0 0x7f7f8414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:58:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f8899b700 1 -- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f84153290 msgr2=0x7f7f8414e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:58:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f8899b700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f84153290 0x7f7f8414e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f8899b700 1 -- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f841593c0 msgr2=0x7f7f8414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:58:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f8899b700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f841593c0 0x7f7f8414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f8899b700 1 -- 172.21.15.102:0/123854480 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f78009550 con 0x7f7f840cd8e0 2023-03-03T05:58:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f83fff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f84153290 0x7f7f8414e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T05:58:53.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f8899b700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f840cd8e0 0x7f7f840c88f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7f84074540 tx=0x7f7f78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:58:53.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f817fa700 1 -- 172.21.15.102:0/123854480 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f78004400 con 0x7f7f840cd8e0 2023-03-03T05:58:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f817fa700 1 -- 172.21.15.102:0/123854480 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7801f070 con 0x7f7f840cd8e0 2023-03-03T05:58:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f817fa700 1 -- 172.21.15.102:0/123854480 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f780038e0 con 0x7f7f840cd8e0 2023-03-03T05:58:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.749+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f840ccde0 con 0x7f7f840cd8e0 2023-03-03T05:58:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.750+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8415d160 con 0x7f7f840cd8e0 2023-03-03T05:58:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.751+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f840cd0c0 con 0x7f7f840cd8e0 2023-03-03T05:58:53.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.752+0000 7f7f817fa700 1 -- 172.21.15.102:0/123854480 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f7802a020 con 0x7f7f840cd8e0 2023-03-03T05:58:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.753+0000 7f7f817fa700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f680682f0 0x7f7f6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T05:58:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.753+0000 7f7f817fa700 1 -- 172.21.15.102:0/123854480 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7f78087330 con 0x7f7f840cd8e0 2023-03-03T05:58:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.754+0000 7f7f83fff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f680682f0 0x7f7f6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T05:58:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.754+0000 7f7f83fff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f680682f0 0x7f7f6806a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7f840c5a20 tx=0x7f7f70009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T05:58:53.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.758+0000 7f7f817fa700 1 -- 172.21.15.102:0/123854480 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f78053a40 con 0x7f7f840cd8e0 2023-03-03T05:58:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.990+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f8404e6e0 con 0x7f7f840cd8e0 2023-03-03T05:58:53.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.991+0000 7f7f817fa700 1 -- 172.21.15.102:0/123854480 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7f78057060 con 0x7f7f840cd8e0 2023-03-03T05:58:53.993 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T05:58:53.993 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T05:58:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f680682f0 msgr2=0x7f7f6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:58:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f680682f0 0x7f7f6806a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7f840c5a20 tx=0x7f7f70009210 comp rx=0 tx=0).stop 2023-03-03T05:58:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f840cd8e0 msgr2=0x7f7f840c88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T05:58:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f840cd8e0 0x7f7f840c88f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7f84074540 tx=0x7f7f78004750 comp rx=0 tx=0).stop 2023-03-03T05:58:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 shutdown_connections 2023-03-03T05:58:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f841593c0 0x7f7f8414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f840cd8e0 0x7f7f840c88f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f680682f0 0x7f7f6806a7a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 --2- 172.21.15.102:0/123854480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f84153290 0x7f7f8414e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T05:58:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 >> 172.21.15.102:0/123854480 conn(0x7f7f84065fb0 msgr2=0x7f7f840cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T05:58:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 shutdown_connections 2023-03-03T05:58:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T05:58:53.994+0000 7f7f8abff700 1 -- 172.21.15.102:0/123854480 wait complete. 2023-03-03T05:58:53.998 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T05:58:54.023 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2023-03-03T05:59:01.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:59:01.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:59:31.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T05:59:31.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:00:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T05:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:00:01.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:00:01.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:00:31.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:00:31.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:00:51.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:00:51.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.291+0000 7f4a3dbff700 1 Processor -- start 2023-03-03T06:00:51.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.291+0000 7f4a3dbff700 1 -- start start 2023-03-03T06:00:51.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a3dbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4a38153290 0x7f4a38158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:00:51.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a3dbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4a380cd8e0 0x7f4a380ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:00:51.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a3dbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a38153660 con 0x7f4a381593c0 2023-03-03T06:00:51.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a3dbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3806c360 con 0x7f4a38153290 2023-03-03T06:00:51.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a3dbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3815cea0 con 0x7f4a380cd8e0 2023-03-03T06:00:51.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a377fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4a38153290 0x7f4a38158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:00:51.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a377fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4a38153290 0x7f4a38158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46882/0 (socket says 172.21.15.102:46882) 2023-03-03T06:00:51.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a377fe700 1 -- 172.21.15.102:0/1450509897 learned_addr learned my addr 172.21.15.102:0/1450509897 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:00:51.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.293+0000 7f4a37fff700 1 --2- 172.21.15.102:0/1450509897 >> v2:172.21.15.172:3300/0 conn(0x7f4a380cd8e0 0x7f4a380ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:00:51.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a377fe700 1 -- 172.21.15.102:0/1450509897 >> v2:172.21.15.172:3300/0 conn(0x7f4a380cd8e0 msgr2=0x7f4a380ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:00:51.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a377fe700 1 --2- 172.21.15.102:0/1450509897 >> v2:172.21.15.172:3300/0 conn(0x7f4a380cd8e0 0x7f4a380ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a377fe700 1 -- 172.21.15.102:0/1450509897 >> v1:172.21.15.172:6789/0 conn(0x7f4a381593c0 legacy=0x7f4a380cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:00:51.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a377fe700 1 -- 172.21.15.102:0/1450509897 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3806c7f0 con 0x7f4a38153290 2023-03-03T06:00:51.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a367fc700 1 -- 172.21.15.102:0/1450509897 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482565255 0 0) 0x7f4a38153660 con 0x7f4a381593c0 2023-03-03T06:00:51.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a377fe700 1 --2- 172.21.15.102:0/1450509897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a38153290 0x7f4a38158e80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4a380c9390 tx=0x7f4a28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a728dcc43c51a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:00:51.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a367fc700 1 -- 172.21.15.102:0/1450509897 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28011830 con 0x7f4a38153290 2023-03-03T06:00:51.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a367fc700 1 -- 172.21.15.102:0/1450509897 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a28011990 con 0x7f4a38153290 2023-03-03T06:00:51.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.294+0000 7f4a367fc700 1 -- 172.21.15.102:0/1450509897 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28011d20 con 0x7f4a38153290 2023-03-03T06:00:51.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 -- 172.21.15.102:0/1450509897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a38153290 msgr2=0x7f4a38158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:00:51.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/1450509897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a38153290 0x7f4a38158e80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4a380c9390 tx=0x7f4a28009bd0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 -- 172.21.15.102:0/1450509897 shutdown_connections 2023-03-03T06:00:51.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/1450509897 >> v2:172.21.15.172:3300/0 conn(0x7f4a380cd8e0 0x7f4a380ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/1450509897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a38153290 0x7f4a38158e80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 -- 172.21.15.102:0/1450509897 >> 172.21.15.102:0/1450509897 conn(0x7f4a38065fb0 msgr2=0x7f4a380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:00:51.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 -- 172.21.15.102:0/1450509897 shutdown_connections 2023-03-03T06:00:51.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.295+0000 7f4a3dbff700 1 -- 172.21.15.102:0/1450509897 wait complete. 2023-03-03T06:00:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.297+0000 7f4a3dbff700 1 Processor -- start 2023-03-03T06:00:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.297+0000 7f4a3dbff700 1 -- start start 2023-03-03T06:00:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.298+0000 7f4a3dbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a380cd8e0 0x7f4a380b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:00:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.298+0000 7f4a3dbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a38153290 0x7f4a380c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:00:51.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.298+0000 7f4a3dbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a381593c0 0x7f4a3814dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:00:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.298+0000 7f4a3dbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a381eef20 con 0x7f4a381593c0 2023-03-03T06:00:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.298+0000 7f4a3dbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a381ee650 con 0x7f4a380cd8e0 2023-03-03T06:00:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.298+0000 7f4a3dbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38156130 con 0x7f4a38153290 2023-03-03T06:00:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a380cd8e0 0x7f4a380b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:00:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a380cd8e0 0x7f4a380b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46860/0 (socket says 172.21.15.102:46860) 2023-03-03T06:00:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 -- 172.21.15.102:0/4013206688 learned_addr learned my addr 172.21.15.102:0/4013206688 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:00:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a36ffd700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a38153290 0x7f4a380c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:00:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a37fff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a381593c0 0x7f4a3814dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:00:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 -- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a38153290 msgr2=0x7f4a380c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:00:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a38153290 0x7f4a380c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 -- 172.21.15.102:0/4013206688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a381593c0 msgr2=0x7f4a3814dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:00:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a381593c0 0x7f4a3814dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.299+0000 7f4a377fe700 1 -- 172.21.15.102:0/4013206688 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a28009550 con 0x7f4a380cd8e0 2023-03-03T06:00:51.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.300+0000 7f4a377fe700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a380cd8e0 0x7f4a380b2cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009fd0 tx=0x7f4a28004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:00:51.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.300+0000 7f4a1ffff700 1 -- 172.21.15.102:0/4013206688 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280036a0 con 0x7f4a380cd8e0 2023-03-03T06:00:51.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.300+0000 7f4a1ffff700 1 -- 172.21.15.102:0/4013206688 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2801f070 con 0x7f4a380cd8e0 2023-03-03T06:00:51.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.300+0000 7f4a1ffff700 1 -- 172.21.15.102:0/4013206688 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28003a20 con 0x7f4a380cd8e0 2023-03-03T06:00:51.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.300+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a380ccde0 con 0x7f4a380cd8e0 2023-03-03T06:00:51.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.300+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3815d160 con 0x7f4a380cd8e0 2023-03-03T06:00:51.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.301+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a380cd0c0 con 0x7f4a380cd8e0 2023-03-03T06:00:51.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.302+0000 7f4a1ffff700 1 -- 172.21.15.102:0/4013206688 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4a2800f050 con 0x7f4a380cd8e0 2023-03-03T06:00:51.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.303+0000 7f4a1ffff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4a24068320 0x7f4a2406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:00:51.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.304+0000 7f4a1ffff700 1 -- 172.21.15.102:0/4013206688 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4a28021070 con 0x7f4a380cd8e0 2023-03-03T06:00:51.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.304+0000 7f4a36ffd700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4a24068320 0x7f4a2406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:00:51.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.305+0000 7f4a36ffd700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4a24068320 0x7f4a2406a7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a38157320 tx=0x7f4a2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:00:51.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.308+0000 7f4a1ffff700 1 -- 172.21.15.102:0/4013206688 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a28053be0 con 0x7f4a380cd8e0 2023-03-03T06:00:51.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.543+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a38157370 con 0x7f4a380cd8e0 2023-03-03T06:00:51.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.544+0000 7f4a1ffff700 1 -- 172.21.15.102:0/4013206688 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4a28015090 con 0x7f4a380cd8e0 2023-03-03T06:00:51.548 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:00:51.548 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:00:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4a24068320 msgr2=0x7f4a2406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:00:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4a24068320 0x7f4a2406a7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a38157320 tx=0x7f4a2c009040 comp rx=0 tx=0).stop 2023-03-03T06:00:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a380cd8e0 msgr2=0x7f4a380b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:00:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a380cd8e0 0x7f4a380b2cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009fd0 tx=0x7f4a28004750 comp rx=0 tx=0).stop 2023-03-03T06:00:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 shutdown_connections 2023-03-03T06:00:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a381593c0 0x7f4a3814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a380cd8e0 0x7f4a380b2cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4a24068320 0x7f4a2406a7d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 --2- 172.21.15.102:0/4013206688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4a38153290 0x7f4a380c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:00:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 >> 172.21.15.102:0/4013206688 conn(0x7f4a38065fb0 msgr2=0x7f4a38150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:00:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 shutdown_connections 2023-03-03T06:00:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:00:51.547+0000 7f4a3dbff700 1 -- 172.21.15.102:0/4013206688 wait complete. 2023-03-03T06:00:51.552 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:00:51.579 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 50 secs before thrashing 2023-03-03T06:01:02.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:01:02.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:01:32.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:01:32.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:01:41.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:01:41.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.844+0000 7f6506dff700 1 Processor -- start 2023-03-03T06:01:41.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.845+0000 7f6506dff700 1 -- start start 2023-03-03T06:01:41.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f6506dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6500153290 0x7f6500158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:41.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f6506dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65001593c0 0x7f65000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:41.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f6506dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65000cd8e0 0x7f65000ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:41.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f6506dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6500153660 con 0x7f65001593c0 2023-03-03T06:01:41.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f6506dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f650006c360 con 0x7f65000cd8e0 2023-03-03T06:01:41.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f6506dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650015cea0 con 0x7f6500153290 2023-03-03T06:01:41.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f650539c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65000cd8e0 0x7f65000ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:41.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f6504b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6500153290 0x7f6500158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:41.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f64fffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65001593c0 0x7f65000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:41.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f650539c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65000cd8e0 0x7f65000ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:49708/0 (socket says 172.21.15.102:49708) 2023-03-03T06:01:41.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.846+0000 7f650539c700 1 -- 172.21.15.102:0/4158818011 learned_addr learned my addr 172.21.15.102:0/4158818011 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:01:41.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.847+0000 7f650539c700 1 -- 172.21.15.102:0/4158818011 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6500153290 msgr2=0x7f6500158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:01:41.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.847+0000 7f650539c700 1 --2- 172.21.15.102:0/4158818011 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6500153290 0x7f6500158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:41.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.847+0000 7f650539c700 1 -- 172.21.15.102:0/4158818011 >> v2:172.21.15.102:3300/0 conn(0x7f65001593c0 msgr2=0x7f65000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:01:41.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.847+0000 7f650539c700 1 --2- 172.21.15.102:0/4158818011 >> v2:172.21.15.102:3300/0 conn(0x7f65001593c0 0x7f65000cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:41.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.847+0000 7f650539c700 1 -- 172.21.15.102:0/4158818011 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650006c7f0 con 0x7f65000cd8e0 2023-03-03T06:01:41.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.847+0000 7f650539c700 1 --2- 172.21.15.102:0/4158818011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 0x7f65000ce360 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f650006d0e0 tx=0x7f64f800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38697cfec9755be0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:01:41.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.847+0000 7f64ff7fe700 1 -- 172.21.15.102:0/4158818011 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f800e070 con 0x7f65000cd8e0 2023-03-03T06:01:41.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f64ff7fe700 1 -- 172.21.15.102:0/4158818011 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f800b910 con 0x7f65000cd8e0 2023-03-03T06:01:41.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f64ff7fe700 1 -- 172.21.15.102:0/4158818011 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f800bcc0 con 0x7f65000cd8e0 2023-03-03T06:01:41.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 -- 172.21.15.102:0/4158818011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 msgr2=0x7f65000ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:41.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 --2- 172.21.15.102:0/4158818011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 0x7f65000ce360 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f650006d0e0 tx=0x7f64f800b330 comp rx=0 tx=0).stop 2023-03-03T06:01:41.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 -- 172.21.15.102:0/4158818011 shutdown_connections 2023-03-03T06:01:41.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 --2- 172.21.15.102:0/4158818011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 0x7f65000ce360 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 --2- 172.21.15.102:0/4158818011 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6500153290 0x7f6500158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 --2- 172.21.15.102:0/4158818011 >> v2:172.21.15.102:3300/0 conn(0x7f65001593c0 0x7f65000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 -- 172.21.15.102:0/4158818011 >> 172.21.15.102:0/4158818011 conn(0x7f6500065fb0 msgr2=0x7f65000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:01:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 -- 172.21.15.102:0/4158818011 shutdown_connections 2023-03-03T06:01:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.848+0000 7f6506dff700 1 -- 172.21.15.102:0/4158818011 wait complete. 2023-03-03T06:01:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.850+0000 7f6506dff700 1 Processor -- start 2023-03-03T06:01:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.851+0000 7f6506dff700 1 -- start start 2023-03-03T06:01:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f6506dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 0x7f65000b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f6506dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6500153290 0x7f65000b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f6506dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65001593c0 0x7f65000b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f6506dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650014ffd0 con 0x7f6500153290 2023-03-03T06:01:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f6506dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65000b2360 con 0x7f65000cd8e0 2023-03-03T06:01:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f6506dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65000c6330 con 0x7f65001593c0 2023-03-03T06:01:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f64fffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6500153290 0x7f65000b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f64fffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6500153290 0x7f65000b27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50728/0 (socket says 172.21.15.102:50728) 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f64fffff700 1 -- 172.21.15.102:0/1756818606 learned_addr learned my addr 172.21.15.102:0/1756818606 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f650539c700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65001593c0 0x7f65000b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.852+0000 7f6504b9b700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 0x7f65000b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fffff700 1 -- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65001593c0 msgr2=0x7f65000b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fffff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65001593c0 0x7f65000b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fffff700 1 -- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 msgr2=0x7f65000b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fffff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 0x7f65000b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fffff700 1 -- 172.21.15.102:0/1756818606 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f8009b30 con 0x7f6500153290 2023-03-03T06:01:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fffff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6500153290 0x7f65000b27f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00e9e0 tx=0x7f64ec00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:01:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fd7fa700 1 -- 172.21.15.102:0/1756818606 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec009710 con 0x7f6500153290 2023-03-03T06:01:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fd7fa700 1 -- 172.21.15.102:0/1756818606 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec00ee80 con 0x7f6500153290 2023-03-03T06:01:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f64fd7fa700 1 -- 172.21.15.102:0/1756818606 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec00ca50 con 0x7f6500153290 2023-03-03T06:01:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65000ccde0 con 0x7f6500153290 2023-03-03T06:01:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.853+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650015d160 con 0x7f6500153290 2023-03-03T06:01:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.854+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65000cd0c0 con 0x7f6500153290 2023-03-03T06:01:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.856+0000 7f64fd7fa700 1 -- 172.21.15.102:0/1756818606 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f64ec019030 con 0x7f6500153290 2023-03-03T06:01:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.857+0000 7f64fd7fa700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64e4068320 0x7f64e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:41.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.857+0000 7f64fd7fa700 1 -- 172.21.15.102:0/1756818606 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f64ec085be0 con 0x7f6500153290 2023-03-03T06:01:41.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.857+0000 7f6504b9b700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64e4068320 0x7f64e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:41.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.858+0000 7f6504b9b700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64e4068320 0x7f64e406a7d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64f4009b40 tx=0x7f64f4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:01:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:41.861+0000 7f64fd7fa700 1 -- 172.21.15.102:0/1756818606 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f64ec052290 con 0x7f6500153290 2023-03-03T06:01:42.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.093+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6500156cf0 con 0x7f6500153290 2023-03-03T06:01:42.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.094+0000 7f64fd7fa700 1 -- 172.21.15.102:0/1756818606 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f64ec0558b0 con 0x7f6500153290 2023-03-03T06:01:42.097 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:01:42.097 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:01:42.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64e4068320 msgr2=0x7f64e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:42.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64e4068320 0x7f64e406a7d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64f4009b40 tx=0x7f64f4009210 comp rx=0 tx=0).stop 2023-03-03T06:01:42.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6500153290 msgr2=0x7f65000b27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:42.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6500153290 0x7f65000b27f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00e9e0 tx=0x7f64ec00ecf0 comp rx=0 tx=0).stop 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 shutdown_connections 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64e4068320 0x7f64e406a7d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65001593c0 0x7f65000b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65000cd8e0 0x7f65000b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 --2- 172.21.15.102:0/1756818606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6500153290 0x7f65000b27f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 >> 172.21.15.102:0/1756818606 conn(0x7f6500065fb0 msgr2=0x7f6500157770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 shutdown_connections 2023-03-03T06:01:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:42.097+0000 7f6506dff700 1 -- 172.21.15.102:0/1756818606 wait complete. 2023-03-03T06:01:42.101 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:01:42.126 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T06:01:51.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:01:51.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.376+0000 7fa9c259d700 1 Processor -- start 2023-03-03T06:01:51.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.376+0000 7fa9c259d700 1 -- start start 2023-03-03T06:01:51.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.378+0000 7fa9c259d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc0cab50 0x7fa9bc1646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:51.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.378+0000 7fa9c259d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc0c9ca0 con 0x7fa9bc1564b0 2023-03-03T06:01:51.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.378+0000 7fa9c259d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc0b46f0 con 0x7fa9bc0c9610 2023-03-03T06:01:51.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.378+0000 7fa9c259d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc0b4920 con 0x7fa9bc0cab50 2023-03-03T06:01:51.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.378+0000 7fa9c159b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa9bc1564b0 0x7fa9bc0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46576/0 (socket says 172.21.15.102:46576) 2023-03-03T06:01:51.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.378+0000 7fa9c159b700 1 -- 172.21.15.102:0/2038947177 learned_addr learned my addr 172.21.15.102:0/2038947177 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:01:51.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.378+0000 7fa9c1d9c700 1 --2- 172.21.15.102:0/2038947177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc0cab50 0x7fa9bc1646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:51.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.379+0000 7fa9b3fff700 1 -- 172.21.15.102:0/2038947177 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709463174 0 0) 0x7fa9bc0b46f0 con 0x7fa9bc0c9610 2023-03-03T06:01:51.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.379+0000 7fa9b3fff700 1 -- 172.21.15.102:0/2038947177 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a80035b0 con 0x7fa9bc0c9610 2023-03-03T06:01:51.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.379+0000 7fa9c1d9c700 1 -- 172.21.15.102:0/2038947177 >> v1:172.21.15.172:6789/0 conn(0x7fa9bc0c9610 legacy=0x7fa9bc0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:51.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.379+0000 7fa9b3fff700 1 -- 172.21.15.102:0/2038947177 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466956983 0 0) 0x7fa9bc0c9ca0 con 0x7fa9bc1564b0 2023-03-03T06:01:51.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.379+0000 7fa9c1d9c700 1 -- 172.21.15.102:0/2038947177 >> v1:172.21.15.102:6789/0 conn(0x7fa9bc1564b0 legacy=0x7fa9bc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:51.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.379+0000 7fa9c1d9c700 1 -- 172.21.15.102:0/2038947177 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc0b4c00 con 0x7fa9bc0cab50 2023-03-03T06:01:51.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.379+0000 7fa9c1d9c700 1 --2- 172.21.15.102:0/2038947177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc0cab50 0x7fa9bc1646b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009dc0 tx=0x7fa9b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16697ea4f64d6356 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:01:51.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9b3fff700 1 -- 172.21.15.102:0/2038947177 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b400e070 con 0x7fa9bc0cab50 2023-03-03T06:01:51.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9b3fff700 1 -- 172.21.15.102:0/2038947177 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b400bb20 con 0x7fa9bc0cab50 2023-03-03T06:01:51.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9b3fff700 1 -- 172.21.15.102:0/2038947177 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b400bed0 con 0x7fa9bc0cab50 2023-03-03T06:01:51.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9c259d700 1 -- 172.21.15.102:0/2038947177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc0cab50 msgr2=0x7fa9bc1646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:51.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2038947177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc0cab50 0x7fa9bc1646b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009dc0 tx=0x7fa9b400b330 comp rx=0 tx=0).stop 2023-03-03T06:01:51.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9c259d700 1 -- 172.21.15.102:0/2038947177 shutdown_connections 2023-03-03T06:01:51.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2038947177 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc0cab50 0x7fa9bc1646b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:51.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9c259d700 1 -- 172.21.15.102:0/2038947177 >> 172.21.15.102:0/2038947177 conn(0x7fa9bc065fb0 msgr2=0x7fa9bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:01:51.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9c259d700 1 -- 172.21.15.102:0/2038947177 shutdown_connections 2023-03-03T06:01:51.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.380+0000 7fa9c259d700 1 -- 172.21.15.102:0/2038947177 wait complete. 2023-03-03T06:01:51.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.382+0000 7fa9c259d700 1 Processor -- start 2023-03-03T06:01:51.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.383+0000 7fa9c259d700 1 -- start start 2023-03-03T06:01:51.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c259d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9bc0c9610 0x7fa9bc1ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:51.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c259d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc0cab50 0x7fa9bc0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c259d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc1564b0 0x7fa9bc1f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c259d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc152e50 con 0x7fa9bc0cab50 2023-03-03T06:01:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c259d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc1597c0 con 0x7fa9bc0c9610 2023-03-03T06:01:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c259d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc1ee8f0 con 0x7fa9bc1564b0 2023-03-03T06:01:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9bc0c9610 0x7fa9bc1ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9bc0c9610 0x7fa9bc1ee2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59596/0 (socket says 172.21.15.102:59596) 2023-03-03T06:01:51.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 -- 172.21.15.102:0/2275656684 learned_addr learned my addr 172.21.15.102:0/2275656684 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:01:51.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c0d9a700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc0cab50 0x7fa9bc0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:51.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c1d9c700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc1564b0 0x7fa9bc1f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:51.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 -- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc1564b0 msgr2=0x7fa9bc1f2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:51.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc1564b0 0x7fa9bc1f2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:51.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 -- 172.21.15.102:0/2275656684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc0cab50 msgr2=0x7fa9bc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:51.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc0cab50 0x7fa9bc0b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:51.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.384+0000 7fa9c159b700 1 -- 172.21.15.102:0/2275656684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b4009a70 con 0x7fa9bc0c9610 2023-03-03T06:01:51.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.385+0000 7fa9c159b700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9bc0c9610 0x7fa9bc1ee2d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80011e0 tx=0x7fa9b800ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:01:51.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.385+0000 7fa9b1ffb700 1 -- 172.21.15.102:0/2275656684 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8004c80 con 0x7fa9bc0c9610 2023-03-03T06:01:51.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.385+0000 7fa9b1ffb700 1 -- 172.21.15.102:0/2275656684 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b8004e00 con 0x7fa9bc0c9610 2023-03-03T06:01:51.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.385+0000 7fa9b1ffb700 1 -- 172.21.15.102:0/2275656684 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8007310 con 0x7fa9bc0c9610 2023-03-03T06:01:51.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.385+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9bc0b2210 con 0x7fa9bc0c9610 2023-03-03T06:01:51.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.385+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9bc0b1f30 con 0x7fa9bc0c9610 2023-03-03T06:01:51.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.386+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9bc156880 con 0x7fa9bc0c9610 2023-03-03T06:01:51.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.387+0000 7fa9b1ffb700 1 -- 172.21.15.102:0/2275656684 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa9b801a020 con 0x7fa9bc0c9610 2023-03-03T06:01:51.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.389+0000 7fa9b1ffb700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9a0068320 0x7fa9a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:01:51.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.389+0000 7fa9b1ffb700 1 -- 172.21.15.102:0/2275656684 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa9b8085840 con 0x7fa9bc0c9610 2023-03-03T06:01:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.389+0000 7fa9c0d9a700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9a0068320 0x7fa9a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:01:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.390+0000 7fa9c0d9a700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9a0068320 0x7fa9a006a7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0c8f30 tx=0x7fa9ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:01:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.393+0000 7fa9b1ffb700 1 -- 172.21.15.102:0/2275656684 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9b8052000 con 0x7fa9bc0c9610 2023-03-03T06:01:51.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.629+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9bc0c9e60 con 0x7fa9bc0c9610 2023-03-03T06:01:51.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.630+0000 7fa9b1ffb700 1 -- 172.21.15.102:0/2275656684 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa9b8055620 con 0x7fa9bc0c9610 2023-03-03T06:01:51.632 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:01:51.632 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:01:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9a0068320 msgr2=0x7fa9a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9a0068320 0x7fa9a006a7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0c8f30 tx=0x7fa9ac009040 comp rx=0 tx=0).stop 2023-03-03T06:01:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9bc0c9610 msgr2=0x7fa9bc1ee2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:01:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9bc0c9610 0x7fa9bc1ee2d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80011e0 tx=0x7fa9b800ca30 comp rx=0 tx=0).stop 2023-03-03T06:01:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 shutdown_connections 2023-03-03T06:01:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9a0068320 0x7fa9a006a7d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9bc1564b0 0x7fa9bc1f2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9bc0c9610 0x7fa9bc1ee2d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:51.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 --2- 172.21.15.102:0/2275656684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc0cab50 0x7fa9bc0b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:01:51.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 >> 172.21.15.102:0/2275656684 conn(0x7fa9bc065fb0 msgr2=0x7fa9bc066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:01:51.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 shutdown_connections 2023-03-03T06:01:51.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:01:51.633+0000 7fa9c259d700 1 -- 172.21.15.102:0/2275656684 wait complete. 2023-03-03T06:01:51.636 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:01:51.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-03-03T06:02:02.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:02:02.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:02:32.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:02:32.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:03:02.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:03:02.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:03:33.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:03:33.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:03:33.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:03:33.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.918+0000 7fae9cfff700 1 Processor -- start 2023-03-03T06:03:33.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.919+0000 7fae9cfff700 1 -- start start 2023-03-03T06:03:33.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.920+0000 7fae9cfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae980cd390 0x7fae980ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:03:33.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.920+0000 7fae9cfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae98153660 con 0x7fae98153290 2023-03-03T06:03:33.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.920+0000 7fae9cfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9806c360 con 0x7fae980cd8e0 2023-03-03T06:03:33.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.920+0000 7fae9cfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9815cea0 con 0x7fae980cd390 2023-03-03T06:03:33.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.920+0000 7fae96d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae980cd390 0x7fae980ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:03:33.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.921+0000 7fae9659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fae98153290 0x7fae98158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56380/0 (socket says 172.21.15.102:56380) 2023-03-03T06:03:33.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.921+0000 7fae9659c700 1 -- 172.21.15.102:0/1056985059 learned_addr learned my addr 172.21.15.102:0/1056985059 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:03:33.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.921+0000 7fae96d9d700 1 -- 172.21.15.102:0/1056985059 >> v1:172.21.15.172:6789/0 conn(0x7fae980cd8e0 legacy=0x7fae980cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:03:33.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.921+0000 7fae96d9d700 1 -- 172.21.15.102:0/1056985059 >> v1:172.21.15.102:6789/0 conn(0x7fae98153290 legacy=0x7fae98158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:03:33.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.921+0000 7fae9559a700 1 -- 172.21.15.102:0/1056985059 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174372889 0 0) 0x7fae9806c360 con 0x7fae980cd8e0 2023-03-03T06:03:33.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.921+0000 7fae96d9d700 1 -- 172.21.15.102:0/1056985059 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9806c7f0 con 0x7fae980cd390 2023-03-03T06:03:33.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.921+0000 7fae96d9d700 1 --2- 172.21.15.102:0/1056985059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980ce530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fae90009d90 tx=0x7fae9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26b1fda04471aee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:03:33.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9559a700 1 -- 172.21.15.102:0/1056985059 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9000e070 con 0x7fae980cd390 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9559a700 1 -- 172.21.15.102:0/1056985059 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9000bbd0 con 0x7fae980cd390 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9559a700 1 -- 172.21.15.102:0/1056985059 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90012430 con 0x7fae980cd390 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9cfff700 1 -- 172.21.15.102:0/1056985059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 msgr2=0x7fae980ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1056985059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980ce530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fae90009d90 tx=0x7fae9000b330 comp rx=0 tx=0).stop 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9cfff700 1 -- 172.21.15.102:0/1056985059 shutdown_connections 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1056985059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980ce530 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9cfff700 1 -- 172.21.15.102:0/1056985059 >> 172.21.15.102:0/1056985059 conn(0x7fae98065fb0 msgr2=0x7fae980663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:03:33.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9cfff700 1 -- 172.21.15.102:0/1056985059 shutdown_connections 2023-03-03T06:03:33.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.922+0000 7fae9cfff700 1 -- 172.21.15.102:0/1056985059 wait complete. 2023-03-03T06:03:33.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.924+0000 7fae9cfff700 1 Processor -- start 2023-03-03T06:03:33.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.925+0000 7fae9cfff700 1 -- start start 2023-03-03T06:03:33.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.925+0000 7fae9cfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980cf790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:03:33.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.925+0000 7fae9cfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae980cd8e0 0x7fae9814e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae98153290 0x7fae981ec910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9cfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9814eab0 con 0x7fae980cd390 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9cfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae980cf500 con 0x7fae980cd8e0 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae980b4110 con 0x7fae98153290 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52302/0 (socket says 172.21.15.102:52302) 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 -- 172.21.15.102:0/1180730669 learned_addr learned my addr 172.21.15.102:0/1180730669 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:03:33.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae95d9b700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae980cd8e0 0x7fae9814e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:03:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae96d9d700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae98153290 0x7fae981ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:03:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 -- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae98153290 msgr2=0x7fae981ec910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:03:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae98153290 0x7fae981ec910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:03:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 -- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae980cd8e0 msgr2=0x7fae9814e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:03:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae980cd8e0 0x7fae9814e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:03:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.926+0000 7fae9659c700 1 -- 172.21.15.102:0/1180730669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae90009a70 con 0x7fae980cd390 2023-03-03T06:03:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.927+0000 7fae9659c700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980cf790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fae8800c480 tx=0x7fae8800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:03:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.927+0000 7fae86ffd700 1 -- 172.21.15.102:0/1180730669 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88005d70 con 0x7fae980cd390 2023-03-03T06:03:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.927+0000 7fae86ffd700 1 -- 172.21.15.102:0/1180730669 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8800ce30 con 0x7fae980cd390 2023-03-03T06:03:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.927+0000 7fae86ffd700 1 -- 172.21.15.102:0/1180730669 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae880072d0 con 0x7fae980cd390 2023-03-03T06:03:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.927+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae980ccde0 con 0x7fae980cd390 2023-03-03T06:03:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.927+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9815d160 con 0x7fae980cd390 2023-03-03T06:03:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.928+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae980cd0c0 con 0x7fae980cd390 2023-03-03T06:03:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.930+0000 7fae86ffd700 1 -- 172.21.15.102:0/1180730669 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae88019030 con 0x7fae980cd390 2023-03-03T06:03:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.931+0000 7fae86ffd700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae780682f0 0x7fae7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:03:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.931+0000 7fae86ffd700 1 -- 172.21.15.102:0/1180730669 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fae88084e90 con 0x7fae980cd390 2023-03-03T06:03:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.931+0000 7fae95d9b700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae780682f0 0x7fae7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:03:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.932+0000 7fae95d9b700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae780682f0 0x7fae7806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fae980cb970 tx=0x7fae8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:03:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:33.935+0000 7fae86ffd700 1 -- 172.21.15.102:0/1180730669 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae8804b5e0 con 0x7fae980cd390 2023-03-03T06:03:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.166+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae9804e6e0 con 0x7fae980cd390 2023-03-03T06:03:34.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.167+0000 7fae86ffd700 1 -- 172.21.15.102:0/1180730669 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fae88017020 con 0x7fae980cd390 2023-03-03T06:03:34.168 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:03:34.169 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:03:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae780682f0 msgr2=0x7fae7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:03:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae780682f0 0x7fae7806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fae980cb970 tx=0x7fae8c009040 comp rx=0 tx=0).stop 2023-03-03T06:03:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 msgr2=0x7fae980cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:03:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980cf790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fae8800c480 tx=0x7fae8800c790 comp rx=0 tx=0).stop 2023-03-03T06:03:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 shutdown_connections 2023-03-03T06:03:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae780682f0 0x7fae7806a7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:03:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae98153290 0x7fae981ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:03:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae980cd390 0x7fae980cf790 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:03:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 --2- 172.21.15.102:0/1180730669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae980cd8e0 0x7fae9814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:03:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 >> 172.21.15.102:0/1180730669 conn(0x7fae98065fb0 msgr2=0x7fae98156ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:03:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 shutdown_connections 2023-03-03T06:03:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:03:34.169+0000 7fae9cfff700 1 -- 172.21.15.102:0/1180730669 wait complete. 2023-03-03T06:03:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:03:34.198 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 72 secs before thrashing 2023-03-03T06:04:03.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:04:03.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:04:33.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:04:33.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:04:46.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:04:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.445+0000 7fd37cdff700 1 Processor -- start 2023-03-03T06:04:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.446+0000 7fd37cdff700 1 -- start start 2023-03-03T06:04:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.447+0000 7fd37cdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3780c9cb0 0x7fd3780cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:04:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.447+0000 7fd37cdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3780cf940 con 0x7fd378153260 2023-03-03T06:04:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.447+0000 7fd37cdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3780cfb90 con 0x7fd3780ca190 2023-03-03T06:04:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.447+0000 7fd37cdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780cfe70 con 0x7fd3780c9cb0 2023-03-03T06:04:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.447+0000 7fd36ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3780c9cb0 0x7fd3780cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:04:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.447+0000 7fd36ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3780c9cb0 0x7fd3780cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45204/0 (socket says 172.21.15.102:45204) 2023-03-03T06:04:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.447+0000 7fd36ffff700 1 -- 172.21.15.102:0/4264369968 learned_addr learned my addr 172.21.15.102:0/4264369968 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:04:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.448+0000 7fd36ffff700 1 -- 172.21.15.102:0/4264369968 >> v1:172.21.15.172:6789/0 conn(0x7fd3780ca190 legacy=0x7fd378163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:04:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.448+0000 7fd36ffff700 1 -- 172.21.15.102:0/4264369968 >> v1:172.21.15.102:6789/0 conn(0x7fd378153260 legacy=0x7fd3780ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:04:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.448+0000 7fd36ffff700 1 -- 172.21.15.102:0/4264369968 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd378158f80 con 0x7fd3780c9cb0 2023-03-03T06:04:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.448+0000 7fd375d9b700 1 -- 172.21.15.102:0/4264369968 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658571545 0 0) 0x7fd3780cf940 con 0x7fd378153260 2023-03-03T06:04:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.448+0000 7fd36ffff700 1 --2- 172.21.15.102:0/4264369968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780c9cb0 0x7fd3780cb880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd3600098f0 tx=0x7fd360009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710c5c7900e6622e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:04:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd375d9b700 1 -- 172.21.15.102:0/4264369968 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000b8a0 con 0x7fd3780c9cb0 2023-03-03T06:04:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd375d9b700 1 -- 172.21.15.102:0/4264369968 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36000ba00 con 0x7fd3780c9cb0 2023-03-03T06:04:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd375d9b700 1 -- 172.21.15.102:0/4264369968 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000bd90 con 0x7fd3780c9cb0 2023-03-03T06:04:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd37cdff700 1 -- 172.21.15.102:0/4264369968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780c9cb0 msgr2=0x7fd3780cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:04:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd37cdff700 1 --2- 172.21.15.102:0/4264369968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780c9cb0 0x7fd3780cb880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd3600098f0 tx=0x7fd360009c00 comp rx=0 tx=0).stop 2023-03-03T06:04:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd37cdff700 1 -- 172.21.15.102:0/4264369968 shutdown_connections 2023-03-03T06:04:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd37cdff700 1 --2- 172.21.15.102:0/4264369968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780c9cb0 0x7fd3780cb880 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:04:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd37cdff700 1 -- 172.21.15.102:0/4264369968 >> 172.21.15.102:0/4264369968 conn(0x7fd378066c00 msgr2=0x7fd3780ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:04:46.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd37cdff700 1 -- 172.21.15.102:0/4264369968 shutdown_connections 2023-03-03T06:04:46.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.449+0000 7fd37cdff700 1 -- 172.21.15.102:0/4264369968 wait complete. 2023-03-03T06:04:46.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.451+0000 7fd37cdff700 1 Processor -- start 2023-03-03T06:04:46.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.452+0000 7fd37cdff700 1 -- start start 2023-03-03T06:04:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37cdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ca190 0x7fd3781e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:04:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37cdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd378153260 0x7fd3781e3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:04:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37cdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3781e24f0 0x7fd3781e3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:04:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37cdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781ee760 con 0x7fd3780ca190 2023-03-03T06:04:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37cdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781eea00 con 0x7fd3781e24f0 2023-03-03T06:04:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37cdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781eeca0 con 0x7fd378153260 2023-03-03T06:04:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ca190 0x7fd3781e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:04:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd36ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd378153260 0x7fd3781e3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:04:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ca190 0x7fd3781e5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45218/0 (socket says 172.21.15.102:45218) 2023-03-03T06:04:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd37659c700 1 -- 172.21.15.102:0/1207314592 learned_addr learned my addr 172.21.15.102:0/1207314592 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:04:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.453+0000 7fd376d9d700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3781e24f0 0x7fd3781e3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:04:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd37659c700 1 -- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd378153260 msgr2=0x7fd3781e3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:04:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd37659c700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd378153260 0x7fd3781e3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:04:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd37659c700 1 -- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3781e24f0 msgr2=0x7fd3781e3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:04:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd37659c700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3781e24f0 0x7fd3781e3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:04:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd37659c700 1 -- 172.21.15.102:0/1207314592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360009550 con 0x7fd3780ca190 2023-03-03T06:04:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd37659c700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ca190 0x7fd3781e5950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd36800c450 tx=0x7fd36800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:04:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd36effd700 1 -- 172.21.15.102:0/1207314592 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680054e0 con 0x7fd3780ca190 2023-03-03T06:04:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd36effd700 1 -- 172.21.15.102:0/1207314592 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36800ce60 con 0x7fd3780ca190 2023-03-03T06:04:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd36effd700 1 -- 172.21.15.102:0/1207314592 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680038c0 con 0x7fd3780ca190 2023-03-03T06:04:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.454+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd378159540 con 0x7fd3780ca190 2023-03-03T06:04:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.455+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd378159260 con 0x7fd3780ca190 2023-03-03T06:04:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.456+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd378159820 con 0x7fd3780ca190 2023-03-03T06:04:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.457+0000 7fd36effd700 1 -- 172.21.15.102:0/1207314592 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd368019030 con 0x7fd3780ca190 2023-03-03T06:04:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.458+0000 7fd36effd700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3580682f0 0x7fd35806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:04:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.458+0000 7fd36effd700 1 -- 172.21.15.102:0/1207314592 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd368084fe0 con 0x7fd3780ca190 2023-03-03T06:04:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.458+0000 7fd36ffff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3580682f0 0x7fd35806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:04:46.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.459+0000 7fd36ffff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3580682f0 0x7fd35806a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd360000c00 tx=0x7fd360008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:04:46.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.462+0000 7fd36effd700 1 -- 172.21.15.102:0/1207314592 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd368051710 con 0x7fd3780ca190 2023-03-03T06:04:46.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.697+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3781e3490 con 0x7fd3780ca190 2023-03-03T06:04:46.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.698+0000 7fd36effd700 1 -- 172.21.15.102:0/1207314592 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd368017090 con 0x7fd3780ca190 2023-03-03T06:04:46.701 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:04:46.701 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:04:46.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3580682f0 msgr2=0x7fd35806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:04:46.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3580682f0 0x7fd35806a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd360000c00 tx=0x7fd360008030 comp rx=0 tx=0).stop 2023-03-03T06:04:46.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ca190 msgr2=0x7fd3781e5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:04:46.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ca190 0x7fd3781e5950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd36800c450 tx=0x7fd36800c760 comp rx=0 tx=0).stop 2023-03-03T06:04:46.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 shutdown_connections 2023-03-03T06:04:46.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3781e24f0 0x7fd3781e3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:04:46.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ca190 0x7fd3781e5950 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:04:46.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3580682f0 0x7fd35806a7a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:04:46.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 --2- 172.21.15.102:0/1207314592 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd378153260 0x7fd3781e3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:04:46.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 >> 172.21.15.102:0/1207314592 conn(0x7fd3780c9cb0 msgr2=0x7fd37806ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:04:46.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 shutdown_connections 2023-03-03T06:04:46.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:04:46.701+0000 7fd37cdff700 1 -- 172.21.15.102:0/1207314592 wait complete. 2023-03-03T06:04:46.706 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:04:46.731 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 50 secs before thrashing 2023-03-03T06:05:03.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:05:03.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:05:33.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:05:33.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:05:36.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:05:36.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.980+0000 7f146b1ff700 1 Processor -- start 2023-03-03T06:05:36.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.981+0000 7f146b1ff700 1 -- start start 2023-03-03T06:05:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.982+0000 7f146b1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1464150c80 0x7f146406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:05:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.982+0000 7f146b1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14640c2020 0x7f14640c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:05:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.982+0000 7f146b1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640c5390 con 0x7f14640c5d60 2023-03-03T06:05:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.982+0000 7f146b1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1464157ad0 con 0x7f14640c2020 2023-03-03T06:05:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.982+0000 7f146b1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14640c61e0 con 0x7f1464150c80 2023-03-03T06:05:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1463fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14640c2020 0x7f14640c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:05:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f146979c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f14640c5d60 0x7f14640c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40664/0 (socket says 172.21.15.102:40664) 2023-03-03T06:05:36.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1463fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14640c2020 0x7f14640c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48424/0 (socket says 172.21.15.102:48424) 2023-03-03T06:05:36.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1468f9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1464150c80 0x7f146406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:05:36.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f146979c700 1 -- 172.21.15.102:0/1365276573 learned_addr learned my addr 172.21.15.102:0/1365276573 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:05:36.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f14637fe700 1 -- 172.21.15.102:0/1365276573 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502137083 0 0) 0x7f14640c5390 con 0x7f14640c5d60 2023-03-03T06:05:36.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f14637fe700 1 -- 172.21.15.102:0/1365276573 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14540035b0 con 0x7f14640c5d60 2023-03-03T06:05:36.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1463fff700 1 -- 172.21.15.102:0/1365276573 >> v2:172.21.15.172:3300/0 conn(0x7f1464150c80 msgr2=0x7f146406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:05:36.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1463fff700 1 --2- 172.21.15.102:0/1365276573 >> v2:172.21.15.172:3300/0 conn(0x7f1464150c80 0x7f146406d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:36.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1463fff700 1 -- 172.21.15.102:0/1365276573 >> v1:172.21.15.172:6789/0 conn(0x7f14640c5d60 legacy=0x7f14640c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:05:36.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1463fff700 1 -- 172.21.15.102:0/1365276573 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640c4a20 con 0x7f14640c2020 2023-03-03T06:05:36.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.983+0000 7f1468f9b700 1 --2- 172.21.15.102:0/1365276573 >> v2:172.21.15.172:3300/0 conn(0x7f1464150c80 0x7f146406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:05:36.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.984+0000 7f1463fff700 1 --2- 172.21.15.102:0/1365276573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c2020 0x7f14640c5820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1464072a90 tx=0x7f1450009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193f128092b37631 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:05:36.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.984+0000 7f14637fe700 1 -- 172.21.15.102:0/1365276573 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450011830 con 0x7f14640c2020 2023-03-03T06:05:36.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.984+0000 7f14637fe700 1 -- 172.21.15.102:0/1365276573 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1450011990 con 0x7f14640c2020 2023-03-03T06:05:36.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.984+0000 7f14637fe700 1 -- 172.21.15.102:0/1365276573 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450011d20 con 0x7f14640c2020 2023-03-03T06:05:36.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 -- 172.21.15.102:0/1365276573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c2020 msgr2=0x7f14640c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:05:36.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 --2- 172.21.15.102:0/1365276573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c2020 0x7f14640c5820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1464072a90 tx=0x7f1450009b20 comp rx=0 tx=0).stop 2023-03-03T06:05:36.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 -- 172.21.15.102:0/1365276573 shutdown_connections 2023-03-03T06:05:36.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 --2- 172.21.15.102:0/1365276573 >> v2:172.21.15.172:3300/0 conn(0x7f1464150c80 0x7f146406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:36.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 --2- 172.21.15.102:0/1365276573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c2020 0x7f14640c5820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:36.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 -- 172.21.15.102:0/1365276573 >> 172.21.15.102:0/1365276573 conn(0x7f146406af90 msgr2=0x7f146406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:05:36.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 -- 172.21.15.102:0/1365276573 shutdown_connections 2023-03-03T06:05:36.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.985+0000 7f146b1ff700 1 -- 172.21.15.102:0/1365276573 wait complete. 2023-03-03T06:05:36.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.987+0000 7f146b1ff700 1 Processor -- start 2023-03-03T06:05:36.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.987+0000 7f146b1ff700 1 -- start start 2023-03-03T06:05:36.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f146b1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14640c2020 0x7f146415aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:05:36.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f146b1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c5d60 0x7f14641482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:05:36.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f146b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1464150c80 0x7f146414f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:05:36.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f146b1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1464158bb0 con 0x7f14640c5d60 2023-03-03T06:05:36.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f146b1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1464158720 con 0x7f14640c2020 2023-03-03T06:05:36.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f146b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14640ca500 con 0x7f1464150c80 2023-03-03T06:05:36.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f1463fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c5d60 0x7f14641482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f1463fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c5d60 0x7f14641482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48440/0 (socket says 172.21.15.102:48440) 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f1463fff700 1 -- 172.21.15.102:0/2777477769 learned_addr learned my addr 172.21.15.102:0/2777477769 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.988+0000 7f146979c700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1464150c80 0x7f146414f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f1468f9b700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14640c2020 0x7f146415aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f1463fff700 1 -- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1464150c80 msgr2=0x7f146414f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f1463fff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1464150c80 0x7f146414f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f1463fff700 1 -- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14640c2020 msgr2=0x7f146415aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:05:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f1463fff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14640c2020 0x7f146415aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:36.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f1463fff700 1 -- 172.21.15.102:0/2777477769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14580094a0 con 0x7f14640c5d60 2023-03-03T06:05:36.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f1463fff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c5d60 0x7f14641482d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1450000c00 tx=0x7f1450004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:05:36.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f14617fa700 1 -- 172.21.15.102:0/2777477769 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14500036a0 con 0x7f14640c5d60 2023-03-03T06:05:36.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.989+0000 7f14617fa700 1 -- 172.21.15.102:0/2777477769 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145001f070 con 0x7f14640c5d60 2023-03-03T06:05:36.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.990+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500094a0 con 0x7f14640c5d60 2023-03-03T06:05:36.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.990+0000 7f14617fa700 1 -- 172.21.15.102:0/2777477769 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450003a60 con 0x7f14640c5d60 2023-03-03T06:05:36.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.990+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1464157510 con 0x7f14640c5d60 2023-03-03T06:05:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.991+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146406f010 con 0x7f14640c5d60 2023-03-03T06:05:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.992+0000 7f14617fa700 1 -- 172.21.15.102:0/2777477769 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f145000f050 con 0x7f14640c5d60 2023-03-03T06:05:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.993+0000 7f14617fa700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1448068320 0x7f144806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:05:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.993+0000 7f14617fa700 1 -- 172.21.15.102:0/2777477769 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1450021070 con 0x7f14640c5d60 2023-03-03T06:05:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.993+0000 7f1468f9b700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1448068320 0x7f144806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:05:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.994+0000 7f1468f9b700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1448068320 0x7f144806a7d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1458009cd0 tx=0x7f1458009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:05:36.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:36.997+0000 7f14617fa700 1 -- 172.21.15.102:0/2777477769 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1450053c00 con 0x7f14640c5d60 2023-03-03T06:05:37.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.232+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14641577f0 con 0x7f14640c5d60 2023-03-03T06:05:37.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.233+0000 7f14617fa700 1 -- 172.21.15.102:0/2777477769 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1450057220 con 0x7f14640c5d60 2023-03-03T06:05:37.236 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:05:37.236 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:05:37.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1448068320 msgr2=0x7f144806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:05:37.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1448068320 0x7f144806a7d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1458009cd0 tx=0x7f1458009210 comp rx=0 tx=0).stop 2023-03-03T06:05:37.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c5d60 msgr2=0x7f14641482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:05:37.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c5d60 0x7f14641482d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1450000c00 tx=0x7f1450004750 comp rx=0 tx=0).stop 2023-03-03T06:05:37.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 shutdown_connections 2023-03-03T06:05:37.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1448068320 0x7f144806a7d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:37.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1464150c80 0x7f146414f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:37.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14640c2020 0x7f146415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:37.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 --2- 172.21.15.102:0/2777477769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14640c5d60 0x7f14641482d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:05:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 >> 172.21.15.102:0/2777477769 conn(0x7f146406af90 msgr2=0x7f146414ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:05:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 shutdown_connections 2023-03-03T06:05:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:05:37.236+0000 7f146b1ff700 1 -- 172.21.15.102:0/2777477769 wait complete. 2023-03-03T06:05:37.240 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:05:37.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T06:06:04.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:06:04.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:06:11.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:06:11.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.532+0000 7f00b89ff700 1 Processor -- start 2023-03-03T06:06:11.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.532+0000 7f00b89ff700 1 -- start start 2023-03-03T06:06:11.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.533+0000 7f00b89ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b4153290 0x7f00b4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:06:11.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.533+0000 7f00b89ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f00b41593c0 0x7f00b40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:06:11.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.533+0000 7f00b89ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b4153660 con 0x7f00b40cd8e0 2023-03-03T06:06:11.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.533+0000 7f00b89ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b406c360 con 0x7f00b41593c0 2023-03-03T06:06:11.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.533+0000 7f00b89ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b415cea0 con 0x7f00b4153290 2023-03-03T06:06:11.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b2d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f00b40cd8e0 0x7f00b40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51840/0 (socket says 172.21.15.102:51840) 2023-03-03T06:06:11.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b2d9d700 1 -- 172.21.15.102:0/1162534838 learned_addr learned my addr 172.21.15.102:0/1162534838 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:06:11.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b259c700 1 --2- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b4153290 0x7f00b4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:06:11.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b1d9b700 1 --2- 172.21.15.102:0/1162534838 >> v2:172.21.15.172:3300/0 conn(0x7f00b41593c0 0x7f00b40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:06:11.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b159a700 1 -- 172.21.15.102:0/1162534838 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598979389 0 0) 0x7f00b4153660 con 0x7f00b40cd8e0 2023-03-03T06:06:11.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b159a700 1 -- 172.21.15.102:0/1162534838 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c0035b0 con 0x7f00b40cd8e0 2023-03-03T06:06:11.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b1d9b700 1 -- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b4153290 msgr2=0x7f00b4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:06:11.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b1d9b700 1 --2- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b4153290 0x7f00b4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b1d9b700 1 -- 172.21.15.102:0/1162534838 >> v1:172.21.15.102:6789/0 conn(0x7f00b40cd8e0 legacy=0x7f00b40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:06:11.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b1d9b700 1 -- 172.21.15.102:0/1162534838 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b406c7f0 con 0x7f00b41593c0 2023-03-03T06:06:11.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.534+0000 7f00b259c700 1 --2- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b4153290 0x7f00b4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:06:11.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b1d9b700 1 --2- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41593c0 0x7f00b40cde20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009840 tx=0x7f00a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b3dc563dbefbd3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:06:11.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b159a700 1 -- 172.21.15.102:0/1162534838 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8011830 con 0x7f00b41593c0 2023-03-03T06:06:11.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b159a700 1 -- 172.21.15.102:0/1162534838 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a8011990 con 0x7f00b41593c0 2023-03-03T06:06:11.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b159a700 1 -- 172.21.15.102:0/1162534838 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8011d20 con 0x7f00b41593c0 2023-03-03T06:06:11.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b89ff700 1 -- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41593c0 msgr2=0x7f00b40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:06:11.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b89ff700 1 --2- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41593c0 0x7f00b40cde20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009840 tx=0x7f00a8009b50 comp rx=0 tx=0).stop 2023-03-03T06:06:11.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b89ff700 1 -- 172.21.15.102:0/1162534838 shutdown_connections 2023-03-03T06:06:11.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b89ff700 1 --2- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b4153290 0x7f00b4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b89ff700 1 --2- 172.21.15.102:0/1162534838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b41593c0 0x7f00b40cde20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.535+0000 7f00b89ff700 1 -- 172.21.15.102:0/1162534838 >> 172.21.15.102:0/1162534838 conn(0x7f00b4065fb0 msgr2=0x7f00b40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:06:11.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.536+0000 7f00b89ff700 1 -- 172.21.15.102:0/1162534838 shutdown_connections 2023-03-03T06:06:11.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.536+0000 7f00b89ff700 1 -- 172.21.15.102:0/1162534838 wait complete. 2023-03-03T06:06:11.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.538+0000 7f00b89ff700 1 Processor -- start 2023-03-03T06:06:11.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.538+0000 7f00b89ff700 1 -- start start 2023-03-03T06:06:11.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b89ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00b40cd8e0 0x7f00b40c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:06:11.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b89ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b4153290 0x7f00b41eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:06:11.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b89ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41593c0 0x7f00b41ef410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:06:11.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b89ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b414f730 con 0x7f00b40cd8e0 2023-03-03T06:06:11.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b89ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40c71a0 con 0x7f00b4153290 2023-03-03T06:06:11.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b89ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4151cc0 con 0x7f00b41593c0 2023-03-03T06:06:11.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00b40cd8e0 0x7f00b40c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:06:11.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00b40cd8e0 0x7f00b40c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34402/0 (socket says 172.21.15.102:34402) 2023-03-03T06:06:11.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b259c700 1 -- 172.21.15.102:0/3356784941 learned_addr learned my addr 172.21.15.102:0/3356784941 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:06:11.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b1d9b700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b4153290 0x7f00b41eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:06:11.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.539+0000 7f00b2d9d700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41593c0 0x7f00b41ef410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:06:11.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b259c700 1 -- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41593c0 msgr2=0x7f00b41ef410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:06:11.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b259c700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41593c0 0x7f00b41ef410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b259c700 1 -- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b4153290 msgr2=0x7f00b41eeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b259c700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b4153290 0x7f00b41eeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b259c700 1 -- 172.21.15.102:0/3356784941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40094a0 con 0x7f00b40cd8e0 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b1d9b700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b4153290 0x7f00b41eeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b259c700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00b40cd8e0 0x7f00b40c63b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f00a400ebe0 tx=0x7f00a400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00a2ffd700 1 -- 172.21.15.102:0/3356784941 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a400c850 con 0x7f00b40cd8e0 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00a2ffd700 1 -- 172.21.15.102:0/3356784941 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a400c9b0 con 0x7f00b40cd8e0 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00a2ffd700 1 -- 172.21.15.102:0/3356784941 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a400cd40 con 0x7f00b40cd8e0 2023-03-03T06:06:11.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.540+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a80094a0 con 0x7f00b40cd8e0 2023-03-03T06:06:11.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.541+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b40ccde0 con 0x7f00b40cd8e0 2023-03-03T06:06:11.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.542+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b415d160 con 0x7f00b40cd8e0 2023-03-03T06:06:11.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.543+0000 7f00a2ffd700 1 -- 172.21.15.102:0/3356784941 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f00a4019030 con 0x7f00b40cd8e0 2023-03-03T06:06:11.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.544+0000 7f00a2ffd700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00940682c0 0x7f009406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:06:11.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.544+0000 7f00a2ffd700 1 -- 172.21.15.102:0/3356784941 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f00a4085dd0 con 0x7f00b40cd8e0 2023-03-03T06:06:11.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.544+0000 7f00b1d9b700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00940682c0 0x7f009406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:06:11.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.545+0000 7f00b1d9b700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00940682c0 0x7f009406a770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f00a8000c00 tx=0x7f00a8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:06:11.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.548+0000 7f00a2ffd700 1 -- 172.21.15.102:0/3356784941 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f00a4052450 con 0x7f00b40cd8e0 2023-03-03T06:06:11.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.783+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00b40cd0c0 con 0x7f00b40cd8e0 2023-03-03T06:06:11.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.784+0000 7f00a2ffd700 1 -- 172.21.15.102:0/3356784941 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f00a4017090 con 0x7f00b40cd8e0 2023-03-03T06:06:11.787 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:06:11.788 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:06:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.786+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00940682c0 msgr2=0x7f009406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:06:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.786+0000 7f00b89ff700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00940682c0 0x7f009406a770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f00a8000c00 tx=0x7f00a8008030 comp rx=0 tx=0).stop 2023-03-03T06:06:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.786+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00b40cd8e0 msgr2=0x7f00b40c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:06:11.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.786+0000 7f00b89ff700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00b40cd8e0 0x7f00b40c63b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f00a400ebe0 tx=0x7f00a400eef0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 shutdown_connections 2023-03-03T06:06:11.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00940682c0 0x7f009406a770 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00b41593c0 0x7f00b41ef410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00b40cd8e0 0x7f00b40c63b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 --2- 172.21.15.102:0/3356784941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00b4153290 0x7f00b41eeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:06:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 >> 172.21.15.102:0/3356784941 conn(0x7f00b4065fb0 msgr2=0x7f00b40cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:06:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 shutdown_connections 2023-03-03T06:06:11.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:06:11.787+0000 7f00b89ff700 1 -- 172.21.15.102:0/3356784941 wait complete. 2023-03-03T06:06:11.793 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:06:11.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T06:06:34.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:06:34.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:07:04.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:07:04.543 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:07:34.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:07:34.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:07:48.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:07:49.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.084+0000 7f1a4759e700 1 Processor -- start 2023-03-03T06:07:49.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.084+0000 7f1a4759e700 1 -- start start 2023-03-03T06:07:49.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.086+0000 7f1a4759e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1a480cab50 0x7f1a481646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:07:49.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.086+0000 7f1a4759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a480c9ca0 con 0x7f1a480c9610 2023-03-03T06:07:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.086+0000 7f1a4759e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a480b46f0 con 0x7f1a481564b0 2023-03-03T06:07:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.086+0000 7f1a4759e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a480b4920 con 0x7f1a480cab50 2023-03-03T06:07:49.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.086+0000 7f1a45d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1a480c9610 0x7f1a480ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52814/0 (socket says 172.21.15.102:52814) 2023-03-03T06:07:49.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.086+0000 7f1a45d9b700 1 -- 172.21.15.102:0/2064619373 learned_addr learned my addr 172.21.15.102:0/2064619373 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:07:49.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.086+0000 7f1a46d9d700 1 --2- 172.21.15.102:0/2064619373 >> v2:172.21.15.102:3300/0 conn(0x7f1a480cab50 0x7f1a481646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:07:49.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a4559a700 1 -- 172.21.15.102:0/2064619373 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075542087 0 0) 0x7f1a480b46f0 con 0x7f1a481564b0 2023-03-03T06:07:49.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a4559a700 1 -- 172.21.15.102:0/2064619373 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a300035b0 con 0x7f1a481564b0 2023-03-03T06:07:49.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a4559a700 1 -- 172.21.15.102:0/2064619373 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043373537 0 0) 0x7f1a480c9ca0 con 0x7f1a480c9610 2023-03-03T06:07:49.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a4559a700 1 -- 172.21.15.102:0/2064619373 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a480b46f0 con 0x7f1a480c9610 2023-03-03T06:07:49.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a46d9d700 1 -- 172.21.15.102:0/2064619373 >> v1:172.21.15.172:6789/0 conn(0x7f1a481564b0 legacy=0x7f1a480cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:07:49.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a46d9d700 1 -- 172.21.15.102:0/2064619373 >> v1:172.21.15.102:6789/0 conn(0x7f1a480c9610 legacy=0x7f1a480ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:07:49.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a46d9d700 1 -- 172.21.15.102:0/2064619373 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a480b4c00 con 0x7f1a480cab50 2023-03-03T06:07:49.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.087+0000 7f1a46d9d700 1 --2- 172.21.15.102:0/2064619373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a480cab50 0x7f1a481646b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009d90 tx=0x7f1a4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c95daf57294f03 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:07:49.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4559a700 1 -- 172.21.15.102:0/2064619373 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4000e070 con 0x7f1a480cab50 2023-03-03T06:07:49.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4559a700 1 -- 172.21.15.102:0/2064619373 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4000bb20 con 0x7f1a480cab50 2023-03-03T06:07:49.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4559a700 1 -- 172.21.15.102:0/2064619373 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40012430 con 0x7f1a480cab50 2023-03-03T06:07:49.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4759e700 1 -- 172.21.15.102:0/2064619373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a480cab50 msgr2=0x7f1a481646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:07:49.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2064619373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a480cab50 0x7f1a481646b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009d90 tx=0x7f1a4000b330 comp rx=0 tx=0).stop 2023-03-03T06:07:49.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4759e700 1 -- 172.21.15.102:0/2064619373 shutdown_connections 2023-03-03T06:07:49.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2064619373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a480cab50 0x7f1a481646b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4759e700 1 -- 172.21.15.102:0/2064619373 >> 172.21.15.102:0/2064619373 conn(0x7f1a48065fb0 msgr2=0x7f1a480663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:07:49.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.088+0000 7f1a4759e700 1 -- 172.21.15.102:0/2064619373 shutdown_connections 2023-03-03T06:07:49.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.089+0000 7f1a4759e700 1 -- 172.21.15.102:0/2064619373 wait complete. 2023-03-03T06:07:49.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.090+0000 7f1a4759e700 1 Processor -- start 2023-03-03T06:07:49.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.091+0000 7f1a4759e700 1 -- start start 2023-03-03T06:07:49.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a4759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a480c9610 0x7f1a48158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:07:49.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a4759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a480cab50 0x7f1a481592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:07:49.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a4759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a481564b0 0x7f1a4814ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:07:49.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a4759e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a481e3090 con 0x7f1a481564b0 2023-03-03T06:07:49.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a4759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a481588f0 con 0x7f1a480cab50 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a4759e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a48152be0 con 0x7f1a480c9610 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a481564b0 0x7f1a4814ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a481564b0 0x7f1a4814ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54914/0 (socket says 172.21.15.102:54914) 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 -- 172.21.15.102:0/2698388872 learned_addr learned my addr 172.21.15.102:0/2698388872 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a45d9b700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a480cab50 0x7f1a481592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a4659c700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a480c9610 0x7f1a48158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 -- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a480c9610 msgr2=0x7f1a48158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:07:49.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a480c9610 0x7f1a48158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 -- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a480cab50 msgr2=0x7f1a481592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:07:49.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a480cab50 0x7f1a481592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.092+0000 7f1a46d9d700 1 -- 172.21.15.102:0/2698388872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a40009a70 con 0x7f1a481564b0 2023-03-03T06:07:49.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.093+0000 7f1a46d9d700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a481564b0 0x7f1a4814ef30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1a4806c8c0 tx=0x7f1a40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:07:49.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.093+0000 7f1a36ffd700 1 -- 172.21.15.102:0/2698388872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40012430 con 0x7f1a481564b0 2023-03-03T06:07:49.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.093+0000 7f1a36ffd700 1 -- 172.21.15.102:0/2698388872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a40020070 con 0x7f1a481564b0 2023-03-03T06:07:49.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.093+0000 7f1a36ffd700 1 -- 172.21.15.102:0/2698388872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4001ba50 con 0x7f1a481564b0 2023-03-03T06:07:49.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.093+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a480b2210 con 0x7f1a481564b0 2023-03-03T06:07:49.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.093+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a480b1f30 con 0x7f1a481564b0 2023-03-03T06:07:49.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.094+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a48156880 con 0x7f1a481564b0 2023-03-03T06:07:49.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.096+0000 7f1a36ffd700 1 -- 172.21.15.102:0/2698388872 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1a4002e070 con 0x7f1a481564b0 2023-03-03T06:07:49.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.097+0000 7f1a36ffd700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1a280682f0 0x7f1a2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:07:49.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.097+0000 7f1a36ffd700 1 -- 172.21.15.102:0/2698388872 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1a40022070 con 0x7f1a481564b0 2023-03-03T06:07:49.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.097+0000 7f1a4659c700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1a280682f0 0x7f1a2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:07:49.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.098+0000 7f1a4659c700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1a280682f0 0x7f1a2806a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1a380011e0 tx=0x7f1a38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:07:49.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.101+0000 7f1a36ffd700 1 -- 172.21.15.102:0/2698388872 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1a400543e0 con 0x7f1a481564b0 2023-03-03T06:07:49.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.335+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a48158a60 con 0x7f1a481564b0 2023-03-03T06:07:49.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.335+0000 7f1a36ffd700 1 -- 172.21.15.102:0/2698388872 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1a4002c090 con 0x7f1a481564b0 2023-03-03T06:07:49.338 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:07:49.338 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:07:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1a280682f0 msgr2=0x7f1a2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:07:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1a280682f0 0x7f1a2806a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1a380011e0 tx=0x7f1a38009040 comp rx=0 tx=0).stop 2023-03-03T06:07:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a481564b0 msgr2=0x7f1a4814ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:07:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a481564b0 0x7f1a4814ef30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1a4806c8c0 tx=0x7f1a40004ad0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 shutdown_connections 2023-03-03T06:07:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1a280682f0 0x7f1a2806a7a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a480c9610 0x7f1a48158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a481564b0 0x7f1a4814ef30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 --2- 172.21.15.102:0/2698388872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a480cab50 0x7f1a481592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:07:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.338+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 >> 172.21.15.102:0/2698388872 conn(0x7f1a48065fb0 msgr2=0x7f1a480cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:07:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.339+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 shutdown_connections 2023-03-03T06:07:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:07:49.339+0000 7f1a4759e700 1 -- 172.21.15.102:0/2698388872 wait complete. 2023-03-03T06:07:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:07:49.368 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T06:08:04.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:08:04.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:08:28.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:08:28.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.627+0000 7fc88e1ff700 1 Processor -- start 2023-03-03T06:08:28.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.627+0000 7fc88e1ff700 1 -- start start 2023-03-03T06:08:28.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.628+0000 7fc88e1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc888153290 0x7fc888158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:08:28.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.628+0000 7fc88e1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc8881593c0 0x7fc8880cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:08:28.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.628+0000 7fc88e1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc888153660 con 0x7fc8880cd8e0 2023-03-03T06:08:28.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.628+0000 7fc88e1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88806c360 con 0x7fc8881593c0 2023-03-03T06:08:28.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.628+0000 7fc88e1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88815cea0 con 0x7fc888153290 2023-03-03T06:08:28.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc886ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc8881593c0 0x7fc8880cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:08:28.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc887fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc8880cd8e0 0x7fc8880ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36032/0 (socket says 172.21.15.102:36032) 2023-03-03T06:08:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc887fff700 1 -- 172.21.15.102:0/3766158506 learned_addr learned my addr 172.21.15.102:0/3766158506 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:08:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc8877fe700 1 --2- 172.21.15.102:0/3766158506 >> v2:172.21.15.172:3300/0 conn(0x7fc888153290 0x7fc888158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:08:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc886ffd700 1 -- 172.21.15.102:0/3766158506 >> v2:172.21.15.172:3300/0 conn(0x7fc888153290 msgr2=0x7fc888158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:08:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc886ffd700 1 --2- 172.21.15.102:0/3766158506 >> v2:172.21.15.172:3300/0 conn(0x7fc888153290 0x7fc888158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc886ffd700 1 -- 172.21.15.102:0/3766158506 >> v1:172.21.15.102:6789/0 conn(0x7fc8880cd8e0 legacy=0x7fc8880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:08:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc886ffd700 1 -- 172.21.15.102:0/3766158506 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88806c7f0 con 0x7fc8881593c0 2023-03-03T06:08:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.629+0000 7fc8867fc700 1 -- 172.21.15.102:0/3766158506 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496826680 0 0) 0x7fc888153660 con 0x7fc8880cd8e0 2023-03-03T06:08:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.630+0000 7fc886ffd700 1 --2- 172.21.15.102:0/3766158506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8881593c0 0x7fc8880cde20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc88808cc70 tx=0x7fc87c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d6f935b3ae3531 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:08:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.630+0000 7fc8867fc700 1 -- 172.21.15.102:0/3766158506 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c011830 con 0x7fc8881593c0 2023-03-03T06:08:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.630+0000 7fc8867fc700 1 -- 172.21.15.102:0/3766158506 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c011990 con 0x7fc8881593c0 2023-03-03T06:08:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.630+0000 7fc8867fc700 1 -- 172.21.15.102:0/3766158506 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c011d20 con 0x7fc8881593c0 2023-03-03T06:08:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.630+0000 7fc88e1ff700 1 -- 172.21.15.102:0/3766158506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8881593c0 msgr2=0x7fc8880cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:08:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.630+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/3766158506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8881593c0 0x7fc8880cde20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc88808cc70 tx=0x7fc87c009b20 comp rx=0 tx=0).stop 2023-03-03T06:08:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.630+0000 7fc88e1ff700 1 -- 172.21.15.102:0/3766158506 shutdown_connections 2023-03-03T06:08:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.631+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/3766158506 >> v2:172.21.15.172:3300/0 conn(0x7fc888153290 0x7fc888158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.631+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/3766158506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8881593c0 0x7fc8880cde20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.631+0000 7fc88e1ff700 1 -- 172.21.15.102:0/3766158506 >> 172.21.15.102:0/3766158506 conn(0x7fc888065fb0 msgr2=0x7fc8880663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:08:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.631+0000 7fc88e1ff700 1 -- 172.21.15.102:0/3766158506 shutdown_connections 2023-03-03T06:08:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.631+0000 7fc88e1ff700 1 -- 172.21.15.102:0/3766158506 wait complete. 2023-03-03T06:08:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.633+0000 7fc88e1ff700 1 Processor -- start 2023-03-03T06:08:28.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.633+0000 7fc88e1ff700 1 -- start start 2023-03-03T06:08:28.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc88e1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8880cd8e0 0x7fc8880c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:08:28.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc88e1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc888153290 0x7fc8880c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:08:28.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc88e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8881593c0 0x7fc8880c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:08:28.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc88e1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8880c7270 con 0x7fc888153290 2023-03-03T06:08:28.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc88e1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8881ef0d0 con 0x7fc8880cd8e0 2023-03-03T06:08:28.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc88e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8881ef600 con 0x7fc8881593c0 2023-03-03T06:08:28.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc886ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc888153290 0x7fc8880c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:08:28.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc886ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc888153290 0x7fc8880c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47430/0 (socket says 172.21.15.102:47430) 2023-03-03T06:08:28.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc886ffd700 1 -- 172.21.15.102:0/4094078743 learned_addr learned my addr 172.21.15.102:0/4094078743 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:08:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc8877fe700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8880cd8e0 0x7fc8880c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:08:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.634+0000 7fc887fff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8881593c0 0x7fc8880c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:08:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc886ffd700 1 -- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8881593c0 msgr2=0x7fc8880c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:08:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc886ffd700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8881593c0 0x7fc8880c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc886ffd700 1 -- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8880cd8e0 msgr2=0x7fc8880c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:08:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc886ffd700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8880cd8e0 0x7fc8880c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc886ffd700 1 -- 172.21.15.102:0/4094078743 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0094a0 con 0x7fc888153290 2023-03-03T06:08:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc886ffd700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc888153290 0x7fc8880c76c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0090b0 tx=0x7fc87c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:08:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc86ffff700 1 -- 172.21.15.102:0/4094078743 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0036a0 con 0x7fc888153290 2023-03-03T06:08:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc86ffff700 1 -- 172.21.15.102:0/4094078743 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c01f070 con 0x7fc888153290 2023-03-03T06:08:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc86ffff700 1 -- 172.21.15.102:0/4094078743 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c003a20 con 0x7fc888153290 2023-03-03T06:08:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.635+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8880ccde0 con 0x7fc888153290 2023-03-03T06:08:28.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.636+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88815d160 con 0x7fc888153290 2023-03-03T06:08:28.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.637+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8880cd0c0 con 0x7fc888153290 2023-03-03T06:08:28.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.638+0000 7fc86ffff700 1 -- 172.21.15.102:0/4094078743 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc87c00f050 con 0x7fc888153290 2023-03-03T06:08:28.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.639+0000 7fc86ffff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8740682f0 0x7fc87406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:08:28.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.639+0000 7fc86ffff700 1 -- 172.21.15.102:0/4094078743 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc87c021070 con 0x7fc888153290 2023-03-03T06:08:28.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.639+0000 7fc8877fe700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8740682f0 0x7fc87406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:08:28.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.640+0000 7fc8877fe700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8740682f0 0x7fc87406a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc878009be0 tx=0x7fc878009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:08:28.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.643+0000 7fc86ffff700 1 -- 172.21.15.102:0/4094078743 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc87c053cf0 con 0x7fc888153290 2023-03-03T06:08:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.878+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc88808bee0 con 0x7fc888153290 2023-03-03T06:08:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.879+0000 7fc86ffff700 1 -- 172.21.15.102:0/4094078743 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc87c057310 con 0x7fc888153290 2023-03-03T06:08:28.882 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:08:28.882 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:08:28.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.881+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8740682f0 msgr2=0x7fc87406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:08:28.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.881+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8740682f0 0x7fc87406a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc878009be0 tx=0x7fc878009210 comp rx=0 tx=0).stop 2023-03-03T06:08:28.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.881+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc888153290 msgr2=0x7fc8880c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:08:28.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.881+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc888153290 0x7fc8880c76c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0090b0 tx=0x7fc87c004750 comp rx=0 tx=0).stop 2023-03-03T06:08:28.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 shutdown_connections 2023-03-03T06:08:28.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8740682f0 0x7fc87406a7a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8881593c0 0x7fc8880c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8880cd8e0 0x7fc8880c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 --2- 172.21.15.102:0/4094078743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc888153290 0x7fc8880c76c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:08:28.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 >> 172.21.15.102:0/4094078743 conn(0x7fc888065fb0 msgr2=0x7fc8880cc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:08:28.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 shutdown_connections 2023-03-03T06:08:28.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:08:28.882+0000 7fc88e1ff700 1 -- 172.21.15.102:0/4094078743 wait complete. 2023-03-03T06:08:28.887 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:08:28.913 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-03-03T06:08:35.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:08:35.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:09:05.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:09:05.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:09:35.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:09:35.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:10:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T06:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:10:05.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:10:05.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:10:17.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:10:18.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.181+0000 7f57c81ff700 1 Processor -- start 2023-03-03T06:10:18.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.182+0000 7f57c81ff700 1 -- start start 2023-03-03T06:10:18.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.183+0000 7f57c81ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f57c0153290 0x7f57c0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:10:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.183+0000 7f57c81ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c0153660 con 0x7f57c00cd8e0 2023-03-03T06:10:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.183+0000 7f57c81ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c006c360 con 0x7f57c01593c0 2023-03-03T06:10:18.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.183+0000 7f57c81ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c00cec40 con 0x7f57c0153290 2023-03-03T06:10:18.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c579a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f57c01593c0 0x7f57c00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35338/0 (socket says 172.21.15.102:35338) 2023-03-03T06:10:18.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c579a700 1 -- 172.21.15.102:0/2643194934 learned_addr learned my addr 172.21.15.102:0/2643194934 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:10:18.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c5f9b700 1 --2- 172.21.15.102:0/2643194934 >> v2:172.21.15.102:3300/0 conn(0x7f57c0153290 0x7f57c0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c4f99700 1 -- 172.21.15.102:0/2643194934 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769847098 0 0) 0x7f57c006c360 con 0x7f57c01593c0 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c4f99700 1 -- 172.21.15.102:0/2643194934 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57ac0035b0 con 0x7f57c01593c0 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c4f99700 1 -- 172.21.15.102:0/2643194934 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913316699 0 0) 0x7f57c0153660 con 0x7f57c00cd8e0 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c4f99700 1 -- 172.21.15.102:0/2643194934 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c006c360 con 0x7f57c00cd8e0 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c5f9b700 1 -- 172.21.15.102:0/2643194934 >> v1:172.21.15.172:6789/0 conn(0x7f57c01593c0 legacy=0x7f57c00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c5f9b700 1 -- 172.21.15.102:0/2643194934 >> v1:172.21.15.102:6789/0 conn(0x7f57c00cd8e0 legacy=0x7f57c00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.184+0000 7f57c5f9b700 1 -- 172.21.15.102:0/2643194934 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c015cea0 con 0x7f57c0153290 2023-03-03T06:10:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.185+0000 7f57c5f9b700 1 --2- 172.21.15.102:0/2643194934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c0158e80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009840 tx=0x7f57bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfa96668567a62f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:10:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.185+0000 7f57c4f99700 1 -- 172.21.15.102:0/2643194934 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc011830 con 0x7f57c0153290 2023-03-03T06:10:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.185+0000 7f57c4f99700 1 -- 172.21.15.102:0/2643194934 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc011990 con 0x7f57c0153290 2023-03-03T06:10:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.185+0000 7f57c4f99700 1 -- 172.21.15.102:0/2643194934 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc011d20 con 0x7f57c0153290 2023-03-03T06:10:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.185+0000 7f57c81ff700 1 -- 172.21.15.102:0/2643194934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 msgr2=0x7f57c0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:10:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.185+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2643194934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c0158e80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009840 tx=0x7f57bc009b50 comp rx=0 tx=0).stop 2023-03-03T06:10:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.186+0000 7f57c81ff700 1 -- 172.21.15.102:0/2643194934 shutdown_connections 2023-03-03T06:10:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.186+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2643194934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c0158e80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:10:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.186+0000 7f57c81ff700 1 -- 172.21.15.102:0/2643194934 >> 172.21.15.102:0/2643194934 conn(0x7f57c0065fb0 msgr2=0x7f57c00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:10:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.186+0000 7f57c81ff700 1 -- 172.21.15.102:0/2643194934 shutdown_connections 2023-03-03T06:10:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.186+0000 7f57c81ff700 1 -- 172.21.15.102:0/2643194934 wait complete. 2023-03-03T06:10:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.188+0000 7f57c81ff700 1 Processor -- start 2023-03-03T06:10:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.188+0000 7f57c81ff700 1 -- start start 2023-03-03T06:10:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c81ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57c00cd8e0 0x7f57c01e5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:10:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c81ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c00b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:10:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c81ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57c01593c0 0x7f57c00b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:10:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c81ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00c9020 con 0x7f57c0153290 2023-03-03T06:10:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c81ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c014f0f0 con 0x7f57c01593c0 2023-03-03T06:10:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c81ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00c59c0 con 0x7f57c00cd8e0 2023-03-03T06:10:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c579a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c00b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:10:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c5f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57c00cd8e0 0x7f57c01e5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:10:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c579a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c00b3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36228/0 (socket says 172.21.15.102:36228) 2023-03-03T06:10:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c579a700 1 -- 172.21.15.102:0/2996318993 learned_addr learned my addr 172.21.15.102:0/2996318993 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:10:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.189+0000 7f57c679c700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57c01593c0 0x7f57c00b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:10:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c5f9b700 1 -- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57c01593c0 msgr2=0x7f57c00b2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:10:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c5f9b700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57c01593c0 0x7f57c00b2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:10:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c5f9b700 1 -- 172.21.15.102:0/2996318993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 msgr2=0x7f57c00b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:10:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c5f9b700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c00b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:10:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c5f9b700 1 -- 172.21.15.102:0/2996318993 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc0094a0 con 0x7f57c00cd8e0 2023-03-03T06:10:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c579a700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c00b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:10:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c5f9b700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57c00cd8e0 0x7f57c01e5c50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f57c00c9390 tx=0x7f57bc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:10:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57b67fc700 1 -- 172.21.15.102:0/2996318993 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc01a8f0 con 0x7f57c00cd8e0 2023-03-03T06:10:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c015d160 con 0x7f57c00cd8e0 2023-03-03T06:10:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57b67fc700 1 -- 172.21.15.102:0/2996318993 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc01aa70 con 0x7f57c00cd8e0 2023-03-03T06:10:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.190+0000 7f57b67fc700 1 -- 172.21.15.102:0/2996318993 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc01ae20 con 0x7f57c00cd8e0 2023-03-03T06:10:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.191+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c006c7f0 con 0x7f57c00cd8e0 2023-03-03T06:10:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.192+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c00ccde0 con 0x7f57c00cd8e0 2023-03-03T06:10:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.193+0000 7f57b67fc700 1 -- 172.21.15.102:0/2996318993 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57bc00f050 con 0x7f57c00cd8e0 2023-03-03T06:10:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.194+0000 7f57b67fc700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a40682f0 0x7f57a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:10:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.194+0000 7f57b67fc700 1 -- 172.21.15.102:0/2996318993 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f57bc086f20 con 0x7f57c00cd8e0 2023-03-03T06:10:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.194+0000 7f57c579a700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a40682f0 0x7f57a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:10:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.195+0000 7f57c579a700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a40682f0 0x7f57a406a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f57c00b4d50 tx=0x7f57b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:10:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.198+0000 7f57b67fc700 1 -- 172.21.15.102:0/2996318993 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57bc0535b0 con 0x7f57c00cd8e0 2023-03-03T06:10:18.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.434+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57c00cc2d0 con 0x7f57c00cd8e0 2023-03-03T06:10:18.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.435+0000 7f57b67fc700 1 -- 172.21.15.102:0/2996318993 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f57bc015090 con 0x7f57c00cd8e0 2023-03-03T06:10:18.438 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:10:18.438 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:10:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a40682f0 msgr2=0x7f57a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:10:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a40682f0 0x7f57a406a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f57c00b4d50 tx=0x7f57b0009040 comp rx=0 tx=0).stop 2023-03-03T06:10:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57c00cd8e0 msgr2=0x7f57c01e5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:10:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57c00cd8e0 0x7f57c01e5c50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f57c00c9390 tx=0x7f57bc004750 comp rx=0 tx=0).stop 2023-03-03T06:10:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 shutdown_connections 2023-03-03T06:10:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a40682f0 0x7f57a406a7a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:10:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57c00cd8e0 0x7f57c01e5c50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:10:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57c01593c0 0x7f57c00b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:10:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 --2- 172.21.15.102:0/2996318993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57c0153290 0x7f57c00b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:10:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.438+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 >> 172.21.15.102:0/2996318993 conn(0x7f57c0065fb0 msgr2=0x7f57c015ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:10:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.439+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 shutdown_connections 2023-03-03T06:10:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:10:18.439+0000 7f57c81ff700 1 -- 172.21.15.102:0/2996318993 wait complete. 2023-03-03T06:10:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:10:18.468 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 103 secs before thrashing 2023-03-03T06:10:35.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:10:35.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:11:05.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:11:05.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:11:35.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:11:35.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:12:01.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:12:01.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.736+0000 7fbb269ff700 1 Processor -- start 2023-03-03T06:12:01.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.736+0000 7fbb269ff700 1 -- start start 2023-03-03T06:12:01.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.737+0000 7fbb269ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb20153260 0x7fbb200ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:01.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.737+0000 7fbb269ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb20163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:01.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.737+0000 7fbb269ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb200cf940 con 0x7fbb200c9cb0 2023-03-03T06:12:01.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.737+0000 7fbb269ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb200cfe70 con 0x7fbb20153260 2023-03-03T06:12:01.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.737+0000 7fbb269ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20158d70 con 0x7fbb200cac20 2023-03-03T06:12:01.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb20153260 0x7fbb200ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:01.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbb200c9cb0 0x7fbb200cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36152/0 (socket says 172.21.15.102:36152) 2023-03-03T06:12:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1f7fe700 1 -- 172.21.15.102:0/3036036485 learned_addr learned my addr 172.21.15.102:0/3036036485 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:12:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb24f9c700 1 --2- 172.21.15.102:0/3036036485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb20163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1effd700 1 -- 172.21.15.102:0/3036036485 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062890238 0 0) 0x7fbb200cf940 con 0x7fbb200c9cb0 2023-03-03T06:12:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1effd700 1 -- 172.21.15.102:0/3036036485 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb080035b0 con 0x7fbb200c9cb0 2023-03-03T06:12:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1ffff700 1 -- 172.21.15.102:0/3036036485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 msgr2=0x7fbb20163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:12:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1ffff700 1 --2- 172.21.15.102:0/3036036485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb20163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1ffff700 1 -- 172.21.15.102:0/3036036485 >> v1:172.21.15.102:6789/0 conn(0x7fbb200c9cb0 legacy=0x7fbb200cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:01.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb1ffff700 1 -- 172.21.15.102:0/3036036485 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200cfb90 con 0x7fbb20153260 2023-03-03T06:12:01.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.738+0000 7fbb24f9c700 1 --2- 172.21.15.102:0/3036036485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb20163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:12:01.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.739+0000 7fbb1ffff700 1 --2- 172.21.15.102:0/3036036485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20153260 0x7fbb200ca6e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009840 tx=0x7fbb10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1eac95c72eb3d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:01.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.739+0000 7fbb1effd700 1 -- 172.21.15.102:0/3036036485 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10011830 con 0x7fbb20153260 2023-03-03T06:12:01.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.739+0000 7fbb1effd700 1 -- 172.21.15.102:0/3036036485 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb10011990 con 0x7fbb20153260 2023-03-03T06:12:01.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.739+0000 7fbb1effd700 1 -- 172.21.15.102:0/3036036485 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10011d20 con 0x7fbb20153260 2023-03-03T06:12:01.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.739+0000 7fbb269ff700 1 -- 172.21.15.102:0/3036036485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20153260 msgr2=0x7fbb200ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:01.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.739+0000 7fbb269ff700 1 --2- 172.21.15.102:0/3036036485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20153260 0x7fbb200ca6e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009840 tx=0x7fbb10009b50 comp rx=0 tx=0).stop 2023-03-03T06:12:01.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.740+0000 7fbb269ff700 1 -- 172.21.15.102:0/3036036485 shutdown_connections 2023-03-03T06:12:01.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.740+0000 7fbb269ff700 1 --2- 172.21.15.102:0/3036036485 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb20163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.740+0000 7fbb269ff700 1 --2- 172.21.15.102:0/3036036485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20153260 0x7fbb200ca6e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.740+0000 7fbb269ff700 1 -- 172.21.15.102:0/3036036485 >> 172.21.15.102:0/3036036485 conn(0x7fbb20066c00 msgr2=0x7fbb200ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:12:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.740+0000 7fbb269ff700 1 -- 172.21.15.102:0/3036036485 shutdown_connections 2023-03-03T06:12:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.740+0000 7fbb269ff700 1 -- 172.21.15.102:0/3036036485 wait complete. 2023-03-03T06:12:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.742+0000 7fbb269ff700 1 Processor -- start 2023-03-03T06:12:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.742+0000 7fbb269ff700 1 -- start start 2023-03-03T06:12:01.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb269ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb200b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:01.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb269ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb20153260 0x7fbb20148550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:01.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb269ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb2006ba30 0x7fbb2015b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:01.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb269ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2015a810 con 0x7fbb2006ba30 2023-03-03T06:12:01.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb269ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb201536e0 con 0x7fbb20153260 2023-03-03T06:12:01.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb269ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2015b440 con 0x7fbb200cac20 2023-03-03T06:12:01.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb1ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb200b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:01.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb1f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb20153260 0x7fbb20148550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:01.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb1ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb200b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34110/0 (socket says 172.21.15.102:34110) 2023-03-03T06:12:01.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb1ffff700 1 -- 172.21.15.102:0/424559731 learned_addr learned my addr 172.21.15.102:0/424559731 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:12:01.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.743+0000 7fbb24f9c700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb2006ba30 0x7fbb2015b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:01.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb24f9c700 1 -- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 msgr2=0x7fbb200b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb24f9c700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb200b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb24f9c700 1 -- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb20153260 msgr2=0x7fbb20148550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb24f9c700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb20153260 0x7fbb20148550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb24f9c700 1 -- 172.21.15.102:0/424559731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb18009a70 con 0x7fbb2006ba30 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb24f9c700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb2006ba30 0x7fbb2015b700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800efc0 tx=0x7fbb1800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb1cff9700 1 -- 172.21.15.102:0/424559731 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800ca30 con 0x7fbb2006ba30 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb1cff9700 1 -- 172.21.15.102:0/424559731 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1800f040 con 0x7fbb2006ba30 2023-03-03T06:12:01.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb1cff9700 1 -- 172.21.15.102:0/424559731 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800cdd0 con 0x7fbb2006ba30 2023-03-03T06:12:01.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.744+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb100094a0 con 0x7fbb2006ba30 2023-03-03T06:12:01.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.745+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb20159260 con 0x7fbb2006ba30 2023-03-03T06:12:01.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.746+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb20158f80 con 0x7fbb2006ba30 2023-03-03T06:12:01.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.747+0000 7fbb1cff9700 1 -- 172.21.15.102:0/424559731 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb18014480 con 0x7fbb2006ba30 2023-03-03T06:12:01.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.748+0000 7fbb1cff9700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb0c068320 0x7fbb0c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:01.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.748+0000 7fbb1cff9700 1 -- 172.21.15.102:0/424559731 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbb180863b0 con 0x7fbb2006ba30 2023-03-03T06:12:01.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.748+0000 7fbb1ffff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb0c068320 0x7fbb0c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:01.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.749+0000 7fbb1ffff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb0c068320 0x7fbb0c06a7d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009b90 tx=0x7fbb1001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:01.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.752+0000 7fbb1cff9700 1 -- 172.21.15.102:0/424559731 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb18052ac0 con 0x7fbb2006ba30 2023-03-03T06:12:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.986+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb20159540 con 0x7fbb2006ba30 2023-03-03T06:12:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.987+0000 7fbb1cff9700 1 -- 172.21.15.102:0/424559731 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbb180560e0 con 0x7fbb2006ba30 2023-03-03T06:12:01.988 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:12:01.989 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:12:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb0c068320 msgr2=0x7fbb0c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb0c068320 0x7fbb0c06a7d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009b90 tx=0x7fbb1001c040 comp rx=0 tx=0).stop 2023-03-03T06:12:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb2006ba30 msgr2=0x7fbb2015b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb2006ba30 0x7fbb2015b700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800efc0 tx=0x7fbb1800c3a0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 shutdown_connections 2023-03-03T06:12:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb0c068320 0x7fbb0c06a7d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb200cac20 0x7fbb200b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb2006ba30 0x7fbb2015b700 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 --2- 172.21.15.102:0/424559731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb20153260 0x7fbb20148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 >> 172.21.15.102:0/424559731 conn(0x7fbb200c9cb0 msgr2=0x7fbb200b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:12:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 shutdown_connections 2023-03-03T06:12:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:01.989+0000 7fbb269ff700 1 -- 172.21.15.102:0/424559731 wait complete. 2023-03-03T06:12:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:12:02.018 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-03-03T06:12:06.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:12:06.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:12:36.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:12:36.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:12:48.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:12:48.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.268+0000 7efeb6d9d700 1 Processor -- start 2023-03-03T06:12:48.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.268+0000 7efeb6d9d700 1 -- start start 2023-03-03T06:12:48.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.269+0000 7efeb6d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efeb01564b0 0x7efeb00cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:48.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.269+0000 7efeb6d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb00c9ca0 con 0x7efeb00c9610 2023-03-03T06:12:48.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb6d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb00b46f0 con 0x7efeb00cab50 2023-03-03T06:12:48.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb6d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb00b4920 con 0x7efeb01564b0 2023-03-03T06:12:48.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb5d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efeb01564b0 0x7efeb00cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:48.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb559a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efeb00c9610 0x7efeb00ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33214/0 (socket says 172.21.15.102:33214) 2023-03-03T06:12:48.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb559a700 1 -- 172.21.15.102:0/2576667787 learned_addr learned my addr 172.21.15.102:0/2576667787 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:12:48.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb4d99700 1 -- 172.21.15.102:0/2576667787 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558000162 0 0) 0x7efeb00c9ca0 con 0x7efeb00c9610 2023-03-03T06:12:48.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb4d99700 1 -- 172.21.15.102:0/2576667787 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c0035b0 con 0x7efeb00c9610 2023-03-03T06:12:48.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb5d9b700 1 -- 172.21.15.102:0/2576667787 >> v1:172.21.15.172:6789/0 conn(0x7efeb00cab50 legacy=0x7efeb01636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:48.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb4d99700 1 -- 172.21.15.102:0/2576667787 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732788651 0 0) 0x7efeb00b46f0 con 0x7efeb00cab50 2023-03-03T06:12:48.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb5d9b700 1 -- 172.21.15.102:0/2576667787 >> v1:172.21.15.102:6789/0 conn(0x7efeb00c9610 legacy=0x7efeb00ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:48.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.270+0000 7efeb5d9b700 1 -- 172.21.15.102:0/2576667787 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00b4c00 con 0x7efeb01564b0 2023-03-03T06:12:48.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.271+0000 7efeb5d9b700 1 --2- 172.21.15.102:0/2576667787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb01564b0 0x7efeb00cb660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efeac009840 tx=0x7efeac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b863dbb0664d687 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:48.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.271+0000 7efeb4d99700 1 -- 172.21.15.102:0/2576667787 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac011830 con 0x7efeb01564b0 2023-03-03T06:12:48.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.271+0000 7efeb4d99700 1 -- 172.21.15.102:0/2576667787 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac011990 con 0x7efeb01564b0 2023-03-03T06:12:48.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.271+0000 7efeb4d99700 1 -- 172.21.15.102:0/2576667787 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac011d20 con 0x7efeb01564b0 2023-03-03T06:12:48.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.272+0000 7efeb6d9d700 1 -- 172.21.15.102:0/2576667787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb01564b0 msgr2=0x7efeb00cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:48.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.272+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/2576667787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb01564b0 0x7efeb00cb660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efeac009840 tx=0x7efeac009b50 comp rx=0 tx=0).stop 2023-03-03T06:12:48.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.272+0000 7efeb6d9d700 1 -- 172.21.15.102:0/2576667787 shutdown_connections 2023-03-03T06:12:48.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.272+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/2576667787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb01564b0 0x7efeb00cb660 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:48.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.272+0000 7efeb6d9d700 1 -- 172.21.15.102:0/2576667787 >> 172.21.15.102:0/2576667787 conn(0x7efeb0065fb0 msgr2=0x7efeb00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.272+0000 7efeb6d9d700 1 -- 172.21.15.102:0/2576667787 shutdown_connections 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.272+0000 7efeb6d9d700 1 -- 172.21.15.102:0/2576667787 wait complete. 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.274+0000 7efeb6d9d700 1 Processor -- start 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.274+0000 7efeb6d9d700 1 -- start start 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb6d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeb00c9610 0x7efeb00c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb6d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb00cab50 0x7efeb01ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb6d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efeb01564b0 0x7efeb01f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:48.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb6d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb015d160 con 0x7efeb01564b0 2023-03-03T06:12:48.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb6d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb00c2b60 con 0x7efeb00cab50 2023-03-03T06:12:48.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb6d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb00c2e20 con 0x7efeb00c9610 2023-03-03T06:12:48.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efeb01564b0 0x7efeb01f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:48.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb559a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb00cab50 0x7efeb01ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:48.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb5d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeb00c9610 0x7efeb00c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:48.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efeb01564b0 0x7efeb01f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52928/0 (socket says 172.21.15.102:52928) 2023-03-03T06:12:48.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.275+0000 7efeb659c700 1 -- 172.21.15.102:0/3977282136 learned_addr learned my addr 172.21.15.102:0/3977282136 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:12:48.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb659c700 1 -- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeb00c9610 msgr2=0x7efeb00c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:48.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb659c700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeb00c9610 0x7efeb00c3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:48.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb659c700 1 -- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb00cab50 msgr2=0x7efeb01ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:48.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb659c700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb00cab50 0x7efeb01ee6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:48.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb659c700 1 -- 172.21.15.102:0/3977282136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac0094a0 con 0x7efeb01564b0 2023-03-03T06:12:48.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb659c700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efeb01564b0 0x7efeb01f1c50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efea8001560 tx=0x7efea800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:48.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efea67fc700 1 -- 172.21.15.102:0/3977282136 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8003e30 con 0x7efeb01564b0 2023-03-03T06:12:48.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efea67fc700 1 -- 172.21.15.102:0/3977282136 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea8005070 con 0x7efeb01564b0 2023-03-03T06:12:48.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efea67fc700 1 -- 172.21.15.102:0/3977282136 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8005420 con 0x7efeb01564b0 2023-03-03T06:12:48.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb00b2210 con 0x7efeb01564b0 2023-03-03T06:12:48.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.276+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb00b1f30 con 0x7efeb01564b0 2023-03-03T06:12:48.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.278+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb0156880 con 0x7efeb01564b0 2023-03-03T06:12:48.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.279+0000 7efea67fc700 1 -- 172.21.15.102:0/3977282136 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efea8019030 con 0x7efeb01564b0 2023-03-03T06:12:48.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.280+0000 7efea67fc700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe940682f0 0x7efe9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:48.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.280+0000 7efea67fc700 1 -- 172.21.15.102:0/3977282136 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7efea8085360 con 0x7efeb01564b0 2023-03-03T06:12:48.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.280+0000 7efeb5d9b700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe940682f0 0x7efe9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:48.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.281+0000 7efeb5d9b700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe940682f0 0x7efe9406a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efeac009fd0 tx=0x7efeac01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:48.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.284+0000 7efea67fc700 1 -- 172.21.15.102:0/3977282136 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efea80519e0 con 0x7efeb01564b0 2023-03-03T06:12:48.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.516+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efeb00c5690 con 0x7efeb01564b0 2023-03-03T06:12:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.517+0000 7efea67fc700 1 -- 172.21.15.102:0/3977282136 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7efeb00c5690 con 0x7efeb01564b0 2023-03-03T06:12:48.519 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:12:48.519 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:12:48.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe940682f0 msgr2=0x7efe9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe940682f0 0x7efe9406a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efeac009fd0 tx=0x7efeac01c040 comp rx=0 tx=0).stop 2023-03-03T06:12:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efeb01564b0 msgr2=0x7efeb01f1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efeb01564b0 0x7efeb01f1c50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efea8001560 tx=0x7efea800cb50 comp rx=0 tx=0).stop 2023-03-03T06:12:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 shutdown_connections 2023-03-03T06:12:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe940682f0 0x7efe9406a7a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efeb00c9610 0x7efeb00c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efeb01564b0 0x7efeb01f1c50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 --2- 172.21.15.102:0/3977282136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efeb00cab50 0x7efeb01ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 >> 172.21.15.102:0/3977282136 conn(0x7efeb0065fb0 msgr2=0x7efeb0066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:12:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 shutdown_connections 2023-03-03T06:12:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:48.520+0000 7efeb6d9d700 1 -- 172.21.15.102:0/3977282136 wait complete. 2023-03-03T06:12:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:12:48.549 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before thrashing 2023-03-03T06:12:52.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:12:52.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.797+0000 7fe8cb3ff700 1 Processor -- start 2023-03-03T06:12:52.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.797+0000 7fe8cb3ff700 1 -- start start 2023-03-03T06:12:52.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.798+0000 7fe8cb3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe8c40cf9a0 0x7fe8c40cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:52.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.798+0000 7fe8cb3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c40cb9f0 con 0x7fe8c40ced80 2023-03-03T06:12:52.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.799+0000 7fe8cb3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c4159c80 con 0x7fe8c41597f0 2023-03-03T06:12:52.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.799+0000 7fe8cb3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c40ca750 con 0x7fe8c40cf9a0 2023-03-03T06:12:52.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.799+0000 7fe8c919b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe8c40ced80 0x7fe8c40cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33228/0 (socket says 172.21.15.102:33228) 2023-03-03T06:12:52.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.799+0000 7fe8c919b700 1 -- 172.21.15.102:0/1328616667 learned_addr learned my addr 172.21.15.102:0/1328616667 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:12:52.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.799+0000 7fe8c999c700 1 --2- 172.21.15.102:0/1328616667 >> v2:172.21.15.172:3300/0 conn(0x7fe8c40cf9a0 0x7fe8c40cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:52.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.799+0000 7fe8bbfff700 1 -- 172.21.15.102:0/1328616667 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406199893 0 0) 0x7fe8c40cb9f0 con 0x7fe8c40ced80 2023-03-03T06:12:52.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.799+0000 7fe8bbfff700 1 -- 172.21.15.102:0/1328616667 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b00035b0 con 0x7fe8c40ced80 2023-03-03T06:12:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8c999c700 1 -- 172.21.15.102:0/1328616667 >> v1:172.21.15.172:6789/0 conn(0x7fe8c41597f0 legacy=0x7fe8c4161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8c999c700 1 -- 172.21.15.102:0/1328616667 >> v1:172.21.15.102:6789/0 conn(0x7fe8c40ced80 legacy=0x7fe8c40cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8bbfff700 1 -- 172.21.15.102:0/1328616667 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800989469 0 0) 0x7fe8c4159c80 con 0x7fe8c41597f0 2023-03-03T06:12:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8c999c700 1 -- 172.21.15.102:0/1328616667 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c40cade0 con 0x7fe8c40cf9a0 2023-03-03T06:12:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8c999c700 1 --2- 172.21.15.102:0/1328616667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c40cf9a0 0x7fe8c40cf150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009d90 tx=0x7fe8bc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87dbdc1d4732ce48 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8bbfff700 1 -- 172.21.15.102:0/1328616667 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc00e070 con 0x7fe8c40cf9a0 2023-03-03T06:12:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8bbfff700 1 -- 172.21.15.102:0/1328616667 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8bc00bb20 con 0x7fe8c40cf9a0 2023-03-03T06:12:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.800+0000 7fe8bbfff700 1 -- 172.21.15.102:0/1328616667 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc00bed0 con 0x7fe8c40cf9a0 2023-03-03T06:12:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.801+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/1328616667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c40cf9a0 msgr2=0x7fe8c40cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.801+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/1328616667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c40cf9a0 0x7fe8c40cf150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009d90 tx=0x7fe8bc00b330 comp rx=0 tx=0).stop 2023-03-03T06:12:52.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.801+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/1328616667 shutdown_connections 2023-03-03T06:12:52.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.801+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/1328616667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c40cf9a0 0x7fe8c40cf150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:52.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.801+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/1328616667 >> 172.21.15.102:0/1328616667 conn(0x7fe8c4065fb0 msgr2=0x7fe8c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:12:52.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.801+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/1328616667 shutdown_connections 2023-03-03T06:12:52.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.801+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/1328616667 wait complete. 2023-03-03T06:12:52.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.804+0000 7fe8cb3ff700 1 Processor -- start 2023-03-03T06:12:52.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.804+0000 7fe8cb3ff700 1 -- start start 2023-03-03T06:12:52.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.805+0000 7fe8cb3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8c40ced80 0x7fe8c414d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:52.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.805+0000 7fe8cb3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8c40cf9a0 0x7fe8c40b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:52.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.805+0000 7fe8cb3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c41597f0 0x7fe8c41f1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:52.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.805+0000 7fe8cb3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c414b910 con 0x7fe8c40ced80 2023-03-03T06:12:52.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.805+0000 7fe8cb3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c414cbc0 con 0x7fe8c41597f0 2023-03-03T06:12:52.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.805+0000 7fe8cb3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c414c770 con 0x7fe8c40cf9a0 2023-03-03T06:12:52.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8c40ced80 0x7fe8c414d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:52.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c899a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8c40cf9a0 0x7fe8c40b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8c40ced80 0x7fe8c414d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52934/0 (socket says 172.21.15.102:52934) 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 -- 172.21.15.102:0/876487595 learned_addr learned my addr 172.21.15.102:0/876487595 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c999c700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c41597f0 0x7fe8c41f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 -- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8c40cf9a0 msgr2=0x7fe8c40b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8c40cf9a0 0x7fe8c40b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 -- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c41597f0 msgr2=0x7fe8c41f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c41597f0 0x7fe8c41f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:52.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c919b700 1 -- 172.21.15.102:0/876487595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc009a70 con 0x7fe8c40ced80 2023-03-03T06:12:52.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.806+0000 7fe8c999c700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c41597f0 0x7fe8c41f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:12:52.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.807+0000 7fe8c919b700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8c40ced80 0x7fe8c414d050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c420 tx=0x7fe8c000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:52.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.807+0000 7fe8b9ffb700 1 -- 172.21.15.102:0/876487595 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0005d70 con 0x7fe8c40ced80 2023-03-03T06:12:52.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.807+0000 7fe8b9ffb700 1 -- 172.21.15.102:0/876487595 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c000ce30 con 0x7fe8c40ced80 2023-03-03T06:12:52.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.807+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c40cb3a0 con 0x7fe8c40ced80 2023-03-03T06:12:52.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.807+0000 7fe8b9ffb700 1 -- 172.21.15.102:0/876487595 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c00072d0 con 0x7fe8c40ced80 2023-03-03T06:12:52.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.807+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c40cb0c0 con 0x7fe8c40ced80 2023-03-03T06:12:52.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.808+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c40cb680 con 0x7fe8c40ced80 2023-03-03T06:12:52.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.809+0000 7fe8b9ffb700 1 -- 172.21.15.102:0/876487595 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8c0019030 con 0x7fe8c40ced80 2023-03-03T06:12:52.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.810+0000 7fe8b9ffb700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8a80682f0 0x7fe8a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:12:52.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.811+0000 7fe8b9ffb700 1 -- 172.21.15.102:0/876487595 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe8c0084f00 con 0x7fe8c40ced80 2023-03-03T06:12:52.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.811+0000 7fe8c899a700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8a80682f0 0x7fe8a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:12:52.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.812+0000 7fe8c899a700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8a80682f0 0x7fe8a806a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40c93a0 tx=0x7fe8b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:12:52.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:52.815+0000 7fe8b9ffb700 1 -- 172.21.15.102:0/876487595 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8c004b5e0 con 0x7fe8c40ced80 2023-03-03T06:12:53.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.049+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8c40677f0 con 0x7fe8c40ced80 2023-03-03T06:12:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.049+0000 7fe8b9ffb700 1 -- 172.21.15.102:0/876487595 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe8c0017020 con 0x7fe8c40ced80 2023-03-03T06:12:53.051 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:12:53.052 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:12:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8a80682f0 msgr2=0x7fe8a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8a80682f0 0x7fe8a806a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40c93a0 tx=0x7fe8b4009040 comp rx=0 tx=0).stop 2023-03-03T06:12:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8c40ced80 msgr2=0x7fe8c414d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:12:53.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8c40ced80 0x7fe8c414d050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c420 tx=0x7fe8c000c730 comp rx=0 tx=0).stop 2023-03-03T06:12:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 shutdown_connections 2023-03-03T06:12:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8c41597f0 0x7fe8c41f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8c40ced80 0x7fe8c414d050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8a80682f0 0x7fe8a806a7a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 --2- 172.21.15.102:0/876487595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8c40cf9a0 0x7fe8c40b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:12:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 >> 172.21.15.102:0/876487595 conn(0x7fe8c4065fb0 msgr2=0x7fe8c40cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:12:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 shutdown_connections 2023-03-03T06:12:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:12:53.052+0000 7fe8cb3ff700 1 -- 172.21.15.102:0/876487595 wait complete. 2023-03-03T06:12:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:12:53.081 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T06:13:06.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:13:06.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:13:36.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:13:36.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:13:41.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:13:41.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.336+0000 7ff159d9d700 1 Processor -- start 2023-03-03T06:13:41.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.336+0000 7ff159d9d700 1 -- start start 2023-03-03T06:13:41.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff159d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff1540ced00 0x7ff1540c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:13:41.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff159d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540b3970 0x7ff1540c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:13:41.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff159d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540c3d20 con 0x7ff1540cf0d0 2023-03-03T06:13:41.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff159d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540c4930 con 0x7ff1540ced00 2023-03-03T06:13:41.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff159d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15415cba0 con 0x7ff1540b3970 2023-03-03T06:13:41.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff158d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff1540ced00 0x7ff1540c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:13:41.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff153fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540b3970 0x7ff1540c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:13:41.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff15959c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff1540cf0d0 0x7ff154158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37476/0 (socket says 172.21.15.102:37476) 2023-03-03T06:13:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.338+0000 7ff15959c700 1 -- 172.21.15.102:0/2412813026 learned_addr learned my addr 172.21.15.102:0/2412813026 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:13:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff1537fe700 1 -- 172.21.15.102:0/2412813026 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428851503 0 0) 0x7ff1540c3d20 con 0x7ff1540cf0d0 2023-03-03T06:13:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff1537fe700 1 -- 172.21.15.102:0/2412813026 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1440035b0 con 0x7ff1540cf0d0 2023-03-03T06:13:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff153fff700 1 -- 172.21.15.102:0/2412813026 >> v2:172.21.15.102:3300/0 conn(0x7ff1540ced00 msgr2=0x7ff1540c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:13:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff153fff700 1 --2- 172.21.15.102:0/2412813026 >> v2:172.21.15.102:3300/0 conn(0x7ff1540ced00 0x7ff1540c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff153fff700 1 -- 172.21.15.102:0/2412813026 >> v1:172.21.15.172:6789/0 conn(0x7ff1540cf0d0 legacy=0x7ff154158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:13:41.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff153fff700 1 -- 172.21.15.102:0/2412813026 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15415ce80 con 0x7ff1540b3970 2023-03-03T06:13:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff158d9b700 1 --2- 172.21.15.102:0/2412813026 >> v2:172.21.15.102:3300/0 conn(0x7ff1540ced00 0x7ff1540c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:13:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff153fff700 1 --2- 172.21.15.102:0/2412813026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540b3970 0x7ff1540c41b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff140009840 tx=0x7ff140009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa2390c96194374 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:13:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.339+0000 7ff1537fe700 1 -- 172.21.15.102:0/2412813026 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140011830 con 0x7ff1540b3970 2023-03-03T06:13:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff1537fe700 1 -- 172.21.15.102:0/2412813026 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff140011990 con 0x7ff1540b3970 2023-03-03T06:13:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff1537fe700 1 -- 172.21.15.102:0/2412813026 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140011d20 con 0x7ff1540b3970 2023-03-03T06:13:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 -- 172.21.15.102:0/2412813026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540b3970 msgr2=0x7ff1540c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:13:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 --2- 172.21.15.102:0/2412813026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540b3970 0x7ff1540c41b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff140009840 tx=0x7ff140009b50 comp rx=0 tx=0).stop 2023-03-03T06:13:41.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 -- 172.21.15.102:0/2412813026 shutdown_connections 2023-03-03T06:13:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 --2- 172.21.15.102:0/2412813026 >> v2:172.21.15.102:3300/0 conn(0x7ff1540ced00 0x7ff1540c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 --2- 172.21.15.102:0/2412813026 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540b3970 0x7ff1540c41b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 -- 172.21.15.102:0/2412813026 >> 172.21.15.102:0/2412813026 conn(0x7ff15408b1c0 msgr2=0x7ff154071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:13:41.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 -- 172.21.15.102:0/2412813026 shutdown_connections 2023-03-03T06:13:41.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.340+0000 7ff159d9d700 1 -- 172.21.15.102:0/2412813026 wait complete. 2023-03-03T06:13:41.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.342+0000 7ff159d9d700 1 Processor -- start 2023-03-03T06:13:41.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.342+0000 7ff159d9d700 1 -- start start 2023-03-03T06:13:41.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.343+0000 7ff159d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1540b3970 0x7ff1541e3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:13:41.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.343+0000 7ff159d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540ced00 0x7ff1541e3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:13:41.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.343+0000 7ff159d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1540cf0d0 0x7ff154146a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:13:41.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.343+0000 7ff159d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541e32e0 con 0x7ff1540b3970 2023-03-03T06:13:41.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.343+0000 7ff159d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541494c0 con 0x7ff1540cf0d0 2023-03-03T06:13:41.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.343+0000 7ff159d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541e2fb0 con 0x7ff1540ced00 2023-03-03T06:13:41.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1540b3970 0x7ff1541e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:13:41.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1540b3970 0x7ff1541e3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51930/0 (socket says 172.21.15.102:51930) 2023-03-03T06:13:41.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 -- 172.21.15.102:0/1408351384 learned_addr learned my addr 172.21.15.102:0/1408351384 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:13:41.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff153fff700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540ced00 0x7ff1541e3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:13:41.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff15959c700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1540cf0d0 0x7ff154146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:13:41.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 -- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540ced00 msgr2=0x7ff1541e3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:13:41.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540ced00 0x7ff1541e3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 -- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1540cf0d0 msgr2=0x7ff154146a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:13:41.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1540cf0d0 0x7ff154146a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.344+0000 7ff158d9b700 1 -- 172.21.15.102:0/1408351384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1480094a0 con 0x7ff1540b3970 2023-03-03T06:13:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.345+0000 7ff158d9b700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1540b3970 0x7ff1541e3770 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff14800ebe0 tx=0x7ff14800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:13:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.345+0000 7ff1517fa700 1 -- 172.21.15.102:0/1408351384 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800c850 con 0x7ff1540b3970 2023-03-03T06:13:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.345+0000 7ff1517fa700 1 -- 172.21.15.102:0/1408351384 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14800c9b0 con 0x7ff1540b3970 2023-03-03T06:13:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.345+0000 7ff1517fa700 1 -- 172.21.15.102:0/1408351384 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800cd40 con 0x7ff1540b3970 2023-03-03T06:13:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.345+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1400094a0 con 0x7ff1540b3970 2023-03-03T06:13:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.345+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15406c200 con 0x7ff1540b3970 2023-03-03T06:13:41.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.346+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15406bf20 con 0x7ff1540b3970 2023-03-03T06:13:41.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.347+0000 7ff1517fa700 1 -- 172.21.15.102:0/1408351384 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff148023070 con 0x7ff1540b3970 2023-03-03T06:13:41.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.349+0000 7ff1517fa700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1380682f0 0x7ff13806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:13:41.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.349+0000 7ff1517fa700 1 -- 172.21.15.102:0/1408351384 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff1480861a0 con 0x7ff1540b3970 2023-03-03T06:13:41.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.349+0000 7ff153fff700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1380682f0 0x7ff13806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:13:41.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.350+0000 7ff153fff700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1380682f0 0x7ff13806a7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff140009840 tx=0x7ff14001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:13:41.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.353+0000 7ff1517fa700 1 -- 172.21.15.102:0/1408351384 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff148052820 con 0x7ff1540b3970 2023-03-03T06:13:41.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.590+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1540cf540 con 0x7ff1540b3970 2023-03-03T06:13:41.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.591+0000 7ff1517fa700 1 -- 172.21.15.102:0/1408351384 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff14800cea0 con 0x7ff1540b3970 2023-03-03T06:13:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:13:41.596 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:13:41.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1380682f0 msgr2=0x7ff13806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:13:41.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1380682f0 0x7ff13806a7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff140009840 tx=0x7ff14001c040 comp rx=0 tx=0).stop 2023-03-03T06:13:41.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1540b3970 msgr2=0x7ff1541e3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:13:41.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1540b3970 0x7ff1541e3770 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff14800ebe0 tx=0x7ff14800eef0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 shutdown_connections 2023-03-03T06:13:41.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1540cf0d0 0x7ff154146a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1540b3970 0x7ff1541e3770 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1380682f0 0x7ff13806a7a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 --2- 172.21.15.102:0/1408351384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1540ced00 0x7ff1541e3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:13:41.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 >> 172.21.15.102:0/1408351384 conn(0x7ff15408b1c0 msgr2=0x7ff154071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:13:41.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 shutdown_connections 2023-03-03T06:13:41.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:13:41.593+0000 7ff159d9d700 1 -- 172.21.15.102:0/1408351384 wait complete. 2023-03-03T06:13:41.599 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:13:41.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-03-03T06:14:04.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:14:04.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.871+0000 7f4328b9d700 1 Processor -- start 2023-03-03T06:14:04.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.871+0000 7f4328b9d700 1 -- start start 2023-03-03T06:14:04.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.873+0000 7f4328b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4324153030 0x7f43240c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:04.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.873+0000 7f4328b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432406bee0 con 0x7f43240b3f30 2023-03-03T06:14:04.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.873+0000 7f4328b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432406c1c0 con 0x7f43240b1e90 2023-03-03T06:14:04.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.873+0000 7f4328b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432406c780 con 0x7f4324153030 2023-03-03T06:14:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.873+0000 7f43237fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4324153030 0x7f43240c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.873+0000 7f4323fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f43240b3f30 0x7f43240cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55454/0 (socket says 172.21.15.102:55454) 2023-03-03T06:14:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.873+0000 7f4323fff700 1 -- 172.21.15.102:0/1438124378 learned_addr learned my addr 172.21.15.102:0/1438124378 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:14:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.874+0000 7f43237fe700 1 -- 172.21.15.102:0/1438124378 >> v1:172.21.15.172:6789/0 conn(0x7f43240b1e90 legacy=0x7f432415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.874+0000 7f43237fe700 1 -- 172.21.15.102:0/1438124378 >> v1:172.21.15.102:6789/0 conn(0x7f43240b3f30 legacy=0x7f43240cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.874+0000 7f43237fe700 1 -- 172.21.15.102:0/1438124378 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432406c4a0 con 0x7f4324153030 2023-03-03T06:14:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.874+0000 7f43237fe700 1 --2- 172.21.15.102:0/1438124378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4324153030 0x7f43240c5b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4314009870 tx=0x7f4314009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b4d619ebd535517 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:14:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.874+0000 7f43227fc700 1 -- 172.21.15.102:0/1438124378 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314011830 con 0x7f4324153030 2023-03-03T06:14:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f43227fc700 1 -- 172.21.15.102:0/1438124378 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4314011990 con 0x7f4324153030 2023-03-03T06:14:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f43227fc700 1 -- 172.21.15.102:0/1438124378 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314011d20 con 0x7f4324153030 2023-03-03T06:14:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f4328b9d700 1 -- 172.21.15.102:0/1438124378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4324153030 msgr2=0x7f43240c5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1438124378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4324153030 0x7f43240c5b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4314009870 tx=0x7f4314009b80 comp rx=0 tx=0).stop 2023-03-03T06:14:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f4328b9d700 1 -- 172.21.15.102:0/1438124378 shutdown_connections 2023-03-03T06:14:04.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1438124378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4324153030 0x7f43240c5b00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:04.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f4328b9d700 1 -- 172.21.15.102:0/1438124378 >> 172.21.15.102:0/1438124378 conn(0x7f432408b1c0 msgr2=0x7f4324071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:14:04.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f4328b9d700 1 -- 172.21.15.102:0/1438124378 shutdown_connections 2023-03-03T06:14:04.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.875+0000 7f4328b9d700 1 -- 172.21.15.102:0/1438124378 wait complete. 2023-03-03T06:14:04.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.877+0000 7f4328b9d700 1 Processor -- start 2023-03-03T06:14:04.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.877+0000 7f4328b9d700 1 -- start start 2023-03-03T06:14:04.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.878+0000 7f4328b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43240b1e90 0x7f43240c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:04.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.878+0000 7f4328b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43240b3f30 0x7f432414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:04.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f4328b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4324153030 0x7f43241f0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:04.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f4328b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432415c620 con 0x7f43240b1e90 2023-03-03T06:14:04.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f4328b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43240cb8f0 con 0x7f43240b3f30 2023-03-03T06:14:04.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f4328b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43240c6260 con 0x7f4324153030 2023-03-03T06:14:04.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43240b1e90 0x7f43240c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:04.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43240b1e90 0x7f43240c5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33448/0 (socket says 172.21.15.102:33448) 2023-03-03T06:14:04.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 -- 172.21.15.102:0/1467940826 learned_addr learned my addr 172.21.15.102:0/1467940826 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:14:04.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f4322ffd700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43240b3f30 0x7f432414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:04.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f4323fff700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4324153030 0x7f43241f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:04.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 -- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4324153030 msgr2=0x7f43241f0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:04.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4324153030 0x7f43241f0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:04.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 -- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43240b3f30 msgr2=0x7f432414f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:04.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43240b3f30 0x7f432414f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:04.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f43237fe700 1 -- 172.21.15.102:0/1467940826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43140094a0 con 0x7f43240b1e90 2023-03-03T06:14:04.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.879+0000 7f4322ffd700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43240b3f30 0x7f432414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:14:04.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.880+0000 7f43237fe700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43240b1e90 0x7f43240c5d20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4314009bc0 tx=0x7f4314004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:14:04.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.880+0000 7f430bfff700 1 -- 172.21.15.102:0/1467940826 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43140036a0 con 0x7f43240b1e90 2023-03-03T06:14:04.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.880+0000 7f430bfff700 1 -- 172.21.15.102:0/1467940826 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431401e070 con 0x7f43240b1e90 2023-03-03T06:14:04.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.880+0000 7f430bfff700 1 -- 172.21.15.102:0/1467940826 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314003a60 con 0x7f43240b1e90 2023-03-03T06:14:04.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.880+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43240cf3f0 con 0x7f43240b1e90 2023-03-03T06:14:04.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.881+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432415c240 con 0x7f43240b1e90 2023-03-03T06:14:04.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.882+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43240cf580 con 0x7f43240b1e90 2023-03-03T06:14:04.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.883+0000 7f430bfff700 1 -- 172.21.15.102:0/1467940826 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f431400f050 con 0x7f43240b1e90 2023-03-03T06:14:04.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.884+0000 7f430bfff700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43100682f0 0x7f431006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:04.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.884+0000 7f430bfff700 1 -- 172.21.15.102:0/1467940826 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4314086e70 con 0x7f43240b1e90 2023-03-03T06:14:04.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.884+0000 7f4322ffd700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43100682f0 0x7f431006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.885+0000 7f4322ffd700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43100682f0 0x7f431006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f432415c210 tx=0x7f4318009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:14:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:04.888+0000 7f430bfff700 1 -- 172.21.15.102:0/1467940826 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4314053580 con 0x7f43240b1e90 2023-03-03T06:14:05.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.121+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43241f03b0 con 0x7f43240b1e90 2023-03-03T06:14:05.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.122+0000 7f430bfff700 1 -- 172.21.15.102:0/1467940826 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4314015090 con 0x7f43240b1e90 2023-03-03T06:14:05.125 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:14:05.126 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:14:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43100682f0 msgr2=0x7f431006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43100682f0 0x7f431006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f432415c210 tx=0x7f4318009040 comp rx=0 tx=0).stop 2023-03-03T06:14:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43240b1e90 msgr2=0x7f43240c5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:05.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43240b1e90 0x7f43240c5d20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4314009bc0 tx=0x7f4314004750 comp rx=0 tx=0).stop 2023-03-03T06:14:05.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 shutdown_connections 2023-03-03T06:14:05.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43100682f0 0x7f431006a7a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:05.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4324153030 0x7f43241f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:05.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43240b1e90 0x7f43240c5d20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:05.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 --2- 172.21.15.102:0/1467940826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43240b3f30 0x7f432414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:05.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.124+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 >> 172.21.15.102:0/1467940826 conn(0x7f432408b1c0 msgr2=0x7f43241e5470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:14:05.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.125+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 shutdown_connections 2023-03-03T06:14:05.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:05.125+0000 7f4328b9d700 1 -- 172.21.15.102:0/1467940826 wait complete. 2023-03-03T06:14:05.130 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:14:05.155 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2023-03-03T06:14:06.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:14:07.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:14:37.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:14:37.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:14:54.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:14:54.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.402+0000 7f732b3ff700 1 Processor -- start 2023-03-03T06:14:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.402+0000 7f732b3ff700 1 -- start start 2023-03-03T06:14:54.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.403+0000 7f732b3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7324153290 0x7f7324158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:54.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.403+0000 7f732b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73240cd8e0 0x7f73240ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.403+0000 7f732b3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7324153660 con 0x7f73241593c0 2023-03-03T06:14:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732b3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732406c360 con 0x7f7324153290 2023-03-03T06:14:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732415cea0 con 0x7f73240cd8e0 2023-03-03T06:14:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732919b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7324153290 0x7f7324158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732899a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f73241593c0 0x7f73240cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:39976/0 (socket says 172.21.15.102:39976) 2023-03-03T06:14:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732899a700 1 -- 172.21.15.102:0/2628440301 learned_addr learned my addr 172.21.15.102:0/2628440301 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:14:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732999c700 1 --2- 172.21.15.102:0/2628440301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73240cd8e0 0x7f73240ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:54.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732999c700 1 -- 172.21.15.102:0/2628440301 >> v2:172.21.15.102:3300/0 conn(0x7f7324153290 msgr2=0x7f7324158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:14:54.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f731bfff700 1 -- 172.21.15.102:0/2628440301 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895212304 0 0) 0x7f7324153660 con 0x7f73241593c0 2023-03-03T06:14:54.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732999c700 1 --2- 172.21.15.102:0/2628440301 >> v2:172.21.15.102:3300/0 conn(0x7f7324153290 0x7f7324158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732999c700 1 -- 172.21.15.102:0/2628440301 >> v1:172.21.15.172:6789/0 conn(0x7f73241593c0 legacy=0x7f73240cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:54.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732999c700 1 -- 172.21.15.102:0/2628440301 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732406c7f0 con 0x7f73240cd8e0 2023-03-03T06:14:54.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.404+0000 7f732919b700 1 --2- 172.21.15.102:0/2628440301 >> v2:172.21.15.102:3300/0 conn(0x7f7324153290 0x7f7324158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:14:54.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.405+0000 7f732999c700 1 --2- 172.21.15.102:0/2628440301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73240cd8e0 0x7f73240ce530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f731c009d90 tx=0x7f731c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f6d9195472d5042 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:14:54.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.405+0000 7f731bfff700 1 -- 172.21.15.102:0/2628440301 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c00e070 con 0x7f73240cd8e0 2023-03-03T06:14:54.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.405+0000 7f731bfff700 1 -- 172.21.15.102:0/2628440301 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c00bb20 con 0x7f73240cd8e0 2023-03-03T06:14:54.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.405+0000 7f731bfff700 1 -- 172.21.15.102:0/2628440301 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c012430 con 0x7f73240cd8e0 2023-03-03T06:14:54.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.405+0000 7f732b3ff700 1 -- 172.21.15.102:0/2628440301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73240cd8e0 msgr2=0x7f73240ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:54.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.405+0000 7f732b3ff700 1 --2- 172.21.15.102:0/2628440301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73240cd8e0 0x7f73240ce530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f731c009d90 tx=0x7f731c00b330 comp rx=0 tx=0).stop 2023-03-03T06:14:54.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.406+0000 7f732b3ff700 1 -- 172.21.15.102:0/2628440301 shutdown_connections 2023-03-03T06:14:54.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.406+0000 7f732b3ff700 1 --2- 172.21.15.102:0/2628440301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73240cd8e0 0x7f73240ce530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.406+0000 7f732b3ff700 1 --2- 172.21.15.102:0/2628440301 >> v2:172.21.15.102:3300/0 conn(0x7f7324153290 0x7f7324158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.406+0000 7f732b3ff700 1 -- 172.21.15.102:0/2628440301 >> 172.21.15.102:0/2628440301 conn(0x7f7324065fb0 msgr2=0x7f73240663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:14:54.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.406+0000 7f732b3ff700 1 -- 172.21.15.102:0/2628440301 shutdown_connections 2023-03-03T06:14:54.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.406+0000 7f732b3ff700 1 -- 172.21.15.102:0/2628440301 wait complete. 2023-03-03T06:14:54.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.408+0000 7f732b3ff700 1 Processor -- start 2023-03-03T06:14:54.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.408+0000 7f732b3ff700 1 -- start start 2023-03-03T06:14:54.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732b3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73240cd8e0 0x7f73240c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:54.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732b3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7324153290 0x7f73241f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73241593c0 0x7f73240c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732b3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240c83d0 con 0x7f73240cd8e0 2023-03-03T06:14:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732b3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240cbe90 con 0x7f7324153290 2023-03-03T06:14:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73241e60d0 con 0x7f73241593c0 2023-03-03T06:14:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73240cd8e0 0x7f73240c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73240cd8e0 0x7f73240c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55398/0 (socket says 172.21.15.102:55398) 2023-03-03T06:14:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732919b700 1 -- 172.21.15.102:0/420598796 learned_addr learned my addr 172.21.15.102:0/420598796 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:14:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732899a700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7324153290 0x7f73241f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.409+0000 7f732999c700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73241593c0 0x7f73240c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f732919b700 1 -- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73241593c0 msgr2=0x7f73240c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f732919b700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73241593c0 0x7f73240c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f732919b700 1 -- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7324153290 msgr2=0x7f73241f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f732919b700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7324153290 0x7f73241f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f732919b700 1 -- 172.21.15.102:0/420598796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200094a0 con 0x7f73240cd8e0 2023-03-03T06:14:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f732919b700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73240cd8e0 0x7f73240c62f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f732000ebe0 tx=0x7f732000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:14:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f7319ffb700 1 -- 172.21.15.102:0/420598796 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320009760 con 0x7f73240cd8e0 2023-03-03T06:14:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f7319ffb700 1 -- 172.21.15.102:0/420598796 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732000c850 con 0x7f73240cd8e0 2023-03-03T06:14:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c009a70 con 0x7f73240cd8e0 2023-03-03T06:14:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.410+0000 7f7319ffb700 1 -- 172.21.15.102:0/420598796 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732000cbe0 con 0x7f73240cd8e0 2023-03-03T06:14:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.411+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73240ccde0 con 0x7f73240cd8e0 2023-03-03T06:14:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.412+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732415d160 con 0x7f73240cd8e0 2023-03-03T06:14:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.413+0000 7f7319ffb700 1 -- 172.21.15.102:0/420598796 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7320023070 con 0x7f73240cd8e0 2023-03-03T06:14:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.414+0000 7f7319ffb700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f73080682f0 0x7f730806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:14:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.414+0000 7f7319ffb700 1 -- 172.21.15.102:0/420598796 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7320085ff0 con 0x7f73240cd8e0 2023-03-03T06:14:54.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.414+0000 7f732899a700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f73080682f0 0x7f730806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:14:54.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.415+0000 7f732899a700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f73080682f0 0x7f730806a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f732414e2d0 tx=0x7f7314007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:14:54.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.418+0000 7f7319ffb700 1 -- 172.21.15.102:0/420598796 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7320052700 con 0x7f73240cd8e0 2023-03-03T06:14:54.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.656+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73240cd0c0 con 0x7f73240cd8e0 2023-03-03T06:14:54.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.656+0000 7f7319ffb700 1 -- 172.21.15.102:0/420598796 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7320055d20 con 0x7f73240cd8e0 2023-03-03T06:14:54.660 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:14:54.660 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:14:54.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f73080682f0 msgr2=0x7f730806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:54.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f73080682f0 0x7f730806a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f732414e2d0 tx=0x7f7314007de0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73240cd8e0 msgr2=0x7f73240c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:14:54.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73240cd8e0 0x7f73240c62f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f732000ebe0 tx=0x7f732000eef0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 shutdown_connections 2023-03-03T06:14:54.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f73080682f0 0x7f730806a7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f73241593c0 0x7f73240c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73240cd8e0 0x7f73240c62f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 --2- 172.21.15.102:0/420598796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7324153290 0x7f73241f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:14:54.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 >> 172.21.15.102:0/420598796 conn(0x7f7324065fb0 msgr2=0x7f7324156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:14:54.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 shutdown_connections 2023-03-03T06:14:54.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:14:54.659+0000 7f732b3ff700 1 -- 172.21.15.102:0/420598796 wait complete. 2023-03-03T06:14:54.666 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:14:54.689 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T06:15:07.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:15:07.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:15:37.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:15:37.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:16:04.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:16:04.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.940+0000 7f02989ff700 1 Processor -- start 2023-03-03T06:16:04.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.941+0000 7f02989ff700 1 -- start start 2023-03-03T06:16:04.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.942+0000 7f02989ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd390 0x7f02940ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:16:04.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.942+0000 7f02989ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0294153660 con 0x7f02940cd8e0 2023-03-03T06:16:04.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.942+0000 7f02989ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029406c360 con 0x7f0294153290 2023-03-03T06:16:04.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.942+0000 7f02989ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029415cea0 con 0x7f02940cd390 2023-03-03T06:16:04.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.942+0000 7f029259c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0294153290 0x7f0294158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59344/0 (socket says 172.21.15.102:59344) 2023-03-03T06:16:04.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.942+0000 7f029259c700 1 -- 172.21.15.102:0/3187890042 learned_addr learned my addr 172.21.15.102:0/3187890042 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:16:04.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.942+0000 7f0292d9d700 1 --2- 172.21.15.102:0/3187890042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd390 0x7f02940ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:16:04.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.943+0000 7f029159a700 1 -- 172.21.15.102:0/3187890042 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259742159 0 0) 0x7f029406c360 con 0x7f0294153290 2023-03-03T06:16:04.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.943+0000 7f029159a700 1 -- 172.21.15.102:0/3187890042 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f027c0035b0 con 0x7f0294153290 2023-03-03T06:16:04.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.943+0000 7f0292d9d700 1 -- 172.21.15.102:0/3187890042 >> v1:172.21.15.172:6789/0 conn(0x7f0294153290 legacy=0x7f0294158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:16:04.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.943+0000 7f029159a700 1 -- 172.21.15.102:0/3187890042 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285569481 0 0) 0x7f0294153660 con 0x7f02940cd8e0 2023-03-03T06:16:04.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.943+0000 7f0292d9d700 1 -- 172.21.15.102:0/3187890042 >> v1:172.21.15.102:6789/0 conn(0x7f02940cd8e0 legacy=0x7f02940cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:16:04.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.943+0000 7f0292d9d700 1 -- 172.21.15.102:0/3187890042 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029406c7f0 con 0x7f02940cd390 2023-03-03T06:16:04.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.943+0000 7f0292d9d700 1 --2- 172.21.15.102:0/3187890042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd390 0x7f02940ce530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f028c009dc0 tx=0x7f028c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71c75211629e6a83 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:16:04.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f029159a700 1 -- 172.21.15.102:0/3187890042 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c00e070 con 0x7f02940cd390 2023-03-03T06:16:04.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f029159a700 1 -- 172.21.15.102:0/3187890042 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c00bb20 con 0x7f02940cd390 2023-03-03T06:16:04.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f029159a700 1 -- 172.21.15.102:0/3187890042 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c00bed0 con 0x7f02940cd390 2023-03-03T06:16:04.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f02989ff700 1 -- 172.21.15.102:0/3187890042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd390 msgr2=0x7f02940ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:16:04.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f02989ff700 1 --2- 172.21.15.102:0/3187890042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd390 0x7f02940ce530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f028c009dc0 tx=0x7f028c00b330 comp rx=0 tx=0).stop 2023-03-03T06:16:04.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f02989ff700 1 -- 172.21.15.102:0/3187890042 shutdown_connections 2023-03-03T06:16:04.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f02989ff700 1 --2- 172.21.15.102:0/3187890042 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd390 0x7f02940ce530 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:16:04.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f02989ff700 1 -- 172.21.15.102:0/3187890042 >> 172.21.15.102:0/3187890042 conn(0x7f0294065fb0 msgr2=0x7f02940663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:16:04.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f02989ff700 1 -- 172.21.15.102:0/3187890042 shutdown_connections 2023-03-03T06:16:04.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.944+0000 7f02989ff700 1 -- 172.21.15.102:0/3187890042 wait complete. 2023-03-03T06:16:04.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.946+0000 7f02989ff700 1 Processor -- start 2023-03-03T06:16:04.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.947+0000 7f02989ff700 1 -- start start 2023-03-03T06:16:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f02989ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02940cd390 0x7f0294157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:16:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f02989ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd8e0 0x7f02940b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:16:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f02989ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294153290 0x7f02941ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:16:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f02989ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0294156710 con 0x7f02940cd390 2023-03-03T06:16:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f02989ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02940c8a00 con 0x7f0294153290 2023-03-03T06:16:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f02989ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02940c5c30 con 0x7f02940cd8e0 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02940cd390 0x7f0294157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02940cd390 0x7f0294157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38636/0 (socket says 172.21.15.102:38636) 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 -- 172.21.15.102:0/516435769 learned_addr learned my addr 172.21.15.102:0/516435769 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f0292d9d700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294153290 0x7f02941ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f0291d9b700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd8e0 0x7f02940b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 -- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd8e0 msgr2=0x7f02940b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd8e0 0x7f02940b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:16:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 -- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294153290 msgr2=0x7f02941ef870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:16:04.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294153290 0x7f02941ef870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:16:04.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.948+0000 7f029259c700 1 -- 172.21.15.102:0/516435769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c009a70 con 0x7f02940cd390 2023-03-03T06:16:04.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.949+0000 7f029259c700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02940cd390 0x7f0294157c70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f028400c4d0 tx=0x7f028400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:16:04.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.949+0000 7f0282ffd700 1 -- 172.21.15.102:0/516435769 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284005d70 con 0x7f02940cd390 2023-03-03T06:16:04.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.949+0000 7f0282ffd700 1 -- 172.21.15.102:0/516435769 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02840070d0 con 0x7f02940cd390 2023-03-03T06:16:04.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.949+0000 7f0282ffd700 1 -- 172.21.15.102:0/516435769 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284007460 con 0x7f02940cd390 2023-03-03T06:16:04.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.949+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02940ccde0 con 0x7f02940cd390 2023-03-03T06:16:04.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.949+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029415d160 con 0x7f02940cd390 2023-03-03T06:16:04.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.950+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02940cd0c0 con 0x7f02940cd390 2023-03-03T06:16:04.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.952+0000 7f0282ffd700 1 -- 172.21.15.102:0/516435769 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0284019030 con 0x7f02940cd390 2023-03-03T06:16:04.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.953+0000 7f0282ffd700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02740682f0 0x7f027406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:16:04.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.953+0000 7f0282ffd700 1 -- 172.21.15.102:0/516435769 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f02840850e0 con 0x7f02940cd390 2023-03-03T06:16:04.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.953+0000 7f0291d9b700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02740682f0 0x7f027406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:16:04.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.954+0000 7f0291d9b700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02740682f0 0x7f027406a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0294150360 tx=0x7f0288009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:16:04.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:04.957+0000 7f0282ffd700 1 -- 172.21.15.102:0/516435769 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f02840517f0 con 0x7f02940cd390 2023-03-03T06:16:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.190+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f029404e6e0 con 0x7f02940cd390 2023-03-03T06:16:05.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.191+0000 7f0282ffd700 1 -- 172.21.15.102:0/516435769 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0284054e10 con 0x7f02940cd390 2023-03-03T06:16:05.193 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:16:05.194 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:16:05.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.193+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02740682f0 msgr2=0x7f027406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:16:05.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.193+0000 7f02989ff700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02740682f0 0x7f027406a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0294150360 tx=0x7f0288009040 comp rx=0 tx=0).stop 2023-03-03T06:16:05.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.193+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02940cd390 msgr2=0x7f0294157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:16:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.193+0000 7f02989ff700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02940cd390 0x7f0294157c70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f028400c4d0 tx=0x7f028400c7e0 comp rx=0 tx=0).stop 2023-03-03T06:16:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 shutdown_connections 2023-03-03T06:16:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294153290 0x7f02941ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:16:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02940cd390 0x7f0294157c70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:16:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02740682f0 0x7f027406a7a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:16:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 --2- 172.21.15.102:0/516435769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02940cd8e0 0x7f02940b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:16:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 >> 172.21.15.102:0/516435769 conn(0x7f0294065fb0 msgr2=0x7f0294157560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:16:05.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 shutdown_connections 2023-03-03T06:16:05.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:16:05.194+0000 7f02989ff700 1 -- 172.21.15.102:0/516435769 wait complete. 2023-03-03T06:16:05.197 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:16:05.222 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 100 secs before thrashing 2023-03-03T06:16:07.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:16:07.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:16:37.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:16:37.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:08.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:08.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:38.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:38.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:45.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:17:45.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.471+0000 7fe7937ff700 1 Processor -- start 2023-03-03T06:17:45.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.471+0000 7fe7937ff700 1 -- start start 2023-03-03T06:17:45.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe7937ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe78c153290 0x7fe78c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:45.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe7937ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe78c1593c0 0x7fe78c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:45.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe7937ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c153660 con 0x7fe78c0cd8e0 2023-03-03T06:17:45.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe7937ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe78c06c360 con 0x7fe78c1593c0 2023-03-03T06:17:45.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe7937ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe78c15cea0 con 0x7fe78c153290 2023-03-03T06:17:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe790d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe78c1593c0 0x7fe78c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe79159b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe78c153290 0x7fe78c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe791d9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe78c0cd8e0 0x7fe78c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45620/0 (socket says 172.21.15.102:45620) 2023-03-03T06:17:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.473+0000 7fe791d9c700 1 -- 172.21.15.102:0/3417292774 learned_addr learned my addr 172.21.15.102:0/3417292774 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:17:45.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe79159b700 1 -- 172.21.15.102:0/3417292774 >> v2:172.21.15.102:3300/0 conn(0x7fe78c1593c0 msgr2=0x7fe78c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe79159b700 1 --2- 172.21.15.102:0/3417292774 >> v2:172.21.15.102:3300/0 conn(0x7fe78c1593c0 0x7fe78c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe79159b700 1 -- 172.21.15.102:0/3417292774 >> v1:172.21.15.172:6789/0 conn(0x7fe78c0cd8e0 legacy=0x7fe78c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe79159b700 1 -- 172.21.15.102:0/3417292774 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c06c7f0 con 0x7fe78c153290 2023-03-03T06:17:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe790d9a700 1 --2- 172.21.15.102:0/3417292774 >> v2:172.21.15.102:3300/0 conn(0x7fe78c1593c0 0x7fe78c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe783fff700 1 -- 172.21.15.102:0/3417292774 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407828068 0 0) 0x7fe78c153660 con 0x7fe78c0cd8e0 2023-03-03T06:17:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe79159b700 1 --2- 172.21.15.102:0/3417292774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c153290 0x7fe78c158e80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe78c08cc70 tx=0x7fe788009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b98ba97e1a37ef6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:45.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe783fff700 1 -- 172.21.15.102:0/3417292774 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788011830 con 0x7fe78c153290 2023-03-03T06:17:45.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.474+0000 7fe783fff700 1 -- 172.21.15.102:0/3417292774 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788011990 con 0x7fe78c153290 2023-03-03T06:17:45.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe783fff700 1 -- 172.21.15.102:0/3417292774 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788011d20 con 0x7fe78c153290 2023-03-03T06:17:45.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 -- 172.21.15.102:0/3417292774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c153290 msgr2=0x7fe78c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3417292774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c153290 0x7fe78c158e80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe78c08cc70 tx=0x7fe788009b20 comp rx=0 tx=0).stop 2023-03-03T06:17:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 -- 172.21.15.102:0/3417292774 shutdown_connections 2023-03-03T06:17:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3417292774 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c153290 0x7fe78c158e80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3417292774 >> v2:172.21.15.102:3300/0 conn(0x7fe78c1593c0 0x7fe78c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 -- 172.21.15.102:0/3417292774 >> 172.21.15.102:0/3417292774 conn(0x7fe78c065fb0 msgr2=0x7fe78c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:45.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 -- 172.21.15.102:0/3417292774 shutdown_connections 2023-03-03T06:17:45.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.475+0000 7fe7937ff700 1 -- 172.21.15.102:0/3417292774 wait complete. 2023-03-03T06:17:45.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.478+0000 7fe7937ff700 1 Processor -- start 2023-03-03T06:17:45.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.478+0000 7fe7937ff700 1 -- start start 2023-03-03T06:17:45.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe7937ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe78c0cd8e0 0x7fe78c1ef690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe7937ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe78c153290 0x7fe78c0b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe7937ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c1593c0 0x7fe78c0cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe7937ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c0c65a0 con 0x7fe78c153290 2023-03-03T06:17:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe7937ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c15b3b0 con 0x7fe78c1593c0 2023-03-03T06:17:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe7937ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c0b2360 con 0x7fe78c0cd8e0 2023-03-03T06:17:45.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe790d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe78c153290 0x7fe78c0b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe790d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe78c153290 0x7fe78c0b27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40638/0 (socket says 172.21.15.102:40638) 2023-03-03T06:17:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe790d9a700 1 -- 172.21.15.102:0/3281088508 learned_addr learned my addr 172.21.15.102:0/3281088508 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:17:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe791d9c700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c1593c0 0x7fe78c0cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:45.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe79159b700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe78c0cd8e0 0x7fe78c1ef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe790d9a700 1 -- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe78c0cd8e0 msgr2=0x7fe78c1ef690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe790d9a700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe78c0cd8e0 0x7fe78c1ef690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.480+0000 7fe790d9a700 1 -- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c1593c0 msgr2=0x7fe78c0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe790d9a700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c1593c0 0x7fe78c0cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe790d9a700 1 -- 172.21.15.102:0/3281088508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c0094a0 con 0x7fe78c153290 2023-03-03T06:17:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe790d9a700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe78c153290 0x7fe78c0b27f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00ec10 tx=0x7fe77c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe781ffb700 1 -- 172.21.15.102:0/3281088508 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c00c7f0 con 0x7fe78c153290 2023-03-03T06:17:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe781ffb700 1 -- 172.21.15.102:0/3281088508 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77c00c950 con 0x7fe78c153290 2023-03-03T06:17:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7880094a0 con 0x7fe78c153290 2023-03-03T06:17:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe781ffb700 1 -- 172.21.15.102:0/3281088508 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c00ccc0 con 0x7fe78c153290 2023-03-03T06:17:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.481+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c0ccde0 con 0x7fe78c153290 2023-03-03T06:17:45.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.483+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe78c15d160 con 0x7fe78c153290 2023-03-03T06:17:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.484+0000 7fe781ffb700 1 -- 172.21.15.102:0/3281088508 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe77c019030 con 0x7fe78c153290 2023-03-03T06:17:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.485+0000 7fe781ffb700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe770068320 0x7fe77006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:45.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.485+0000 7fe781ffb700 1 -- 172.21.15.102:0/3281088508 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe77c085c70 con 0x7fe78c153290 2023-03-03T06:17:45.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.485+0000 7fe79159b700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe770068320 0x7fe77006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:45.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.486+0000 7fe79159b700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe770068320 0x7fe77006a7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe7880090b0 tx=0x7fe788008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:45.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.489+0000 7fe781ffb700 1 -- 172.21.15.102:0/3281088508 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe77c052380 con 0x7fe78c153290 2023-03-03T06:17:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.723+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe78c0cd0c0 con 0x7fe78c153290 2023-03-03T06:17:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.724+0000 7fe781ffb700 1 -- 172.21.15.102:0/3281088508 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe77c0559a0 con 0x7fe78c153290 2023-03-03T06:17:45.726 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:17:45.726 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:17:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe770068320 msgr2=0x7fe77006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe770068320 0x7fe77006a7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe7880090b0 tx=0x7fe788008030 comp rx=0 tx=0).stop 2023-03-03T06:17:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe78c153290 msgr2=0x7fe78c0b27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe78c153290 0x7fe78c0b27f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00ec10 tx=0x7fe77c00ef20 comp rx=0 tx=0).stop 2023-03-03T06:17:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 shutdown_connections 2023-03-03T06:17:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe770068320 0x7fe77006a7d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe78c0cd8e0 0x7fe78c1ef690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe78c1593c0 0x7fe78c0cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 --2- 172.21.15.102:0/3281088508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe78c153290 0x7fe78c0b27f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 >> 172.21.15.102:0/3281088508 conn(0x7fe78c065fb0 msgr2=0x7fe78c150e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 shutdown_connections 2023-03-03T06:17:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:17:45.727+0000 7fe7937ff700 1 -- 172.21.15.102:0/3281088508 wait complete. 2023-03-03T06:17:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:17:45.756 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2023-03-03T06:18:08.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:18:08.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:18:38.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:18:38.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:05.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:19:06.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.007+0000 7fc5c49ff700 1 Processor -- start 2023-03-03T06:19:06.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.007+0000 7fc5c49ff700 1 -- start start 2023-03-03T06:19:06.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5c49ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc5c014fbe0 0x7fc5c00c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:06.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5c49ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc5c014ffb0 0x7fc5c00c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:06.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5c49ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c00c3c80 con 0x7fc5c00716c0 2023-03-03T06:19:06.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5c49ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c00c41a0 con 0x7fc5c014fbe0 2023-03-03T06:19:06.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5c49ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c0071cd0 con 0x7fc5c014ffb0 2023-03-03T06:19:06.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5be59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc5c014fbe0 0x7fc5c00c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:06.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5bed9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc5c00716c0 0x7fc5c00706f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:58014/0 (socket says 172.21.15.102:58014) 2023-03-03T06:19:06.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5bed9d700 1 -- 172.21.15.102:0/485430389 learned_addr learned my addr 172.21.15.102:0/485430389 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:19:06.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.009+0000 7fc5bdd9b700 1 --2- 172.21.15.102:0/485430389 >> v2:172.21.15.172:3300/0 conn(0x7fc5c014ffb0 0x7fc5c00c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:06.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.010+0000 7fc5be59c700 1 -- 172.21.15.102:0/485430389 >> v2:172.21.15.172:3300/0 conn(0x7fc5c014ffb0 msgr2=0x7fc5c00c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:19:06.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.010+0000 7fc5be59c700 1 --2- 172.21.15.102:0/485430389 >> v2:172.21.15.172:3300/0 conn(0x7fc5c014ffb0 0x7fc5c00c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.010+0000 7fc5be59c700 1 -- 172.21.15.102:0/485430389 >> v1:172.21.15.172:6789/0 conn(0x7fc5c00716c0 legacy=0x7fc5c00706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:06.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.010+0000 7fc5bd59a700 1 -- 172.21.15.102:0/485430389 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464003874 0 0) 0x7fc5b8001490 con 0x7fc5c00716c0 2023-03-03T06:19:06.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.010+0000 7fc5be59c700 1 -- 172.21.15.102:0/485430389 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c00c3ec0 con 0x7fc5c014fbe0 2023-03-03T06:19:06.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.010+0000 7fc5be59c700 1 --2- 172.21.15.102:0/485430389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c00c3740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00c2850 tx=0x7fc5b0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742c8d9e615acaba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:19:06.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5bd59a700 1 -- 172.21.15.102:0/485430389 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000f8a0 con 0x7fc5c014fbe0 2023-03-03T06:19:06.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5bd59a700 1 -- 172.21.15.102:0/485430389 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000fa00 con 0x7fc5c014fbe0 2023-03-03T06:19:06.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5bd59a700 1 -- 172.21.15.102:0/485430389 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000fd90 con 0x7fc5c014fbe0 2023-03-03T06:19:06.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 -- 172.21.15.102:0/485430389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 msgr2=0x7fc5c00c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:06.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/485430389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c00c3740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00c2850 tx=0x7fc5b0009b20 comp rx=0 tx=0).stop 2023-03-03T06:19:06.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 -- 172.21.15.102:0/485430389 shutdown_connections 2023-03-03T06:19:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/485430389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c00c3740 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/485430389 >> v2:172.21.15.172:3300/0 conn(0x7fc5c014ffb0 0x7fc5c00c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 -- 172.21.15.102:0/485430389 >> 172.21.15.102:0/485430389 conn(0x7fc5c006af90 msgr2=0x7fc5c006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:19:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 -- 172.21.15.102:0/485430389 shutdown_connections 2023-03-03T06:19:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.011+0000 7fc5c49ff700 1 -- 172.21.15.102:0/485430389 wait complete. 2023-03-03T06:19:06.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.013+0000 7fc5c49ff700 1 Processor -- start 2023-03-03T06:19:06.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.013+0000 7fc5c49ff700 1 -- start start 2023-03-03T06:19:06.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.014+0000 7fc5c49ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c00716c0 0x7fc5c0156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:06.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.014+0000 7fc5c49ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c0143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:06.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.014+0000 7fc5c49ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5c014ffb0 0x7fc5c0154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:06.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.014+0000 7fc5c49ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0156840 con 0x7fc5c014fbe0 2023-03-03T06:19:06.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.014+0000 7fc5c49ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c01ead40 con 0x7fc5c00716c0 2023-03-03T06:19:06.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.014+0000 7fc5c49ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c01eb270 con 0x7fc5c014ffb0 2023-03-03T06:19:06.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bdd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c0143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:06.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5be59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c00716c0 0x7fc5c0156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:06.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5be59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c00716c0 0x7fc5c0156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39150/0 (socket says 172.21.15.102:39150) 2023-03-03T06:19:06.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5be59c700 1 -- 172.21.15.102:0/369252884 learned_addr learned my addr 172.21.15.102:0/369252884 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:19:06.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bed9d700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5c014ffb0 0x7fc5c0154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:06.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bdd9b700 1 -- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5c014ffb0 msgr2=0x7fc5c0154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:06.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bdd9b700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5c014ffb0 0x7fc5c0154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bdd9b700 1 -- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c00716c0 msgr2=0x7fc5c0156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:06.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bdd9b700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c00716c0 0x7fc5c0156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bdd9b700 1 -- 172.21.15.102:0/369252884 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00094a0 con 0x7fc5c014fbe0 2023-03-03T06:19:06.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5be59c700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c00716c0 0x7fc5c0156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:19:06.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.015+0000 7fc5bdd9b700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c0143280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e8a0 tx=0x7fc5b400ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:19:06.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.016+0000 7fc5aeffd700 1 -- 172.21.15.102:0/369252884 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4013680 con 0x7fc5c014fbe0 2023-03-03T06:19:06.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.016+0000 7fc5aeffd700 1 -- 172.21.15.102:0/369252884 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b40137e0 con 0x7fc5c014fbe0 2023-03-03T06:19:06.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.016+0000 7fc5aeffd700 1 -- 172.21.15.102:0/369252884 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4013b70 con 0x7fc5c014fbe0 2023-03-03T06:19:06.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.016+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c0072250 con 0x7fc5c014fbe0 2023-03-03T06:19:06.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.016+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c0071f70 con 0x7fc5c014fbe0 2023-03-03T06:19:06.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.017+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c0072530 con 0x7fc5c014fbe0 2023-03-03T06:19:06.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.018+0000 7fc5aeffd700 1 -- 172.21.15.102:0/369252884 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5b4019030 con 0x7fc5c014fbe0 2023-03-03T06:19:06.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.019+0000 7fc5aeffd700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5a00682f0 0x7fc5a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:06.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.019+0000 7fc5aeffd700 1 -- 172.21.15.102:0/369252884 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc5b4085a40 con 0x7fc5c014fbe0 2023-03-03T06:19:06.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.020+0000 7fc5be59c700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5a00682f0 0x7fc5a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:06.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.021+0000 7fc5be59c700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5a00682f0 0x7fc5a006a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b001b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:19:06.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.024+0000 7fc5aeffd700 1 -- 172.21.15.102:0/369252884 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5b4052150 con 0x7fc5c014fbe0 2023-03-03T06:19:06.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.259+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5c004e6e0 con 0x7fc5c014fbe0 2023-03-03T06:19:06.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.259+0000 7fc5aeffd700 1 -- 172.21.15.102:0/369252884 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc5b4055770 con 0x7fc5c014fbe0 2023-03-03T06:19:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:19:06.261 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:19:06.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5a00682f0 msgr2=0x7fc5a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:06.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5a00682f0 0x7fc5a006a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b001b040 comp rx=0 tx=0).stop 2023-03-03T06:19:06.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 msgr2=0x7fc5c0143280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:06.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c0143280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e8a0 tx=0x7fc5b400ebb0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 shutdown_connections 2023-03-03T06:19:06.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5a00682f0 0x7fc5a006a7a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5c014ffb0 0x7fc5c0154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5c00716c0 0x7fc5c0156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 --2- 172.21.15.102:0/369252884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5c014fbe0 0x7fc5c0143280 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 >> 172.21.15.102:0/369252884 conn(0x7fc5c006af90 msgr2=0x7fc5c00cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:19:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 shutdown_connections 2023-03-03T06:19:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:06.262+0000 7fc5c49ff700 1 -- 172.21.15.102:0/369252884 wait complete. 2023-03-03T06:19:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:19:06.290 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-03-03T06:19:08.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:08.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:38.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:38.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:59.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:19:59.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.538+0000 7f2d39bff700 1 Processor -- start 2023-03-03T06:19:59.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.538+0000 7f2d39bff700 1 -- start start 2023-03-03T06:19:59.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.539+0000 7f2d39bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2d34153290 0x7f2d34158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d39bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d341593c0 0x7f2d340cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d39bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d34153660 con 0x7f2d340cd8e0 2023-03-03T06:19:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d39bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3406c360 con 0x7f2d34153290 2023-03-03T06:19:59.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d39bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3415cea0 con 0x7f2d341593c0 2023-03-03T06:19:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d337fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2d34153290 0x7f2d34158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d337fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2d34153290 0x7f2d34158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46318/0 (socket says 172.21.15.102:46318) 2023-03-03T06:19:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d337fe700 1 -- 172.21.15.102:0/901972770 learned_addr learned my addr 172.21.15.102:0/901972770 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:19:59.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d32ffd700 1 --2- 172.21.15.102:0/901972770 >> v2:172.21.15.172:3300/0 conn(0x7f2d341593c0 0x7f2d340cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:59.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d337fe700 1 -- 172.21.15.102:0/901972770 >> v2:172.21.15.172:3300/0 conn(0x7f2d341593c0 msgr2=0x7f2d340cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:19:59.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d337fe700 1 --2- 172.21.15.102:0/901972770 >> v2:172.21.15.172:3300/0 conn(0x7f2d341593c0 0x7f2d340cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.541+0000 7f2d337fe700 1 -- 172.21.15.102:0/901972770 >> v1:172.21.15.172:6789/0 conn(0x7f2d340cd8e0 legacy=0x7f2d340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.541+0000 7f2d337fe700 1 -- 172.21.15.102:0/901972770 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3406c7f0 con 0x7f2d34153290 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.540+0000 7f2d327fc700 1 -- 172.21.15.102:0/901972770 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024464145 0 0) 0x7f2d34153660 con 0x7f2d340cd8e0 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.541+0000 7f2d337fe700 1 --2- 172.21.15.102:0/901972770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d34158e80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2d34159fb0 tx=0x7f2d24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c094184373595c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.541+0000 7f2d327fc700 1 -- 172.21.15.102:0/901972770 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24011830 con 0x7f2d34153290 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.541+0000 7f2d327fc700 1 -- 172.21.15.102:0/901972770 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24011990 con 0x7f2d34153290 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.541+0000 7f2d327fc700 1 -- 172.21.15.102:0/901972770 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24011d20 con 0x7f2d34153290 2023-03-03T06:19:59.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 -- 172.21.15.102:0/901972770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 msgr2=0x7f2d34158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 --2- 172.21.15.102:0/901972770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d34158e80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2d34159fb0 tx=0x7f2d24009bd0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 -- 172.21.15.102:0/901972770 shutdown_connections 2023-03-03T06:19:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 --2- 172.21.15.102:0/901972770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d34158e80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 --2- 172.21.15.102:0/901972770 >> v2:172.21.15.172:3300/0 conn(0x7f2d341593c0 0x7f2d340cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 -- 172.21.15.102:0/901972770 >> 172.21.15.102:0/901972770 conn(0x7f2d34065fb0 msgr2=0x7f2d340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:19:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 -- 172.21.15.102:0/901972770 shutdown_connections 2023-03-03T06:19:59.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.542+0000 7f2d39bff700 1 -- 172.21.15.102:0/901972770 wait complete. 2023-03-03T06:19:59.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.545+0000 7f2d39bff700 1 Processor -- start 2023-03-03T06:19:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.545+0000 7f2d39bff700 1 -- start start 2023-03-03T06:19:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.546+0000 7f2d39bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340cd8e0 0x7f2d3414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.546+0000 7f2d39bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d341e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.546+0000 7f2d39bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d341593c0 0x7f2d341e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:59.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.546+0000 7f2d39bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3414ea90 con 0x7f2d34153290 2023-03-03T06:19:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.546+0000 7f2d39bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341f02b0 con 0x7f2d341593c0 2023-03-03T06:19:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.546+0000 7f2d39bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340b3e20 con 0x7f2d340cd8e0 2023-03-03T06:19:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d341e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d341e5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46330/0 (socket says 172.21.15.102:46330) 2023-03-03T06:19:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d33fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d341593c0 0x7f2d341e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 -- 172.21.15.102:0/3669985111 learned_addr learned my addr 172.21.15.102:0/3669985111 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:19:59.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d337fe700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340cd8e0 0x7f2d3414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 -- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340cd8e0 msgr2=0x7f2d3414e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340cd8e0 0x7f2d3414e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 -- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d341593c0 msgr2=0x7f2d341e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d341593c0 0x7f2d341e5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d32ffd700 1 -- 172.21.15.102:0/3669985111 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d24009550 con 0x7f2d34153290 2023-03-03T06:19:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d33fff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d341593c0 0x7f2d341e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:19:59.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.547+0000 7f2d337fe700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340cd8e0 0x7f2d3414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:19:59.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.548+0000 7f2d32ffd700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d341e5950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800e9b0 tx=0x7f2d2800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:19:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.548+0000 7f2d1bfff700 1 -- 172.21.15.102:0/3669985111 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28009710 con 0x7f2d34153290 2023-03-03T06:19:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.548+0000 7f2d1bfff700 1 -- 172.21.15.102:0/3669985111 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2800ee50 con 0x7f2d34153290 2023-03-03T06:19:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.548+0000 7f2d1bfff700 1 -- 172.21.15.102:0/3669985111 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2800ca50 con 0x7f2d34153290 2023-03-03T06:19:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.548+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d340ccde0 con 0x7f2d34153290 2023-03-03T06:19:59.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.548+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3415d160 con 0x7f2d34153290 2023-03-03T06:19:59.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.549+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d340cd0c0 con 0x7f2d34153290 2023-03-03T06:19:59.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.553+0000 7f2d1bfff700 1 -- 172.21.15.102:0/3669985111 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d28019030 con 0x7f2d34153290 2023-03-03T06:19:59.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.555+0000 7f2d1bfff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d20068320 0x7f2d2006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:19:59.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.555+0000 7f2d1bfff700 1 -- 172.21.15.102:0/3669985111 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2d28086380 con 0x7f2d34153290 2023-03-03T06:19:59.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.555+0000 7f2d337fe700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d20068320 0x7f2d2006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:19:59.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.556+0000 7f2d337fe700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d20068320 0x7f2d2006a7d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009c10 tx=0x7f2d24008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:19:59.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.558+0000 7f2d1bfff700 1 -- 172.21.15.102:0/3669985111 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d28052b10 con 0x7f2d34153290 2023-03-03T06:19:59.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.793+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d3404e6e0 con 0x7f2d34153290 2023-03-03T06:19:59.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.794+0000 7f2d1bfff700 1 -- 172.21.15.102:0/3669985111 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2d28056130 con 0x7f2d34153290 2023-03-03T06:19:59.796 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:19:59.796 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:19:59.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d20068320 msgr2=0x7f2d2006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:59.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d20068320 0x7f2d2006a7d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009c10 tx=0x7f2d24008030 comp rx=0 tx=0).stop 2023-03-03T06:19:59.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 msgr2=0x7f2d341e5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:19:59.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d341e5950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800e9b0 tx=0x7f2d2800ecc0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 shutdown_connections 2023-03-03T06:19:59.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d20068320 0x7f2d2006a7d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d340cd8e0 0x7f2d3414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d341593c0 0x7f2d341e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 --2- 172.21.15.102:0/3669985111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d34153290 0x7f2d341e5950 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:19:59.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 >> 172.21.15.102:0/3669985111 conn(0x7f2d34065fb0 msgr2=0x7f2d3415ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:19:59.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 shutdown_connections 2023-03-03T06:19:59.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:19:59.797+0000 7f2d39bff700 1 -- 172.21.15.102:0/3669985111 wait complete. 2023-03-03T06:19:59.800 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:19:59.825 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T06:20:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T06:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:20:09.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:09.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:39.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:39.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:55.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:20:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.078+0000 7f4951f9d700 1 Processor -- start 2023-03-03T06:20:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.078+0000 7f4951f9d700 1 -- start start 2023-03-03T06:20:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f4951f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f494c0b3970 0x7f494c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:20:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f4951f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f494c0c41b0 0x7f494c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:20:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f4951f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c0c3d20 con 0x7f494c0ced00 2023-03-03T06:20:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f4951f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c0c4930 con 0x7f494c0c41b0 2023-03-03T06:20:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f4951f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c0cf610 con 0x7f494c0b3970 2023-03-03T06:20:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f494bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f494c0b3970 0x7f494c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:20:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f495179c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f494c0c41b0 0x7f494c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:20:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f4950f9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f494c0ced00 0x7f494c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36904/0 (socket says 172.21.15.102:36904) 2023-03-03T06:20:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.080+0000 7f4950f9b700 1 -- 172.21.15.102:0/254353852 learned_addr learned my addr 172.21.15.102:0/254353852 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:20:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f495179c700 1 -- 172.21.15.102:0/254353852 >> v2:172.21.15.172:3300/0 conn(0x7f494c0b3970 msgr2=0x7f494c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:20:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f495179c700 1 --2- 172.21.15.102:0/254353852 >> v2:172.21.15.172:3300/0 conn(0x7f494c0b3970 0x7f494c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f495179c700 1 -- 172.21.15.102:0/254353852 >> v1:172.21.15.102:6789/0 conn(0x7f494c0ced00 legacy=0x7f494c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:20:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f495179c700 1 -- 172.21.15.102:0/254353852 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c15cba0 con 0x7f494c0c41b0 2023-03-03T06:20:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f495179c700 1 --2- 172.21.15.102:0/254353852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0c41b0 0x7f494c0cf0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f494c0c4ba0 tx=0x7f494400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e62199cdb9e8cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:20:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f494b7fe700 1 -- 172.21.15.102:0/254353852 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494400e070 con 0x7f494c0c41b0 2023-03-03T06:20:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f494b7fe700 1 -- 172.21.15.102:0/254353852 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494400bae0 con 0x7f494c0c41b0 2023-03-03T06:20:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.081+0000 7f494b7fe700 1 -- 172.21.15.102:0/254353852 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494400be90 con 0x7f494c0c41b0 2023-03-03T06:20:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 -- 172.21.15.102:0/254353852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0c41b0 msgr2=0x7f494c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:20:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 --2- 172.21.15.102:0/254353852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0c41b0 0x7f494c0cf0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f494c0c4ba0 tx=0x7f494400b330 comp rx=0 tx=0).stop 2023-03-03T06:20:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 -- 172.21.15.102:0/254353852 shutdown_connections 2023-03-03T06:20:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 --2- 172.21.15.102:0/254353852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0c41b0 0x7f494c0cf0d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 --2- 172.21.15.102:0/254353852 >> v2:172.21.15.172:3300/0 conn(0x7f494c0b3970 0x7f494c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 -- 172.21.15.102:0/254353852 >> 172.21.15.102:0/254353852 conn(0x7f494c08b1c0 msgr2=0x7f494c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:20:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 -- 172.21.15.102:0/254353852 shutdown_connections 2023-03-03T06:20:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.082+0000 7f4951f9d700 1 -- 172.21.15.102:0/254353852 wait complete. 2023-03-03T06:20:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.084+0000 7f4951f9d700 1 Processor -- start 2023-03-03T06:20:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.084+0000 7f4951f9d700 1 -- start start 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.085+0000 7f4951f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f494c0b3970 0x7f494c1f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.085+0000 7f4951f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494c0c41b0 0x7f494c15b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.085+0000 7f4951f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0ced00 0x7f494c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.085+0000 7f4951f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c1f1fd0 con 0x7f494c0ced00 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.085+0000 7f4951f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c150e60 con 0x7f494c0b3970 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.085+0000 7f4951f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c151120 con 0x7f494c0c41b0 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f4950f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f494c0b3970 0x7f494c1f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:20:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f495179c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0ced00 0x7f494c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:20:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f494bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494c0c41b0 0x7f494c15b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:20:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f4950f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f494c0b3970 0x7f494c1f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55054/0 (socket says 172.21.15.102:55054) 2023-03-03T06:20:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f4950f9b700 1 -- 172.21.15.102:0/1151915548 learned_addr learned my addr 172.21.15.102:0/1151915548 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:20:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f494bfff700 1 -- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f494c0b3970 msgr2=0x7f494c1f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:20:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f494bfff700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f494c0b3970 0x7f494c1f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f494bfff700 1 -- 172.21.15.102:0/1151915548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0ced00 msgr2=0x7f494c0b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:20:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f494bfff700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0ced00 0x7f494c0b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f494bfff700 1 -- 172.21.15.102:0/1151915548 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4944009a70 con 0x7f494c0c41b0 2023-03-03T06:20:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f495179c700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0ced00 0x7f494c0b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:20:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.086+0000 7f4950f9b700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f494c0b3970 0x7f494c1f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:20:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.087+0000 7f494bfff700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494c0c41b0 0x7f494c15b1c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f493800e870 tx=0x7f493800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:20:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.087+0000 7f49497fa700 1 -- 172.21.15.102:0/1151915548 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938013680 con 0x7f494c0c41b0 2023-03-03T06:20:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.087+0000 7f49497fa700 1 -- 172.21.15.102:0/1151915548 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49380137e0 con 0x7f494c0c41b0 2023-03-03T06:20:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.087+0000 7f49497fa700 1 -- 172.21.15.102:0/1151915548 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938013b90 con 0x7f494c0c41b0 2023-03-03T06:20:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.087+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c06bf20 con 0x7f494c0c41b0 2023-03-03T06:20:56.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.087+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c15ce80 con 0x7f494c0c41b0 2023-03-03T06:20:56.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.088+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c06c200 con 0x7f494c0c41b0 2023-03-03T06:20:56.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.089+0000 7f49497fa700 1 -- 172.21.15.102:0/1151915548 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4938019030 con 0x7f494c0c41b0 2023-03-03T06:20:56.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.091+0000 7f49497fa700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4930068320 0x7f493006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:20:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.091+0000 7f49497fa700 1 -- 172.21.15.102:0/1151915548 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4938085ad0 con 0x7f494c0c41b0 2023-03-03T06:20:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.091+0000 7f4950f9b700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4930068320 0x7f493006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:20:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.092+0000 7f4950f9b700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4930068320 0x7f493006a7d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4940005910 tx=0x7f4940009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:20:56.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.095+0000 7f49497fa700 1 -- 172.21.15.102:0/1151915548 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4938052150 con 0x7f494c0c41b0 2023-03-03T06:20:56.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.327+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f494c0c9ef0 con 0x7f494c0c41b0 2023-03-03T06:20:56.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.328+0000 7f49497fa700 1 -- 172.21.15.102:0/1151915548 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4938055770 con 0x7f494c0c41b0 2023-03-03T06:20:56.330 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:20:56.330 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:20:56.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4930068320 msgr2=0x7f493006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:20:56.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4930068320 0x7f493006a7d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4940005910 tx=0x7f4940009040 comp rx=0 tx=0).stop 2023-03-03T06:20:56.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494c0c41b0 msgr2=0x7f494c15b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:20:56.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494c0c41b0 0x7f494c15b1c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f493800e870 tx=0x7f493800eb80 comp rx=0 tx=0).stop 2023-03-03T06:20:56.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 shutdown_connections 2023-03-03T06:20:56.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f494c0ced00 0x7f494c0b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f494c0b3970 0x7f494c1f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4930068320 0x7f493006a7d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 --2- 172.21.15.102:0/1151915548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f494c0c41b0 0x7f494c15b1c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:20:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 >> 172.21.15.102:0/1151915548 conn(0x7f494c08b1c0 msgr2=0x7f494c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:20:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 shutdown_connections 2023-03-03T06:20:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:20:56.331+0000 7f4951f9d700 1 -- 172.21.15.102:0/1151915548 wait complete. 2023-03-03T06:20:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:20:56.360 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-03-03T06:21:04.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:21:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.606+0000 7f872e1ff700 1 Processor -- start 2023-03-03T06:21:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.607+0000 7f872e1ff700 1 -- start start 2023-03-03T06:21:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.608+0000 7f872e1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f87280cd8e0 0x7f87280cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.608+0000 7f872e1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8728153660 con 0x7f8728153290 2023-03-03T06:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.608+0000 7f872e1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872806c360 con 0x7f87280cd390 2023-03-03T06:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.608+0000 7f872e1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872815cea0 con 0x7f87280cd8e0 2023-03-03T06:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.608+0000 7f87277fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8728153290 0x7f8728158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52606/0 (socket says 172.21.15.102:52606) 2023-03-03T06:21:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.608+0000 7f87277fe700 1 -- 172.21.15.102:0/13784636 learned_addr learned my addr 172.21.15.102:0/13784636 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:21:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.608+0000 7f8726ffd700 1 --2- 172.21.15.102:0/13784636 >> v2:172.21.15.172:3300/0 conn(0x7f87280cd8e0 0x7f87280cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:21:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.609+0000 7f8726ffd700 1 -- 172.21.15.102:0/13784636 >> v1:172.21.15.172:6789/0 conn(0x7f87280cd390 legacy=0x7f87280ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:21:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.609+0000 7f8726ffd700 1 -- 172.21.15.102:0/13784636 >> v1:172.21.15.102:6789/0 conn(0x7f8728153290 legacy=0x7f8728158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:21:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.609+0000 7f87267fc700 1 -- 172.21.15.102:0/13784636 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304309869 0 0) 0x7f872806c360 con 0x7f87280cd390 2023-03-03T06:21:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.609+0000 7f8726ffd700 1 -- 172.21.15.102:0/13784636 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872806c7f0 con 0x7f87280cd8e0 2023-03-03T06:21:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.609+0000 7f8726ffd700 1 --2- 172.21.15.102:0/13784636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87280cde20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f871c009840 tx=0x7f871c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c870bc9b211d33e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f87267fc700 1 -- 172.21.15.102:0/13784636 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c011830 con 0x7f87280cd8e0 2023-03-03T06:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f87267fc700 1 -- 172.21.15.102:0/13784636 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c011990 con 0x7f87280cd8e0 2023-03-03T06:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f87267fc700 1 -- 172.21.15.102:0/13784636 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c011d00 con 0x7f87280cd8e0 2023-03-03T06:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f872e1ff700 1 -- 172.21.15.102:0/13784636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 msgr2=0x7f87280cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f872e1ff700 1 --2- 172.21.15.102:0/13784636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87280cde20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f871c009840 tx=0x7f871c009b50 comp rx=0 tx=0).stop 2023-03-03T06:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f872e1ff700 1 -- 172.21.15.102:0/13784636 shutdown_connections 2023-03-03T06:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f872e1ff700 1 --2- 172.21.15.102:0/13784636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87280cde20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f872e1ff700 1 -- 172.21.15.102:0/13784636 >> 172.21.15.102:0/13784636 conn(0x7f8728065fb0 msgr2=0x7f87280663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f872e1ff700 1 -- 172.21.15.102:0/13784636 shutdown_connections 2023-03-03T06:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.610+0000 7f872e1ff700 1 -- 172.21.15.102:0/13784636 wait complete. 2023-03-03T06:21:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.612+0000 7f872e1ff700 1 Processor -- start 2023-03-03T06:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.612+0000 7f872e1ff700 1 -- start start 2023-03-03T06:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.613+0000 7f872e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87280cd390 0x7f87281efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.613+0000 7f872e1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87281f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.613+0000 7f872e1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8728153290 0x7f87281f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.613+0000 7f872e1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280b36d0 con 0x7f8728153290 2023-03-03T06:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.613+0000 7f872e1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280b2820 con 0x7f87280cd8e0 2023-03-03T06:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.613+0000 7f872e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872815cb10 con 0x7f87280cd390 2023-03-03T06:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87281f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f87277fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87280cd390 0x7f87281efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87281f0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:51448/0 (socket says 172.21.15.102:51448) 2023-03-03T06:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 -- 172.21.15.102:0/1605559170 learned_addr learned my addr 172.21.15.102:0/1605559170 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8727fff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8728153290 0x7f87281f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 -- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87280cd390 msgr2=0x7f87281efac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87280cd390 0x7f87281efac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 -- 172.21.15.102:0/1605559170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8728153290 msgr2=0x7f87281f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8728153290 0x7f87281f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.614+0000 7f8726ffd700 1 -- 172.21.15.102:0/1605559170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c0094a0 con 0x7f87280cd8e0 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.615+0000 7f8726ffd700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87281f0000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8728074540 tx=0x7f871c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.615+0000 7f870ffff700 1 -- 172.21.15.102:0/1605559170 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c01a790 con 0x7f87280cd8e0 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.615+0000 7f870ffff700 1 -- 172.21.15.102:0/1605559170 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c01a910 con 0x7f87280cd8e0 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.615+0000 7f870ffff700 1 -- 172.21.15.102:0/1605559170 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c01acc0 con 0x7f87280cd8e0 2023-03-03T06:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.615+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87280ccde0 con 0x7f87280cd8e0 2023-03-03T06:21:04.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.615+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872815d160 con 0x7f87280cd8e0 2023-03-03T06:21:04.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.616+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87280cd0c0 con 0x7f87280cd8e0 2023-03-03T06:21:04.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.618+0000 7f870ffff700 1 -- 172.21.15.102:0/1605559170 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f871c00f050 con 0x7f87280cd8e0 2023-03-03T06:21:04.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.619+0000 7f870ffff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8714068320 0x7f871406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:21:04.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.619+0000 7f870ffff700 1 -- 172.21.15.102:0/1605559170 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f871c086b30 con 0x7f87280cd8e0 2023-03-03T06:21:04.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.619+0000 7f87277fe700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8714068320 0x7f871406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:21:04.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.620+0000 7f87277fe700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8714068320 0x7f871406a7d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8718005910 tx=0x7f8718009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:21:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.624+0000 7f870ffff700 1 -- 172.21.15.102:0/1605559170 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f871c053240 con 0x7f87280cd8e0 2023-03-03T06:21:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.860+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f872804e6e0 con 0x7f87280cd8e0 2023-03-03T06:21:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.861+0000 7f870ffff700 1 -- 172.21.15.102:0/1605559170 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f871c015090 con 0x7f87280cd8e0 2023-03-03T06:21:04.863 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:21:04.864 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:21:04.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8714068320 msgr2=0x7f871406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:21:04.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8714068320 0x7f871406a7d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8718005910 tx=0x7f8718009040 comp rx=0 tx=0).stop 2023-03-03T06:21:04.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 msgr2=0x7f87281f0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:21:04.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87281f0000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8728074540 tx=0x7f871c004750 comp rx=0 tx=0).stop 2023-03-03T06:21:04.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 shutdown_connections 2023-03-03T06:21:04.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8714068320 0x7f871406a7d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:21:04.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87280cd390 0x7f87281efac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:21:04.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8728153290 0x7f87281f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:21:04.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 --2- 172.21.15.102:0/1605559170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87280cd8e0 0x7f87281f0000 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:21:04.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.863+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 >> 172.21.15.102:0/1605559170 conn(0x7f8728065fb0 msgr2=0x7f87280cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:21:04.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.864+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 shutdown_connections 2023-03-03T06:21:04.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:21:04.864+0000 7f872e1ff700 1 -- 172.21.15.102:0/1605559170 wait complete. 2023-03-03T06:21:04.868 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:21:04.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-03-03T06:21:09.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:21:09.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:21:39.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:21:39.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:09.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:09.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:40.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:40.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:48.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:22:49.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.142+0000 7faf103ff700 1 Processor -- start 2023-03-03T06:22:49.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.143+0000 7faf103ff700 1 -- start start 2023-03-03T06:22:49.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf103ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf08153290 0x7faf08158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:22:49.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf103ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf081593c0 0x7faf080cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:22:49.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf103ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faf080cd8e0 0x7faf080ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:22:49.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf103ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf08153660 con 0x7faf08153290 2023-03-03T06:22:49.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf103ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0806c360 con 0x7faf080cd8e0 2023-03-03T06:22:49.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf103ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0815cea0 con 0x7faf081593c0 2023-03-03T06:22:49.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf0e19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf08153290 0x7faf08158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:22:49.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf0e99c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faf080cd8e0 0x7faf080ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:22:49.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf0d99a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf081593c0 0x7faf080cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:22:49.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf0e19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf08153290 0x7faf08158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54350/0 (socket says 172.21.15.102:54350) 2023-03-03T06:22:49.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.144+0000 7faf0e19b700 1 -- 172.21.15.102:0/2421912666 learned_addr learned my addr 172.21.15.102:0/2421912666 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:22:49.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0e99c700 1 -- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf081593c0 msgr2=0x7faf080cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:22:49.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0e99c700 1 --2- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf081593c0 0x7faf080cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0e99c700 1 -- 172.21.15.102:0/2421912666 >> v2:172.21.15.102:3300/0 conn(0x7faf08153290 msgr2=0x7faf08158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:22:49.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0e99c700 1 --2- 172.21.15.102:0/2421912666 >> v2:172.21.15.102:3300/0 conn(0x7faf08153290 0x7faf08158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0e99c700 1 -- 172.21.15.102:0/2421912666 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0806c7f0 con 0x7faf080cd8e0 2023-03-03T06:22:49.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0e19b700 1 --2- 172.21.15.102:0/2421912666 >> v2:172.21.15.102:3300/0 conn(0x7faf08153290 0x7faf08158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:22:49.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0d99a700 1 --2- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf081593c0 0x7faf080cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:22:49.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0e99c700 1 --2- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf080cd8e0 0x7faf080ce360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf0806d0e0 tx=0x7faf0000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc75b7c17712950f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:22:49.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.145+0000 7faf0d199700 1 -- 172.21.15.102:0/2421912666 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0000e070 con 0x7faf080cd8e0 2023-03-03T06:22:49.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf0d199700 1 -- 172.21.15.102:0/2421912666 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0000bb20 con 0x7faf080cd8e0 2023-03-03T06:22:49.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf0d199700 1 -- 172.21.15.102:0/2421912666 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00012430 con 0x7faf080cd8e0 2023-03-03T06:22:49.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 -- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf080cd8e0 msgr2=0x7faf080ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:22:49.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 --2- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf080cd8e0 0x7faf080ce360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf0806d0e0 tx=0x7faf0000b330 comp rx=0 tx=0).stop 2023-03-03T06:22:49.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 -- 172.21.15.102:0/2421912666 shutdown_connections 2023-03-03T06:22:49.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 --2- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf080cd8e0 0x7faf080ce360 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 --2- 172.21.15.102:0/2421912666 >> v2:172.21.15.102:3300/0 conn(0x7faf08153290 0x7faf08158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 --2- 172.21.15.102:0/2421912666 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf081593c0 0x7faf080cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 -- 172.21.15.102:0/2421912666 >> 172.21.15.102:0/2421912666 conn(0x7faf08065fb0 msgr2=0x7faf080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:22:49.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 -- 172.21.15.102:0/2421912666 shutdown_connections 2023-03-03T06:22:49.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.146+0000 7faf103ff700 1 -- 172.21.15.102:0/2421912666 wait complete. 2023-03-03T06:22:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.148+0000 7faf103ff700 1 Processor -- start 2023-03-03T06:22:49.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.148+0000 7faf103ff700 1 -- start start 2023-03-03T06:22:49.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.149+0000 7faf103ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf080cd8e0 0x7faf080b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:22:49.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.149+0000 7faf103ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf08153290 0x7faf081577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:22:49.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf103ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf081593c0 0x7faf08157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:22:49.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf103ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf080c6140 con 0x7faf08153290 2023-03-03T06:22:49.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf103ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf081eef50 con 0x7faf081593c0 2023-03-03T06:22:49.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf103ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf080c6370 con 0x7faf080cd8e0 2023-03-03T06:22:49.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0d99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf08153290 0x7faf081577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:22:49.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0e19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf080cd8e0 0x7faf080b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:22:49.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0d99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf08153290 0x7faf081577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54364/0 (socket says 172.21.15.102:54364) 2023-03-03T06:22:49.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0d99a700 1 -- 172.21.15.102:0/3130863843 learned_addr learned my addr 172.21.15.102:0/3130863843 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:22:49.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0e99c700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf081593c0 0x7faf08157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:22:49.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0e19b700 1 -- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf081593c0 msgr2=0x7faf08157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:22:49.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0e19b700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf081593c0 0x7faf08157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0e19b700 1 -- 172.21.15.102:0/3130863843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf08153290 msgr2=0x7faf081577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:22:49.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0e19b700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf08153290 0x7faf081577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0e19b700 1 -- 172.21.15.102:0/3130863843 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf04009550 con 0x7faf080cd8e0 2023-03-03T06:22:49.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.150+0000 7faf0d99a700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf08153290 0x7faf081577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:22:49.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.151+0000 7faf0e19b700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf080cd8e0 0x7faf080b44c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf0400e980 tx=0x7faf0400ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:22:49.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.151+0000 7faefeffd700 1 -- 172.21.15.102:0/3130863843 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04004030 con 0x7faf080cd8e0 2023-03-03T06:22:49.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.151+0000 7faefeffd700 1 -- 172.21.15.102:0/3130863843 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf04004190 con 0x7faf080cd8e0 2023-03-03T06:22:49.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.151+0000 7faefeffd700 1 -- 172.21.15.102:0/3130863843 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04004520 con 0x7faf080cd8e0 2023-03-03T06:22:49.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.151+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef80094a0 con 0x7faf080cd8e0 2023-03-03T06:22:49.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.151+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf00009a70 con 0x7faf080cd8e0 2023-03-03T06:22:49.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.152+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf080ccde0 con 0x7faf080cd8e0 2023-03-03T06:22:49.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.154+0000 7faefeffd700 1 -- 172.21.15.102:0/3130863843 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faf04022070 con 0x7faf080cd8e0 2023-03-03T06:22:49.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.155+0000 7faefeffd700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faeec0682f0 0x7faeec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:22:49.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.155+0000 7faefeffd700 1 -- 172.21.15.102:0/3130863843 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7faf04085d60 con 0x7faf080cd8e0 2023-03-03T06:22:49.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.155+0000 7faf0d99a700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faeec0682f0 0x7faeec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:22:49.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.156+0000 7faf0d99a700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faeec0682f0 0x7faeec06a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faef8009fd0 tx=0x7faef8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:22:49.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.159+0000 7faefeffd700 1 -- 172.21.15.102:0/3130863843 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faf04052470 con 0x7faf080cd8e0 2023-03-03T06:22:49.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.392+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf080cd0c0 con 0x7faf080cd8e0 2023-03-03T06:22:49.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.394+0000 7faefeffd700 1 -- 172.21.15.102:0/3130863843 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7faf04020090 con 0x7faf080cd8e0 2023-03-03T06:22:49.396 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:22:49.396 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:22:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.396+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faeec0682f0 msgr2=0x7faeec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:22:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.396+0000 7faf103ff700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faeec0682f0 0x7faeec06a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faef8009fd0 tx=0x7faef8009210 comp rx=0 tx=0).stop 2023-03-03T06:22:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.396+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf080cd8e0 msgr2=0x7faf080b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:22:49.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.396+0000 7faf103ff700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf080cd8e0 0x7faf080b44c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf0400e980 tx=0x7faf0400ec90 comp rx=0 tx=0).stop 2023-03-03T06:22:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.396+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 shutdown_connections 2023-03-03T06:22:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.397+0000 7faf103ff700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faeec0682f0 0x7faeec06a7a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.397+0000 7faf103ff700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf080cd8e0 0x7faf080b44c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.397+0000 7faf103ff700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf081593c0 0x7faf08157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.397+0000 7faf103ff700 1 --2- 172.21.15.102:0/3130863843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf08153290 0x7faf081577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:22:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.397+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 >> 172.21.15.102:0/3130863843 conn(0x7faf08065fb0 msgr2=0x7faf0815ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:22:49.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.397+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 shutdown_connections 2023-03-03T06:22:49.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:22:49.397+0000 7faf103ff700 1 -- 172.21.15.102:0/3130863843 wait complete. 2023-03-03T06:22:49.401 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:22:49.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 25 secs before thrashing 2023-03-03T06:23:10.259 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:23:10.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:23:14.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:23:14.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.676+0000 7f065e59d700 1 Processor -- start 2023-03-03T06:23:14.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.676+0000 7f065e59d700 1 -- start start 2023-03-03T06:23:14.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065e59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06581564b0 0x7f06580cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:23:14.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065e59d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f06580cab50 0x7f06580cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:23:14.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065e59d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06580c9ca0 con 0x7f06580c9610 2023-03-03T06:23:14.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065e59d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06580b4920 con 0x7f06580cab50 2023-03-03T06:23:14.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065e59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06580b4c00 con 0x7f06581564b0 2023-03-03T06:23:14.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065d59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06581564b0 0x7f06580cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:23:14.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065dd9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f06580cab50 0x7f06580cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:23:14.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065cd9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f06580c9610 0x7f06580ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37986/0 (socket says 172.21.15.102:37986) 2023-03-03T06:23:14.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.678+0000 7f065cd9a700 1 -- 172.21.15.102:0/2860948180 learned_addr learned my addr 172.21.15.102:0/2860948180 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:23:14.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f064ffff700 1 -- 172.21.15.102:0/2860948180 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047558924 0 0) 0x7f06580c9ca0 con 0x7f06580c9610 2023-03-03T06:23:14.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f065d59b700 1 -- 172.21.15.102:0/2860948180 >> v2:172.21.15.102:3300/0 conn(0x7f06580cab50 msgr2=0x7f06580cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:23:14.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f065d59b700 1 --2- 172.21.15.102:0/2860948180 >> v2:172.21.15.102:3300/0 conn(0x7f06580cab50 0x7f06580cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f065d59b700 1 -- 172.21.15.102:0/2860948180 >> v1:172.21.15.172:6789/0 conn(0x7f06580c9610 legacy=0x7f06580ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:23:14.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f065d59b700 1 -- 172.21.15.102:0/2860948180 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06580b46f0 con 0x7f06581564b0 2023-03-03T06:23:14.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f065d59b700 1 --2- 172.21.15.102:0/2860948180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06581564b0 0x7f06580cb660 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0654009840 tx=0x7f0654009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcd0384fa10405fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:23:14.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f064ffff700 1 -- 172.21.15.102:0/2860948180 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654011830 con 0x7f06581564b0 2023-03-03T06:23:14.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f064ffff700 1 -- 172.21.15.102:0/2860948180 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654011990 con 0x7f06581564b0 2023-03-03T06:23:14.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.679+0000 7f064ffff700 1 -- 172.21.15.102:0/2860948180 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654011d20 con 0x7f06581564b0 2023-03-03T06:23:14.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 -- 172.21.15.102:0/2860948180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06581564b0 msgr2=0x7f06580cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:23:14.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 --2- 172.21.15.102:0/2860948180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06581564b0 0x7f06580cb660 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0654009840 tx=0x7f0654009b50 comp rx=0 tx=0).stop 2023-03-03T06:23:14.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 -- 172.21.15.102:0/2860948180 shutdown_connections 2023-03-03T06:23:14.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 --2- 172.21.15.102:0/2860948180 >> v2:172.21.15.102:3300/0 conn(0x7f06580cab50 0x7f06580cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 --2- 172.21.15.102:0/2860948180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06581564b0 0x7f06580cb660 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 -- 172.21.15.102:0/2860948180 >> 172.21.15.102:0/2860948180 conn(0x7f0658065fb0 msgr2=0x7f06580663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:23:14.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 -- 172.21.15.102:0/2860948180 shutdown_connections 2023-03-03T06:23:14.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.680+0000 7f065e59d700 1 -- 172.21.15.102:0/2860948180 wait complete. 2023-03-03T06:23:14.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.682+0000 7f065e59d700 1 Processor -- start 2023-03-03T06:23:14.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.682+0000 7f065e59d700 1 -- start start 2023-03-03T06:23:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.683+0000 7f065e59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06580c9610 0x7f06581ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:23:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.683+0000 7f065e59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06580cab50 0x7f06581e1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:23:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.683+0000 7f065e59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06581564b0 0x7f065814ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:23:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.683+0000 7f065e59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065814f3c0 con 0x7f06581564b0 2023-03-03T06:23:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.683+0000 7f065e59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0658152870 con 0x7f06580cab50 2023-03-03T06:23:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.683+0000 7f065e59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06581e3a60 con 0x7f06580c9610 2023-03-03T06:23:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06581564b0 0x7f065814ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:23:14.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065cd9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06580cab50 0x7f06581e1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:23:14.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065d59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06580c9610 0x7f06581ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:23:14.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06581564b0 0x7f065814ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36224/0 (socket says 172.21.15.102:36224) 2023-03-03T06:23:14.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 -- 172.21.15.102:0/88841449 learned_addr learned my addr 172.21.15.102:0/88841449 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:23:14.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 -- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06580c9610 msgr2=0x7f06581ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:23:14.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06580c9610 0x7f06581ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 -- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06580cab50 msgr2=0x7f06581e1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:23:14.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06580cab50 0x7f06581e1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 -- 172.21.15.102:0/88841449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540094a0 con 0x7f06581564b0 2023-03-03T06:23:14.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.684+0000 7f065dd9c700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06581564b0 0x7f065814ddb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f065000efc0 tx=0x7f065000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:23:14.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.685+0000 7f064dffb700 1 -- 172.21.15.102:0/88841449 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650009ea0 con 0x7f06581564b0 2023-03-03T06:23:14.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.685+0000 7f064dffb700 1 -- 172.21.15.102:0/88841449 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065000f040 con 0x7f06581564b0 2023-03-03T06:23:14.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.685+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06580b2210 con 0x7f06581564b0 2023-03-03T06:23:14.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.685+0000 7f064dffb700 1 -- 172.21.15.102:0/88841449 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065000ced0 con 0x7f06581564b0 2023-03-03T06:23:14.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.685+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06580b1f30 con 0x7f06581564b0 2023-03-03T06:23:14.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.686+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0658156880 con 0x7f06581564b0 2023-03-03T06:23:14.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.687+0000 7f064dffb700 1 -- 172.21.15.102:0/88841449 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0650014480 con 0x7f06581564b0 2023-03-03T06:23:14.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.689+0000 7f064dffb700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f063c0682f0 0x7f063c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:23:14.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.689+0000 7f064dffb700 1 -- 172.21.15.102:0/88841449 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0650086970 con 0x7f06581564b0 2023-03-03T06:23:14.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.689+0000 7f065d59b700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f063c0682f0 0x7f063c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:23:14.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.690+0000 7f065d59b700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f063c0682f0 0x7f063c06a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f06540090b0 tx=0x7f0654008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:23:14.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.693+0000 7f064dffb700 1 -- 172.21.15.102:0/88841449 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f06500530b0 con 0x7f06581564b0 2023-03-03T06:23:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.930+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06580677f0 con 0x7f06581564b0 2023-03-03T06:23:14.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.931+0000 7f064dffb700 1 -- 172.21.15.102:0/88841449 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f06500566d0 con 0x7f06581564b0 2023-03-03T06:23:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:23:14.932 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:23:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f063c0682f0 msgr2=0x7f063c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:23:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f063c0682f0 0x7f063c06a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f06540090b0 tx=0x7f0654008030 comp rx=0 tx=0).stop 2023-03-03T06:23:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06581564b0 msgr2=0x7f065814ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:23:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06581564b0 0x7f065814ddb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f065000efc0 tx=0x7f065000c3a0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 shutdown_connections 2023-03-03T06:23:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f063c0682f0 0x7f063c06a7a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06580c9610 0x7f06581ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06581564b0 0x7f065814ddb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 --2- 172.21.15.102:0/88841449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06580cab50 0x7f06581e1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:23:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 >> 172.21.15.102:0/88841449 conn(0x7f0658065fb0 msgr2=0x7f06580c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:23:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 shutdown_connections 2023-03-03T06:23:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:23:14.933+0000 7f065e59d700 1 -- 172.21.15.102:0/88841449 wait complete. 2023-03-03T06:23:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:23:14.961 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 93 secs before thrashing 2023-03-03T06:23:40.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:23:40.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:10.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:10.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:40.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:40.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:47.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:24:48.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.228+0000 7f9bedbff700 1 Processor -- start 2023-03-03T06:24:48.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.228+0000 7f9bedbff700 1 -- start start 2023-03-03T06:24:48.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.230+0000 7f9bedbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9be8153290 0x7f9be8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:24:48.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.230+0000 7f9bedbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be8153660 con 0x7f9be80cd8e0 2023-03-03T06:24:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.230+0000 7f9bedbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be806c360 con 0x7f9be81593c0 2023-03-03T06:24:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.230+0000 7f9bedbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be80cec40 con 0x7f9be8153290 2023-03-03T06:24:48.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.230+0000 7f9be7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9be80cd8e0 0x7f9be80ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33728/0 (socket says 172.21.15.102:33728) 2023-03-03T06:24:48.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.230+0000 7f9be7fff700 1 -- 172.21.15.102:0/244726497 learned_addr learned my addr 172.21.15.102:0/244726497 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:24:48.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.230+0000 7f9be77fe700 1 --2- 172.21.15.102:0/244726497 >> v2:172.21.15.172:3300/0 conn(0x7f9be8153290 0x7f9be8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:24:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be67fc700 1 -- 172.21.15.102:0/244726497 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244760633 0 0) 0x7f9be806c360 con 0x7f9be81593c0 2023-03-03T06:24:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be67fc700 1 -- 172.21.15.102:0/244726497 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd00035b0 con 0x7f9be81593c0 2023-03-03T06:24:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be77fe700 1 -- 172.21.15.102:0/244726497 >> v1:172.21.15.172:6789/0 conn(0x7f9be81593c0 legacy=0x7f9be80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:24:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be77fe700 1 -- 172.21.15.102:0/244726497 >> v1:172.21.15.102:6789/0 conn(0x7f9be80cd8e0 legacy=0x7f9be80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:24:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be67fc700 1 -- 172.21.15.102:0/244726497 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993523132 0 0) 0x7f9be8153660 con 0x7f9be80cd8e0 2023-03-03T06:24:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be77fe700 1 -- 172.21.15.102:0/244726497 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be815cea0 con 0x7f9be8153290 2023-03-03T06:24:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be77fe700 1 --2- 172.21.15.102:0/244726497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be8153290 0x7f9be8158e80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009840 tx=0x7f9bd8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa9de3373d1fae3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:24:48.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be67fc700 1 -- 172.21.15.102:0/244726497 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8011830 con 0x7f9be8153290 2023-03-03T06:24:48.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be67fc700 1 -- 172.21.15.102:0/244726497 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd8011990 con 0x7f9be8153290 2023-03-03T06:24:48.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.231+0000 7f9be67fc700 1 -- 172.21.15.102:0/244726497 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8011d20 con 0x7f9be8153290 2023-03-03T06:24:48.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.232+0000 7f9bedbff700 1 -- 172.21.15.102:0/244726497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be8153290 msgr2=0x7f9be8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:24:48.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.232+0000 7f9bedbff700 1 --2- 172.21.15.102:0/244726497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be8153290 0x7f9be8158e80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009840 tx=0x7f9bd8009b50 comp rx=0 tx=0).stop 2023-03-03T06:24:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.232+0000 7f9bedbff700 1 -- 172.21.15.102:0/244726497 shutdown_connections 2023-03-03T06:24:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.232+0000 7f9bedbff700 1 --2- 172.21.15.102:0/244726497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be8153290 0x7f9be8158e80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:24:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.232+0000 7f9bedbff700 1 -- 172.21.15.102:0/244726497 >> 172.21.15.102:0/244726497 conn(0x7f9be8065fb0 msgr2=0x7f9be80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:24:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.232+0000 7f9bedbff700 1 -- 172.21.15.102:0/244726497 shutdown_connections 2023-03-03T06:24:48.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.232+0000 7f9bedbff700 1 -- 172.21.15.102:0/244726497 wait complete. 2023-03-03T06:24:48.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.234+0000 7f9bedbff700 1 Processor -- start 2023-03-03T06:24:48.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.234+0000 7f9bedbff700 1 -- start start 2023-03-03T06:24:48.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.235+0000 7f9bedbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9be80cd8e0 0x7f9be80b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:24:48.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.235+0000 7f9bedbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9be8153290 0x7f9be8151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:24:48.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.235+0000 7f9bedbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be81593c0 0x7f9be814f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:24:48.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.235+0000 7f9bedbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be80b35e0 con 0x7f9be8153290 2023-03-03T06:24:48.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.235+0000 7f9bedbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be80b3f40 con 0x7f9be81593c0 2023-03-03T06:24:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.235+0000 7f9bedbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be8150790 con 0x7f9be80cd8e0 2023-03-03T06:24:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9be8153290 0x7f9be8151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:24:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9be8153290 0x7f9be8151090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43068/0 (socket says 172.21.15.102:43068) 2023-03-03T06:24:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be7fff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be81593c0 0x7f9be814f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:24:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be6ffd700 1 -- 172.21.15.102:0/2004337504 learned_addr learned my addr 172.21.15.102:0/2004337504 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:24:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be77fe700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9be80cd8e0 0x7f9be80b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:24:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be7fff700 1 -- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9be80cd8e0 msgr2=0x7f9be80b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:24:48.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be7fff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9be80cd8e0 0x7f9be80b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:24:48.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be7fff700 1 -- 172.21.15.102:0/2004337504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9be8153290 msgr2=0x7f9be8151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:24:48.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be7fff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9be8153290 0x7f9be8151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:24:48.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.236+0000 7f9be7fff700 1 -- 172.21.15.102:0/2004337504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80094a0 con 0x7f9be81593c0 2023-03-03T06:24:48.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.237+0000 7f9be7fff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be81593c0 0x7f9be814f530 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9be000caf0 tx=0x7f9be000ce00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:24:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.237+0000 7f9bcffff700 1 -- 172.21.15.102:0/2004337504 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be000a9e0 con 0x7f9be81593c0 2023-03-03T06:24:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.237+0000 7f9bcffff700 1 -- 172.21.15.102:0/2004337504 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be000ab60 con 0x7f9be81593c0 2023-03-03T06:24:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.237+0000 7f9bcffff700 1 -- 172.21.15.102:0/2004337504 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be00070d0 con 0x7f9be81593c0 2023-03-03T06:24:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.237+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be815d160 con 0x7f9be81593c0 2023-03-03T06:24:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.237+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be806c7f0 con 0x7f9be81593c0 2023-03-03T06:24:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.238+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be80ccde0 con 0x7f9be81593c0 2023-03-03T06:24:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.240+0000 7f9bcffff700 1 -- 172.21.15.102:0/2004337504 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9be0010070 con 0x7f9be81593c0 2023-03-03T06:24:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.241+0000 7f9bcffff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9bd40682f0 0x7f9bd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:24:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.241+0000 7f9bcffff700 1 -- 172.21.15.102:0/2004337504 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9be0085550 con 0x7f9be81593c0 2023-03-03T06:24:48.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.241+0000 7f9be77fe700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9bd40682f0 0x7f9bd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:24:48.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.242+0000 7f9be77fe700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9bd40682f0 0x7f9bd406a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009fd0 tx=0x7f9bd801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:24:48.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.245+0000 7f9bcffff700 1 -- 172.21.15.102:0/2004337504 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9be0051c00 con 0x7f9be81593c0 2023-03-03T06:24:48.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.478+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9be80b3720 con 0x7f9be81593c0 2023-03-03T06:24:48.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.479+0000 7f9bcffff700 1 -- 172.21.15.102:0/2004337504 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9be80b3720 con 0x7f9be81593c0 2023-03-03T06:24:48.481 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:24:48.481 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:24:48.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9bd40682f0 msgr2=0x7f9bd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:24:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9bd40682f0 0x7f9bd406a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009fd0 tx=0x7f9bd801c040 comp rx=0 tx=0).stop 2023-03-03T06:24:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be81593c0 msgr2=0x7f9be814f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:24:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be81593c0 0x7f9be814f530 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9be000caf0 tx=0x7f9be000ce00 comp rx=0 tx=0).stop 2023-03-03T06:24:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 shutdown_connections 2023-03-03T06:24:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9bd40682f0 0x7f9bd406a7a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:24:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9be80cd8e0 0x7f9be80b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:24:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9be81593c0 0x7f9be814f530 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:24:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 --2- 172.21.15.102:0/2004337504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9be8153290 0x7f9be8151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:24:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 >> 172.21.15.102:0/2004337504 conn(0x7f9be8065fb0 msgr2=0x7f9be815ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:24:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 shutdown_connections 2023-03-03T06:24:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:24:48.482+0000 7f9bedbff700 1 -- 172.21.15.102:0/2004337504 wait complete. 2023-03-03T06:24:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:24:48.511 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T06:25:11.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:11.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:41.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:41.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:57.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:25:57.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.760+0000 7f4c6d3ff700 1 Processor -- start 2023-03-03T06:25:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.760+0000 7f4c6d3ff700 1 -- start start 2023-03-03T06:25:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.761+0000 7f4c6d3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c68153290 0x7f4c68158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:25:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.761+0000 7f4c6d3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c681593c0 0x7f4c680cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:25:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.761+0000 7f4c6d3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c68153660 con 0x7f4c680cd8e0 2023-03-03T06:25:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.761+0000 7f4c6d3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6806c360 con 0x7f4c68153290 2023-03-03T06:25:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.761+0000 7f4c6d3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6815cea0 con 0x7f4c681593c0 2023-03-03T06:25:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c667fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c681593c0 0x7f4c680cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:25:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c66ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c68153290 0x7f4c68158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:25:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c677fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4c680cd8e0 0x7f4c680ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35494/0 (socket says 172.21.15.102:35494) 2023-03-03T06:25:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c677fe700 1 -- 172.21.15.102:0/2819132688 learned_addr learned my addr 172.21.15.102:0/2819132688 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:25:57.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c65ffb700 1 -- 172.21.15.102:0/2819132688 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643334682 0 0) 0x7f4c68153660 con 0x7f4c680cd8e0 2023-03-03T06:25:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c66ffd700 1 -- 172.21.15.102:0/2819132688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c681593c0 msgr2=0x7f4c680cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:25:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c66ffd700 1 --2- 172.21.15.102:0/2819132688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c681593c0 0x7f4c680cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c66ffd700 1 -- 172.21.15.102:0/2819132688 >> v1:172.21.15.172:6789/0 conn(0x7f4c680cd8e0 legacy=0x7f4c680ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:25:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c66ffd700 1 -- 172.21.15.102:0/2819132688 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6806c7f0 con 0x7f4c68153290 2023-03-03T06:25:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.762+0000 7f4c667fc700 1 --2- 172.21.15.102:0/2819132688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c681593c0 0x7f4c680cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:25:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.763+0000 7f4c66ffd700 1 --2- 172.21.15.102:0/2819132688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c68153290 0x7f4c68158e80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009840 tx=0x7f4c58009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2729514af68b8c26 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:25:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.763+0000 7f4c65ffb700 1 -- 172.21.15.102:0/2819132688 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58011830 con 0x7f4c68153290 2023-03-03T06:25:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.763+0000 7f4c65ffb700 1 -- 172.21.15.102:0/2819132688 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58011990 con 0x7f4c68153290 2023-03-03T06:25:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.763+0000 7f4c65ffb700 1 -- 172.21.15.102:0/2819132688 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58011d20 con 0x7f4c68153290 2023-03-03T06:25:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.763+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/2819132688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c68153290 msgr2=0x7f4c68158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:25:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.763+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/2819132688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c68153290 0x7f4c68158e80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009840 tx=0x7f4c58009b50 comp rx=0 tx=0).stop 2023-03-03T06:25:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.763+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/2819132688 shutdown_connections 2023-03-03T06:25:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.764+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/2819132688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c68153290 0x7f4c68158e80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.764+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/2819132688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c681593c0 0x7f4c680cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:57.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.764+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/2819132688 >> 172.21.15.102:0/2819132688 conn(0x7f4c68065fb0 msgr2=0x7f4c680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:25:57.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.764+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/2819132688 shutdown_connections 2023-03-03T06:25:57.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.764+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/2819132688 wait complete. 2023-03-03T06:25:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.766+0000 7f4c6d3ff700 1 Processor -- start 2023-03-03T06:25:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.766+0000 7f4c6d3ff700 1 -- start start 2023-03-03T06:25:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c6d3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c680cd8e0 0x7f4c6814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:25:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c6d3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c68153290 0x7f4c680b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:25:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c6d3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c681593c0 0x7f4c680b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:25:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c6d3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681e58f0 con 0x7f4c681593c0 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c6d3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6814fef0 con 0x7f4c680cd8e0 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c6d3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681e5610 con 0x7f4c68153290 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c66ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c680cd8e0 0x7f4c6814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c66ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c680cd8e0 0x7f4c6814f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55554/0 (socket says 172.21.15.102:55554) 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c66ffd700 1 -- 172.21.15.102:0/4159901168 learned_addr learned my addr 172.21.15.102:0/4159901168 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c667fc700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c68153290 0x7f4c680b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.767+0000 7f4c677fe700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c681593c0 0x7f4c680b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:25:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c66ffd700 1 -- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c68153290 msgr2=0x7f4c680b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:25:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c66ffd700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c68153290 0x7f4c680b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c66ffd700 1 -- 172.21.15.102:0/4159901168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c681593c0 msgr2=0x7f4c680b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:25:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c66ffd700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c681593c0 0x7f4c680b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c66ffd700 1 -- 172.21.15.102:0/4159901168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0094a0 con 0x7f4c680cd8e0 2023-03-03T06:25:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c667fc700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c68153290 0x7f4c680b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:25:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c66ffd700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c680cd8e0 0x7f4c6814f9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009fd0 tx=0x7f4c58004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:25:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c4f7fe700 1 -- 172.21.15.102:0/4159901168 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58004550 con 0x7f4c680cd8e0 2023-03-03T06:25:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c4f7fe700 1 -- 172.21.15.102:0/4159901168 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5801e070 con 0x7f4c680cd8e0 2023-03-03T06:25:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c4f7fe700 1 -- 172.21.15.102:0/4159901168 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c580038e0 con 0x7f4c680cd8e0 2023-03-03T06:25:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.768+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c580094a0 con 0x7f4c680cd8e0 2023-03-03T06:25:57.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.769+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c680ccde0 con 0x7f4c680cd8e0 2023-03-03T06:25:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.770+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6815d160 con 0x7f4c680cd8e0 2023-03-03T06:25:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.771+0000 7f4c4f7fe700 1 -- 172.21.15.102:0/4159901168 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c5800f050 con 0x7f4c680cd8e0 2023-03-03T06:25:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.772+0000 7f4c4f7fe700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c540682f0 0x7f4c5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:25:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.772+0000 7f4c4f7fe700 1 -- 172.21.15.102:0/4159901168 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4c58086cf0 con 0x7f4c680cd8e0 2023-03-03T06:25:57.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.772+0000 7f4c667fc700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c540682f0 0x7f4c5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:25:57.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.773+0000 7f4c667fc700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c540682f0 0x7f4c5406a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c680b37c0 tx=0x7f4c5c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:25:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:57.777+0000 7f4c4f7fe700 1 -- 172.21.15.102:0/4159901168 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c58053380 con 0x7f4c680cd8e0 2023-03-03T06:25:58.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.010+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c680cd0c0 con 0x7f4c680cd8e0 2023-03-03T06:25:58.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.012+0000 7f4c4f7fe700 1 -- 172.21.15.102:0/4159901168 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4c58020090 con 0x7f4c680cd8e0 2023-03-03T06:25:58.014 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:25:58.015 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:25:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c540682f0 msgr2=0x7f4c5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:25:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c540682f0 0x7f4c5406a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c680b37c0 tx=0x7f4c5c009210 comp rx=0 tx=0).stop 2023-03-03T06:25:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c680cd8e0 msgr2=0x7f4c6814f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:25:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c680cd8e0 0x7f4c6814f9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009fd0 tx=0x7f4c58004750 comp rx=0 tx=0).stop 2023-03-03T06:25:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 shutdown_connections 2023-03-03T06:25:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c681593c0 0x7f4c680b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c680cd8e0 0x7f4c6814f9b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c540682f0 0x7f4c5406a7a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 --2- 172.21.15.102:0/4159901168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c68153290 0x7f4c680b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:25:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 >> 172.21.15.102:0/4159901168 conn(0x7f4c68065fb0 msgr2=0x7f4c680cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:25:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 shutdown_connections 2023-03-03T06:25:58.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:25:58.014+0000 7f4c6d3ff700 1 -- 172.21.15.102:0/4159901168 wait complete. 2023-03-03T06:25:58.019 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:25:58.045 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2023-03-03T06:26:11.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:11.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:12.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:26:12.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.296+0000 7fbf859ff700 1 Processor -- start 2023-03-03T06:26:12.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.296+0000 7fbf859ff700 1 -- start start 2023-03-03T06:26:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.297+0000 7fbf859ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf80153290 0x7fbf80158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.297+0000 7fbf859ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf801593c0 0x7fbf800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.297+0000 7fbf859ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbf800cd8e0 0x7fbf800ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.297+0000 7fbf859ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf80153660 con 0x7fbf800cd8e0 2023-03-03T06:26:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.297+0000 7fbf859ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8006c360 con 0x7fbf801593c0 2023-03-03T06:26:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.297+0000 7fbf859ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8015cea0 con 0x7fbf80153290 2023-03-03T06:26:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbf800cd8e0 0x7fbf800ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7e7fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf801593c0 0x7fbf800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf80153290 0x7fbf80158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf80153290 0x7fbf80158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60076/0 (socket says 172.21.15.102:60076) 2023-03-03T06:26:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbf800cd8e0 0x7fbf800ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47810/0 (socket says 172.21.15.102:47810) 2023-03-03T06:26:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7f7fe700 1 -- 172.21.15.102:0/1727320852 learned_addr learned my addr 172.21.15.102:0/1727320852 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:26:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7e7fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf801593c0 0x7fbf800cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38140/0 (socket says 172.21.15.102:38140) 2023-03-03T06:26:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7e7fc700 1 -- 172.21.15.102:0/1727320852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf80153290 msgr2=0x7fbf80158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:26:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7e7fc700 1 --2- 172.21.15.102:0/1727320852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf80153290 0x7fbf80158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7e7fc700 1 -- 172.21.15.102:0/1727320852 >> v2:172.21.15.102:3300/0 conn(0x7fbf800cd8e0 msgr2=0x7fbf800ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7e7fc700 1 --2- 172.21.15.102:0/1727320852 >> v2:172.21.15.102:3300/0 conn(0x7fbf800cd8e0 0x7fbf800ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.298+0000 7fbf7e7fc700 1 -- 172.21.15.102:0/1727320852 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8006c7f0 con 0x7fbf801593c0 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.299+0000 7fbf7e7fc700 1 --2- 172.21.15.102:0/1727320852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf801593c0 0x7fbf800cde20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbf800b1970 tx=0x7fbf74009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e98f6daa6897b47f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.299+0000 7fbf7dffb700 1 -- 172.21.15.102:0/1727320852 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf74011830 con 0x7fbf801593c0 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.299+0000 7fbf7dffb700 1 -- 172.21.15.102:0/1727320852 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf74011990 con 0x7fbf801593c0 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.299+0000 7fbf7dffb700 1 -- 172.21.15.102:0/1727320852 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf74011d20 con 0x7fbf801593c0 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.299+0000 7fbf859ff700 1 -- 172.21.15.102:0/1727320852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf801593c0 msgr2=0x7fbf800cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.299+0000 7fbf859ff700 1 --2- 172.21.15.102:0/1727320852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf801593c0 0x7fbf800cde20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbf800b1970 tx=0x7fbf74009b20 comp rx=0 tx=0).stop 2023-03-03T06:26:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.300+0000 7fbf859ff700 1 -- 172.21.15.102:0/1727320852 shutdown_connections 2023-03-03T06:26:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.300+0000 7fbf859ff700 1 --2- 172.21.15.102:0/1727320852 >> v2:172.21.15.102:3300/0 conn(0x7fbf800cd8e0 0x7fbf800ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.300+0000 7fbf859ff700 1 --2- 172.21.15.102:0/1727320852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf80153290 0x7fbf80158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.300+0000 7fbf859ff700 1 --2- 172.21.15.102:0/1727320852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf801593c0 0x7fbf800cde20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.300+0000 7fbf859ff700 1 -- 172.21.15.102:0/1727320852 >> 172.21.15.102:0/1727320852 conn(0x7fbf80065fb0 msgr2=0x7fbf800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:26:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.300+0000 7fbf859ff700 1 -- 172.21.15.102:0/1727320852 shutdown_connections 2023-03-03T06:26:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.300+0000 7fbf859ff700 1 -- 172.21.15.102:0/1727320852 wait complete. 2023-03-03T06:26:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.302+0000 7fbf859ff700 1 Processor -- start 2023-03-03T06:26:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.302+0000 7fbf859ff700 1 -- start start 2023-03-03T06:26:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf859ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf800cd8e0 0x7fbf800cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf859ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf80153290 0x7fbf800b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf859ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf801593c0 0x7fbf801f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf859ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf800c75b0 con 0x7fbf800cd8e0 2023-03-03T06:26:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf859ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf800b1c70 con 0x7fbf80153290 2023-03-03T06:26:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf859ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf800b1ea0 con 0x7fbf801593c0 2023-03-03T06:26:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf7effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf800cd8e0 0x7fbf800cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf7effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf800cd8e0 0x7fbf800cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47814/0 (socket says 172.21.15.102:47814) 2023-03-03T06:26:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.303+0000 7fbf7effd700 1 -- 172.21.15.102:0/3383531152 learned_addr learned my addr 172.21.15.102:0/3383531152 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7e7fc700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf80153290 0x7fbf800b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7f7fe700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf801593c0 0x7fbf801f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7e7fc700 1 -- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf801593c0 msgr2=0x7fbf801f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7e7fc700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf801593c0 0x7fbf801f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7e7fc700 1 -- 172.21.15.102:0/3383531152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf800cd8e0 msgr2=0x7fbf800cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7e7fc700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf800cd8e0 0x7fbf800cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7e7fc700 1 -- 172.21.15.102:0/3383531152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740094a0 con 0x7fbf80153290 2023-03-03T06:26:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf7e7fc700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf80153290 0x7fbf800b2e80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf74016000 tx=0x7fbf74004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:26:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.304+0000 7fbf67fff700 1 -- 172.21.15.102:0/3383531152 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf74004510 con 0x7fbf80153290 2023-03-03T06:26:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.305+0000 7fbf67fff700 1 -- 172.21.15.102:0/3383531152 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7401f070 con 0x7fbf80153290 2023-03-03T06:26:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.305+0000 7fbf67fff700 1 -- 172.21.15.102:0/3383531152 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf740038e0 con 0x7fbf80153290 2023-03-03T06:26:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.305+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf800ccde0 con 0x7fbf80153290 2023-03-03T06:26:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.305+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8015d160 con 0x7fbf80153290 2023-03-03T06:26:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.306+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf800cd0c0 con 0x7fbf80153290 2023-03-03T06:26:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.307+0000 7fbf67fff700 1 -- 172.21.15.102:0/3383531152 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf74048070 con 0x7fbf80153290 2023-03-03T06:26:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.308+0000 7fbf67fff700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf6c068320 0x7fbf6c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.308+0000 7fbf67fff700 1 -- 172.21.15.102:0/3383531152 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbf74021070 con 0x7fbf80153290 2023-03-03T06:26:12.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.309+0000 7fbf7effd700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf6c068320 0x7fbf6c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:12.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.309+0000 7fbf7effd700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf6c068320 0x7fbf6c06a7d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009b80 tx=0x7fbf70009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:26:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.311+0000 7fbf67fff700 1 -- 172.21.15.102:0/3383531152 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf740545b0 con 0x7fbf80153290 2023-03-03T06:26:12.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.542+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf800c76f0 con 0x7fbf80153290 2023-03-03T06:26:12.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.543+0000 7fbf67fff700 1 -- 172.21.15.102:0/3383531152 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbf800c76f0 con 0x7fbf80153290 2023-03-03T06:26:12.545 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:26:12.545 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:26:12.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf6c068320 msgr2=0x7fbf6c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:12.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf6c068320 0x7fbf6c06a7d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009b80 tx=0x7fbf70009210 comp rx=0 tx=0).stop 2023-03-03T06:26:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf80153290 msgr2=0x7fbf800b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf80153290 0x7fbf800b2e80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf74016000 tx=0x7fbf74004750 comp rx=0 tx=0).stop 2023-03-03T06:26:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 shutdown_connections 2023-03-03T06:26:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf6c068320 0x7fbf6c06a7d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf801593c0 0x7fbf801f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf800cd8e0 0x7fbf800cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 --2- 172.21.15.102:0/3383531152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf80153290 0x7fbf800b2e80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:12.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 >> 172.21.15.102:0/3383531152 conn(0x7fbf80065fb0 msgr2=0x7fbf80150e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:26:12.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 shutdown_connections 2023-03-03T06:26:12.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:12.546+0000 7fbf859ff700 1 -- 172.21.15.102:0/3383531152 wait complete. 2023-03-03T06:26:12.549 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:26:12.575 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T06:26:41.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:41.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:56.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:26:56.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.820+0000 7f929a19d700 1 Processor -- start 2023-03-03T06:26:56.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.820+0000 7f929a19d700 1 -- start start 2023-03-03T06:26:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f929a19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92940c9610 0x7f92940ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f929a19d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92940c9ca0 con 0x7f92940ca980 2023-03-03T06:26:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f929a19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92940b46f0 con 0x7f92941564b0 2023-03-03T06:26:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f929a19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92940b4920 con 0x7f92940c9610 2023-03-03T06:26:56.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f929899a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92940c9610 0x7f92940ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f929999c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f92940ca980 0x7f92941636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37838/0 (socket says 172.21.15.102:37838) 2023-03-03T06:26:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f929999c700 1 -- 172.21.15.102:0/1038963885 learned_addr learned my addr 172.21.15.102:0/1038963885 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:26:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.822+0000 7f928bfff700 1 -- 172.21.15.102:0/1038963885 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139390114 0 0) 0x7f92940b46f0 con 0x7f92941564b0 2023-03-03T06:26:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.823+0000 7f929899a700 1 -- 172.21.15.102:0/1038963885 >> v1:172.21.15.172:6789/0 conn(0x7f92941564b0 legacy=0x7f92940cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.823+0000 7f929899a700 1 -- 172.21.15.102:0/1038963885 >> v1:172.21.15.102:6789/0 conn(0x7f92940ca980 legacy=0x7f92941636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.823+0000 7f929899a700 1 -- 172.21.15.102:0/1038963885 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92940b4c00 con 0x7f92940c9610 2023-03-03T06:26:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.823+0000 7f929899a700 1 --2- 172.21.15.102:0/1038963885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940c9610 0x7f92940ca440 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9284009840 tx=0x7f9284009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1db1c44890c405 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:26:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.823+0000 7f928bfff700 1 -- 172.21.15.102:0/1038963885 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284004230 con 0x7f92940c9610 2023-03-03T06:26:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.823+0000 7f928bfff700 1 -- 172.21.15.102:0/1038963885 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9284004390 con 0x7f92940c9610 2023-03-03T06:26:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.823+0000 7f928bfff700 1 -- 172.21.15.102:0/1038963885 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840036a0 con 0x7f92940c9610 2023-03-03T06:26:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.824+0000 7f929a19d700 1 -- 172.21.15.102:0/1038963885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940c9610 msgr2=0x7f92940ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.824+0000 7f929a19d700 1 --2- 172.21.15.102:0/1038963885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940c9610 0x7f92940ca440 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9284009840 tx=0x7f9284009b50 comp rx=0 tx=0).stop 2023-03-03T06:26:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.824+0000 7f929a19d700 1 -- 172.21.15.102:0/1038963885 shutdown_connections 2023-03-03T06:26:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.824+0000 7f929a19d700 1 --2- 172.21.15.102:0/1038963885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940c9610 0x7f92940ca440 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.824+0000 7f929a19d700 1 -- 172.21.15.102:0/1038963885 >> 172.21.15.102:0/1038963885 conn(0x7f9294065fb0 msgr2=0x7f92940663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:26:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.824+0000 7f929a19d700 1 -- 172.21.15.102:0/1038963885 shutdown_connections 2023-03-03T06:26:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.824+0000 7f929a19d700 1 -- 172.21.15.102:0/1038963885 wait complete. 2023-03-03T06:26:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.826+0000 7f929a19d700 1 Processor -- start 2023-03-03T06:26:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.826+0000 7f929a19d700 1 -- start start 2023-03-03T06:26:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929a19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92940c9610 0x7f929415c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929a19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940ca980 0x7f92940c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929a19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92941564b0 0x7f92940c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929a19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929415d2a0 con 0x7f92940ca980 2023-03-03T06:26:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929a19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92941eec60 con 0x7f92940c9610 2023-03-03T06:26:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929a19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92941eef40 con 0x7f92941564b0 2023-03-03T06:26:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929899a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940ca980 0x7f92940c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.827+0000 7f929919b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92940c9610 0x7f929415c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940ca980 0x7f92940c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39150/0 (socket says 172.21.15.102:39150) 2023-03-03T06:26:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929999c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92941564b0 0x7f92940c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 -- 172.21.15.102:0/1734334614 learned_addr learned my addr 172.21.15.102:0/1734334614 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:26:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 -- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92941564b0 msgr2=0x7f92940c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92941564b0 0x7f92940c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 -- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92940c9610 msgr2=0x7f929415c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92940c9610 0x7f929415c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 -- 172.21.15.102:0/1734334614 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92840094a0 con 0x7f92940ca980 2023-03-03T06:26:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.828+0000 7f929899a700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940ca980 0x7f92940c6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92840090b0 tx=0x7f9284004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:26:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.829+0000 7f9289ffb700 1 -- 172.21.15.102:0/1734334614 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284003c10 con 0x7f92940ca980 2023-03-03T06:26:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.829+0000 7f9289ffb700 1 -- 172.21.15.102:0/1734334614 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928401f070 con 0x7f92940ca980 2023-03-03T06:26:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.829+0000 7f9289ffb700 1 -- 172.21.15.102:0/1734334614 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284011630 con 0x7f92940ca980 2023-03-03T06:26:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.829+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92940b2210 con 0x7f92940ca980 2023-03-03T06:26:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.829+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92940b1f30 con 0x7f92940ca980 2023-03-03T06:26:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.830+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9294156880 con 0x7f92940ca980 2023-03-03T06:26:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.831+0000 7f9289ffb700 1 -- 172.21.15.102:0/1734334614 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9284017020 con 0x7f92940ca980 2023-03-03T06:26:56.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.832+0000 7f9289ffb700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92780682f0 0x7f927806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:26:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.832+0000 7f9289ffb700 1 -- 172.21.15.102:0/1734334614 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9284086b60 con 0x7f92940ca980 2023-03-03T06:26:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.833+0000 7f929919b700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92780682f0 0x7f927806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:26:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.833+0000 7f929919b700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92780682f0 0x7f927806a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9290000f60 tx=0x7f9290009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:26:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:56.837+0000 7f9289ffb700 1 -- 172.21.15.102:0/1734334614 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9284053270 con 0x7f92940ca980 2023-03-03T06:26:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.072+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92940c5f30 con 0x7f92940ca980 2023-03-03T06:26:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.073+0000 7f9289ffb700 1 -- 172.21.15.102:0/1734334614 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f92940c5f30 con 0x7f92940ca980 2023-03-03T06:26:57.075 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:26:57.076 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:26:57.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92780682f0 msgr2=0x7f927806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:57.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92780682f0 0x7f927806a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9290000f60 tx=0x7f9290009040 comp rx=0 tx=0).stop 2023-03-03T06:26:57.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940ca980 msgr2=0x7f92940c6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:26:57.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940ca980 0x7f92940c6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92840090b0 tx=0x7f9284004750 comp rx=0 tx=0).stop 2023-03-03T06:26:57.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 shutdown_connections 2023-03-03T06:26:57.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92780682f0 0x7f927806a7a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:57.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92941564b0 0x7f92940c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:57.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92940c9610 0x7f929415c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:57.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 --2- 172.21.15.102:0/1734334614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92940ca980 0x7f92940c6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:26:57.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 >> 172.21.15.102:0/1734334614 conn(0x7f9294065fb0 msgr2=0x7f92941534a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:26:57.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.075+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 shutdown_connections 2023-03-03T06:26:57.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:26:57.076+0000 7f929a19d700 1 -- 172.21.15.102:0/1734334614 wait complete. 2023-03-03T06:26:57.082 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:26:57.104 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T06:27:11.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:27:11.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:27:42.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:27:42.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:12.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:12.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:42.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:42.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:51.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:28:51.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.353+0000 7f3912fff700 1 Processor -- start 2023-03-03T06:28:51.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.353+0000 7f3912fff700 1 -- start start 2023-03-03T06:28:51.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.354+0000 7f3912fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f390c153260 0x7f390c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:28:51.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.354+0000 7f3912fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c0c9cb0 0x7f390c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:28:51.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.354+0000 7f3912fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390c0cf940 con 0x7f390c0cac20 2023-03-03T06:28:51.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.354+0000 7f3912fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c0cfe70 con 0x7f390c153260 2023-03-03T06:28:51.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.354+0000 7f3912fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c158d70 con 0x7f390c0c9cb0 2023-03-03T06:28:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f390bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c0c9cb0 0x7f390c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:28:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f3910d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f390c153260 0x7f390c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:28:51.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f390bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c0c9cb0 0x7f390c0cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57260/0 (socket says 172.21.15.102:57260) 2023-03-03T06:28:51.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f390bfff700 1 -- 172.21.15.102:0/867833495 learned_addr learned my addr 172.21.15.102:0/867833495 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:28:51.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f390bfff700 1 -- 172.21.15.102:0/867833495 >> v2:172.21.15.102:3300/0 conn(0x7f390c153260 msgr2=0x7f390c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:28:51.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f390bfff700 1 --2- 172.21.15.102:0/867833495 >> v2:172.21.15.102:3300/0 conn(0x7f390c153260 0x7f390c0ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f390bfff700 1 -- 172.21.15.102:0/867833495 >> v1:172.21.15.102:6789/0 conn(0x7f390c0cac20 legacy=0x7f390c15c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:28:51.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.355+0000 7f390bfff700 1 -- 172.21.15.102:0/867833495 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c0cfb90 con 0x7f390c0c9cb0 2023-03-03T06:28:51.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f390bfff700 1 --2- 172.21.15.102:0/867833495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c0c9cb0 0x7f390c0cb880 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38f80098f0 tx=0x7f38f8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5c59dca8ae0f344 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:28:51.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f390b7fe700 1 -- 172.21.15.102:0/867833495 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f8011830 con 0x7f390c0c9cb0 2023-03-03T06:28:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f390b7fe700 1 -- 172.21.15.102:0/867833495 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f8011990 con 0x7f390c0c9cb0 2023-03-03T06:28:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f390b7fe700 1 -- 172.21.15.102:0/867833495 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f8011d20 con 0x7f390c0c9cb0 2023-03-03T06:28:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f3912fff700 1 -- 172.21.15.102:0/867833495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c0c9cb0 msgr2=0x7f390c0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:28:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f3912fff700 1 --2- 172.21.15.102:0/867833495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c0c9cb0 0x7f390c0cb880 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38f80098f0 tx=0x7f38f8009c00 comp rx=0 tx=0).stop 2023-03-03T06:28:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f3912fff700 1 -- 172.21.15.102:0/867833495 shutdown_connections 2023-03-03T06:28:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f3912fff700 1 --2- 172.21.15.102:0/867833495 >> v2:172.21.15.102:3300/0 conn(0x7f390c153260 0x7f390c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.356+0000 7f3912fff700 1 --2- 172.21.15.102:0/867833495 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c0c9cb0 0x7f390c0cb880 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.357+0000 7f3912fff700 1 -- 172.21.15.102:0/867833495 >> 172.21.15.102:0/867833495 conn(0x7f390c066c00 msgr2=0x7f390c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:28:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.357+0000 7f3912fff700 1 -- 172.21.15.102:0/867833495 shutdown_connections 2023-03-03T06:28:51.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.357+0000 7f3912fff700 1 -- 172.21.15.102:0/867833495 wait complete. 2023-03-03T06:28:51.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.359+0000 7f3912fff700 1 Processor -- start 2023-03-03T06:28:51.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.359+0000 7f3912fff700 1 -- start start 2023-03-03T06:28:51.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3912fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f390c0cac20 0x7f390c1f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:28:51.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3912fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c153260 0x7f390c150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:28:51.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3912fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f390c1564b0 0x7f390c14d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:28:51.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3912fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c14ef80 con 0x7f390c0cac20 2023-03-03T06:28:51.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3912fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1f2710 con 0x7f390c1564b0 2023-03-03T06:28:51.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3912fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1f2430 con 0x7f390c153260 2023-03-03T06:28:51.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3910d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f390c0cac20 0x7f390c1f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:28:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3910d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f390c0cac20 0x7f390c1f1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46686/0 (socket says 172.21.15.102:46686) 2023-03-03T06:28:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f3910d9b700 1 -- 172.21.15.102:0/2865389143 learned_addr learned my addr 172.21.15.102:0/2865389143 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:28:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f390bfff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c153260 0x7f390c150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:28:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.360+0000 7f391159c700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f390c1564b0 0x7f390c14d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:28:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f390bfff700 1 -- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f390c1564b0 msgr2=0x7f390c14d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:28:51.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f390bfff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f390c1564b0 0x7f390c14d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f390bfff700 1 -- 172.21.15.102:0/2865389143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f390c0cac20 msgr2=0x7f390c1f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:28:51.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f390bfff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f390c0cac20 0x7f390c1f1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f390bfff700 1 -- 172.21.15.102:0/2865389143 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f8009550 con 0x7f390c153260 2023-03-03T06:28:51.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f3910d9b700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f390c0cac20 0x7f390c1f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:28:51.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f390bfff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c153260 0x7f390c150270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38f8016000 tx=0x7f38f8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:28:51.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f39097fa700 1 -- 172.21.15.102:0/2865389143 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f8004450 con 0x7f390c153260 2023-03-03T06:28:51.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f39097fa700 1 -- 172.21.15.102:0/2865389143 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f801e070 con 0x7f390c153260 2023-03-03T06:28:51.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f39097fa700 1 -- 172.21.15.102:0/2865389143 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80038e0 con 0x7f390c153260 2023-03-03T06:28:51.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c159260 con 0x7f390c153260 2023-03-03T06:28:51.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.361+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c158f80 con 0x7f390c153260 2023-03-03T06:28:51.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.363+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c159540 con 0x7f390c153260 2023-03-03T06:28:51.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.364+0000 7f39097fa700 1 -- 172.21.15.102:0/2865389143 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f38f800f050 con 0x7f390c153260 2023-03-03T06:28:51.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.365+0000 7f39097fa700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38f00682f0 0x7f38f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:28:51.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.365+0000 7f39097fa700 1 -- 172.21.15.102:0/2865389143 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f38f8086c10 con 0x7f390c153260 2023-03-03T06:28:51.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.365+0000 7f3910d9b700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38f00682f0 0x7f38f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:28:51.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.366+0000 7f3910d9b700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38f00682f0 0x7f38f006a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f39000113d0 tx=0x7f3900009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:28:51.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.370+0000 7f39097fa700 1 -- 172.21.15.102:0/2865389143 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f38f80532a0 con 0x7f390c153260 2023-03-03T06:28:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.604+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f390c06d020 con 0x7f390c153260 2023-03-03T06:28:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.605+0000 7f39097fa700 1 -- 172.21.15.102:0/2865389143 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f38f80045b0 con 0x7f390c153260 2023-03-03T06:28:51.607 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:28:51.607 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:28:51.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38f00682f0 msgr2=0x7f38f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:28:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38f00682f0 0x7f38f006a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f39000113d0 tx=0x7f3900009210 comp rx=0 tx=0).stop 2023-03-03T06:28:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c153260 msgr2=0x7f390c150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:28:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c153260 0x7f390c150270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38f8016000 tx=0x7f38f8004750 comp rx=0 tx=0).stop 2023-03-03T06:28:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 shutdown_connections 2023-03-03T06:28:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f390c1564b0 0x7f390c14d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f390c0cac20 0x7f390c1f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38f00682f0 0x7f38f006a7a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 --2- 172.21.15.102:0/2865389143 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f390c153260 0x7f390c150270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:28:51.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 >> 172.21.15.102:0/2865389143 conn(0x7f390c0c9cb0 msgr2=0x7f390c0b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:28:51.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 shutdown_connections 2023-03-03T06:28:51.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:28:51.608+0000 7f3912fff700 1 -- 172.21.15.102:0/2865389143 wait complete. 2023-03-03T06:28:51.611 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:28:51.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 20 secs before thrashing 2023-03-03T06:29:11.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:29:11.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.901+0000 7f07d759e700 1 Processor -- start 2023-03-03T06:29:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.901+0000 7f07d759e700 1 -- start start 2023-03-03T06:29:11.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.902+0000 7f07d759e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f07d80c9610 0x7f07d80ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:29:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d80c9ca0 con 0x7f07d81564b0 2023-03-03T06:29:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d759e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d80b46f0 con 0x7f07d80ca980 2023-03-03T06:29:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d759e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d80b4920 con 0x7f07d80c9610 2023-03-03T06:29:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d5d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f07d80c9610 0x7f07d80ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:29:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d6d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f07d80ca980 0x7f07d81636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54116/0 (socket says 172.21.15.102:54116) 2023-03-03T06:29:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d6d9d700 1 -- 172.21.15.102:0/2770280327 learned_addr learned my addr 172.21.15.102:0/2770280327 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:29:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d559a700 1 -- 172.21.15.102:0/2770280327 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733241903 0 0) 0x7f07d80b46f0 con 0x7f07d80ca980 2023-03-03T06:29:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d559a700 1 -- 172.21.15.102:0/2770280327 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c00035b0 con 0x7f07d80ca980 2023-03-03T06:29:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d5d9b700 1 -- 172.21.15.102:0/2770280327 >> v1:172.21.15.172:6789/0 conn(0x7f07d80ca980 legacy=0x7f07d81636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:29:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d5d9b700 1 -- 172.21.15.102:0/2770280327 >> v1:172.21.15.102:6789/0 conn(0x7f07d81564b0 legacy=0x7f07d80cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:29:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d559a700 1 -- 172.21.15.102:0/2770280327 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548720977 0 0) 0x7f07d80c9ca0 con 0x7f07d81564b0 2023-03-03T06:29:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.903+0000 7f07d5d9b700 1 -- 172.21.15.102:0/2770280327 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80b4c00 con 0x7f07d80c9610 2023-03-03T06:29:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.904+0000 7f07d5d9b700 1 --2- 172.21.15.102:0/2770280327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80c9610 0x7f07d80ca440 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009840 tx=0x7f07cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c337f21c105e78 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:29:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.904+0000 7f07d559a700 1 -- 172.21.15.102:0/2770280327 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc004230 con 0x7f07d80c9610 2023-03-03T06:29:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.904+0000 7f07d559a700 1 -- 172.21.15.102:0/2770280327 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc004390 con 0x7f07d80c9610 2023-03-03T06:29:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.904+0000 7f07d559a700 1 -- 172.21.15.102:0/2770280327 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc0036a0 con 0x7f07d80c9610 2023-03-03T06:29:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.905+0000 7f07d759e700 1 -- 172.21.15.102:0/2770280327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80c9610 msgr2=0x7f07d80ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:29:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.905+0000 7f07d759e700 1 --2- 172.21.15.102:0/2770280327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80c9610 0x7f07d80ca440 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009840 tx=0x7f07cc009b50 comp rx=0 tx=0).stop 2023-03-03T06:29:11.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.905+0000 7f07d759e700 1 -- 172.21.15.102:0/2770280327 shutdown_connections 2023-03-03T06:29:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.905+0000 7f07d759e700 1 --2- 172.21.15.102:0/2770280327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80c9610 0x7f07d80ca440 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:29:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.905+0000 7f07d759e700 1 -- 172.21.15.102:0/2770280327 >> 172.21.15.102:0/2770280327 conn(0x7f07d8065fb0 msgr2=0x7f07d80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:29:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.905+0000 7f07d759e700 1 -- 172.21.15.102:0/2770280327 shutdown_connections 2023-03-03T06:29:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.905+0000 7f07d759e700 1 -- 172.21.15.102:0/2770280327 wait complete. 2023-03-03T06:29:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.907+0000 7f07d759e700 1 Processor -- start 2023-03-03T06:29:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.907+0000 7f07d759e700 1 -- start start 2023-03-03T06:29:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.908+0000 7f07d759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d80c9610 0x7f07d80cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:29:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.908+0000 7f07d759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80ca980 0x7f07d81eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:29:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.908+0000 7f07d759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d81564b0 0x7f07d81f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:29:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.908+0000 7f07d759e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8159460 con 0x7f07d80c9610 2023-03-03T06:29:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.908+0000 7f07d759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8157d10 con 0x7f07d80ca980 2023-03-03T06:29:11.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.908+0000 7f07d759e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d81ef340 con 0x7f07d81564b0 2023-03-03T06:29:11.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80ca980 0x7f07d81eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:29:11.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d80c9610 0x7f07d80cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:29:11.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80ca980 0x7f07d81eee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45694/0 (socket says 172.21.15.102:45694) 2023-03-03T06:29:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 -- 172.21.15.102:0/1019888306 learned_addr learned my addr 172.21.15.102:0/1019888306 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:29:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d6d9d700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d81564b0 0x7f07d81f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:29:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 -- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d81564b0 msgr2=0x7f07d81f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:29:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d81564b0 0x7f07d81f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:29:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 -- 172.21.15.102:0/1019888306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d80c9610 msgr2=0x7f07d80cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:29:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d80c9610 0x7f07d80cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:29:11.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 -- 172.21.15.102:0/1019888306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc0094a0 con 0x7f07d80ca980 2023-03-03T06:29:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.909+0000 7f07d5d9b700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80ca980 0x7f07d81eee00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07cc014040 tx=0x7f07cc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:29:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.910+0000 7f07c6ffd700 1 -- 172.21.15.102:0/1019888306 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc003d30 con 0x7f07d80ca980 2023-03-03T06:29:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.910+0000 7f07c6ffd700 1 -- 172.21.15.102:0/1019888306 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc003e90 con 0x7f07d80ca980 2023-03-03T06:29:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.910+0000 7f07c6ffd700 1 -- 172.21.15.102:0/1019888306 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc011830 con 0x7f07d80ca980 2023-03-03T06:29:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.910+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d80b2210 con 0x7f07d80ca980 2023-03-03T06:29:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.910+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80b1f30 con 0x7f07d80ca980 2023-03-03T06:29:11.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.911+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d8156880 con 0x7f07d80ca980 2023-03-03T06:29:11.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.912+0000 7f07c6ffd700 1 -- 172.21.15.102:0/1019888306 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f07cc02d070 con 0x7f07d80ca980 2023-03-03T06:29:11.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.913+0000 7f07c6ffd700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b80682f0 0x7f07b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:29:11.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.913+0000 7f07c6ffd700 1 -- 172.21.15.102:0/1019888306 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f07cc0866e0 con 0x7f07d80ca980 2023-03-03T06:29:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.914+0000 7f07d659c700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b80682f0 0x7f07b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:29:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.914+0000 7f07d659c700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b80682f0 0x7f07b806a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f07c8000f60 tx=0x7f07c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:29:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:11.917+0000 7f07c6ffd700 1 -- 172.21.15.102:0/1019888306 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07cc052df0 con 0x7f07d80ca980 2023-03-03T06:29:12.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.156+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07d80677f0 con 0x7f07d80ca980 2023-03-03T06:29:12.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.158+0000 7f07c6ffd700 1 -- 172.21.15.102:0/1019888306 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f07cc056410 con 0x7f07d80ca980 2023-03-03T06:29:12.159 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:29:12.159 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:29:12.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.160+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b80682f0 msgr2=0x7f07b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:29:12.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.160+0000 7f07d759e700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b80682f0 0x7f07b806a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f07c8000f60 tx=0x7f07c8009040 comp rx=0 tx=0).stop 2023-03-03T06:29:12.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.160+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80ca980 msgr2=0x7f07d81eee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:29:12.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.160+0000 7f07d759e700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80ca980 0x7f07d81eee00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07cc014040 tx=0x7f07cc004750 comp rx=0 tx=0).stop 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.160+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 shutdown_connections 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.160+0000 7f07d759e700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b80682f0 0x7f07b806a7a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.160+0000 7f07d759e700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d81564b0 0x7f07d81f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.161+0000 7f07d759e700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d80c9610 0x7f07d80cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.161+0000 7f07d759e700 1 --2- 172.21.15.102:0/1019888306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d80ca980 0x7f07d81eee00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.161+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 >> 172.21.15.102:0/1019888306 conn(0x7f07d8065fb0 msgr2=0x7f07d81556f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.161+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 shutdown_connections 2023-03-03T06:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:29:12.161+0000 7f07d759e700 1 -- 172.21.15.102:0/1019888306 wait complete. 2023-03-03T06:29:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:29:12.189 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-03-03T06:29:12.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:29:12.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:29:42.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:29:42.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T06:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:30:13.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:13.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:18.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:30:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.442+0000 7f9620b9d700 1 Processor -- start 2023-03-03T06:30:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.442+0000 7f9620b9d700 1 -- start start 2023-03-03T06:30:18.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f9620b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f961c0ced00 0x7f961c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:30:18.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f9620b9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f961c0b3970 0x7f961c0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:30:18.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f9620b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:30:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f9620b9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961c0c4930 con 0x7f961c0b3970 2023-03-03T06:30:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f9620b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961c0c3d20 con 0x7f961c0ced00 2023-03-03T06:30:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f9620b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0cf570 con 0x7f961c0cf0d0 2023-03-03T06:30:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f961affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f961c0b3970 0x7f961c0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:30:18.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f961bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f961affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f961c0b3970 0x7f961c0c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58792/0 (socket says 172.21.15.102:58792) 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f961affd700 1 -- 172.21.15.102:0/471549555 learned_addr learned my addr 172.21.15.102:0/471549555 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.444+0000 7f961b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f961c0ced00 0x7f961c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961affd700 1 -- 172.21.15.102:0/471549555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 msgr2=0x7f961c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961affd700 1 --2- 172.21.15.102:0/471549555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961affd700 1 -- 172.21.15.102:0/471549555 >> v2:172.21.15.172:3300/0 conn(0x7f961c0ced00 msgr2=0x7f961c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961affd700 1 --2- 172.21.15.102:0/471549555 >> v2:172.21.15.172:3300/0 conn(0x7f961c0ced00 0x7f961c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961affd700 1 -- 172.21.15.102:0/471549555 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c15cba0 con 0x7f961c0b3970 2023-03-03T06:30:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961affd700 1 --2- 172.21.15.102:0/471549555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0b3970 0x7f961c0c41b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f961c0b1b40 tx=0x7f9610009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d1f2663dbcea5a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:30:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961a7fc700 1 -- 172.21.15.102:0/471549555 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610011830 con 0x7f961c0b3970 2023-03-03T06:30:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961a7fc700 1 -- 172.21.15.102:0/471549555 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9610011990 con 0x7f961c0b3970 2023-03-03T06:30:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.445+0000 7f961a7fc700 1 -- 172.21.15.102:0/471549555 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610011d20 con 0x7f961c0b3970 2023-03-03T06:30:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 -- 172.21.15.102:0/471549555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0b3970 msgr2=0x7f961c0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:30:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 --2- 172.21.15.102:0/471549555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0b3970 0x7f961c0c41b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f961c0b1b40 tx=0x7f9610009bd0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 -- 172.21.15.102:0/471549555 shutdown_connections 2023-03-03T06:30:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 --2- 172.21.15.102:0/471549555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 --2- 172.21.15.102:0/471549555 >> v2:172.21.15.172:3300/0 conn(0x7f961c0ced00 0x7f961c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 --2- 172.21.15.102:0/471549555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0b3970 0x7f961c0c41b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 -- 172.21.15.102:0/471549555 >> 172.21.15.102:0/471549555 conn(0x7f961c08b1c0 msgr2=0x7f961c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:30:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 -- 172.21.15.102:0/471549555 shutdown_connections 2023-03-03T06:30:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.446+0000 7f9620b9d700 1 -- 172.21.15.102:0/471549555 wait complete. 2023-03-03T06:30:18.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.448+0000 7f9620b9d700 1 Processor -- start 2023-03-03T06:30:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.448+0000 7f9620b9d700 1 -- start start 2023-03-03T06:30:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.449+0000 7f9620b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f961c0b3970 0x7f961c1e2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:30:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.449+0000 7f9620b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0ced00 0x7f961c151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:30:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.449+0000 7f9620b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:30:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.449+0000 7f9620b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c15a690 con 0x7f961c0ced00 2023-03-03T06:30:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.449+0000 7f9620b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c151960 con 0x7f961c0b3970 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.449+0000 7f9620b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c1516a0 con 0x7f961c0cf0d0 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0ced00 0x7f961c151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f961c0b3970 0x7f961c1e2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0ced00 0x7f961c151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58796/0 (socket says 172.21.15.102:58796) 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 -- 172.21.15.102:0/1891173580 learned_addr learned my addr 172.21.15.102:0/1891173580 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 -- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 msgr2=0x7f961c148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:30:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 -- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f961c0b3970 msgr2=0x7f961c1e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:30:18.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f961c0b3970 0x7f961c1e2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 -- 172.21.15.102:0/1891173580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9610009550 con 0x7f961c0ced00 2023-03-03T06:30:18.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.450+0000 7f961affd700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0ced00 0x7f961c151db0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9610000c00 tx=0x7f9610004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:30:18.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.451+0000 7f9603fff700 1 -- 172.21.15.102:0/1891173580 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610004450 con 0x7f961c0ced00 2023-03-03T06:30:18.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.451+0000 7f9603fff700 1 -- 172.21.15.102:0/1891173580 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961001e070 con 0x7f961c0ced00 2023-03-03T06:30:18.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.451+0000 7f9603fff700 1 -- 172.21.15.102:0/1891173580 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96100038c0 con 0x7f961c0ced00 2023-03-03T06:30:18.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.451+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c06bf20 con 0x7f961c0ced00 2023-03-03T06:30:18.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.451+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c15ce80 con 0x7f961c0ced00 2023-03-03T06:30:18.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.452+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c06c200 con 0x7f961c0ced00 2023-03-03T06:30:18.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.453+0000 7f9603fff700 1 -- 172.21.15.102:0/1891173580 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f961000f050 con 0x7f961c0ced00 2023-03-03T06:30:18.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.454+0000 7f9603fff700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96080682f0 0x7f960806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:30:18.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.455+0000 7f9603fff700 1 -- 172.21.15.102:0/1891173580 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9610086a60 con 0x7f961c0ced00 2023-03-03T06:30:18.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.455+0000 7f961b7fe700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96080682f0 0x7f960806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:30:18.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.455+0000 7f961b7fe700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96080682f0 0x7f960806a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f960c009b60 tx=0x7f960c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:30:18.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.459+0000 7f9603fff700 1 -- 172.21.15.102:0/1891173580 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9610053170 con 0x7f961c0ced00 2023-03-03T06:30:18.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.692+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f961c14da40 con 0x7f961c0ced00 2023-03-03T06:30:18.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.692+0000 7f9603fff700 1 -- 172.21.15.102:0/1891173580 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f96100045b0 con 0x7f961c0ced00 2023-03-03T06:30:18.694 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:30:18.694 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:30:18.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96080682f0 msgr2=0x7f960806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:30:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96080682f0 0x7f960806a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f960c009b60 tx=0x7f960c009210 comp rx=0 tx=0).stop 2023-03-03T06:30:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0ced00 msgr2=0x7f961c151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:30:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0ced00 0x7f961c151db0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9610000c00 tx=0x7f9610004750 comp rx=0 tx=0).stop 2023-03-03T06:30:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 shutdown_connections 2023-03-03T06:30:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96080682f0 0x7f960806a7a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f961c0cf0d0 0x7f961c148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f961c0b3970 0x7f961c1e2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 --2- 172.21.15.102:0/1891173580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f961c0ced00 0x7f961c151db0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:30:18.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 >> 172.21.15.102:0/1891173580 conn(0x7f961c08b1c0 msgr2=0x7f961c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:30:18.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 shutdown_connections 2023-03-03T06:30:18.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:30:18.695+0000 7f9620b9d700 1 -- 172.21.15.102:0/1891173580 wait complete. 2023-03-03T06:30:18.698 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:30:18.724 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 118 secs before thrashing 2023-03-03T06:30:43.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:43.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:13.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:13.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:43.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:43.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:13.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:13.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:16.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:32:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.977+0000 7f42e85ff700 1 Processor -- start 2023-03-03T06:32:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.977+0000 7f42e85ff700 1 -- start start 2023-03-03T06:32:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.978+0000 7f42e85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e00cd8e0 0x7f42e00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.978+0000 7f42e85ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42e00cd390 0x7f42e00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.978+0000 7f42e85ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e0153660 con 0x7f42e0153290 2023-03-03T06:32:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.978+0000 7f42e85ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e006c360 con 0x7f42e00cd390 2023-03-03T06:32:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.978+0000 7f42e85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e015cea0 con 0x7f42e00cd8e0 2023-03-03T06:32:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e639b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f42e0153290 0x7f42e0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57512/0 (socket says 172.21.15.102:57512) 2023-03-03T06:32:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e639b700 1 -- 172.21.15.102:0/1748734867 learned_addr learned my addr 172.21.15.102:0/1748734867 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:32:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e6b9c700 1 --2- 172.21.15.102:0/1748734867 >> v2:172.21.15.102:3300/0 conn(0x7f42e00cd390 0x7f42e00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e5b9a700 1 --2- 172.21.15.102:0/1748734867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e00cd8e0 0x7f42e00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e5399700 1 -- 172.21.15.102:0/1748734867 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305197274 0 0) 0x7f42e0153660 con 0x7f42e0153290 2023-03-03T06:32:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e6b9c700 1 -- 172.21.15.102:0/1748734867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e00cd8e0 msgr2=0x7f42e00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:32:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e6b9c700 1 --2- 172.21.15.102:0/1748734867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e00cd8e0 0x7f42e00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e6b9c700 1 -- 172.21.15.102:0/1748734867 >> v1:172.21.15.102:6789/0 conn(0x7f42e0153290 legacy=0x7f42e0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.979+0000 7f42e6b9c700 1 -- 172.21.15.102:0/1748734867 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e006c7f0 con 0x7f42e00cd390 2023-03-03T06:32:16.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.980+0000 7f42e6b9c700 1 --2- 172.21.15.102:0/1748734867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e00ce360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009d90 tx=0x7f42d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f508339906ca02b4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:32:16.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.980+0000 7f42e5399700 1 -- 172.21.15.102:0/1748734867 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d800e070 con 0x7f42e00cd390 2023-03-03T06:32:16.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.980+0000 7f42e5399700 1 -- 172.21.15.102:0/1748734867 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d800bb20 con 0x7f42e00cd390 2023-03-03T06:32:16.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.980+0000 7f42e5399700 1 -- 172.21.15.102:0/1748734867 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8012430 con 0x7f42e00cd390 2023-03-03T06:32:16.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.980+0000 7f42e85ff700 1 -- 172.21.15.102:0/1748734867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 msgr2=0x7f42e00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:16.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.980+0000 7f42e85ff700 1 --2- 172.21.15.102:0/1748734867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e00ce360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009d90 tx=0x7f42d800b330 comp rx=0 tx=0).stop 2023-03-03T06:32:16.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.981+0000 7f42e85ff700 1 -- 172.21.15.102:0/1748734867 shutdown_connections 2023-03-03T06:32:16.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.981+0000 7f42e85ff700 1 --2- 172.21.15.102:0/1748734867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e00ce360 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:16.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.981+0000 7f42e85ff700 1 --2- 172.21.15.102:0/1748734867 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e00cd8e0 0x7f42e00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:16.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.981+0000 7f42e85ff700 1 -- 172.21.15.102:0/1748734867 >> 172.21.15.102:0/1748734867 conn(0x7f42e0065fb0 msgr2=0x7f42e00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:32:16.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.981+0000 7f42e85ff700 1 -- 172.21.15.102:0/1748734867 shutdown_connections 2023-03-03T06:32:16.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.981+0000 7f42e85ff700 1 -- 172.21.15.102:0/1748734867 wait complete. 2023-03-03T06:32:16.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.983+0000 7f42e85ff700 1 Processor -- start 2023-03-03T06:32:16.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.984+0000 7f42e85ff700 1 -- start start 2023-03-03T06:32:16.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e85ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e0155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e85ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e00cd8e0 0x7f42e01f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e0153290 0x7f42e01f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e85ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e01e4950 con 0x7f42e00cd390 2023-03-03T06:32:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e85ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0159460 con 0x7f42e00cd8e0 2023-03-03T06:32:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0159740 con 0x7f42e0153290 2023-03-03T06:32:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e0155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:16.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e0155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49858/0 (socket says 172.21.15.102:49858) 2023-03-03T06:32:16.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e639b700 1 -- 172.21.15.102:0/695755341 learned_addr learned my addr 172.21.15.102:0/695755341 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:32:16.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e5b9a700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e00cd8e0 0x7f42e01f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:16.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.985+0000 7f42e6b9c700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e0153290 0x7f42e01f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:16.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e5b9a700 1 -- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e0153290 msgr2=0x7f42e01f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:16.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e5b9a700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e0153290 0x7f42e01f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:16.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e5b9a700 1 -- 172.21.15.102:0/695755341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 msgr2=0x7f42e0155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e5b9a700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e0155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e5b9a700 1 -- 172.21.15.102:0/695755341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d8009a70 con 0x7f42e00cd8e0 2023-03-03T06:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e639b700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e0155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e6b9c700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e0153290 0x7f42e01f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.986+0000 7f42e5b9a700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e00cd8e0 0x7f42e01f1860 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f42d000e870 tx=0x7f42d000eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.987+0000 7f42d6ffd700 1 -- 172.21.15.102:0/695755341 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0013680 con 0x7f42e00cd8e0 2023-03-03T06:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.987+0000 7f42d6ffd700 1 -- 172.21.15.102:0/695755341 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d00137e0 con 0x7f42e00cd8e0 2023-03-03T06:32:16.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.987+0000 7f42d6ffd700 1 -- 172.21.15.102:0/695755341 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0013b90 con 0x7f42e00cd8e0 2023-03-03T06:32:16.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.987+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e00ccde0 con 0x7f42e00cd8e0 2023-03-03T06:32:16.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.987+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e015d160 con 0x7f42e00cd8e0 2023-03-03T06:32:16.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.988+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e00cd0c0 con 0x7f42e00cd8e0 2023-03-03T06:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.989+0000 7f42d6ffd700 1 -- 172.21.15.102:0/695755341 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f42d0019030 con 0x7f42e00cd8e0 2023-03-03T06:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.990+0000 7f42d6ffd700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c4068320 0x7f42c406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.990+0000 7f42d6ffd700 1 -- 172.21.15.102:0/695755341 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f42d00859f0 con 0x7f42e00cd8e0 2023-03-03T06:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.991+0000 7f42e639b700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c4068320 0x7f42c406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.991+0000 7f42e639b700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c4068320 0x7f42c406a7d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f42dc000f60 tx=0x7f42dc007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:32:16.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:16.995+0000 7f42d6ffd700 1 -- 172.21.15.102:0/695755341 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42d0052100 con 0x7f42e00cd8e0 2023-03-03T06:32:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.227+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42e01e4ac0 con 0x7f42e00cd8e0 2023-03-03T06:32:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.228+0000 7f42d6ffd700 1 -- 172.21.15.102:0/695755341 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f42d0055720 con 0x7f42e00cd8e0 2023-03-03T06:32:17.231 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:32:17.231 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:32:17.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c4068320 msgr2=0x7f42c406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:17.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c4068320 0x7f42c406a7d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f42dc000f60 tx=0x7f42dc007ea0 comp rx=0 tx=0).stop 2023-03-03T06:32:17.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e00cd8e0 msgr2=0x7f42e01f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:17.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e00cd8e0 0x7f42e01f1860 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f42d000e870 tx=0x7f42d000eb80 comp rx=0 tx=0).stop 2023-03-03T06:32:17.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 shutdown_connections 2023-03-03T06:32:17.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c4068320 0x7f42c406a7d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:17.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e0153290 0x7f42e01f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e00cd390 0x7f42e0155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 --2- 172.21.15.102:0/695755341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e00cd8e0 0x7f42e01f1860 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 >> 172.21.15.102:0/695755341 conn(0x7f42e0065fb0 msgr2=0x7f42e0073810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:32:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 shutdown_connections 2023-03-03T06:32:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:17.231+0000 7f42e85ff700 1 -- 172.21.15.102:0/695755341 wait complete. 2023-03-03T06:32:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:32:17.261 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-03-03T06:32:22.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:32:22.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.510+0000 7fa98dd9d700 1 Processor -- start 2023-03-03T06:32:22.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.510+0000 7fa98dd9d700 1 -- start start 2023-03-03T06:32:22.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa98dd9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9880b3970 0x7fa9880c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:22.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa98dd9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9880c41b0 0x7fa9880cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:22.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa98dd9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9880c3d20 con 0x7fa9880ced00 2023-03-03T06:32:22.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa98dd9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9880c4930 con 0x7fa9880b3970 2023-03-03T06:32:22.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa98dd9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9880cf610 con 0x7fa9880c41b0 2023-03-03T06:32:22.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa987fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9880b3970 0x7fa9880c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:22.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa98d59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9880c41b0 0x7fa9880cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:22.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa987fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9880b3970 0x7fa9880c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49870/0 (socket says 172.21.15.102:49870) 2023-03-03T06:32:22.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa987fff700 1 -- 172.21.15.102:0/1845784556 learned_addr learned my addr 172.21.15.102:0/1845784556 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:32:22.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.512+0000 7fa98cd9b700 1 --1- 172.21.15.102:0/1845784556 >> v1:172.21.15.172:6789/0 conn(0x7fa9880ced00 0x7fa988158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41994/0 (socket says 172.21.15.102:41994) 2023-03-03T06:32:22.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa98d59c700 1 -- 172.21.15.102:0/1845784556 >> v2:172.21.15.102:3300/0 conn(0x7fa9880b3970 msgr2=0x7fa9880c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:32:22.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa98d59c700 1 --2- 172.21.15.102:0/1845784556 >> v2:172.21.15.102:3300/0 conn(0x7fa9880b3970 0x7fa9880c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa98d59c700 1 -- 172.21.15.102:0/1845784556 >> v1:172.21.15.172:6789/0 conn(0x7fa9880ced00 legacy=0x7fa988158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:22.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa98d59c700 1 -- 172.21.15.102:0/1845784556 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98815cba0 con 0x7fa9880c41b0 2023-03-03T06:32:22.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa9877fe700 1 -- 172.21.15.102:0/1845784556 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240910852 0 0) 0x7fa9880c3d20 con 0x7fa9880ced00 2023-03-03T06:32:22.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa98d59c700 1 --2- 172.21.15.102:0/1845784556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa9880cf0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa98808ce30 tx=0x7fa98000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65482f56371860fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:32:22.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa9877fe700 1 -- 172.21.15.102:0/1845784556 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98000e070 con 0x7fa9880c41b0 2023-03-03T06:32:22.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa9877fe700 1 -- 172.21.15.102:0/1845784556 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98000bb20 con 0x7fa9880c41b0 2023-03-03T06:32:22.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.513+0000 7fa9877fe700 1 -- 172.21.15.102:0/1845784556 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980012430 con 0x7fa9880c41b0 2023-03-03T06:32:22.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1845784556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 msgr2=0x7fa9880cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:22.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1845784556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa9880cf0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa98808ce30 tx=0x7fa98000b330 comp rx=0 tx=0).stop 2023-03-03T06:32:22.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1845784556 shutdown_connections 2023-03-03T06:32:22.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1845784556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa9880cf0d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1845784556 >> v2:172.21.15.102:3300/0 conn(0x7fa9880b3970 0x7fa9880c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1845784556 >> 172.21.15.102:0/1845784556 conn(0x7fa98808b1c0 msgr2=0x7fa988071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:32:22.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1845784556 shutdown_connections 2023-03-03T06:32:22.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.514+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1845784556 wait complete. 2023-03-03T06:32:22.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.516+0000 7fa98dd9d700 1 Processor -- start 2023-03-03T06:32:22.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.516+0000 7fa98dd9d700 1 -- start start 2023-03-03T06:32:22.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.517+0000 7fa98dd9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9880b3970 0x7fa9881e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:22.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.517+0000 7fa98dd9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa988151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:22.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.517+0000 7fa98dd9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880ced00 0x7fa988151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:22.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.517+0000 7fa98dd9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98815af40 con 0x7fa9880ced00 2023-03-03T06:32:22.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.517+0000 7fa98dd9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98814cda0 con 0x7fa9880c41b0 2023-03-03T06:32:22.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.517+0000 7fa98dd9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98814e5f0 con 0x7fa9880b3970 2023-03-03T06:32:22.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98cd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9880b3970 0x7fa9881e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:22.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98d59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880ced00 0x7fa988151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:22.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa987fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa988151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:22.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa987fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa988151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46182/0 (socket says 172.21.15.102:46182) 2023-03-03T06:32:22.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98cd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9880b3970 0x7fa9881e5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48968/0 (socket says 172.21.15.102:48968) 2023-03-03T06:32:22.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa987fff700 1 -- 172.21.15.102:0/1487417791 learned_addr learned my addr 172.21.15.102:0/1487417791 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:32:22.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98d59c700 1 -- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9880b3970 msgr2=0x7fa9881e5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:22.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98d59c700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9880b3970 0x7fa9881e5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98d59c700 1 -- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 msgr2=0x7fa988151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:22.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98d59c700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa988151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98d59c700 1 -- 172.21.15.102:0/1487417791 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa980009a70 con 0x7fa9880ced00 2023-03-03T06:32:22.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.518+0000 7fa98d59c700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880ced00 0x7fa988151ac0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa980016000 tx=0x7fa980004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:32:22.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.519+0000 7fa9857fa700 1 -- 172.21.15.102:0/1487417791 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98001b810 con 0x7fa9880ced00 2023-03-03T06:32:22.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.519+0000 7fa9857fa700 1 -- 172.21.15.102:0/1487417791 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa980020070 con 0x7fa9880ced00 2023-03-03T06:32:22.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.519+0000 7fa9857fa700 1 -- 172.21.15.102:0/1487417791 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98001bbb0 con 0x7fa9880ced00 2023-03-03T06:32:22.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.519+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98806bf20 con 0x7fa9880ced00 2023-03-03T06:32:22.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.519+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98815ce80 con 0x7fa9880ced00 2023-03-03T06:32:22.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.520+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98806c200 con 0x7fa9880ced00 2023-03-03T06:32:22.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.521+0000 7fa9857fa700 1 -- 172.21.15.102:0/1487417791 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa98001bd10 con 0x7fa9880ced00 2023-03-03T06:32:22.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.523+0000 7fa9857fa700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa96c0682f0 0x7fa96c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:32:22.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.523+0000 7fa9857fa700 1 -- 172.21.15.102:0/1487417791 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa980022070 con 0x7fa9880ced00 2023-03-03T06:32:22.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.523+0000 7fa98cd9b700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa96c0682f0 0x7fa96c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:32:22.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.524+0000 7fa98cd9b700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa96c0682f0 0x7fa96c06a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007fd0 tx=0x7fa97c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:32:22.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.528+0000 7fa9857fa700 1 -- 172.21.15.102:0/1487417791 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa980054a80 con 0x7fa9880ced00 2023-03-03T06:32:22.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.762+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa98815baf0 con 0x7fa9880ced00 2023-03-03T06:32:22.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.762+0000 7fa9857fa700 1 -- 172.21.15.102:0/1487417791 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa98815baf0 con 0x7fa9880ced00 2023-03-03T06:32:22.766 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:32:22.766 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa96c0682f0 msgr2=0x7fa96c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:22.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa96c0682f0 0x7fa96c06a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007fd0 tx=0x7fa97c007de0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880ced00 msgr2=0x7fa988151ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:32:22.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880ced00 0x7fa988151ac0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa980016000 tx=0x7fa980004b20 comp rx=0 tx=0).stop 2023-03-03T06:32:22.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 shutdown_connections 2023-03-03T06:32:22.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa96c0682f0 0x7fa96c06a7a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9880b3970 0x7fa9881e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880ced00 0x7fa988151ac0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 --2- 172.21.15.102:0/1487417791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9880c41b0 0x7fa988151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:32:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 >> 172.21.15.102:0/1487417791 conn(0x7fa98808b1c0 msgr2=0x7fa9880b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:32:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 shutdown_connections 2023-03-03T06:32:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:32:22.765+0000 7fa98dd9d700 1 -- 172.21.15.102:0/1487417791 wait complete. 2023-03-03T06:32:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:32:22.794 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T06:32:44.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:44.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:14.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:14.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:22.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:33:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.051+0000 7f4bc7fff700 1 Processor -- start 2023-03-03T06:33:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.051+0000 7f4bc7fff700 1 -- start start 2023-03-03T06:33:23.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.053+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80b3970 0x7f4bc80c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:33:23.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.053+0000 7f4bc7fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80c3d20 con 0x7f4bc80c41b0 2023-03-03T06:33:23.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.053+0000 7f4bc7fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc815cba0 con 0x7f4bc80ced00 2023-03-03T06:33:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.053+0000 7f4bc7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80c4930 con 0x7f4bc80b3970 2023-03-03T06:33:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.053+0000 7f4bc67fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80b3970 0x7f4bc80c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:33:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.053+0000 7f4bc77fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4bc80c41b0 0x7f4bc8163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41304/0 (socket says 172.21.15.102:41304) 2023-03-03T06:33:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.053+0000 7f4bc77fe700 1 -- 172.21.15.102:0/2634883724 learned_addr learned my addr 172.21.15.102:0/2634883724 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:33:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc5ffb700 1 -- 172.21.15.102:0/2634883724 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679638138 0 0) 0x7f4bc815cba0 con 0x7f4bc80ced00 2023-03-03T06:33:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc67fc700 1 -- 172.21.15.102:0/2634883724 >> v1:172.21.15.172:6789/0 conn(0x7f4bc80ced00 legacy=0x7f4bc8158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:33:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc67fc700 1 -- 172.21.15.102:0/2634883724 >> v1:172.21.15.102:6789/0 conn(0x7f4bc80c41b0 legacy=0x7f4bc8163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:33:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc67fc700 1 -- 172.21.15.102:0/2634883724 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc815ce80 con 0x7f4bc80b3970 2023-03-03T06:33:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc67fc700 1 --2- 172.21.15.102:0/2634883724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80b3970 0x7f4bc80c5a90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009870 tx=0x7f4bbc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41f73a571452ac69 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:33:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc5ffb700 1 -- 172.21.15.102:0/2634883724 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc004230 con 0x7f4bc80b3970 2023-03-03T06:33:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc5ffb700 1 -- 172.21.15.102:0/2634883724 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc004390 con 0x7f4bc80b3970 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.054+0000 7f4bc5ffb700 1 -- 172.21.15.102:0/2634883724 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0036a0 con 0x7f4bc80b3970 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.055+0000 7f4bc7fff700 1 -- 172.21.15.102:0/2634883724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80b3970 msgr2=0x7f4bc80c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.055+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/2634883724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80b3970 0x7f4bc80c5a90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009870 tx=0x7f4bbc009b80 comp rx=0 tx=0).stop 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.055+0000 7f4bc7fff700 1 -- 172.21.15.102:0/2634883724 shutdown_connections 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.055+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/2634883724 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80b3970 0x7f4bc80c5a90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.055+0000 7f4bc7fff700 1 -- 172.21.15.102:0/2634883724 >> 172.21.15.102:0/2634883724 conn(0x7f4bc808b1c0 msgr2=0x7f4bc8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.055+0000 7f4bc7fff700 1 -- 172.21.15.102:0/2634883724 shutdown_connections 2023-03-03T06:33:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.055+0000 7f4bc7fff700 1 -- 172.21.15.102:0/2634883724 wait complete. 2023-03-03T06:33:23.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.058+0000 7f4bc7fff700 1 Processor -- start 2023-03-03T06:33:23.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.058+0000 7f4bc7fff700 1 -- start start 2023-03-03T06:33:23.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.059+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80b3970 0x7f4bc81f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:33:23.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.059+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80c41b0 0x7f4bc814cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:33:23.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80ced00 0x7f4bc8150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:33:23.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc7fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8151a70 con 0x7f4bc80b3970 2023-03-03T06:33:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc815b010 con 0x7f4bc80ced00 2023-03-03T06:33:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80cd740 con 0x7f4bc80c41b0 2023-03-03T06:33:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80b3970 0x7f4bc81f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:33:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80b3970 0x7f4bc81f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60660/0 (socket says 172.21.15.102:60660) 2023-03-03T06:33:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 -- 172.21.15.102:0/3524008849 learned_addr learned my addr 172.21.15.102:0/3524008849 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:33:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc77fe700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80ced00 0x7f4bc8150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:33:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc67fc700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80c41b0 0x7f4bc814cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:33:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 -- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80c41b0 msgr2=0x7f4bc814cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:33:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80c41b0 0x7f4bc814cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:33:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 -- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80ced00 msgr2=0x7f4bc8150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:33:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80ced00 0x7f4bc8150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:33:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc6ffd700 1 -- 172.21.15.102:0/3524008849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0094a0 con 0x7f4bc80b3970 2023-03-03T06:33:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc77fe700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80ced00 0x7f4bc8150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:33:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.060+0000 7f4bc67fc700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80c41b0 0x7f4bc814cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:33:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.061+0000 7f4bc6ffd700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80b3970 0x7f4bc81f23c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800c500 tx=0x7f4bb800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:33:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.061+0000 7f4baf7fe700 1 -- 172.21.15.102:0/3524008849 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80054e0 con 0x7f4bc80b3970 2023-03-03T06:33:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.061+0000 7f4baf7fe700 1 -- 172.21.15.102:0/3524008849 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8003680 con 0x7f4bc80b3970 2023-03-03T06:33:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.061+0000 7f4baf7fe700 1 -- 172.21.15.102:0/3524008849 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8003a30 con 0x7f4bc80b3970 2023-03-03T06:33:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.061+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc806c200 con 0x7f4bc80b3970 2023-03-03T06:33:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.061+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc806bf20 con 0x7f4bc80b3970 2023-03-03T06:33:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.062+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc815c890 con 0x7f4bc80b3970 2023-03-03T06:33:23.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.063+0000 7f4baf7fe700 1 -- 172.21.15.102:0/3524008849 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4bb8019030 con 0x7f4bc80b3970 2023-03-03T06:33:23.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.065+0000 7f4baf7fe700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bb40682f0 0x7f4bb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:33:23.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.065+0000 7f4baf7fe700 1 -- 172.21.15.102:0/3524008849 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4bb8085150 con 0x7f4bc80b3970 2023-03-03T06:33:23.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.065+0000 7f4bc67fc700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bb40682f0 0x7f4bb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:33:23.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.066+0000 7f4bc67fc700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bb40682f0 0x7f4bb406a7a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0097c0 tx=0x7f4bbc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:33:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.069+0000 7f4baf7fe700 1 -- 172.21.15.102:0/3524008849 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4bb8051750 con 0x7f4bc80b3970 2023-03-03T06:33:23.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.300+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc804e6e0 con 0x7f4bc80b3970 2023-03-03T06:33:23.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.301+0000 7f4baf7fe700 1 -- 172.21.15.102:0/3524008849 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4bb8017020 con 0x7f4bc80b3970 2023-03-03T06:33:23.303 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:33:23.303 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:33:23.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bb40682f0 msgr2=0x7f4bb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:33:23.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bb40682f0 0x7f4bb406a7a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0097c0 tx=0x7f4bbc008030 comp rx=0 tx=0).stop 2023-03-03T06:33:23.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80b3970 msgr2=0x7f4bc81f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:33:23.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80b3970 0x7f4bc81f23c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800c500 tx=0x7f4bb800c810 comp rx=0 tx=0).stop 2023-03-03T06:33:23.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 shutdown_connections 2023-03-03T06:33:23.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80ced00 0x7f4bc8150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:33:23.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80b3970 0x7f4bc81f23c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:33:23.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bb40682f0 0x7f4bb406a7a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:33:23.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 --2- 172.21.15.102:0/3524008849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80c41b0 0x7f4bc814cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:33:23.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 >> 172.21.15.102:0/3524008849 conn(0x7f4bc808b1c0 msgr2=0x7f4bc806b190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:33:23.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.303+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 shutdown_connections 2023-03-03T06:33:23.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:33:23.304+0000 7f4bc7fff700 1 -- 172.21.15.102:0/3524008849 wait complete. 2023-03-03T06:33:23.307 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:33:23.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2023-03-03T06:33:44.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:44.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:14.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:14.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:45.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:45.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:08.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:35:08.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.575+0000 7f65519ff700 1 Processor -- start 2023-03-03T06:35:08.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.575+0000 7f65519ff700 1 -- start start 2023-03-03T06:35:08.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f65519ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f654c0cd8e0 0x7f654c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:08.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f65519ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c153660 con 0x7f654c153290 2023-03-03T06:35:08.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f65519ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c06c360 con 0x7f654c0cd390 2023-03-03T06:35:08.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f65519ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654c15cea0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f654affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f654c153290 0x7f654c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36394/0 (socket says 172.21.15.102:36394) 2023-03-03T06:35:08.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f654affd700 1 -- 172.21.15.102:0/744190802 learned_addr learned my addr 172.21.15.102:0/744190802 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f654a7fc700 1 --2- 172.21.15.102:0/744190802 >> v2:172.21.15.172:3300/0 conn(0x7f654c0cd8e0 0x7f654c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.577+0000 7f6549ffb700 1 -- 172.21.15.102:0/744190802 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952935790 0 0) 0x7f654c06c360 con 0x7f654c0cd390 2023-03-03T06:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f6549ffb700 1 -- 172.21.15.102:0/744190802 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65340035b0 con 0x7f654c0cd390 2023-03-03T06:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f6549ffb700 1 -- 172.21.15.102:0/744190802 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406939113 0 0) 0x7f654c153660 con 0x7f654c153290 2023-03-03T06:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f654a7fc700 1 -- 172.21.15.102:0/744190802 >> v1:172.21.15.172:6789/0 conn(0x7f654c0cd390 legacy=0x7f654c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f654a7fc700 1 -- 172.21.15.102:0/744190802 >> v1:172.21.15.102:6789/0 conn(0x7f654c153290 legacy=0x7f654c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f654a7fc700 1 -- 172.21.15.102:0/744190802 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c06c7f0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f654a7fc700 1 --2- 172.21.15.102:0/744190802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c0cde20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6540009840 tx=0x7f6540009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d1b5ca66a5361bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f6549ffb700 1 -- 172.21.15.102:0/744190802 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540011830 con 0x7f654c0cd8e0 2023-03-03T06:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f6549ffb700 1 -- 172.21.15.102:0/744190802 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6540011990 con 0x7f654c0cd8e0 2023-03-03T06:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.578+0000 7f6549ffb700 1 -- 172.21.15.102:0/744190802 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540011d20 con 0x7f654c0cd8e0 2023-03-03T06:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.579+0000 7f65519ff700 1 -- 172.21.15.102:0/744190802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 msgr2=0x7f654c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.579+0000 7f65519ff700 1 --2- 172.21.15.102:0/744190802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c0cde20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6540009840 tx=0x7f6540009b50 comp rx=0 tx=0).stop 2023-03-03T06:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.579+0000 7f65519ff700 1 -- 172.21.15.102:0/744190802 shutdown_connections 2023-03-03T06:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.579+0000 7f65519ff700 1 --2- 172.21.15.102:0/744190802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c0cde20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.579+0000 7f65519ff700 1 -- 172.21.15.102:0/744190802 >> 172.21.15.102:0/744190802 conn(0x7f654c065fb0 msgr2=0x7f654c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.579+0000 7f65519ff700 1 -- 172.21.15.102:0/744190802 shutdown_connections 2023-03-03T06:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.579+0000 7f65519ff700 1 -- 172.21.15.102:0/744190802 wait complete. 2023-03-03T06:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.581+0000 7f65519ff700 1 Processor -- start 2023-03-03T06:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.581+0000 7f65519ff700 1 -- start start 2023-03-03T06:35:08.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f65519ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f654c0cd390 0x7f654c1f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:08.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f65519ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c1590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:08.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f65519ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f654c153290 0x7f654c0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:08.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f65519ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c0c8bb0 con 0x7f654c153290 2023-03-03T06:35:08.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f65519ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c0c8e20 con 0x7f654c0cd8e0 2023-03-03T06:35:08.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f65519ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654c0c6510 con 0x7f654c0cd390 2023-03-03T06:35:08.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f654b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f654c153290 0x7f654c0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:08.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f654a7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c1590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f654affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f654c0cd390 0x7f654c1f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f654b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f654c153290 0x7f654c0c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47964/0 (socket says 172.21.15.102:47964) 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f654a7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c1590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37450/0 (socket says 172.21.15.102:37450) 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.582+0000 7f654b7fe700 1 -- 172.21.15.102:0/4121756167 learned_addr learned my addr 172.21.15.102:0/4121756167 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.583+0000 7f654a7fc700 1 -- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f654c0cd390 msgr2=0x7f654c1f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.583+0000 7f654a7fc700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f654c0cd390 0x7f654c1f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.583+0000 7f654a7fc700 1 -- 172.21.15.102:0/4121756167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f654c153290 msgr2=0x7f654c0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.583+0000 7f654a7fc700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f654c153290 0x7f654c0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.583+0000 7f654a7fc700 1 -- 172.21.15.102:0/4121756167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65400094a0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.583+0000 7f654b7fe700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f654c153290 0x7f654c0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.583+0000 7f654a7fc700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c1590f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6540016000 tx=0x7f6540004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.584+0000 7f6533fff700 1 -- 172.21.15.102:0/4121756167 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654001a7e0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.584+0000 7f6533fff700 1 -- 172.21.15.102:0/4121756167 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654001a960 con 0x7f654c0cd8e0 2023-03-03T06:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.584+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654c0ccde0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.584+0000 7f6533fff700 1 -- 172.21.15.102:0/4121756167 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654001ad10 con 0x7f654c0cd8e0 2023-03-03T06:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.584+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654c15d160 con 0x7f654c0cd8e0 2023-03-03T06:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.585+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654c0cd0c0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.586+0000 7f6533fff700 1 -- 172.21.15.102:0/4121756167 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f654002b030 con 0x7f654c0cd8e0 2023-03-03T06:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.587+0000 7f6533fff700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65380682f0 0x7f653806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.588+0000 7f6533fff700 1 -- 172.21.15.102:0/4121756167 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f65400871e0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.588+0000 7f654affd700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65380682f0 0x7f653806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.588+0000 7f654affd700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65380682f0 0x7f653806a7a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f653c000f60 tx=0x7f653c004a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:35:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.590+0000 7f6533fff700 1 -- 172.21.15.102:0/4121756167 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6540053970 con 0x7f654c0cd8e0 2023-03-03T06:35:08.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.826+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f654c04e6e0 con 0x7f654c0cd8e0 2023-03-03T06:35:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.826+0000 7f6533fff700 1 -- 172.21.15.102:0/4121756167 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6540056f90 con 0x7f654c0cd8e0 2023-03-03T06:35:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:35:08.828 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:35:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65380682f0 msgr2=0x7f653806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65380682f0 0x7f653806a7a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f653c000f60 tx=0x7f653c004a50 comp rx=0 tx=0).stop 2023-03-03T06:35:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 msgr2=0x7f654c1590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c1590f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6540016000 tx=0x7f6540004750 comp rx=0 tx=0).stop 2023-03-03T06:35:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 shutdown_connections 2023-03-03T06:35:08.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65380682f0 0x7f653806a7a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:08.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f654c0cd390 0x7f654c1f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:08.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f654c153290 0x7f654c0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:08.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 --2- 172.21.15.102:0/4121756167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c0cd8e0 0x7f654c1590f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:08.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 >> 172.21.15.102:0/4121756167 conn(0x7f654c065fb0 msgr2=0x7f654c158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:35:08.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.829+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 shutdown_connections 2023-03-03T06:35:08.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:08.830+0000 7f65519ff700 1 -- 172.21.15.102:0/4121756167 wait complete. 2023-03-03T06:35:08.833 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:35:08.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-03-03T06:35:15.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:15.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:37.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:35:38.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.110+0000 7f2fc43ff700 1 Processor -- start 2023-03-03T06:35:38.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.110+0000 7f2fc43ff700 1 -- start start 2023-03-03T06:35:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.111+0000 7f2fc43ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fbc153290 0x7f2fbc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.111+0000 7f2fc43ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2fbc0cd8e0 0x7f2fbc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.111+0000 7f2fc43ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fbc153660 con 0x7f2fbc1593c0 2023-03-03T06:35:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.111+0000 7f2fc43ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fbc06c360 con 0x7f2fbc153290 2023-03-03T06:35:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.111+0000 7f2fc43ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fbc15cea0 con 0x7f2fbc0cd8e0 2023-03-03T06:35:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc219b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fbc153290 0x7f2fbc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc199a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2fbc1593c0 0x7f2fbc0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32900/0 (socket says 172.21.15.102:32900) 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc199a700 1 -- 172.21.15.102:0/3988489871 learned_addr learned my addr 172.21.15.102:0/3988489871 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc299c700 1 --2- 172.21.15.102:0/3988489871 >> v2:172.21.15.172:3300/0 conn(0x7f2fbc0cd8e0 0x7f2fbc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc219b700 1 -- 172.21.15.102:0/3988489871 >> v2:172.21.15.172:3300/0 conn(0x7f2fbc0cd8e0 msgr2=0x7f2fbc0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc219b700 1 --2- 172.21.15.102:0/3988489871 >> v2:172.21.15.172:3300/0 conn(0x7f2fbc0cd8e0 0x7f2fbc0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc219b700 1 -- 172.21.15.102:0/3988489871 >> v1:172.21.15.102:6789/0 conn(0x7f2fbc1593c0 legacy=0x7f2fbc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.112+0000 7f2fc219b700 1 -- 172.21.15.102:0/3988489871 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc06c7f0 con 0x7f2fbc153290 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.113+0000 7f2fc219b700 1 --2- 172.21.15.102:0/3988489871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc153290 0x7f2fbc158e80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0c9390 tx=0x7f2fb8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be93217060aeffd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:35:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.113+0000 7f2fc1199700 1 -- 172.21.15.102:0/3988489871 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8011830 con 0x7f2fbc153290 2023-03-03T06:35:38.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.113+0000 7f2fc1199700 1 -- 172.21.15.102:0/3988489871 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8011990 con 0x7f2fbc153290 2023-03-03T06:35:38.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.113+0000 7f2fc1199700 1 -- 172.21.15.102:0/3988489871 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8011d20 con 0x7f2fbc153290 2023-03-03T06:35:38.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.113+0000 7f2fc43ff700 1 -- 172.21.15.102:0/3988489871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc153290 msgr2=0x7f2fbc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.113+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/3988489871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc153290 0x7f2fbc158e80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0c9390 tx=0x7f2fb8009b20 comp rx=0 tx=0).stop 2023-03-03T06:35:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.114+0000 7f2fc43ff700 1 -- 172.21.15.102:0/3988489871 shutdown_connections 2023-03-03T06:35:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.114+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/3988489871 >> v2:172.21.15.172:3300/0 conn(0x7f2fbc0cd8e0 0x7f2fbc0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.114+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/3988489871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc153290 0x7f2fbc158e80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.114+0000 7f2fc43ff700 1 -- 172.21.15.102:0/3988489871 >> 172.21.15.102:0/3988489871 conn(0x7f2fbc065fb0 msgr2=0x7f2fbc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:35:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.114+0000 7f2fc43ff700 1 -- 172.21.15.102:0/3988489871 shutdown_connections 2023-03-03T06:35:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.114+0000 7f2fc43ff700 1 -- 172.21.15.102:0/3988489871 wait complete. 2023-03-03T06:35:38.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.116+0000 7f2fc43ff700 1 Processor -- start 2023-03-03T06:35:38.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.116+0000 7f2fc43ff700 1 -- start start 2023-03-03T06:35:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc43ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fbc0cd8e0 0x7f2fbc156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc43ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fbc153290 0x7f2fbc0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc43ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc1593c0 0x7f2fbc0c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc43ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc15b900 con 0x7f2fbc1593c0 2023-03-03T06:35:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc43ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc156a80 con 0x7f2fbc153290 2023-03-03T06:35:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc43ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc153990 con 0x7f2fbc0cd8e0 2023-03-03T06:35:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc299c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc1593c0 0x7f2fbc0c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc199a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fbc153290 0x7f2fbc0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc219b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fbc0cd8e0 0x7f2fbc156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc199a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fbc153290 0x7f2fbc0c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50184/0 (socket says 172.21.15.102:50184) 2023-03-03T06:35:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.117+0000 7f2fc199a700 1 -- 172.21.15.102:0/2155512953 learned_addr learned my addr 172.21.15.102:0/2155512953 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:35:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc199a700 1 -- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fbc0cd8e0 msgr2=0x7f2fbc156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc199a700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fbc0cd8e0 0x7f2fbc156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc199a700 1 -- 172.21.15.102:0/2155512953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc1593c0 msgr2=0x7f2fbc0c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc199a700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc1593c0 0x7f2fbc0c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc199a700 1 -- 172.21.15.102:0/2155512953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80094a0 con 0x7f2fbc153290 2023-03-03T06:35:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc299c700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc1593c0 0x7f2fbc0c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:35:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc199a700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fbc153290 0x7f2fbc0c63b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00c550 tx=0x7f2fac00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:35:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2155512953 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac005d70 con 0x7f2fbc153290 2023-03-03T06:35:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2155512953 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac0070d0 con 0x7f2fbc153290 2023-03-03T06:35:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc0ccde0 con 0x7f2fbc153290 2023-03-03T06:35:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2155512953 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac007440 con 0x7f2fbc153290 2023-03-03T06:35:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.118+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fbc15d160 con 0x7f2fbc153290 2023-03-03T06:35:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.120+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fbc0cd0c0 con 0x7f2fbc153290 2023-03-03T06:35:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.121+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2155512953 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2fac019030 con 0x7f2fbc153290 2023-03-03T06:35:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.122+0000 7f2fb2ffd700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2fa00682f0 0x7f2fa006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:35:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.122+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2155512953 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2fac084fc0 con 0x7f2fbc153290 2023-03-03T06:35:38.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.122+0000 7f2fc219b700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2fa00682f0 0x7f2fa006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:35:38.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.123+0000 7f2fc219b700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2fa00682f0 0x7f2fa006a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80090b0 tx=0x7f2fb8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:35:38.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.126+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2155512953 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2fac04b5e0 con 0x7f2fbc153290 2023-03-03T06:35:38.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.360+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fbc04e6e0 con 0x7f2fbc153290 2023-03-03T06:35:38.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.361+0000 7f2fb2ffd700 1 -- 172.21.15.102:0/2155512953 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2fac017020 con 0x7f2fbc153290 2023-03-03T06:35:38.362 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:35:38.363 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:35:38.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2fa00682f0 msgr2=0x7f2fa006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:38.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2fa00682f0 0x7f2fa006a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80090b0 tx=0x7f2fb8008030 comp rx=0 tx=0).stop 2023-03-03T06:35:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fbc153290 msgr2=0x7f2fbc0c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:35:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fbc153290 0x7f2fbc0c63b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00c550 tx=0x7f2fac00c860 comp rx=0 tx=0).stop 2023-03-03T06:35:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 shutdown_connections 2023-03-03T06:35:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2fa00682f0 0x7f2fa006a7a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2fbc0cd8e0 0x7f2fbc156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fbc1593c0 0x7f2fbc0c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 --2- 172.21.15.102:0/2155512953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2fbc153290 0x7f2fbc0c63b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:35:38.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 >> 172.21.15.102:0/2155512953 conn(0x7f2fbc065fb0 msgr2=0x7f2fbc156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:35:38.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 shutdown_connections 2023-03-03T06:35:38.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:35:38.364+0000 7f2fc43ff700 1 -- 172.21.15.102:0/2155512953 wait complete. 2023-03-03T06:35:38.367 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:35:38.393 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2023-03-03T06:35:45.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:45.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:15.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:15.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:33.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:36:33.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.644+0000 7f63111ff700 1 Processor -- start 2023-03-03T06:36:33.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.644+0000 7f63111ff700 1 -- start start 2023-03-03T06:36:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.645+0000 7f63111ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f630c153260 0x7f630c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:36:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.645+0000 7f63111ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0c9cb0 0x7f630c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:36:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.645+0000 7f63111ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f630c0cac20 0x7f630c0cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:36:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.645+0000 7f63111ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c0cfe70 con 0x7f630c153260 2023-03-03T06:36:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.645+0000 7f63111ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c158d70 con 0x7f630c0cac20 2023-03-03T06:36:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.645+0000 7f63111ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c0cf940 con 0x7f630c0c9cb0 2023-03-03T06:36:33.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630ad9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f630c153260 0x7f630c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:36:33.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0c9cb0 0x7f630c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:36:33.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630ad9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f630c153260 0x7f630c0ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38216/0 (socket says 172.21.15.102:38216) 2023-03-03T06:36:33.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630ad9d700 1 -- 172.21.15.102:0/250365472 learned_addr learned my addr 172.21.15.102:0/250365472 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:36:33.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630b59e700 1 --2- 172.21.15.102:0/250365472 >> v2:172.21.15.172:3300/0 conn(0x7f630c0cac20 0x7f630c0cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:36:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630a59c700 1 -- 172.21.15.102:0/250365472 >> v2:172.21.15.172:3300/0 conn(0x7f630c0cac20 msgr2=0x7f630c0cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:36:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630a59c700 1 --2- 172.21.15.102:0/250365472 >> v2:172.21.15.172:3300/0 conn(0x7f630c0cac20 0x7f630c0cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630a59c700 1 -- 172.21.15.102:0/250365472 >> v2:172.21.15.102:3300/0 conn(0x7f630c153260 msgr2=0x7f630c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:36:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630a59c700 1 --2- 172.21.15.102:0/250365472 >> v2:172.21.15.102:3300/0 conn(0x7f630c153260 0x7f630c0ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630a59c700 1 -- 172.21.15.102:0/250365472 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c0cfb90 con 0x7f630c0c9cb0 2023-03-03T06:36:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630ad9d700 1 --2- 172.21.15.102:0/250365472 >> v2:172.21.15.102:3300/0 conn(0x7f630c153260 0x7f630c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:36:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.646+0000 7f630a59c700 1 --2- 172.21.15.102:0/250365472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0c9cb0 0x7f630c0cb880 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f630c06ad60 tx=0x7f6300009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c2457a11b8f218 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f6309d9b700 1 -- 172.21.15.102:0/250365472 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300011830 con 0x7f630c0c9cb0 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f6309d9b700 1 -- 172.21.15.102:0/250365472 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6300011990 con 0x7f630c0c9cb0 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f6309d9b700 1 -- 172.21.15.102:0/250365472 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300011d20 con 0x7f630c0c9cb0 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 -- 172.21.15.102:0/250365472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0c9cb0 msgr2=0x7f630c0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 --2- 172.21.15.102:0/250365472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0c9cb0 0x7f630c0cb880 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f630c06ad60 tx=0x7f6300009b20 comp rx=0 tx=0).stop 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 -- 172.21.15.102:0/250365472 shutdown_connections 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 --2- 172.21.15.102:0/250365472 >> v2:172.21.15.172:3300/0 conn(0x7f630c0cac20 0x7f630c0cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 --2- 172.21.15.102:0/250365472 >> v2:172.21.15.102:3300/0 conn(0x7f630c153260 0x7f630c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 --2- 172.21.15.102:0/250365472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0c9cb0 0x7f630c0cb880 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 -- 172.21.15.102:0/250365472 >> 172.21.15.102:0/250365472 conn(0x7f630c066c00 msgr2=0x7f630c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:36:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.647+0000 7f63111ff700 1 -- 172.21.15.102:0/250365472 shutdown_connections 2023-03-03T06:36:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.648+0000 7f63111ff700 1 -- 172.21.15.102:0/250365472 wait complete. 2023-03-03T06:36:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.650+0000 7f63111ff700 1 Processor -- start 2023-03-03T06:36:33.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.651+0000 7f63111ff700 1 -- start start 2023-03-03T06:36:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f63111ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0cac20 0x7f630c0cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:36:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f63111ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f630c153260 0x7f630c0c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:36:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f63111ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f630c0c6df0 0x7f630c0c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f63111ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c155730 con 0x7f630c153260 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f63111ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c1548d0 con 0x7f630c0c6df0 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f63111ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c154bb0 con 0x7f630c0cac20 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f630a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f630c153260 0x7f630c0c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f630a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f630c153260 0x7f630c0c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38222/0 (socket says 172.21.15.102:38222) 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f630a59c700 1 -- 172.21.15.102:0/1199578065 learned_addr learned my addr 172.21.15.102:0/1199578065 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f630ad9d700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0cac20 0x7f630c0cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:36:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.652+0000 7f630b59e700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f630c0c6df0 0x7f630c0c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:36:33.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f630a59c700 1 -- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0cac20 msgr2=0x7f630c0cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:36:33.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f630a59c700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0cac20 0x7f630c0cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f630a59c700 1 -- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f630c0c6df0 msgr2=0x7f630c0c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:36:33.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f630a59c700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f630c0c6df0 0x7f630c0c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f630a59c700 1 -- 172.21.15.102:0/1199578065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc009550 con 0x7f630c153260 2023-03-03T06:36:33.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f630a59c700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f630c153260 0x7f630c0c68b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6300000c00 tx=0x7f6300004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:36:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f62f37fe700 1 -- 172.21.15.102:0/1199578065 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300004510 con 0x7f630c153260 2023-03-03T06:36:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f62f37fe700 1 -- 172.21.15.102:0/1199578065 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630001e070 con 0x7f630c153260 2023-03-03T06:36:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f62f37fe700 1 -- 172.21.15.102:0/1199578065 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63000038e0 con 0x7f630c153260 2023-03-03T06:36:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.653+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63000094a0 con 0x7f630c153260 2023-03-03T06:36:33.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.654+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c159260 con 0x7f630c153260 2023-03-03T06:36:33.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.655+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c158f80 con 0x7f630c153260 2023-03-03T06:36:33.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.656+0000 7f62f37fe700 1 -- 172.21.15.102:0/1199578065 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f630000f050 con 0x7f630c153260 2023-03-03T06:36:33.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.658+0000 7f62f37fe700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62f80682f0 0x7f62f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:36:33.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.658+0000 7f62f37fe700 1 -- 172.21.15.102:0/1199578065 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6300086a60 con 0x7f630c153260 2023-03-03T06:36:33.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.658+0000 7f630ad9d700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62f80682f0 0x7f62f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:36:33.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.659+0000 7f630ad9d700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62f80682f0 0x7f62f806a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006d30 tx=0x7f62fc0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:36:33.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.662+0000 7f62f37fe700 1 -- 172.21.15.102:0/1199578065 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6300053170 con 0x7f630c153260 2023-03-03T06:36:33.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.901+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f630c159540 con 0x7f630c153260 2023-03-03T06:36:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.902+0000 7f62f37fe700 1 -- 172.21.15.102:0/1199578065 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6300020090 con 0x7f630c153260 2023-03-03T06:36:33.903 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:36:33.904 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:36:33.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62f80682f0 msgr2=0x7f62f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:36:33.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62f80682f0 0x7f62f806a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006d30 tx=0x7f62fc0092c0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f630c153260 msgr2=0x7f630c0c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:36:33.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f630c153260 0x7f630c0c68b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6300000c00 tx=0x7f6300004750 comp rx=0 tx=0).stop 2023-03-03T06:36:33.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 shutdown_connections 2023-03-03T06:36:33.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62f80682f0 0x7f62f806a7a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f630c0cac20 0x7f630c0cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f630c0c6df0 0x7f630c0c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 --2- 172.21.15.102:0/1199578065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f630c153260 0x7f630c0c68b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:36:33.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 >> 172.21.15.102:0/1199578065 conn(0x7f630c0c9cb0 msgr2=0x7f630c150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:36:33.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 shutdown_connections 2023-03-03T06:36:33.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:36:33.904+0000 7f63111ff700 1 -- 172.21.15.102:0/1199578065 wait complete. 2023-03-03T06:36:33.907 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:36:33.932 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T06:36:45.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:45.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:15.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:16.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:46.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:46.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:16.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:16.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:27.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:38:28.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.187+0000 7f89bc7ff700 1 Processor -- start 2023-03-03T06:38:28.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.187+0000 7f89bc7ff700 1 -- start start 2023-03-03T06:38:28.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.188+0000 7f89bc7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b4153290 0x7f89b4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:28.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.188+0000 7f89bc7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f89b40cd8e0 0x7f89b40ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:28.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.188+0000 7f89bc7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b4153660 con 0x7f89b41593c0 2023-03-03T06:38:28.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.188+0000 7f89bc7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b406c360 con 0x7f89b40cd8e0 2023-03-03T06:38:28.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.188+0000 7f89bc7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b415cea0 con 0x7f89b4153290 2023-03-03T06:38:28.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89bad9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f89b40cd8e0 0x7f89b40ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:28.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89ba59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b4153290 0x7f89b4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:28.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89b9d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f89b41593c0 0x7f89b40cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57144/0 (socket says 172.21.15.102:57144) 2023-03-03T06:38:28.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89b9d9a700 1 -- 172.21.15.102:0/399397142 learned_addr learned my addr 172.21.15.102:0/399397142 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:38:28.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89bad9c700 1 -- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b4153290 msgr2=0x7f89b4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:38:28.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89bad9c700 1 --2- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b4153290 0x7f89b4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89bad9c700 1 -- 172.21.15.102:0/399397142 >> v1:172.21.15.172:6789/0 conn(0x7f89b41593c0 legacy=0x7f89b40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:28.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89bad9c700 1 -- 172.21.15.102:0/399397142 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b406c7f0 con 0x7f89b40cd8e0 2023-03-03T06:38:28.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89ba59b700 1 --2- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b4153290 0x7f89b4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:38:28.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.189+0000 7f89bad9c700 1 --2- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b40cd8e0 0x7f89b40ce530 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009d90 tx=0x7f89ac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e326d0f88631741 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:38:28.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89b9599700 1 -- 172.21.15.102:0/399397142 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac00e070 con 0x7f89b40cd8e0 2023-03-03T06:38:28.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89b9599700 1 -- 172.21.15.102:0/399397142 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ac00bb20 con 0x7f89b40cd8e0 2023-03-03T06:38:28.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89b9599700 1 -- 172.21.15.102:0/399397142 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac012430 con 0x7f89b40cd8e0 2023-03-03T06:38:28.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 -- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b40cd8e0 msgr2=0x7f89b40ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:28.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b40cd8e0 0x7f89b40ce530 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009d90 tx=0x7f89ac00b330 comp rx=0 tx=0).stop 2023-03-03T06:38:28.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 -- 172.21.15.102:0/399397142 shutdown_connections 2023-03-03T06:38:28.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b40cd8e0 0x7f89b40ce530 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/399397142 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b4153290 0x7f89b4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 -- 172.21.15.102:0/399397142 >> 172.21.15.102:0/399397142 conn(0x7f89b4065fb0 msgr2=0x7f89b40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:38:28.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 -- 172.21.15.102:0/399397142 shutdown_connections 2023-03-03T06:38:28.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.190+0000 7f89bc7ff700 1 -- 172.21.15.102:0/399397142 wait complete. 2023-03-03T06:38:28.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.192+0000 7f89bc7ff700 1 Processor -- start 2023-03-03T06:38:28.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.192+0000 7f89bc7ff700 1 -- start start 2023-03-03T06:38:28.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.193+0000 7f89bc7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b40cd8e0 0x7f89b41e3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.193+0000 7f89bc7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89b4153290 0x7f89b40b4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.193+0000 7f89bc7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b41593c0 0x7f89b4151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.193+0000 7f89bc7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b41e4440 con 0x7f89b4153290 2023-03-03T06:38:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.193+0000 7f89bc7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b41eeb40 con 0x7f89b41593c0 2023-03-03T06:38:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.193+0000 7f89bc7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b41e31d0 con 0x7f89b40cd8e0 2023-03-03T06:38:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89b9d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89b4153290 0x7f89b40b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b40cd8e0 0x7f89b41e3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89bad9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b41593c0 0x7f89b4151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b40cd8e0 0x7f89b41e3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44254/0 (socket says 172.21.15.102:44254) 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 -- 172.21.15.102:0/3550810087 learned_addr learned my addr 172.21.15.102:0/3550810087 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 -- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b41593c0 msgr2=0x7f89b4151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b41593c0 0x7f89b4151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 -- 172.21.15.102:0/3550810087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89b4153290 msgr2=0x7f89b40b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89b4153290 0x7f89b40b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89ba59b700 1 -- 172.21.15.102:0/3550810087 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b00094a0 con 0x7f89b40cd8e0 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89b9d9a700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89b4153290 0x7f89b40b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.194+0000 7f89bad9c700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b41593c0 0x7f89b4151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.195+0000 7f89ba59b700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b40cd8e0 0x7f89b41e3f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f89b000ebe0 tx=0x7f89b000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.195+0000 7f89aaffd700 1 -- 172.21.15.102:0/3550810087 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0009760 con 0x7f89b40cd8e0 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.195+0000 7f89aaffd700 1 -- 172.21.15.102:0/3550810087 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b000c850 con 0x7f89b40cd8e0 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.195+0000 7f89aaffd700 1 -- 172.21.15.102:0/3550810087 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b000cbe0 con 0x7f89b40cd8e0 2023-03-03T06:38:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.195+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac009a70 con 0x7f89b40cd8e0 2023-03-03T06:38:28.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.195+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b40ccde0 con 0x7f89b40cd8e0 2023-03-03T06:38:28.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.196+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b415d160 con 0x7f89b40cd8e0 2023-03-03T06:38:28.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.197+0000 7f89aaffd700 1 -- 172.21.15.102:0/3550810087 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f89b0019030 con 0x7f89b40cd8e0 2023-03-03T06:38:28.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.198+0000 7f89aaffd700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89980682f0 0x7f899806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:28.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.199+0000 7f89aaffd700 1 -- 172.21.15.102:0/3550810087 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f89b0085c40 con 0x7f89b40cd8e0 2023-03-03T06:38:28.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.199+0000 7f89b9d9a700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89980682f0 0x7f899806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:28.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.199+0000 7f89b9d9a700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89980682f0 0x7f899806a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f89b41e6030 tx=0x7f89a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:38:28.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.203+0000 7f89aaffd700 1 -- 172.21.15.102:0/3550810087 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f89b0052350 con 0x7f89b40cd8e0 2023-03-03T06:38:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.436+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89b40cd0c0 con 0x7f89b40cd8e0 2023-03-03T06:38:28.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.437+0000 7f89aaffd700 1 -- 172.21.15.102:0/3550810087 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f89b0055970 con 0x7f89b40cd8e0 2023-03-03T06:38:28.439 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:38:28.440 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:38:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89980682f0 msgr2=0x7f899806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89980682f0 0x7f899806a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f89b41e6030 tx=0x7f89a4009040 comp rx=0 tx=0).stop 2023-03-03T06:38:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b40cd8e0 msgr2=0x7f89b41e3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b40cd8e0 0x7f89b41e3f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f89b000ebe0 tx=0x7f89b000eef0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 shutdown_connections 2023-03-03T06:38:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89980682f0 0x7f899806a7a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89b40cd8e0 0x7f89b41e3f00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89b41593c0 0x7f89b4151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 --2- 172.21.15.102:0/3550810087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89b4153290 0x7f89b40b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:28.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 >> 172.21.15.102:0/3550810087 conn(0x7f89b4065fb0 msgr2=0x7f89b415bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:38:28.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 shutdown_connections 2023-03-03T06:38:28.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:28.440+0000 7f89bc7ff700 1 -- 172.21.15.102:0/3550810087 wait complete. 2023-03-03T06:38:28.444 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:38:28.469 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2023-03-03T06:38:46.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:46.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:55.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:38:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.717+0000 7efe3f7ff700 1 Processor -- start 2023-03-03T06:38:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.718+0000 7efe3f7ff700 1 -- start start 2023-03-03T06:38:55.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.719+0000 7efe3f7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:55.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.719+0000 7efe3f7ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe38153660 con 0x7efe38153290 2023-03-03T06:38:55.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.719+0000 7efe3f7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3806c360 con 0x7efe380cd8e0 2023-03-03T06:38:55.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.719+0000 7efe3f7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3815cea0 con 0x7efe380cd390 2023-03-03T06:38:55.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.719+0000 7efe3d59b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efe38153290 0x7efe38158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51732/0 (socket says 172.21.15.102:51732) 2023-03-03T06:38:55.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.719+0000 7efe3d59b700 1 -- 172.21.15.102:0/3655149709 learned_addr learned my addr 172.21.15.102:0/3655149709 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:38:55.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.719+0000 7efe3dd9c700 1 --2- 172.21.15.102:0/3655149709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe2ffff700 1 -- 172.21.15.102:0/3655149709 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505119342 0 0) 0x7efe3806c360 con 0x7efe380cd8e0 2023-03-03T06:38:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe2ffff700 1 -- 172.21.15.102:0/3655149709 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe240035b0 con 0x7efe380cd8e0 2023-03-03T06:38:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe2ffff700 1 -- 172.21.15.102:0/3655149709 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028630329 0 0) 0x7efe38153660 con 0x7efe38153290 2023-03-03T06:38:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe3dd9c700 1 -- 172.21.15.102:0/3655149709 >> v1:172.21.15.172:6789/0 conn(0x7efe380cd8e0 legacy=0x7efe380cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe3dd9c700 1 -- 172.21.15.102:0/3655149709 >> v1:172.21.15.102:6789/0 conn(0x7efe38153290 legacy=0x7efe38158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe3dd9c700 1 -- 172.21.15.102:0/3655149709 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3806c7f0 con 0x7efe380cd390 2023-03-03T06:38:55.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe3dd9c700 1 --2- 172.21.15.102:0/3655149709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380ce530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efe30009dc0 tx=0x7efe3000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab3687cbe54361c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:38:55.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe2ffff700 1 -- 172.21.15.102:0/3655149709 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3000e070 con 0x7efe380cd390 2023-03-03T06:38:55.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe2ffff700 1 -- 172.21.15.102:0/3655149709 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3000bb20 con 0x7efe380cd390 2023-03-03T06:38:55.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.720+0000 7efe2ffff700 1 -- 172.21.15.102:0/3655149709 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3000bed0 con 0x7efe380cd390 2023-03-03T06:38:55.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.721+0000 7efe3f7ff700 1 -- 172.21.15.102:0/3655149709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 msgr2=0x7efe380ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:55.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.721+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/3655149709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380ce530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efe30009dc0 tx=0x7efe3000b330 comp rx=0 tx=0).stop 2023-03-03T06:38:55.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.721+0000 7efe3f7ff700 1 -- 172.21.15.102:0/3655149709 shutdown_connections 2023-03-03T06:38:55.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.721+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/3655149709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380ce530 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.721+0000 7efe3f7ff700 1 -- 172.21.15.102:0/3655149709 >> 172.21.15.102:0/3655149709 conn(0x7efe38065fb0 msgr2=0x7efe380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:38:55.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.721+0000 7efe3f7ff700 1 -- 172.21.15.102:0/3655149709 shutdown_connections 2023-03-03T06:38:55.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.721+0000 7efe3f7ff700 1 -- 172.21.15.102:0/3655149709 wait complete. 2023-03-03T06:38:55.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.723+0000 7efe3f7ff700 1 Processor -- start 2023-03-03T06:38:55.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.723+0000 7efe3f7ff700 1 -- start start 2023-03-03T06:38:55.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.724+0000 7efe3f7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:55.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.724+0000 7efe3f7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe380cd8e0 0x7efe380b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:55.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.724+0000 7efe3f7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe38153290 0x7efe381f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:55.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.724+0000 7efe3f7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38158890 con 0x7efe380cd8e0 2023-03-03T06:38:55.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.724+0000 7efe3f7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe381e42d0 con 0x7efe38153290 2023-03-03T06:38:55.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.724+0000 7efe3f7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe380b30c0 con 0x7efe380cd390 2023-03-03T06:38:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3cd9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe380cd8e0 0x7efe380b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:55.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3dd9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe38153290 0x7efe381f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:55.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3d59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:55.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3dd9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe38153290 0x7efe381f2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50442/0 (socket says 172.21.15.102:50442) 2023-03-03T06:38:55.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3dd9c700 1 -- 172.21.15.102:0/2734199754 learned_addr learned my addr 172.21.15.102:0/2734199754 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:38:55.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3cd9a700 1 -- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 msgr2=0x7efe380b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:55.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3cd9a700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3cd9a700 1 -- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe38153290 msgr2=0x7efe381f2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:55.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3cd9a700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe38153290 0x7efe381f2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3cd9a700 1 -- 172.21.15.102:0/2734199754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30009a70 con 0x7efe380cd8e0 2023-03-03T06:38:55.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3d59b700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:38:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.725+0000 7efe3dd9c700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe38153290 0x7efe381f2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:38:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.726+0000 7efe3cd9a700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe380cd8e0 0x7efe380b3b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7efe2800c450 tx=0x7efe2800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:38:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.726+0000 7efe2dffb700 1 -- 172.21.15.102:0/2734199754 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe280052d0 con 0x7efe380cd8e0 2023-03-03T06:38:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.726+0000 7efe2dffb700 1 -- 172.21.15.102:0/2734199754 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe28005430 con 0x7efe380cd8e0 2023-03-03T06:38:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.726+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe380ccde0 con 0x7efe380cd8e0 2023-03-03T06:38:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.726+0000 7efe2dffb700 1 -- 172.21.15.102:0/2734199754 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe280038a0 con 0x7efe380cd8e0 2023-03-03T06:38:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.726+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3815d160 con 0x7efe380cd8e0 2023-03-03T06:38:55.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.727+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe380cd0c0 con 0x7efe380cd8e0 2023-03-03T06:38:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.728+0000 7efe2dffb700 1 -- 172.21.15.102:0/2734199754 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efe28019030 con 0x7efe380cd8e0 2023-03-03T06:38:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.730+0000 7efe2dffb700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe1c0682f0 0x7efe1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:38:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.730+0000 7efe2dffb700 1 -- 172.21.15.102:0/2734199754 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7efe28084ff0 con 0x7efe380cd8e0 2023-03-03T06:38:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.730+0000 7efe3d59b700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe1c0682f0 0x7efe1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:38:55.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.731+0000 7efe3d59b700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe1c0682f0 0x7efe1c06a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efe34001110 tx=0x7efe340010a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:38:55.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.734+0000 7efe2dffb700 1 -- 172.21.15.102:0/2734199754 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efe28051700 con 0x7efe380cd8e0 2023-03-03T06:38:55.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.965+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe3804e6e0 con 0x7efe380cd8e0 2023-03-03T06:38:55.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.965+0000 7efe2dffb700 1 -- 172.21.15.102:0/2734199754 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7efe28017090 con 0x7efe380cd8e0 2023-03-03T06:38:55.968 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:38:55.968 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:38:55.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.969+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe1c0682f0 msgr2=0x7efe1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:55.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.969+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe1c0682f0 0x7efe1c06a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efe34001110 tx=0x7efe340010a0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.969+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe380cd8e0 msgr2=0x7efe380b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:38:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.969+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe380cd8e0 0x7efe380b3b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7efe2800c450 tx=0x7efe2800c760 comp rx=0 tx=0).stop 2023-03-03T06:38:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 shutdown_connections 2023-03-03T06:38:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efe1c0682f0 0x7efe1c06a7a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efe380cd390 0x7efe380b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe38153290 0x7efe381f2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 --2- 172.21.15.102:0/2734199754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe380cd8e0 0x7efe380b3b00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:38:55.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 >> 172.21.15.102:0/2734199754 conn(0x7efe38065fb0 msgr2=0x7efe380cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:38:55.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 shutdown_connections 2023-03-03T06:38:55.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:38:55.970+0000 7efe3f7ff700 1 -- 172.21.15.102:0/2734199754 wait complete. 2023-03-03T06:38:55.973 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:38:55.998 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-03-03T06:39:16.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:39:16.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:39:47.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:39:47.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T06:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:40:17.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:17.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:46.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:40:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.248+0000 7fe723fff700 1 Processor -- start 2023-03-03T06:40:46.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.248+0000 7fe723fff700 1 -- start start 2023-03-03T06:40:46.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe723fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe7241564b0 0x7fe7240cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:40:46.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe723fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7240c9ca0 con 0x7fe7240c9610 2023-03-03T06:40:46.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe723fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7240b46f0 con 0x7fe7240cab50 2023-03-03T06:40:46.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe723fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7240b4920 con 0x7fe7241564b0 2023-03-03T06:40:46.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe7237fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe7240cab50 0x7fe7241636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47764/0 (socket says 172.21.15.102:47764) 2023-03-03T06:40:46.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe7237fe700 1 -- 172.21.15.102:0/2246856730 learned_addr learned my addr 172.21.15.102:0/2246856730 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:40:46.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe722ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe7241564b0 0x7fe7240cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:40:46.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe7227fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe7240c9610 0x7fe7240ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47086/0 (socket says 172.21.15.102:47086) 2023-03-03T06:40:46.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.250+0000 7fe721ffb700 1 -- 172.21.15.102:0/2246856730 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050046060 0 0) 0x7fe7240b46f0 con 0x7fe7240cab50 2023-03-03T06:40:46.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe721ffb700 1 -- 172.21.15.102:0/2246856730 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c0035b0 con 0x7fe7240cab50 2023-03-03T06:40:46.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe722ffd700 1 -- 172.21.15.102:0/2246856730 >> v1:172.21.15.172:6789/0 conn(0x7fe7240cab50 legacy=0x7fe7241636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:40:46.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe721ffb700 1 -- 172.21.15.102:0/2246856730 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099045959 0 0) 0x7fe7240c9ca0 con 0x7fe7240c9610 2023-03-03T06:40:46.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe722ffd700 1 -- 172.21.15.102:0/2246856730 >> v1:172.21.15.102:6789/0 conn(0x7fe7240c9610 legacy=0x7fe7240ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:40:46.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe722ffd700 1 -- 172.21.15.102:0/2246856730 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7240b4c00 con 0x7fe7241564b0 2023-03-03T06:40:46.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe722ffd700 1 --2- 172.21.15.102:0/2246856730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7241564b0 0x7fe7240cb660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe714009840 tx=0x7fe714009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7d4a8437595f7c4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:40:46.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe721ffb700 1 -- 172.21.15.102:0/2246856730 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714011830 con 0x7fe7241564b0 2023-03-03T06:40:46.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe721ffb700 1 -- 172.21.15.102:0/2246856730 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe714011990 con 0x7fe7241564b0 2023-03-03T06:40:46.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.251+0000 7fe721ffb700 1 -- 172.21.15.102:0/2246856730 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714011d00 con 0x7fe7241564b0 2023-03-03T06:40:46.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.252+0000 7fe723fff700 1 -- 172.21.15.102:0/2246856730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7241564b0 msgr2=0x7fe7240cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:40:46.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.252+0000 7fe723fff700 1 --2- 172.21.15.102:0/2246856730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7241564b0 0x7fe7240cb660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe714009840 tx=0x7fe714009b50 comp rx=0 tx=0).stop 2023-03-03T06:40:46.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.252+0000 7fe723fff700 1 -- 172.21.15.102:0/2246856730 shutdown_connections 2023-03-03T06:40:46.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.252+0000 7fe723fff700 1 --2- 172.21.15.102:0/2246856730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7241564b0 0x7fe7240cb660 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:40:46.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.252+0000 7fe723fff700 1 -- 172.21.15.102:0/2246856730 >> 172.21.15.102:0/2246856730 conn(0x7fe724065fb0 msgr2=0x7fe7240663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:40:46.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.252+0000 7fe723fff700 1 -- 172.21.15.102:0/2246856730 shutdown_connections 2023-03-03T06:40:46.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.252+0000 7fe723fff700 1 -- 172.21.15.102:0/2246856730 wait complete. 2023-03-03T06:40:46.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.254+0000 7fe723fff700 1 Processor -- start 2023-03-03T06:40:46.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.254+0000 7fe723fff700 1 -- start start 2023-03-03T06:40:46.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe723fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7240c9610 0x7fe72415c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:40:46.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe723fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7240cab50 0x7fe724159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:40:46.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe723fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7241564b0 0x7fe724150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:40:46.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe723fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72415d1b0 con 0x7fe7240c9610 2023-03-03T06:40:46.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe723fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe724158280 con 0x7fe7240cab50 2023-03-03T06:40:46.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe723fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7241ee910 con 0x7fe7241564b0 2023-03-03T06:40:46.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe722ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7240c9610 0x7fe72415c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:40:46.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe722ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7240c9610 0x7fe72415c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34910/0 (socket says 172.21.15.102:34910) 2023-03-03T06:40:46.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe722ffd700 1 -- 172.21.15.102:0/184834424 learned_addr learned my addr 172.21.15.102:0/184834424 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:40:46.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe7227fc700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7240cab50 0x7fe724159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:40:46.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.255+0000 7fe7237fe700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7241564b0 0x7fe724150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:40:46.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe722ffd700 1 -- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7241564b0 msgr2=0x7fe724150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:40:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe722ffd700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7241564b0 0x7fe724150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:40:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe722ffd700 1 -- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7240cab50 msgr2=0x7fe724159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:40:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe722ffd700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7240cab50 0x7fe724159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:40:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe722ffd700 1 -- 172.21.15.102:0/184834424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140094a0 con 0x7fe7240c9610 2023-03-03T06:40:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe722ffd700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7240c9610 0x7fe72415c7d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe714009fd0 tx=0x7fe714004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:40:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe70b7fe700 1 -- 172.21.15.102:0/184834424 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140045d0 con 0x7fe7240c9610 2023-03-03T06:40:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.256+0000 7fe70b7fe700 1 -- 172.21.15.102:0/184834424 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71401f070 con 0x7fe7240c9610 2023-03-03T06:40:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.257+0000 7fe70b7fe700 1 -- 172.21.15.102:0/184834424 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140038e0 con 0x7fe7240c9610 2023-03-03T06:40:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.257+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7240b2210 con 0x7fe7240c9610 2023-03-03T06:40:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.257+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7240b1f30 con 0x7fe7240c9610 2023-03-03T06:40:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.258+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe724156880 con 0x7fe7240c9610 2023-03-03T06:40:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.259+0000 7fe70b7fe700 1 -- 172.21.15.102:0/184834424 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe71400f050 con 0x7fe7240c9610 2023-03-03T06:40:46.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.260+0000 7fe70b7fe700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7100682f0 0x7fe71006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:40:46.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.260+0000 7fe70b7fe700 1 -- 172.21.15.102:0/184834424 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe714021070 con 0x7fe7240c9610 2023-03-03T06:40:46.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.260+0000 7fe7227fc700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7100682f0 0x7fe71006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:40:46.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.261+0000 7fe7227fc700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7100682f0 0x7fe71006a7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe724155d00 tx=0x7fe718007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:40:46.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.264+0000 7fe70b7fe700 1 -- 172.21.15.102:0/184834424 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe714053d40 con 0x7fe7240c9610 2023-03-03T06:40:46.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.502+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7240b2850 con 0x7fe7240c9610 2023-03-03T06:40:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.502+0000 7fe70b7fe700 1 -- 172.21.15.102:0/184834424 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe714057360 con 0x7fe7240c9610 2023-03-03T06:40:46.505 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:40:46.505 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:40:46.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7100682f0 msgr2=0x7fe71006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:40:46.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7100682f0 0x7fe71006a7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe724155d00 tx=0x7fe718007e70 comp rx=0 tx=0).stop 2023-03-03T06:40:46.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7240c9610 msgr2=0x7fe72415c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:40:46.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7240c9610 0x7fe72415c7d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe714009fd0 tx=0x7fe714004750 comp rx=0 tx=0).stop 2023-03-03T06:40:46.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 shutdown_connections 2023-03-03T06:40:46.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7100682f0 0x7fe71006a7a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:40:46.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7241564b0 0x7fe724150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:40:46.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7240c9610 0x7fe72415c7d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:40:46.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 --2- 172.21.15.102:0/184834424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7240cab50 0x7fe724159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:40:46.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.505+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 >> 172.21.15.102:0/184834424 conn(0x7fe724065fb0 msgr2=0x7fe724153380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:40:46.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.506+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 shutdown_connections 2023-03-03T06:40:46.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:40:46.506+0000 7fe723fff700 1 -- 172.21.15.102:0/184834424 wait complete. 2023-03-03T06:40:46.510 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:40:46.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-03-03T06:40:47.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:47.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:17.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:17.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:47.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:47.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:17.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:17.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:33.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:42:33.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.786+0000 7f4f7f7ff700 1 Processor -- start 2023-03-03T06:42:33.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.786+0000 7f4f7f7ff700 1 -- start start 2023-03-03T06:42:33.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.787+0000 7f4f7f7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4f78153290 0x7f4f78158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:42:33.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.787+0000 7f4f7f7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f781593c0 0x7f4f780cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:42:33.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.787+0000 7f4f7f7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4f780cd8e0 0x7f4f780ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:42:33.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.787+0000 7f4f7f7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f78153660 con 0x7f4f78153290 2023-03-03T06:42:33.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.787+0000 7f4f7f7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7806c360 con 0x7f4f780cd8e0 2023-03-03T06:42:33.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.787+0000 7f4f7f7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7815cea0 con 0x7f4f781593c0 2023-03-03T06:42:33.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4f78153290 0x7f4f78158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:42:33.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4f78153290 0x7f4f78158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38020/0 (socket says 172.21.15.102:38020) 2023-03-03T06:42:33.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7cd9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f781593c0 0x7f4f780cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:42:33.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 -- 172.21.15.102:0/775787873 learned_addr learned my addr 172.21.15.102:0/775787873 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:42:33.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7dd9c700 1 --2- 172.21.15.102:0/775787873 >> v2:172.21.15.172:3300/0 conn(0x7f4f780cd8e0 0x7f4f780ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:42:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 -- 172.21.15.102:0/775787873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f781593c0 msgr2=0x7f4f780cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:42:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 --2- 172.21.15.102:0/775787873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f781593c0 0x7f4f780cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 -- 172.21.15.102:0/775787873 >> v2:172.21.15.172:3300/0 conn(0x7f4f780cd8e0 msgr2=0x7f4f780ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:42:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 --2- 172.21.15.102:0/775787873 >> v2:172.21.15.172:3300/0 conn(0x7f4f780cd8e0 0x7f4f780ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.788+0000 7f4f7d59b700 1 -- 172.21.15.102:0/775787873 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7806c7f0 con 0x7f4f78153290 2023-03-03T06:42:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f7d59b700 1 --2- 172.21.15.102:0/775787873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f78153290 0x7f4f78158e80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4f7806d0e0 tx=0x7f4f74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d54e7cc2d2968d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:42:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f6ffff700 1 -- 172.21.15.102:0/775787873 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74011830 con 0x7f4f78153290 2023-03-03T06:42:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f6ffff700 1 -- 172.21.15.102:0/775787873 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f74011990 con 0x7f4f78153290 2023-03-03T06:42:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f6ffff700 1 -- 172.21.15.102:0/775787873 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74011d20 con 0x7f4f78153290 2023-03-03T06:42:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/775787873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f78153290 msgr2=0x7f4f78158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:42:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/775787873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f78153290 0x7f4f78158e80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4f7806d0e0 tx=0x7f4f74009bd0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/775787873 shutdown_connections 2023-03-03T06:42:33.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/775787873 >> v2:172.21.15.172:3300/0 conn(0x7f4f780cd8e0 0x7f4f780ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/775787873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f78153290 0x7f4f78158e80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.789+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/775787873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f781593c0 0x7f4f780cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.790+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/775787873 >> 172.21.15.102:0/775787873 conn(0x7f4f78065fb0 msgr2=0x7f4f780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:42:33.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.790+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/775787873 shutdown_connections 2023-03-03T06:42:33.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.790+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/775787873 wait complete. 2023-03-03T06:42:33.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.792+0000 7f4f7f7ff700 1 Processor -- start 2023-03-03T06:42:33.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.792+0000 7f4f7f7ff700 1 -- start start 2023-03-03T06:42:33.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7f7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f780cd8e0 0x7f4f78158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:42:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7f7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f78153290 0x7f4f781564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:42:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7f7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f781593c0 0x7f4f78157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:42:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7f7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f781e31d0 con 0x7f4f781593c0 2023-03-03T06:42:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7f7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f78154da0 con 0x7f4f780cd8e0 2023-03-03T06:42:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7f7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f78153890 con 0x7f4f78153290 2023-03-03T06:42:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7dd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f781593c0 0x7f4f78157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:42:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7d59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f780cd8e0 0x7f4f78158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:42:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7cd9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f78153290 0x7f4f781564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:42:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7dd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f781593c0 0x7f4f78157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38034/0 (socket says 172.21.15.102:38034) 2023-03-03T06:42:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.793+0000 7f4f7dd9c700 1 -- 172.21.15.102:0/476642328 learned_addr learned my addr 172.21.15.102:0/476642328 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:42:33.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.794+0000 7f4f7cd9a700 1 -- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f780cd8e0 msgr2=0x7f4f78158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:42:33.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.794+0000 7f4f7cd9a700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f780cd8e0 0x7f4f78158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.794+0000 7f4f7cd9a700 1 -- 172.21.15.102:0/476642328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f781593c0 msgr2=0x7f4f78157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:42:33.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.794+0000 7f4f7cd9a700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f781593c0 0x7f4f78157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:33.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.794+0000 7f4f7cd9a700 1 -- 172.21.15.102:0/476642328 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74009550 con 0x7f4f78153290 2023-03-03T06:42:33.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.794+0000 7f4f7d59b700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f780cd8e0 0x7f4f78158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:42:33.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.794+0000 7f4f7cd9a700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f78153290 0x7f4f781564e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800e900 tx=0x7f4f6800ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:42:33.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.795+0000 7f4f6dffb700 1 -- 172.21.15.102:0/476642328 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68013680 con 0x7f4f78153290 2023-03-03T06:42:33.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.795+0000 7f4f6dffb700 1 -- 172.21.15.102:0/476642328 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f680137e0 con 0x7f4f78153290 2023-03-03T06:42:33.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.795+0000 7f4f6dffb700 1 -- 172.21.15.102:0/476642328 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68013b50 con 0x7f4f78153290 2023-03-03T06:42:33.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.795+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f780ccde0 con 0x7f4f78153290 2023-03-03T06:42:33.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.795+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7815d160 con 0x7f4f78153290 2023-03-03T06:42:33.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.796+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f780cd0c0 con 0x7f4f78153290 2023-03-03T06:42:33.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.797+0000 7f4f6dffb700 1 -- 172.21.15.102:0/476642328 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f68019030 con 0x7f4f78153290 2023-03-03T06:42:33.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.798+0000 7f4f6dffb700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4f5c0682f0 0x7f4f5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:42:33.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.798+0000 7f4f6dffb700 1 -- 172.21.15.102:0/476642328 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4f68085b20 con 0x7f4f78153290 2023-03-03T06:42:33.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.799+0000 7f4f7d59b700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4f5c0682f0 0x7f4f5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:42:33.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.799+0000 7f4f7d59b700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4f5c0682f0 0x7f4f5c06a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f74014040 tx=0x7f4f74008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:42:33.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:33.803+0000 7f4f6dffb700 1 -- 172.21.15.102:0/476642328 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f680521d0 con 0x7f4f78153290 2023-03-03T06:42:34.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.037+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f7804e6e0 con 0x7f4f78153290 2023-03-03T06:42:34.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.038+0000 7f4f6dffb700 1 -- 172.21.15.102:0/476642328 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4f68017090 con 0x7f4f78153290 2023-03-03T06:42:34.040 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:42:34.040 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:42:34.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4f5c0682f0 msgr2=0x7f4f5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:42:34.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4f5c0682f0 0x7f4f5c06a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f74014040 tx=0x7f4f74008030 comp rx=0 tx=0).stop 2023-03-03T06:42:34.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f78153290 msgr2=0x7f4f781564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:42:34.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f78153290 0x7f4f781564e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800e900 tx=0x7f4f6800ec10 comp rx=0 tx=0).stop 2023-03-03T06:42:34.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 shutdown_connections 2023-03-03T06:42:34.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f781593c0 0x7f4f78157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:34.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f780cd8e0 0x7f4f78158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:34.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4f5c0682f0 0x7f4f5c06a7a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:34.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 --2- 172.21.15.102:0/476642328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4f78153290 0x7f4f781564e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:42:34.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 >> 172.21.15.102:0/476642328 conn(0x7f4f78065fb0 msgr2=0x7f4f780cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:42:34.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 shutdown_connections 2023-03-03T06:42:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:42:34.040+0000 7f4f7f7ff700 1 -- 172.21.15.102:0/476642328 wait complete. 2023-03-03T06:42:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:42:34.070 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-03-03T06:42:48.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:48.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:07.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:43:07.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.320+0000 7f0b787ff700 1 Processor -- start 2023-03-03T06:43:07.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.320+0000 7f0b787ff700 1 -- start start 2023-03-03T06:43:07.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.321+0000 7f0b787ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0b70153290 0x7f0b70158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:43:07.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b787ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0b701593c0 0x7f0b700cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:43:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b787ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b70153660 con 0x7f0b700cd8e0 2023-03-03T06:43:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b787ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7006c360 con 0x7f0b701593c0 2023-03-03T06:43:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b787ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7015cea0 con 0x7f0b70153290 2023-03-03T06:43:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b75d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0b701593c0 0x7f0b700cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:43:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b76d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0b700cd8e0 0x7f0b700ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49174/0 (socket says 172.21.15.102:49174) 2023-03-03T06:43:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b76d9c700 1 -- 172.21.15.102:0/88019439 learned_addr learned my addr 172.21.15.102:0/88019439 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:43:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.322+0000 7f0b7659b700 1 --2- 172.21.15.102:0/88019439 >> v2:172.21.15.172:3300/0 conn(0x7f0b70153290 0x7f0b70158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:43:07.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75d9a700 1 -- 172.21.15.102:0/88019439 >> v2:172.21.15.172:3300/0 conn(0x7f0b70153290 msgr2=0x7f0b70158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:43:07.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75d9a700 1 --2- 172.21.15.102:0/88019439 >> v2:172.21.15.172:3300/0 conn(0x7f0b70153290 0x7f0b70158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75d9a700 1 -- 172.21.15.102:0/88019439 >> v1:172.21.15.102:6789/0 conn(0x7f0b700cd8e0 legacy=0x7f0b700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:43:07.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75d9a700 1 -- 172.21.15.102:0/88019439 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7006c7f0 con 0x7f0b701593c0 2023-03-03T06:43:07.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75599700 1 -- 172.21.15.102:0/88019439 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901454633 0 0) 0x7f0b70153660 con 0x7f0b700cd8e0 2023-03-03T06:43:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75d9a700 1 --2- 172.21.15.102:0/88019439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b701593c0 0x7f0b700cde20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0b7008cc70 tx=0x7f0b60009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2413542451cdeb3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:43:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75599700 1 -- 172.21.15.102:0/88019439 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60011830 con 0x7f0b701593c0 2023-03-03T06:43:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75599700 1 -- 172.21.15.102:0/88019439 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60011990 con 0x7f0b701593c0 2023-03-03T06:43:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.323+0000 7f0b75599700 1 -- 172.21.15.102:0/88019439 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60011d20 con 0x7f0b701593c0 2023-03-03T06:43:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 -- 172.21.15.102:0/88019439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b701593c0 msgr2=0x7f0b700cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:43:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 --2- 172.21.15.102:0/88019439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b701593c0 0x7f0b700cde20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0b7008cc70 tx=0x7f0b60009b20 comp rx=0 tx=0).stop 2023-03-03T06:43:07.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 -- 172.21.15.102:0/88019439 shutdown_connections 2023-03-03T06:43:07.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 --2- 172.21.15.102:0/88019439 >> v2:172.21.15.172:3300/0 conn(0x7f0b70153290 0x7f0b70158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 --2- 172.21.15.102:0/88019439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b701593c0 0x7f0b700cde20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 -- 172.21.15.102:0/88019439 >> 172.21.15.102:0/88019439 conn(0x7f0b70065fb0 msgr2=0x7f0b700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:43:07.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 -- 172.21.15.102:0/88019439 shutdown_connections 2023-03-03T06:43:07.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.324+0000 7f0b787ff700 1 -- 172.21.15.102:0/88019439 wait complete. 2023-03-03T06:43:07.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.326+0000 7f0b787ff700 1 Processor -- start 2023-03-03T06:43:07.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.326+0000 7f0b787ff700 1 -- start start 2023-03-03T06:43:07.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.327+0000 7f0b787ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b700cd8e0 0x7f0b700c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:43:07.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.327+0000 7f0b787ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b70153290 0x7f0b700c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.327+0000 7f0b787ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b701593c0 0x7f0b700c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.327+0000 7f0b787ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700c7270 con 0x7f0b700cd8e0 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.327+0000 7f0b787ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701ef0d0 con 0x7f0b70153290 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.327+0000 7f0b787ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701ef600 con 0x7f0b701593c0 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b70153290 0x7f0b700c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b7659b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b700cd8e0 0x7f0b700c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b76d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b701593c0 0x7f0b700c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:43:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b70153290 0x7f0b700c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57176/0 (socket says 172.21.15.102:57176) 2023-03-03T06:43:07.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 -- 172.21.15.102:0/19258324 learned_addr learned my addr 172.21.15.102:0/19258324 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:43:07.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 -- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b701593c0 msgr2=0x7f0b700c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:43:07.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b701593c0 0x7f0b700c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 -- 172.21.15.102:0/19258324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b700cd8e0 msgr2=0x7f0b700c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:43:07.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b700cd8e0 0x7f0b700c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 -- 172.21.15.102:0/19258324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b600094a0 con 0x7f0b70153290 2023-03-03T06:43:07.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.328+0000 7f0b75d9a700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b70153290 0x7f0b700c76c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b600090b0 tx=0x7f0b60004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:43:07.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.329+0000 7f0b66ffd700 1 -- 172.21.15.102:0/19258324 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b600036a0 con 0x7f0b70153290 2023-03-03T06:43:07.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.329+0000 7f0b66ffd700 1 -- 172.21.15.102:0/19258324 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6001f070 con 0x7f0b70153290 2023-03-03T06:43:07.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.329+0000 7f0b66ffd700 1 -- 172.21.15.102:0/19258324 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60003a20 con 0x7f0b70153290 2023-03-03T06:43:07.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.329+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b700ccde0 con 0x7f0b70153290 2023-03-03T06:43:07.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.329+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7015d160 con 0x7f0b70153290 2023-03-03T06:43:07.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.330+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b700cd0c0 con 0x7f0b70153290 2023-03-03T06:43:07.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.331+0000 7f0b66ffd700 1 -- 172.21.15.102:0/19258324 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b6000f050 con 0x7f0b70153290 2023-03-03T06:43:07.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.332+0000 7f0b66ffd700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:43:07.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.332+0000 7f0b66ffd700 1 -- 172.21.15.102:0/19258324 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0b60021070 con 0x7f0b70153290 2023-03-03T06:43:07.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.333+0000 7f0b7659b700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:43:07.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.333+0000 7f0b7659b700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009be0 tx=0x7f0b6c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:43:07.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.337+0000 7f0b66ffd700 1 -- 172.21.15.102:0/19258324 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b60053c70 con 0x7f0b70153290 2023-03-03T06:43:07.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.571+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b7008bee0 con 0x7f0b70153290 2023-03-03T06:43:07.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.572+0000 7f0b66ffd700 1 -- 172.21.15.102:0/19258324 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0b60057290 con 0x7f0b70153290 2023-03-03T06:43:07.574 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:43:07.574 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:43:07.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.574+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 msgr2=0x7f0b5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:43:07.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.574+0000 7f0b787ff700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009be0 tx=0x7f0b6c009210 comp rx=0 tx=0).stop 2023-03-03T06:43:07.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.574+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b70153290 msgr2=0x7f0b700c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:43:07.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.574+0000 7f0b787ff700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b70153290 0x7f0b700c76c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b600090b0 tx=0x7f0b60004750 comp rx=0 tx=0).stop 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 shutdown_connections 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b701593c0 0x7f0b700c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b700cd8e0 0x7f0b700c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 --2- 172.21.15.102:0/19258324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b70153290 0x7f0b700c76c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 >> 172.21.15.102:0/19258324 conn(0x7f0b70065fb0 msgr2=0x7f0b700cc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 shutdown_connections 2023-03-03T06:43:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:43:07.575+0000 7f0b787ff700 1 -- 172.21.15.102:0/19258324 wait complete. 2023-03-03T06:43:07.579 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:43:07.604 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T06:43:18.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:18.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:48.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:48.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:05.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:44:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.851+0000 7f9f0fbff700 1 Processor -- start 2023-03-03T06:44:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.851+0000 7f9f0fbff700 1 -- start start 2023-03-03T06:44:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.852+0000 7f9f0fbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f08153290 0x7f9f08158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.852+0000 7f9f0fbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f080cd8e0 0x7f9f080ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.852+0000 7f9f0fbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f08153660 con 0x7f9f081593c0 2023-03-03T06:44:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.852+0000 7f9f0fbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0806c360 con 0x7f9f080cd8e0 2023-03-03T06:44:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.852+0000 7f9f0fbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0815cea0 con 0x7f9f08153290 2023-03-03T06:44:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0e19c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f080cd8e0 0x7f9f080ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0d99b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f08153290 0x7f9f08158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:05.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0d99b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f08153290 0x7f9f08158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39828/0 (socket says 172.21.15.102:39828) 2023-03-03T06:44:05.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0d99b700 1 -- 172.21.15.102:0/4205905744 learned_addr learned my addr 172.21.15.102:0/4205905744 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:44:05.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0e19c700 1 -- 172.21.15.102:0/4205905744 >> v2:172.21.15.172:3300/0 conn(0x7f9f08153290 msgr2=0x7f9f08158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:44:05.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0e19c700 1 --2- 172.21.15.102:0/4205905744 >> v2:172.21.15.172:3300/0 conn(0x7f9f08153290 0x7f9f08158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:05.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0e19c700 1 -- 172.21.15.102:0/4205905744 >> v1:172.21.15.102:6789/0 conn(0x7f9f081593c0 legacy=0x7f9f080cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:05.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.853+0000 7f9f0e19c700 1 -- 172.21.15.102:0/4205905744 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0806c7f0 con 0x7f9f080cd8e0 2023-03-03T06:44:05.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.854+0000 7f9f0e19c700 1 --2- 172.21.15.102:0/4205905744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f080ce530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9f0808cc70 tx=0x7f9f0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48897d69d2ec8dff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:44:05.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.854+0000 7f9f0c999700 1 -- 172.21.15.102:0/4205905744 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000e070 con 0x7f9f080cd8e0 2023-03-03T06:44:05.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.854+0000 7f9f0c999700 1 -- 172.21.15.102:0/4205905744 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0000bb20 con 0x7f9f080cd8e0 2023-03-03T06:44:05.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.854+0000 7f9f0c999700 1 -- 172.21.15.102:0/4205905744 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00012430 con 0x7f9f080cd8e0 2023-03-03T06:44:05.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.854+0000 7f9f0fbff700 1 -- 172.21.15.102:0/4205905744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 msgr2=0x7f9f080ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:05.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.854+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/4205905744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f080ce530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9f0808cc70 tx=0x7f9f0000b330 comp rx=0 tx=0).stop 2023-03-03T06:44:05.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.855+0000 7f9f0fbff700 1 -- 172.21.15.102:0/4205905744 shutdown_connections 2023-03-03T06:44:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.855+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/4205905744 >> v2:172.21.15.172:3300/0 conn(0x7f9f08153290 0x7f9f08158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.855+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/4205905744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f080ce530 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.855+0000 7f9f0fbff700 1 -- 172.21.15.102:0/4205905744 >> 172.21.15.102:0/4205905744 conn(0x7f9f08065fb0 msgr2=0x7f9f080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:44:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.855+0000 7f9f0fbff700 1 -- 172.21.15.102:0/4205905744 shutdown_connections 2023-03-03T06:44:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.855+0000 7f9f0fbff700 1 -- 172.21.15.102:0/4205905744 wait complete. 2023-03-03T06:44:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.857+0000 7f9f0fbff700 1 Processor -- start 2023-03-03T06:44:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.857+0000 7f9f0fbff700 1 -- start start 2023-03-03T06:44:05.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0fbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f081f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:05.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0fbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f08153290 0x7f9f080c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:05.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0fbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f081593c0 0x7f9f080c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:05.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0fbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f081f2e90 con 0x7f9f080cd8e0 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0fbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f081e5520 con 0x7f9f08153290 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0fbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f081e6060 con 0x7f9f081593c0 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0d99b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f081f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0d99b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f081f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51236/0 (socket says 172.21.15.102:51236) 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0d99b700 1 -- 172.21.15.102:0/1181642086 learned_addr learned my addr 172.21.15.102:0/1181642086 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0d19a700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f08153290 0x7f9f080c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.858+0000 7f9f0e19c700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f081593c0 0x7f9f080c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9f0d99b700 1 -- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f081593c0 msgr2=0x7f9f080c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9f0d99b700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f081593c0 0x7f9f080c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9f0d99b700 1 -- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f08153290 msgr2=0x7f9f080c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9f0d99b700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f08153290 0x7f9f080c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9f0d99b700 1 -- 172.21.15.102:0/1181642086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00009a70 con 0x7f9f080cd8e0 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9f0d99b700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f081f2950 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400e910 tx=0x7f9f0400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9efe7fc700 1 -- 172.21.15.102:0/1181642086 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f040096a0 con 0x7f9f080cd8e0 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9efe7fc700 1 -- 172.21.15.102:0/1181642086 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f04013680 con 0x7f9f080cd8e0 2023-03-03T06:44:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f080ccde0 con 0x7f9f080cd8e0 2023-03-03T06:44:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.859+0000 7f9efe7fc700 1 -- 172.21.15.102:0/1181642086 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04013a30 con 0x7f9f080cd8e0 2023-03-03T06:44:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.860+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0815d160 con 0x7f9f080cd8e0 2023-03-03T06:44:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.861+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f080cd0c0 con 0x7f9f080cd8e0 2023-03-03T06:44:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.862+0000 7f9efe7fc700 1 -- 172.21.15.102:0/1181642086 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9f04019030 con 0x7f9f080cd8e0 2023-03-03T06:44:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.863+0000 7f9efe7fc700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9eec0682f0 0x7f9eec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.863+0000 7f9efe7fc700 1 -- 172.21.15.102:0/1181642086 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9f04085c00 con 0x7f9f080cd8e0 2023-03-03T06:44:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.863+0000 7f9f0d19a700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9eec0682f0 0x7f9eec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.864+0000 7f9f0d19a700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9eec0682f0 0x7f9eec06a7a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9f0806d110 tx=0x7f9ef8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:44:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:05.867+0000 7f9efe7fc700 1 -- 172.21.15.102:0/1181642086 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9f04052310 con 0x7f9f080cd8e0 2023-03-03T06:44:06.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.102+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f0808c780 con 0x7f9f080cd8e0 2023-03-03T06:44:06.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.103+0000 7f9efe7fc700 1 -- 172.21.15.102:0/1181642086 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9f04055930 con 0x7f9f080cd8e0 2023-03-03T06:44:06.105 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:44:06.106 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:44:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9eec0682f0 msgr2=0x7f9eec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9eec0682f0 0x7f9eec06a7a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9f0806d110 tx=0x7f9ef8009040 comp rx=0 tx=0).stop 2023-03-03T06:44:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 msgr2=0x7f9f081f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f081f2950 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400e910 tx=0x7f9f0400ec20 comp rx=0 tx=0).stop 2023-03-03T06:44:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 shutdown_connections 2023-03-03T06:44:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9eec0682f0 0x7f9eec06a7a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f081593c0 0x7f9f080c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f080cd8e0 0x7f9f081f2950 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 --2- 172.21.15.102:0/1181642086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f08153290 0x7f9f080c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.105+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 >> 172.21.15.102:0/1181642086 conn(0x7f9f08065fb0 msgr2=0x7f9f0815b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:44:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.106+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 shutdown_connections 2023-03-03T06:44:06.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:06.106+0000 7f9f0fbff700 1 -- 172.21.15.102:0/1181642086 wait complete. 2023-03-03T06:44:06.110 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:44:06.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 45 secs before thrashing 2023-03-03T06:44:18.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:18.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:49.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:49.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:51.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:44:51.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.395+0000 7f58d6f9d700 1 Processor -- start 2023-03-03T06:44:51.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.396+0000 7f58d6f9d700 1 -- start start 2023-03-03T06:44:51.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d6f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00c5a90 0x7f58d00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d6f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d00c3d20 con 0x7f58d00b3970 2023-03-03T06:44:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d6f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d015cba0 con 0x7f58d00ced00 2023-03-03T06:44:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d6f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00c4930 con 0x7f58d00c5a90 2023-03-03T06:44:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d579a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f58d00b3970 0x7f58d0161470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43162/0 (socket says 172.21.15.102:43162) 2023-03-03T06:44:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d579a700 1 -- 172.21.15.102:0/310192237 learned_addr learned my addr 172.21.15.102:0/310192237 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:44:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d5f9b700 1 --1- 172.21.15.102:0/310192237 >> v1:172.21.15.172:6789/0 conn(0x7f58d00ced00 0x7f58d0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40592/0 (socket says 172.21.15.102:40592) 2023-03-03T06:44:51.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.397+0000 7f58d679c700 1 --2- 172.21.15.102:0/310192237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00c5a90 0x7f58d00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d4f99700 1 -- 172.21.15.102:0/310192237 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787871270 0 0) 0x7f58d00c3d20 con 0x7f58d00b3970 2023-03-03T06:44:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d4f99700 1 -- 172.21.15.102:0/310192237 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58bc0035b0 con 0x7f58d00b3970 2023-03-03T06:44:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d4f99700 1 -- 172.21.15.102:0/310192237 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547136229 0 0) 0x7f58d015cba0 con 0x7f58d00ced00 2023-03-03T06:44:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d4f99700 1 -- 172.21.15.102:0/310192237 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d00c3d20 con 0x7f58d00ced00 2023-03-03T06:44:51.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d679c700 1 -- 172.21.15.102:0/310192237 >> v1:172.21.15.172:6789/0 conn(0x7f58d00ced00 legacy=0x7f58d0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d679c700 1 -- 172.21.15.102:0/310192237 >> v1:172.21.15.102:6789/0 conn(0x7f58d00b3970 legacy=0x7f58d0161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d679c700 1 -- 172.21.15.102:0/310192237 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d015ce80 con 0x7f58d00c5a90 2023-03-03T06:44:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.398+0000 7f58d679c700 1 --2- 172.21.15.102:0/310192237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00c5a90 0x7f58d00c41b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009dc0 tx=0x7f58c800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415f2b9018ce9147 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:44:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d4f99700 1 -- 172.21.15.102:0/310192237 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c800e070 con 0x7f58d00c5a90 2023-03-03T06:44:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d4f99700 1 -- 172.21.15.102:0/310192237 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c800bb20 con 0x7f58d00c5a90 2023-03-03T06:44:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d4f99700 1 -- 172.21.15.102:0/310192237 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8012430 con 0x7f58d00c5a90 2023-03-03T06:44:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d6f9d700 1 -- 172.21.15.102:0/310192237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00c5a90 msgr2=0x7f58d00c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/310192237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00c5a90 0x7f58d00c41b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009dc0 tx=0x7f58c800b330 comp rx=0 tx=0).stop 2023-03-03T06:44:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d6f9d700 1 -- 172.21.15.102:0/310192237 shutdown_connections 2023-03-03T06:44:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/310192237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00c5a90 0x7f58d00c41b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d6f9d700 1 -- 172.21.15.102:0/310192237 >> 172.21.15.102:0/310192237 conn(0x7f58d008b1c0 msgr2=0x7f58d0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:44:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.399+0000 7f58d6f9d700 1 -- 172.21.15.102:0/310192237 shutdown_connections 2023-03-03T06:44:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.400+0000 7f58d6f9d700 1 -- 172.21.15.102:0/310192237 wait complete. 2023-03-03T06:44:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.401+0000 7f58d6f9d700 1 Processor -- start 2023-03-03T06:44:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.402+0000 7f58d6f9d700 1 -- start start 2023-03-03T06:44:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d6f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00b3970 0x7f58d01510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d6f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d00c5a90 0x7f58d00ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d6f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58d00ced00 0x7f58d01f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d6f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d014fbd0 con 0x7f58d00c5a90 2023-03-03T06:44:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d6f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00c7f20 con 0x7f58d00ced00 2023-03-03T06:44:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d6f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00c4680 con 0x7f58d00b3970 2023-03-03T06:44:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d579a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d00c5a90 0x7f58d00ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d5f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00b3970 0x7f58d01510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d579a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d00c5a90 0x7f58d00ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49528/0 (socket says 172.21.15.102:49528) 2023-03-03T06:44:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d579a700 1 -- 172.21.15.102:0/3060545994 learned_addr learned my addr 172.21.15.102:0/3060545994 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:44:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.403+0000 7f58d679c700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58d00ced00 0x7f58d01f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d5f9b700 1 -- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58d00ced00 msgr2=0x7f58d01f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d5f9b700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58d00ced00 0x7f58d01f1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d5f9b700 1 -- 172.21.15.102:0/3060545994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d00c5a90 msgr2=0x7f58d00ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d5f9b700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d00c5a90 0x7f58d00ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d5f9b700 1 -- 172.21.15.102:0/3060545994 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c8009a70 con 0x7f58d00b3970 2023-03-03T06:44:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d579a700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d00c5a90 0x7f58d00ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:44:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d5f9b700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00b3970 0x7f58d01510d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00c4d0 tx=0x7f58cc00c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:44:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58c67fc700 1 -- 172.21.15.102:0/3060545994 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc005cb0 con 0x7f58d00b3970 2023-03-03T06:44:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58c67fc700 1 -- 172.21.15.102:0/3060545994 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc005e10 con 0x7f58d00b3970 2023-03-03T06:44:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d006c200 con 0x7f58d00b3970 2023-03-03T06:44:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.404+0000 7f58c67fc700 1 -- 172.21.15.102:0/3060545994 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0072d0 con 0x7f58d00b3970 2023-03-03T06:44:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.405+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d006bf20 con 0x7f58d00b3970 2023-03-03T06:44:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.406+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d015c890 con 0x7f58d00b3970 2023-03-03T06:44:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.407+0000 7f58c67fc700 1 -- 172.21.15.102:0/3060545994 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f58cc019030 con 0x7f58d00b3970 2023-03-03T06:44:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.408+0000 7f58c67fc700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58b40682f0 0x7f58b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:44:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.409+0000 7f58c67fc700 1 -- 172.21.15.102:0/3060545994 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f58cc085370 con 0x7f58d00b3970 2023-03-03T06:44:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.409+0000 7f58d579a700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58b40682f0 0x7f58b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:44:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.409+0000 7f58d579a700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58b40682f0 0x7f58b406a7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f58d0150720 tx=0x7f58c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:44:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.411+0000 7f58c67fc700 1 -- 172.21.15.102:0/3060545994 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f58cc051ac0 con 0x7f58d00b3970 2023-03-03T06:44:51.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.645+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58d014fe40 con 0x7f58d00b3970 2023-03-03T06:44:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.646+0000 7f58c67fc700 1 -- 172.21.15.102:0/3060545994 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f58cc0550e0 con 0x7f58d00b3970 2023-03-03T06:44:51.649 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:44:51.649 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:44:51.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58b40682f0 msgr2=0x7f58b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:51.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58b40682f0 0x7f58b406a7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f58d0150720 tx=0x7f58c0009040 comp rx=0 tx=0).stop 2023-03-03T06:44:51.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00b3970 msgr2=0x7f58d01510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:44:51.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00b3970 0x7f58d01510d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00c4d0 tx=0x7f58cc00c7e0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 shutdown_connections 2023-03-03T06:44:51.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58b40682f0 0x7f58b406a7a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58d00b3970 0x7f58d01510d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58d00ced00 0x7f58d01f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 --2- 172.21.15.102:0/3060545994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d00c5a90 0x7f58d00ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:44:51.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 >> 172.21.15.102:0/3060545994 conn(0x7f58d008b1c0 msgr2=0x7f58d0151ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:44:51.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 shutdown_connections 2023-03-03T06:44:51.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:44:51.649+0000 7f58d6f9d700 1 -- 172.21.15.102:0/3060545994 wait complete. 2023-03-03T06:44:51.656 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:44:51.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T06:45:19.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:45:19.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:45:35.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:45:35.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.927+0000 7f9a6d79d700 1 Processor -- start 2023-03-03T06:45:35.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.927+0000 7f9a6d79d700 1 -- start start 2023-03-03T06:45:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a6d79d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:45:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a6d79d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:45:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a6d79d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a680c3d20 con 0x7f9a680ced00 2023-03-03T06:45:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a6d79d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a680c4930 con 0x7f9a680b3970 2023-03-03T06:45:35.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a6d79d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a680cf610 con 0x7f9a680c41b0 2023-03-03T06:45:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a677fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:45:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a6cf9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:45:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a677fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53978/0 (socket says 172.21.15.102:53978) 2023-03-03T06:45:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a67fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9a680ced00 0x7f9a68158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51050/0 (socket says 172.21.15.102:51050) 2023-03-03T06:45:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.929+0000 7f9a677fe700 1 -- 172.21.15.102:0/3968674175 learned_addr learned my addr 172.21.15.102:0/3968674175 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:45:35.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a6cf9c700 1 -- 172.21.15.102:0/3968674175 >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 msgr2=0x7f9a680c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:45:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a6cf9c700 1 --2- 172.21.15.102:0/3968674175 >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a6cf9c700 1 -- 172.21.15.102:0/3968674175 >> v1:172.21.15.172:6789/0 conn(0x7f9a680ced00 legacy=0x7f9a68158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:45:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a6cf9c700 1 -- 172.21.15.102:0/3968674175 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6815cba0 con 0x7f9a680c41b0 2023-03-03T06:45:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3968674175 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436807441 0 0) 0x7f9a680c3d20 con 0x7f9a680ced00 2023-03-03T06:45:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a6cf9c700 1 --2- 172.21.15.102:0/3968674175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a680cf0d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9a6808ce30 tx=0x7f9a6000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83a0563518785ce1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:45:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3968674175 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6000e070 con 0x7f9a680c41b0 2023-03-03T06:45:35.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3968674175 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6000bb20 con 0x7f9a680c41b0 2023-03-03T06:45:35.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.930+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3968674175 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60012430 con 0x7f9a680c41b0 2023-03-03T06:45:35.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 -- 172.21.15.102:0/3968674175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 msgr2=0x7f9a680cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:45:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/3968674175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a680cf0d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9a6808ce30 tx=0x7f9a6000b330 comp rx=0 tx=0).stop 2023-03-03T06:45:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 -- 172.21.15.102:0/3968674175 shutdown_connections 2023-03-03T06:45:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/3968674175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/3968674175 >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 -- 172.21.15.102:0/3968674175 >> 172.21.15.102:0/3968674175 conn(0x7f9a6808b1c0 msgr2=0x7f9a68071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:45:35.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 -- 172.21.15.102:0/3968674175 shutdown_connections 2023-03-03T06:45:35.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.931+0000 7f9a6d79d700 1 -- 172.21.15.102:0/3968674175 wait complete. 2023-03-03T06:45:35.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.933+0000 7f9a6d79d700 1 Processor -- start 2023-03-03T06:45:35.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.933+0000 7f9a6d79d700 1 -- start start 2023-03-03T06:45:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.934+0000 7f9a6d79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a681e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:45:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.934+0000 7f9a6d79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a68151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:45:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.934+0000 7f9a6d79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:45:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.934+0000 7f9a6d79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6815af40 con 0x7f9a680b3970 2023-03-03T06:45:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.934+0000 7f9a6d79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6814cda0 con 0x7f9a680c41b0 2023-03-03T06:45:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.934+0000 7f9a6d79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6814e5f0 con 0x7f9a680ced00 2023-03-03T06:45:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a681e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:45:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a677fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a68151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:45:35.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a681e5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53992/0 (socket says 172.21.15.102:53992) 2023-03-03T06:45:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 -- 172.21.15.102:0/856299286 learned_addr learned my addr 172.21.15.102:0/856299286 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:45:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a6cf9c700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:45:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 -- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 msgr2=0x7f9a68151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:45:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 -- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 msgr2=0x7f9a68151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:45:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a68151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:35.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a67fff700 1 -- 172.21.15.102:0/856299286 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60009a70 con 0x7f9a680b3970 2023-03-03T06:45:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.935+0000 7f9a677fe700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a68151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:45:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.936+0000 7f9a67fff700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a681e5310 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c750 tx=0x7f9a5800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:45:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.936+0000 7f9a64ff9700 1 -- 172.21.15.102:0/856299286 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58004c80 con 0x7f9a680b3970 2023-03-03T06:45:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.936+0000 7f9a64ff9700 1 -- 172.21.15.102:0/856299286 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58004e00 con 0x7f9a680b3970 2023-03-03T06:45:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.936+0000 7f9a64ff9700 1 -- 172.21.15.102:0/856299286 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58007310 con 0x7f9a680b3970 2023-03-03T06:45:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.936+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6806bf20 con 0x7f9a680b3970 2023-03-03T06:45:35.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.936+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6815ce80 con 0x7f9a680b3970 2023-03-03T06:45:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.937+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6806c200 con 0x7f9a680b3970 2023-03-03T06:45:35.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.938+0000 7f9a64ff9700 1 -- 172.21.15.102:0/856299286 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a5801a020 con 0x7f9a680b3970 2023-03-03T06:45:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.939+0000 7f9a64ff9700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:45:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.940+0000 7f9a64ff9700 1 -- 172.21.15.102:0/856299286 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9a58085920 con 0x7f9a680b3970 2023-03-03T06:45:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.940+0000 7f9a677fe700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:45:35.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.941+0000 7f9a677fe700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a680cfc50 tx=0x7f9a5c0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:45:35.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:35.944+0000 7f9a64ff9700 1 -- 172.21.15.102:0/856299286 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a58052080 con 0x7f9a680b3970 2023-03-03T06:45:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.182+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6815baf0 con 0x7f9a680b3970 2023-03-03T06:45:36.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.182+0000 7f9a64ff9700 1 -- 172.21.15.102:0/856299286 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9a6815baf0 con 0x7f9a680b3970 2023-03-03T06:45:36.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:45:36.185 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:45:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 msgr2=0x7f9a5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:45:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a680cfc50 tx=0x7f9a5c0092c0 comp rx=0 tx=0).stop 2023-03-03T06:45:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 msgr2=0x7f9a681e5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:45:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a681e5310 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c750 tx=0x7f9a5800ca60 comp rx=0 tx=0).stop 2023-03-03T06:45:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 shutdown_connections 2023-03-03T06:45:36.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a681e5310 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 --2- 172.21.15.102:0/856299286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a68151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:45:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 >> 172.21.15.102:0/856299286 conn(0x7f9a6808b1c0 msgr2=0x7f9a680b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:45:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.185+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 shutdown_connections 2023-03-03T06:45:36.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:45:36.186+0000 7f9a6d79d700 1 -- 172.21.15.102:0/856299286 wait complete. 2023-03-03T06:45:36.190 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:45:36.215 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-03-03T06:45:49.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:45:49.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:19.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:19.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:49.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:49.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:05.931 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-03-03T06:47:05.933 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-03-03T06:47:20.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:20.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:26.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:47:26.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.474+0000 7f195dfff700 1 Processor -- start 2023-03-03T06:47:26.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.474+0000 7f195dfff700 1 -- start start 2023-03-03T06:47:26.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.475+0000 7f195dfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19580cd390 0x7f19580ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:47:26.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.475+0000 7f195dfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1958153660 con 0x7f19580cd8e0 2023-03-03T06:47:26.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.475+0000 7f195dfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195806c360 con 0x7f1958153290 2023-03-03T06:47:26.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.475+0000 7f195dfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195815cea0 con 0x7f19580cd390 2023-03-03T06:47:26.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f1956ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f19580cd8e0 0x7f19580cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43732/0 (socket says 172.21.15.102:43732) 2023-03-03T06:47:26.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f1956ffd700 1 -- 172.21.15.102:0/3590876879 learned_addr learned my addr 172.21.15.102:0/3590876879 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:47:26.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f1957fff700 1 --2- 172.21.15.102:0/3590876879 >> v2:172.21.15.102:3300/0 conn(0x7f19580cd390 0x7f19580ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:47:26.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f19567fc700 1 -- 172.21.15.102:0/3590876879 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499114060 0 0) 0x7f195806c360 con 0x7f1958153290 2023-03-03T06:47:26.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f19567fc700 1 -- 172.21.15.102:0/3590876879 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19400035b0 con 0x7f1958153290 2023-03-03T06:47:26.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f19567fc700 1 -- 172.21.15.102:0/3590876879 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073537012 0 0) 0x7f1958153660 con 0x7f19580cd8e0 2023-03-03T06:47:26.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f19567fc700 1 -- 172.21.15.102:0/3590876879 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195806c360 con 0x7f19580cd8e0 2023-03-03T06:47:26.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f1957fff700 1 -- 172.21.15.102:0/3590876879 >> v1:172.21.15.172:6789/0 conn(0x7f1958153290 legacy=0x7f1958158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:47:26.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f1957fff700 1 -- 172.21.15.102:0/3590876879 >> v1:172.21.15.102:6789/0 conn(0x7f19580cd8e0 legacy=0x7f19580cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:47:26.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.476+0000 7f1957fff700 1 -- 172.21.15.102:0/3590876879 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195806c7f0 con 0x7f19580cd390 2023-03-03T06:47:26.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.477+0000 7f1957fff700 1 --2- 172.21.15.102:0/3590876879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd390 0x7f19580ce530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1950009d90 tx=0x7f195000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8f1aeb8824f987 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:47:26.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.477+0000 7f19567fc700 1 -- 172.21.15.102:0/3590876879 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195000e070 con 0x7f19580cd390 2023-03-03T06:47:26.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.477+0000 7f19567fc700 1 -- 172.21.15.102:0/3590876879 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195000bb20 con 0x7f19580cd390 2023-03-03T06:47:26.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.477+0000 7f19567fc700 1 -- 172.21.15.102:0/3590876879 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950012430 con 0x7f19580cd390 2023-03-03T06:47:26.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.477+0000 7f195dfff700 1 -- 172.21.15.102:0/3590876879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd390 msgr2=0x7f19580ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:47:26.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.477+0000 7f195dfff700 1 --2- 172.21.15.102:0/3590876879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd390 0x7f19580ce530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1950009d90 tx=0x7f195000b330 comp rx=0 tx=0).stop 2023-03-03T06:47:26.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.478+0000 7f195dfff700 1 -- 172.21.15.102:0/3590876879 shutdown_connections 2023-03-03T06:47:26.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.478+0000 7f195dfff700 1 --2- 172.21.15.102:0/3590876879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd390 0x7f19580ce530 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:47:26.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.478+0000 7f195dfff700 1 -- 172.21.15.102:0/3590876879 >> 172.21.15.102:0/3590876879 conn(0x7f1958065fb0 msgr2=0x7f19580663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:47:26.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.478+0000 7f195dfff700 1 -- 172.21.15.102:0/3590876879 shutdown_connections 2023-03-03T06:47:26.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.478+0000 7f195dfff700 1 -- 172.21.15.102:0/3590876879 wait complete. 2023-03-03T06:47:26.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.481+0000 7f195dfff700 1 Processor -- start 2023-03-03T06:47:26.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.481+0000 7f195dfff700 1 -- start start 2023-03-03T06:47:26.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.482+0000 7f195dfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19580cd390 0x7f19580cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:47:26.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.482+0000 7f195dfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd8e0 0x7f19581e3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:47:26.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.482+0000 7f195dfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1958153290 0x7f19581e3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:47:26.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.482+0000 7f195dfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1958153d70 con 0x7f19580cd8e0 2023-03-03T06:47:26.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.482+0000 7f195dfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19580b2140 con 0x7f19580cd390 2023-03-03T06:47:26.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.482+0000 7f195dfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19580b3ce0 con 0x7f1958153290 2023-03-03T06:47:26.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.482+0000 7f1956ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd8e0 0x7f19581e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:47:26.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f19577fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19580cd390 0x7f19580cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:47:26.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd8e0 0x7f19581e3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40550/0 (socket says 172.21.15.102:40550) 2023-03-03T06:47:26.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 -- 172.21.15.102:0/1782954938 learned_addr learned my addr 172.21.15.102:0/1782954938 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:47:26.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1957fff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1958153290 0x7f19581e3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:47:26.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 -- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1958153290 msgr2=0x7f19581e3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:47:26.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1958153290 0x7f19581e3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:47:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 -- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19580cd390 msgr2=0x7f19580cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:47:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19580cd390 0x7f19580cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:47:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 -- 172.21.15.102:0/1782954938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1950009a70 con 0x7f19580cd8e0 2023-03-03T06:47:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.483+0000 7f1956ffd700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd8e0 0x7f19581e3530 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f194c00c700 tx=0x7f194c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:47:26.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.484+0000 7f193ffff700 1 -- 172.21.15.102:0/1782954938 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c005d70 con 0x7f19580cd8e0 2023-03-03T06:47:26.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.484+0000 7f193ffff700 1 -- 172.21.15.102:0/1782954938 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c0070d0 con 0x7f19580cd8e0 2023-03-03T06:47:26.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.484+0000 7f193ffff700 1 -- 172.21.15.102:0/1782954938 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c007460 con 0x7f19580cd8e0 2023-03-03T06:47:26.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.484+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19580ccde0 con 0x7f19580cd8e0 2023-03-03T06:47:26.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.484+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195815d160 con 0x7f19580cd8e0 2023-03-03T06:47:26.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.485+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19580cd0c0 con 0x7f19580cd8e0 2023-03-03T06:47:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.486+0000 7f193ffff700 1 -- 172.21.15.102:0/1782954938 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f194c019030 con 0x7f19580cd8e0 2023-03-03T06:47:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.488+0000 7f193ffff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:47:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.488+0000 7f193ffff700 1 -- 172.21.15.102:0/1782954938 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f194c085090 con 0x7f19580cd8e0 2023-03-03T06:47:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.489+0000 7f19577fe700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:47:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.489+0000 7f19577fe700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1948001010 tx=0x7f1948009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:47:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.493+0000 7f193ffff700 1 -- 172.21.15.102:0/1782954938 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f194c0517a0 con 0x7f19580cd8e0 2023-03-03T06:47:26.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.726+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f195804e6e0 con 0x7f19580cd8e0 2023-03-03T06:47:26.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.726+0000 7f193ffff700 1 -- 172.21.15.102:0/1782954938 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f194c017070 con 0x7f19580cd8e0 2023-03-03T06:47:26.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:47:26.728 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:47:26.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.729+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 msgr2=0x7f194406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:47:26.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.729+0000 7f195dfff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1948001010 tx=0x7f1948009040 comp rx=0 tx=0).stop 2023-03-03T06:47:26.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.729+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd8e0 msgr2=0x7f19581e3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:47:26.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.729+0000 7f195dfff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd8e0 0x7f19581e3530 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f194c00c700 tx=0x7f194c00ca10 comp rx=0 tx=0).stop 2023-03-03T06:47:26.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.729+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 shutdown_connections 2023-03-03T06:47:26.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.729+0000 7f195dfff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:47:26.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.730+0000 7f195dfff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1958153290 0x7f19581e3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:47:26.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.730+0000 7f195dfff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19580cd390 0x7f19580cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:47:26.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.730+0000 7f195dfff700 1 --2- 172.21.15.102:0/1782954938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19580cd8e0 0x7f19581e3530 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:47:26.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.730+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 >> 172.21.15.102:0/1782954938 conn(0x7f1958065fb0 msgr2=0x7f19581587d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:47:26.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.730+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 shutdown_connections 2023-03-03T06:47:26.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:47:26.730+0000 7f195dfff700 1 -- 172.21.15.102:0/1782954938 wait complete. 2023-03-03T06:47:26.733 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:47:26.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T06:47:50.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:50.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:00.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:48:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.009+0000 7fdb7ffff700 1 Processor -- start 2023-03-03T06:48:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.010+0000 7fdb7ffff700 1 -- start start 2023-03-03T06:48:01.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdb801564b0 0x7fdb800cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:48:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb800cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:48:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb800c9ca0 con 0x7fdb800c9610 2023-03-03T06:48:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7ffff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb800b4920 con 0x7fdb801564b0 2023-03-03T06:48:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800b4c00 con 0x7fdb800cab50 2023-03-03T06:48:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdb801564b0 0x7fdb800cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:48:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdb801564b0 0x7fdb800cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42832/0 (socket says 172.21.15.102:42832) 2023-03-03T06:48:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7effd700 1 -- 172.21.15.102:0/600329216 learned_addr learned my addr 172.21.15.102:0/600329216 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:48:01.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.011+0000 7fdb7f7fe700 1 --2- 172.21.15.102:0/600329216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb800cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:48:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.012+0000 7fdb7effd700 1 -- 172.21.15.102:0/600329216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 msgr2=0x7fdb800cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:48:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.012+0000 7fdb7effd700 1 --2- 172.21.15.102:0/600329216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb800cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.012+0000 7fdb7effd700 1 -- 172.21.15.102:0/600329216 >> v1:172.21.15.172:6789/0 conn(0x7fdb800c9610 legacy=0x7fdb800ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:48:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.012+0000 7fdb7dffb700 1 -- 172.21.15.102:0/600329216 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88677843 0 0) 0x7fdb800c9ca0 con 0x7fdb800c9610 2023-03-03T06:48:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.012+0000 7fdb7effd700 1 -- 172.21.15.102:0/600329216 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb800b46f0 con 0x7fdb801564b0 2023-03-03T06:48:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.012+0000 7fdb7effd700 1 --2- 172.21.15.102:0/600329216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 0x7fdb800cb660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdb700098f0 tx=0x7fdb70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e73702cbfc79006 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:48:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.012+0000 7fdb7dffb700 1 -- 172.21.15.102:0/600329216 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70011830 con 0x7fdb801564b0 2023-03-03T06:48:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7dffb700 1 -- 172.21.15.102:0/600329216 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb70011990 con 0x7fdb801564b0 2023-03-03T06:48:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7dffb700 1 -- 172.21.15.102:0/600329216 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70011d20 con 0x7fdb801564b0 2023-03-03T06:48:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 -- 172.21.15.102:0/600329216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 msgr2=0x7fdb800cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:48:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/600329216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 0x7fdb800cb660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdb700098f0 tx=0x7fdb70009c00 comp rx=0 tx=0).stop 2023-03-03T06:48:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 -- 172.21.15.102:0/600329216 shutdown_connections 2023-03-03T06:48:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/600329216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb800cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/600329216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 0x7fdb800cb660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 -- 172.21.15.102:0/600329216 >> 172.21.15.102:0/600329216 conn(0x7fdb80065fb0 msgr2=0x7fdb800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:48:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 -- 172.21.15.102:0/600329216 shutdown_connections 2023-03-03T06:48:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.013+0000 7fdb7ffff700 1 -- 172.21.15.102:0/600329216 wait complete. 2023-03-03T06:48:01.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.015+0000 7fdb7ffff700 1 Processor -- start 2023-03-03T06:48:01.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.015+0000 7fdb7ffff700 1 -- start start 2023-03-03T06:48:01.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.016+0000 7fdb7ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb800c9610 0x7fdb800ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:48:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.016+0000 7fdb7ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb80153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:48:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.016+0000 7fdb7ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 0x7fdb801ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:48:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.016+0000 7fdb7ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb801ee2a0 con 0x7fdb801564b0 2023-03-03T06:48:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.016+0000 7fdb7ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8015a0e0 con 0x7fdb800c9610 2023-03-03T06:48:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.016+0000 7fdb7ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8015a380 con 0x7fdb800cab50 2023-03-03T06:48:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb800c9610 0x7fdb800ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:48:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb800c9610 0x7fdb800ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36212/0 (socket says 172.21.15.102:36212) 2023-03-03T06:48:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 -- 172.21.15.102:0/884668206 learned_addr learned my addr 172.21.15.102:0/884668206 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:48:01.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7e7fc700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb80153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:48:01.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7f7fe700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 0x7fdb801ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:48:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 -- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 msgr2=0x7fdb80153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:48:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb80153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 -- 172.21.15.102:0/884668206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 msgr2=0x7fdb801ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:48:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 0x7fdb801ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7effd700 1 -- 172.21.15.102:0/884668206 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb70009550 con 0x7fdb800c9610 2023-03-03T06:48:01.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.017+0000 7fdb7e7fc700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb80153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:48:01.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.018+0000 7fdb7effd700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb800c9610 0x7fdb800ce400 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009c40 tx=0x7fdb70004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:48:01.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.018+0000 7fdb677fe700 1 -- 172.21.15.102:0/884668206 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70004480 con 0x7fdb800c9610 2023-03-03T06:48:01.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.018+0000 7fdb677fe700 1 -- 172.21.15.102:0/884668206 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7001f070 con 0x7fdb800c9610 2023-03-03T06:48:01.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.018+0000 7fdb677fe700 1 -- 172.21.15.102:0/884668206 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb700038a0 con 0x7fdb800c9610 2023-03-03T06:48:01.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.018+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb800b2210 con 0x7fdb800c9610 2023-03-03T06:48:01.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.018+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb800b1f30 con 0x7fdb800c9610 2023-03-03T06:48:01.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.019+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb80156880 con 0x7fdb800c9610 2023-03-03T06:48:01.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.020+0000 7fdb677fe700 1 -- 172.21.15.102:0/884668206 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdb7002a020 con 0x7fdb800c9610 2023-03-03T06:48:01.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.022+0000 7fdb677fe700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb6c0682f0 0x7fdb6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:48:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.022+0000 7fdb677fe700 1 -- 172.21.15.102:0/884668206 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdb700873f0 con 0x7fdb800c9610 2023-03-03T06:48:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.022+0000 7fdb7e7fc700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb6c0682f0 0x7fdb6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:48:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.022+0000 7fdb7e7fc700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb6c0682f0 0x7fdb6c06a7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb800c5980 tx=0x7fdb74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:48:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.026+0000 7fdb677fe700 1 -- 172.21.15.102:0/884668206 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb70053b00 con 0x7fdb800c9610 2023-03-03T06:48:01.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.262+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb801ee580 con 0x7fdb800c9610 2023-03-03T06:48:01.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.263+0000 7fdb677fe700 1 -- 172.21.15.102:0/884668206 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdb70057120 con 0x7fdb800c9610 2023-03-03T06:48:01.266 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:48:01.266 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:48:01.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb6c0682f0 msgr2=0x7fdb6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:48:01.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb6c0682f0 0x7fdb6c06a7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb800c5980 tx=0x7fdb74009040 comp rx=0 tx=0).stop 2023-03-03T06:48:01.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb800c9610 msgr2=0x7fdb800ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:48:01.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb800c9610 0x7fdb800ce400 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009c40 tx=0x7fdb70004750 comp rx=0 tx=0).stop 2023-03-03T06:48:01.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 shutdown_connections 2023-03-03T06:48:01.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb801564b0 0x7fdb801ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb800c9610 0x7fdb800ce400 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb6c0682f0 0x7fdb6c06a7a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 --2- 172.21.15.102:0/884668206 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdb800cab50 0x7fdb80153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:48:01.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 >> 172.21.15.102:0/884668206 conn(0x7fdb80065fb0 msgr2=0x7fdb80151bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:48:01.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 shutdown_connections 2023-03-03T06:48:01.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:48:01.266+0000 7fdb7ffff700 1 -- 172.21.15.102:0/884668206 wait complete. 2023-03-03T06:48:01.270 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:48:01.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 93 secs before thrashing 2023-03-03T06:48:20.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:20.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:50.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:50.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:20.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:20.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:34.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:49:34.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.543+0000 7fd3149ff700 1 Processor -- start 2023-03-03T06:49:34.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.544+0000 7fd3149ff700 1 -- start start 2023-03-03T06:49:34.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.545+0000 7fd3149ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd310153290 0x7fd310158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:49:34.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.545+0000 7fd3149ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3100cd8e0 0x7fd3100ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:49:34.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.545+0000 7fd3149ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd310153660 con 0x7fd3101593c0 2023-03-03T06:49:34.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.545+0000 7fd3149ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31006c360 con 0x7fd310153290 2023-03-03T06:49:34.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.545+0000 7fd3149ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31015cea0 con 0x7fd3100cd8e0 2023-03-03T06:49:34.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.545+0000 7fd30e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd310153290 0x7fd310158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:49:34.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd310153290 0x7fd310158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57542/0 (socket says 172.21.15.102:57542) 2023-03-03T06:49:34.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30e59c700 1 -- 172.21.15.102:0/1962300787 learned_addr learned my addr 172.21.15.102:0/1962300787 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:49:34.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30ed9d700 1 --2- 172.21.15.102:0/1962300787 >> v2:172.21.15.172:3300/0 conn(0x7fd3100cd8e0 0x7fd3100ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:49:34.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30e59c700 1 -- 172.21.15.102:0/1962300787 >> v2:172.21.15.172:3300/0 conn(0x7fd3100cd8e0 msgr2=0x7fd3100ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:49:34.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30e59c700 1 --2- 172.21.15.102:0/1962300787 >> v2:172.21.15.172:3300/0 conn(0x7fd3100cd8e0 0x7fd3100ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30e59c700 1 -- 172.21.15.102:0/1962300787 >> v1:172.21.15.172:6789/0 conn(0x7fd3101593c0 legacy=0x7fd3100cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:49:34.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30e59c700 1 -- 172.21.15.102:0/1962300787 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31006c7f0 con 0x7fd310153290 2023-03-03T06:49:34.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30dd9b700 1 -- 172.21.15.102:0/1962300787 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45194043 0 0) 0x7fd310153660 con 0x7fd3101593c0 2023-03-03T06:49:34.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.546+0000 7fd30e59c700 1 --2- 172.21.15.102:0/1962300787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd310153290 0x7fd310158e80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd3100c9390 tx=0x7fd300009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8369297f4ca44e0c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:49:34.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd30dd9b700 1 -- 172.21.15.102:0/1962300787 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300011830 con 0x7fd310153290 2023-03-03T06:49:34.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd30dd9b700 1 -- 172.21.15.102:0/1962300787 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd300011990 con 0x7fd310153290 2023-03-03T06:49:34.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd30dd9b700 1 -- 172.21.15.102:0/1962300787 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300011d20 con 0x7fd310153290 2023-03-03T06:49:34.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd3149ff700 1 -- 172.21.15.102:0/1962300787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd310153290 msgr2=0x7fd310158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:49:34.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd3149ff700 1 --2- 172.21.15.102:0/1962300787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd310153290 0x7fd310158e80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd3100c9390 tx=0x7fd300009bd0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd3149ff700 1 -- 172.21.15.102:0/1962300787 shutdown_connections 2023-03-03T06:49:34.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd3149ff700 1 --2- 172.21.15.102:0/1962300787 >> v2:172.21.15.172:3300/0 conn(0x7fd3100cd8e0 0x7fd3100ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd3149ff700 1 --2- 172.21.15.102:0/1962300787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd310153290 0x7fd310158e80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd3149ff700 1 -- 172.21.15.102:0/1962300787 >> 172.21.15.102:0/1962300787 conn(0x7fd310065fb0 msgr2=0x7fd3100663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:49:34.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.547+0000 7fd3149ff700 1 -- 172.21.15.102:0/1962300787 shutdown_connections 2023-03-03T06:49:34.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.548+0000 7fd3149ff700 1 -- 172.21.15.102:0/1962300787 wait complete. 2023-03-03T06:49:34.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.550+0000 7fd3149ff700 1 Processor -- start 2023-03-03T06:49:34.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.550+0000 7fd3149ff700 1 -- start start 2023-03-03T06:49:34.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd3149ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3100cd8e0 0x7fd3100b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:49:34.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd3149ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd310153290 0x7fd3100c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:49:34.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd3149ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3101593c0 0x7fd31014dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd3149ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3101eef20 con 0x7fd3101593c0 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd3149ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3101ee650 con 0x7fd3100cd8e0 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd3149ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd310156130 con 0x7fd310153290 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd30e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3100cd8e0 0x7fd3100b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd30e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3100cd8e0 0x7fd3100b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45312/0 (socket says 172.21.15.102:45312) 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd30e59c700 1 -- 172.21.15.102:0/2299144181 learned_addr learned my addr 172.21.15.102:0/2299144181 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd307fff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd310153290 0x7fd3100c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:49:34.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.551+0000 7fd30ed9d700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3101593c0 0x7fd31014dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:49:34.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd30e59c700 1 -- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd310153290 msgr2=0x7fd3100c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:49:34.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd30e59c700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd310153290 0x7fd3100c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd30e59c700 1 -- 172.21.15.102:0/2299144181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3101593c0 msgr2=0x7fd31014dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:49:34.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd30e59c700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3101593c0 0x7fd31014dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd30e59c700 1 -- 172.21.15.102:0/2299144181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd300009550 con 0x7fd3100cd8e0 2023-03-03T06:49:34.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd30e59c700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3100cd8e0 0x7fd3100b2cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd300009c10 tx=0x7fd300004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:49:34.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd306ffd700 1 -- 172.21.15.102:0/2299144181 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300004510 con 0x7fd3100cd8e0 2023-03-03T06:49:34.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd306ffd700 1 -- 172.21.15.102:0/2299144181 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3000036a0 con 0x7fd3100cd8e0 2023-03-03T06:49:34.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd306ffd700 1 -- 172.21.15.102:0/2299144181 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300003a10 con 0x7fd3100cd8e0 2023-03-03T06:49:34.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.552+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3100ccde0 con 0x7fd3100cd8e0 2023-03-03T06:49:34.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.553+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31015d160 con 0x7fd3100cd8e0 2023-03-03T06:49:34.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.554+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3100cd0c0 con 0x7fd3100cd8e0 2023-03-03T06:49:34.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.555+0000 7fd306ffd700 1 -- 172.21.15.102:0/2299144181 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd30002d070 con 0x7fd3100cd8e0 2023-03-03T06:49:34.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.556+0000 7fd306ffd700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2f0068320 0x7fd2f006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:49:34.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.556+0000 7fd306ffd700 1 -- 172.21.15.102:0/2299144181 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd300021030 con 0x7fd3100cd8e0 2023-03-03T06:49:34.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.556+0000 7fd307fff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2f0068320 0x7fd2f006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:49:34.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.557+0000 7fd307fff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2f0068320 0x7fd2f006a7d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd310157320 tx=0x7fd2f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:49:34.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.560+0000 7fd306ffd700 1 -- 172.21.15.102:0/2299144181 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd300053b10 con 0x7fd3100cd8e0 2023-03-03T06:49:34.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.793+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd310157370 con 0x7fd3100cd8e0 2023-03-03T06:49:34.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.794+0000 7fd306ffd700 1 -- 172.21.15.102:0/2299144181 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd300015090 con 0x7fd3100cd8e0 2023-03-03T06:49:34.797 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:49:34.797 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:49:34.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.796+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2f0068320 msgr2=0x7fd2f006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:49:34.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.796+0000 7fd3149ff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2f0068320 0x7fd2f006a7d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd310157320 tx=0x7fd2f8009040 comp rx=0 tx=0).stop 2023-03-03T06:49:34.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3100cd8e0 msgr2=0x7fd3100b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:49:34.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3100cd8e0 0x7fd3100b2cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd300009c10 tx=0x7fd300004750 comp rx=0 tx=0).stop 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 shutdown_connections 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3101593c0 0x7fd31014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3100cd8e0 0x7fd3100b2cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2f0068320 0x7fd2f006a7d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 --2- 172.21.15.102:0/2299144181 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd310153290 0x7fd3100c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 >> 172.21.15.102:0/2299144181 conn(0x7fd310065fb0 msgr2=0x7fd310150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 shutdown_connections 2023-03-03T06:49:34.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:49:34.797+0000 7fd3149ff700 1 -- 172.21.15.102:0/2299144181 wait complete. 2023-03-03T06:49:34.801 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:49:34.827 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2023-03-03T06:49:51.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:51.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T06:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:50:21.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:21.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:51.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:51.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:21.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:21.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:26.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:51:27.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.081+0000 7f0c957ff700 1 Processor -- start 2023-03-03T06:51:27.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.081+0000 7f0c957ff700 1 -- start start 2023-03-03T06:51:27.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c957ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0c900cd8e0 0x7f0c900cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:51:27.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c957ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:51:27.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c957ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c90153660 con 0x7f0c90153290 2023-03-03T06:51:27.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c957ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9006c360 con 0x7f0c900cd8e0 2023-03-03T06:51:27.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c957ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9015cea0 con 0x7f0c900cd390 2023-03-03T06:51:27.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c8effd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0c90153290 0x7f0c90158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40140/0 (socket says 172.21.15.102:40140) 2023-03-03T06:51:27.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c8effd700 1 -- 172.21.15.102:0/3758015066 learned_addr learned my addr 172.21.15.102:0/3758015066 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:51:27.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c8f7fe700 1 --2- 172.21.15.102:0/3758015066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:51:27.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.083+0000 7f0c8e7fc700 1 --2- 172.21.15.102:0/3758015066 >> v2:172.21.15.172:3300/0 conn(0x7f0c900cd8e0 0x7f0c900cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:51:27.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.084+0000 7f0c8f7fe700 1 -- 172.21.15.102:0/3758015066 >> v2:172.21.15.172:3300/0 conn(0x7f0c900cd8e0 msgr2=0x7f0c900cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:51:27.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.084+0000 7f0c8f7fe700 1 --2- 172.21.15.102:0/3758015066 >> v2:172.21.15.172:3300/0 conn(0x7f0c900cd8e0 0x7f0c900cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.084+0000 7f0c8f7fe700 1 -- 172.21.15.102:0/3758015066 >> v1:172.21.15.172:6789/0 conn(0x7f0c90153290 legacy=0x7f0c90158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:51:27.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.084+0000 7f0c8f7fe700 1 -- 172.21.15.102:0/3758015066 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9006c7f0 con 0x7f0c900cd390 2023-03-03T06:51:27.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.084+0000 7f0c8e7fc700 1 --2- 172.21.15.102:0/3758015066 >> v2:172.21.15.172:3300/0 conn(0x7f0c900cd8e0 0x7f0c900cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:51:27.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.084+0000 7f0c8f7fe700 1 --2- 172.21.15.102:0/3758015066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900ce360 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009d90 tx=0x7f0c8800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6955bbb5520913d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:51:27.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c8dffb700 1 -- 172.21.15.102:0/3758015066 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8800e070 con 0x7f0c900cd390 2023-03-03T06:51:27.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c8dffb700 1 -- 172.21.15.102:0/3758015066 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8800bb20 con 0x7f0c900cd390 2023-03-03T06:51:27.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c8dffb700 1 -- 172.21.15.102:0/3758015066 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88012430 con 0x7f0c900cd390 2023-03-03T06:51:27.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 -- 172.21.15.102:0/3758015066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 msgr2=0x7f0c900ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:51:27.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 --2- 172.21.15.102:0/3758015066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900ce360 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009d90 tx=0x7f0c8800b330 comp rx=0 tx=0).stop 2023-03-03T06:51:27.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 -- 172.21.15.102:0/3758015066 shutdown_connections 2023-03-03T06:51:27.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 --2- 172.21.15.102:0/3758015066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900ce360 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 --2- 172.21.15.102:0/3758015066 >> v2:172.21.15.172:3300/0 conn(0x7f0c900cd8e0 0x7f0c900cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 -- 172.21.15.102:0/3758015066 >> 172.21.15.102:0/3758015066 conn(0x7f0c90065fb0 msgr2=0x7f0c900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:51:27.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 -- 172.21.15.102:0/3758015066 shutdown_connections 2023-03-03T06:51:27.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.086+0000 7f0c957ff700 1 -- 172.21.15.102:0/3758015066 wait complete. 2023-03-03T06:51:27.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.089+0000 7f0c957ff700 1 Processor -- start 2023-03-03T06:51:27.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.089+0000 7f0c957ff700 1 -- start start 2023-03-03T06:51:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.090+0000 7f0c957ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900c3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:51:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.090+0000 7f0c957ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c900cd8e0 0x7f0c900b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:51:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.090+0000 7f0c957ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c90153290 0x7f0c900c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:51:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.090+0000 7f0c957ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9014dd10 con 0x7f0c900cd8e0 2023-03-03T06:51:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.090+0000 7f0c957ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c900c2ec0 con 0x7f0c90153290 2023-03-03T06:51:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.090+0000 7f0c957ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9014df60 con 0x7f0c900cd390 2023-03-03T06:51:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.090+0000 7f0c8e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c900cd8e0 0x7f0c900b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:51:27.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c90153290 0x7f0c900c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:51:27.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900c3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:51:27.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c90153290 0x7f0c900c7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50120/0 (socket says 172.21.15.102:50120) 2023-03-03T06:51:27.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900c3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60102/0 (socket says 172.21.15.102:60102) 2023-03-03T06:51:27.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8f7fe700 1 -- 172.21.15.102:0/1056353854 learned_addr learned my addr 172.21.15.102:0/1056353854 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:51:27.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 -- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c90153290 msgr2=0x7f0c900c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c90153290 0x7f0c900c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 -- 172.21.15.102:0/1056353854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c900cd8e0 msgr2=0x7f0c900b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c900cd8e0 0x7f0c900b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 -- 172.21.15.102:0/1056353854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c840094a0 con 0x7f0c900cd390 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8f7fe700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c90153290 0x7f0c900c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.091+0000 7f0c8effd700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900c3da0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000c530 tx=0x7f0c8000c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.092+0000 7f0c777fe700 1 -- 172.21.15.102:0/1056353854 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80005d70 con 0x7f0c900cd390 2023-03-03T06:51:27.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.092+0000 7f0c777fe700 1 -- 172.21.15.102:0/1056353854 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8000ce40 con 0x7f0c900cd390 2023-03-03T06:51:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.092+0000 7f0c777fe700 1 -- 172.21.15.102:0/1056353854 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c800072d0 con 0x7f0c900cd390 2023-03-03T06:51:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.092+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c88009a70 con 0x7f0c900cd390 2023-03-03T06:51:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.092+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c900ccde0 con 0x7f0c900cd390 2023-03-03T06:51:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.093+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9015d160 con 0x7f0c900cd390 2023-03-03T06:51:27.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.094+0000 7f0c777fe700 1 -- 172.21.15.102:0/1056353854 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0c80019030 con 0x7f0c900cd390 2023-03-03T06:51:27.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.095+0000 7f0c777fe700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0c7c0682f0 0x7f0c7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:51:27.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.096+0000 7f0c777fe700 1 -- 172.21.15.102:0/1056353854 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0c80084de0 con 0x7f0c900cd390 2023-03-03T06:51:27.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.096+0000 7f0c8e7fc700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0c7c0682f0 0x7f0c7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:51:27.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.096+0000 7f0c8e7fc700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0c7c0682f0 0x7f0c7c06a7a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0c900cb0f0 tx=0x7f0c84009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:51:27.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.100+0000 7f0c777fe700 1 -- 172.21.15.102:0/1056353854 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0c8004b5e0 con 0x7f0c900cd390 2023-03-03T06:51:27.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.331+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c900cd0c0 con 0x7f0c900cd390 2023-03-03T06:51:27.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.332+0000 7f0c777fe700 1 -- 172.21.15.102:0/1056353854 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0c80017090 con 0x7f0c900cd390 2023-03-03T06:51:27.333 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:51:27.334 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:51:27.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0c7c0682f0 msgr2=0x7f0c7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:51:27.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0c7c0682f0 0x7f0c7c06a7a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0c900cb0f0 tx=0x7f0c84009210 comp rx=0 tx=0).stop 2023-03-03T06:51:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 msgr2=0x7f0c900c3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:51:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900c3da0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000c530 tx=0x7f0c8000c840 comp rx=0 tx=0).stop 2023-03-03T06:51:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 shutdown_connections 2023-03-03T06:51:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0c7c0682f0 0x7f0c7c06a7a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0c900cd390 0x7f0c900c3da0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0c90153290 0x7f0c900c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 --2- 172.21.15.102:0/1056353854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c900cd8e0 0x7f0c900b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:51:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 >> 172.21.15.102:0/1056353854 conn(0x7f0c90065fb0 msgr2=0x7f0c900cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:51:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 shutdown_connections 2023-03-03T06:51:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:51:27.335+0000 7f0c957ff700 1 -- 172.21.15.102:0/1056353854 wait complete. 2023-03-03T06:51:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:51:27.363 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T06:51:51.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:51.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:22.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:22.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:52.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:52.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:04.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:53:04.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.622+0000 7fed0cfff700 1 Processor -- start 2023-03-03T06:53:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.623+0000 7fed0cfff700 1 -- start start 2023-03-03T06:53:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed0cfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fed08153290 0x7fed08158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed0cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed081593c0 0x7fed080cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed0cfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed08153660 con 0x7fed080cd8e0 2023-03-03T06:53:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed0cfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0806c360 con 0x7fed08153290 2023-03-03T06:53:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed0cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0815cea0 con 0x7fed081593c0 2023-03-03T06:53:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed06d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fed080cd8e0 0x7fed080ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56272/0 (socket says 172.21.15.102:56272) 2023-03-03T06:53:04.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed06d9d700 1 -- 172.21.15.102:0/823448266 learned_addr learned my addr 172.21.15.102:0/823448266 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:53:04.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed05d9b700 1 --2- 172.21.15.102:0/823448266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed081593c0 0x7fed080cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:04.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.624+0000 7fed0659c700 1 --2- 172.21.15.102:0/823448266 >> v2:172.21.15.172:3300/0 conn(0x7fed08153290 0x7fed08158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:04.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed05d9b700 1 -- 172.21.15.102:0/823448266 >> v2:172.21.15.172:3300/0 conn(0x7fed08153290 msgr2=0x7fed08158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:53:04.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed05d9b700 1 --2- 172.21.15.102:0/823448266 >> v2:172.21.15.172:3300/0 conn(0x7fed08153290 0x7fed08158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed05d9b700 1 -- 172.21.15.102:0/823448266 >> v1:172.21.15.102:6789/0 conn(0x7fed080cd8e0 legacy=0x7fed080ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed05d9b700 1 -- 172.21.15.102:0/823448266 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0806c7f0 con 0x7fed081593c0 2023-03-03T06:53:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed0559a700 1 -- 172.21.15.102:0/823448266 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921729383 0 0) 0x7fed08153660 con 0x7fed080cd8e0 2023-03-03T06:53:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed05d9b700 1 --2- 172.21.15.102:0/823448266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed081593c0 0x7fed080cde20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009840 tx=0x7fecfc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a1196d6697ccbf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed0559a700 1 -- 172.21.15.102:0/823448266 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc011830 con 0x7fed081593c0 2023-03-03T06:53:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed0559a700 1 -- 172.21.15.102:0/823448266 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc011990 con 0x7fed081593c0 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.625+0000 7fed0559a700 1 -- 172.21.15.102:0/823448266 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc011d20 con 0x7fed081593c0 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 -- 172.21.15.102:0/823448266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed081593c0 msgr2=0x7fed080cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 --2- 172.21.15.102:0/823448266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed081593c0 0x7fed080cde20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009840 tx=0x7fecfc009b50 comp rx=0 tx=0).stop 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 -- 172.21.15.102:0/823448266 shutdown_connections 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 --2- 172.21.15.102:0/823448266 >> v2:172.21.15.172:3300/0 conn(0x7fed08153290 0x7fed08158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 --2- 172.21.15.102:0/823448266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed081593c0 0x7fed080cde20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 -- 172.21.15.102:0/823448266 >> 172.21.15.102:0/823448266 conn(0x7fed08065fb0 msgr2=0x7fed080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:53:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 -- 172.21.15.102:0/823448266 shutdown_connections 2023-03-03T06:53:04.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.626+0000 7fed0cfff700 1 -- 172.21.15.102:0/823448266 wait complete. 2023-03-03T06:53:04.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.628+0000 7fed0cfff700 1 Processor -- start 2023-03-03T06:53:04.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.628+0000 7fed0cfff700 1 -- start start 2023-03-03T06:53:04.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.629+0000 7fed0cfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080cd8e0 0x7fed0814e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.629+0000 7fed0cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed08153290 0x7fed080c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.629+0000 7fed0cfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed081593c0 0x7fed081eee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.629+0000 7fed0cfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0814ea90 con 0x7fed080cd8e0 2023-03-03T06:53:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.629+0000 7fed0cfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080c8bc0 con 0x7fed081593c0 2023-03-03T06:53:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.629+0000 7fed0cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed080c7be0 con 0x7fed08153290 2023-03-03T06:53:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080cd8e0 0x7fed0814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080cd8e0 0x7fed0814e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41036/0 (socket says 172.21.15.102:41036) 2023-03-03T06:53:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 -- 172.21.15.102:0/1210847526 learned_addr learned my addr 172.21.15.102:0/1210847526 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:53:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed05d9b700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed08153290 0x7fed080c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed06d9d700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed081593c0 0x7fed081eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 -- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed08153290 msgr2=0x7fed080c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed08153290 0x7fed080c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 -- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed081593c0 msgr2=0x7fed081eee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed081593c0 0x7fed081eee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed0659c700 1 -- 172.21.15.102:0/1210847526 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0094a0 con 0x7fed080cd8e0 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.630+0000 7fed05d9b700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed08153290 0x7fed080c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.631+0000 7fed0659c700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080cd8e0 0x7fed0814e550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fecf800e8a0 tx=0x7fecf800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.631+0000 7fecf6ffd700 1 -- 172.21.15.102:0/1210847526 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8009590 con 0x7fed080cd8e0 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.631+0000 7fecf6ffd700 1 -- 172.21.15.102:0/1210847526 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf8013680 con 0x7fed080cd8e0 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.631+0000 7fecf6ffd700 1 -- 172.21.15.102:0/1210847526 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf80139f0 con 0x7fed080cd8e0 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.631+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed080ccde0 con 0x7fed080cd8e0 2023-03-03T06:53:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.631+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0815d160 con 0x7fed080cd8e0 2023-03-03T06:53:04.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.632+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed080cd0c0 con 0x7fed080cd8e0 2023-03-03T06:53:04.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.634+0000 7fecf6ffd700 1 -- 172.21.15.102:0/1210847526 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fecf8019030 con 0x7fed080cd8e0 2023-03-03T06:53:04.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.636+0000 7fecf6ffd700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fece80682f0 0x7fece806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:04.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.636+0000 7fecf6ffd700 1 -- 172.21.15.102:0/1210847526 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fecf8085b00 con 0x7fed080cd8e0 2023-03-03T06:53:04.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.636+0000 7fed05d9b700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fece80682f0 0x7fece806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:04.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.637+0000 7fed05d9b700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fece80682f0 0x7fece806a7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fecfc000c00 tx=0x7fecfc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:04.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.640+0000 7fecf6ffd700 1 -- 172.21.15.102:0/1210847526 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fecf8087e60 con 0x7fed080cd8e0 2023-03-03T06:53:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.876+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed080c9490 con 0x7fed080cd8e0 2023-03-03T06:53:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.877+0000 7fecf6ffd700 1 -- 172.21.15.102:0/1210847526 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fecf8021090 con 0x7fed080cd8e0 2023-03-03T06:53:04.879 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:53:04.879 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:53:04.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fece80682f0 msgr2=0x7fece806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:04.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fece80682f0 0x7fece806a7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fecfc000c00 tx=0x7fecfc008030 comp rx=0 tx=0).stop 2023-03-03T06:53:04.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080cd8e0 msgr2=0x7fed0814e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:04.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080cd8e0 0x7fed0814e550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fecf800e8a0 tx=0x7fecf800ebb0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 shutdown_connections 2023-03-03T06:53:04.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed081593c0 0x7fed081eee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080cd8e0 0x7fed0814e550 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fece80682f0 0x7fece806a7a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 --2- 172.21.15.102:0/1210847526 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed08153290 0x7fed080c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 >> 172.21.15.102:0/1210847526 conn(0x7fed08065fb0 msgr2=0x7fed080cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:53:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 shutdown_connections 2023-03-03T06:53:04.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:04.880+0000 7fed0cfff700 1 -- 172.21.15.102:0/1210847526 wait complete. 2023-03-03T06:53:04.884 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:53:04.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T06:53:22.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:22.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:32.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:53:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.175+0000 7fdad07ff700 1 Processor -- start 2023-03-03T06:53:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.176+0000 7fdad07ff700 1 -- start start 2023-03-03T06:53:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdad07ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdac80cd8e0 0x7fdac80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdad07ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac8153660 con 0x7fdac8153290 2023-03-03T06:53:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdad07ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac806c360 con 0x7fdac80cd390 2023-03-03T06:53:33.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdad07ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac815cea0 con 0x7fdac80cd8e0 2023-03-03T06:53:33.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdace59b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdac8153290 0x7fdac8158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33350/0 (socket says 172.21.15.102:33350) 2023-03-03T06:53:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdacdd9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdac80cd8e0 0x7fdac80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdace59b700 1 -- 172.21.15.102:0/2465150634 learned_addr learned my addr 172.21.15.102:0/2465150634 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:53:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.177+0000 7fdaced9c700 1 --1- 172.21.15.102:0/2465150634 >> v1:172.21.15.172:6789/0 conn(0x7fdac80cd390 0x7fdac80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47178/0 (socket says 172.21.15.102:47178) 2023-03-03T06:53:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacd599700 1 -- 172.21.15.102:0/2465150634 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996813288 0 0) 0x7fdac8153660 con 0x7fdac8153290 2023-03-03T06:53:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacd599700 1 -- 172.21.15.102:0/2465150634 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab40035b0 con 0x7fdac8153290 2023-03-03T06:53:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacd599700 1 -- 172.21.15.102:0/2465150634 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713791225 0 0) 0x7fdac806c360 con 0x7fdac80cd390 2023-03-03T06:53:33.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacd599700 1 -- 172.21.15.102:0/2465150634 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac8153660 con 0x7fdac80cd390 2023-03-03T06:53:33.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacdd9a700 1 -- 172.21.15.102:0/2465150634 >> v1:172.21.15.172:6789/0 conn(0x7fdac80cd390 legacy=0x7fdac80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:33.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacdd9a700 1 -- 172.21.15.102:0/2465150634 >> v1:172.21.15.102:6789/0 conn(0x7fdac8153290 legacy=0x7fdac8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:33.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacdd9a700 1 -- 172.21.15.102:0/2465150634 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac806c7f0 con 0x7fdac80cd8e0 2023-03-03T06:53:33.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.178+0000 7fdacdd9a700 1 --2- 172.21.15.102:0/2465150634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac80cd8e0 0x7fdac80cde20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009840 tx=0x7fdab8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=393e5c6bf552b93c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:33.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdacd599700 1 -- 172.21.15.102:0/2465150634 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab8011830 con 0x7fdac80cd8e0 2023-03-03T06:53:33.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdacd599700 1 -- 172.21.15.102:0/2465150634 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab8011990 con 0x7fdac80cd8e0 2023-03-03T06:53:33.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdacd599700 1 -- 172.21.15.102:0/2465150634 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab8011d20 con 0x7fdac80cd8e0 2023-03-03T06:53:33.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdad07ff700 1 -- 172.21.15.102:0/2465150634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac80cd8e0 msgr2=0x7fdac80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:33.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdad07ff700 1 --2- 172.21.15.102:0/2465150634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac80cd8e0 0x7fdac80cde20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009840 tx=0x7fdab8009b50 comp rx=0 tx=0).stop 2023-03-03T06:53:33.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdad07ff700 1 -- 172.21.15.102:0/2465150634 shutdown_connections 2023-03-03T06:53:33.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdad07ff700 1 --2- 172.21.15.102:0/2465150634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac80cd8e0 0x7fdac80cde20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdad07ff700 1 -- 172.21.15.102:0/2465150634 >> 172.21.15.102:0/2465150634 conn(0x7fdac8065fb0 msgr2=0x7fdac80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:53:33.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.179+0000 7fdad07ff700 1 -- 172.21.15.102:0/2465150634 shutdown_connections 2023-03-03T06:53:33.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.180+0000 7fdad07ff700 1 -- 172.21.15.102:0/2465150634 wait complete. 2023-03-03T06:53:33.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.181+0000 7fdad07ff700 1 Processor -- start 2023-03-03T06:53:33.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.182+0000 7fdad07ff700 1 -- start start 2023-03-03T06:53:33.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdad07ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdac80cd390 0x7fdac8155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:33.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdad07ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdac80cd8e0 0x7fdac80c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:33.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdad07ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac8153290 0x7fdac814e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:33.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdad07ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac80c7d90 con 0x7fdac80cd390 2023-03-03T06:53:33.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdad07ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac80c8e20 con 0x7fdac8153290 2023-03-03T06:53:33.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdad07ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac80c9100 con 0x7fdac80cd8e0 2023-03-03T06:53:33.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdace59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdac80cd390 0x7fdac8155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:33.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdace59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdac80cd390 0x7fdac8155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53174/0 (socket says 172.21.15.102:53174) 2023-03-03T06:53:33.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdace59b700 1 -- 172.21.15.102:0/1709492473 learned_addr learned my addr 172.21.15.102:0/1709492473 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:53:33.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdaced9c700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac8153290 0x7fdac814e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:33.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.183+0000 7fdacdd9a700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdac80cd8e0 0x7fdac80c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:33.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdace59b700 1 -- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdac80cd8e0 msgr2=0x7fdac80c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:33.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdace59b700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdac80cd8e0 0x7fdac80c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdace59b700 1 -- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac8153290 msgr2=0x7fdac814e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:33.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdace59b700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac8153290 0x7fdac814e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdace59b700 1 -- 172.21.15.102:0/1709492473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab80094a0 con 0x7fdac80cd390 2023-03-03T06:53:33.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdace59b700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdac80cd390 0x7fdac8155a20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdac400c4d0 tx=0x7fdac400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:33.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdabeffd700 1 -- 172.21.15.102:0/1709492473 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4005d70 con 0x7fdac80cd390 2023-03-03T06:53:33.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdabeffd700 1 -- 172.21.15.102:0/1709492473 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac400ce60 con 0x7fdac80cd390 2023-03-03T06:53:33.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac80ccde0 con 0x7fdac80cd390 2023-03-03T06:53:33.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdabeffd700 1 -- 172.21.15.102:0/1709492473 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40072f0 con 0x7fdac80cd390 2023-03-03T06:53:33.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.184+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac815d160 con 0x7fdac80cd390 2023-03-03T06:53:33.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.185+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac80cd0c0 con 0x7fdac80cd390 2023-03-03T06:53:33.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.187+0000 7fdabeffd700 1 -- 172.21.15.102:0/1709492473 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdac4019030 con 0x7fdac80cd390 2023-03-03T06:53:33.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.188+0000 7fdabeffd700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdaac0682f0 0x7fdaac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:33.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.188+0000 7fdabeffd700 1 -- 172.21.15.102:0/1709492473 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdac4084fe0 con 0x7fdac80cd390 2023-03-03T06:53:33.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.189+0000 7fdacdd9a700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdaac0682f0 0x7fdaac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:33.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.189+0000 7fdacdd9a700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdaac0682f0 0x7fdaac06a7a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009fd0 tx=0x7fdab801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:33.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.192+0000 7fdabeffd700 1 -- 172.21.15.102:0/1709492473 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdac4051750 con 0x7fdac80cd390 2023-03-03T06:53:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.422+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdac804e6e0 con 0x7fdac80cd390 2023-03-03T06:53:33.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.423+0000 7fdabeffd700 1 -- 172.21.15.102:0/1709492473 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdac4017090 con 0x7fdac80cd390 2023-03-03T06:53:33.426 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:53:33.426 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:53:33.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdaac0682f0 msgr2=0x7fdaac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:33.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdaac0682f0 0x7fdaac06a7a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009fd0 tx=0x7fdab801c040 comp rx=0 tx=0).stop 2023-03-03T06:53:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdac80cd390 msgr2=0x7fdac8155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdac80cd390 0x7fdac8155a20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdac400c4d0 tx=0x7fdac400c7e0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 shutdown_connections 2023-03-03T06:53:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdac8153290 0x7fdac814e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdac80cd390 0x7fdac8155a20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdaac0682f0 0x7fdaac06a7a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 --2- 172.21.15.102:0/1709492473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdac80cd8e0 0x7fdac80c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:33.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 >> 172.21.15.102:0/1709492473 conn(0x7fdac8065fb0 msgr2=0x7fdac8158b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:53:33.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 shutdown_connections 2023-03-03T06:53:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:33.426+0000 7fdad07ff700 1 -- 172.21.15.102:0/1709492473 wait complete. 2023-03-03T06:53:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:53:33.455 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-03-03T06:53:36.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:53:36.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.710+0000 7f30cfbff700 1 Processor -- start 2023-03-03T06:53:36.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.710+0000 7f30cfbff700 1 -- start start 2023-03-03T06:53:36.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cfbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30c80ced80 0x7f30c80cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:36.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cfbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f30c80cf9a0 0x7f30c8163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:36.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cfbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c80cb9f0 con 0x7f30c81597f0 2023-03-03T06:53:36.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cfbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80ca750 con 0x7f30c80cf9a0 2023-03-03T06:53:36.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cfbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c815a230 con 0x7f30c80ced80 2023-03-03T06:53:36.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cd99b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30c80ced80 0x7f30c80cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:36.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30ce19c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f30c80cf9a0 0x7f30c8163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cd19a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f30c81597f0 0x7f30c80cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42780/0 (socket says 172.21.15.102:42780) 2023-03-03T06:53:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.712+0000 7f30cd19a700 1 -- 172.21.15.102:0/2571320383 learned_addr learned my addr 172.21.15.102:0/2571320383 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:53:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cd99b700 1 -- 172.21.15.102:0/2571320383 >> v2:172.21.15.102:3300/0 conn(0x7f30c80cf9a0 msgr2=0x7f30c8163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:53:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cd99b700 1 --2- 172.21.15.102:0/2571320383 >> v2:172.21.15.102:3300/0 conn(0x7f30c80cf9a0 0x7f30c8163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cd99b700 1 -- 172.21.15.102:0/2571320383 >> v1:172.21.15.102:6789/0 conn(0x7f30c81597f0 legacy=0x7f30c80cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cc999700 1 -- 172.21.15.102:0/2571320383 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450396795 0 0) 0x7f30c80cb9f0 con 0x7f30c81597f0 2023-03-03T06:53:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cd99b700 1 -- 172.21.15.102:0/2571320383 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c8159c80 con 0x7f30c80ced80 2023-03-03T06:53:36.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cd99b700 1 --2- 172.21.15.102:0/2571320383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80cf150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30c80699d0 tx=0x7f30c4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ef7e318acde862e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:36.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cc999700 1 -- 172.21.15.102:0/2571320383 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4011830 con 0x7f30c80ced80 2023-03-03T06:53:36.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cc999700 1 -- 172.21.15.102:0/2571320383 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4011990 con 0x7f30c80ced80 2023-03-03T06:53:36.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.713+0000 7f30cc999700 1 -- 172.21.15.102:0/2571320383 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4011d20 con 0x7f30c80ced80 2023-03-03T06:53:36.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 -- 172.21.15.102:0/2571320383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 msgr2=0x7f30c80cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:36.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 --2- 172.21.15.102:0/2571320383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80cf150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30c80699d0 tx=0x7f30c4009b20 comp rx=0 tx=0).stop 2023-03-03T06:53:36.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 -- 172.21.15.102:0/2571320383 shutdown_connections 2023-03-03T06:53:36.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 --2- 172.21.15.102:0/2571320383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80cf150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 --2- 172.21.15.102:0/2571320383 >> v2:172.21.15.102:3300/0 conn(0x7f30c80cf9a0 0x7f30c8163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 -- 172.21.15.102:0/2571320383 >> 172.21.15.102:0/2571320383 conn(0x7f30c8065fb0 msgr2=0x7f30c80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:53:36.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 -- 172.21.15.102:0/2571320383 shutdown_connections 2023-03-03T06:53:36.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.714+0000 7f30cfbff700 1 -- 172.21.15.102:0/2571320383 wait complete. 2023-03-03T06:53:36.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.716+0000 7f30cfbff700 1 Processor -- start 2023-03-03T06:53:36.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.716+0000 7f30cfbff700 1 -- start start 2023-03-03T06:53:36.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.717+0000 7f30cfbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:36.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.717+0000 7f30cfbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cf9a0 0x7f30c81f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:36.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.717+0000 7f30cfbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c81597f0 0x7f30c80afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:36.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.717+0000 7f30cfbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c80c6f10 con 0x7f30c81597f0 2023-03-03T06:53:36.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.717+0000 7f30cfbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81f2900 con 0x7f30c80ced80 2023-03-03T06:53:36.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.717+0000 7f30cfbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81f2be0 con 0x7f30c80cf9a0 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30ce19c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c81597f0 0x7f30c80afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30cd99b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30cd19a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cf9a0 0x7f30c81f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30cd19a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cf9a0 0x7f30c81f2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51532/0 (socket says 172.21.15.102:51532) 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30cd19a700 1 -- 172.21.15.102:0/4211220443 learned_addr learned my addr 172.21.15.102:0/4211220443 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30cd99b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60958/0 (socket says 172.21.15.102:60958) 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30ce19c700 1 -- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cf9a0 msgr2=0x7f30c81f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:36.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30ce19c700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cf9a0 0x7f30c81f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30ce19c700 1 -- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 msgr2=0x7f30c80b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:36.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30ce19c700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.718+0000 7f30ce19c700 1 -- 172.21.15.102:0/4211220443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c40094a0 con 0x7f30c81597f0 2023-03-03T06:53:36.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.719+0000 7f30ce19c700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c81597f0 0x7f30c80afd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30c000eba0 tx=0x7f30c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:36.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.719+0000 7f30be7fc700 1 -- 172.21.15.102:0/4211220443 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0009a80 con 0x7f30c81597f0 2023-03-03T06:53:36.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.719+0000 7f30be7fc700 1 -- 172.21.15.102:0/4211220443 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c000f040 con 0x7f30c81597f0 2023-03-03T06:53:36.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.719+0000 7f30be7fc700 1 -- 172.21.15.102:0/4211220443 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c000cc40 con 0x7f30c81597f0 2023-03-03T06:53:36.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.719+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c80cb0c0 con 0x7f30c81597f0 2023-03-03T06:53:36.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.719+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c80cade0 con 0x7f30c81597f0 2023-03-03T06:53:36.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.720+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c80cb3a0 con 0x7f30c81597f0 2023-03-03T06:53:36.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.721+0000 7f30be7fc700 1 -- 172.21.15.102:0/4211220443 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30c0014480 con 0x7f30c81597f0 2023-03-03T06:53:36.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.723+0000 7f30be7fc700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac0682f0 0x7f30ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:53:36.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.723+0000 7f30be7fc700 1 -- 172.21.15.102:0/4211220443 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f30c00869f0 con 0x7f30c81597f0 2023-03-03T06:53:36.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.723+0000 7f30cd99b700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac0682f0 0x7f30ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:53:36.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.724+0000 7f30cd99b700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac0682f0 0x7f30ac06a7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f30c40090b0 tx=0x7f30c401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:53:36.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.727+0000 7f30be7fc700 1 -- 172.21.15.102:0/4211220443 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30c0053010 con 0x7f30c81597f0 2023-03-03T06:53:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.958+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c80677f0 con 0x7f30c81597f0 2023-03-03T06:53:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.959+0000 7f30be7fc700 1 -- 172.21.15.102:0/4211220443 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f30c0018070 con 0x7f30c81597f0 2023-03-03T06:53:36.961 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:53:36.961 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:53:36.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac0682f0 msgr2=0x7f30ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac0682f0 0x7f30ac06a7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f30c40090b0 tx=0x7f30c401c040 comp rx=0 tx=0).stop 2023-03-03T06:53:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c81597f0 msgr2=0x7f30c80afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:53:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c81597f0 0x7f30c80afd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30c000eba0 tx=0x7f30c000eeb0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 shutdown_connections 2023-03-03T06:53:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c81597f0 0x7f30c80afd90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80ced80 0x7f30c80b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac0682f0 0x7f30ac06a7a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 --2- 172.21.15.102:0/4211220443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cf9a0 0x7f30c81f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:53:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 >> 172.21.15.102:0/4211220443 conn(0x7f30c8065fb0 msgr2=0x7f30c80cbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:53:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 shutdown_connections 2023-03-03T06:53:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:53:36.961+0000 7f30cfbff700 1 -- 172.21.15.102:0/4211220443 wait complete. 2023-03-03T06:53:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:53:36.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T06:53:52.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:52.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:22.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:22.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:32.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:54:33.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.258+0000 7f119dd9d700 1 Processor -- start 2023-03-03T06:54:33.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.258+0000 7f119dd9d700 1 -- start start 2023-03-03T06:54:33.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.260+0000 7f119dd9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11980cb2d0 0x7f11980c6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:54:33.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.260+0000 7f119dd9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119806b160 con 0x7f1198071470 2023-03-03T06:54:33.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.260+0000 7f119dd9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1198071d40 con 0x7f119814fb20 2023-03-03T06:54:33.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.260+0000 7f119dd9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11980c2810 con 0x7f11980cb2d0 2023-03-03T06:54:33.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.260+0000 7f119cd9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f119814fb20 0x7f11980721d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:55148/0 (socket says 172.21.15.102:55148) 2023-03-03T06:54:33.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.260+0000 7f119cd9b700 1 -- 172.21.15.102:0/1639065822 learned_addr learned my addr 172.21.15.102:0/1639065822 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:54:33.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.260+0000 7f1197fff700 1 --2- 172.21.15.102:0/1639065822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11980cb2d0 0x7f11980c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:54:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.261+0000 7f11977fe700 1 -- 172.21.15.102:0/1639065822 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633188922 0 0) 0x7f1198071d40 con 0x7f119814fb20 2023-03-03T06:54:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.261+0000 7f1197fff700 1 -- 172.21.15.102:0/1639065822 >> v1:172.21.15.172:6789/0 conn(0x7f119814fb20 legacy=0x7f11980721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:54:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.261+0000 7f1197fff700 1 -- 172.21.15.102:0/1639065822 >> v1:172.21.15.102:6789/0 conn(0x7f1198071470 legacy=0x7f11980c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:54:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.261+0000 7f1197fff700 1 -- 172.21.15.102:0/1639065822 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11980728e0 con 0x7f11980cb2d0 2023-03-03T06:54:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.261+0000 7f1197fff700 1 --2- 172.21.15.102:0/1639065822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11980cb2d0 0x7f11980c6250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1184009870 tx=0x7f1184009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d96b3df8b9cb34a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:54:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f11977fe700 1 -- 172.21.15.102:0/1639065822 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184011830 con 0x7f11980cb2d0 2023-03-03T06:54:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f11977fe700 1 -- 172.21.15.102:0/1639065822 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1184011990 con 0x7f11980cb2d0 2023-03-03T06:54:33.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f11977fe700 1 -- 172.21.15.102:0/1639065822 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184011d20 con 0x7f11980cb2d0 2023-03-03T06:54:33.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f119dd9d700 1 -- 172.21.15.102:0/1639065822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11980cb2d0 msgr2=0x7f11980c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:54:33.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f119dd9d700 1 --2- 172.21.15.102:0/1639065822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11980cb2d0 0x7f11980c6250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1184009870 tx=0x7f1184009b80 comp rx=0 tx=0).stop 2023-03-03T06:54:33.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f119dd9d700 1 -- 172.21.15.102:0/1639065822 shutdown_connections 2023-03-03T06:54:33.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f119dd9d700 1 --2- 172.21.15.102:0/1639065822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11980cb2d0 0x7f11980c6250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:54:33.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f119dd9d700 1 -- 172.21.15.102:0/1639065822 >> 172.21.15.102:0/1639065822 conn(0x7f11980ca7c0 msgr2=0x7f11980cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:54:33.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.262+0000 7f119dd9d700 1 -- 172.21.15.102:0/1639065822 shutdown_connections 2023-03-03T06:54:33.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.263+0000 7f119dd9d700 1 -- 172.21.15.102:0/1639065822 wait complete. 2023-03-03T06:54:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.265+0000 7f119dd9d700 1 Processor -- start 2023-03-03T06:54:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.266+0000 7f119dd9d700 1 -- start start 2023-03-03T06:54:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119dd9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1198071470 0x7f1198150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:54:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119dd9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11980cb2d0 0x7f119814c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:54:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119dd9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119814fb20 0x7f1198144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:54:33.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119dd9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1198151d70 con 0x7f119814fb20 2023-03-03T06:54:33.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119dd9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11981503d0 con 0x7f11980cb2d0 2023-03-03T06:54:33.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119dd9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1198150da0 con 0x7f1198071470 2023-03-03T06:54:33.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119d59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119814fb20 0x7f1198144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:54:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f119cd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1198071470 0x7f1198150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:54:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f1197fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11980cb2d0 0x7f119814c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:54:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f1197fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11980cb2d0 0x7f119814c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33864/0 (socket says 172.21.15.102:33864) 2023-03-03T06:54:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.267+0000 7f1197fff700 1 -- 172.21.15.102:0/4289695332 learned_addr learned my addr 172.21.15.102:0/4289695332 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:54:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.268+0000 7f1197fff700 1 -- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1198071470 msgr2=0x7f1198150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:54:33.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.268+0000 7f1197fff700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1198071470 0x7f1198150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:54:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.268+0000 7f1197fff700 1 -- 172.21.15.102:0/4289695332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119814fb20 msgr2=0x7f1198144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:54:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.268+0000 7f1197fff700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119814fb20 0x7f1198144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:54:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.268+0000 7f1197fff700 1 -- 172.21.15.102:0/4289695332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11840094a0 con 0x7f11980cb2d0 2023-03-03T06:54:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.268+0000 7f119d59c700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119814fb20 0x7f1198144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:54:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.268+0000 7f1197fff700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11980cb2d0 0x7f119814c980 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1184000c00 tx=0x7f1184004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:54:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.269+0000 7f11957fa700 1 -- 172.21.15.102:0/4289695332 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184004590 con 0x7f11980cb2d0 2023-03-03T06:54:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.269+0000 7f11957fa700 1 -- 172.21.15.102:0/4289695332 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118401f070 con 0x7f11980cb2d0 2023-03-03T06:54:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.269+0000 7f11957fa700 1 -- 172.21.15.102:0/4289695332 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11840038a0 con 0x7f11980cb2d0 2023-03-03T06:54:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.269+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1198159a90 con 0x7f11980cb2d0 2023-03-03T06:54:33.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.269+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11980706f0 con 0x7f11980cb2d0 2023-03-03T06:54:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.270+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11980719d0 con 0x7f11980cb2d0 2023-03-03T06:54:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.271+0000 7f11957fa700 1 -- 172.21.15.102:0/4289695332 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f118402d070 con 0x7f11980cb2d0 2023-03-03T06:54:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.272+0000 7f11957fa700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f117c068320 0x7f117c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:54:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.272+0000 7f11957fa700 1 -- 172.21.15.102:0/4289695332 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1184021070 con 0x7f11980cb2d0 2023-03-03T06:54:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.273+0000 7f119cd9b700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f117c068320 0x7f117c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:54:33.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.273+0000 7f119cd9b700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f117c068320 0x7f117c06a7d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f118c000f60 tx=0x7f118c007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:54:33.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.277+0000 7f11957fa700 1 -- 172.21.15.102:0/4289695332 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1184053960 con 0x7f11980cb2d0 2023-03-03T06:54:33.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.513+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f119806f4d0 con 0x7f11980cb2d0 2023-03-03T06:54:33.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.514+0000 7f11957fa700 1 -- 172.21.15.102:0/4289695332 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1184056f80 con 0x7f11980cb2d0 2023-03-03T06:54:33.516 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:54:33.516 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:54:33.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.516+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f117c068320 msgr2=0x7f117c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:54:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.516+0000 7f119dd9d700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f117c068320 0x7f117c06a7d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f118c000f60 tx=0x7f118c007e70 comp rx=0 tx=0).stop 2023-03-03T06:54:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.516+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11980cb2d0 msgr2=0x7f119814c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:54:33.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.516+0000 7f119dd9d700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11980cb2d0 0x7f119814c980 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1184000c00 tx=0x7f1184004750 comp rx=0 tx=0).stop 2023-03-03T06:54:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 shutdown_connections 2023-03-03T06:54:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f117c068320 0x7f117c06a7d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:54:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1198071470 0x7f1198150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:54:33.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119814fb20 0x7f1198144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:54:33.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 --2- 172.21.15.102:0/4289695332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11980cb2d0 0x7f119814c980 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:54:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 >> 172.21.15.102:0/4289695332 conn(0x7f11980ca7c0 msgr2=0x7f119814b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:54:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 shutdown_connections 2023-03-03T06:54:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:54:33.517+0000 7f119dd9d700 1 -- 172.21.15.102:0/4289695332 wait complete. 2023-03-03T06:54:33.526 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:54:33.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 54 secs before thrashing 2023-03-03T06:54:52.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:52.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:23.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:23.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:27.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:55:27.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.793+0000 7f2c335ff700 1 Processor -- start 2023-03-03T06:55:27.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.793+0000 7f2c335ff700 1 -- start start 2023-03-03T06:55:27.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.794+0000 7f2c335ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c2c0cd8e0 0x7f2c2c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:55:27.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.794+0000 7f2c335ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c153660 con 0x7f2c2c0cd390 2023-03-03T06:55:27.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.794+0000 7f2c335ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c06c360 con 0x7f2c2c153290 2023-03-03T06:55:27.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.794+0000 7f2c335ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2c15cea0 con 0x7f2c2c0cd8e0 2023-03-03T06:55:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.795+0000 7f2c30b9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c2c0cd8e0 0x7f2c2c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:55:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.795+0000 7f2c3139b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2c2c153290 0x7f2c2c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38000/0 (socket says 172.21.15.102:38000) 2023-03-03T06:55:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.795+0000 7f2c3139b700 1 -- 172.21.15.102:0/4261910010 learned_addr learned my addr 172.21.15.102:0/4261910010 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:55:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.795+0000 7f2c30b9a700 1 -- 172.21.15.102:0/4261910010 >> v1:172.21.15.172:6789/0 conn(0x7f2c2c153290 legacy=0x7f2c2c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:55:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.795+0000 7f2c30b9a700 1 -- 172.21.15.102:0/4261910010 >> v1:172.21.15.102:6789/0 conn(0x7f2c2c0cd390 legacy=0x7f2c2c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:55:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.795+0000 7f2c30b9a700 1 -- 172.21.15.102:0/4261910010 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c06c7f0 con 0x7f2c2c0cd8e0 2023-03-03T06:55:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.795+0000 7f2c30b9a700 1 --2- 172.21.15.102:0/4261910010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c0cde20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009840 tx=0x7f2c1c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bb883f20edcb0b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:55:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c23fff700 1 -- 172.21.15.102:0/4261910010 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c011830 con 0x7f2c2c0cd8e0 2023-03-03T06:55:27.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c23fff700 1 -- 172.21.15.102:0/4261910010 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c011990 con 0x7f2c2c0cd8e0 2023-03-03T06:55:27.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c23fff700 1 -- 172.21.15.102:0/4261910010 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c011d20 con 0x7f2c2c0cd8e0 2023-03-03T06:55:27.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c335ff700 1 -- 172.21.15.102:0/4261910010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd8e0 msgr2=0x7f2c2c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:55:27.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c335ff700 1 --2- 172.21.15.102:0/4261910010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c0cde20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009840 tx=0x7f2c1c009b50 comp rx=0 tx=0).stop 2023-03-03T06:55:27.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c335ff700 1 -- 172.21.15.102:0/4261910010 shutdown_connections 2023-03-03T06:55:27.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c335ff700 1 --2- 172.21.15.102:0/4261910010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c0cde20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:55:27.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c335ff700 1 -- 172.21.15.102:0/4261910010 >> 172.21.15.102:0/4261910010 conn(0x7f2c2c065fb0 msgr2=0x7f2c2c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:55:27.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c335ff700 1 -- 172.21.15.102:0/4261910010 shutdown_connections 2023-03-03T06:55:27.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.796+0000 7f2c335ff700 1 -- 172.21.15.102:0/4261910010 wait complete. 2023-03-03T06:55:27.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.798+0000 7f2c335ff700 1 Processor -- start 2023-03-03T06:55:27.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.798+0000 7f2c335ff700 1 -- start start 2023-03-03T06:55:27.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.799+0000 7f2c335ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd390 0x7f2c2c155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:55:27.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.799+0000 7f2c335ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:55:27.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.799+0000 7f2c335ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c2c153290 0x7f2c2c154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:55:27.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.799+0000 7f2c335ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1eee00 con 0x7f2c2c0cd8e0 2023-03-03T06:55:27.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.799+0000 7f2c335ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c0c53d0 con 0x7f2c2c0cd390 2023-03-03T06:55:27.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.799+0000 7f2c335ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1ef0c0 con 0x7f2c2c153290 2023-03-03T06:55:27.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd390 0x7f2c2c155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:55:27.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c30b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:55:27.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd390 0x7f2c2c155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48154/0 (socket says 172.21.15.102:48154) 2023-03-03T06:55:27.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 -- 172.21.15.102:0/392135094 learned_addr learned my addr 172.21.15.102:0/392135094 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:55:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c31b9c700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c2c153290 0x7f2c2c154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:55:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 -- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c2c153290 msgr2=0x7f2c2c154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:55:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c2c153290 0x7f2c2c154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:55:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 -- 172.21.15.102:0/392135094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2c0cd8e0 msgr2=0x7f2c2c154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:55:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:55:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 -- 172.21.15.102:0/392135094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0094a0 con 0x7f2c2c0cd390 2023-03-03T06:55:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c30b9a700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:55:27.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.800+0000 7f2c3139b700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd390 0x7f2c2c155e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800c480 tx=0x7f2c2800c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:55:27.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.801+0000 7f2c21ffb700 1 -- 172.21.15.102:0/392135094 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28005d70 con 0x7f2c2c0cd390 2023-03-03T06:55:27.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.801+0000 7f2c21ffb700 1 -- 172.21.15.102:0/392135094 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2800ce90 con 0x7f2c2c0cd390 2023-03-03T06:55:27.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.801+0000 7f2c21ffb700 1 -- 172.21.15.102:0/392135094 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c280072d0 con 0x7f2c2c0cd390 2023-03-03T06:55:27.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.801+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c0ccde0 con 0x7f2c2c0cd390 2023-03-03T06:55:27.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.801+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2c15d160 con 0x7f2c2c0cd390 2023-03-03T06:55:27.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.802+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c0cd0c0 con 0x7f2c2c0cd390 2023-03-03T06:55:27.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.803+0000 7f2c21ffb700 1 -- 172.21.15.102:0/392135094 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c28019030 con 0x7f2c2c0cd390 2023-03-03T06:55:27.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.805+0000 7f2c21ffb700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c100682f0 0x7f2c1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:55:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.805+0000 7f2c21ffb700 1 -- 172.21.15.102:0/392135094 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2c28084e80 con 0x7f2c2c0cd390 2023-03-03T06:55:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.805+0000 7f2c30b9a700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c100682f0 0x7f2c1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:55:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.805+0000 7f2c30b9a700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c100682f0 0x7f2c1006a7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0090b0 tx=0x7f2c1c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:55:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:27.809+0000 7f2c21ffb700 1 -- 172.21.15.102:0/392135094 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c2804b5e0 con 0x7f2c2c0cd390 2023-03-03T06:55:28.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.045+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c2c154840 con 0x7f2c2c0cd390 2023-03-03T06:55:28.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.046+0000 7f2c21ffb700 1 -- 172.21.15.102:0/392135094 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2c28017090 con 0x7f2c2c0cd390 2023-03-03T06:55:28.048 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:55:28.048 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:55:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.048+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c100682f0 msgr2=0x7f2c1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:55:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.048+0000 7f2c335ff700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c100682f0 0x7f2c1006a7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0090b0 tx=0x7f2c1c008030 comp rx=0 tx=0).stop 2023-03-03T06:55:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.048+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd390 msgr2=0x7f2c2c155e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:55:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.048+0000 7f2c335ff700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd390 0x7f2c2c155e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800c480 tx=0x7f2c2800c790 comp rx=0 tx=0).stop 2023-03-03T06:55:28.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 shutdown_connections 2023-03-03T06:55:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c100682f0 0x7f2c1006a7a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:55:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c2c153290 0x7f2c2c154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:55:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c2c0cd390 0x7f2c2c155e30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:55:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 --2- 172.21.15.102:0/392135094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2c0cd8e0 0x7f2c2c154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:55:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 >> 172.21.15.102:0/392135094 conn(0x7f2c2c065fb0 msgr2=0x7f2c2c158a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:55:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 shutdown_connections 2023-03-03T06:55:28.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:55:28.049+0000 7f2c335ff700 1 -- 172.21.15.102:0/392135094 wait complete. 2023-03-03T06:55:28.052 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:55:28.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T06:55:53.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:53.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:02.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:56:02.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.326+0000 7f03df5ff700 1 Processor -- start 2023-03-03T06:56:02.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.326+0000 7f03df5ff700 1 -- start start 2023-03-03T06:56:02.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03df5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d8153290 0x7f03d8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:02.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03df5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f03d81593c0 0x7f03d80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03df5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03d80cd8e0 0x7f03d80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03df5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d8153660 con 0x7f03d80cd8e0 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03df5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d806c360 con 0x7f03d81593c0 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03df5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815cea0 con 0x7f03d8153290 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03dd39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d8153290 0x7f03d8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03ddb9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03d80cd8e0 0x7f03d80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03dd39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d8153290 0x7f03d8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51012/0 (socket says 172.21.15.102:51012) 2023-03-03T06:56:02.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03dd39b700 1 -- 172.21.15.102:0/321690654 learned_addr learned my addr 172.21.15.102:0/321690654 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:56:02.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.328+0000 7f03dcb9a700 1 --2- 172.21.15.102:0/321690654 >> v2:172.21.15.172:3300/0 conn(0x7f03d81593c0 0x7f03d80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:02.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03dd39b700 1 -- 172.21.15.102:0/321690654 >> v2:172.21.15.172:3300/0 conn(0x7f03d81593c0 msgr2=0x7f03d80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:56:02.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03dd39b700 1 --2- 172.21.15.102:0/321690654 >> v2:172.21.15.172:3300/0 conn(0x7f03d81593c0 0x7f03d80cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03dd39b700 1 -- 172.21.15.102:0/321690654 >> v2:172.21.15.102:3300/0 conn(0x7f03d80cd8e0 msgr2=0x7f03d80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:56:02.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03dd39b700 1 --2- 172.21.15.102:0/321690654 >> v2:172.21.15.102:3300/0 conn(0x7f03d80cd8e0 0x7f03d80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03dd39b700 1 -- 172.21.15.102:0/321690654 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d806c7f0 con 0x7f03d8153290 2023-03-03T06:56:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03ddb9c700 1 --2- 172.21.15.102:0/321690654 >> v2:172.21.15.102:3300/0 conn(0x7f03d80cd8e0 0x7f03d80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:56:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03dd39b700 1 --2- 172.21.15.102:0/321690654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d8153290 0x7f03d8158e80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f03d80b1970 tx=0x7f03d4009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a01e955fef6de60 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:56:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03cffff700 1 -- 172.21.15.102:0/321690654 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4011830 con 0x7f03d8153290 2023-03-03T06:56:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03cffff700 1 -- 172.21.15.102:0/321690654 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d4011990 con 0x7f03d8153290 2023-03-03T06:56:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.329+0000 7f03cffff700 1 -- 172.21.15.102:0/321690654 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4011d20 con 0x7f03d8153290 2023-03-03T06:56:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 -- 172.21.15.102:0/321690654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d8153290 msgr2=0x7f03d8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 --2- 172.21.15.102:0/321690654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d8153290 0x7f03d8158e80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f03d80b1970 tx=0x7f03d4009bd0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 -- 172.21.15.102:0/321690654 shutdown_connections 2023-03-03T06:56:02.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 --2- 172.21.15.102:0/321690654 >> v2:172.21.15.102:3300/0 conn(0x7f03d80cd8e0 0x7f03d80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 --2- 172.21.15.102:0/321690654 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d8153290 0x7f03d8158e80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 --2- 172.21.15.102:0/321690654 >> v2:172.21.15.172:3300/0 conn(0x7f03d81593c0 0x7f03d80cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 -- 172.21.15.102:0/321690654 >> 172.21.15.102:0/321690654 conn(0x7f03d8065fb0 msgr2=0x7f03d80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:56:02.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 -- 172.21.15.102:0/321690654 shutdown_connections 2023-03-03T06:56:02.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.330+0000 7f03df5ff700 1 -- 172.21.15.102:0/321690654 wait complete. 2023-03-03T06:56:02.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.332+0000 7f03df5ff700 1 Processor -- start 2023-03-03T06:56:02.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.332+0000 7f03df5ff700 1 -- start start 2023-03-03T06:56:02.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.333+0000 7f03df5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d80cd8e0 0x7f03d81f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:02.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.333+0000 7f03df5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03d8153290 0x7f03d80cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:02.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.333+0000 7f03df5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03d81593c0 0x7f03d8152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.333+0000 7f03df5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d80c6fb0 con 0x7f03d8153290 2023-03-03T06:56:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.333+0000 7f03df5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d80c8a00 con 0x7f03d81593c0 2023-03-03T06:56:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.333+0000 7f03df5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d80c8ce0 con 0x7f03d80cd8e0 2023-03-03T06:56:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.333+0000 7f03dd39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d80cd8e0 0x7f03d81f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dcb9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03d8153290 0x7f03d80cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03ddb9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03d81593c0 0x7f03d8152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dd39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d80cd8e0 0x7f03d81f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51020/0 (socket says 172.21.15.102:51020) 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dd39b700 1 -- 172.21.15.102:0/3761511551 learned_addr learned my addr 172.21.15.102:0/3761511551 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dcb9a700 1 -- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d80cd8e0 msgr2=0x7f03d81f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dcb9a700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d80cd8e0 0x7f03d81f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dcb9a700 1 -- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03d81593c0 msgr2=0x7f03d8152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dcb9a700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03d81593c0 0x7f03d8152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dcb9a700 1 -- 172.21.15.102:0/3761511551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0009a70 con 0x7f03d8153290 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03dcb9a700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03d8153290 0x7f03d80cf0b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f03c800e900 tx=0x7f03c800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:56:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.334+0000 7f03cdffb700 1 -- 172.21.15.102:0/3761511551 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c800c640 con 0x7f03d8153290 2023-03-03T06:56:02.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.335+0000 7f03cdffb700 1 -- 172.21.15.102:0/3761511551 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c800c7a0 con 0x7f03d8153290 2023-03-03T06:56:02.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.335+0000 7f03cdffb700 1 -- 172.21.15.102:0/3761511551 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c800cb10 con 0x7f03d8153290 2023-03-03T06:56:02.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.335+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d4009550 con 0x7f03d8153290 2023-03-03T06:56:02.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.335+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d80ccde0 con 0x7f03d8153290 2023-03-03T06:56:02.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.336+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d815d160 con 0x7f03d8153290 2023-03-03T06:56:02.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.337+0000 7f03cdffb700 1 -- 172.21.15.102:0/3761511551 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f03c8019030 con 0x7f03d8153290 2023-03-03T06:56:02.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.338+0000 7f03cdffb700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03bc0682f0 0x7f03bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:02.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.338+0000 7f03cdffb700 1 -- 172.21.15.102:0/3761511551 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f03c8085b60 con 0x7f03d8153290 2023-03-03T06:56:02.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.339+0000 7f03dd39b700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03bc0682f0 0x7f03bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:02.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.339+0000 7f03dd39b700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03bc0682f0 0x7f03bc06a7a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009fd0 tx=0x7f03d4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:56:02.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.342+0000 7f03cdffb700 1 -- 172.21.15.102:0/3761511551 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f03c8052270 con 0x7f03d8153290 2023-03-03T06:56:02.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.578+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03d80cd0c0 con 0x7f03d8153290 2023-03-03T06:56:02.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.578+0000 7f03cdffb700 1 -- 172.21.15.102:0/3761511551 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f03c8055890 con 0x7f03d8153290 2023-03-03T06:56:02.581 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:56:02.581 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:56:02.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03bc0682f0 msgr2=0x7f03bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:02.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03bc0682f0 0x7f03bc06a7a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009fd0 tx=0x7f03d4008030 comp rx=0 tx=0).stop 2023-03-03T06:56:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03d8153290 msgr2=0x7f03d80cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03d8153290 0x7f03d80cf0b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f03c800e900 tx=0x7f03c800ec10 comp rx=0 tx=0).stop 2023-03-03T06:56:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 shutdown_connections 2023-03-03T06:56:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03bc0682f0 0x7f03bc06a7a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03d80cd8e0 0x7f03d81f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03d81593c0 0x7f03d8152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 --2- 172.21.15.102:0/3761511551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03d8153290 0x7f03d80cf0b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 >> 172.21.15.102:0/3761511551 conn(0x7f03d8065fb0 msgr2=0x7f03d80cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:56:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 shutdown_connections 2023-03-03T06:56:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:02.581+0000 7f03df5ff700 1 -- 172.21.15.102:0/3761511551 wait complete. 2023-03-03T06:56:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:56:02.610 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T06:56:23.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:23.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:30.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:56:30.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.864+0000 7f05c8bff700 1 Processor -- start 2023-03-03T06:56:30.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.864+0000 7f05c8bff700 1 -- start start 2023-03-03T06:56:30.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.865+0000 7f05c8bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f05c4153290 0x7f05c4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:30.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.865+0000 7f05c8bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05c41593c0 0x7f05c40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:30.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.865+0000 7f05c8bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c4153660 con 0x7f05c40cd8e0 2023-03-03T06:56:30.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.865+0000 7f05c8bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c406c360 con 0x7f05c4153290 2023-03-03T06:56:30.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.865+0000 7f05c8bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c415cea0 con 0x7f05c41593c0 2023-03-03T06:56:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c259c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f05c4153290 0x7f05c4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c2d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f05c40cd8e0 0x7f05c40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53644/0 (socket says 172.21.15.102:53644) 2023-03-03T06:56:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c1d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05c41593c0 0x7f05c40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:30.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c2d9d700 1 -- 172.21.15.102:0/4017483735 learned_addr learned my addr 172.21.15.102:0/4017483735 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:56:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c259c700 1 -- 172.21.15.102:0/4017483735 >> v2:172.21.15.172:3300/0 conn(0x7f05c41593c0 msgr2=0x7f05c40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:56:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c259c700 1 --2- 172.21.15.102:0/4017483735 >> v2:172.21.15.172:3300/0 conn(0x7f05c41593c0 0x7f05c40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c259c700 1 -- 172.21.15.102:0/4017483735 >> v1:172.21.15.102:6789/0 conn(0x7f05c40cd8e0 legacy=0x7f05c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.866+0000 7f05c259c700 1 -- 172.21.15.102:0/4017483735 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c406c7f0 con 0x7f05c4153290 2023-03-03T06:56:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.867+0000 7f05c259c700 1 --2- 172.21.15.102:0/4017483735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c4153290 0x7f05c4158e80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f05c4159fb0 tx=0x7f05b4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444ffc85ad589bc2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:56:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.867+0000 7f05c159a700 1 -- 172.21.15.102:0/4017483735 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4011830 con 0x7f05c4153290 2023-03-03T06:56:30.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.867+0000 7f05c159a700 1 -- 172.21.15.102:0/4017483735 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4011990 con 0x7f05c4153290 2023-03-03T06:56:30.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.867+0000 7f05c159a700 1 -- 172.21.15.102:0/4017483735 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4011d20 con 0x7f05c4153290 2023-03-03T06:56:30.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.867+0000 7f05c8bff700 1 -- 172.21.15.102:0/4017483735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c4153290 msgr2=0x7f05c4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:30.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.867+0000 7f05c8bff700 1 --2- 172.21.15.102:0/4017483735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c4153290 0x7f05c4158e80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f05c4159fb0 tx=0x7f05b4009b20 comp rx=0 tx=0).stop 2023-03-03T06:56:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.868+0000 7f05c8bff700 1 -- 172.21.15.102:0/4017483735 shutdown_connections 2023-03-03T06:56:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.868+0000 7f05c8bff700 1 --2- 172.21.15.102:0/4017483735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c4153290 0x7f05c4158e80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.868+0000 7f05c8bff700 1 --2- 172.21.15.102:0/4017483735 >> v2:172.21.15.172:3300/0 conn(0x7f05c41593c0 0x7f05c40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.868+0000 7f05c8bff700 1 -- 172.21.15.102:0/4017483735 >> 172.21.15.102:0/4017483735 conn(0x7f05c4065fb0 msgr2=0x7f05c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:56:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.868+0000 7f05c8bff700 1 -- 172.21.15.102:0/4017483735 shutdown_connections 2023-03-03T06:56:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.868+0000 7f05c8bff700 1 -- 172.21.15.102:0/4017483735 wait complete. 2023-03-03T06:56:30.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.870+0000 7f05c8bff700 1 Processor -- start 2023-03-03T06:56:30.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.870+0000 7f05c8bff700 1 -- start start 2023-03-03T06:56:30.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c8bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c40cd8e0 0x7f05c414c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:30.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c8bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05c4153290 0x7f05c4155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:30.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c8bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05c41593c0 0x7f05c4156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:30.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c8bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c41ee930 con 0x7f05c40cd8e0 2023-03-03T06:56:30.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c8bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40b45f0 con 0x7f05c41593c0 2023-03-03T06:56:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c8bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40b1d10 con 0x7f05c4153290 2023-03-03T06:56:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c40cd8e0 0x7f05c414c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c40cd8e0 0x7f05c414c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47886/0 (socket says 172.21.15.102:47886) 2023-03-03T06:56:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c259c700 1 -- 172.21.15.102:0/2675703690 learned_addr learned my addr 172.21.15.102:0/2675703690 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:56:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c1d9b700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05c4153290 0x7f05c4155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.871+0000 7f05c2d9d700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05c41593c0 0x7f05c4156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:30.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c259c700 1 -- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05c4153290 msgr2=0x7f05c4155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c259c700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05c4153290 0x7f05c4155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c259c700 1 -- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05c41593c0 msgr2=0x7f05c4156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c259c700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05c41593c0 0x7f05c4156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c259c700 1 -- 172.21.15.102:0/2675703690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b40094a0 con 0x7f05c40cd8e0 2023-03-03T06:56:30.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c2d9d700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05c41593c0 0x7f05c4156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:56:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c259c700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c40cd8e0 0x7f05c414c770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f05b40090b0 tx=0x7f05b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:56:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05b2ffd700 1 -- 172.21.15.102:0/2675703690 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b40036a0 con 0x7f05c40cd8e0 2023-03-03T06:56:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05b2ffd700 1 -- 172.21.15.102:0/2675703690 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b401f070 con 0x7f05c40cd8e0 2023-03-03T06:56:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c40ccde0 con 0x7f05c40cd8e0 2023-03-03T06:56:30.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.872+0000 7f05b2ffd700 1 -- 172.21.15.102:0/2675703690 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4003a40 con 0x7f05c40cd8e0 2023-03-03T06:56:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.873+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c415d160 con 0x7f05c40cd8e0 2023-03-03T06:56:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.874+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c40cd0c0 con 0x7f05c40cd8e0 2023-03-03T06:56:30.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.875+0000 7f05b2ffd700 1 -- 172.21.15.102:0/2675703690 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f05b400f050 con 0x7f05c40cd8e0 2023-03-03T06:56:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.876+0000 7f05b2ffd700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05a40682f0 0x7f05a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:56:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.876+0000 7f05b2ffd700 1 -- 172.21.15.102:0/2675703690 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f05b4021070 con 0x7f05c40cd8e0 2023-03-03T06:56:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.876+0000 7f05c1d9b700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05a40682f0 0x7f05a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:56:30.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.877+0000 7f05c1d9b700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05a40682f0 0x7f05a406a7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f05c415a4d0 tx=0x7f05b8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:56:30.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:30.880+0000 7f05b2ffd700 1 -- 172.21.15.102:0/2675703690 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05b4053db0 con 0x7f05c40cd8e0 2023-03-03T06:56:31.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.114+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05c404e6e0 con 0x7f05c40cd8e0 2023-03-03T06:56:31.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.115+0000 7f05b2ffd700 1 -- 172.21.15.102:0/2675703690 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f05b4015090 con 0x7f05c40cd8e0 2023-03-03T06:56:31.116 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:56:31.116 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:56:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05a40682f0 msgr2=0x7f05a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05a40682f0 0x7f05a406a7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f05c415a4d0 tx=0x7f05b8009210 comp rx=0 tx=0).stop 2023-03-03T06:56:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c40cd8e0 msgr2=0x7f05c414c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:56:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c40cd8e0 0x7f05c414c770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f05b40090b0 tx=0x7f05b4004750 comp rx=0 tx=0).stop 2023-03-03T06:56:31.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 shutdown_connections 2023-03-03T06:56:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05c41593c0 0x7f05c4156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05c40cd8e0 0x7f05c414c770 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.117+0000 7f05c8bff700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05a40682f0 0x7f05a406a7a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.118+0000 7f05c8bff700 1 --2- 172.21.15.102:0/2675703690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f05c4153290 0x7f05c4155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:56:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.118+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 >> 172.21.15.102:0/2675703690 conn(0x7f05c4065fb0 msgr2=0x7f05c40cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:56:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.118+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 shutdown_connections 2023-03-03T06:56:31.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:56:31.118+0000 7f05c8bff700 1 -- 172.21.15.102:0/2675703690 wait complete. 2023-03-03T06:56:31.121 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:56:31.146 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T06:56:53.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:53.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:23.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:23.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:53.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:53.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:03.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:58:03.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.396+0000 7f2e7efff700 1 Processor -- start 2023-03-03T06:58:03.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.396+0000 7f2e7efff700 1 -- start start 2023-03-03T06:58:03.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e7efff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e78153290 0x7f2e78158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:03.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e7efff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e780cd8e0 0x7f2e780ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:03.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e7efff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e78153660 con 0x7f2e781593c0 2023-03-03T06:58:03.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e7efff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7806c360 con 0x7f2e780cd8e0 2023-03-03T06:58:03.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e7efff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7815cea0 con 0x7f2e78153290 2023-03-03T06:58:03.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e7cd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e78153290 0x7f2e78158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:03.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e7d59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e780cd8e0 0x7f2e780ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:03.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e77fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2e781593c0 0x7f2e780cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44334/0 (socket says 172.21.15.102:44334) 2023-03-03T06:58:03.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.398+0000 7f2e77fff700 1 -- 172.21.15.102:0/944375802 learned_addr learned my addr 172.21.15.102:0/944375802 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T06:58:03.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.399+0000 7f2e7cd9b700 1 -- 172.21.15.102:0/944375802 >> v2:172.21.15.102:3300/0 conn(0x7f2e780cd8e0 msgr2=0x7f2e780ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:58:03.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.399+0000 7f2e7cd9b700 1 --2- 172.21.15.102:0/944375802 >> v2:172.21.15.102:3300/0 conn(0x7f2e780cd8e0 0x7f2e780ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.399+0000 7f2e7cd9b700 1 -- 172.21.15.102:0/944375802 >> v1:172.21.15.172:6789/0 conn(0x7f2e781593c0 legacy=0x7f2e780cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:03.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.399+0000 7f2e7cd9b700 1 -- 172.21.15.102:0/944375802 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7806c7f0 con 0x7f2e78153290 2023-03-03T06:58:03.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.399+0000 7f2e7d59c700 1 --2- 172.21.15.102:0/944375802 >> v2:172.21.15.102:3300/0 conn(0x7f2e780cd8e0 0x7f2e780ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:58:03.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.399+0000 7f2e7cd9b700 1 --2- 172.21.15.102:0/944375802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e78153290 0x7f2e78158e80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2e7808cc70 tx=0x7f2e6c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa89416a88292467 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:58:03.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.399+0000 7f2e777fe700 1 -- 172.21.15.102:0/944375802 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c011830 con 0x7f2e78153290 2023-03-03T06:58:03.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e777fe700 1 -- 172.21.15.102:0/944375802 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c011990 con 0x7f2e78153290 2023-03-03T06:58:03.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e777fe700 1 -- 172.21.15.102:0/944375802 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c011d20 con 0x7f2e78153290 2023-03-03T06:58:03.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 -- 172.21.15.102:0/944375802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e78153290 msgr2=0x7f2e78158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:03.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 --2- 172.21.15.102:0/944375802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e78153290 0x7f2e78158e80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2e7808cc70 tx=0x7f2e6c009b20 comp rx=0 tx=0).stop 2023-03-03T06:58:03.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 -- 172.21.15.102:0/944375802 shutdown_connections 2023-03-03T06:58:03.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 --2- 172.21.15.102:0/944375802 >> v2:172.21.15.102:3300/0 conn(0x7f2e780cd8e0 0x7f2e780ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 --2- 172.21.15.102:0/944375802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e78153290 0x7f2e78158e80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 -- 172.21.15.102:0/944375802 >> 172.21.15.102:0/944375802 conn(0x7f2e78065fb0 msgr2=0x7f2e780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:58:03.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 -- 172.21.15.102:0/944375802 shutdown_connections 2023-03-03T06:58:03.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.400+0000 7f2e7efff700 1 -- 172.21.15.102:0/944375802 wait complete. 2023-03-03T06:58:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.402+0000 7f2e7efff700 1 Processor -- start 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.403+0000 7f2e7efff700 1 -- start start 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.403+0000 7f2e7efff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cd8e0 0x7f2e78155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.403+0000 7f2e7efff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e78153290 0x7f2e780b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.403+0000 7f2e7efff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e781593c0 0x7f2e780b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e7efff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e78154af0 con 0x7f2e78153290 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e7efff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780c8c00 con 0x7f2e781593c0 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e7efff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780c9180 con 0x7f2e780cd8e0 2023-03-03T06:58:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e78153290 0x7f2e780b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e78153290 0x7f2e780b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60668/0 (socket says 172.21.15.102:60668) 2023-03-03T06:58:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 -- 172.21.15.102:0/1261323849 learned_addr learned my addr 172.21.15.102:0/1261323849 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:58:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e7cd9b700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cd8e0 0x7f2e78155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e7d59c700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e781593c0 0x7f2e780b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 -- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cd8e0 msgr2=0x7f2e78155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cd8e0 0x7f2e78155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 -- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e781593c0 msgr2=0x7f2e780b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:03.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e781593c0 0x7f2e780b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.404+0000 7f2e77fff700 1 -- 172.21.15.102:0/1261323849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e70009a70 con 0x7f2e78153290 2023-03-03T06:58:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.405+0000 7f2e77fff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e78153290 0x7f2e780b24c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400c520 tx=0x7f2e6400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:58:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.405+0000 7f2e757fa700 1 -- 172.21.15.102:0/1261323849 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64005d70 con 0x7f2e78153290 2023-03-03T06:58:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.405+0000 7f2e757fa700 1 -- 172.21.15.102:0/1261323849 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e640070d0 con 0x7f2e78153290 2023-03-03T06:58:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.405+0000 7f2e757fa700 1 -- 172.21.15.102:0/1261323849 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64007440 con 0x7f2e78153290 2023-03-03T06:58:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.405+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c0094a0 con 0x7f2e78153290 2023-03-03T06:58:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.405+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e780ccde0 con 0x7f2e78153290 2023-03-03T06:58:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.406+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7815d160 con 0x7f2e78153290 2023-03-03T06:58:03.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.407+0000 7f2e757fa700 1 -- 172.21.15.102:0/1261323849 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e64019030 con 0x7f2e78153290 2023-03-03T06:58:03.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.409+0000 7f2e757fa700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e5c0682f0 0x7f2e5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:03.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.409+0000 7f2e757fa700 1 -- 172.21.15.102:0/1261323849 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2e64084fe0 con 0x7f2e78153290 2023-03-03T06:58:03.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.409+0000 7f2e7cd9b700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e5c0682f0 0x7f2e5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:03.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.410+0000 7f2e7cd9b700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e5c0682f0 0x7f2e5c06a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0090b0 tx=0x7f2e6c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:58:03.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.413+0000 7f2e757fa700 1 -- 172.21.15.102:0/1261323849 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e6404b5e0 con 0x7f2e78153290 2023-03-03T06:58:03.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.647+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e780cd0c0 con 0x7f2e78153290 2023-03-03T06:58:03.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.648+0000 7f2e757fa700 1 -- 172.21.15.102:0/1261323849 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2e64017020 con 0x7f2e78153290 2023-03-03T06:58:03.649 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:58:03.649 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:58:03.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.650+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e5c0682f0 msgr2=0x7f2e5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:03.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.650+0000 7f2e7efff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e5c0682f0 0x7f2e5c06a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0090b0 tx=0x7f2e6c008030 comp rx=0 tx=0).stop 2023-03-03T06:58:03.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.650+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e78153290 msgr2=0x7f2e780b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:03.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.650+0000 7f2e7efff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e78153290 0x7f2e780b24c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400c520 tx=0x7f2e6400c830 comp rx=0 tx=0).stop 2023-03-03T06:58:03.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 shutdown_connections 2023-03-03T06:58:03.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e5c0682f0 0x7f2e5c06a7a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e780cd8e0 0x7f2e78155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e781593c0 0x7f2e780b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 --2- 172.21.15.102:0/1261323849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e78153290 0x7f2e780b24c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:03.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 >> 172.21.15.102:0/1261323849 conn(0x7f2e78065fb0 msgr2=0x7f2e780cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:58:03.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 shutdown_connections 2023-03-03T06:58:03.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:03.651+0000 7f2e7efff700 1 -- 172.21.15.102:0/1261323849 wait complete. 2023-03-03T06:58:03.654 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:58:03.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-03-03T06:58:24.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:24.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:49.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:58:49.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.942+0000 7fc05a1ff700 1 Processor -- start 2023-03-03T06:58:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.942+0000 7fc05a1ff700 1 -- start start 2023-03-03T06:58:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.943+0000 7fc05a1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc054153290 0x7fc054158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.943+0000 7fc05a1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0541593c0 0x7fc0540cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.943+0000 7fc05a1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0540cd8e0 0x7fc0540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.943+0000 7fc05a1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc054153660 con 0x7fc0541593c0 2023-03-03T06:58:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc05a1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc05406c360 con 0x7fc054153290 2023-03-03T06:58:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc05a1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05415cea0 con 0x7fc0540cd8e0 2023-03-03T06:58:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0541593c0 0x7fc0540cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0541593c0 0x7fc0540cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56610/0 (socket says 172.21.15.102:56610) 2023-03-03T06:58:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 -- 172.21.15.102:0/3186226412 learned_addr learned my addr 172.21.15.102:0/3186226412 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:58:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc0537fe700 1 --2- 172.21.15.102:0/3186226412 >> v2:172.21.15.172:3300/0 conn(0x7fc054153290 0x7fc054158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc053fff700 1 --2- 172.21.15.102:0/3186226412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0540cd8e0 0x7fc0540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 -- 172.21.15.102:0/3186226412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0540cd8e0 msgr2=0x7fc0540ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 --2- 172.21.15.102:0/3186226412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0540cd8e0 0x7fc0540ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 -- 172.21.15.102:0/3186226412 >> v2:172.21.15.172:3300/0 conn(0x7fc054153290 msgr2=0x7fc054158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 --2- 172.21.15.102:0/3186226412 >> v2:172.21.15.172:3300/0 conn(0x7fc054153290 0x7fc054158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.944+0000 7fc04bfff700 1 -- 172.21.15.102:0/3186226412 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05406c7f0 con 0x7fc0541593c0 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.945+0000 7fc04bfff700 1 --2- 172.21.15.102:0/3186226412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0540cde20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc05408cc70 tx=0x7fc03c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110ddcaf05efec7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.945+0000 7fc052ffd700 1 -- 172.21.15.102:0/3186226412 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c011830 con 0x7fc0541593c0 2023-03-03T06:58:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.945+0000 7fc052ffd700 1 -- 172.21.15.102:0/3186226412 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c011990 con 0x7fc0541593c0 2023-03-03T06:58:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.945+0000 7fc052ffd700 1 -- 172.21.15.102:0/3186226412 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c011d20 con 0x7fc0541593c0 2023-03-03T06:58:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 -- 172.21.15.102:0/3186226412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 msgr2=0x7fc0540cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/3186226412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0540cde20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc05408cc70 tx=0x7fc03c009bd0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 -- 172.21.15.102:0/3186226412 shutdown_connections 2023-03-03T06:58:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/3186226412 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0540cd8e0 0x7fc0540ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/3186226412 >> v2:172.21.15.172:3300/0 conn(0x7fc054153290 0x7fc054158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/3186226412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0540cde20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 -- 172.21.15.102:0/3186226412 >> 172.21.15.102:0/3186226412 conn(0x7fc054065fb0 msgr2=0x7fc0540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:58:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 -- 172.21.15.102:0/3186226412 shutdown_connections 2023-03-03T06:58:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.946+0000 7fc05a1ff700 1 -- 172.21.15.102:0/3186226412 wait complete. 2023-03-03T06:58:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.948+0000 7fc05a1ff700 1 Processor -- start 2023-03-03T06:58:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.948+0000 7fc05a1ff700 1 -- start start 2023-03-03T06:58:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc05a1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0540cd8e0 0x7fc05414f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc05a1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc054153290 0x7fc0541f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc05a1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0541f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc05a1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc054156560 con 0x7fc0541593c0 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc05a1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0541f2c60 con 0x7fc0540cd8e0 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc05a1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0541e2c70 con 0x7fc054153290 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc0537fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0540cd8e0 0x7fc05414f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc0537fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0540cd8e0 0x7fc05414f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53510/0 (socket says 172.21.15.102:53510) 2023-03-03T06:58:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc053fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0541f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc0537fe700 1 -- 172.21.15.102:0/770449963 learned_addr learned my addr 172.21.15.102:0/770449963 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T06:58:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.949+0000 7fc04bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc054153290 0x7fc0541f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc04bfff700 1 -- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0540cd8e0 msgr2=0x7fc05414f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc04bfff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0540cd8e0 0x7fc05414f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc04bfff700 1 -- 172.21.15.102:0/770449963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 msgr2=0x7fc0541f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc04bfff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0541f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc04bfff700 1 -- 172.21.15.102:0/770449963 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c009550 con 0x7fc054153290 2023-03-03T06:58:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc0537fe700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0540cd8e0 0x7fc05414f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:58:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc053fff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0541f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:58:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc04bfff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc054153290 0x7fc0541f2080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc03c016000 tx=0x7fc03c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:58:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.950+0000 7fc050ff9700 1 -- 172.21.15.102:0/770449963 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c004450 con 0x7fc054153290 2023-03-03T06:58:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.951+0000 7fc050ff9700 1 -- 172.21.15.102:0/770449963 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c01e070 con 0x7fc054153290 2023-03-03T06:58:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.951+0000 7fc050ff9700 1 -- 172.21.15.102:0/770449963 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0038e0 con 0x7fc054153290 2023-03-03T06:58:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.951+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0540ccde0 con 0x7fc054153290 2023-03-03T06:58:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.951+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05415d160 con 0x7fc054153290 2023-03-03T06:58:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.952+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0540cd0c0 con 0x7fc054153290 2023-03-03T06:58:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.953+0000 7fc050ff9700 1 -- 172.21.15.102:0/770449963 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc03c00f050 con 0x7fc054153290 2023-03-03T06:58:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.954+0000 7fc050ff9700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0340682f0 0x7fc03406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:58:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.954+0000 7fc050ff9700 1 -- 172.21.15.102:0/770449963 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc03c087350 con 0x7fc054153290 2023-03-03T06:58:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.954+0000 7fc0537fe700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0340682f0 0x7fc03406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:58:49.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.955+0000 7fc0537fe700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0340682f0 0x7fc03406a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc044009f70 tx=0x7fc044009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:58:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:49.957+0000 7fc050ff9700 1 -- 172.21.15.102:0/770449963 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc03c053ae0 con 0x7fc054153290 2023-03-03T06:58:50.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.197+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc05414f5b0 con 0x7fc054153290 2023-03-03T06:58:50.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.198+0000 7fc050ff9700 1 -- 172.21.15.102:0/770449963 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc03c057100 con 0x7fc054153290 2023-03-03T06:58:50.200 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T06:58:50.200 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:58:50.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0340682f0 msgr2=0x7fc03406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:50.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0340682f0 0x7fc03406a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc044009f70 tx=0x7fc044009210 comp rx=0 tx=0).stop 2023-03-03T06:58:50.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc054153290 msgr2=0x7fc0541f2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:58:50.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc054153290 0x7fc0541f2080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc03c016000 tx=0x7fc03c004750 comp rx=0 tx=0).stop 2023-03-03T06:58:50.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 shutdown_connections 2023-03-03T06:58:50.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541593c0 0x7fc0541f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:50.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0540cd8e0 0x7fc05414f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:50.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0340682f0 0x7fc03406a7a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:50.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 --2- 172.21.15.102:0/770449963 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc054153290 0x7fc0541f2080 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:58:50.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 >> 172.21.15.102:0/770449963 conn(0x7fc054065fb0 msgr2=0x7fc05415ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:58:50.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 shutdown_connections 2023-03-03T06:58:50.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T06:58:50.201+0000 7fc05a1ff700 1 -- 172.21.15.102:0/770449963 wait complete. 2023-03-03T06:58:50.204 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T06:58:50.229 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-03-03T06:58:54.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:54.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:24.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:24.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:54.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:54.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T06:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:00:24.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:24.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:34.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:00:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.482+0000 7fc5d1dff700 1 Processor -- start 2023-03-03T07:00:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.483+0000 7fc5d1dff700 1 -- start start 2023-03-03T07:00:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.484+0000 7fc5d1dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc5cc0cd8e0 0x7fc5cc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:00:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.484+0000 7fc5d1dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:00:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.484+0000 7fc5d1dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc153660 con 0x7fc5cc153290 2023-03-03T07:00:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.484+0000 7fc5d1dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5cc06c360 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.484+0000 7fc5d1dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc15cea0 con 0x7fc5cc0cd390 2023-03-03T07:00:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5cb7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc5cc153290 0x7fc5cc158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37078/0 (socket says 172.21.15.102:37078) 2023-03-03T07:00:34.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5cb7fe700 1 -- 172.21.15.102:0/1226852849 learned_addr learned my addr 172.21.15.102:0/1226852849 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:00:34.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5cbfff700 1 --2- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:00:34.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5caffd700 1 --2- 172.21.15.102:0/1226852849 >> v2:172.21.15.172:3300/0 conn(0x7fc5cc0cd8e0 0x7fc5cc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:00:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5ca7fc700 1 -- 172.21.15.102:0/1226852849 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724782495 0 0) 0x7fc5cc153660 con 0x7fc5cc153290 2023-03-03T07:00:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5ca7fc700 1 -- 172.21.15.102:0/1226852849 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b40035b0 con 0x7fc5cc153290 2023-03-03T07:00:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5caffd700 1 -- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 msgr2=0x7fc5cc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:00:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5caffd700 1 --2- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5caffd700 1 -- 172.21.15.102:0/1226852849 >> v1:172.21.15.172:6789/0 conn(0x7fc5cc153290 legacy=0x7fc5cc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:00:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5caffd700 1 -- 172.21.15.102:0/1226852849 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc06c7f0 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.485+0000 7fc5cbfff700 1 --2- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:00:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5caffd700 1 --2- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0cde20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009840 tx=0x7fc5c0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ceb123446449067 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:00:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5ca7fc700 1 -- 172.21.15.102:0/1226852849 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0011830 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5ca7fc700 1 -- 172.21.15.102:0/1226852849 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c0011990 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5ca7fc700 1 -- 172.21.15.102:0/1226852849 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0011d00 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5d1dff700 1 -- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 msgr2=0x7fc5cc0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:00:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0cde20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009840 tx=0x7fc5c0009b50 comp rx=0 tx=0).stop 2023-03-03T07:00:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5d1dff700 1 -- 172.21.15.102:0/1226852849 shutdown_connections 2023-03-03T07:00:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/1226852849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0cde20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.486+0000 7fc5d1dff700 1 -- 172.21.15.102:0/1226852849 >> 172.21.15.102:0/1226852849 conn(0x7fc5cc065fb0 msgr2=0x7fc5cc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:00:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.487+0000 7fc5d1dff700 1 -- 172.21.15.102:0/1226852849 shutdown_connections 2023-03-03T07:00:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.487+0000 7fc5d1dff700 1 -- 172.21.15.102:0/1226852849 wait complete. 2023-03-03T07:00:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.489+0000 7fc5d1dff700 1 Processor -- start 2023-03-03T07:00:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.489+0000 7fc5d1dff700 1 -- start start 2023-03-03T07:00:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5d1dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc1e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:00:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5d1dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:00:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5d1dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5cc153290 0x7fc5cc15bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:00:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5d1dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1f23d0 con 0x7fc5cc153290 2023-03-03T07:00:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5d1dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1f2670 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5d1dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1e3f30 con 0x7fc5cc0cd390 2023-03-03T07:00:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5caffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:00:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5cb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc1e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:00:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5caffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43988/0 (socket says 172.21.15.102:43988) 2023-03-03T07:00:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5caffd700 1 -- 172.21.15.102:0/3599322048 learned_addr learned my addr 172.21.15.102:0/3599322048 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:00:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.490+0000 7fc5cbfff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5cc153290 0x7fc5cc15bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:00:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5caffd700 1 -- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 msgr2=0x7fc5cc1e4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:00:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5caffd700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc1e4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5caffd700 1 -- 172.21.15.102:0/3599322048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5cc153290 msgr2=0x7fc5cc15bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:00:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5caffd700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5cc153290 0x7fc5cc15bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5caffd700 1 -- 172.21.15.102:0/3599322048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c4009a70 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5caffd700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0b1c70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009fd0 tx=0x7fc5c0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:00:34.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5b3fff700 1 -- 172.21.15.102:0/3599322048 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0004550 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5b3fff700 1 -- 172.21.15.102:0/3599322048 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c001e070 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5b3fff700 1 -- 172.21.15.102:0/3599322048 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00038c0 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.491+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c00094a0 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.492+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc0ccde0 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.492+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc15d160 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.494+0000 7fc5b3fff700 1 -- 172.21.15.102:0/3599322048 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5c000f050 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.495+0000 7fc5b3fff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5b80682f0 0x7fc5b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:00:34.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.495+0000 7fc5b3fff700 1 -- 172.21.15.102:0/3599322048 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc5c00874b0 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.495+0000 7fc5cb7fe700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5b80682f0 0x7fc5b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:00:34.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.496+0000 7fc5cb7fe700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5b80682f0 0x7fc5b806a7a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc007fd0 tx=0x7fc5bc007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:00:34.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.498+0000 7fc5b3fff700 1 -- 172.21.15.102:0/3599322048 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5c0053c40 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.732+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5cc0cd0c0 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.733+0000 7fc5b3fff700 1 -- 172.21.15.102:0/3599322048 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc5c0057260 con 0x7fc5cc0cd8e0 2023-03-03T07:00:34.735 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:00:34.736 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:00:34.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5b80682f0 msgr2=0x7fc5b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:00:34.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5b80682f0 0x7fc5b806a7a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc007fd0 tx=0x7fc5bc007dc0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 msgr2=0x7fc5cc0b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:00:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0b1c70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009fd0 tx=0x7fc5c0004750 comp rx=0 tx=0).stop 2023-03-03T07:00:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 shutdown_connections 2023-03-03T07:00:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5b80682f0 0x7fc5b806a7a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5cc0cd390 0x7fc5cc1e4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5cc153290 0x7fc5cc15bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 --2- 172.21.15.102:0/3599322048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5cc0cd8e0 0x7fc5cc0b1c70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:00:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 >> 172.21.15.102:0/3599322048 conn(0x7fc5cc065fb0 msgr2=0x7fc5cc158690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:00:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 shutdown_connections 2023-03-03T07:00:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:00:34.736+0000 7fc5d1dff700 1 -- 172.21.15.102:0/3599322048 wait complete. 2023-03-03T07:00:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:00:34.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-03-03T07:00:55.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:55.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:25.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:25.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:55.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:55.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:22.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:02:23.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.016+0000 7fdfb3fff700 1 Processor -- start 2023-03-03T07:02:23.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.016+0000 7fdfb3fff700 1 -- start start 2023-03-03T07:02:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdfb40ced00 0x7fdfb40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:02:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdfb40c41b0 0x7fdfb40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:02:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb3fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40c3d20 con 0x7fdfb40b3970 2023-03-03T07:02:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb3fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb40c4930 con 0x7fdfb40ced00 2023-03-03T07:02:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb40cf610 con 0x7fdfb40c41b0 2023-03-03T07:02:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb2ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdfb40ced00 0x7fdfb40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:02:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb27fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdfb40b3970 0x7fdfb4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54800/0 (socket says 172.21.15.102:54800) 2023-03-03T07:02:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb27fc700 1 -- 172.21.15.102:0/4083984712 learned_addr learned my addr 172.21.15.102:0/4083984712 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:02:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.018+0000 7fdfb37fe700 1 --2- 172.21.15.102:0/4083984712 >> v2:172.21.15.172:3300/0 conn(0x7fdfb40c41b0 0x7fdfb40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:02:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb37fe700 1 -- 172.21.15.102:0/4083984712 >> v2:172.21.15.102:3300/0 conn(0x7fdfb40ced00 msgr2=0x7fdfb40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:02:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb37fe700 1 --2- 172.21.15.102:0/4083984712 >> v2:172.21.15.102:3300/0 conn(0x7fdfb40ced00 0x7fdfb40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb37fe700 1 -- 172.21.15.102:0/4083984712 >> v1:172.21.15.172:6789/0 conn(0x7fdfb40b3970 legacy=0x7fdfb4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:02:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb37fe700 1 -- 172.21.15.102:0/4083984712 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb415cba0 con 0x7fdfb40c41b0 2023-03-03T07:02:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb1ffb700 1 -- 172.21.15.102:0/4083984712 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293651102 0 0) 0x7fdfb40c3d20 con 0x7fdfb40b3970 2023-03-03T07:02:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb37fe700 1 --2- 172.21.15.102:0/4083984712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40c41b0 0x7fdfb40cf0d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40ca080 tx=0x7fdfac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=262fa1d8e9977e32 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:02:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb1ffb700 1 -- 172.21.15.102:0/4083984712 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac00e070 con 0x7fdfb40c41b0 2023-03-03T07:02:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb1ffb700 1 -- 172.21.15.102:0/4083984712 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac00bb20 con 0x7fdfb40c41b0 2023-03-03T07:02:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.019+0000 7fdfb1ffb700 1 -- 172.21.15.102:0/4083984712 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac012430 con 0x7fdfb40c41b0 2023-03-03T07:02:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4083984712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40c41b0 msgr2=0x7fdfb40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:02:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4083984712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40c41b0 0x7fdfb40cf0d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40ca080 tx=0x7fdfac00b330 comp rx=0 tx=0).stop 2023-03-03T07:02:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4083984712 shutdown_connections 2023-03-03T07:02:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4083984712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40c41b0 0x7fdfb40cf0d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4083984712 >> v2:172.21.15.102:3300/0 conn(0x7fdfb40ced00 0x7fdfb40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4083984712 >> 172.21.15.102:0/4083984712 conn(0x7fdfb408b1c0 msgr2=0x7fdfb4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:02:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4083984712 shutdown_connections 2023-03-03T07:02:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.020+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4083984712 wait complete. 2023-03-03T07:02:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.022+0000 7fdfb3fff700 1 Processor -- start 2023-03-03T07:02:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.022+0000 7fdfb3fff700 1 -- start start 2023-03-03T07:02:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.023+0000 7fdfb3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfb40b3970 0x7fdfb415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:02:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.023+0000 7fdfb3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfb40c41b0 0x7fdfb414dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:02:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.023+0000 7fdfb3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40ced00 0x7fdfb414e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:02:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.023+0000 7fdfb3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb414e7f0 con 0x7fdfb40b3970 2023-03-03T07:02:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.023+0000 7fdfb3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40c8320 con 0x7fdfb40ced00 2023-03-03T07:02:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.023+0000 7fdfb3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40c9080 con 0x7fdfb40c41b0 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfb40b3970 0x7fdfb415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb27fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfb40c41b0 0x7fdfb414dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40ced00 0x7fdfb414e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb27fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfb40c41b0 0x7fdfb414dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:32870/0 (socket says 172.21.15.102:32870) 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb27fc700 1 -- 172.21.15.102:0/4044858504 learned_addr learned my addr 172.21.15.102:0/4044858504 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb37fe700 1 -- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfb40c41b0 msgr2=0x7fdfb414dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb37fe700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfb40c41b0 0x7fdfb414dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb37fe700 1 -- 172.21.15.102:0/4044858504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfb40b3970 msgr2=0x7fdfb415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:02:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb37fe700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfb40b3970 0x7fdfb415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb37fe700 1 -- 172.21.15.102:0/4044858504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac009a70 con 0x7fdfb40ced00 2023-03-03T07:02:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb27fc700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfb40c41b0 0x7fdfb414dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:02:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.024+0000 7fdfb37fe700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40ced00 0x7fdfb414e2b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40ca080 tx=0x7fdfac004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:02:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.025+0000 7fdf9b7fe700 1 -- 172.21.15.102:0/4044858504 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac01b810 con 0x7fdfb40ced00 2023-03-03T07:02:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.025+0000 7fdf9b7fe700 1 -- 172.21.15.102:0/4044858504 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac020070 con 0x7fdfb40ced00 2023-03-03T07:02:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.025+0000 7fdf9b7fe700 1 -- 172.21.15.102:0/4044858504 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac01bbd0 con 0x7fdfb40ced00 2023-03-03T07:02:23.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.025+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb406bf20 con 0x7fdfb40ced00 2023-03-03T07:02:23.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.025+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb415ce80 con 0x7fdfb40ced00 2023-03-03T07:02:23.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.026+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb406c200 con 0x7fdfb40ced00 2023-03-03T07:02:23.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.027+0000 7fdf9b7fe700 1 -- 172.21.15.102:0/4044858504 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdfac01bd30 con 0x7fdfb40ced00 2023-03-03T07:02:23.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.028+0000 7fdf9b7fe700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdfa0068320 0x7fdfa006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:02:23.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.028+0000 7fdf9b7fe700 1 -- 172.21.15.102:0/4044858504 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdfac022070 con 0x7fdfb40ced00 2023-03-03T07:02:23.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.029+0000 7fdfb2ffd700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdfa0068320 0x7fdfa006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:02:23.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.029+0000 7fdfb2ffd700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdfa0068320 0x7fdfa006a7d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009be0 tx=0x7fdfa4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:02:23.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.033+0000 7fdf9b7fe700 1 -- 172.21.15.102:0/4044858504 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdfac054290 con 0x7fdfb40ced00 2023-03-03T07:02:23.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.266+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfb404e6e0 con 0x7fdfb40ced00 2023-03-03T07:02:23.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.268+0000 7fdf9b7fe700 1 -- 172.21.15.102:0/4044858504 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdfac0578b0 con 0x7fdfb40ced00 2023-03-03T07:02:23.269 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:02:23.270 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:02:23.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdfa0068320 msgr2=0x7fdfa006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:02:23.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdfa0068320 0x7fdfa006a7d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009be0 tx=0x7fdfa4009210 comp rx=0 tx=0).stop 2023-03-03T07:02:23.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40ced00 msgr2=0x7fdfb414e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:02:23.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40ced00 0x7fdfb414e2b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40ca080 tx=0x7fdfac004b20 comp rx=0 tx=0).stop 2023-03-03T07:02:23.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 shutdown_connections 2023-03-03T07:02:23.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb40ced00 0x7fdfb414e2b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfb40b3970 0x7fdfb415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdfa0068320 0x7fdfa006a7d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 --2- 172.21.15.102:0/4044858504 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfb40c41b0 0x7fdfb414dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:02:23.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 >> 172.21.15.102:0/4044858504 conn(0x7fdfb408b1c0 msgr2=0x7fdfb4066930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:02:23.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 shutdown_connections 2023-03-03T07:02:23.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:02:23.270+0000 7fdfb3fff700 1 -- 172.21.15.102:0/4044858504 wait complete. 2023-03-03T07:02:23.274 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:02:23.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-03-03T07:02:25.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:25.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:55.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:55.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:25.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:25.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:52.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:03:52.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.568+0000 7fda705ff700 1 Processor -- start 2023-03-03T07:03:52.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.568+0000 7fda705ff700 1 -- start start 2023-03-03T07:03:52.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.569+0000 7fda705ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda68153290 0x7fda68158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:03:52.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.569+0000 7fda705ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda681593c0 0x7fda680cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:03:52.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda705ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda68153660 con 0x7fda680cd8e0 2023-03-03T07:03:52.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda705ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6806c360 con 0x7fda681593c0 2023-03-03T07:03:52.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda705ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6815cea0 con 0x7fda68153290 2023-03-03T07:03:52.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6db9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda681593c0 0x7fda680cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:03:52.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6e39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda68153290 0x7fda68158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:03:52.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6db9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fda681593c0 0x7fda680cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39266/0 (socket says 172.21.15.102:39266) 2023-03-03T07:03:52.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6eb9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fda680cd8e0 0x7fda680ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59602/0 (socket says 172.21.15.102:59602) 2023-03-03T07:03:52.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6db9a700 1 -- 172.21.15.102:0/3505812507 learned_addr learned my addr 172.21.15.102:0/3505812507 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:03:52.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6db9a700 1 -- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda68153290 msgr2=0x7fda68158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:03:52.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6db9a700 1 --2- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda68153290 0x7fda68158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6db9a700 1 -- 172.21.15.102:0/3505812507 >> v1:172.21.15.102:6789/0 conn(0x7fda680cd8e0 legacy=0x7fda680ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:03:52.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6d399700 1 -- 172.21.15.102:0/3505812507 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782082540 0 0) 0x7fda68153660 con 0x7fda680cd8e0 2023-03-03T07:03:52.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6db9a700 1 -- 172.21.15.102:0/3505812507 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6806c7f0 con 0x7fda681593c0 2023-03-03T07:03:52.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.570+0000 7fda6e39b700 1 --2- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda68153290 0x7fda68158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:03:52.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.571+0000 7fda6db9a700 1 --2- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda680cde20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fda580098f0 tx=0x7fda58009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=346a9d72b564154f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:03:52.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.571+0000 7fda6d399700 1 -- 172.21.15.102:0/3505812507 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58011830 con 0x7fda681593c0 2023-03-03T07:03:52.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.571+0000 7fda6d399700 1 -- 172.21.15.102:0/3505812507 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda58011990 con 0x7fda681593c0 2023-03-03T07:03:52.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.571+0000 7fda6d399700 1 -- 172.21.15.102:0/3505812507 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58011d20 con 0x7fda681593c0 2023-03-03T07:03:52.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 -- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 msgr2=0x7fda680cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:03:52.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 --2- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda680cde20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fda580098f0 tx=0x7fda58009c00 comp rx=0 tx=0).stop 2023-03-03T07:03:52.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 -- 172.21.15.102:0/3505812507 shutdown_connections 2023-03-03T07:03:52.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 --2- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda68153290 0x7fda68158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 --2- 172.21.15.102:0/3505812507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda680cde20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 -- 172.21.15.102:0/3505812507 >> 172.21.15.102:0/3505812507 conn(0x7fda68065fb0 msgr2=0x7fda680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:03:52.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 -- 172.21.15.102:0/3505812507 shutdown_connections 2023-03-03T07:03:52.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.572+0000 7fda705ff700 1 -- 172.21.15.102:0/3505812507 wait complete. 2023-03-03T07:03:52.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.574+0000 7fda705ff700 1 Processor -- start 2023-03-03T07:03:52.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.574+0000 7fda705ff700 1 -- start start 2023-03-03T07:03:52.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.575+0000 7fda705ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda680cd8e0 0x7fda680cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.575+0000 7fda705ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda68153290 0x7fda68154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.575+0000 7fda705ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda6814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.575+0000 7fda705ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda68153f50 con 0x7fda68153290 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.575+0000 7fda705ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda681501a0 con 0x7fda681593c0 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.575+0000 7fda705ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda680b2740 con 0x7fda680cd8e0 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda680cd8e0 0x7fda680cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6db9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda68153290 0x7fda68154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:03:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6eb9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda6814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda680cd8e0 0x7fda680cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60008/0 (socket says 172.21.15.102:60008) 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 -- 172.21.15.102:0/1375025655 learned_addr learned my addr 172.21.15.102:0/1375025655 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 -- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 msgr2=0x7fda6814fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda6814fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 -- 172.21.15.102:0/1375025655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda68153290 msgr2=0x7fda68154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda68153290 0x7fda68154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 -- 172.21.15.102:0/1375025655 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640094a0 con 0x7fda680cd8e0 2023-03-03T07:03:52.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6eb9c700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda6814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:03:52.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6db9a700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda68153290 0x7fda68154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:03:52.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.576+0000 7fda6e39b700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda680cd8e0 0x7fda680cad00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda6400ebe0 tx=0x7fda6400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:03:52.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.577+0000 7fda5effd700 1 -- 172.21.15.102:0/1375025655 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6400c850 con 0x7fda680cd8e0 2023-03-03T07:03:52.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.577+0000 7fda5effd700 1 -- 172.21.15.102:0/1375025655 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6400c9b0 con 0x7fda680cd8e0 2023-03-03T07:03:52.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.577+0000 7fda5effd700 1 -- 172.21.15.102:0/1375025655 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6400cd40 con 0x7fda680cd8e0 2023-03-03T07:03:52.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.577+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda58009550 con 0x7fda680cd8e0 2023-03-03T07:03:52.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.577+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda680ccde0 con 0x7fda680cd8e0 2023-03-03T07:03:52.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.578+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6815d160 con 0x7fda680cd8e0 2023-03-03T07:03:52.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.579+0000 7fda5effd700 1 -- 172.21.15.102:0/1375025655 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda64019030 con 0x7fda680cd8e0 2023-03-03T07:03:52.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.581+0000 7fda5effd700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda4c0682f0 0x7fda4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:03:52.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.581+0000 7fda5effd700 1 -- 172.21.15.102:0/1375025655 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fda64085d40 con 0x7fda680cd8e0 2023-03-03T07:03:52.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.581+0000 7fda6db9a700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda4c0682f0 0x7fda4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:03:52.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.582+0000 7fda6db9a700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda4c0682f0 0x7fda4c06a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fda58009130 tx=0x7fda58008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:03:52.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.585+0000 7fda5effd700 1 -- 172.21.15.102:0/1375025655 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda64052450 con 0x7fda680cd8e0 2023-03-03T07:03:52.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.818+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda680cd0c0 con 0x7fda680cd8e0 2023-03-03T07:03:52.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.819+0000 7fda5effd700 1 -- 172.21.15.102:0/1375025655 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fda6400cea0 con 0x7fda680cd8e0 2023-03-03T07:03:52.822 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:03:52.822 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:03:52.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda4c0682f0 msgr2=0x7fda4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:03:52.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda4c0682f0 0x7fda4c06a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fda58009130 tx=0x7fda58008030 comp rx=0 tx=0).stop 2023-03-03T07:03:52.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda680cd8e0 msgr2=0x7fda680cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:03:52.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda680cd8e0 0x7fda680cad00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda6400ebe0 tx=0x7fda6400eef0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 shutdown_connections 2023-03-03T07:03:52.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda4c0682f0 0x7fda4c06a7a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fda680cd8e0 0x7fda680cad00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda681593c0 0x7fda6814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 --2- 172.21.15.102:0/1375025655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda68153290 0x7fda68154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:03:52.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 >> 172.21.15.102:0/1375025655 conn(0x7fda68065fb0 msgr2=0x7fda681e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:03:52.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 shutdown_connections 2023-03-03T07:03:52.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:03:52.822+0000 7fda705ff700 1 -- 172.21.15.102:0/1375025655 wait complete. 2023-03-03T07:03:52.826 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:03:52.851 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 25 secs before thrashing 2023-03-03T07:03:56.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:56.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:17.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:04:18.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.102+0000 7f84e55ff700 1 Processor -- start 2023-03-03T07:04:18.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.102+0000 7f84e55ff700 1 -- start start 2023-03-03T07:04:18.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.104+0000 7f84e55ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e015ce50 0x7f84e00c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:04:18.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.104+0000 7f84e55ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e00c83c0 0x7f84e00c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:04:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.104+0000 7f84e55ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f84e00c9680 0x7f84e00c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:04:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.104+0000 7f84e55ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e0070010 con 0x7f84e015ce50 2023-03-03T07:04:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.104+0000 7f84e55ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e00ca670 con 0x7f84e00c9680 2023-03-03T07:04:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.104+0000 7f84e55ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e006fd30 con 0x7f84e00c83c0 2023-03-03T07:04:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84deffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e015ce50 0x7f84e00c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:04:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84deffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e015ce50 0x7f84e00c1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60246/0 (socket says 172.21.15.102:60246) 2023-03-03T07:04:18.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84deffd700 1 -- 172.21.15.102:0/1185961679 learned_addr learned my addr 172.21.15.102:0/1185961679 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:04:18.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84df7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f84e00c9680 0x7f84e00c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:04:18.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84de7fc700 1 --2- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e00c83c0 0x7f84e00c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:04:18.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84df7fe700 1 -- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e00c83c0 msgr2=0x7f84e00c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:04:18.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84df7fe700 1 --2- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e00c83c0 0x7f84e00c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84df7fe700 1 -- 172.21.15.102:0/1185961679 >> v2:172.21.15.102:3300/0 conn(0x7f84e015ce50 msgr2=0x7f84e00c1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:04:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84df7fe700 1 --2- 172.21.15.102:0/1185961679 >> v2:172.21.15.102:3300/0 conn(0x7f84e015ce50 0x7f84e00c1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.105+0000 7f84df7fe700 1 -- 172.21.15.102:0/1185961679 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00c8f90 con 0x7f84e00c9680 2023-03-03T07:04:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.106+0000 7f84df7fe700 1 --2- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 0x7f84e00c9aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f84e0071470 tx=0x7f84d800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212ec6dc8748f693 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:04:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.106+0000 7f84ddffb700 1 -- 172.21.15.102:0/1185961679 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800e070 con 0x7f84e00c9680 2023-03-03T07:04:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.106+0000 7f84ddffb700 1 -- 172.21.15.102:0/1185961679 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d800bb20 con 0x7f84e00c9680 2023-03-03T07:04:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.106+0000 7f84ddffb700 1 -- 172.21.15.102:0/1185961679 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8012430 con 0x7f84e00c9680 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 -- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 msgr2=0x7f84e00c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 --2- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 0x7f84e00c9aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f84e0071470 tx=0x7f84d800b330 comp rx=0 tx=0).stop 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 -- 172.21.15.102:0/1185961679 shutdown_connections 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 --2- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 0x7f84e00c9aa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 --2- 172.21.15.102:0/1185961679 >> v2:172.21.15.102:3300/0 conn(0x7f84e015ce50 0x7f84e00c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 --2- 172.21.15.102:0/1185961679 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e00c83c0 0x7f84e00c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 -- 172.21.15.102:0/1185961679 >> 172.21.15.102:0/1185961679 conn(0x7f84e0070540 msgr2=0x7f84e0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:04:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 -- 172.21.15.102:0/1185961679 shutdown_connections 2023-03-03T07:04:18.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.107+0000 7f84e55ff700 1 -- 172.21.15.102:0/1185961679 wait complete. 2023-03-03T07:04:18.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.109+0000 7f84e55ff700 1 Processor -- start 2023-03-03T07:04:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.109+0000 7f84e55ff700 1 -- start start 2023-03-03T07:04:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84e55ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00c83c0 0x7f84e01505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:04:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84e55ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 0x7f84e00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:04:18.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84e55ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e015ce50 0x7f84e00c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:04:18.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84e55ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00c46f0 con 0x7f84e00c83c0 2023-03-03T07:04:18.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84e55ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00c6950 con 0x7f84e00c9680 2023-03-03T07:04:18.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84e55ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e0159df0 con 0x7f84e015ce50 2023-03-03T07:04:18.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84deffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00c83c0 0x7f84e01505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:04:18.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84deffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00c83c0 0x7f84e01505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60260/0 (socket says 172.21.15.102:60260) 2023-03-03T07:04:18.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84deffd700 1 -- 172.21.15.102:0/606707773 learned_addr learned my addr 172.21.15.102:0/606707773 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:04:18.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84de7fc700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 0x7f84e00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:04:18.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.110+0000 7f84df7fe700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e015ce50 0x7f84e00c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:04:18.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84deffd700 1 -- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e015ce50 msgr2=0x7f84e00c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:04:18.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84deffd700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e015ce50 0x7f84e00c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84deffd700 1 -- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 msgr2=0x7f84e00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:04:18.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84deffd700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 0x7f84e00c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84deffd700 1 -- 172.21.15.102:0/606707773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8009a70 con 0x7f84e00c83c0 2023-03-03T07:04:18.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84deffd700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00c83c0 0x7f84e01505d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f84d000e910 tx=0x7f84d000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:04:18.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84c77fe700 1 -- 172.21.15.102:0/606707773 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d00096a0 con 0x7f84e00c83c0 2023-03-03T07:04:18.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84c77fe700 1 -- 172.21.15.102:0/606707773 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d0013680 con 0x7f84e00c83c0 2023-03-03T07:04:18.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e00c2450 con 0x7f84e00c83c0 2023-03-03T07:04:18.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.111+0000 7f84c77fe700 1 -- 172.21.15.102:0/606707773 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0013a30 con 0x7f84e00c83c0 2023-03-03T07:04:18.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.112+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e00c9270 con 0x7f84e00c83c0 2023-03-03T07:04:18.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.113+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e00c8020 con 0x7f84e00c83c0 2023-03-03T07:04:18.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.114+0000 7f84c77fe700 1 -- 172.21.15.102:0/606707773 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f84d0023070 con 0x7f84e00c83c0 2023-03-03T07:04:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.115+0000 7f84c77fe700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84cc0682f0 0x7f84cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:04:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.115+0000 7f84c77fe700 1 -- 172.21.15.102:0/606707773 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f84d0085e90 con 0x7f84e00c83c0 2023-03-03T07:04:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.116+0000 7f84de7fc700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84cc0682f0 0x7f84cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:04:18.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.116+0000 7f84de7fc700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84cc0682f0 0x7f84cc06a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f84e014def0 tx=0x7f84d4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:04:18.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.119+0000 7f84c77fe700 1 -- 172.21.15.102:0/606707773 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84d00525a0 con 0x7f84e00c83c0 2023-03-03T07:04:18.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.354+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e00677f0 con 0x7f84e00c83c0 2023-03-03T07:04:18.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.354+0000 7f84c77fe700 1 -- 172.21.15.102:0/606707773 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f84d0055bc0 con 0x7f84e00c83c0 2023-03-03T07:04:18.356 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:04:18.356 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:04:18.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84cc0682f0 msgr2=0x7f84cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:04:18.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84cc0682f0 0x7f84cc06a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f84e014def0 tx=0x7f84d4009210 comp rx=0 tx=0).stop 2023-03-03T07:04:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00c83c0 msgr2=0x7f84e01505d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:04:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00c83c0 0x7f84e01505d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f84d000e910 tx=0x7f84d000ec20 comp rx=0 tx=0).stop 2023-03-03T07:04:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 shutdown_connections 2023-03-03T07:04:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84cc0682f0 0x7f84cc06a7a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84e015ce50 0x7f84e00c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00c83c0 0x7f84e01505d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 --2- 172.21.15.102:0/606707773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84e00c9680 0x7f84e00c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:04:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 >> 172.21.15.102:0/606707773 conn(0x7f84e0070540 msgr2=0x7f84e015b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:04:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 shutdown_connections 2023-03-03T07:04:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:04:18.357+0000 7f84e55ff700 1 -- 172.21.15.102:0/606707773 wait complete. 2023-03-03T07:04:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:04:18.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-03-03T07:04:26.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:26.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:56.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:56.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:26.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:26.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:35.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:05:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.636+0000 7f065359e700 1 Processor -- start 2023-03-03T07:05:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.636+0000 7f065359e700 1 -- start start 2023-03-03T07:05:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f065359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0654153030 0x7f06540c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:05:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f065359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f06540b3f30 0x7f06540cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f065359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065406bee0 con 0x7f06540b1e90 2023-03-03T07:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f065359e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065406c780 con 0x7f06540b3f30 2023-03-03T07:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f065359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065415c620 con 0x7f0654153030 2023-03-03T07:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f0652d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f06540b3f30 0x7f06540cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f064bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f06540b1e90 0x7f065415ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46934/0 (socket says 172.21.15.102:46934) 2023-03-03T07:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f064bfff700 1 -- 172.21.15.102:0/4237924658 learned_addr learned my addr 172.21.15.102:0/4237924658 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:05:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.638+0000 7f065259c700 1 --2- 172.21.15.102:0/4237924658 >> v2:172.21.15.172:3300/0 conn(0x7f0654153030 0x7f06540c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0652d9d700 1 -- 172.21.15.102:0/4237924658 >> v2:172.21.15.172:3300/0 conn(0x7f0654153030 msgr2=0x7f06540c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0652d9d700 1 --2- 172.21.15.102:0/4237924658 >> v2:172.21.15.172:3300/0 conn(0x7f0654153030 0x7f06540c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0652d9d700 1 -- 172.21.15.102:0/4237924658 >> v1:172.21.15.102:6789/0 conn(0x7f06540b1e90 legacy=0x7f065415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0652d9d700 1 -- 172.21.15.102:0/4237924658 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065406c1c0 con 0x7f06540b3f30 2023-03-03T07:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0651d9b700 1 -- 172.21.15.102:0/4237924658 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483411590 0 0) 0x7f065406bee0 con 0x7f06540b1e90 2023-03-03T07:05:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0652d9d700 1 --2- 172.21.15.102:0/4237924658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b3f30 0x7f06540cece0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f064c009d90 tx=0x7f064c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd55844d7d99065 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0651d9b700 1 -- 172.21.15.102:0/4237924658 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c00e070 con 0x7f06540b3f30 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0651d9b700 1 -- 172.21.15.102:0/4237924658 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c00bae0 con 0x7f06540b3f30 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.639+0000 7f0651d9b700 1 -- 172.21.15.102:0/4237924658 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c00be90 con 0x7f06540b3f30 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 -- 172.21.15.102:0/4237924658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b3f30 msgr2=0x7f06540cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 --2- 172.21.15.102:0/4237924658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b3f30 0x7f06540cece0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f064c009d90 tx=0x7f064c00b330 comp rx=0 tx=0).stop 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 -- 172.21.15.102:0/4237924658 shutdown_connections 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 --2- 172.21.15.102:0/4237924658 >> v2:172.21.15.172:3300/0 conn(0x7f0654153030 0x7f06540c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 --2- 172.21.15.102:0/4237924658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b3f30 0x7f06540cece0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 -- 172.21.15.102:0/4237924658 >> 172.21.15.102:0/4237924658 conn(0x7f065408b1c0 msgr2=0x7f0654071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:05:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 -- 172.21.15.102:0/4237924658 shutdown_connections 2023-03-03T07:05:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.640+0000 7f065359e700 1 -- 172.21.15.102:0/4237924658 wait complete. 2023-03-03T07:05:35.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.642+0000 7f065359e700 1 Processor -- start 2023-03-03T07:05:35.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.642+0000 7f065359e700 1 -- start start 2023-03-03T07:05:35.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.643+0000 7f065359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b1e90 0x7f065415b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:05:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.643+0000 7f065359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06540b3f30 0x7f0654156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:05:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.643+0000 7f065359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0654153030 0x7f06541f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:05:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.643+0000 7f065359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c84c0 con 0x7f06540b1e90 2023-03-03T07:05:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.643+0000 7f065359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06541ef0e0 con 0x7f0654153030 2023-03-03T07:05:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.643+0000 7f065359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0654158270 con 0x7f06540b3f30 2023-03-03T07:05:35.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f064bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06540b3f30 0x7f0654156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f065259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b1e90 0x7f065415b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f064bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06540b3f30 0x7f0654156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:32796/0 (socket says 172.21.15.102:32796) 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f064bfff700 1 -- 172.21.15.102:0/815077489 learned_addr learned my addr 172.21.15.102:0/815077489 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f0652d9d700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0654153030 0x7f06541f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f065259c700 1 -- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06540b3f30 msgr2=0x7f0654156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f065259c700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06540b3f30 0x7f0654156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f065259c700 1 -- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0654153030 msgr2=0x7f06541f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:05:35.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f065259c700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0654153030 0x7f06541f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f065259c700 1 -- 172.21.15.102:0/815077489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c009a70 con 0x7f06540b1e90 2023-03-03T07:05:35.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.644+0000 7f0652d9d700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0654153030 0x7f06541f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:05:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.645+0000 7f065259c700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b1e90 0x7f065415b140 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f064400e870 tx=0x7f064400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:05:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.645+0000 7f064affd700 1 -- 172.21.15.102:0/815077489 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644013680 con 0x7f06540b1e90 2023-03-03T07:05:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.645+0000 7f064affd700 1 -- 172.21.15.102:0/815077489 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06440137e0 con 0x7f06540b1e90 2023-03-03T07:05:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.645+0000 7f064affd700 1 -- 172.21.15.102:0/815077489 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644013b70 con 0x7f06540b1e90 2023-03-03T07:05:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.645+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065415c240 con 0x7f06540b1e90 2023-03-03T07:05:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.645+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065406c4a0 con 0x7f06540b1e90 2023-03-03T07:05:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.646+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06540b39e0 con 0x7f06540b1e90 2023-03-03T07:05:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.647+0000 7f064affd700 1 -- 172.21.15.102:0/815077489 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0644019030 con 0x7f06540b1e90 2023-03-03T07:05:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.649+0000 7f064affd700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f06340682f0 0x7f063406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:05:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.649+0000 7f064affd700 1 -- 172.21.15.102:0/815077489 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0644085bf0 con 0x7f06540b1e90 2023-03-03T07:05:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.649+0000 7f064bfff700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f06340682f0 0x7f063406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:05:35.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.650+0000 7f064bfff700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f06340682f0 0x7f063406a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0654150fc0 tx=0x7f063c007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:05:35.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.653+0000 7f064affd700 1 -- 172.21.15.102:0/815077489 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0644052270 con 0x7f06540b1e90 2023-03-03T07:05:35.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.889+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f065404e6e0 con 0x7f06540b1e90 2023-03-03T07:05:35.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.890+0000 7f064affd700 1 -- 172.21.15.102:0/815077489 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0644021090 con 0x7f06540b1e90 2023-03-03T07:05:35.893 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:05:35.893 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:05:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f06340682f0 msgr2=0x7f063406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:05:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f06340682f0 0x7f063406a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0654150fc0 tx=0x7f063c007e10 comp rx=0 tx=0).stop 2023-03-03T07:05:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b1e90 msgr2=0x7f065415b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:05:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b1e90 0x7f065415b140 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f064400e870 tx=0x7f064400eb80 comp rx=0 tx=0).stop 2023-03-03T07:05:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 shutdown_connections 2023-03-03T07:05:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0654153030 0x7f06541f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06540b1e90 0x7f065415b140 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f06340682f0 0x7f063406a7a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 --2- 172.21.15.102:0/815077489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f06540b3f30 0x7f0654156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:05:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 >> 172.21.15.102:0/815077489 conn(0x7f065408b1c0 msgr2=0x7f06540cc120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:05:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 shutdown_connections 2023-03-03T07:05:35.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:05:35.893+0000 7f065359e700 1 -- 172.21.15.102:0/815077489 wait complete. 2023-03-03T07:05:35.897 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:05:35.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-03-03T07:05:56.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:56.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:27.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:27.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:57.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:57.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:04.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:07:05.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.177+0000 7f55207ff700 1 Processor -- start 2023-03-03T07:07:05.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.177+0000 7f55207ff700 1 -- start start 2023-03-03T07:07:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.178+0000 7f55207ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5518153290 0x7f5518158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:07:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.178+0000 7f55207ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5518153660 con 0x7f55180cd8e0 2023-03-03T07:07:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.178+0000 7f55207ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551806c360 con 0x7f55181593c0 2023-03-03T07:07:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.178+0000 7f55207ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55180cec40 con 0x7f5518153290 2023-03-03T07:07:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551e59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5518153290 0x7f5518158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:07:05.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551e59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5518153290 0x7f5518158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:32916/0 (socket says 172.21.15.102:32916) 2023-03-03T07:07:05.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551dd9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f55181593c0 0x7f55180cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50126/0 (socket says 172.21.15.102:50126) 2023-03-03T07:07:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551e59b700 1 -- 172.21.15.102:0/2927998682 learned_addr learned my addr 172.21.15.102:0/2927998682 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:07:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551e59b700 1 -- 172.21.15.102:0/2927998682 >> v1:172.21.15.172:6789/0 conn(0x7f55181593c0 legacy=0x7f55180cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:07:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551e59b700 1 -- 172.21.15.102:0/2927998682 >> v1:172.21.15.102:6789/0 conn(0x7f55180cd8e0 legacy=0x7f55180ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:07:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551e59b700 1 -- 172.21.15.102:0/2927998682 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551815cea0 con 0x7f5518153290 2023-03-03T07:07:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.179+0000 7f551d599700 1 -- 172.21.15.102:0/2927998682 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049611563 0 0) 0x7f551806c360 con 0x7f55181593c0 2023-03-03T07:07:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.180+0000 7f551e59b700 1 --2- 172.21.15.102:0/2927998682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f5518158e80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55140098f0 tx=0x7f5514009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3681d5b9b1e6b0fa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:07:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.180+0000 7f551d599700 1 -- 172.21.15.102:0/2927998682 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514004230 con 0x7f5518153290 2023-03-03T07:07:05.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.180+0000 7f551d599700 1 -- 172.21.15.102:0/2927998682 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5514004390 con 0x7f5518153290 2023-03-03T07:07:05.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.180+0000 7f551d599700 1 -- 172.21.15.102:0/2927998682 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55140036a0 con 0x7f5518153290 2023-03-03T07:07:05.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.181+0000 7f55207ff700 1 -- 172.21.15.102:0/2927998682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 msgr2=0x7f5518158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:07:05.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.181+0000 7f55207ff700 1 --2- 172.21.15.102:0/2927998682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f5518158e80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55140098f0 tx=0x7f5514009c00 comp rx=0 tx=0).stop 2023-03-03T07:07:05.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.181+0000 7f55207ff700 1 -- 172.21.15.102:0/2927998682 shutdown_connections 2023-03-03T07:07:05.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.181+0000 7f55207ff700 1 --2- 172.21.15.102:0/2927998682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f5518158e80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.181+0000 7f55207ff700 1 -- 172.21.15.102:0/2927998682 >> 172.21.15.102:0/2927998682 conn(0x7f5518065fb0 msgr2=0x7f55180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:07:05.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.181+0000 7f55207ff700 1 -- 172.21.15.102:0/2927998682 shutdown_connections 2023-03-03T07:07:05.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.181+0000 7f55207ff700 1 -- 172.21.15.102:0/2927998682 wait complete. 2023-03-03T07:07:05.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.183+0000 7f55207ff700 1 Processor -- start 2023-03-03T07:07:05.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.183+0000 7f55207ff700 1 -- start start 2023-03-03T07:07:05.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f55207ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55180cd8e0 0x7f55180cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:07:05.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f55207ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f551814e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:07:05.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f55207ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55181593c0 0x7f551814e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:07:05.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f55207ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55180c6b10 con 0x7f5518153290 2023-03-03T07:07:05.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f55207ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55181ef8e0 con 0x7f55181593c0 2023-03-03T07:07:05.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f55207ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55181ef380 con 0x7f55180cd8e0 2023-03-03T07:07:05.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f551dd9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f551814e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:07:05.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f551dd9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f551814e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:32922/0 (socket says 172.21.15.102:32922) 2023-03-03T07:07:05.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f551dd9a700 1 -- 172.21.15.102:0/1023966049 learned_addr learned my addr 172.21.15.102:0/1023966049 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:07:05.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f551e59b700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55180cd8e0 0x7f55180cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:07:05.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.184+0000 7f551ed9c700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55181593c0 0x7f551814e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:07:05.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f551dd9a700 1 -- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55180cd8e0 msgr2=0x7f55180cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:07:05.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f551dd9a700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55180cd8e0 0x7f55180cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f551dd9a700 1 -- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55181593c0 msgr2=0x7f551814e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:07:05.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f551dd9a700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55181593c0 0x7f551814e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f551dd9a700 1 -- 172.21.15.102:0/1023966049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5514009550 con 0x7f5518153290 2023-03-03T07:07:05.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f551dd9a700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f551814e140 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f550800c4a0 tx=0x7f550800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:07:05.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f550effd700 1 -- 172.21.15.102:0/1023966049 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508004c80 con 0x7f5518153290 2023-03-03T07:07:05.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f550effd700 1 -- 172.21.15.102:0/1023966049 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5508004de0 con 0x7f5518153290 2023-03-03T07:07:05.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551815d160 con 0x7f5518153290 2023-03-03T07:07:05.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.185+0000 7f550effd700 1 -- 172.21.15.102:0/1023966049 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508007310 con 0x7f5518153290 2023-03-03T07:07:05.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.186+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551806c7f0 con 0x7f5518153290 2023-03-03T07:07:05.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.187+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55180ccde0 con 0x7f5518153290 2023-03-03T07:07:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.188+0000 7f550effd700 1 -- 172.21.15.102:0/1023966049 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f550801a020 con 0x7f5518153290 2023-03-03T07:07:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.189+0000 7f550effd700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54fc068320 0x7f54fc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:07:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.189+0000 7f550effd700 1 -- 172.21.15.102:0/1023966049 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5508085850 con 0x7f5518153290 2023-03-03T07:07:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.189+0000 7f551e59b700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54fc068320 0x7f54fc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:07:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.190+0000 7f551e59b700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54fc068320 0x7f54fc06a7d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5514009130 tx=0x7f5514008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:07:05.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.193+0000 7f550effd700 1 -- 172.21.15.102:0/1023966049 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5508051f60 con 0x7f5518153290 2023-03-03T07:07:05.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.431+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5518158810 con 0x7f5518153290 2023-03-03T07:07:05.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.432+0000 7f550effd700 1 -- 172.21.15.102:0/1023966049 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5508055580 con 0x7f5518153290 2023-03-03T07:07:05.435 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:07:05.435 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:07:05.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54fc068320 msgr2=0x7f54fc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:07:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54fc068320 0x7f54fc06a7d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5514009130 tx=0x7f5514008030 comp rx=0 tx=0).stop 2023-03-03T07:07:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 msgr2=0x7f551814e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:07:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f551814e140 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f550800c4a0 tx=0x7f550800c7b0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 shutdown_connections 2023-03-03T07:07:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54fc068320 0x7f54fc06a7d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55180cd8e0 0x7f55180cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55181593c0 0x7f551814e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 --2- 172.21.15.102:0/1023966049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5518153290 0x7f551814e140 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:07:05.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 >> 172.21.15.102:0/1023966049 conn(0x7f5518065fb0 msgr2=0x7f551815b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:07:05.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 shutdown_connections 2023-03-03T07:07:05.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:07:05.435+0000 7f55207ff700 1 -- 172.21.15.102:0/1023966049 wait complete. 2023-03-03T07:07:05.440 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:07:05.465 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T07:07:27.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:27.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:57.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:57.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:05.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:08:05.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.716+0000 7f9978f9d700 1 Processor -- start 2023-03-03T07:08:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.716+0000 7f9978f9d700 1 -- start start 2023-03-03T07:08:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9978f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99740ced00 0x7f99740c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9978f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740c41b0 0x7f99740cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9978f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740c3d20 con 0x7f99740b3970 2023-03-03T07:08:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9978f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99740c4930 con 0x7f99740ced00 2023-03-03T07:08:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9978f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740cf610 con 0x7f99740c41b0 2023-03-03T07:08:05.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9973fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740c41b0 0x7f99740cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f99737fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99740ced00 0x7f99740c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9972ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f99740b3970 0x7f9974158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37232/0 (socket says 172.21.15.102:37232) 2023-03-03T07:08:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.718+0000 7f9972ffd700 1 -- 172.21.15.102:0/3786205600 learned_addr learned my addr 172.21.15.102:0/3786205600 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:08:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f9973fff700 1 -- 172.21.15.102:0/3786205600 >> v2:172.21.15.172:3300/0 conn(0x7f99740ced00 msgr2=0x7f99740c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:08:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f9973fff700 1 --2- 172.21.15.102:0/3786205600 >> v2:172.21.15.172:3300/0 conn(0x7f99740ced00 0x7f99740c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f9973fff700 1 -- 172.21.15.102:0/3786205600 >> v1:172.21.15.102:6789/0 conn(0x7f99740b3970 legacy=0x7f9974158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f9973fff700 1 -- 172.21.15.102:0/3786205600 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997415cba0 con 0x7f99740c41b0 2023-03-03T07:08:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f99737fe700 1 --2- 172.21.15.102:0/3786205600 >> v2:172.21.15.172:3300/0 conn(0x7f99740ced00 0x7f99740c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:08:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f9973fff700 1 --2- 172.21.15.102:0/3786205600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740c41b0 0x7f99740cf0d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f996c009d90 tx=0x7f996c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d82f27ee9f37269 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f99727fc700 1 -- 172.21.15.102:0/3786205600 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00e070 con 0x7f99740c41b0 2023-03-03T07:08:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f99727fc700 1 -- 172.21.15.102:0/3786205600 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c00bb20 con 0x7f99740c41b0 2023-03-03T07:08:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.719+0000 7f99727fc700 1 -- 172.21.15.102:0/3786205600 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c012430 con 0x7f99740c41b0 2023-03-03T07:08:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 -- 172.21.15.102:0/3786205600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740c41b0 msgr2=0x7f99740cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3786205600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740c41b0 0x7f99740cf0d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f996c009d90 tx=0x7f996c00b330 comp rx=0 tx=0).stop 2023-03-03T07:08:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 -- 172.21.15.102:0/3786205600 shutdown_connections 2023-03-03T07:08:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3786205600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740c41b0 0x7f99740cf0d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3786205600 >> v2:172.21.15.172:3300/0 conn(0x7f99740ced00 0x7f99740c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 -- 172.21.15.102:0/3786205600 >> 172.21.15.102:0/3786205600 conn(0x7f997408b1c0 msgr2=0x7f9974071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 -- 172.21.15.102:0/3786205600 shutdown_connections 2023-03-03T07:08:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.720+0000 7f9978f9d700 1 -- 172.21.15.102:0/3786205600 wait complete. 2023-03-03T07:08:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.722+0000 7f9978f9d700 1 Processor -- start 2023-03-03T07:08:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.722+0000 7f9978f9d700 1 -- start start 2023-03-03T07:08:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.723+0000 7f9978f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740b3970 0x7f99741f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.723+0000 7f9978f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99740c41b0 0x7f9974150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.723+0000 7f9978f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99740ced00 0x7f99740b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.723+0000 7f9978f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740c66a0 con 0x7f99740c41b0 2023-03-03T07:08:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.723+0000 7f9978f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740c7f50 con 0x7f99740ced00 2023-03-03T07:08:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.723+0000 7f9978f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740c6070 con 0x7f99740b3970 2023-03-03T07:08:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.723+0000 7f99737fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740b3970 0x7f99741f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f9973fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99740ced00 0x7f99740b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f9972ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99740c41b0 0x7f9974150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740b3970 0x7f99741f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44114/0 (socket says 172.21.15.102:44114) 2023-03-03T07:08:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 -- 172.21.15.102:0/3712109903 learned_addr learned my addr 172.21.15.102:0/3712109903 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:08:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f9973fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99740ced00 0x7f99740b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57594/0 (socket says 172.21.15.102:57594) 2023-03-03T07:08:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 -- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99740ced00 msgr2=0x7f99740b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99740ced00 0x7f99740b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 -- 172.21.15.102:0/3712109903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99740c41b0 msgr2=0x7f9974150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99740c41b0 0x7f9974150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 -- 172.21.15.102:0/3712109903 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99640094a0 con 0x7f99740b3970 2023-03-03T07:08:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f9972ffd700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99740c41b0 0x7f9974150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:08:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f9973fff700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99740ced00 0x7f99740b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:08:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.724+0000 7f99737fe700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740b3970 0x7f99741f1740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f996400ebe0 tx=0x7f996400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.725+0000 7f995bfff700 1 -- 172.21.15.102:0/3712109903 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996400c850 con 0x7f99740b3970 2023-03-03T07:08:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.725+0000 7f995bfff700 1 -- 172.21.15.102:0/3712109903 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996400c9b0 con 0x7f99740b3970 2023-03-03T07:08:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.725+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c009a70 con 0x7f99740b3970 2023-03-03T07:08:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.725+0000 7f995bfff700 1 -- 172.21.15.102:0/3712109903 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996400cd40 con 0x7f99740b3970 2023-03-03T07:08:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.725+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997406bf20 con 0x7f99740b3970 2023-03-03T07:08:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.726+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997415ce80 con 0x7f99740b3970 2023-03-03T07:08:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.727+0000 7f995bfff700 1 -- 172.21.15.102:0/3712109903 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9964023070 con 0x7f99740b3970 2023-03-03T07:08:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.729+0000 7f995bfff700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9960068320 0x7f996006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.729+0000 7f995bfff700 1 -- 172.21.15.102:0/3712109903 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f99640864c0 con 0x7f99740b3970 2023-03-03T07:08:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.729+0000 7f9972ffd700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9960068320 0x7f996006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.729+0000 7f9972ffd700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9960068320 0x7f996006a7d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f997414f420 tx=0x7f9968009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:05.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.731+0000 7f995bfff700 1 -- 172.21.15.102:0/3712109903 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9964052bc0 con 0x7f99740b3970 2023-03-03T07:08:05.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.967+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f997406c200 con 0x7f99740b3970 2023-03-03T07:08:05.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.968+0000 7f995bfff700 1 -- 172.21.15.102:0/3712109903 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f996400cea0 con 0x7f99740b3970 2023-03-03T07:08:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:08:05.970 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:08:05.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9960068320 msgr2=0x7f996006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:05.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9960068320 0x7f996006a7d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f997414f420 tx=0x7f9968009040 comp rx=0 tx=0).stop 2023-03-03T07:08:05.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740b3970 msgr2=0x7f99741f1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:05.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740b3970 0x7f99741f1740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f996400ebe0 tx=0x7f996400eef0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 shutdown_connections 2023-03-03T07:08:05.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9960068320 0x7f996006a7d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f99740b3970 0x7f99741f1740 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99740ced00 0x7f99740b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 --2- 172.21.15.102:0/3712109903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99740c41b0 0x7f9974150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:05.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 >> 172.21.15.102:0/3712109903 conn(0x7f997408b1c0 msgr2=0x7f9974065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:05.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 shutdown_connections 2023-03-03T07:08:05.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:05.971+0000 7f9978f9d700 1 -- 172.21.15.102:0/3712109903 wait complete. 2023-03-03T07:08:05.974 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:08:06.000 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 26 secs before thrashing 2023-03-03T07:08:27.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:27.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:32.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:08:32.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.264+0000 7f5c305ff700 1 Processor -- start 2023-03-03T07:08:32.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.264+0000 7f5c305ff700 1 -- start start 2023-03-03T07:08:32.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c305ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5c280cd8e0 0x7f5c280cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:32.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c305ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c280ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:32.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c305ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c28153660 con 0x7f5c28153290 2023-03-03T07:08:32.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c305ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c2806c360 con 0x7f5c280cd8e0 2023-03-03T07:08:32.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c305ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2815cea0 con 0x7f5c280cd390 2023-03-03T07:08:32.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c2e39b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5c28153290 0x7f5c28158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36222/0 (socket says 172.21.15.102:36222) 2023-03-03T07:08:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c2e39b700 1 -- 172.21.15.102:0/1091365045 learned_addr learned my addr 172.21.15.102:0/1091365045 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:08:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c2eb9c700 1 --2- 172.21.15.102:0/1091365045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c280ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.266+0000 7f5c2db9a700 1 --2- 172.21.15.102:0/1091365045 >> v2:172.21.15.172:3300/0 conn(0x7f5c280cd8e0 0x7f5c280cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2d399700 1 -- 172.21.15.102:0/1091365045 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565713150 0 0) 0x7f5c28153660 con 0x7f5c28153290 2023-03-03T07:08:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2d399700 1 -- 172.21.15.102:0/1091365045 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c140035b0 con 0x7f5c28153290 2023-03-03T07:08:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2eb9c700 1 -- 172.21.15.102:0/1091365045 >> v2:172.21.15.172:3300/0 conn(0x7f5c280cd8e0 msgr2=0x7f5c280cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:08:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2eb9c700 1 --2- 172.21.15.102:0/1091365045 >> v2:172.21.15.172:3300/0 conn(0x7f5c280cd8e0 0x7f5c280cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2eb9c700 1 -- 172.21.15.102:0/1091365045 >> v1:172.21.15.102:6789/0 conn(0x7f5c28153290 legacy=0x7f5c28158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:32.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2eb9c700 1 -- 172.21.15.102:0/1091365045 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2806c7f0 con 0x7f5c280cd390 2023-03-03T07:08:32.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2eb9c700 1 --2- 172.21.15.102:0/1091365045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c280ce360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009d90 tx=0x7f5c2000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72b15a1ce862a146 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:32.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2d399700 1 -- 172.21.15.102:0/1091365045 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2000e070 con 0x7f5c280cd390 2023-03-03T07:08:32.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2d399700 1 -- 172.21.15.102:0/1091365045 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2000bb20 con 0x7f5c280cd390 2023-03-03T07:08:32.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.267+0000 7f5c2d399700 1 -- 172.21.15.102:0/1091365045 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20012430 con 0x7f5c280cd390 2023-03-03T07:08:32.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 -- 172.21.15.102:0/1091365045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 msgr2=0x7f5c280ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:32.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 --2- 172.21.15.102:0/1091365045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c280ce360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009d90 tx=0x7f5c2000b330 comp rx=0 tx=0).stop 2023-03-03T07:08:32.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 -- 172.21.15.102:0/1091365045 shutdown_connections 2023-03-03T07:08:32.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 --2- 172.21.15.102:0/1091365045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c280ce360 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 --2- 172.21.15.102:0/1091365045 >> v2:172.21.15.172:3300/0 conn(0x7f5c280cd8e0 0x7f5c280cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 -- 172.21.15.102:0/1091365045 >> 172.21.15.102:0/1091365045 conn(0x7f5c28065fb0 msgr2=0x7f5c280663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:32.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 -- 172.21.15.102:0/1091365045 shutdown_connections 2023-03-03T07:08:32.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.268+0000 7f5c305ff700 1 -- 172.21.15.102:0/1091365045 wait complete. 2023-03-03T07:08:32.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.270+0000 7f5c305ff700 1 Processor -- start 2023-03-03T07:08:32.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.270+0000 7f5c305ff700 1 -- start start 2023-03-03T07:08:32.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.271+0000 7f5c305ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c281f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:32.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.271+0000 7f5c305ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c280cd8e0 0x7f5c280b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:32.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.271+0000 7f5c305ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c28153290 0x7f5c2814e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:32.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.271+0000 7f5c305ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c28153fa0 con 0x7f5c280cd8e0 2023-03-03T07:08:32.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.271+0000 7f5c305ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280c78d0 con 0x7f5c28153290 2023-03-03T07:08:32.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.271+0000 7f5c305ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280c70c0 con 0x7f5c280cd390 2023-03-03T07:08:32.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c280cd8e0 0x7f5c280b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c280cd8e0 0x7f5c280b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43252/0 (socket says 172.21.15.102:43252) 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 -- 172.21.15.102:0/4184181391 learned_addr learned my addr 172.21.15.102:0/4184181391 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2e39b700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c281f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2eb9c700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c28153290 0x7f5c2814e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 -- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 msgr2=0x7f5c281f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c281f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 -- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c28153290 msgr2=0x7f5c2814e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:32.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c28153290 0x7f5c2814e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.272+0000 7f5c2db9a700 1 -- 172.21.15.102:0/4184181391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c20009a70 con 0x7f5c280cd8e0 2023-03-03T07:08:32.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.273+0000 7f5c2db9a700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c280cd8e0 0x7f5c280b40b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800e8a0 tx=0x7f5c1800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:32.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.273+0000 7f5c1effd700 1 -- 172.21.15.102:0/4184181391 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18009590 con 0x7f5c280cd8e0 2023-03-03T07:08:32.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.273+0000 7f5c1effd700 1 -- 172.21.15.102:0/4184181391 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18013680 con 0x7f5c280cd8e0 2023-03-03T07:08:32.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.273+0000 7f5c1effd700 1 -- 172.21.15.102:0/4184181391 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c180139f0 con 0x7f5c280cd8e0 2023-03-03T07:08:32.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.273+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c280ccde0 con 0x7f5c280cd8e0 2023-03-03T07:08:32.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.273+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2815d160 con 0x7f5c280cd8e0 2023-03-03T07:08:32.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.274+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c280cd0c0 con 0x7f5c280cd8e0 2023-03-03T07:08:32.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.275+0000 7f5c1effd700 1 -- 172.21.15.102:0/4184181391 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c18019030 con 0x7f5c280cd8e0 2023-03-03T07:08:32.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.277+0000 7f5c1effd700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c0c0682f0 0x7f5c0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:32.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.277+0000 7f5c1effd700 1 -- 172.21.15.102:0/4184181391 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5c18085a90 con 0x7f5c280cd8e0 2023-03-03T07:08:32.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.277+0000 7f5c2e39b700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c0c0682f0 0x7f5c0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:32.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.278+0000 7f5c2e39b700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c0c0682f0 0x7f5c0c06a7a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5c24007fd0 tx=0x7f5c24007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:32.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.281+0000 7f5c1effd700 1 -- 172.21.15.102:0/4184181391 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c180521a0 con 0x7f5c280cd8e0 2023-03-03T07:08:32.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.516+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c28152ad0 con 0x7f5c280cd8e0 2023-03-03T07:08:32.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.516+0000 7f5c1effd700 1 -- 172.21.15.102:0/4184181391 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5c180557c0 con 0x7f5c280cd8e0 2023-03-03T07:08:32.519 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:08:32.519 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:08:32.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c0c0682f0 msgr2=0x7f5c0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c0c0682f0 0x7f5c0c06a7a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5c24007fd0 tx=0x7f5c24007dc0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c280cd8e0 msgr2=0x7f5c280b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c280cd8e0 0x7f5c280b40b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800e8a0 tx=0x7f5c1800ebb0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 shutdown_connections 2023-03-03T07:08:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c0c0682f0 0x7f5c0c06a7a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c280cd390 0x7f5c281f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c28153290 0x7f5c2814e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 --2- 172.21.15.102:0/4184181391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c280cd8e0 0x7f5c280b40b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.519+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 >> 172.21.15.102:0/4184181391 conn(0x7f5c28065fb0 msgr2=0x7f5c28156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.520+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 shutdown_connections 2023-03-03T07:08:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:32.520+0000 7f5c305ff700 1 -- 172.21.15.102:0/4184181391 wait complete. 2023-03-03T07:08:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:08:32.548 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-03-03T07:08:42.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:08:42.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.798+0000 7f54cf3ff700 1 Processor -- start 2023-03-03T07:08:42.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.798+0000 7f54cf3ff700 1 -- start start 2023-03-03T07:08:42.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.799+0000 7f54cf3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f54c8153290 0x7f54c8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.799+0000 7f54cf3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c81593c0 0x7f54c80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cf3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54c80cd8e0 0x7f54c80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cf3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c8153660 con 0x7f54c8153290 2023-03-03T07:08:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cf3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c806c360 con 0x7f54c80cd8e0 2023-03-03T07:08:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cf3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c815cea0 con 0x7f54c81593c0 2023-03-03T07:08:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cd19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f54c8153290 0x7f54c8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cd99c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54c80cd8e0 0x7f54c80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cc99a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c81593c0 0x7f54c80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cd99c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54c80cd8e0 0x7f54c80ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53102/0 (socket says 172.21.15.102:53102) 2023-03-03T07:08:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cc99a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c81593c0 0x7f54c80cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36828/0 (socket says 172.21.15.102:36828) 2023-03-03T07:08:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.800+0000 7f54cd99c700 1 -- 172.21.15.102:0/2096513869 learned_addr learned my addr 172.21.15.102:0/2096513869 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:08:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54cd19b700 1 -- 172.21.15.102:0/2096513869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c81593c0 msgr2=0x7f54c80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:08:42.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54cd19b700 1 --2- 172.21.15.102:0/2096513869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c81593c0 0x7f54c80cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:42.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54cd19b700 1 -- 172.21.15.102:0/2096513869 >> v2:172.21.15.172:3300/0 conn(0x7f54c80cd8e0 msgr2=0x7f54c80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:08:42.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54cd19b700 1 --2- 172.21.15.102:0/2096513869 >> v2:172.21.15.172:3300/0 conn(0x7f54c80cd8e0 0x7f54c80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:42.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54cd19b700 1 -- 172.21.15.102:0/2096513869 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c806c7f0 con 0x7f54c8153290 2023-03-03T07:08:42.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54cd19b700 1 --2- 172.21.15.102:0/2096513869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c8158e80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54c806d0e0 tx=0x7f54c4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6972776e8e35c5d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:42.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54bffff700 1 -- 172.21.15.102:0/2096513869 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c4011830 con 0x7f54c8153290 2023-03-03T07:08:42.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54bffff700 1 -- 172.21.15.102:0/2096513869 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c4011990 con 0x7f54c8153290 2023-03-03T07:08:42.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.801+0000 7f54bffff700 1 -- 172.21.15.102:0/2096513869 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c4011d20 con 0x7f54c8153290 2023-03-03T07:08:42.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 -- 172.21.15.102:0/2096513869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 msgr2=0x7f54c8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:42.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/2096513869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c8158e80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54c806d0e0 tx=0x7f54c4009b20 comp rx=0 tx=0).stop 2023-03-03T07:08:42.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 -- 172.21.15.102:0/2096513869 shutdown_connections 2023-03-03T07:08:42.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/2096513869 >> v2:172.21.15.172:3300/0 conn(0x7f54c80cd8e0 0x7f54c80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:42.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/2096513869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c8158e80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:42.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/2096513869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c81593c0 0x7f54c80cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:42.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 -- 172.21.15.102:0/2096513869 >> 172.21.15.102:0/2096513869 conn(0x7f54c8065fb0 msgr2=0x7f54c80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:42.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 -- 172.21.15.102:0/2096513869 shutdown_connections 2023-03-03T07:08:42.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.802+0000 7f54cf3ff700 1 -- 172.21.15.102:0/2096513869 wait complete. 2023-03-03T07:08:42.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.804+0000 7f54cf3ff700 1 Processor -- start 2023-03-03T07:08:42.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.804+0000 7f54cf3ff700 1 -- start start 2023-03-03T07:08:42.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.805+0000 7f54cf3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c80cd8e0 0x7f54c81f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:42.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.805+0000 7f54cf3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c81e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.805+0000 7f54cf3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c81593c0 0x7f54c814d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.805+0000 7f54cf3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c80c7fc0 con 0x7f54c8153290 2023-03-03T07:08:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.805+0000 7f54cf3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c81f2b00 con 0x7f54c81593c0 2023-03-03T07:08:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.805+0000 7f54cf3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c80c7ce0 con 0x7f54c80cd8e0 2023-03-03T07:08:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cd19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c80cd8e0 0x7f54c81f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cd99c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c81593c0 0x7f54c814d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cc99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c81e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:42.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cd19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c80cd8e0 0x7f54c81f1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36842/0 (socket says 172.21.15.102:36842) 2023-03-03T07:08:42.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cd19b700 1 -- 172.21.15.102:0/392696227 learned_addr learned my addr 172.21.15.102:0/392696227 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:08:42.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cc99a700 1 -- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c80cd8e0 msgr2=0x7f54c81f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:42.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cc99a700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c80cd8e0 0x7f54c81f1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:42.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cc99a700 1 -- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c81593c0 msgr2=0x7f54c814d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:42.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cc99a700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c81593c0 0x7f54c814d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:42.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cc99a700 1 -- 172.21.15.102:0/392696227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c40094a0 con 0x7f54c8153290 2023-03-03T07:08:42.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cd19b700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c80cd8e0 0x7f54c81f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:08:42.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cd99c700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c81593c0 0x7f54c814d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:08:42.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.806+0000 7f54cc99a700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c81e5b20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f54b800e900 tx=0x7f54b800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:42.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.807+0000 7f54bdffb700 1 -- 172.21.15.102:0/392696227 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8013680 con 0x7f54c8153290 2023-03-03T07:08:42.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.807+0000 7f54bdffb700 1 -- 172.21.15.102:0/392696227 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b80137e0 con 0x7f54c8153290 2023-03-03T07:08:42.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.807+0000 7f54bdffb700 1 -- 172.21.15.102:0/392696227 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8013b50 con 0x7f54c8153290 2023-03-03T07:08:42.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.807+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c80ccde0 con 0x7f54c8153290 2023-03-03T07:08:42.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.807+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c815d160 con 0x7f54c8153290 2023-03-03T07:08:42.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.808+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c80cd0c0 con 0x7f54c8153290 2023-03-03T07:08:42.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.809+0000 7f54bdffb700 1 -- 172.21.15.102:0/392696227 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f54b8019030 con 0x7f54c8153290 2023-03-03T07:08:42.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.811+0000 7f54bdffb700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54ac068320 0x7f54ac06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:42.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.812+0000 7f54bdffb700 1 -- 172.21.15.102:0/392696227 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f54b8085b20 con 0x7f54c8153290 2023-03-03T07:08:42.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.812+0000 7f54cd19b700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54ac068320 0x7f54ac06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:42.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.813+0000 7f54cd19b700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54ac068320 0x7f54ac06a7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f54c4014040 tx=0x7f54c401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:42.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:42.816+0000 7f54bdffb700 1 -- 172.21.15.102:0/392696227 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f54b8052230 con 0x7f54c8153290 2023-03-03T07:08:43.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.052+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54c8073dd0 con 0x7f54c8153290 2023-03-03T07:08:43.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.052+0000 7f54bdffb700 1 -- 172.21.15.102:0/392696227 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f54b8017090 con 0x7f54c8153290 2023-03-03T07:08:43.055 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:08:43.056 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:08:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54ac068320 msgr2=0x7f54ac06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54ac068320 0x7f54ac06a7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f54c4014040 tx=0x7f54c401c040 comp rx=0 tx=0).stop 2023-03-03T07:08:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 msgr2=0x7f54c81e5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c81e5b20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f54b800e900 tx=0x7f54b800ec10 comp rx=0 tx=0).stop 2023-03-03T07:08:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 shutdown_connections 2023-03-03T07:08:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54ac068320 0x7f54ac06a7d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c80cd8e0 0x7f54c81f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c81593c0 0x7f54c814d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 --2- 172.21.15.102:0/392696227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c8153290 0x7f54c81e5b20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 >> 172.21.15.102:0/392696227 conn(0x7f54c8065fb0 msgr2=0x7f54c815b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 shutdown_connections 2023-03-03T07:08:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:43.055+0000 7f54cf3ff700 1 -- 172.21.15.102:0/392696227 wait complete. 2023-03-03T07:08:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:08:43.084 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2023-03-03T07:08:49.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:08:49.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.336+0000 7f0e0959d700 1 Processor -- start 2023-03-03T07:08:49.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.337+0000 7f0e0959d700 1 -- start start 2023-03-03T07:08:49.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e0959d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e040c9610 0x7f0e040ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:49.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e0959d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e040c9ca0 con 0x7f0e040ca980 2023-03-03T07:08:49.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e0959d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e040b46f0 con 0x7f0e041564b0 2023-03-03T07:08:49.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e0959d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040cadc0 con 0x7f0e040c9610 2023-03-03T07:08:49.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e037fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e040c9610 0x7f0e040ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e08d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0e040ca980 0x7f0e041636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38592/0 (socket says 172.21.15.102:38592) 2023-03-03T07:08:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e08d9c700 1 -- 172.21.15.102:0/82864359 learned_addr learned my addr 172.21.15.102:0/82864359 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:08:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.338+0000 7f0e03fff700 1 --1- 172.21.15.102:0/82864359 >> v1:172.21.15.172:6789/0 conn(0x7f0e041564b0 0x7f0e040cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56450/0 (socket says 172.21.15.102:56450) 2023-03-03T07:08:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.339+0000 7f0e037fe700 1 -- 172.21.15.102:0/82864359 >> v1:172.21.15.172:6789/0 conn(0x7f0e041564b0 legacy=0x7f0e040cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.339+0000 7f0e037fe700 1 -- 172.21.15.102:0/82864359 >> v1:172.21.15.102:6789/0 conn(0x7f0e040ca980 legacy=0x7f0e041636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.339+0000 7f0e037fe700 1 -- 172.21.15.102:0/82864359 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e040b4920 con 0x7f0e040c9610 2023-03-03T07:08:49.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.339+0000 7f0e02ffd700 1 -- 172.21.15.102:0/82864359 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356488486 0 0) 0x7f0e040b46f0 con 0x7f0e041564b0 2023-03-03T07:08:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.339+0000 7f0e037fe700 1 --2- 172.21.15.102:0/82864359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e040c9610 0x7f0e040ca440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009870 tx=0x7f0df8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7184ad3bb529439 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.339+0000 7f0e02ffd700 1 -- 172.21.15.102:0/82864359 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8004230 con 0x7f0e040c9610 2023-03-03T07:08:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e02ffd700 1 -- 172.21.15.102:0/82864359 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df8004390 con 0x7f0e040c9610 2023-03-03T07:08:49.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e02ffd700 1 -- 172.21.15.102:0/82864359 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df80036a0 con 0x7f0e040c9610 2023-03-03T07:08:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e0959d700 1 -- 172.21.15.102:0/82864359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e040c9610 msgr2=0x7f0e040ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e0959d700 1 --2- 172.21.15.102:0/82864359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e040c9610 0x7f0e040ca440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009870 tx=0x7f0df8009b80 comp rx=0 tx=0).stop 2023-03-03T07:08:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e0959d700 1 -- 172.21.15.102:0/82864359 shutdown_connections 2023-03-03T07:08:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e0959d700 1 --2- 172.21.15.102:0/82864359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e040c9610 0x7f0e040ca440 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e0959d700 1 -- 172.21.15.102:0/82864359 >> 172.21.15.102:0/82864359 conn(0x7f0e04065fb0 msgr2=0x7f0e040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:49.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e0959d700 1 -- 172.21.15.102:0/82864359 shutdown_connections 2023-03-03T07:08:49.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.340+0000 7f0e0959d700 1 -- 172.21.15.102:0/82864359 wait complete. 2023-03-03T07:08:49.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.342+0000 7f0e0959d700 1 Processor -- start 2023-03-03T07:08:49.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.342+0000 7f0e0959d700 1 -- start start 2023-03-03T07:08:49.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.343+0000 7f0e0959d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e040c9610 0x7f0e040c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:49.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.343+0000 7f0e0959d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e040ca980 0x7f0e040cee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:49.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e0959d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e041564b0 0x7f0e041ee800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:49.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e0959d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04159eb0 con 0x7f0e040ca980 2023-03-03T07:08:49.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e0959d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040c8050 con 0x7f0e040c9610 2023-03-03T07:08:49.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e0959d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040c77b0 con 0x7f0e041564b0 2023-03-03T07:08:49.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e037fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e040ca980 0x7f0e040cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:49.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e040c9610 0x7f0e040c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:49.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e08d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e041564b0 0x7f0e041ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:49.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e040c9610 0x7f0e040c6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57622/0 (socket says 172.21.15.102:57622) 2023-03-03T07:08:49.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 -- 172.21.15.102:0/1938433182 learned_addr learned my addr 172.21.15.102:0/1938433182 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:08:49.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 -- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e041564b0 msgr2=0x7f0e041ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:49.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e041564b0 0x7f0e041ee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:49.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 -- 172.21.15.102:0/1938433182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e040ca980 msgr2=0x7f0e040cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:49.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e040ca980 0x7f0e040cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e03fff700 1 -- 172.21.15.102:0/1938433182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df80094a0 con 0x7f0e040c9610 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e08d9c700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e041564b0 0x7f0e041ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.344+0000 7f0e037fe700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e040ca980 0x7f0e040cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.345+0000 7f0e03fff700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e040c9610 0x7f0e040c6940 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0df400c500 tx=0x7f0df400c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.345+0000 7f0e00ff9700 1 -- 172.21.15.102:0/1938433182 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4004c80 con 0x7f0e040c9610 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.345+0000 7f0e00ff9700 1 -- 172.21.15.102:0/1938433182 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df4004de0 con 0x7f0e040c9610 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.345+0000 7f0e00ff9700 1 -- 172.21.15.102:0/1938433182 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4007310 con 0x7f0e040c9610 2023-03-03T07:08:49.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.345+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e040b1f30 con 0x7f0e040c9610 2023-03-03T07:08:49.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.345+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e040b4c00 con 0x7f0e040c9610 2023-03-03T07:08:49.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.346+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e040b2210 con 0x7f0e040c9610 2023-03-03T07:08:49.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.348+0000 7f0e00ff9700 1 -- 172.21.15.102:0/1938433182 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0df401a020 con 0x7f0e040c9610 2023-03-03T07:08:49.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.349+0000 7f0e00ff9700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0df0068320 0x7f0df006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:08:49.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.349+0000 7f0e00ff9700 1 -- 172.21.15.102:0/1938433182 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0df4085810 con 0x7f0e040c9610 2023-03-03T07:08:49.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.349+0000 7f0e037fe700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0df0068320 0x7f0df006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:08:49.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.350+0000 7f0e037fe700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0df0068320 0x7f0df006a7d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0df80097c0 tx=0x7f0df8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:08:49.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.353+0000 7f0e00ff9700 1 -- 172.21.15.102:0/1938433182 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0df4051f20 con 0x7f0e040c9610 2023-03-03T07:08:49.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.590+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e040677f0 con 0x7f0e040c9610 2023-03-03T07:08:49.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.591+0000 7f0e00ff9700 1 -- 172.21.15.102:0/1938433182 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0df4055540 con 0x7f0e040c9610 2023-03-03T07:08:49.592 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:08:49.592 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:08:49.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0df0068320 msgr2=0x7f0df006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:49.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0df0068320 0x7f0df006a7d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0df80097c0 tx=0x7f0df8008030 comp rx=0 tx=0).stop 2023-03-03T07:08:49.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e040c9610 msgr2=0x7f0e040c6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:08:49.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e040c9610 0x7f0e040c6940 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0df400c500 tx=0x7f0df400c810 comp rx=0 tx=0).stop 2023-03-03T07:08:49.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 shutdown_connections 2023-03-03T07:08:49.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0df0068320 0x7f0df006a7d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:49.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e041564b0 0x7f0e041ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:49.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e040c9610 0x7f0e040c6940 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:49.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 --2- 172.21.15.102:0/1938433182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e040ca980 0x7f0e040cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:08:49.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.593+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 >> 172.21.15.102:0/1938433182 conn(0x7f0e04065fb0 msgr2=0x7f0e04155570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:08:49.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.594+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 shutdown_connections 2023-03-03T07:08:49.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:08:49.594+0000 7f0e0959d700 1 -- 172.21.15.102:0/1938433182 wait complete. 2023-03-03T07:08:49.597 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:08:49.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2023-03-03T07:08:57.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:57.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:28.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:28.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:53.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:09:53.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.879+0000 7f7ad679d700 1 Processor -- start 2023-03-03T07:09:53.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.879+0000 7f7ad679d700 1 -- start start 2023-03-03T07:09:53.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad679d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ad00c22d0 0x7f7ad01569b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:09:53.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad679d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00cb2d0 0x7f7ad00c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:09:53.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad679d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad00c6ad0 con 0x7f7ad00c49c0 2023-03-03T07:09:53.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad679d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad00c7020 con 0x7f7ad00c22d0 2023-03-03T07:09:53.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad679d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad006e660 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad579b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ad00c22d0 0x7f7ad01569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:09:53.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad579b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ad00c22d0 0x7f7ad01569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53628/0 (socket says 172.21.15.102:53628) 2023-03-03T07:09:53.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad579b700 1 -- 172.21.15.102:0/4046649090 learned_addr learned my addr 172.21.15.102:0/4046649090 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:09:53.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad5f9c700 1 --1- 172.21.15.102:0/4046649090 >> v1:172.21.15.102:6789/0 conn(0x7f7ad00c49c0 0x7f7ad01507c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52402/0 (socket says 172.21.15.102:52402) 2023-03-03T07:09:53.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.881+0000 7f7ad4f9a700 1 --2- 172.21.15.102:0/4046649090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00cb2d0 0x7f7ad00c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:09:53.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ad579b700 1 -- 172.21.15.102:0/4046649090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00cb2d0 msgr2=0x7f7ad00c26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:09:53.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ad579b700 1 --2- 172.21.15.102:0/4046649090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00cb2d0 0x7f7ad00c26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:53.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ad579b700 1 -- 172.21.15.102:0/4046649090 >> v1:172.21.15.102:6789/0 conn(0x7f7ad00c49c0 legacy=0x7f7ad01507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:09:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ad579b700 1 -- 172.21.15.102:0/4046649090 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad00c6d40 con 0x7f7ad00c22d0 2023-03-03T07:09:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ad579b700 1 --2- 172.21.15.102:0/4046649090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00c22d0 0x7f7ad01569b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0098f0 tx=0x7f7acc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61854983e52736c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:09:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ac7fff700 1 -- 172.21.15.102:0/4046649090 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc011830 con 0x7f7ad00c22d0 2023-03-03T07:09:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ac7fff700 1 -- 172.21.15.102:0/4046649090 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc011990 con 0x7f7ad00c22d0 2023-03-03T07:09:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.882+0000 7f7ac7fff700 1 -- 172.21.15.102:0/4046649090 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc011d20 con 0x7f7ad00c22d0 2023-03-03T07:09:53.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 -- 172.21.15.102:0/4046649090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00c22d0 msgr2=0x7f7ad01569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:09:53.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 --2- 172.21.15.102:0/4046649090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00c22d0 0x7f7ad01569b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0098f0 tx=0x7f7acc009c00 comp rx=0 tx=0).stop 2023-03-03T07:09:53.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 -- 172.21.15.102:0/4046649090 shutdown_connections 2023-03-03T07:09:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 --2- 172.21.15.102:0/4046649090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00c22d0 0x7f7ad01569b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 --2- 172.21.15.102:0/4046649090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00cb2d0 0x7f7ad00c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 -- 172.21.15.102:0/4046649090 >> 172.21.15.102:0/4046649090 conn(0x7f7ad00ca7c0 msgr2=0x7f7ad00cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:09:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 -- 172.21.15.102:0/4046649090 shutdown_connections 2023-03-03T07:09:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.883+0000 7f7ad679d700 1 -- 172.21.15.102:0/4046649090 wait complete. 2023-03-03T07:09:53.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.885+0000 7f7ad679d700 1 Processor -- start 2023-03-03T07:09:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.886+0000 7f7ad679d700 1 -- start start 2023-03-03T07:09:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.886+0000 7f7ad679d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00c22d0 0x7f7ad0071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:09:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad679d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad00c49c0 0x7f7ad00bb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad679d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00cb2d0 0x7f7ad00bb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad679d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01de780 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad679d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01dcc10 con 0x7f7ad00c49c0 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad679d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01557f0 con 0x7f7ad00c22d0 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00cb2d0 0x7f7ad00bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad579b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00c22d0 0x7f7ad0071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad4f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad00c49c0 0x7f7ad00bb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:09:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00cb2d0 0x7f7ad00bb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53634/0 (socket says 172.21.15.102:53634) 2023-03-03T07:09:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 -- 172.21.15.102:0/3707411195 learned_addr learned my addr 172.21.15.102:0/3707411195 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:09:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 -- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00c22d0 msgr2=0x7f7ad0071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:09:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00c22d0 0x7f7ad0071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 -- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad00c49c0 msgr2=0x7f7ad00bb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:09:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad00c49c0 0x7f7ad00bb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.887+0000 7f7ad5f9c700 1 -- 172.21.15.102:0/3707411195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc009550 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.888+0000 7f7ad5f9c700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00cb2d0 0x7f7ad00bb870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800cb10 tx=0x7f7ac800ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:09:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.888+0000 7f7ac5ffb700 1 -- 172.21.15.102:0/3707411195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac800a9e0 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.888+0000 7f7ac5ffb700 1 -- 172.21.15.102:0/3707411195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac800ab40 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.888+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad0156ef0 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.888+0000 7f7ac5ffb700 1 -- 172.21.15.102:0/3707411195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8012400 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.888+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad00c44a0 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.889+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad0072840 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.891+0000 7f7ac5ffb700 1 -- 172.21.15.102:0/3707411195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ac8019030 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.892+0000 7f7ac5ffb700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7ab40682f0 0x7f7ab406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:09:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.892+0000 7f7ac5ffb700 1 -- 172.21.15.102:0/3707411195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7ac8085920 con 0x7f7ad00cb2d0 2023-03-03T07:09:53.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.892+0000 7f7ad579b700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7ab40682f0 0x7f7ab406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:09:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.893+0000 7f7ad579b700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7ab40682f0 0x7f7ab406a7a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009c40 tx=0x7f7acc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:09:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:53.896+0000 7f7ac5ffb700 1 -- 172.21.15.102:0/3707411195 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ac8052030 con 0x7f7ad00cb2d0 2023-03-03T07:09:54.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.132+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ad00748c0 con 0x7f7ad00cb2d0 2023-03-03T07:09:54.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.132+0000 7f7ac5ffb700 1 -- 172.21.15.102:0/3707411195 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7ac8001c20 con 0x7f7ad00cb2d0 2023-03-03T07:09:54.134 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:09:54.134 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:09:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7ab40682f0 msgr2=0x7f7ab406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:09:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7ab40682f0 0x7f7ab406a7a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009c40 tx=0x7f7acc008030 comp rx=0 tx=0).stop 2023-03-03T07:09:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00cb2d0 msgr2=0x7f7ad00bb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:09:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00cb2d0 0x7f7ad00bb870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800cb10 tx=0x7f7ac800ce20 comp rx=0 tx=0).stop 2023-03-03T07:09:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 shutdown_connections 2023-03-03T07:09:54.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7ab40682f0 0x7f7ab406a7a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ad00c22d0 0x7f7ad0071760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ad00cb2d0 0x7f7ad00bb870 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 --2- 172.21.15.102:0/3707411195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad00c49c0 0x7f7ad00bb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:09:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.135+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 >> 172.21.15.102:0/3707411195 conn(0x7f7ad00ca7c0 msgr2=0x7f7ad014fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:09:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.136+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 shutdown_connections 2023-03-03T07:09:54.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:09:54.136+0000 7f7ad679d700 1 -- 172.21.15.102:0/3707411195 wait complete. 2023-03-03T07:09:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:09:54.164 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 25 secs before thrashing 2023-03-03T07:09:58.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:58.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T07:10:00.000+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:10:19.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:10:19.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.415+0000 7fbd87fff700 1 Processor -- start 2023-03-03T07:10:19.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.415+0000 7fbd87fff700 1 -- start start 2023-03-03T07:10:19.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.417+0000 7fbd87fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbd80153290 0x7fbd80158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:19.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.417+0000 7fbd87fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd80153660 con 0x7fbd800cd8e0 2023-03-03T07:10:19.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.417+0000 7fbd87fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8006c360 con 0x7fbd801593c0 2023-03-03T07:10:19.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.417+0000 7fbd87fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd800cec40 con 0x7fbd80153290 2023-03-03T07:10:19.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.417+0000 7fbd8659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbd800cd8e0 0x7fbd800ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41944/0 (socket says 172.21.15.102:41944) 2023-03-03T07:10:19.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.417+0000 7fbd8659c700 1 -- 172.21.15.102:0/2645348701 learned_addr learned my addr 172.21.15.102:0/2645348701 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:10:19.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.417+0000 7fbd85d9b700 1 --2- 172.21.15.102:0/2645348701 >> v2:172.21.15.172:3300/0 conn(0x7fbd80153290 0x7fbd80158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd84d99700 1 -- 172.21.15.102:0/2645348701 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155984543 0 0) 0x7fbd8006c360 con 0x7fbd801593c0 2023-03-03T07:10:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd84d99700 1 -- 172.21.15.102:0/2645348701 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd6c0035b0 con 0x7fbd801593c0 2023-03-03T07:10:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd84d99700 1 -- 172.21.15.102:0/2645348701 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616727592 0 0) 0x7fbd80153660 con 0x7fbd800cd8e0 2023-03-03T07:10:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd85d9b700 1 -- 172.21.15.102:0/2645348701 >> v1:172.21.15.172:6789/0 conn(0x7fbd801593c0 legacy=0x7fbd800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd85d9b700 1 -- 172.21.15.102:0/2645348701 >> v1:172.21.15.102:6789/0 conn(0x7fbd800cd8e0 legacy=0x7fbd800ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:19.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd85d9b700 1 -- 172.21.15.102:0/2645348701 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8015cea0 con 0x7fbd80153290 2023-03-03T07:10:19.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd85d9b700 1 --2- 172.21.15.102:0/2645348701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd80153290 0x7fbd80158e80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009840 tx=0x7fbd7c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d61fc5f8278217c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:10:19.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd84d99700 1 -- 172.21.15.102:0/2645348701 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c011830 con 0x7fbd80153290 2023-03-03T07:10:19.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd84d99700 1 -- 172.21.15.102:0/2645348701 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7c011990 con 0x7fbd80153290 2023-03-03T07:10:19.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.418+0000 7fbd84d99700 1 -- 172.21.15.102:0/2645348701 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c011d20 con 0x7fbd80153290 2023-03-03T07:10:19.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.419+0000 7fbd87fff700 1 -- 172.21.15.102:0/2645348701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd80153290 msgr2=0x7fbd80158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:19.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.419+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2645348701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd80153290 0x7fbd80158e80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009840 tx=0x7fbd7c009b50 comp rx=0 tx=0).stop 2023-03-03T07:10:19.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.419+0000 7fbd87fff700 1 -- 172.21.15.102:0/2645348701 shutdown_connections 2023-03-03T07:10:19.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.419+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2645348701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd80153290 0x7fbd80158e80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:19.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.419+0000 7fbd87fff700 1 -- 172.21.15.102:0/2645348701 >> 172.21.15.102:0/2645348701 conn(0x7fbd80065fb0 msgr2=0x7fbd800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:10:19.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.419+0000 7fbd87fff700 1 -- 172.21.15.102:0/2645348701 shutdown_connections 2023-03-03T07:10:19.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.419+0000 7fbd87fff700 1 -- 172.21.15.102:0/2645348701 wait complete. 2023-03-03T07:10:19.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.421+0000 7fbd87fff700 1 Processor -- start 2023-03-03T07:10:19.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.421+0000 7fbd87fff700 1 -- start start 2023-03-03T07:10:19.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.422+0000 7fbd87fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd800cd8e0 0x7fbd800b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:19.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.422+0000 7fbd87fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80153290 0x7fbd80151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:19.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.422+0000 7fbd87fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd801593c0 0x7fbd8014f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:19.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.422+0000 7fbd87fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800b35e0 con 0x7fbd80153290 2023-03-03T07:10:19.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.422+0000 7fbd87fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800b3f40 con 0x7fbd800cd8e0 2023-03-03T07:10:19.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.422+0000 7fbd87fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd80150790 con 0x7fbd801593c0 2023-03-03T07:10:19.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd85d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd800cd8e0 0x7fbd800b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:19.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8559a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80153290 0x7fbd80151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:19.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd85d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd800cd8e0 0x7fbd800b3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54244/0 (socket says 172.21.15.102:54244) 2023-03-03T07:10:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd85d9b700 1 -- 172.21.15.102:0/2728381527 learned_addr learned my addr 172.21.15.102:0/2728381527 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:10:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8659c700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd801593c0 0x7fbd8014f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8559a700 1 -- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd801593c0 msgr2=0x7fbd8014f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8559a700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd801593c0 0x7fbd8014f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8559a700 1 -- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd800cd8e0 msgr2=0x7fbd800b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8559a700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd800cd8e0 0x7fbd800b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8559a700 1 -- 172.21.15.102:0/2728381527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c0094a0 con 0x7fbd80153290 2023-03-03T07:10:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd85d9b700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd800cd8e0 0x7fbd800b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:10:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.423+0000 7fbd8559a700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80153290 0x7fbd80151090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000c420 tx=0x7fbd7000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:10:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.424+0000 7fbd767fc700 1 -- 172.21.15.102:0/2728381527 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd700052d0 con 0x7fbd80153290 2023-03-03T07:10:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.424+0000 7fbd767fc700 1 -- 172.21.15.102:0/2728381527 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd70005430 con 0x7fbd80153290 2023-03-03T07:10:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.424+0000 7fbd767fc700 1 -- 172.21.15.102:0/2728381527 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd700038c0 con 0x7fbd80153290 2023-03-03T07:10:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.424+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8015d160 con 0x7fbd80153290 2023-03-03T07:10:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.424+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8006c7f0 con 0x7fbd80153290 2023-03-03T07:10:19.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.425+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd800ccde0 con 0x7fbd80153290 2023-03-03T07:10:19.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.426+0000 7fbd767fc700 1 -- 172.21.15.102:0/2728381527 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd70019030 con 0x7fbd80153290 2023-03-03T07:10:19.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.427+0000 7fbd767fc700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbd640682f0 0x7fbd6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.428+0000 7fbd767fc700 1 -- 172.21.15.102:0/2728381527 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbd70084f40 con 0x7fbd80153290 2023-03-03T07:10:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.428+0000 7fbd85d9b700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbd640682f0 0x7fbd6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.429+0000 7fbd85d9b700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbd640682f0 0x7fbd6406a7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000c00 tx=0x7fbd7c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:10:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.432+0000 7fbd767fc700 1 -- 172.21.15.102:0/2728381527 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd70051650 con 0x7fbd80153290 2023-03-03T07:10:19.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.665+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd800b3720 con 0x7fbd80153290 2023-03-03T07:10:19.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.666+0000 7fbd767fc700 1 -- 172.21.15.102:0/2728381527 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbd800b3720 con 0x7fbd80153290 2023-03-03T07:10:19.668 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:10:19.668 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:10:19.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbd640682f0 msgr2=0x7fbd6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:19.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbd640682f0 0x7fbd6406a7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000c00 tx=0x7fbd7c01c040 comp rx=0 tx=0).stop 2023-03-03T07:10:19.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80153290 msgr2=0x7fbd80151090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:19.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80153290 0x7fbd80151090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000c420 tx=0x7fbd7000c730 comp rx=0 tx=0).stop 2023-03-03T07:10:19.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 shutdown_connections 2023-03-03T07:10:19.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbd640682f0 0x7fbd6406a7a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:19.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbd801593c0 0x7fbd8014f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:19.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd800cd8e0 0x7fbd800b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:19.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 --2- 172.21.15.102:0/2728381527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80153290 0x7fbd80151090 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:19.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 >> 172.21.15.102:0/2728381527 conn(0x7fbd80065fb0 msgr2=0x7fbd8015ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:10:19.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 shutdown_connections 2023-03-03T07:10:19.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:19.668+0000 7fbd87fff700 1 -- 172.21.15.102:0/2728381527 wait complete. 2023-03-03T07:10:19.672 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:10:19.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-03-03T07:10:26.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:10:26.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.963+0000 7f1633fff700 1 Processor -- start 2023-03-03T07:10:26.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.964+0000 7f1633fff700 1 -- start start 2023-03-03T07:10:26.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.965+0000 7f1633fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f16340ced00 0x7f16340c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:26.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.965+0000 7f1633fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340c41b0 0x7f16340cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:26.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.965+0000 7f1633fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16340c3d20 con 0x7f16340b3970 2023-03-03T07:10:26.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.965+0000 7f1633fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340c4930 con 0x7f16340ced00 2023-03-03T07:10:26.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.965+0000 7f1633fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16340cf610 con 0x7f16340c41b0 2023-03-03T07:10:26.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.965+0000 7f16327fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f16340b3970 0x7f1634158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46740/0 (socket says 172.21.15.102:46740) 2023-03-03T07:10:26.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f16327fc700 1 -- 172.21.15.102:0/2928893992 learned_addr learned my addr 172.21.15.102:0/2928893992 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:10:26.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f16337fe700 1 --2- 172.21.15.102:0/2928893992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340c41b0 0x7f16340cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:26.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f1632ffd700 1 --2- 172.21.15.102:0/2928893992 >> v2:172.21.15.172:3300/0 conn(0x7f16340ced00 0x7f16340c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:26.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f1631ffb700 1 -- 172.21.15.102:0/2928893992 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176132478 0 0) 0x7f16340c3d20 con 0x7f16340b3970 2023-03-03T07:10:26.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f1631ffb700 1 -- 172.21.15.102:0/2928893992 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f161c0035b0 con 0x7f16340b3970 2023-03-03T07:10:26.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f16337fe700 1 -- 172.21.15.102:0/2928893992 >> v2:172.21.15.172:3300/0 conn(0x7f16340ced00 msgr2=0x7f16340c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:10:26.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f16337fe700 1 --2- 172.21.15.102:0/2928893992 >> v2:172.21.15.172:3300/0 conn(0x7f16340ced00 0x7f16340c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:26.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f16337fe700 1 -- 172.21.15.102:0/2928893992 >> v1:172.21.15.102:6789/0 conn(0x7f16340b3970 legacy=0x7f1634158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:26.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f16337fe700 1 -- 172.21.15.102:0/2928893992 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163415cba0 con 0x7f16340c41b0 2023-03-03T07:10:26.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.966+0000 7f16337fe700 1 --2- 172.21.15.102:0/2928893992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340c41b0 0x7f16340cf0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f162c009d90 tx=0x7f162c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=515d03579d9a17d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:10:26.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1631ffb700 1 -- 172.21.15.102:0/2928893992 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00e070 con 0x7f16340c41b0 2023-03-03T07:10:26.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1631ffb700 1 -- 172.21.15.102:0/2928893992 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c00bb20 con 0x7f16340c41b0 2023-03-03T07:10:26.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1631ffb700 1 -- 172.21.15.102:0/2928893992 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c012430 con 0x7f16340c41b0 2023-03-03T07:10:26.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1633fff700 1 -- 172.21.15.102:0/2928893992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340c41b0 msgr2=0x7f16340cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:26.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1633fff700 1 --2- 172.21.15.102:0/2928893992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340c41b0 0x7f16340cf0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f162c009d90 tx=0x7f162c00b330 comp rx=0 tx=0).stop 2023-03-03T07:10:26.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1633fff700 1 -- 172.21.15.102:0/2928893992 shutdown_connections 2023-03-03T07:10:26.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1633fff700 1 --2- 172.21.15.102:0/2928893992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340c41b0 0x7f16340cf0d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:26.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1633fff700 1 --2- 172.21.15.102:0/2928893992 >> v2:172.21.15.172:3300/0 conn(0x7f16340ced00 0x7f16340c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:26.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.967+0000 7f1633fff700 1 -- 172.21.15.102:0/2928893992 >> 172.21.15.102:0/2928893992 conn(0x7f163408b1c0 msgr2=0x7f1634071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:10:26.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.968+0000 7f1633fff700 1 -- 172.21.15.102:0/2928893992 shutdown_connections 2023-03-03T07:10:26.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.968+0000 7f1633fff700 1 -- 172.21.15.102:0/2928893992 wait complete. 2023-03-03T07:10:26.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.970+0000 7f1633fff700 1 Processor -- start 2023-03-03T07:10:26.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.970+0000 7f1633fff700 1 -- start start 2023-03-03T07:10:26.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f1633fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16340b3970 0x7f163414f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:26.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f1633fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16340c41b0 0x7f1634148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:26.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f1633fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340ced00 0x7f1634151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:26.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f1633fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16340cadf0 con 0x7f16340b3970 2023-03-03T07:10:26.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f1633fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16341eef40 con 0x7f16340c41b0 2023-03-03T07:10:26.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f1633fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16341ee630 con 0x7f16340ced00 2023-03-03T07:10:26.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f1632ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16340b3970 0x7f163414f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:26.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f16337fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340ced00 0x7f1634151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:26.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f16327fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16340c41b0 0x7f1634148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:26.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f16337fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340ced00 0x7f1634151990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:33710/0 (socket says 172.21.15.102:33710) 2023-03-03T07:10:26.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.971+0000 7f16337fe700 1 -- 172.21.15.102:0/356862904 learned_addr learned my addr 172.21.15.102:0/356862904 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:10:26.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f16337fe700 1 -- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16340c41b0 msgr2=0x7f1634148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f16337fe700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16340c41b0 0x7f1634148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f16337fe700 1 -- 172.21.15.102:0/356862904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16340b3970 msgr2=0x7f163414f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f16337fe700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16340b3970 0x7f163414f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f16337fe700 1 -- 172.21.15.102:0/356862904 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c009a70 con 0x7f16340ced00 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f16337fe700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340ced00 0x7f1634151990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f162c00b370 tx=0x7f162c004b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f161b7fe700 1 -- 172.21.15.102:0/356862904 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c012430 con 0x7f16340ced00 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f161b7fe700 1 -- 172.21.15.102:0/356862904 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c020070 con 0x7f16340ced00 2023-03-03T07:10:26.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f161b7fe700 1 -- 172.21.15.102:0/356862904 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c004280 con 0x7f16340ced00 2023-03-03T07:10:26.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.972+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163406bf20 con 0x7f16340ced00 2023-03-03T07:10:26.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.973+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163415ce80 con 0x7f16340ced00 2023-03-03T07:10:26.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.974+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163406c200 con 0x7f16340ced00 2023-03-03T07:10:26.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.975+0000 7f161b7fe700 1 -- 172.21.15.102:0/356862904 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f162c02e070 con 0x7f16340ced00 2023-03-03T07:10:26.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.976+0000 7f161b7fe700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16200682f0 0x7f162006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:10:26.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.976+0000 7f161b7fe700 1 -- 172.21.15.102:0/356862904 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f162c022070 con 0x7f16340ced00 2023-03-03T07:10:26.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.976+0000 7f1632ffd700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16200682f0 0x7f162006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:10:26.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.977+0000 7f1632ffd700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16200682f0 0x7f162006a7a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1624009be0 tx=0x7f1624009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:10:26.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:26.981+0000 7f161b7fe700 1 -- 172.21.15.102:0/356862904 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f162c0540f0 con 0x7f16340ced00 2023-03-03T07:10:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.214+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f163408b590 con 0x7f16340ced00 2023-03-03T07:10:27.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.215+0000 7f161b7fe700 1 -- 172.21.15.102:0/356862904 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f162c02c090 con 0x7f16340ced00 2023-03-03T07:10:27.216 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:10:27.217 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:10:27.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16200682f0 msgr2=0x7f162006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:27.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16200682f0 0x7f162006a7a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1624009be0 tx=0x7f1624009210 comp rx=0 tx=0).stop 2023-03-03T07:10:27.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340ced00 msgr2=0x7f1634151990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:10:27.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340ced00 0x7f1634151990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f162c00b370 tx=0x7f162c004b20 comp rx=0 tx=0).stop 2023-03-03T07:10:27.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 shutdown_connections 2023-03-03T07:10:27.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16200682f0 0x7f162006a7a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:27.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16340ced00 0x7f1634151990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:27.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16340b3970 0x7f163414f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:27.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 --2- 172.21.15.102:0/356862904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16340c41b0 0x7f1634148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:10:27.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 >> 172.21.15.102:0/356862904 conn(0x7f163408b1c0 msgr2=0x7f1634071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:10:27.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 shutdown_connections 2023-03-03T07:10:27.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:10:27.218+0000 7f1633fff700 1 -- 172.21.15.102:0/356862904 wait complete. 2023-03-03T07:10:27.222 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:10:27.246 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T07:10:28.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:28.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:58.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:58.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:28.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:28.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:46.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:11:46.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.511+0000 7f791999d700 1 Processor -- start 2023-03-03T07:11:46.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.511+0000 7f791999d700 1 -- start start 2023-03-03T07:11:46.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.513+0000 7f791999d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79140c5a90 0x7f79140c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:11:46.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.513+0000 7f791999d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79140c3d20 con 0x7f79140ced00 2023-03-03T07:11:46.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.513+0000 7f791999d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791415cba0 con 0x7f79140b3970 2023-03-03T07:11:46.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.513+0000 7f791999d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79140c4930 con 0x7f79140c5a90 2023-03-03T07:11:46.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.513+0000 7f791899b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f79140ced00 0x7f7914158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49438/0 (socket says 172.21.15.102:49438) 2023-03-03T07:11:46.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.513+0000 7f791899b700 1 -- 172.21.15.102:0/2064302965 learned_addr learned my addr 172.21.15.102:0/2064302965 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:11:46.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.513+0000 7f791919c700 1 --2- 172.21.15.102:0/2064302965 >> v2:172.21.15.102:3300/0 conn(0x7f79140c5a90 0x7f79140c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:11:46.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f79137fe700 1 -- 172.21.15.102:0/2064302965 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817888063 0 0) 0x7f79140c3d20 con 0x7f79140ced00 2023-03-03T07:11:46.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f79137fe700 1 -- 172.21.15.102:0/2064302965 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79040035b0 con 0x7f79140ced00 2023-03-03T07:11:46.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f79137fe700 1 -- 172.21.15.102:0/2064302965 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311282249 0 0) 0x7f791415cba0 con 0x7f79140b3970 2023-03-03T07:11:46.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f79137fe700 1 -- 172.21.15.102:0/2064302965 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79140c3d20 con 0x7f79140b3970 2023-03-03T07:11:46.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f791919c700 1 -- 172.21.15.102:0/2064302965 >> v1:172.21.15.172:6789/0 conn(0x7f79140b3970 legacy=0x7f7914161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:11:46.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f791919c700 1 -- 172.21.15.102:0/2064302965 >> v1:172.21.15.102:6789/0 conn(0x7f79140ced00 legacy=0x7f7914158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:11:46.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f791919c700 1 -- 172.21.15.102:0/2064302965 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791415ce80 con 0x7f79140c5a90 2023-03-03T07:11:46.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f791919c700 1 --2- 172.21.15.102:0/2064302965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140c5a90 0x7f79140c41b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f790c009d90 tx=0x7f790c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6996661c7be86d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:11:46.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.514+0000 7f79137fe700 1 -- 172.21.15.102:0/2064302965 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c00e070 con 0x7f79140c5a90 2023-03-03T07:11:46.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f79137fe700 1 -- 172.21.15.102:0/2064302965 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c00bb20 con 0x7f79140c5a90 2023-03-03T07:11:46.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f79137fe700 1 -- 172.21.15.102:0/2064302965 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c012430 con 0x7f79140c5a90 2023-03-03T07:11:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f791999d700 1 -- 172.21.15.102:0/2064302965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140c5a90 msgr2=0x7f79140c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:11:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f791999d700 1 --2- 172.21.15.102:0/2064302965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140c5a90 0x7f79140c41b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f790c009d90 tx=0x7f790c00b330 comp rx=0 tx=0).stop 2023-03-03T07:11:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f791999d700 1 -- 172.21.15.102:0/2064302965 shutdown_connections 2023-03-03T07:11:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f791999d700 1 --2- 172.21.15.102:0/2064302965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140c5a90 0x7f79140c41b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f791999d700 1 -- 172.21.15.102:0/2064302965 >> 172.21.15.102:0/2064302965 conn(0x7f791408b1c0 msgr2=0x7f7914071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:11:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f791999d700 1 -- 172.21.15.102:0/2064302965 shutdown_connections 2023-03-03T07:11:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.515+0000 7f791999d700 1 -- 172.21.15.102:0/2064302965 wait complete. 2023-03-03T07:11:46.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.518+0000 7f791999d700 1 Processor -- start 2023-03-03T07:11:46.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.518+0000 7f791999d700 1 -- start start 2023-03-03T07:11:46.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791999d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140b3970 0x7f7914148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:11:46.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791999d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79140c5a90 0x7f7914151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:11:46.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791999d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79140ced00 0x7f79141519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:11:46.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791999d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914159cc0 con 0x7f79140b3970 2023-03-03T07:11:46.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791999d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79141eef10 con 0x7f79140ced00 2023-03-03T07:11:46.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791999d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79141ef440 con 0x7f79140c5a90 2023-03-03T07:11:46.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f7913fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79140c5a90 0x7f7914151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:11:46.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791919c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79140ced00 0x7f79141519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:11:46.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791899b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140b3970 0x7f7914148850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:11:46.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791919c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79140ced00 0x7f79141519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55036/0 (socket says 172.21.15.102:55036) 2023-03-03T07:11:46.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.519+0000 7f791919c700 1 -- 172.21.15.102:0/1428782841 learned_addr learned my addr 172.21.15.102:0/1428782841 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:11:46.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791919c700 1 -- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79140c5a90 msgr2=0x7f7914151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:11:46.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791919c700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79140c5a90 0x7f7914151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791919c700 1 -- 172.21.15.102:0/1428782841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140b3970 msgr2=0x7f7914148850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:11:46.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791919c700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140b3970 0x7f7914148850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791919c700 1 -- 172.21.15.102:0/1428782841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c009a70 con 0x7f79140ced00 2023-03-03T07:11:46.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f7913fff700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79140c5a90 0x7f7914151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:11:46.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791899b700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140b3970 0x7f7914148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:11:46.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791919c700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79140ced00 0x7f79141519c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f790c014040 tx=0x7f790c004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:11:46.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f79117fa700 1 -- 172.21.15.102:0/1428782841 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c012430 con 0x7f79140ced00 2023-03-03T07:11:46.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f79117fa700 1 -- 172.21.15.102:0/1428782841 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c020070 con 0x7f79140ced00 2023-03-03T07:11:46.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f79117fa700 1 -- 172.21.15.102:0/1428782841 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c01ba50 con 0x7f79140ced00 2023-03-03T07:11:46.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.520+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791406c200 con 0x7f79140ced00 2023-03-03T07:11:46.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.521+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791406bf20 con 0x7f79140ced00 2023-03-03T07:11:46.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.522+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791415c890 con 0x7f79140ced00 2023-03-03T07:11:46.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.523+0000 7f79117fa700 1 -- 172.21.15.102:0/1428782841 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f790c02e070 con 0x7f79140ced00 2023-03-03T07:11:46.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.524+0000 7f79117fa700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f80682f0 0x7f78f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:11:46.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.524+0000 7f79117fa700 1 -- 172.21.15.102:0/1428782841 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f790c022070 con 0x7f79140ced00 2023-03-03T07:11:46.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.524+0000 7f791899b700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f80682f0 0x7f78f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:11:46.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.525+0000 7f791899b700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f80682f0 0x7f78f806a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7908007fd0 tx=0x7f7908007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:11:46.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.528+0000 7f79117fa700 1 -- 172.21.15.102:0/1428782841 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f790c054370 con 0x7f79140ced00 2023-03-03T07:11:46.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.764+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7914159e30 con 0x7f79140ced00 2023-03-03T07:11:46.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.765+0000 7f79117fa700 1 -- 172.21.15.102:0/1428782841 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7914159e30 con 0x7f79140ced00 2023-03-03T07:11:46.767 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:11:46.767 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:11:46.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f80682f0 msgr2=0x7f78f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:11:46.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f80682f0 0x7f78f806a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7908007fd0 tx=0x7f7908007de0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79140ced00 msgr2=0x7f79141519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:11:46.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79140ced00 0x7f79141519c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f790c014040 tx=0x7f790c004b20 comp rx=0 tx=0).stop 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 shutdown_connections 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79140ced00 0x7f79141519c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79140b3970 0x7f7914148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f80682f0 0x7f78f806a7a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 --2- 172.21.15.102:0/1428782841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79140c5a90 0x7f7914151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 >> 172.21.15.102:0/1428782841 conn(0x7f791408b1c0 msgr2=0x7f79140cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 shutdown_connections 2023-03-03T07:11:46.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:11:46.768+0000 7f791999d700 1 -- 172.21.15.102:0/1428782841 wait complete. 2023-03-03T07:11:46.772 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:11:46.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 82 secs before thrashing 2023-03-03T07:11:58.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:58.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:29.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:29.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:59.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:59.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:08.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:13:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.049+0000 7f689f5ff700 1 Processor -- start 2023-03-03T07:13:09.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.049+0000 7f689f5ff700 1 -- start start 2023-03-03T07:13:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.050+0000 7f689f5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6898153290 0x7f6898158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:09.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.050+0000 7f689f5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68981593c0 0x7f68980cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.050+0000 7f689f5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f68980cd8e0 0x7f68980ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.050+0000 7f689f5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6898153660 con 0x7f68980cd8e0 2023-03-03T07:13:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.050+0000 7f689f5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689806c360 con 0x7f6898153290 2023-03-03T07:13:09.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.050+0000 7f689f5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689815cea0 con 0x7f68981593c0 2023-03-03T07:13:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689d39b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6898153290 0x7f6898158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689db9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f68980cd8e0 0x7f68980ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:09.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68981593c0 0x7f68980cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68981593c0 0x7f68980cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:56288/0 (socket says 172.21.15.102:56288) 2023-03-03T07:13:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 -- 172.21.15.102:0/3067250015 learned_addr learned my addr 172.21.15.102:0/3067250015 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:13:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689d39b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6898153290 0x7f6898158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45282/0 (socket says 172.21.15.102:45282) 2023-03-03T07:13:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 -- 172.21.15.102:0/3067250015 >> v2:172.21.15.172:3300/0 conn(0x7f6898153290 msgr2=0x7f6898158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:13:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 --2- 172.21.15.102:0/3067250015 >> v2:172.21.15.172:3300/0 conn(0x7f6898153290 0x7f6898158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 -- 172.21.15.102:0/3067250015 >> v2:172.21.15.102:3300/0 conn(0x7f68980cd8e0 msgr2=0x7f68980ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:13:09.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 --2- 172.21.15.102:0/3067250015 >> v2:172.21.15.102:3300/0 conn(0x7f68980cd8e0 0x7f68980ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689cb9a700 1 -- 172.21.15.102:0/3067250015 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689806c7f0 con 0x7f68981593c0 2023-03-03T07:13:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.051+0000 7f689db9c700 1 --2- 172.21.15.102:0/3067250015 >> v2:172.21.15.102:3300/0 conn(0x7f68980cd8e0 0x7f68980ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:13:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.052+0000 7f689cb9a700 1 --2- 172.21.15.102:0/3067250015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68981593c0 0x7f68980cde20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f689808cc70 tx=0x7f6888009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0fa50391f7d42e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:13:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.052+0000 7f688ffff700 1 -- 172.21.15.102:0/3067250015 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888011830 con 0x7f68981593c0 2023-03-03T07:13:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.052+0000 7f688ffff700 1 -- 172.21.15.102:0/3067250015 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6888011990 con 0x7f68981593c0 2023-03-03T07:13:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.052+0000 7f688ffff700 1 -- 172.21.15.102:0/3067250015 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888011d20 con 0x7f68981593c0 2023-03-03T07:13:09.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.052+0000 7f689f5ff700 1 -- 172.21.15.102:0/3067250015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68981593c0 msgr2=0x7f68980cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.052+0000 7f689f5ff700 1 --2- 172.21.15.102:0/3067250015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68981593c0 0x7f68980cde20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f689808cc70 tx=0x7f6888009bd0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.053+0000 7f689f5ff700 1 -- 172.21.15.102:0/3067250015 shutdown_connections 2023-03-03T07:13:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.053+0000 7f689f5ff700 1 --2- 172.21.15.102:0/3067250015 >> v2:172.21.15.102:3300/0 conn(0x7f68980cd8e0 0x7f68980ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.053+0000 7f689f5ff700 1 --2- 172.21.15.102:0/3067250015 >> v2:172.21.15.172:3300/0 conn(0x7f6898153290 0x7f6898158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.053+0000 7f689f5ff700 1 --2- 172.21.15.102:0/3067250015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68981593c0 0x7f68980cde20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.053+0000 7f689f5ff700 1 -- 172.21.15.102:0/3067250015 >> 172.21.15.102:0/3067250015 conn(0x7f6898065fb0 msgr2=0x7f68980663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:13:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.053+0000 7f689f5ff700 1 -- 172.21.15.102:0/3067250015 shutdown_connections 2023-03-03T07:13:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.053+0000 7f689f5ff700 1 -- 172.21.15.102:0/3067250015 wait complete. 2023-03-03T07:13:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.055+0000 7f689f5ff700 1 Processor -- start 2023-03-03T07:13:09.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.055+0000 7f689f5ff700 1 -- start start 2023-03-03T07:13:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689f5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68980cd8e0 0x7f68980c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689f5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6898153290 0x7f68980c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689f5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68981593c0 0x7f689814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689f5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68980c8ff0 con 0x7f68981593c0 2023-03-03T07:13:09.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689f5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68980b2510 con 0x7f6898153290 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689f5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68980c6140 con 0x7f68980cd8e0 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689db9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68981593c0 0x7f689814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689cb9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6898153290 0x7f68980c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689db9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68981593c0 0x7f689814e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58560/0 (socket says 172.21.15.102:58560) 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689db9c700 1 -- 172.21.15.102:0/1323520443 learned_addr learned my addr 172.21.15.102:0/1323520443 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.056+0000 7f689d39b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68980cd8e0 0x7f68980c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f689db9c700 1 -- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68980cd8e0 msgr2=0x7f68980c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:09.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f689db9c700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68980cd8e0 0x7f68980c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f689db9c700 1 -- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6898153290 msgr2=0x7f68980c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f689db9c700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6898153290 0x7f68980c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f689db9c700 1 -- 172.21.15.102:0/1323520443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6890009a70 con 0x7f68981593c0 2023-03-03T07:13:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f689db9c700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68981593c0 0x7f689814e040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f689000efc0 tx=0x7f689000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:13:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f688dffb700 1 -- 172.21.15.102:0/1323520443 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689000ca30 con 0x7f68981593c0 2023-03-03T07:13:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f688dffb700 1 -- 172.21.15.102:0/1323520443 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689000f040 con 0x7f68981593c0 2023-03-03T07:13:09.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6888009550 con 0x7f68981593c0 2023-03-03T07:13:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.057+0000 7f688dffb700 1 -- 172.21.15.102:0/1323520443 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689000cdd0 con 0x7f68981593c0 2023-03-03T07:13:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.058+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68980ccde0 con 0x7f68981593c0 2023-03-03T07:13:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.058+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689815d160 con 0x7f68981593c0 2023-03-03T07:13:09.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.060+0000 7f688dffb700 1 -- 172.21.15.102:0/1323520443 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6890014480 con 0x7f68981593c0 2023-03-03T07:13:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.062+0000 7f688dffb700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f687c0682f0 0x7f687c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.062+0000 7f688dffb700 1 -- 172.21.15.102:0/1323520443 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f68900865c0 con 0x7f68981593c0 2023-03-03T07:13:09.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.062+0000 7f689d39b700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f687c0682f0 0x7f687c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:09.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.063+0000 7f689d39b700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f687c0682f0 0x7f687c06a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6894009ac0 tx=0x7f6894009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:13:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.066+0000 7f688dffb700 1 -- 172.21.15.102:0/1323520443 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6890052be0 con 0x7f68981593c0 2023-03-03T07:13:09.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.299+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68980cd0c0 con 0x7f68981593c0 2023-03-03T07:13:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.300+0000 7f688dffb700 1 -- 172.21.15.102:0/1323520443 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6890009cd0 con 0x7f68981593c0 2023-03-03T07:13:09.301 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:13:09.301 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:13:09.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f687c0682f0 msgr2=0x7f687c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:09.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f687c0682f0 0x7f687c06a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6894009ac0 tx=0x7f6894009210 comp rx=0 tx=0).stop 2023-03-03T07:13:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68981593c0 msgr2=0x7f689814e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68981593c0 0x7f689814e040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f689000efc0 tx=0x7f689000c3a0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 shutdown_connections 2023-03-03T07:13:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f687c0682f0 0x7f687c06a7a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68980cd8e0 0x7f68980c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68981593c0 0x7f689814e040 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 --2- 172.21.15.102:0/1323520443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6898153290 0x7f68980c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:09.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 >> 172.21.15.102:0/1323520443 conn(0x7f6898065fb0 msgr2=0x7f68980cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:13:09.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 shutdown_connections 2023-03-03T07:13:09.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:09.302+0000 7f689f5ff700 1 -- 172.21.15.102:0/1323520443 wait complete. 2023-03-03T07:13:09.305 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:13:09.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-03-03T07:13:21.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:13:21.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.579+0000 7fb88a99d700 1 Processor -- start 2023-03-03T07:13:21.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.580+0000 7fb88a99d700 1 -- start start 2023-03-03T07:13:21.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88a99d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb8840c9610 0x7fb8840ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:21.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88a99d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840ca980 0x7fb8841636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:21.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88a99d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8840c9ca0 con 0x7fb8841564b0 2023-03-03T07:13:21.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88a99d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8840b4920 con 0x7fb8840c9610 2023-03-03T07:13:21.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88a99d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8840b4c00 con 0x7fb8840ca980 2023-03-03T07:13:21.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88a19c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840ca980 0x7fb8841636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:21.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88999b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb8841564b0 0x7fb8840cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:55534/0 (socket says 172.21.15.102:55534) 2023-03-03T07:13:21.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88919a700 1 --2- 172.21.15.102:0/2879091422 >> v2:172.21.15.102:3300/0 conn(0x7fb8840c9610 0x7fb8840ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:21.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.581+0000 7fb88999b700 1 -- 172.21.15.102:0/2879091422 learned_addr learned my addr 172.21.15.102:0/2879091422 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:13:21.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.582+0000 7fb88a19c700 1 -- 172.21.15.102:0/2879091422 >> v2:172.21.15.102:3300/0 conn(0x7fb8840c9610 msgr2=0x7fb8840ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:13:21.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.582+0000 7fb88a19c700 1 --2- 172.21.15.102:0/2879091422 >> v2:172.21.15.102:3300/0 conn(0x7fb8840c9610 0x7fb8840ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.582+0000 7fb88a19c700 1 -- 172.21.15.102:0/2879091422 >> v1:172.21.15.172:6789/0 conn(0x7fb8841564b0 legacy=0x7fb8840cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.582+0000 7fb88a19c700 1 -- 172.21.15.102:0/2879091422 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8840b46f0 con 0x7fb8840ca980 2023-03-03T07:13:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.582+0000 7fb888999700 1 -- 172.21.15.102:0/2879091422 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095750 0 0) 0x7fb8840c9ca0 con 0x7fb8841564b0 2023-03-03T07:13:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.582+0000 7fb88a19c700 1 --2- 172.21.15.102:0/2879091422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840ca980 0x7fb8841636a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009d90 tx=0x7fb87c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e69f1fdc176b756 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:13:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb888999700 1 -- 172.21.15.102:0/2879091422 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c00e070 con 0x7fb8840ca980 2023-03-03T07:13:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb888999700 1 -- 172.21.15.102:0/2879091422 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c00bb20 con 0x7fb8840ca980 2023-03-03T07:13:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb888999700 1 -- 172.21.15.102:0/2879091422 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c012430 con 0x7fb8840ca980 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 -- 172.21.15.102:0/2879091422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840ca980 msgr2=0x7fb8841636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 --2- 172.21.15.102:0/2879091422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840ca980 0x7fb8841636a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009d90 tx=0x7fb87c00b330 comp rx=0 tx=0).stop 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 -- 172.21.15.102:0/2879091422 shutdown_connections 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 --2- 172.21.15.102:0/2879091422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840ca980 0x7fb8841636a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 --2- 172.21.15.102:0/2879091422 >> v2:172.21.15.102:3300/0 conn(0x7fb8840c9610 0x7fb8840ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 -- 172.21.15.102:0/2879091422 >> 172.21.15.102:0/2879091422 conn(0x7fb884065fb0 msgr2=0x7fb8840663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 -- 172.21.15.102:0/2879091422 shutdown_connections 2023-03-03T07:13:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.583+0000 7fb88a99d700 1 -- 172.21.15.102:0/2879091422 wait complete. 2023-03-03T07:13:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.585+0000 7fb88a99d700 1 Processor -- start 2023-03-03T07:13:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.586+0000 7fb88a99d700 1 -- start start 2023-03-03T07:13:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.586+0000 7fb88a99d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840c9610 0x7fb8840c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88a99d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8840ca980 0x7fb8840c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88a99d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8841564b0 0x7fb8840c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88a99d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8841eee80 con 0x7fb8840ca980 2023-03-03T07:13:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88a99d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8841ef160 con 0x7fb8841564b0 2023-03-03T07:13:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88a99d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb884153780 con 0x7fb8840c9610 2023-03-03T07:13:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8840ca980 0x7fb8840c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8840ca980 0x7fb8840c7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58884/0 (socket says 172.21.15.102:58884) 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 -- 172.21.15.102:0/1197677764 learned_addr learned my addr 172.21.15.102:0/1197677764 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88999b700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840c9610 0x7fb8840c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88a19c700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8841564b0 0x7fb8840c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 -- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840c9610 msgr2=0x7fb8840c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840c9610 0x7fb8840c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 -- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8841564b0 msgr2=0x7fb8840c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:21.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8841564b0 0x7fb8840c4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.587+0000 7fb88919a700 1 -- 172.21.15.102:0/1197677764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c009a70 con 0x7fb8840ca980 2023-03-03T07:13:21.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.588+0000 7fb88919a700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8840ca980 0x7fb8840c7730 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb87400e980 tx=0x7fb87400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:13:21.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.588+0000 7fb87a7fc700 1 -- 172.21.15.102:0/1197677764 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874009710 con 0x7fb8840ca980 2023-03-03T07:13:21.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.588+0000 7fb87a7fc700 1 -- 172.21.15.102:0/1197677764 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87400c690 con 0x7fb8840ca980 2023-03-03T07:13:21.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.588+0000 7fb87a7fc700 1 -- 172.21.15.102:0/1197677764 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87400ca40 con 0x7fb8840ca980 2023-03-03T07:13:21.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.588+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8840b2210 con 0x7fb8840ca980 2023-03-03T07:13:21.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.588+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8840b1f30 con 0x7fb8840ca980 2023-03-03T07:13:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.589+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb884156880 con 0x7fb8840ca980 2023-03-03T07:13:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.590+0000 7fb87a7fc700 1 -- 172.21.15.102:0/1197677764 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb874019030 con 0x7fb8840ca980 2023-03-03T07:13:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.591+0000 7fb87a7fc700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8680682f0 0x7fb86806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:13:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.592+0000 7fb87a7fc700 1 -- 172.21.15.102:0/1197677764 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb874085c40 con 0x7fb8840ca980 2023-03-03T07:13:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.592+0000 7fb88999b700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8680682f0 0x7fb86806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:13:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.592+0000 7fb88999b700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8680682f0 0x7fb86806a7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb880000f60 tx=0x7fb880007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:13:21.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.596+0000 7fb87a7fc700 1 -- 172.21.15.102:0/1197677764 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb874052350 con 0x7fb8840ca980 2023-03-03T07:13:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.830+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8840677f0 con 0x7fb8840ca980 2023-03-03T07:13:21.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.830+0000 7fb87a7fc700 1 -- 172.21.15.102:0/1197677764 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb874055970 con 0x7fb8840ca980 2023-03-03T07:13:21.833 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:13:21.833 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:13:21.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8680682f0 msgr2=0x7fb86806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:21.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8680682f0 0x7fb86806a7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb880000f60 tx=0x7fb880007e70 comp rx=0 tx=0).stop 2023-03-03T07:13:21.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8840ca980 msgr2=0x7fb8840c7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:13:21.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8840ca980 0x7fb8840c7730 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb87400e980 tx=0x7fb87400ec90 comp rx=0 tx=0).stop 2023-03-03T07:13:21.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 shutdown_connections 2023-03-03T07:13:21.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8680682f0 0x7fb86806a7a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8840c9610 0x7fb8840c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8841564b0 0x7fb8840c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 --2- 172.21.15.102:0/1197677764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8840ca980 0x7fb8840c7730 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:13:21.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.833+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 >> 172.21.15.102:0/1197677764 conn(0x7fb884065fb0 msgr2=0x7fb884158e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:13:21.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.834+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 shutdown_connections 2023-03-03T07:13:21.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:13:21.834+0000 7fb88a99d700 1 -- 172.21.15.102:0/1197677764 wait complete. 2023-03-03T07:13:21.838 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:13:21.863 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2023-03-03T07:13:29.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:29.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:59.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:59.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:14:29.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:14:30.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:14:49.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:14:50.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.114+0000 7fd0a59ff700 1 Processor -- start 2023-03-03T07:14:50.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.115+0000 7fd0a59ff700 1 -- start start 2023-03-03T07:14:50.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.116+0000 7fd0a59ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0a0153290 0x7fd0a0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:50.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.116+0000 7fd0a59ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a0153660 con 0x7fd0a01593c0 2023-03-03T07:14:50.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.116+0000 7fd0a59ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a006c360 con 0x7fd0a00cd8e0 2023-03-03T07:14:50.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.116+0000 7fd0a59ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a00cec40 con 0x7fd0a0153290 2023-03-03T07:14:50.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.116+0000 7fd09effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0a0153290 0x7fd0a0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:50.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.116+0000 7fd09effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0a0153290 0x7fd0a0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40312/0 (socket says 172.21.15.102:40312) 2023-03-03T07:14:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.116+0000 7fd09effd700 1 -- 172.21.15.102:0/3339795433 learned_addr learned my addr 172.21.15.102:0/3339795433 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:14:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.117+0000 7fd09effd700 1 -- 172.21.15.102:0/3339795433 >> v1:172.21.15.172:6789/0 conn(0x7fd0a00cd8e0 legacy=0x7fd0a00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.117+0000 7fd09effd700 1 -- 172.21.15.102:0/3339795433 >> v1:172.21.15.102:6789/0 conn(0x7fd0a01593c0 legacy=0x7fd0a00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.117+0000 7fd09effd700 1 -- 172.21.15.102:0/3339795433 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a015cea0 con 0x7fd0a0153290 2023-03-03T07:14:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.117+0000 7fd09effd700 1 --2- 172.21.15.102:0/3339795433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a0153290 0x7fd0a0158e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd0900098f0 tx=0x7fd090009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3c328c6346ebdd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:14:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.117+0000 7fd09dffb700 1 -- 172.21.15.102:0/3339795433 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09000b8a0 con 0x7fd0a0153290 2023-03-03T07:14:50.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.117+0000 7fd09dffb700 1 -- 172.21.15.102:0/3339795433 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09000ba00 con 0x7fd0a0153290 2023-03-03T07:14:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.117+0000 7fd09dffb700 1 -- 172.21.15.102:0/3339795433 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09000bd90 con 0x7fd0a0153290 2023-03-03T07:14:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.118+0000 7fd0a59ff700 1 -- 172.21.15.102:0/3339795433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a0153290 msgr2=0x7fd0a0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.118+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/3339795433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a0153290 0x7fd0a0158e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd0900098f0 tx=0x7fd090009c00 comp rx=0 tx=0).stop 2023-03-03T07:14:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.118+0000 7fd0a59ff700 1 -- 172.21.15.102:0/3339795433 shutdown_connections 2023-03-03T07:14:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.118+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/3339795433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a0153290 0x7fd0a0158e80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.118+0000 7fd0a59ff700 1 -- 172.21.15.102:0/3339795433 >> 172.21.15.102:0/3339795433 conn(0x7fd0a0065fb0 msgr2=0x7fd0a00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:14:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.118+0000 7fd0a59ff700 1 -- 172.21.15.102:0/3339795433 shutdown_connections 2023-03-03T07:14:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.118+0000 7fd0a59ff700 1 -- 172.21.15.102:0/3339795433 wait complete. 2023-03-03T07:14:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.120+0000 7fd0a59ff700 1 Processor -- start 2023-03-03T07:14:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.120+0000 7fd0a59ff700 1 -- start start 2023-03-03T07:14:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.121+0000 7fd0a59ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00cd8e0 0x7fd0a00b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.121+0000 7fd0a59ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a0153290 0x7fd0a00c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.121+0000 7fd0a59ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a01593c0 0x7fd0a00c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.121+0000 7fd0a59ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00c6de0 con 0x7fd0a01593c0 2023-03-03T07:14:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.121+0000 7fd0a59ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00c8760 con 0x7fd0a0153290 2023-03-03T07:14:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.121+0000 7fd0a59ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00c8a40 con 0x7fd0a00cd8e0 2023-03-03T07:14:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09e7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a0153290 0x7fd0a00c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a01593c0 0x7fd0a00c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00cd8e0 0x7fd0a00b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00cd8e0 0x7fd0a00b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34104/0 (socket says 172.21.15.102:34104) 2023-03-03T07:14:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09e7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a0153290 0x7fd0a00c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52796/0 (socket says 172.21.15.102:52796) 2023-03-03T07:14:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09effd700 1 -- 172.21.15.102:0/4140838642 learned_addr learned my addr 172.21.15.102:0/4140838642 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:14:50.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09f7fe700 1 -- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00cd8e0 msgr2=0x7fd0a00b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:50.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09f7fe700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00cd8e0 0x7fd0a00b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09f7fe700 1 -- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a0153290 msgr2=0x7fd0a00c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09f7fe700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a0153290 0x7fd0a00c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09f7fe700 1 -- 172.21.15.102:0/4140838642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd090009550 con 0x7fd0a01593c0 2023-03-03T07:14:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09e7fc700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a0153290 0x7fd0a00c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:14:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.122+0000 7fd09effd700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00cd8e0 0x7fd0a00b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:14:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.123+0000 7fd09f7fe700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a01593c0 0x7fd0a00c7550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd09800cae0 tx=0x7fd09800cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:14:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.123+0000 7fd087fff700 1 -- 172.21.15.102:0/4140838642 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098003c00 con 0x7fd0a01593c0 2023-03-03T07:14:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.123+0000 7fd087fff700 1 -- 172.21.15.102:0/4140838642 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd098003d60 con 0x7fd0a01593c0 2023-03-03T07:14:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.123+0000 7fd087fff700 1 -- 172.21.15.102:0/4140838642 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09800aa20 con 0x7fd0a01593c0 2023-03-03T07:14:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.123+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a015d160 con 0x7fd0a01593c0 2023-03-03T07:14:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.123+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a006c7f0 con 0x7fd0a01593c0 2023-03-03T07:14:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.124+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a00ccde0 con 0x7fd0a01593c0 2023-03-03T07:14:50.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.125+0000 7fd087fff700 1 -- 172.21.15.102:0/4140838642 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd098010050 con 0x7fd0a01593c0 2023-03-03T07:14:50.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.127+0000 7fd087fff700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd08c0682f0 0x7fd08c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:50.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.127+0000 7fd087fff700 1 -- 172.21.15.102:0/4140838642 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd098085b60 con 0x7fd0a01593c0 2023-03-03T07:14:50.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.127+0000 7fd09effd700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd08c0682f0 0x7fd08c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.128+0000 7fd09effd700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd08c0682f0 0x7fd08c06a7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd090009c40 tx=0x7fd090008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:14:50.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.131+0000 7fd087fff700 1 -- 172.21.15.102:0/4140838642 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd098052210 con 0x7fd0a01593c0 2023-03-03T07:14:50.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.367+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0a004e6e0 con 0x7fd0a01593c0 2023-03-03T07:14:50.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.367+0000 7fd087fff700 1 -- 172.21.15.102:0/4140838642 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd098055830 con 0x7fd0a01593c0 2023-03-03T07:14:50.370 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:14:50.370 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:14:50.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd08c0682f0 msgr2=0x7fd08c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:50.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd08c0682f0 0x7fd08c06a7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd090009c40 tx=0x7fd090008030 comp rx=0 tx=0).stop 2023-03-03T07:14:50.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a01593c0 msgr2=0x7fd0a00c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:50.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a01593c0 0x7fd0a00c7550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd09800cae0 tx=0x7fd09800cdf0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 shutdown_connections 2023-03-03T07:14:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd08c0682f0 0x7fd08c06a7a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0a00cd8e0 0x7fd0a00b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a01593c0 0x7fd0a00c7550 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 --2- 172.21.15.102:0/4140838642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0a0153290 0x7fd0a00c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 >> 172.21.15.102:0/4140838642 conn(0x7fd0a0065fb0 msgr2=0x7fd0a00cb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:14:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 shutdown_connections 2023-03-03T07:14:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:50.370+0000 7fd0a59ff700 1 -- 172.21.15.102:0/4140838642 wait complete. 2023-03-03T07:14:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:14:50.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T07:14:59.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:14:59.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.648+0000 7ff195b9d700 1 Processor -- start 2023-03-03T07:14:59.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.649+0000 7ff195b9d700 1 -- start start 2023-03-03T07:14:59.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.650+0000 7ff195b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff1900cab50 0x7ff1901646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:59.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.650+0000 7ff195b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900c9ca0 con 0x7ff1901564b0 2023-03-03T07:14:59.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.650+0000 7ff195b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900b46f0 con 0x7ff1900c9610 2023-03-03T07:14:59.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.650+0000 7ff195b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900b4920 con 0x7ff1900cab50 2023-03-03T07:14:59.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.650+0000 7ff194b9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff1901564b0 0x7ff1900cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54016/0 (socket says 172.21.15.102:54016) 2023-03-03T07:14:59.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.650+0000 7ff194b9b700 1 -- 172.21.15.102:0/1261005899 learned_addr learned my addr 172.21.15.102:0/1261005899 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:14:59.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.650+0000 7ff19539c700 1 --2- 172.21.15.102:0/1261005899 >> v2:172.21.15.172:3300/0 conn(0x7ff1900cab50 0x7ff1901646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:59.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.651+0000 7ff18f7fe700 1 -- 172.21.15.102:0/1261005899 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899387436 0 0) 0x7ff1900c9ca0 con 0x7ff1901564b0 2023-03-03T07:14:59.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.651+0000 7ff19539c700 1 -- 172.21.15.102:0/1261005899 >> v1:172.21.15.172:6789/0 conn(0x7ff1900c9610 legacy=0x7ff1900ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:59.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.651+0000 7ff19539c700 1 -- 172.21.15.102:0/1261005899 >> v1:172.21.15.102:6789/0 conn(0x7ff1901564b0 legacy=0x7ff1900cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:59.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.651+0000 7ff19539c700 1 -- 172.21.15.102:0/1261005899 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900b4c00 con 0x7ff1900cab50 2023-03-03T07:14:59.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.651+0000 7ff19539c700 1 --2- 172.21.15.102:0/1261005899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1900cab50 0x7ff1901646b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff188009d90 tx=0x7ff18800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32d43832ca1b0f2e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:14:59.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.651+0000 7ff18f7fe700 1 -- 172.21.15.102:0/1261005899 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18800e070 con 0x7ff1900cab50 2023-03-03T07:14:59.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff18f7fe700 1 -- 172.21.15.102:0/1261005899 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18800bad0 con 0x7ff1900cab50 2023-03-03T07:14:59.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff18f7fe700 1 -- 172.21.15.102:0/1261005899 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18800be80 con 0x7ff1900cab50 2023-03-03T07:14:59.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff195b9d700 1 -- 172.21.15.102:0/1261005899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1900cab50 msgr2=0x7ff1901646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:59.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff195b9d700 1 --2- 172.21.15.102:0/1261005899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1900cab50 0x7ff1901646b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff188009d90 tx=0x7ff18800b330 comp rx=0 tx=0).stop 2023-03-03T07:14:59.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff195b9d700 1 -- 172.21.15.102:0/1261005899 shutdown_connections 2023-03-03T07:14:59.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff195b9d700 1 --2- 172.21.15.102:0/1261005899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1900cab50 0x7ff1901646b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:59.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff195b9d700 1 -- 172.21.15.102:0/1261005899 >> 172.21.15.102:0/1261005899 conn(0x7ff190065fb0 msgr2=0x7ff1900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:14:59.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff195b9d700 1 -- 172.21.15.102:0/1261005899 shutdown_connections 2023-03-03T07:14:59.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.652+0000 7ff195b9d700 1 -- 172.21.15.102:0/1261005899 wait complete. 2023-03-03T07:14:59.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.654+0000 7ff195b9d700 1 Processor -- start 2023-03-03T07:14:59.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.654+0000 7ff195b9d700 1 -- start start 2023-03-03T07:14:59.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.655+0000 7ff195b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1900c9610 0x7ff190157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:59.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.655+0000 7ff195b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1900cab50 0x7ff1901f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:59.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.655+0000 7ff195b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1901564b0 0x7ff190158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:59.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.655+0000 7ff195b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1901f2970 con 0x7ff1900cab50 2023-03-03T07:14:59.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.655+0000 7ff195b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190158c40 con 0x7ff1901564b0 2023-03-03T07:14:59.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.655+0000 7ff195b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff190154430 con 0x7ff1900c9610 2023-03-03T07:14:59.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1900c9610 0x7ff190157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:59.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1900c9610 0x7ff190157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60600/0 (socket says 172.21.15.102:60600) 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 -- 172.21.15.102:0/723522808 learned_addr learned my addr 172.21.15.102:0/723522808 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff18ffff700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1900cab50 0x7ff1901f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff19539c700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1901564b0 0x7ff190158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 -- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1901564b0 msgr2=0x7ff190158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1901564b0 0x7ff190158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 -- 172.21.15.102:0/723522808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1900cab50 msgr2=0x7ff1901f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1900cab50 0x7ff1901f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 -- 172.21.15.102:0/723522808 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff188009a70 con 0x7ff1900c9610 2023-03-03T07:14:59.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.656+0000 7ff194b9b700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1900c9610 0x7ff190157b00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c550 tx=0x7ff18400c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:14:59.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.657+0000 7ff18d7fa700 1 -- 172.21.15.102:0/723522808 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184004c80 con 0x7ff1900c9610 2023-03-03T07:14:59.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.657+0000 7ff18d7fa700 1 -- 172.21.15.102:0/723522808 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff184004e00 con 0x7ff1900c9610 2023-03-03T07:14:59.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.657+0000 7ff18d7fa700 1 -- 172.21.15.102:0/723522808 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184007310 con 0x7ff1900c9610 2023-03-03T07:14:59.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.657+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1900b2210 con 0x7ff1900c9610 2023-03-03T07:14:59.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.657+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1900b1f30 con 0x7ff1900c9610 2023-03-03T07:14:59.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.658+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff190156880 con 0x7ff1900c9610 2023-03-03T07:14:59.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.660+0000 7ff18d7fa700 1 -- 172.21.15.102:0/723522808 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff18401a020 con 0x7ff1900c9610 2023-03-03T07:14:59.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.661+0000 7ff18d7fa700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1740682f0 0x7ff17406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:14:59.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.661+0000 7ff18d7fa700 1 -- 172.21.15.102:0/723522808 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff1840857f0 con 0x7ff1900c9610 2023-03-03T07:14:59.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.661+0000 7ff18ffff700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1740682f0 0x7ff17406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:14:59.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.662+0000 7ff18ffff700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1740682f0 0x7ff17406a7a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1900b3130 tx=0x7ff17c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:14:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.665+0000 7ff18d7fa700 1 -- 172.21.15.102:0/723522808 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff184051f00 con 0x7ff1900c9610 2023-03-03T07:14:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.902+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff190154c20 con 0x7ff1900c9610 2023-03-03T07:14:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.904+0000 7ff18d7fa700 1 -- 172.21.15.102:0/723522808 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff184055520 con 0x7ff1900c9610 2023-03-03T07:14:59.906 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:14:59.906 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:14:59.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1740682f0 msgr2=0x7ff17406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:59.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1740682f0 0x7ff17406a7a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1900b3130 tx=0x7ff17c009040 comp rx=0 tx=0).stop 2023-03-03T07:14:59.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1900c9610 msgr2=0x7ff190157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:14:59.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1900c9610 0x7ff190157b00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c550 tx=0x7ff18400c860 comp rx=0 tx=0).stop 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 shutdown_connections 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1740682f0 0x7ff17406a7a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1900c9610 0x7ff190157b00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1901564b0 0x7ff190158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 --2- 172.21.15.102:0/723522808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1900cab50 0x7ff1901f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 >> 172.21.15.102:0/723522808 conn(0x7ff190065fb0 msgr2=0x7ff1901dff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 shutdown_connections 2023-03-03T07:14:59.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:14:59.906+0000 7ff195b9d700 1 -- 172.21.15.102:0/723522808 wait complete. 2023-03-03T07:14:59.911 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:14:59.935 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T07:15:00.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:00.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:30.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:30.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:37.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:15:38.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.200+0000 7f54c49ff700 1 Processor -- start 2023-03-03T07:15:38.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.200+0000 7f54c49ff700 1 -- start start 2023-03-03T07:15:38.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.201+0000 7f54c49ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54c00cd8e0 0x7f54c00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:15:38.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54c49ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c0153660 con 0x7f54c00cd390 2023-03-03T07:15:38.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54c49ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c006c360 con 0x7f54c0153290 2023-03-03T07:15:38.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54c49ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c015cea0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54be59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f54c0153290 0x7f54c0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47914/0 (socket says 172.21.15.102:47914) 2023-03-03T07:15:38.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54be59c700 1 -- 172.21.15.102:0/4090502351 learned_addr learned my addr 172.21.15.102:0/4090502351 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:15:38.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54bdd9b700 1 --2- 172.21.15.102:0/4090502351 >> v2:172.21.15.172:3300/0 conn(0x7f54c00cd8e0 0x7f54c00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:15:38.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54bd59a700 1 -- 172.21.15.102:0/4090502351 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941156059 0 0) 0x7f54c006c360 con 0x7f54c0153290 2023-03-03T07:15:38.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54bd59a700 1 -- 172.21.15.102:0/4090502351 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a80035b0 con 0x7f54c0153290 2023-03-03T07:15:38.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54bdd9b700 1 -- 172.21.15.102:0/4090502351 >> v1:172.21.15.172:6789/0 conn(0x7f54c0153290 legacy=0x7f54c0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:15:38.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54bd59a700 1 -- 172.21.15.102:0/4090502351 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866371510 0 0) 0x7f54c0153660 con 0x7f54c00cd390 2023-03-03T07:15:38.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54bdd9b700 1 -- 172.21.15.102:0/4090502351 >> v1:172.21.15.102:6789/0 conn(0x7f54c00cd390 legacy=0x7f54c00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:15:38.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.202+0000 7f54bdd9b700 1 -- 172.21.15.102:0/4090502351 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c006c7f0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.203+0000 7f54bdd9b700 1 --2- 172.21.15.102:0/4090502351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c00cde20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009840 tx=0x7f54b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=857eba7c7011f1f7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:15:38.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.203+0000 7f54bd59a700 1 -- 172.21.15.102:0/4090502351 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4011830 con 0x7f54c00cd8e0 2023-03-03T07:15:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.203+0000 7f54bd59a700 1 -- 172.21.15.102:0/4090502351 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b4011990 con 0x7f54c00cd8e0 2023-03-03T07:15:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.203+0000 7f54bd59a700 1 -- 172.21.15.102:0/4090502351 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4011d20 con 0x7f54c00cd8e0 2023-03-03T07:15:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.204+0000 7f54c49ff700 1 -- 172.21.15.102:0/4090502351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 msgr2=0x7f54c00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:15:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.204+0000 7f54c49ff700 1 --2- 172.21.15.102:0/4090502351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c00cde20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009840 tx=0x7f54b4009b50 comp rx=0 tx=0).stop 2023-03-03T07:15:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.204+0000 7f54c49ff700 1 -- 172.21.15.102:0/4090502351 shutdown_connections 2023-03-03T07:15:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.204+0000 7f54c49ff700 1 --2- 172.21.15.102:0/4090502351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c00cde20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.204+0000 7f54c49ff700 1 -- 172.21.15.102:0/4090502351 >> 172.21.15.102:0/4090502351 conn(0x7f54c0065fb0 msgr2=0x7f54c00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:15:38.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.204+0000 7f54c49ff700 1 -- 172.21.15.102:0/4090502351 shutdown_connections 2023-03-03T07:15:38.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.204+0000 7f54c49ff700 1 -- 172.21.15.102:0/4090502351 wait complete. 2023-03-03T07:15:38.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.206+0000 7f54c49ff700 1 Processor -- start 2023-03-03T07:15:38.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.206+0000 7f54c49ff700 1 -- start start 2023-03-03T07:15:38.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.207+0000 7f54c49ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c00cd390 0x7f54c00c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:15:38.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.207+0000 7f54c49ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c01eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:15:38.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.207+0000 7f54c49ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c0153290 0x7f54c01ef150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:15:38.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.207+0000 7f54c49ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c01ee7f0 con 0x7f54c00cd390 2023-03-03T07:15:38.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.207+0000 7f54c49ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00c6370 con 0x7f54c00cd8e0 2023-03-03T07:15:38.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.207+0000 7f54c49ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00c6610 con 0x7f54c0153290 2023-03-03T07:15:38.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.207+0000 7f54be59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c00cd390 0x7f54c00c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:15:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c01eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:15:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c01eec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33936/0 (socket says 172.21.15.102:33936) 2023-03-03T07:15:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 -- 172.21.15.102:0/1144213796 learned_addr learned my addr 172.21.15.102:0/1144213796 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:15:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bed9d700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c0153290 0x7f54c01ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:15:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 -- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c0153290 msgr2=0x7f54c01ef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:15:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c0153290 0x7f54c01ef150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 -- 172.21.15.102:0/1144213796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c00cd390 msgr2=0x7f54c00c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:15:38.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c00cd390 0x7f54c00c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 -- 172.21.15.102:0/1144213796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b40094a0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bed9d700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c0153290 0x7f54c01ef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:15:38.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54be59c700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c00cd390 0x7f54c00c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:15:38.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.208+0000 7f54bdd9b700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c01eec10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f54b40090b0 tx=0x7f54b4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:15:38.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.209+0000 7f54aeffd700 1 -- 172.21.15.102:0/1144213796 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b401a860 con 0x7f54c00cd8e0 2023-03-03T07:15:38.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.209+0000 7f54aeffd700 1 -- 172.21.15.102:0/1144213796 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b401a9c0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.209+0000 7f54aeffd700 1 -- 172.21.15.102:0/1144213796 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b401ad30 con 0x7f54c00cd8e0 2023-03-03T07:15:38.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.209+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c00ccde0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.209+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c015d160 con 0x7f54c00cd8e0 2023-03-03T07:15:38.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.210+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c00cd0c0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.211+0000 7f54aeffd700 1 -- 172.21.15.102:0/1144213796 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f54b400f050 con 0x7f54c00cd8e0 2023-03-03T07:15:38.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.212+0000 7f54aeffd700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54a0068320 0x7f54a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:15:38.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.212+0000 7f54aeffd700 1 -- 172.21.15.102:0/1144213796 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f54b4086c00 con 0x7f54c00cd8e0 2023-03-03T07:15:38.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.213+0000 7f54be59c700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54a0068320 0x7f54a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:15:38.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.213+0000 7f54be59c700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54a0068320 0x7f54a006a7d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f54b0007fd0 tx=0x7f54b0007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:15:38.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.217+0000 7f54aeffd700 1 -- 172.21.15.102:0/1144213796 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f54b40533c0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.450+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54c004e6e0 con 0x7f54c00cd8e0 2023-03-03T07:15:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.451+0000 7f54aeffd700 1 -- 172.21.15.102:0/1144213796 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f54b4029090 con 0x7f54c00cd8e0 2023-03-03T07:15:38.453 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:15:38.453 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:15:38.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54a0068320 msgr2=0x7f54a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:15:38.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54a0068320 0x7f54a006a7d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f54b0007fd0 tx=0x7f54b0007dc0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 msgr2=0x7f54c01eec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:15:38.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c01eec10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f54b40090b0 tx=0x7f54b4004750 comp rx=0 tx=0).stop 2023-03-03T07:15:38.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 shutdown_connections 2023-03-03T07:15:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54a0068320 0x7f54a006a7d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54c0153290 0x7f54c01ef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54c00cd390 0x7f54c00c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 --2- 172.21.15.102:0/1144213796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54c00cd8e0 0x7f54c01eec10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:15:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 >> 172.21.15.102:0/1144213796 conn(0x7f54c0065fb0 msgr2=0x7f54c00cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:15:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 shutdown_connections 2023-03-03T07:15:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:15:38.454+0000 7f54c49ff700 1 -- 172.21.15.102:0/1144213796 wait complete. 2023-03-03T07:15:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:15:38.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T07:16:00.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:16:00.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:16:30.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:16:30.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:00.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:00.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:29.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:17:29.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.736+0000 7f8eba3ff700 1 Processor -- start 2023-03-03T07:17:29.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.736+0000 7f8eba3ff700 1 -- start start 2023-03-03T07:17:29.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.737+0000 7f8eba3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8eb4153260 0x7f8eb40ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:17:29.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.737+0000 7f8eba3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40cf940 con 0x7f8eb40cac20 2023-03-03T07:17:29.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.737+0000 7f8eba3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40cfb90 con 0x7f8eb40c9cb0 2023-03-03T07:17:29.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.737+0000 7f8eba3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb40cfe70 con 0x7f8eb4153260 2023-03-03T07:17:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8eb4153260 0x7f8eb40ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:17:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb899c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8eb40cac20 0x7f8eb4163610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54308/0 (socket says 172.21.15.102:54308) 2023-03-03T07:17:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb899c700 1 -- 172.21.15.102:0/2508244762 learned_addr learned my addr 172.21.15.102:0/2508244762 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:17:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb37fe700 1 --1- 172.21.15.102:0/2508244762 >> v1:172.21.15.172:6789/0 conn(0x7f8eb40c9cb0 0x7f8eb40cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51698/0 (socket says 172.21.15.102:51698) 2023-03-03T07:17:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb2ffd700 1 -- 172.21.15.102:0/2508244762 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548196475 0 0) 0x7f8eb40cfb90 con 0x7f8eb40c9cb0 2023-03-03T07:17:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb2ffd700 1 -- 172.21.15.102:0/2508244762 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c0035b0 con 0x7f8eb40c9cb0 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb2ffd700 1 -- 172.21.15.102:0/2508244762 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107780016 0 0) 0x7f8eb40cf940 con 0x7f8eb40cac20 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb3fff700 1 -- 172.21.15.102:0/2508244762 >> v1:172.21.15.172:6789/0 conn(0x7f8eb40c9cb0 legacy=0x7f8eb40cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb3fff700 1 -- 172.21.15.102:0/2508244762 >> v1:172.21.15.102:6789/0 conn(0x7f8eb40cac20 legacy=0x7f8eb4163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.738+0000 7f8eb3fff700 1 -- 172.21.15.102:0/2508244762 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4158f80 con 0x7f8eb4153260 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.739+0000 7f8eb3fff700 1 --2- 172.21.15.102:0/2508244762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb4153260 0x7f8eb40ca6e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009840 tx=0x7f8ea4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d16e373bf028b6f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.739+0000 7f8eb2ffd700 1 -- 172.21.15.102:0/2508244762 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4011830 con 0x7f8eb4153260 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.739+0000 7f8eb2ffd700 1 -- 172.21.15.102:0/2508244762 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea4011990 con 0x7f8eb4153260 2023-03-03T07:17:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.739+0000 7f8eb2ffd700 1 -- 172.21.15.102:0/2508244762 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4011d20 con 0x7f8eb4153260 2023-03-03T07:17:29.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.739+0000 7f8eba3ff700 1 -- 172.21.15.102:0/2508244762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb4153260 msgr2=0x7f8eb40ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:17:29.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.739+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/2508244762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb4153260 0x7f8eb40ca6e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009840 tx=0x7f8ea4009b50 comp rx=0 tx=0).stop 2023-03-03T07:17:29.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.740+0000 7f8eba3ff700 1 -- 172.21.15.102:0/2508244762 shutdown_connections 2023-03-03T07:17:29.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.740+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/2508244762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb4153260 0x7f8eb40ca6e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:17:29.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.740+0000 7f8eba3ff700 1 -- 172.21.15.102:0/2508244762 >> 172.21.15.102:0/2508244762 conn(0x7f8eb4066c00 msgr2=0x7f8eb40ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:17:29.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.740+0000 7f8eba3ff700 1 -- 172.21.15.102:0/2508244762 shutdown_connections 2023-03-03T07:17:29.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.740+0000 7f8eba3ff700 1 -- 172.21.15.102:0/2508244762 wait complete. 2023-03-03T07:17:29.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.742+0000 7f8eba3ff700 1 Processor -- start 2023-03-03T07:17:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.742+0000 7f8eba3ff700 1 -- start start 2023-03-03T07:17:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eba3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eb40cac20 0x7f8eb41536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:17:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eba3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eb4153260 0x7f8eb414e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:17:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eba3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb41586f0 0x7f8eb414be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:17:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eba3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb4153c20 con 0x7f8eb41586f0 2023-03-03T07:17:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eba3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40aede0 con 0x7f8eb4153260 2023-03-03T07:17:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eba3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb414dab0 con 0x7f8eb40cac20 2023-03-03T07:17:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eb3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eb40cac20 0x7f8eb41536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:17:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eb37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eb4153260 0x7f8eb414e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:17:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eb3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eb40cac20 0x7f8eb41536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59504/0 (socket says 172.21.15.102:59504) 2023-03-03T07:17:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eb3fff700 1 -- 172.21.15.102:0/4017883454 learned_addr learned my addr 172.21.15.102:0/4017883454 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:17:29.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.743+0000 7f8eb899c700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb41586f0 0x7f8eb414be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:17:29.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb3fff700 1 -- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eb4153260 msgr2=0x7f8eb414e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:17:29.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb3fff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eb4153260 0x7f8eb414e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:17:29.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb3fff700 1 -- 172.21.15.102:0/4017883454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb41586f0 msgr2=0x7f8eb414be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:17:29.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb3fff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb41586f0 0x7f8eb414be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:17:29.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb3fff700 1 -- 172.21.15.102:0/4017883454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40094a0 con 0x7f8eb40cac20 2023-03-03T07:17:29.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb899c700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb41586f0 0x7f8eb414be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:17:29.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb3fff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eb40cac20 0x7f8eb41536e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4014040 tx=0x7f8ea4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:17:29.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb0ff9700 1 -- 172.21.15.102:0/4017883454 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea401a7e0 con 0x7f8eb40cac20 2023-03-03T07:17:29.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb0ff9700 1 -- 172.21.15.102:0/4017883454 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea401a940 con 0x7f8eb40cac20 2023-03-03T07:17:29.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eb0ff9700 1 -- 172.21.15.102:0/4017883454 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea401acd0 con 0x7f8eb40cac20 2023-03-03T07:17:29.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.744+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb4159540 con 0x7f8eb40cac20 2023-03-03T07:17:29.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.745+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb4159260 con 0x7f8eb40cac20 2023-03-03T07:17:29.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.745+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb4159820 con 0x7f8eb40cac20 2023-03-03T07:17:29.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.747+0000 7f8eb0ff9700 1 -- 172.21.15.102:0/4017883454 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8ea400f050 con 0x7f8eb40cac20 2023-03-03T07:17:29.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.748+0000 7f8eb0ff9700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ea00682f0 0x7f8ea006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:17:29.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.748+0000 7f8eb0ff9700 1 -- 172.21.15.102:0/4017883454 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8ea4086c80 con 0x7f8eb40cac20 2023-03-03T07:17:29.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.748+0000 7f8eb37fe700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ea00682f0 0x7f8ea006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:17:29.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.749+0000 7f8eb37fe700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ea00682f0 0x7f8ea006a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8eb41ec490 tx=0x7f8ea8007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:17:29.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.753+0000 7f8eb0ff9700 1 -- 172.21.15.102:0/4017883454 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ea4053310 con 0x7f8eb40cac20 2023-03-03T07:17:29.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.993+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8eb406cad0 con 0x7f8eb40cac20 2023-03-03T07:17:29.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.994+0000 7f8eb0ff9700 1 -- 172.21.15.102:0/4017883454 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8ea4029090 con 0x7f8eb40cac20 2023-03-03T07:17:29.996 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:17:29.996 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:17:29.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ea00682f0 msgr2=0x7f8ea006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:17:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ea00682f0 0x7f8ea006a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8eb41ec490 tx=0x7f8ea8007e30 comp rx=0 tx=0).stop 2023-03-03T07:17:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eb40cac20 msgr2=0x7f8eb41536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:17:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eb40cac20 0x7f8eb41536e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4014040 tx=0x7f8ea4004750 comp rx=0 tx=0).stop 2023-03-03T07:17:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 shutdown_connections 2023-03-03T07:17:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ea00682f0 0x7f8ea006a7a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:17:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8eb40cac20 0x7f8eb41536e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:17:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb41586f0 0x7f8eb414be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:17:30.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 --2- 172.21.15.102:0/4017883454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8eb4153260 0x7f8eb414e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:17:30.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 >> 172.21.15.102:0/4017883454 conn(0x7f8eb40c9cb0 msgr2=0x7f8eb4157980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:17:30.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 shutdown_connections 2023-03-03T07:17:30.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:17:29.997+0000 7f8eba3ff700 1 -- 172.21.15.102:0/4017883454 wait complete. 2023-03-03T07:17:30.001 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:17:30.026 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T07:17:30.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:30.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:01.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:01.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:31.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:31.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:49.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:18:49.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.290+0000 7fce3c5ff700 1 Processor -- start 2023-03-03T07:18:49.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.290+0000 7fce3c5ff700 1 -- start start 2023-03-03T07:18:49.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.291+0000 7fce3c5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fce34153290 0x7fce34158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:18:49.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.291+0000 7fce3c5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce34153660 con 0x7fce340cd8e0 2023-03-03T07:18:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.291+0000 7fce3c5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3406c360 con 0x7fce341593c0 2023-03-03T07:18:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.291+0000 7fce3c5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce340cec40 con 0x7fce34153290 2023-03-03T07:18:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce39b9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fce341593c0 0x7fce340cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46544/0 (socket says 172.21.15.102:46544) 2023-03-03T07:18:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce39b9a700 1 -- 172.21.15.102:0/3824061319 learned_addr learned my addr 172.21.15.102:0/3824061319 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:18:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce3ab9c700 1 --1- 172.21.15.102:0/3824061319 >> v1:172.21.15.102:6789/0 conn(0x7fce340cd8e0 0x7fce340ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56500/0 (socket says 172.21.15.102:56500) 2023-03-03T07:18:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce3a39b700 1 --2- 172.21.15.102:0/3824061319 >> v2:172.21.15.172:3300/0 conn(0x7fce34153290 0x7fce34158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:18:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce3a39b700 1 -- 172.21.15.102:0/3824061319 >> v1:172.21.15.172:6789/0 conn(0x7fce341593c0 legacy=0x7fce340cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:18:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce3a39b700 1 -- 172.21.15.102:0/3824061319 >> v1:172.21.15.102:6789/0 conn(0x7fce340cd8e0 legacy=0x7fce340ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:18:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce3a39b700 1 -- 172.21.15.102:0/3824061319 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3415cea0 con 0x7fce34153290 2023-03-03T07:18:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.292+0000 7fce39399700 1 -- 172.21.15.102:0/3824061319 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534898154 0 0) 0x7fce3406c360 con 0x7fce341593c0 2023-03-03T07:18:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce3a39b700 1 --2- 172.21.15.102:0/3824061319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce34158e80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fce30009840 tx=0x7fce30009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ee0aab5e44cdbef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:18:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce39399700 1 -- 172.21.15.102:0/3824061319 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30004230 con 0x7fce34153290 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce39399700 1 -- 172.21.15.102:0/3824061319 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30004390 con 0x7fce34153290 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce39399700 1 -- 172.21.15.102:0/3824061319 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce300036a0 con 0x7fce34153290 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3824061319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 msgr2=0x7fce34158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3824061319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce34158e80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fce30009840 tx=0x7fce30009b50 comp rx=0 tx=0).stop 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3824061319 shutdown_connections 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3824061319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce34158e80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.293+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3824061319 >> 172.21.15.102:0/3824061319 conn(0x7fce34065fb0 msgr2=0x7fce340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:18:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.294+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3824061319 shutdown_connections 2023-03-03T07:18:49.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.294+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3824061319 wait complete. 2023-03-03T07:18:49.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.295+0000 7fce3c5ff700 1 Processor -- start 2023-03-03T07:18:49.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.296+0000 7fce3c5ff700 1 -- start start 2023-03-03T07:18:49.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3c5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340cd8e0 0x7fce340cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:18:49.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3c5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce3414e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:18:49.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3c5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce341593c0 0x7fce3414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:18:49.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3c5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce340c6b10 con 0x7fce340cd8e0 2023-03-03T07:18:49.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3c5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce341ef8e0 con 0x7fce34153290 2023-03-03T07:18:49.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3c5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce341ef380 con 0x7fce341593c0 2023-03-03T07:18:49.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce39b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce3414e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:18:49.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3ab9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce341593c0 0x7fce3414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:18:49.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce3a39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340cd8e0 0x7fce340cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:18:49.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce39b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce3414e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57948/0 (socket says 172.21.15.102:57948) 2023-03-03T07:18:49.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.297+0000 7fce39b9a700 1 -- 172.21.15.102:0/3800493664 learned_addr learned my addr 172.21.15.102:0/3800493664 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:18:49.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce39b9a700 1 -- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce341593c0 msgr2=0x7fce3414e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:18:49.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce39b9a700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce341593c0 0x7fce3414e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:18:49.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce39b9a700 1 -- 172.21.15.102:0/3800493664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340cd8e0 msgr2=0x7fce340cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:18:49.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce39b9a700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340cd8e0 0x7fce340cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:18:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce39b9a700 1 -- 172.21.15.102:0/3800493664 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300094a0 con 0x7fce34153290 2023-03-03T07:18:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce39b9a700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce3414e140 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fce2400c550 tx=0x7fce2400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:18:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce2affd700 1 -- 172.21.15.102:0/3800493664 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24004c80 con 0x7fce34153290 2023-03-03T07:18:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce2affd700 1 -- 172.21.15.102:0/3800493664 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24004e00 con 0x7fce34153290 2023-03-03T07:18:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce2affd700 1 -- 172.21.15.102:0/3800493664 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24007310 con 0x7fce34153290 2023-03-03T07:18:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3415d160 con 0x7fce34153290 2023-03-03T07:18:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.298+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3406c7f0 con 0x7fce34153290 2023-03-03T07:18:49.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.300+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce340ccde0 con 0x7fce34153290 2023-03-03T07:18:49.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.301+0000 7fce2affd700 1 -- 172.21.15.102:0/3800493664 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fce2401a020 con 0x7fce34153290 2023-03-03T07:18:49.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.302+0000 7fce2affd700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fce18068320 0x7fce1806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:18:49.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.302+0000 7fce2affd700 1 -- 172.21.15.102:0/3800493664 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fce24085850 con 0x7fce34153290 2023-03-03T07:18:49.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.302+0000 7fce3a39b700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fce18068320 0x7fce1806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:18:49.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.303+0000 7fce3a39b700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fce18068320 0x7fce1806a7d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fce300090b0 tx=0x7fce30008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:18:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.306+0000 7fce2affd700 1 -- 172.21.15.102:0/3800493664 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce24051f60 con 0x7fce34153290 2023-03-03T07:18:49.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.544+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce34158810 con 0x7fce34153290 2023-03-03T07:18:49.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.545+0000 7fce2affd700 1 -- 172.21.15.102:0/3800493664 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fce24055580 con 0x7fce34153290 2023-03-03T07:18:49.547 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:18:49.547 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:18:49.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fce18068320 msgr2=0x7fce1806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:18:49.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fce18068320 0x7fce1806a7d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fce300090b0 tx=0x7fce30008030 comp rx=0 tx=0).stop 2023-03-03T07:18:49.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 msgr2=0x7fce3414e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:18:49.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce3414e140 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fce2400c550 tx=0x7fce2400c860 comp rx=0 tx=0).stop 2023-03-03T07:18:49.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 shutdown_connections 2023-03-03T07:18:49.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fce18068320 0x7fce1806a7d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:18:49.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fce341593c0 0x7fce3414e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:18:49.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce340cd8e0 0x7fce340cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:18:49.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 --2- 172.21.15.102:0/3800493664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce34153290 0x7fce3414e140 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:18:49.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 >> 172.21.15.102:0/3800493664 conn(0x7fce34065fb0 msgr2=0x7fce3415b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:18:49.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 shutdown_connections 2023-03-03T07:18:49.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:18:49.548+0000 7fce3c5ff700 1 -- 172.21.15.102:0/3800493664 wait complete. 2023-03-03T07:18:49.552 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:18:49.578 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 101 secs before thrashing 2023-03-03T07:19:01.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:19:01.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:19:31.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:19:31.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T07:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:20:02.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:02.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:30.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:20:30.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.846+0000 7f9c77bff700 1 Processor -- start 2023-03-03T07:20:30.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.846+0000 7f9c77bff700 1 -- start start 2023-03-03T07:20:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c77bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9c70153290 0x7f9c70158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:20:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c77bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c700ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:20:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c77bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c70153660 con 0x7f9c701593c0 2023-03-03T07:20:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c77bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7006c360 con 0x7f9c70153290 2023-03-03T07:20:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c77bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7015cea0 con 0x7f9c700cd8e0 2023-03-03T07:20:30.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c7599b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9c70153290 0x7f9c70158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:20:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c7599b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9c70153290 0x7f9c70158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38104/0 (socket says 172.21.15.102:38104) 2023-03-03T07:20:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c7599b700 1 -- 172.21.15.102:0/720136567 learned_addr learned my addr 172.21.15.102:0/720136567 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:20:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.848+0000 7f9c7619c700 1 --2- 172.21.15.102:0/720136567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c700ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:20:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c7599b700 1 -- 172.21.15.102:0/720136567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 msgr2=0x7f9c700ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:20:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c7599b700 1 --2- 172.21.15.102:0/720136567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c700ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c7599b700 1 -- 172.21.15.102:0/720136567 >> v1:172.21.15.102:6789/0 conn(0x7f9c701593c0 legacy=0x7f9c700cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:20:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c7599b700 1 -- 172.21.15.102:0/720136567 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7006c7f0 con 0x7f9c70153290 2023-03-03T07:20:30.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c7599b700 1 --2- 172.21.15.102:0/720136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c70158e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0098f0 tx=0x7f9c6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cf0248ba037635 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:20:30.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c74999700 1 -- 172.21.15.102:0/720136567 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c011830 con 0x7f9c70153290 2023-03-03T07:20:30.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c74999700 1 -- 172.21.15.102:0/720136567 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c011990 con 0x7f9c70153290 2023-03-03T07:20:30.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.849+0000 7f9c74999700 1 -- 172.21.15.102:0/720136567 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c011d20 con 0x7f9c70153290 2023-03-03T07:20:30.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 -- 172.21.15.102:0/720136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 msgr2=0x7f9c70158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:20:30.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 --2- 172.21.15.102:0/720136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c70158e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0098f0 tx=0x7f9c6c009c00 comp rx=0 tx=0).stop 2023-03-03T07:20:30.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 -- 172.21.15.102:0/720136567 shutdown_connections 2023-03-03T07:20:30.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 --2- 172.21.15.102:0/720136567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c700ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:30.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 --2- 172.21.15.102:0/720136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c70158e80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:30.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 -- 172.21.15.102:0/720136567 >> 172.21.15.102:0/720136567 conn(0x7f9c70065fb0 msgr2=0x7f9c700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:20:30.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 -- 172.21.15.102:0/720136567 shutdown_connections 2023-03-03T07:20:30.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.850+0000 7f9c77bff700 1 -- 172.21.15.102:0/720136567 wait complete. 2023-03-03T07:20:30.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.852+0000 7f9c77bff700 1 Processor -- start 2023-03-03T07:20:30.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.852+0000 7f9c77bff700 1 -- start start 2023-03-03T07:20:30.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.853+0000 7f9c77bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c70156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:20:30.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.853+0000 7f9c77bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c7015b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:20:30.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c77bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c701593c0 0x7f9c70150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:20:30.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c77bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700b44e0 con 0x7f9c70153290 2023-03-03T07:20:30.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c77bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701ee5d0 con 0x7f9c701593c0 2023-03-03T07:20:30.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c77bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701ee8b0 con 0x7f9c700cd8e0 2023-03-03T07:20:30.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c7015b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:20:30.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c7015b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38114/0 (socket says 172.21.15.102:38114) 2023-03-03T07:20:30.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 -- 172.21.15.102:0/2478165263 learned_addr learned my addr 172.21.15.102:0/2478165263 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:20:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7599b700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c70156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:20:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7619c700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c701593c0 0x7f9c70150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:20:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 -- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 msgr2=0x7f9c70156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:20:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c70156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 -- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c701593c0 msgr2=0x7f9c70150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:20:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c701593c0 0x7f9c70150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.854+0000 7f9c7519a700 1 -- 172.21.15.102:0/2478165263 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c009550 con 0x7f9c70153290 2023-03-03T07:20:30.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.855+0000 7f9c7519a700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c7015b010 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000c4e0 tx=0x7f9c6000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:20:30.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.855+0000 7f9c667fc700 1 -- 172.21.15.102:0/2478165263 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c600052d0 con 0x7f9c70153290 2023-03-03T07:20:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.855+0000 7f9c667fc700 1 -- 172.21.15.102:0/2478165263 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c60005430 con 0x7f9c70153290 2023-03-03T07:20:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.855+0000 7f9c667fc700 1 -- 172.21.15.102:0/2478165263 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c600038c0 con 0x7f9c70153290 2023-03-03T07:20:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.855+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c700ccde0 con 0x7f9c70153290 2023-03-03T07:20:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.855+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7015d160 con 0x7f9c70153290 2023-03-03T07:20:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.856+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c700cd0c0 con 0x7f9c70153290 2023-03-03T07:20:30.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.857+0000 7f9c667fc700 1 -- 172.21.15.102:0/2478165263 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c60019030 con 0x7f9c70153290 2023-03-03T07:20:30.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.859+0000 7f9c667fc700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c540682f0 0x7f9c5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:20:30.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.859+0000 7f9c667fc700 1 -- 172.21.15.102:0/2478165263 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9c60084fd0 con 0x7f9c70153290 2023-03-03T07:20:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.859+0000 7f9c7599b700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c540682f0 0x7f9c5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:20:30.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.860+0000 7f9c7599b700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c540682f0 0x7f9c5406a7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c014040 tx=0x7f9c6c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:20:30.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:30.864+0000 7f9c667fc700 1 -- 172.21.15.102:0/2478165263 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c600516e0 con 0x7f9c70153290 2023-03-03T07:20:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.098+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c7004e6e0 con 0x7f9c70153290 2023-03-03T07:20:31.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.099+0000 7f9c667fc700 1 -- 172.21.15.102:0/2478165263 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9c60017090 con 0x7f9c70153290 2023-03-03T07:20:31.101 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:20:31.102 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:20:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.101+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c540682f0 msgr2=0x7f9c5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:20:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.101+0000 7f9c77bff700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c540682f0 0x7f9c5406a7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c014040 tx=0x7f9c6c008030 comp rx=0 tx=0).stop 2023-03-03T07:20:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.101+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 msgr2=0x7f9c7015b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:20:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.101+0000 7f9c77bff700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c7015b010 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000c4e0 tx=0x7f9c6000c7f0 comp rx=0 tx=0).stop 2023-03-03T07:20:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 shutdown_connections 2023-03-03T07:20:31.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c540682f0 0x7f9c5406a7a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:31.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c700cd8e0 0x7f9c70156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:31.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c701593c0 0x7f9c70150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:31.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 --2- 172.21.15.102:0/2478165263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c70153290 0x7f9c7015b010 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:20:31.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 >> 172.21.15.102:0/2478165263 conn(0x7f9c70065fb0 msgr2=0x7f9c70156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:20:31.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 shutdown_connections 2023-03-03T07:20:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:20:31.102+0000 7f9c77bff700 1 -- 172.21.15.102:0/2478165263 wait complete. 2023-03-03T07:20:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:20:31.130 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T07:20:32.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:32.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:02.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:02.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:32.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:32.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:55.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:21:55.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.395+0000 7f3f883ff700 1 Processor -- start 2023-03-03T07:21:55.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.395+0000 7f3f883ff700 1 -- start start 2023-03-03T07:21:55.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f883ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3f800cd390 0x7f3f800ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:21:55.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f883ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f80153660 con 0x7f3f80153290 2023-03-03T07:21:55.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f883ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8006c360 con 0x7f3f800cd8e0 2023-03-03T07:21:55.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f883ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8015cea0 con 0x7f3f800cd390 2023-03-03T07:21:55.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f8699c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3f800cd390 0x7f3f800ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:21:55.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f8619b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3f80153290 0x7f3f80158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42262/0 (socket says 172.21.15.102:42262) 2023-03-03T07:21:55.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f8619b700 1 -- 172.21.15.102:0/3210517637 learned_addr learned my addr 172.21.15.102:0/3210517637 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:21:55.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f8599a700 1 --1- 172.21.15.102:0/3210517637 >> v1:172.21.15.172:6789/0 conn(0x7f3f800cd8e0 0x7f3f800cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52210/0 (socket says 172.21.15.102:52210) 2023-03-03T07:21:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f85199700 1 -- 172.21.15.102:0/3210517637 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952241557 0 0) 0x7f3f8006c360 con 0x7f3f800cd8e0 2023-03-03T07:21:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f85199700 1 -- 172.21.15.102:0/3210517637 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c0035b0 con 0x7f3f800cd8e0 2023-03-03T07:21:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.397+0000 7f3f85199700 1 -- 172.21.15.102:0/3210517637 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876614759 0 0) 0x7f3f80153660 con 0x7f3f80153290 2023-03-03T07:21:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f85199700 1 -- 172.21.15.102:0/3210517637 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8006c360 con 0x7f3f80153290 2023-03-03T07:21:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f8699c700 1 -- 172.21.15.102:0/3210517637 >> v1:172.21.15.172:6789/0 conn(0x7f3f800cd8e0 legacy=0x7f3f800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:21:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f8699c700 1 -- 172.21.15.102:0/3210517637 >> v1:172.21.15.102:6789/0 conn(0x7f3f80153290 legacy=0x7f3f80158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:21:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f8699c700 1 -- 172.21.15.102:0/3210517637 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8006c7f0 con 0x7f3f800cd390 2023-03-03T07:21:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f8699c700 1 --2- 172.21.15.102:0/3210517637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd390 0x7f3f800ce530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3f78009d90 tx=0x7f3f7800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=334a40d5d2a57b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:21:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f85199700 1 -- 172.21.15.102:0/3210517637 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800e070 con 0x7f3f800cd390 2023-03-03T07:21:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f85199700 1 -- 172.21.15.102:0/3210517637 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7800bb20 con 0x7f3f800cd390 2023-03-03T07:21:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.398+0000 7f3f85199700 1 -- 172.21.15.102:0/3210517637 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78012430 con 0x7f3f800cd390 2023-03-03T07:21:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.399+0000 7f3f883ff700 1 -- 172.21.15.102:0/3210517637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd390 msgr2=0x7f3f800ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:21:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.399+0000 7f3f883ff700 1 --2- 172.21.15.102:0/3210517637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd390 0x7f3f800ce530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3f78009d90 tx=0x7f3f7800b330 comp rx=0 tx=0).stop 2023-03-03T07:21:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.399+0000 7f3f883ff700 1 -- 172.21.15.102:0/3210517637 shutdown_connections 2023-03-03T07:21:55.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.399+0000 7f3f883ff700 1 --2- 172.21.15.102:0/3210517637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd390 0x7f3f800ce530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.399+0000 7f3f883ff700 1 -- 172.21.15.102:0/3210517637 >> 172.21.15.102:0/3210517637 conn(0x7f3f80065fb0 msgr2=0x7f3f800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:21:55.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.399+0000 7f3f883ff700 1 -- 172.21.15.102:0/3210517637 shutdown_connections 2023-03-03T07:21:55.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.399+0000 7f3f883ff700 1 -- 172.21.15.102:0/3210517637 wait complete. 2023-03-03T07:21:55.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.401+0000 7f3f883ff700 1 Processor -- start 2023-03-03T07:21:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.401+0000 7f3f883ff700 1 -- start start 2023-03-03T07:21:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.402+0000 7f3f883ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f800cd390 0x7f3f80153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:21:55.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.402+0000 7f3f883ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd8e0 0x7f3f800cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:21:55.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.402+0000 7f3f883ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f80153290 0x7f3f801f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:21:55.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.402+0000 7f3f883ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f800c5cf0 con 0x7f3f80153290 2023-03-03T07:21:55.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.402+0000 7f3f883ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f80156120 con 0x7f3f800cd8e0 2023-03-03T07:21:55.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.402+0000 7f3f883ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801e4790 con 0x7f3f800cd390 2023-03-03T07:21:55.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.402+0000 7f3f8699c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f80153290 0x7f3f801f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:21:55.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f800cd390 0x7f3f80153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:21:55.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8599a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd8e0 0x7f3f800cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:21:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8699c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f80153290 0x7f3f801f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50222/0 (socket says 172.21.15.102:50222) 2023-03-03T07:21:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8699c700 1 -- 172.21.15.102:0/443591077 learned_addr learned my addr 172.21.15.102:0/443591077 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:21:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f800cd390 0x7f3f80153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51712/0 (socket says 172.21.15.102:51712) 2023-03-03T07:21:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 -- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd8e0 msgr2=0x7f3f800cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:21:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd8e0 0x7f3f800cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 -- 172.21.15.102:0/443591077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f80153290 msgr2=0x7f3f801f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:21:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f80153290 0x7f3f801f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 -- 172.21.15.102:0/443591077 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f78009a70 con 0x7f3f800cd390 2023-03-03T07:21:55.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8699c700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f80153290 0x7f3f801f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:21:55.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.403+0000 7f3f8619b700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f800cd390 0x7f3f80153a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00c460 tx=0x7f3f7c00c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:21:55.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.404+0000 7f3f76ffd700 1 -- 172.21.15.102:0/443591077 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c005d70 con 0x7f3f800cd390 2023-03-03T07:21:55.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.404+0000 7f3f76ffd700 1 -- 172.21.15.102:0/443591077 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c0070d0 con 0x7f3f800cd390 2023-03-03T07:21:55.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.404+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f800ccde0 con 0x7f3f800cd390 2023-03-03T07:21:55.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.404+0000 7f3f76ffd700 1 -- 172.21.15.102:0/443591077 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c007460 con 0x7f3f800cd390 2023-03-03T07:21:55.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.404+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8015d160 con 0x7f3f800cd390 2023-03-03T07:21:55.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.405+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f800cd0c0 con 0x7f3f800cd390 2023-03-03T07:21:55.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.406+0000 7f3f76ffd700 1 -- 172.21.15.102:0/443591077 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f7c019030 con 0x7f3f800cd390 2023-03-03T07:21:55.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.407+0000 7f3f76ffd700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3f640682f0 0x7f3f6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:21:55.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.407+0000 7f3f76ffd700 1 -- 172.21.15.102:0/443591077 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3f7c084eb0 con 0x7f3f800cd390 2023-03-03T07:21:55.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.408+0000 7f3f8599a700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3f640682f0 0x7f3f6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:21:55.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.408+0000 7f3f8599a700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3f640682f0 0x7f3f6406a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3f800b4d50 tx=0x7f3f70007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:21:55.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.412+0000 7f3f76ffd700 1 -- 172.21.15.102:0/443591077 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f7c051530 con 0x7f3f800cd390 2023-03-03T07:21:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.645+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f800c5ef0 con 0x7f3f800cd390 2023-03-03T07:21:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.646+0000 7f3f76ffd700 1 -- 172.21.15.102:0/443591077 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3f7c017090 con 0x7f3f800cd390 2023-03-03T07:21:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:21:55.649 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:21:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3f640682f0 msgr2=0x7f3f6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:21:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3f640682f0 0x7f3f6406a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3f800b4d50 tx=0x7f3f70007de0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f800cd390 msgr2=0x7f3f80153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:21:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f800cd390 0x7f3f80153a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00c460 tx=0x7f3f7c00c770 comp rx=0 tx=0).stop 2023-03-03T07:21:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 shutdown_connections 2023-03-03T07:21:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3f640682f0 0x7f3f6406a7a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3f800cd390 0x7f3f80153a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f80153290 0x7f3f801f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 --2- 172.21.15.102:0/443591077 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f800cd8e0 0x7f3f800cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:21:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 >> 172.21.15.102:0/443591077 conn(0x7f3f80065fb0 msgr2=0x7f3f80158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:21:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 shutdown_connections 2023-03-03T07:21:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:21:55.649+0000 7f3f883ff700 1 -- 172.21.15.102:0/443591077 wait complete. 2023-03-03T07:21:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:21:55.678 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T07:22:02.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:02.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:33.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:33.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:33.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:22:33.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.926+0000 7fe218b9d700 1 Processor -- start 2023-03-03T07:22:33.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.926+0000 7fe218b9d700 1 -- start start 2023-03-03T07:22:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe218b9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2140b3970 0x7fe2140c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:22:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe218b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe2140c41b0 0x7fe2140cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:22:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe218b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2140c3d20 con 0x7fe2140ced00 2023-03-03T07:22:33.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe218b9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2140c4930 con 0x7fe2140b3970 2023-03-03T07:22:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe218b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2140cf610 con 0x7fe2140c41b0 2023-03-03T07:22:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe212ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2140b3970 0x7fe2140c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:22:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe2137fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe2140ced00 0x7fe214158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59426/0 (socket says 172.21.15.102:59426) 2023-03-03T07:22:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe2137fe700 1 -- 172.21.15.102:0/2859953814 learned_addr learned my addr 172.21.15.102:0/2859953814 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:22:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.928+0000 7fe213fff700 1 --2- 172.21.15.102:0/2859953814 >> v2:172.21.15.172:3300/0 conn(0x7fe2140c41b0 0x7fe2140cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:22:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe212ffd700 1 -- 172.21.15.102:0/2859953814 >> v2:172.21.15.172:3300/0 conn(0x7fe2140c41b0 msgr2=0x7fe2140cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:22:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe212ffd700 1 --2- 172.21.15.102:0/2859953814 >> v2:172.21.15.172:3300/0 conn(0x7fe2140c41b0 0x7fe2140cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe212ffd700 1 -- 172.21.15.102:0/2859953814 >> v1:172.21.15.172:6789/0 conn(0x7fe2140ced00 legacy=0x7fe214158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:22:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe212ffd700 1 -- 172.21.15.102:0/2859953814 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21415cba0 con 0x7fe2140b3970 2023-03-03T07:22:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe2127fc700 1 -- 172.21.15.102:0/2859953814 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801370815 0 0) 0x7fe2140c3d20 con 0x7fe2140ced00 2023-03-03T07:22:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe212ffd700 1 --2- 172.21.15.102:0/2859953814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2140c5a90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe21408ce30 tx=0x7fe208009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9bf1642f9e0ce1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:22:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe2127fc700 1 -- 172.21.15.102:0/2859953814 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208011830 con 0x7fe2140b3970 2023-03-03T07:22:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe2127fc700 1 -- 172.21.15.102:0/2859953814 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe208011990 con 0x7fe2140b3970 2023-03-03T07:22:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.929+0000 7fe2127fc700 1 -- 172.21.15.102:0/2859953814 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208011d20 con 0x7fe2140b3970 2023-03-03T07:22:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 -- 172.21.15.102:0/2859953814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 msgr2=0x7fe2140c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:22:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 --2- 172.21.15.102:0/2859953814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2140c5a90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe21408ce30 tx=0x7fe208009b50 comp rx=0 tx=0).stop 2023-03-03T07:22:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 -- 172.21.15.102:0/2859953814 shutdown_connections 2023-03-03T07:22:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 --2- 172.21.15.102:0/2859953814 >> v2:172.21.15.172:3300/0 conn(0x7fe2140c41b0 0x7fe2140cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 --2- 172.21.15.102:0/2859953814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2140c5a90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:33.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 -- 172.21.15.102:0/2859953814 >> 172.21.15.102:0/2859953814 conn(0x7fe21408b1c0 msgr2=0x7fe214071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:22:33.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 -- 172.21.15.102:0/2859953814 shutdown_connections 2023-03-03T07:22:33.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.930+0000 7fe218b9d700 1 -- 172.21.15.102:0/2859953814 wait complete. 2023-03-03T07:22:33.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.932+0000 7fe218b9d700 1 Processor -- start 2023-03-03T07:22:33.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.932+0000 7fe218b9d700 1 -- start start 2023-03-03T07:22:33.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.933+0000 7fe218b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2141e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:22:33.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.933+0000 7fe218b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2140c41b0 0x7fe214151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.933+0000 7fe218b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2140ced00 0x7fe2140ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.933+0000 7fe218b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2141e4e80 con 0x7fe2140b3970 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.933+0000 7fe218b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2141e3cc0 con 0x7fe2140c41b0 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.933+0000 7fe218b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2141f2710 con 0x7fe2140ced00 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.933+0000 7fe2137fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2141e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2141e4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35640/0 (socket says 172.21.15.102:35640) 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 -- 172.21.15.102:0/1672300476 learned_addr learned my addr 172.21.15.102:0/1672300476 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:22:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe213fff700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2140ced00 0x7fe2140ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:22:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe212ffd700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2140c41b0 0x7fe214151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:22:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 -- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2140ced00 msgr2=0x7fe2140ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:22:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2140ced00 0x7fe2140ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 -- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2140c41b0 msgr2=0x7fe214151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:22:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2140c41b0 0x7fe214151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 -- 172.21.15.102:0/1672300476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2080094a0 con 0x7fe2140b3970 2023-03-03T07:22:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe2137fe700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2141e4580 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe20400c750 tx=0x7fe20400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:22:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.934+0000 7fe1fbfff700 1 -- 172.21.15.102:0/1672300476 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204004c80 con 0x7fe2140b3970 2023-03-03T07:22:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.935+0000 7fe1fbfff700 1 -- 172.21.15.102:0/1672300476 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe204004e00 con 0x7fe2140b3970 2023-03-03T07:22:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.935+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21406bf20 con 0x7fe2140b3970 2023-03-03T07:22:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.935+0000 7fe1fbfff700 1 -- 172.21.15.102:0/1672300476 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204007310 con 0x7fe2140b3970 2023-03-03T07:22:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.935+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21415ce80 con 0x7fe2140b3970 2023-03-03T07:22:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.936+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21406c200 con 0x7fe2140b3970 2023-03-03T07:22:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.937+0000 7fe1fbfff700 1 -- 172.21.15.102:0/1672300476 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe20401a020 con 0x7fe2140b3970 2023-03-03T07:22:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.939+0000 7fe1fbfff700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2000682f0 0x7fe20006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:22:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.939+0000 7fe1fbfff700 1 -- 172.21.15.102:0/1672300476 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe2040858f0 con 0x7fe2140b3970 2023-03-03T07:22:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.940+0000 7fe212ffd700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2000682f0 0x7fe20006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:22:33.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.940+0000 7fe212ffd700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2000682f0 0x7fe20006a7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe2080097c0 tx=0x7fe20801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:22:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:33.944+0000 7fe1fbfff700 1 -- 172.21.15.102:0/1672300476 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe204051fa0 con 0x7fe2140b3970 2023-03-03T07:22:34.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.179+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2141e4c20 con 0x7fe2140b3970 2023-03-03T07:22:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.180+0000 7fe1fbfff700 1 -- 172.21.15.102:0/1672300476 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe2040555c0 con 0x7fe2140b3970 2023-03-03T07:22:34.181 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:22:34.181 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:22:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.182+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2000682f0 msgr2=0x7fe20006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:22:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.182+0000 7fe218b9d700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2000682f0 0x7fe20006a7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe2080097c0 tx=0x7fe20801c040 comp rx=0 tx=0).stop 2023-03-03T07:22:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.182+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 msgr2=0x7fe2141e4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:22:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.182+0000 7fe218b9d700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2141e4580 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe20400c750 tx=0x7fe20400ca60 comp rx=0 tx=0).stop 2023-03-03T07:22:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 shutdown_connections 2023-03-03T07:22:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2000682f0 0x7fe20006a7a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2140ced00 0x7fe2140ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2140b3970 0x7fe2141e4580 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 --2- 172.21.15.102:0/1672300476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2140c41b0 0x7fe214151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:22:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 >> 172.21.15.102:0/1672300476 conn(0x7fe21408b1c0 msgr2=0x7fe214071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:22:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 shutdown_connections 2023-03-03T07:22:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:22:34.183+0000 7fe218b9d700 1 -- 172.21.15.102:0/1672300476 wait complete. 2023-03-03T07:22:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:22:34.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-03-03T07:23:03.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:23:03.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:23:03.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:23:03.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.482+0000 7f2f2f5ff700 1 Processor -- start 2023-03-03T07:23:03.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.482+0000 7f2f2f5ff700 1 -- start start 2023-03-03T07:23:03.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2f5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f280ca130 0x7f2f2815c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:03.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2f5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2f280c8eb0 0x7f2f2815c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:03.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2f5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280c9590 con 0x7f2f28152e50 2023-03-03T07:23:03.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2f5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f28159880 con 0x7f2f280c8eb0 2023-03-03T07:23:03.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2f5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f280c7ae0 con 0x7f2f280ca130 2023-03-03T07:23:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2db9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2f280c8eb0 0x7f2f2815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2cb9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f280ca130 0x7f2f2815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2db9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2f280c8eb0 0x7f2f2815c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54638/0 (socket says 172.21.15.102:54638) 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.484+0000 7f2f2db9c700 1 -- 172.21.15.102:0/3030552498 learned_addr learned my addr 172.21.15.102:0/3030552498 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f2db9c700 1 -- 172.21.15.102:0/3030552498 >> v2:172.21.15.172:3300/0 conn(0x7f2f280ca130 msgr2=0x7f2f2815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f2db9c700 1 --2- 172.21.15.102:0/3030552498 >> v2:172.21.15.172:3300/0 conn(0x7f2f280ca130 0x7f2f2815c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f2db9c700 1 -- 172.21.15.102:0/3030552498 >> v1:172.21.15.172:6789/0 conn(0x7f2f28152e50 legacy=0x7f2f280c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f2db9c700 1 -- 172.21.15.102:0/3030552498 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280c7dc0 con 0x7f2f280c8eb0 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f2db9c700 1 --2- 172.21.15.102:0/3030552498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f2815c790 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2f280c7280 tx=0x7f2f2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1a5fe2cab2a61c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f1ffff700 1 -- 172.21.15.102:0/3030552498 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000e070 con 0x7f2f280c8eb0 2023-03-03T07:23:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f1ffff700 1 -- 172.21.15.102:0/3030552498 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2000b910 con 0x7f2f280c8eb0 2023-03-03T07:23:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.485+0000 7f2f1ffff700 1 -- 172.21.15.102:0/3030552498 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000bcc0 con 0x7f2f280c8eb0 2023-03-03T07:23:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3030552498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 msgr2=0x7f2f2815c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3030552498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f2815c790 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2f280c7280 tx=0x7f2f2000b330 comp rx=0 tx=0).stop 2023-03-03T07:23:03.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3030552498 shutdown_connections 2023-03-03T07:23:03.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3030552498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f2815c790 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3030552498 >> v2:172.21.15.172:3300/0 conn(0x7f2f280ca130 0x7f2f2815c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3030552498 >> 172.21.15.102:0/3030552498 conn(0x7f2f280704c0 msgr2=0x7f2f280708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:23:03.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3030552498 shutdown_connections 2023-03-03T07:23:03.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.486+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3030552498 wait complete. 2023-03-03T07:23:03.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.488+0000 7f2f2f5ff700 1 Processor -- start 2023-03-03T07:23:03.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.488+0000 7f2f2f5ff700 1 -- start start 2023-03-03T07:23:03.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.489+0000 7f2f2f5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f280b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:03.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.489+0000 7f2f2f5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f280ca130 0x7f2f280b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:03.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.489+0000 7f2f2f5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f28152e50 0x7f2f281eece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:03.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.489+0000 7f2f2f5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f281ee830 con 0x7f2f280c8eb0 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.489+0000 7f2f2f5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280b1700 con 0x7f2f280ca130 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.489+0000 7f2f2f5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f281e5160 con 0x7f2f28152e50 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.489+0000 7f2f2d39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f280b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f280b10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54652/0 (socket says 172.21.15.102:54652) 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 -- 172.21.15.102:0/3415718000 learned_addr learned my addr 172.21.15.102:0/3415718000 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2cb9a700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f280ca130 0x7f2f280b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2db9c700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f28152e50 0x7f2f281eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 -- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f28152e50 msgr2=0x7f2f281eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:03.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f28152e50 0x7f2f281eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 -- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f280ca130 msgr2=0x7f2f280b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:03.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f280ca130 0x7f2f280b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 -- 172.21.15.102:0/3415718000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20009b30 con 0x7f2f280c8eb0 2023-03-03T07:23:03.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.490+0000 7f2f2d39b700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f280b10a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400c450 tx=0x7f2f2400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:23:03.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.491+0000 7f2f1dffb700 1 -- 172.21.15.102:0/3415718000 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f240052d0 con 0x7f2f280c8eb0 2023-03-03T07:23:03.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.491+0000 7f2f1dffb700 1 -- 172.21.15.102:0/3415718000 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f24005430 con 0x7f2f280c8eb0 2023-03-03T07:23:03.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.491+0000 7f2f1dffb700 1 -- 172.21.15.102:0/3415718000 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f240038c0 con 0x7f2f280c8eb0 2023-03-03T07:23:03.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.491+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f28159b60 con 0x7f2f280c8eb0 2023-03-03T07:23:03.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.491+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f280c80a0 con 0x7f2f280c8eb0 2023-03-03T07:23:03.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.492+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f280c7690 con 0x7f2f280c8eb0 2023-03-03T07:23:03.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.493+0000 7f2f1dffb700 1 -- 172.21.15.102:0/3415718000 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f24019030 con 0x7f2f280c8eb0 2023-03-03T07:23:03.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.494+0000 7f2f1dffb700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f0c0682f0 0x7f2f0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:03.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.494+0000 7f2f1dffb700 1 -- 172.21.15.102:0/3415718000 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2f24084f80 con 0x7f2f280c8eb0 2023-03-03T07:23:03.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.495+0000 7f2f2cb9a700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f0c0682f0 0x7f2f0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:03.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.495+0000 7f2f2cb9a700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f0c0682f0 0x7f2f0c06a7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2f28150ab0 tx=0x7f2f18009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:23:03.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.498+0000 7f2f1dffb700 1 -- 172.21.15.102:0/3415718000 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f24051690 con 0x7f2f280c8eb0 2023-03-03T07:23:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.737+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f2804e6e0 con 0x7f2f280c8eb0 2023-03-03T07:23:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.737+0000 7f2f1dffb700 1 -- 172.21.15.102:0/3415718000 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2f24017090 con 0x7f2f280c8eb0 2023-03-03T07:23:03.739 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:23:03.739 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:23:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f0c0682f0 msgr2=0x7f2f0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f0c0682f0 0x7f2f0c06a7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2f28150ab0 tx=0x7f2f18009210 comp rx=0 tx=0).stop 2023-03-03T07:23:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 msgr2=0x7f2f280b10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f280b10a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400c450 tx=0x7f2f2400c760 comp rx=0 tx=0).stop 2023-03-03T07:23:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 shutdown_connections 2023-03-03T07:23:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f0c0682f0 0x7f2f0c06a7a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f28152e50 0x7f2f281eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f280c8eb0 0x7f2f280b10a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 --2- 172.21.15.102:0/3415718000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f280ca130 0x7f2f280b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 >> 172.21.15.102:0/3415718000 conn(0x7f2f280704c0 msgr2=0x7f2f2814e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:23:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 shutdown_connections 2023-03-03T07:23:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:03.741+0000 7f2f2f5ff700 1 -- 172.21.15.102:0/3415718000 wait complete. 2023-03-03T07:23:03.744 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:23:03.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before thrashing 2023-03-03T07:23:04.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:23:05.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.017+0000 7f49e8d9d700 1 Processor -- start 2023-03-03T07:23:05.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.017+0000 7f49e8d9d700 1 -- start start 2023-03-03T07:23:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.018+0000 7f49e8d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40c9610 0x7f49e40ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e8d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40c9ca0 con 0x7f49e40ca980 2023-03-03T07:23:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e8d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40b46f0 con 0x7f49e41564b0 2023-03-03T07:23:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e8d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e40cadc0 con 0x7f49e40c9610 2023-03-03T07:23:05.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e37fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f49e41564b0 0x7f49e40cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:43034/0 (socket says 172.21.15.102:43034) 2023-03-03T07:23:05.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e37fe700 1 -- 172.21.15.102:0/1506334260 learned_addr learned my addr 172.21.15.102:0/1506334260 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:23:05.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e2ffd700 1 --2- 172.21.15.102:0/1506334260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40c9610 0x7f49e40ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:05.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e3fff700 1 --1- 172.21.15.102:0/1506334260 >> v1:172.21.15.102:6789/0 conn(0x7f49e40ca980 0x7f49e41636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58022/0 (socket says 172.21.15.102:58022) 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e27fc700 1 -- 172.21.15.102:0/1506334260 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233030611 0 0) 0x7f49e40b46f0 con 0x7f49e41564b0 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e27fc700 1 -- 172.21.15.102:0/1506334260 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc0035b0 con 0x7f49e41564b0 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e2ffd700 1 -- 172.21.15.102:0/1506334260 >> v1:172.21.15.172:6789/0 conn(0x7f49e41564b0 legacy=0x7f49e40cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e2ffd700 1 -- 172.21.15.102:0/1506334260 >> v1:172.21.15.102:6789/0 conn(0x7f49e40ca980 legacy=0x7f49e41636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.019+0000 7f49e27fc700 1 -- 172.21.15.102:0/1506334260 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618028692 0 0) 0x7f49e40c9ca0 con 0x7f49e40ca980 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.020+0000 7f49e2ffd700 1 -- 172.21.15.102:0/1506334260 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e40b4920 con 0x7f49e40c9610 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.020+0000 7f49e2ffd700 1 --2- 172.21.15.102:0/1506334260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40c9610 0x7f49e40ca440 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009870 tx=0x7f49d8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e34d0a8820c53e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:23:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.020+0000 7f49e27fc700 1 -- 172.21.15.102:0/1506334260 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8011830 con 0x7f49e40c9610 2023-03-03T07:23:05.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.020+0000 7f49e27fc700 1 -- 172.21.15.102:0/1506334260 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d8011990 con 0x7f49e40c9610 2023-03-03T07:23:05.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.020+0000 7f49e27fc700 1 -- 172.21.15.102:0/1506334260 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8011d00 con 0x7f49e40c9610 2023-03-03T07:23:05.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.021+0000 7f49e8d9d700 1 -- 172.21.15.102:0/1506334260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40c9610 msgr2=0x7f49e40ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.021+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/1506334260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40c9610 0x7f49e40ca440 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009870 tx=0x7f49d8009b80 comp rx=0 tx=0).stop 2023-03-03T07:23:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.021+0000 7f49e8d9d700 1 -- 172.21.15.102:0/1506334260 shutdown_connections 2023-03-03T07:23:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.021+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/1506334260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40c9610 0x7f49e40ca440 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.021+0000 7f49e8d9d700 1 -- 172.21.15.102:0/1506334260 >> 172.21.15.102:0/1506334260 conn(0x7f49e4065fb0 msgr2=0x7f49e40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:23:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.021+0000 7f49e8d9d700 1 -- 172.21.15.102:0/1506334260 shutdown_connections 2023-03-03T07:23:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.021+0000 7f49e8d9d700 1 -- 172.21.15.102:0/1506334260 wait complete. 2023-03-03T07:23:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.023+0000 7f49e8d9d700 1 Processor -- start 2023-03-03T07:23:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.023+0000 7f49e8d9d700 1 -- start start 2023-03-03T07:23:05.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e8d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40c9610 0x7f49e41f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:05.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e8d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40ca980 0x7f49e415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:05.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e8d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49e41564b0 0x7f49e415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:05.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e8d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e415b1a0 con 0x7f49e40c9610 2023-03-03T07:23:05.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e8d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e414fc40 con 0x7f49e41564b0 2023-03-03T07:23:05.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e8d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e415a070 con 0x7f49e40ca980 2023-03-03T07:23:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40c9610 0x7f49e41f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40c9610 0x7f49e41f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54666/0 (socket says 172.21.15.102:54666) 2023-03-03T07:23:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e37fe700 1 -- 172.21.15.102:0/2753594681 learned_addr learned my addr 172.21.15.102:0/2753594681 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:23:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49e41564b0 0x7f49e415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.024+0000 7f49e2ffd700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40ca980 0x7f49e415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49e37fe700 1 -- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40ca980 msgr2=0x7f49e415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49e37fe700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40ca980 0x7f49e415bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49e37fe700 1 -- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49e41564b0 msgr2=0x7f49e415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49e37fe700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49e41564b0 0x7f49e415cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49e37fe700 1 -- 172.21.15.102:0/2753594681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d80094a0 con 0x7f49e40c9610 2023-03-03T07:23:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49e3fff700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49e41564b0 0x7f49e415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:23:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49e37fe700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40c9610 0x7f49e41f0fe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f49d400c500 tx=0x7f49d400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:23:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.025+0000 7f49cbfff700 1 -- 172.21.15.102:0/2753594681 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4005d70 con 0x7f49e40c9610 2023-03-03T07:23:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.026+0000 7f49cbfff700 1 -- 172.21.15.102:0/2753594681 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d40070d0 con 0x7f49e40c9610 2023-03-03T07:23:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.026+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e40b1f30 con 0x7f49e40c9610 2023-03-03T07:23:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.026+0000 7f49cbfff700 1 -- 172.21.15.102:0/2753594681 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4007480 con 0x7f49e40c9610 2023-03-03T07:23:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.026+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e40b4c00 con 0x7f49e40c9610 2023-03-03T07:23:05.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.027+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e40b2210 con 0x7f49e40c9610 2023-03-03T07:23:05.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.028+0000 7f49cbfff700 1 -- 172.21.15.102:0/2753594681 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f49d4019030 con 0x7f49e40c9610 2023-03-03T07:23:05.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.029+0000 7f49cbfff700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49d00682f0 0x7f49d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:23:05.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.029+0000 7f49cbfff700 1 -- 172.21.15.102:0/2753594681 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f49d40850a0 con 0x7f49e40c9610 2023-03-03T07:23:05.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.029+0000 7f49e2ffd700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49d00682f0 0x7f49d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:23:05.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.030+0000 7f49e2ffd700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49d00682f0 0x7f49d006a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f49d80090b0 tx=0x7f49d8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:23:05.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.033+0000 7f49cbfff700 1 -- 172.21.15.102:0/2753594681 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49d40516c0 con 0x7f49e40c9610 2023-03-03T07:23:05.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.268+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49e40677f0 con 0x7f49e40c9610 2023-03-03T07:23:05.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.269+0000 7f49cbfff700 1 -- 172.21.15.102:0/2753594681 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f49d4017070 con 0x7f49e40c9610 2023-03-03T07:23:05.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:23:05.270 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:23:05.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49d00682f0 msgr2=0x7f49d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:05.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49d00682f0 0x7f49d006a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f49d80090b0 tx=0x7f49d8008030 comp rx=0 tx=0).stop 2023-03-03T07:23:05.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40c9610 msgr2=0x7f49e41f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:23:05.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40c9610 0x7f49e41f0fe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f49d400c500 tx=0x7f49d400c810 comp rx=0 tx=0).stop 2023-03-03T07:23:05.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 shutdown_connections 2023-03-03T07:23:05.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49e41564b0 0x7f49e415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:05.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40c9610 0x7f49e41f0fe0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:05.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49d00682f0 0x7f49d006a7a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:05.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 --2- 172.21.15.102:0/2753594681 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49e40ca980 0x7f49e415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:23:05.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.271+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 >> 172.21.15.102:0/2753594681 conn(0x7f49e4065fb0 msgr2=0x7f49e40cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:23:05.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.272+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 shutdown_connections 2023-03-03T07:23:05.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:23:05.272+0000 7f49e8d9d700 1 -- 172.21.15.102:0/2753594681 wait complete. 2023-03-03T07:23:05.274 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:23:05.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T07:23:33.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:23:33.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:03.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:03.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:05.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:24:05.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.559+0000 7f77ecbff700 1 Processor -- start 2023-03-03T07:24:05.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.559+0000 7f77ecbff700 1 -- start start 2023-03-03T07:24:05.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.560+0000 7f77ecbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77e8153260 0x7f77e80ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:24:05.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.560+0000 7f77ecbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80c9cb0 0x7f77e80cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:24:05.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77ecbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f77e80cac20 0x7f77e80cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:24:05.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77ecbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e80cfe70 con 0x7f77e80cac20 2023-03-03T07:24:05.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77ecbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e8158d70 con 0x7f77e8153260 2023-03-03T07:24:05.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77ecbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80cf940 con 0x7f77e80c9cb0 2023-03-03T07:24:05.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77e8153260 0x7f77e80ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:24:05.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77e8153260 0x7f77e80ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42168/0 (socket says 172.21.15.102:42168) 2023-03-03T07:24:05.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 -- 172.21.15.102:0/30136362 learned_addr learned my addr 172.21.15.102:0/30136362 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:24:05.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77dffff700 1 --2- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80c9cb0 0x7f77e80cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:24:05.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e6d9d700 1 --2- 172.21.15.102:0/30136362 >> v2:172.21.15.102:3300/0 conn(0x7f77e80cac20 0x7f77e80cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:24:05.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 -- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80c9cb0 msgr2=0x7f77e80cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:24:05.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 --2- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80c9cb0 0x7f77e80cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 -- 172.21.15.102:0/30136362 >> v2:172.21.15.102:3300/0 conn(0x7f77e80cac20 msgr2=0x7f77e80cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:24:05.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 --2- 172.21.15.102:0/30136362 >> v2:172.21.15.102:3300/0 conn(0x7f77e80cac20 0x7f77e80cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.561+0000 7f77e659c700 1 -- 172.21.15.102:0/30136362 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e80cfb90 con 0x7f77e8153260 2023-03-03T07:24:05.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.562+0000 7f77e659c700 1 --2- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8153260 0x7f77e80ca6e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f77e808cc70 tx=0x7f77d8009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0d73dfee2f47d5c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:24:05.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.562+0000 7f77e5d9b700 1 -- 172.21.15.102:0/30136362 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8011830 con 0x7f77e8153260 2023-03-03T07:24:05.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.562+0000 7f77e5d9b700 1 -- 172.21.15.102:0/30136362 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d8011990 con 0x7f77e8153260 2023-03-03T07:24:05.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.562+0000 7f77e5d9b700 1 -- 172.21.15.102:0/30136362 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8011d20 con 0x7f77e8153260 2023-03-03T07:24:05.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.562+0000 7f77ecbff700 1 -- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8153260 msgr2=0x7f77e80ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:24:05.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.562+0000 7f77ecbff700 1 --2- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8153260 0x7f77e80ca6e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f77e808cc70 tx=0x7f77d8009bd0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.563+0000 7f77ecbff700 1 -- 172.21.15.102:0/30136362 shutdown_connections 2023-03-03T07:24:05.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.563+0000 7f77ecbff700 1 --2- 172.21.15.102:0/30136362 >> v2:172.21.15.102:3300/0 conn(0x7f77e80cac20 0x7f77e80cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.563+0000 7f77ecbff700 1 --2- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8153260 0x7f77e80ca6e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.563+0000 7f77ecbff700 1 --2- 172.21.15.102:0/30136362 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80c9cb0 0x7f77e80cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.563+0000 7f77ecbff700 1 -- 172.21.15.102:0/30136362 >> 172.21.15.102:0/30136362 conn(0x7f77e8066c00 msgr2=0x7f77e80ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:24:05.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.563+0000 7f77ecbff700 1 -- 172.21.15.102:0/30136362 shutdown_connections 2023-03-03T07:24:05.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.563+0000 7f77ecbff700 1 -- 172.21.15.102:0/30136362 wait complete. 2023-03-03T07:24:05.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.565+0000 7f77ecbff700 1 Processor -- start 2023-03-03T07:24:05.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.566+0000 7f77ecbff700 1 -- start start 2023-03-03T07:24:05.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.567+0000 7f77ecbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80cac20 0x7f77e81f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:24:05.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.567+0000 7f77ecbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e8153260 0x7f77e80c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:24:05.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.567+0000 7f77ecbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8157ab0 0x7f77e80c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:24:05.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.567+0000 7f77ecbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80c6530 con 0x7f77e8153260 2023-03-03T07:24:05.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.567+0000 7f77ecbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e81f29c0 con 0x7f77e8157ab0 2023-03-03T07:24:05.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.567+0000 7f77ecbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80c6740 con 0x7f77e80cac20 2023-03-03T07:24:05.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.567+0000 7f77e659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80cac20 0x7f77e81f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:24:05.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80cac20 0x7f77e81f1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42404/0 (socket says 172.21.15.102:42404) 2023-03-03T07:24:05.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 -- 172.21.15.102:0/3690124788 learned_addr learned my addr 172.21.15.102:0/3690124788 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:24:05.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e6d9d700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8157ab0 0x7f77e80c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:24:05.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77dffff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e8153260 0x7f77e80c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:24:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 -- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8157ab0 msgr2=0x7f77e80c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:24:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8157ab0 0x7f77e80c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 -- 172.21.15.102:0/3690124788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e8153260 msgr2=0x7f77e80c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:24:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e8153260 0x7f77e80c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 -- 172.21.15.102:0/3690124788 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8009550 con 0x7f77e80cac20 2023-03-03T07:24:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77dffff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e8153260 0x7f77e80c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:24:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77e659c700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80cac20 0x7f77e81f1f50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f77d8000c00 tx=0x7f77d8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:24:05.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.568+0000 7f77deffd700 1 -- 172.21.15.102:0/3690124788 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8004300 con 0x7f77e80cac20 2023-03-03T07:24:05.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.569+0000 7f77deffd700 1 -- 172.21.15.102:0/3690124788 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d801f070 con 0x7f77e80cac20 2023-03-03T07:24:05.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.569+0000 7f77deffd700 1 -- 172.21.15.102:0/3690124788 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d80036a0 con 0x7f77e80cac20 2023-03-03T07:24:05.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.569+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e8159260 con 0x7f77e80cac20 2023-03-03T07:24:05.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.569+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e8158f80 con 0x7f77e80cac20 2023-03-03T07:24:05.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.570+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e8159540 con 0x7f77e80cac20 2023-03-03T07:24:05.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.571+0000 7f77deffd700 1 -- 172.21.15.102:0/3690124788 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f77d802a020 con 0x7f77e80cac20 2023-03-03T07:24:05.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.572+0000 7f77deffd700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77c80682f0 0x7f77c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:24:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.572+0000 7f77deffd700 1 -- 172.21.15.102:0/3690124788 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f77d80870f0 con 0x7f77e80cac20 2023-03-03T07:24:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.573+0000 7f77dffff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77c80682f0 0x7f77c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:24:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.573+0000 7f77dffff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77c80682f0 0x7f77c806a7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f77e81521d0 tx=0x7f77d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:24:05.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.577+0000 7f77deffd700 1 -- 172.21.15.102:0/3690124788 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f77d8053800 con 0x7f77e80cac20 2023-03-03T07:24:05.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.811+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77e8073dd0 con 0x7f77e80cac20 2023-03-03T07:24:05.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.812+0000 7f77deffd700 1 -- 172.21.15.102:0/3690124788 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f77d8056e20 con 0x7f77e80cac20 2023-03-03T07:24:05.813 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:24:05.813 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:24:05.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77c80682f0 msgr2=0x7f77c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:24:05.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77c80682f0 0x7f77c806a7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f77e81521d0 tx=0x7f77d0008040 comp rx=0 tx=0).stop 2023-03-03T07:24:05.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80cac20 msgr2=0x7f77e81f1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:24:05.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80cac20 0x7f77e81f1f50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f77d8000c00 tx=0x7f77d8004750 comp rx=0 tx=0).stop 2023-03-03T07:24:05.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 shutdown_connections 2023-03-03T07:24:05.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77c80682f0 0x7f77c806a7a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77e80cac20 0x7f77e81f1f50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e8157ab0 0x7f77e80c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 --2- 172.21.15.102:0/3690124788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e8153260 0x7f77e80c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:24:05.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.814+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 >> 172.21.15.102:0/3690124788 conn(0x7f77e80c9cb0 msgr2=0x7f77e80cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:24:05.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.815+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 shutdown_connections 2023-03-03T07:24:05.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:24:05.815+0000 7f77ecbff700 1 -- 172.21.15.102:0/3690124788 wait complete. 2023-03-03T07:24:05.818 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:24:05.843 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2023-03-03T07:24:33.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:33.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:02.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:25:03.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.095+0000 7f6302bff700 1 Processor -- start 2023-03-03T07:25:03.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.095+0000 7f6302bff700 1 -- start start 2023-03-03T07:25:03.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.096+0000 7f6302bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc153290 0x7f62fc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:03.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f6302bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f62fc0cd8e0 0x7f62fc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:03.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f6302bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc153660 con 0x7f62fc1593c0 2023-03-03T07:25:03.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f6302bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc06c360 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f6302bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc15cea0 con 0x7f62fc153290 2023-03-03T07:25:03.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f630099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc153290 0x7f62fc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:03.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f630119c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f62fc0cd8e0 0x7f62fc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:03.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f62fbfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f62fc1593c0 0x7f62fc0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49354/0 (socket says 172.21.15.102:49354) 2023-03-03T07:25:03.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f630099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc153290 0x7f62fc158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46130/0 (socket says 172.21.15.102:46130) 2023-03-03T07:25:03.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f62fbfff700 1 -- 172.21.15.102:0/2229009099 learned_addr learned my addr 172.21.15.102:0/2229009099 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:25:03.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.097+0000 7f62fb7fe700 1 -- 172.21.15.102:0/2229009099 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595351686 0 0) 0x7f62fc153660 con 0x7f62fc1593c0 2023-03-03T07:25:03.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f62fb7fe700 1 -- 172.21.15.102:0/2229009099 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec0035b0 con 0x7f62fc1593c0 2023-03-03T07:25:03.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f630119c700 1 -- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc153290 msgr2=0x7f62fc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:25:03.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f630119c700 1 --2- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc153290 0x7f62fc158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f630119c700 1 -- 172.21.15.102:0/2229009099 >> v1:172.21.15.172:6789/0 conn(0x7f62fc1593c0 legacy=0x7f62fc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:03.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f630119c700 1 -- 172.21.15.102:0/2229009099 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc06c7f0 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f630099b700 1 --2- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc153290 0x7f62fc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:25:03.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f630119c700 1 --2- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc0cd8e0 0x7f62fc0ce530 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009d90 tx=0x7f62f400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45158d117f6c0905 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:25:03.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f62fb7fe700 1 -- 172.21.15.102:0/2229009099 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f400e070 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f62fb7fe700 1 -- 172.21.15.102:0/2229009099 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f400bb20 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.098+0000 7f62fb7fe700 1 -- 172.21.15.102:0/2229009099 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4012430 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 -- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc0cd8e0 msgr2=0x7f62fc0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 --2- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc0cd8e0 0x7f62fc0ce530 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009d90 tx=0x7f62f400b330 comp rx=0 tx=0).stop 2023-03-03T07:25:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 -- 172.21.15.102:0/2229009099 shutdown_connections 2023-03-03T07:25:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 --2- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc0cd8e0 0x7f62fc0ce530 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 --2- 172.21.15.102:0/2229009099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc153290 0x7f62fc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 -- 172.21.15.102:0/2229009099 >> 172.21.15.102:0/2229009099 conn(0x7f62fc065fb0 msgr2=0x7f62fc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:25:03.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 -- 172.21.15.102:0/2229009099 shutdown_connections 2023-03-03T07:25:03.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.099+0000 7f6302bff700 1 -- 172.21.15.102:0/2229009099 wait complete. 2023-03-03T07:25:03.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.101+0000 7f6302bff700 1 Processor -- start 2023-03-03T07:25:03.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.101+0000 7f6302bff700 1 -- start start 2023-03-03T07:25:03.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.102+0000 7f6302bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc0cd8e0 0x7f62fc0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:03.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.102+0000 7f6302bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc153290 0x7f62fc153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:03.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.102+0000 7f6302bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62fc1593c0 0x7f62fc0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:03.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.102+0000 7f6302bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0c3d20 con 0x7f62fc1593c0 2023-03-03T07:25:03.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.102+0000 7f6302bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0c8380 con 0x7f62fc153290 2023-03-03T07:25:03.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.102+0000 7f6302bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0c8660 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc0cd8e0 0x7f62fc0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f62fbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc153290 0x7f62fc153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc0cd8e0 0x7f62fc0c7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46144/0 (socket says 172.21.15.102:46144) 2023-03-03T07:25:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 -- 172.21.15.102:0/3530924789 learned_addr learned my addr 172.21.15.102:0/3530924789 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:25:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630119c700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62fc1593c0 0x7f62fc0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 -- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc153290 msgr2=0x7f62fc153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc153290 0x7f62fc153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 -- 172.21.15.102:0/3530924789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62fc1593c0 msgr2=0x7f62fc0c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62fc1593c0 0x7f62fc0c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 -- 172.21.15.102:0/3530924789 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f00094a0 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f62fbfff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc153290 0x7f62fc153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:25:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.103+0000 7f630099b700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc0cd8e0 0x7f62fc0c7cd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62f000ebe0 tx=0x7f62f000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:25:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.104+0000 7f62f97fa700 1 -- 172.21.15.102:0/3530924789 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f000c850 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.104+0000 7f62f97fa700 1 -- 172.21.15.102:0/3530924789 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f000c9b0 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.104+0000 7f62f97fa700 1 -- 172.21.15.102:0/3530924789 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f000cd40 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.104+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f4009a70 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.104+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc0ccde0 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.105+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc15d160 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.106+0000 7f62f97fa700 1 -- 172.21.15.102:0/3530924789 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f62f0019030 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.108+0000 7f62f97fa700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62e00682f0 0x7f62e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.108+0000 7f62f97fa700 1 -- 172.21.15.102:0/3530924789 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f62f0085da0 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.108+0000 7f62fbfff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62e00682f0 0x7f62e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:03.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.108+0000 7f62fbfff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62e00682f0 0x7f62e006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f62fc1581d0 tx=0x7f62e8007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:25:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.112+0000 7f62f97fa700 1 -- 172.21.15.102:0/3530924789 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f62f0052420 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.349+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62fc0cd0c0 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.350+0000 7f62f97fa700 1 -- 172.21.15.102:0/3530924789 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f62f0017090 con 0x7f62fc0cd8e0 2023-03-03T07:25:03.351 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:25:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:25:03.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62e00682f0 msgr2=0x7f62e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:03.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62e00682f0 0x7f62e006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f62fc1581d0 tx=0x7f62e8007dc0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc0cd8e0 msgr2=0x7f62fc0c7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:03.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc0cd8e0 0x7f62fc0c7cd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62f000ebe0 tx=0x7f62f000eef0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 shutdown_connections 2023-03-03T07:25:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62e00682f0 0x7f62e006a7a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62fc0cd8e0 0x7f62fc0c7cd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62fc1593c0 0x7f62fc0c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 --2- 172.21.15.102:0/3530924789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62fc153290 0x7f62fc153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 >> 172.21.15.102:0/3530924789 conn(0x7f62fc065fb0 msgr2=0x7f62fc15be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:25:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 shutdown_connections 2023-03-03T07:25:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:03.353+0000 7f6302bff700 1 -- 172.21.15.102:0/3530924789 wait complete. 2023-03-03T07:25:03.356 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:25:03.382 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T07:25:04.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:04.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:34.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:34.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:51.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:25:51.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.635+0000 7f320759e700 1 Processor -- start 2023-03-03T07:25:51.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.635+0000 7f320759e700 1 -- start start 2023-03-03T07:25:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320759e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32080ced00 0x7f32080c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320759e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32080b3970 0x7f32080c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320759e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32080c3d20 con 0x7f32080cf0d0 2023-03-03T07:25:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320759e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32080c4930 con 0x7f32080ced00 2023-03-03T07:25:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320759e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32080cf570 con 0x7f32080b3970 2023-03-03T07:25:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32080ced00 0x7f32080c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32080ced00 0x7f32080c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47182/0 (socket says 172.21.15.102:47182) 2023-03-03T07:25:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f320659c700 1 -- 172.21.15.102:0/1979774275 learned_addr learned my addr 172.21.15.102:0/1979774275 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:25:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.637+0000 7f3205d9b700 1 --2- 172.21.15.102:0/1979774275 >> v2:172.21.15.172:3300/0 conn(0x7f32080b3970 0x7f32080c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320659c700 1 -- 172.21.15.102:0/1979774275 >> v2:172.21.15.172:3300/0 conn(0x7f32080b3970 msgr2=0x7f32080c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:25:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320659c700 1 --2- 172.21.15.102:0/1979774275 >> v2:172.21.15.172:3300/0 conn(0x7f32080b3970 0x7f32080c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320659c700 1 -- 172.21.15.102:0/1979774275 >> v1:172.21.15.172:6789/0 conn(0x7f32080cf0d0 legacy=0x7f3208158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320659c700 1 -- 172.21.15.102:0/1979774275 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320815cba0 con 0x7f32080ced00 2023-03-03T07:25:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320559a700 1 -- 172.21.15.102:0/1979774275 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537187445 0 0) 0x7f32080c3d20 con 0x7f32080cf0d0 2023-03-03T07:25:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320659c700 1 --2- 172.21.15.102:0/1979774275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c5a90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f32080c4ba0 tx=0x7f31f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57713a10b772dc35 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:25:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320559a700 1 -- 172.21.15.102:0/1979774275 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8011830 con 0x7f32080ced00 2023-03-03T07:25:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320559a700 1 -- 172.21.15.102:0/1979774275 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f8011990 con 0x7f32080ced00 2023-03-03T07:25:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.638+0000 7f320559a700 1 -- 172.21.15.102:0/1979774275 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8011d20 con 0x7f32080ced00 2023-03-03T07:25:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 -- 172.21.15.102:0/1979774275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 msgr2=0x7f32080c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 --2- 172.21.15.102:0/1979774275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c5a90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f32080c4ba0 tx=0x7f31f8009bd0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 -- 172.21.15.102:0/1979774275 shutdown_connections 2023-03-03T07:25:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 --2- 172.21.15.102:0/1979774275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c5a90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 --2- 172.21.15.102:0/1979774275 >> v2:172.21.15.172:3300/0 conn(0x7f32080b3970 0x7f32080c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 -- 172.21.15.102:0/1979774275 >> 172.21.15.102:0/1979774275 conn(0x7f320808b1c0 msgr2=0x7f3208071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:25:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 -- 172.21.15.102:0/1979774275 shutdown_connections 2023-03-03T07:25:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.639+0000 7f320759e700 1 -- 172.21.15.102:0/1979774275 wait complete. 2023-03-03T07:25:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.642+0000 7f320759e700 1 Processor -- start 2023-03-03T07:25:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.642+0000 7f320759e700 1 -- start start 2023-03-03T07:25:51.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32080b3970 0x7f3208150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:51.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:51.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32080cf0d0 0x7f32080c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:51.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320759e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32080c79f0 con 0x7f32080ced00 2023-03-03T07:25:51.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32080c63d0 con 0x7f32080cf0d0 2023-03-03T07:25:51.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320759e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32080c4f60 con 0x7f32080b3970 2023-03-03T07:25:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f3205d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f3206d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32080cf0d0 0x7f32080c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32080b3970 0x7f3208150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32080b3970 0x7f3208150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38376/0 (socket says 172.21.15.102:38376) 2023-03-03T07:25:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320659c700 1 -- 172.21.15.102:0/765231759 learned_addr learned my addr 172.21.15.102:0/765231759 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:25:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f3206d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32080cf0d0 0x7f32080c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:32942/0 (socket says 172.21.15.102:32942) 2023-03-03T07:25:51.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f3205d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47186/0 (socket says 172.21.15.102:47186) 2023-03-03T07:25:51.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320659c700 1 -- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32080cf0d0 msgr2=0x7f32080c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:51.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.644+0000 7f320659c700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32080cf0d0 0x7f32080c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f320659c700 1 -- 172.21.15.102:0/765231759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 msgr2=0x7f32080c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:51.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f320659c700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f320659c700 1 -- 172.21.15.102:0/765231759 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f8009550 con 0x7f32080b3970 2023-03-03T07:25:51.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f3205d9b700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:25:51.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f3206d9d700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32080cf0d0 0x7f32080c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:25:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f320659c700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32080b3970 0x7f3208150a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f31f8000c00 tx=0x7f31f8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:25:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f31f6ffd700 1 -- 172.21.15.102:0/765231759 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f80044d0 con 0x7f32080b3970 2023-03-03T07:25:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f31f6ffd700 1 -- 172.21.15.102:0/765231759 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f801e070 con 0x7f32080b3970 2023-03-03T07:25:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320806bf20 con 0x7f32080b3970 2023-03-03T07:25:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f31f6ffd700 1 -- 172.21.15.102:0/765231759 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f80038e0 con 0x7f32080b3970 2023-03-03T07:25:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.645+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320815ce80 con 0x7f32080b3970 2023-03-03T07:25:51.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.646+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320806c200 con 0x7f32080b3970 2023-03-03T07:25:51.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.648+0000 7f31f6ffd700 1 -- 172.21.15.102:0/765231759 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f31f800f050 con 0x7f32080b3970 2023-03-03T07:25:51.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.649+0000 7f31f6ffd700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31e80682f0 0x7f31e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:25:51.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.649+0000 7f31f6ffd700 1 -- 172.21.15.102:0/765231759 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f31f8086d50 con 0x7f32080b3970 2023-03-03T07:25:51.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.649+0000 7f3205d9b700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31e80682f0 0x7f31e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:25:51.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.650+0000 7f3205d9b700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31e80682f0 0x7f31e806a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f32081e4470 tx=0x7f31fc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:25:51.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.653+0000 7f31f6ffd700 1 -- 172.21.15.102:0/765231759 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31f8053460 con 0x7f32080b3970 2023-03-03T07:25:51.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.886+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32080c66b0 con 0x7f32080b3970 2023-03-03T07:25:51.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.887+0000 7f31f6ffd700 1 -- 172.21.15.102:0/765231759 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f31f8020020 con 0x7f32080b3970 2023-03-03T07:25:51.888 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:25:51.888 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:25:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31e80682f0 msgr2=0x7f31e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31e80682f0 0x7f31e806a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f32081e4470 tx=0x7f31fc009210 comp rx=0 tx=0).stop 2023-03-03T07:25:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32080b3970 msgr2=0x7f3208150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:25:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32080b3970 0x7f3208150a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f31f8000c00 tx=0x7f31f8004750 comp rx=0 tx=0).stop 2023-03-03T07:25:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 shutdown_connections 2023-03-03T07:25:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31e80682f0 0x7f31e806a7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32080b3970 0x7f3208150a00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32080cf0d0 0x7f32080c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 --2- 172.21.15.102:0/765231759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32080ced00 0x7f32080c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:25:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 >> 172.21.15.102:0/765231759 conn(0x7f320808b1c0 msgr2=0x7f3208071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:25:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 shutdown_connections 2023-03-03T07:25:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:25:51.889+0000 7f320759e700 1 -- 172.21.15.102:0/765231759 wait complete. 2023-03-03T07:25:51.893 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:25:51.917 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 76 secs before thrashing 2023-03-03T07:26:04.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:26:04.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:26:34.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:26:34.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:04.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:04.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:07.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:27:08.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.166+0000 7fc8dc7ff700 1 Processor -- start 2023-03-03T07:27:08.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.166+0000 7fc8dc7ff700 1 -- start start 2023-03-03T07:27:08.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8dc7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d4153290 0x7fc8d4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:27:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8dc7ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d4153660 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8dc7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d406c360 con 0x7fc8d41593c0 2023-03-03T07:27:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8dc7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d40cec40 con 0x7fc8d4153290 2023-03-03T07:27:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8dad9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc8d40cd8e0 0x7fc8d40ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49142/0 (socket says 172.21.15.102:49142) 2023-03-03T07:27:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8d9d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc8d41593c0 0x7fc8d40cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:55004/0 (socket says 172.21.15.102:55004) 2023-03-03T07:27:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8da59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d4153290 0x7fc8d4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:27:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8dad9c700 1 -- 172.21.15.102:0/116952735 learned_addr learned my addr 172.21.15.102:0/116952735 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:27:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.168+0000 7fc8d9599700 1 -- 172.21.15.102:0/116952735 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296973568 0 0) 0x7fc8d406c360 con 0x7fc8d41593c0 2023-03-03T07:27:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8d9599700 1 -- 172.21.15.102:0/116952735 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c00035b0 con 0x7fc8d41593c0 2023-03-03T07:27:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8d9599700 1 -- 172.21.15.102:0/116952735 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569776534 0 0) 0x7fc8d4153660 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8da59b700 1 -- 172.21.15.102:0/116952735 >> v1:172.21.15.172:6789/0 conn(0x7fc8d41593c0 legacy=0x7fc8d40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:27:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8da59b700 1 -- 172.21.15.102:0/116952735 >> v1:172.21.15.102:6789/0 conn(0x7fc8d40cd8e0 legacy=0x7fc8d40ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:27:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8da59b700 1 -- 172.21.15.102:0/116952735 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d415cea0 con 0x7fc8d4153290 2023-03-03T07:27:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8da59b700 1 --2- 172.21.15.102:0/116952735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d4153290 0x7fc8d4158e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009870 tx=0x7fc8d0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd83807812c39beb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:27:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8d9599700 1 -- 172.21.15.102:0/116952735 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0011830 con 0x7fc8d4153290 2023-03-03T07:27:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8d9599700 1 -- 172.21.15.102:0/116952735 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d0011990 con 0x7fc8d4153290 2023-03-03T07:27:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.169+0000 7fc8d9599700 1 -- 172.21.15.102:0/116952735 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0011d20 con 0x7fc8d4153290 2023-03-03T07:27:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.170+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/116952735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d4153290 msgr2=0x7fc8d4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:27:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.170+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/116952735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d4153290 0x7fc8d4158e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009870 tx=0x7fc8d0009b80 comp rx=0 tx=0).stop 2023-03-03T07:27:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.170+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/116952735 shutdown_connections 2023-03-03T07:27:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.170+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/116952735 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d4153290 0x7fc8d4158e80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.170+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/116952735 >> 172.21.15.102:0/116952735 conn(0x7fc8d4065fb0 msgr2=0x7fc8d40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:27:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.170+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/116952735 shutdown_connections 2023-03-03T07:27:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.170+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/116952735 wait complete. 2023-03-03T07:27:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.172+0000 7fc8dc7ff700 1 Processor -- start 2023-03-03T07:27:08.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.172+0000 7fc8dc7ff700 1 -- start start 2023-03-03T07:27:08.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8dc7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d40cd8e0 0x7fc8d414e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:27:08.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8dc7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d4153290 0x7fc8d41f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:27:08.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8dc7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d41593c0 0x7fc8d40b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:27:08.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8dc7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d40b3380 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8dc7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d414eb20 con 0x7fc8d4153290 2023-03-03T07:27:08.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8dc7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d41f2cd0 con 0x7fc8d41593c0 2023-03-03T07:27:08.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8d9d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d4153290 0x7fc8d41f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:27:08.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.173+0000 7fc8da59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d40cd8e0 0x7fc8d414e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:27:08.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8dad9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d41593c0 0x7fc8d40b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:27:08.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8d9d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d4153290 0x7fc8d41f2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47814/0 (socket says 172.21.15.102:47814) 2023-03-03T07:27:08.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8d9d9a700 1 -- 172.21.15.102:0/500876368 learned_addr learned my addr 172.21.15.102:0/500876368 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:27:08.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8da59b700 1 -- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d41593c0 msgr2=0x7fc8d40b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:27:08.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8da59b700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d41593c0 0x7fc8d40b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8da59b700 1 -- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d4153290 msgr2=0x7fc8d41f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:27:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8da59b700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d4153290 0x7fc8d41f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8da59b700 1 -- 172.21.15.102:0/500876368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d00094a0 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8d9d9a700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d4153290 0x7fc8d41f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:27:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8dad9c700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d41593c0 0x7fc8d40b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:27:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.174+0000 7fc8da59b700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d40cd8e0 0x7fc8d414e5e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0016000 tx=0x7fc8d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:27:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.175+0000 7fc8caffd700 1 -- 172.21.15.102:0/500876368 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d001a7e0 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.175+0000 7fc8caffd700 1 -- 172.21.15.102:0/500876368 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d001a960 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.175+0000 7fc8caffd700 1 -- 172.21.15.102:0/500876368 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d001ad10 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.175+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d415d160 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.175+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d406c7f0 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.176+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d40ccde0 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.177+0000 7fc8caffd700 1 -- 172.21.15.102:0/500876368 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc8d000f050 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.178+0000 7fc8caffd700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8b80682f0 0x7fc8b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:27:08.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.178+0000 7fc8caffd700 1 -- 172.21.15.102:0/500876368 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc8d0086b40 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.179+0000 7fc8d9d9a700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8b80682f0 0x7fc8b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:27:08.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.179+0000 7fc8d9d9a700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8b80682f0 0x7fc8b806a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40c9320 tx=0x7fc8c4007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:27:08.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.183+0000 7fc8caffd700 1 -- 172.21.15.102:0/500876368 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8d0053250 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.414+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8d41e21c0 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.415+0000 7fc8caffd700 1 -- 172.21.15.102:0/500876368 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc8d41e21c0 con 0x7fc8d40cd8e0 2023-03-03T07:27:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:27:08.417 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:27:08.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8b80682f0 msgr2=0x7fc8b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:27:08.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8b80682f0 0x7fc8b806a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40c9320 tx=0x7fc8c4007de0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d40cd8e0 msgr2=0x7fc8d414e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:27:08.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d40cd8e0 0x7fc8d414e5e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0016000 tx=0x7fc8d0004750 comp rx=0 tx=0).stop 2023-03-03T07:27:08.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 shutdown_connections 2023-03-03T07:27:08.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc8b80682f0 0x7fc8b806a7a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc8d41593c0 0x7fc8d40b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d40cd8e0 0x7fc8d414e5e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 --2- 172.21.15.102:0/500876368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d4153290 0x7fc8d41f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:27:08.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 >> 172.21.15.102:0/500876368 conn(0x7fc8d4065fb0 msgr2=0x7fc8d40cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:27:08.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 shutdown_connections 2023-03-03T07:27:08.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:27:08.418+0000 7fc8dc7ff700 1 -- 172.21.15.102:0/500876368 wait complete. 2023-03-03T07:27:08.424 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:27:08.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-03-03T07:27:34.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:34.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:05.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:05.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:35.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:35.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:43.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:28:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.713+0000 7fab50dff700 1 Processor -- start 2023-03-03T07:28:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.713+0000 7fab50dff700 1 -- start start 2023-03-03T07:28:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.715+0000 7fab50dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fab4c0ca190 0x7fab4c164620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:28:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.715+0000 7fab50dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c0cf940 con 0x7fab4c0c9cb0 2023-03-03T07:28:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.715+0000 7fab50dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c0cfb90 con 0x7fab4c153260 2023-03-03T07:28:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.715+0000 7fab50dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4c0cfe70 con 0x7fab4c0ca190 2023-03-03T07:28:43.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.715+0000 7fab4a59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fab4c153260 0x7fab4c0ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:58764/0 (socket says 172.21.15.102:58764) 2023-03-03T07:28:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.715+0000 7fab4a59c700 1 -- 172.21.15.102:0/2532350174 learned_addr learned my addr 172.21.15.102:0/2532350174 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:28:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.715+0000 7fab4ad9d700 1 --2- 172.21.15.102:0/2532350174 >> v2:172.21.15.102:3300/0 conn(0x7fab4c0ca190 0x7fab4c164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:28:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4959a700 1 -- 172.21.15.102:0/2532350174 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142403901 0 0) 0x7fab4c0cfb90 con 0x7fab4c153260 2023-03-03T07:28:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4959a700 1 -- 172.21.15.102:0/2532350174 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab340035b0 con 0x7fab4c153260 2023-03-03T07:28:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4959a700 1 -- 172.21.15.102:0/2532350174 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501823464 0 0) 0x7fab4c0cf940 con 0x7fab4c0c9cb0 2023-03-03T07:28:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4959a700 1 -- 172.21.15.102:0/2532350174 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c0cfb90 con 0x7fab4c0c9cb0 2023-03-03T07:28:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4ad9d700 1 -- 172.21.15.102:0/2532350174 >> v1:172.21.15.172:6789/0 conn(0x7fab4c153260 legacy=0x7fab4c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:28:43.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4ad9d700 1 -- 172.21.15.102:0/2532350174 >> v1:172.21.15.102:6789/0 conn(0x7fab4c0c9cb0 legacy=0x7fab4c0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:28:43.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4ad9d700 1 -- 172.21.15.102:0/2532350174 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c158f80 con 0x7fab4c0ca190 2023-03-03T07:28:43.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.716+0000 7fab4ad9d700 1 --2- 172.21.15.102:0/2532350174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c0ca190 0x7fab4c164620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fab44009d90 tx=0x7fab4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae536a05139bf63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:28:43.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab4959a700 1 -- 172.21.15.102:0/2532350174 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4400e070 con 0x7fab4c0ca190 2023-03-03T07:28:43.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab4959a700 1 -- 172.21.15.102:0/2532350174 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4400bb20 con 0x7fab4c0ca190 2023-03-03T07:28:43.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab4959a700 1 -- 172.21.15.102:0/2532350174 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44012430 con 0x7fab4c0ca190 2023-03-03T07:28:43.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab50dff700 1 -- 172.21.15.102:0/2532350174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c0ca190 msgr2=0x7fab4c164620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:28:43.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab50dff700 1 --2- 172.21.15.102:0/2532350174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c0ca190 0x7fab4c164620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fab44009d90 tx=0x7fab4400b330 comp rx=0 tx=0).stop 2023-03-03T07:28:43.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab50dff700 1 -- 172.21.15.102:0/2532350174 shutdown_connections 2023-03-03T07:28:43.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab50dff700 1 --2- 172.21.15.102:0/2532350174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c0ca190 0x7fab4c164620 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab50dff700 1 -- 172.21.15.102:0/2532350174 >> 172.21.15.102:0/2532350174 conn(0x7fab4c066c00 msgr2=0x7fab4c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:28:43.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab50dff700 1 -- 172.21.15.102:0/2532350174 shutdown_connections 2023-03-03T07:28:43.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.717+0000 7fab50dff700 1 -- 172.21.15.102:0/2532350174 wait complete. 2023-03-03T07:28:43.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.719+0000 7fab50dff700 1 Processor -- start 2023-03-03T07:28:43.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.720+0000 7fab50dff700 1 -- start start 2023-03-03T07:28:43.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab50dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab4c0ca190 0x7fab4c15c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:28:43.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab50dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab4c153260 0x7fab4c154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:28:43.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab50dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c06ba30 0x7fab4c155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab50dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c1ef270 con 0x7fab4c06ba30 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab50dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c15b0f0 con 0x7fab4c0ca190 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab50dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c15c1f0 con 0x7fab4c153260 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab4ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c06ba30 0x7fab4c155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab4a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab4c0ca190 0x7fab4c15c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab49d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab4c153260 0x7fab4c154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab49d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab4c153260 0x7fab4c154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60560/0 (socket says 172.21.15.102:60560) 2023-03-03T07:28:43.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab49d9b700 1 -- 172.21.15.102:0/3392734755 learned_addr learned my addr 172.21.15.102:0/3392734755 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:28:43.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab4ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c06ba30 0x7fab4c155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51592/0 (socket says 172.21.15.102:51592) 2023-03-03T07:28:43.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.721+0000 7fab4a59c700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab4c0ca190 0x7fab4c15c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57456/0 (socket says 172.21.15.102:57456) 2023-03-03T07:28:43.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab49d9b700 1 -- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab4c0ca190 msgr2=0x7fab4c15c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:28:43.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab49d9b700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab4c0ca190 0x7fab4c15c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab49d9b700 1 -- 172.21.15.102:0/3392734755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c06ba30 msgr2=0x7fab4c155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:28:43.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab49d9b700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c06ba30 0x7fab4c155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab49d9b700 1 -- 172.21.15.102:0/3392734755 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab44009a70 con 0x7fab4c153260 2023-03-03T07:28:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab49d9b700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab4c153260 0x7fab4c154f40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fab4000c520 tx=0x7fab4000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:28:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab3affd700 1 -- 172.21.15.102:0/3392734755 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab400052d0 con 0x7fab4c153260 2023-03-03T07:28:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab3affd700 1 -- 172.21.15.102:0/3392734755 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab40005430 con 0x7fab4c153260 2023-03-03T07:28:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c159540 con 0x7fab4c153260 2023-03-03T07:28:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab3affd700 1 -- 172.21.15.102:0/3392734755 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab400038c0 con 0x7fab4c153260 2023-03-03T07:28:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.722+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c159260 con 0x7fab4c153260 2023-03-03T07:28:43.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.723+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c159820 con 0x7fab4c153260 2023-03-03T07:28:43.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.725+0000 7fab3affd700 1 -- 172.21.15.102:0/3392734755 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab40019030 con 0x7fab4c153260 2023-03-03T07:28:43.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.726+0000 7fab3affd700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c068320 0x7fab2c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:28:43.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.726+0000 7fab3affd700 1 -- 172.21.15.102:0/3392734755 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fab40085090 con 0x7fab4c153260 2023-03-03T07:28:43.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.726+0000 7fab4a59c700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c068320 0x7fab2c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:28:43.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.727+0000 7fab4a59c700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c068320 0x7fab2c06a7d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fab3c007fd0 tx=0x7fab3c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:28:43.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.731+0000 7fab3affd700 1 -- 172.21.15.102:0/3392734755 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab400516b0 con 0x7fab4c153260 2023-03-03T07:28:43.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.967+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab4c15c380 con 0x7fab4c153260 2023-03-03T07:28:43.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.968+0000 7fab3affd700 1 -- 172.21.15.102:0/3392734755 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fab40017020 con 0x7fab4c153260 2023-03-03T07:28:43.969 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:28:43.969 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:28:43.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c068320 msgr2=0x7fab2c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:28:43.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c068320 0x7fab2c06a7d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fab3c007fd0 tx=0x7fab3c007de0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab4c153260 msgr2=0x7fab4c154f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:28:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab4c153260 0x7fab4c154f40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fab4000c520 tx=0x7fab4000c830 comp rx=0 tx=0).stop 2023-03-03T07:28:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 shutdown_connections 2023-03-03T07:28:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab4c06ba30 0x7fab4c155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab4c0ca190 0x7fab4c15c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c068320 0x7fab2c06a7d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 --2- 172.21.15.102:0/3392734755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab4c153260 0x7fab4c154f40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:28:43.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.970+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 >> 172.21.15.102:0/3392734755 conn(0x7fab4c0c9cb0 msgr2=0x7fab4c157ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:28:43.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.971+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 shutdown_connections 2023-03-03T07:28:43.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:28:43.971+0000 7fab50dff700 1 -- 172.21.15.102:0/3392734755 wait complete. 2023-03-03T07:28:43.973 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:28:43.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2023-03-03T07:29:05.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:29:05.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:29:11.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:29:11.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.251+0000 7f8094bff700 1 Processor -- start 2023-03-03T07:29:11.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.251+0000 7f8094bff700 1 -- start start 2023-03-03T07:29:11.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f8094bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8090153290 0x7f8090158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:29:11.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f8094bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8090153660 con 0x7f80900cd8e0 2023-03-03T07:29:11.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f8094bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809006c360 con 0x7f80901593c0 2023-03-03T07:29:11.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f8094bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80900cec40 con 0x7f8090153290 2023-03-03T07:29:11.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f808ed9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f80900cd8e0 0x7f80900ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38670/0 (socket says 172.21.15.102:38670) 2023-03-03T07:29:11.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f808ed9d700 1 -- 172.21.15.102:0/4027023988 learned_addr learned my addr 172.21.15.102:0/4027023988 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:29:11.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f808e59c700 1 --2- 172.21.15.102:0/4027023988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8090153290 0x7f8090158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:29:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.253+0000 7f808d59a700 1 -- 172.21.15.102:0/4027023988 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971862695 0 0) 0x7f8090153660 con 0x7f80900cd8e0 2023-03-03T07:29:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808d59a700 1 -- 172.21.15.102:0/4027023988 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80780035b0 con 0x7f80900cd8e0 2023-03-03T07:29:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808d59a700 1 -- 172.21.15.102:0/4027023988 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419389284 0 0) 0x7f809006c360 con 0x7f80901593c0 2023-03-03T07:29:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808d59a700 1 -- 172.21.15.102:0/4027023988 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8090153660 con 0x7f80901593c0 2023-03-03T07:29:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808e59c700 1 -- 172.21.15.102:0/4027023988 >> v1:172.21.15.172:6789/0 conn(0x7f80901593c0 legacy=0x7f80900cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:29:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808e59c700 1 -- 172.21.15.102:0/4027023988 >> v1:172.21.15.102:6789/0 conn(0x7f80900cd8e0 legacy=0x7f80900ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:29:11.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808e59c700 1 -- 172.21.15.102:0/4027023988 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809015cea0 con 0x7f8090153290 2023-03-03T07:29:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808e59c700 1 --2- 172.21.15.102:0/4027023988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8090153290 0x7f8090158e80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8080009870 tx=0x7f8080009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f78f7f3ffdbf7a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:29:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808d59a700 1 -- 172.21.15.102:0/4027023988 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080011830 con 0x7f8090153290 2023-03-03T07:29:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808d59a700 1 -- 172.21.15.102:0/4027023988 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080011990 con 0x7f8090153290 2023-03-03T07:29:11.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.254+0000 7f808d59a700 1 -- 172.21.15.102:0/4027023988 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080011d20 con 0x7f8090153290 2023-03-03T07:29:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.255+0000 7f8094bff700 1 -- 172.21.15.102:0/4027023988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8090153290 msgr2=0x7f8090158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:29:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.255+0000 7f8094bff700 1 --2- 172.21.15.102:0/4027023988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8090153290 0x7f8090158e80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8080009870 tx=0x7f8080009b80 comp rx=0 tx=0).stop 2023-03-03T07:29:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.255+0000 7f8094bff700 1 -- 172.21.15.102:0/4027023988 shutdown_connections 2023-03-03T07:29:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.255+0000 7f8094bff700 1 --2- 172.21.15.102:0/4027023988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8090153290 0x7f8090158e80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.255+0000 7f8094bff700 1 -- 172.21.15.102:0/4027023988 >> 172.21.15.102:0/4027023988 conn(0x7f8090065fb0 msgr2=0x7f80900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:29:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.255+0000 7f8094bff700 1 -- 172.21.15.102:0/4027023988 shutdown_connections 2023-03-03T07:29:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.255+0000 7f8094bff700 1 -- 172.21.15.102:0/4027023988 wait complete. 2023-03-03T07:29:11.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.257+0000 7f8094bff700 1 Processor -- start 2023-03-03T07:29:11.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.257+0000 7f8094bff700 1 -- start start 2023-03-03T07:29:11.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.258+0000 7f8094bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80900cd8e0 0x7f80900b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:29:11.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.258+0000 7f8094bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8090153290 0x7f80900c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:29:11.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.258+0000 7f8094bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80901593c0 0x7f80901f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:29:11.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.258+0000 7f8094bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80900b3b10 con 0x7f80901593c0 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.258+0000 7f8094bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80901f23a0 con 0x7f8090153290 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f8094bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80900b3d60 con 0x7f80900cd8e0 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80901593c0 0x7f80901f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80900cd8e0 0x7f80900b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80901593c0 0x7f80901f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33974/0 (socket says 172.21.15.102:33974) 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 -- 172.21.15.102:0/1300919377 learned_addr learned my addr 172.21.15.102:0/1300919377 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808dd9b700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8090153290 0x7f80900c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:29:11.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 -- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80900cd8e0 msgr2=0x7f80900b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:29:11.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80900cd8e0 0x7f80900b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 -- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8090153290 msgr2=0x7f80900c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:29:11.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8090153290 0x7f80900c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.259+0000 7f808ed9d700 1 -- 172.21.15.102:0/1300919377 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80800094a0 con 0x7f80901593c0 2023-03-03T07:29:11.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.260+0000 7f808ed9d700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80901593c0 0x7f80901f28a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f808800caa0 tx=0x7f808800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:29:11.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.260+0000 7f807effd700 1 -- 172.21.15.102:0/1300919377 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808800a9e0 con 0x7f80901593c0 2023-03-03T07:29:11.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.260+0000 7f807effd700 1 -- 172.21.15.102:0/1300919377 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808800ab60 con 0x7f80901593c0 2023-03-03T07:29:11.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.260+0000 7f807effd700 1 -- 172.21.15.102:0/1300919377 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088012400 con 0x7f80901593c0 2023-03-03T07:29:11.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.260+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809015d160 con 0x7f80901593c0 2023-03-03T07:29:11.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.260+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809006c7f0 con 0x7f80901593c0 2023-03-03T07:29:11.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.261+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80900ccde0 con 0x7f80901593c0 2023-03-03T07:29:11.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.262+0000 7f807effd700 1 -- 172.21.15.102:0/1300919377 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8088019030 con 0x7f80901593c0 2023-03-03T07:29:11.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.263+0000 7f807effd700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80700682f0 0x7f807006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:29:11.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.263+0000 7f807effd700 1 -- 172.21.15.102:0/1300919377 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8088085860 con 0x7f80901593c0 2023-03-03T07:29:11.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.264+0000 7f808e59c700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80700682f0 0x7f807006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:29:11.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.264+0000 7f808e59c700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80700682f0 0x7f807006a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f80800097c0 tx=0x7f808001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:29:11.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.268+0000 7f807effd700 1 -- 172.21.15.102:0/1300919377 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8088052030 con 0x7f80901593c0 2023-03-03T07:29:11.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.502+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80901f2560 con 0x7f80901593c0 2023-03-03T07:29:11.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.503+0000 7f807effd700 1 -- 172.21.15.102:0/1300919377 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8088055650 con 0x7f80901593c0 2023-03-03T07:29:11.505 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:29:11.505 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:29:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80700682f0 msgr2=0x7f807006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:29:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80700682f0 0x7f807006a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f80800097c0 tx=0x7f808001c040 comp rx=0 tx=0).stop 2023-03-03T07:29:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80901593c0 msgr2=0x7f80901f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:29:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80901593c0 0x7f80901f28a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f808800caa0 tx=0x7f808800cdb0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 shutdown_connections 2023-03-03T07:29:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80700682f0 0x7f807006a7a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80900cd8e0 0x7f80900b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80901593c0 0x7f80901f28a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 --2- 172.21.15.102:0/1300919377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8090153290 0x7f80900c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:29:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 >> 172.21.15.102:0/1300919377 conn(0x7f8090065fb0 msgr2=0x7f80900cc120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:29:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 shutdown_connections 2023-03-03T07:29:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:29:11.506+0000 7f8094bff700 1 -- 172.21.15.102:0/1300919377 wait complete. 2023-03-03T07:29:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:29:11.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T07:29:35.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:29:35.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T07:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:30:05.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:05.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:35.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:35.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:49.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:30:49.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.788+0000 7f11073ff700 1 Processor -- start 2023-03-03T07:30:49.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.788+0000 7f11073ff700 1 -- start start 2023-03-03T07:30:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.789+0000 7f11073ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11000cd390 0x7f11000ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:30:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.789+0000 7f11073ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1100153660 con 0x7f1100153290 2023-03-03T07:30:49.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.789+0000 7f11073ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110006c360 con 0x7f11000cd8e0 2023-03-03T07:30:49.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.789+0000 7f11073ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110015cea0 con 0x7f11000cd390 2023-03-03T07:30:49.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f110499a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f11000cd8e0 0x7f11000cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40724/0 (socket says 172.21.15.102:40724) 2023-03-03T07:30:49.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f110499a700 1 -- 172.21.15.102:0/673193411 learned_addr learned my addr 172.21.15.102:0/673193411 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:30:49.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f110519b700 1 --1- 172.21.15.102:0/673193411 >> v1:172.21.15.102:6789/0 conn(0x7f1100153290 0x7f1100158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49914/0 (socket says 172.21.15.102:49914) 2023-03-03T07:30:49.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f110599c700 1 --2- 172.21.15.102:0/673193411 >> v2:172.21.15.102:3300/0 conn(0x7f11000cd390 0x7f11000ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:30:49.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f10f7fff700 1 -- 172.21.15.102:0/673193411 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467151592 0 0) 0x7f110006c360 con 0x7f11000cd8e0 2023-03-03T07:30:49.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f10f7fff700 1 -- 172.21.15.102:0/673193411 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec0035b0 con 0x7f11000cd8e0 2023-03-03T07:30:49.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f10f7fff700 1 -- 172.21.15.102:0/673193411 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689518990 0 0) 0x7f1100153660 con 0x7f1100153290 2023-03-03T07:30:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f10f7fff700 1 -- 172.21.15.102:0/673193411 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110006c360 con 0x7f1100153290 2023-03-03T07:30:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f110599c700 1 -- 172.21.15.102:0/673193411 >> v1:172.21.15.172:6789/0 conn(0x7f11000cd8e0 legacy=0x7f11000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:30:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f110599c700 1 -- 172.21.15.102:0/673193411 >> v1:172.21.15.102:6789/0 conn(0x7f1100153290 legacy=0x7f1100158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:30:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.790+0000 7f110599c700 1 -- 172.21.15.102:0/673193411 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110006c7f0 con 0x7f11000cd390 2023-03-03T07:30:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.791+0000 7f110599c700 1 --2- 172.21.15.102:0/673193411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd390 0x7f11000ce530 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009d90 tx=0x7f10f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1888662663b39572 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:30:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.791+0000 7f10f7fff700 1 -- 172.21.15.102:0/673193411 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f800e070 con 0x7f11000cd390 2023-03-03T07:30:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.791+0000 7f10f7fff700 1 -- 172.21.15.102:0/673193411 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f800bb20 con 0x7f11000cd390 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.791+0000 7f10f7fff700 1 -- 172.21.15.102:0/673193411 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8012430 con 0x7f11000cd390 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.791+0000 7f11073ff700 1 -- 172.21.15.102:0/673193411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd390 msgr2=0x7f11000ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.791+0000 7f11073ff700 1 --2- 172.21.15.102:0/673193411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd390 0x7f11000ce530 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009d90 tx=0x7f10f800b330 comp rx=0 tx=0).stop 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.792+0000 7f11073ff700 1 -- 172.21.15.102:0/673193411 shutdown_connections 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.792+0000 7f11073ff700 1 --2- 172.21.15.102:0/673193411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd390 0x7f11000ce530 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.792+0000 7f11073ff700 1 -- 172.21.15.102:0/673193411 >> 172.21.15.102:0/673193411 conn(0x7f1100065fb0 msgr2=0x7f11000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.792+0000 7f11073ff700 1 -- 172.21.15.102:0/673193411 shutdown_connections 2023-03-03T07:30:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.792+0000 7f11073ff700 1 -- 172.21.15.102:0/673193411 wait complete. 2023-03-03T07:30:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.794+0000 7f11073ff700 1 Processor -- start 2023-03-03T07:30:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.794+0000 7f11073ff700 1 -- start start 2023-03-03T07:30:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f11073ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11000cd390 0x7f1100153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:30:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f11073ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd8e0 0x7f11000cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:30:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f11073ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1100153290 0x7f11001f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:30:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f11073ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11000c5cf0 con 0x7f11000cd8e0 2023-03-03T07:30:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f11073ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1100156120 con 0x7f11000cd390 2023-03-03T07:30:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f11073ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11001e4790 con 0x7f1100153290 2023-03-03T07:30:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f110499a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd8e0 0x7f11000cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:30:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f110519b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11000cd390 0x7f1100153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:30:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f110599c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1100153290 0x7f11001f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:30:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f110499a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd8e0 0x7f11000cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56648/0 (socket says 172.21.15.102:56648) 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.795+0000 7f110499a700 1 -- 172.21.15.102:0/993475555 learned_addr learned my addr 172.21.15.102:0/993475555 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f110499a700 1 -- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1100153290 msgr2=0x7f11001f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f110499a700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1100153290 0x7f11001f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f110499a700 1 -- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11000cd390 msgr2=0x7f1100153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f110499a700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11000cd390 0x7f1100153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f110499a700 1 -- 172.21.15.102:0/993475555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f8009a70 con 0x7f11000cd8e0 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f110519b700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11000cd390 0x7f1100153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:30:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f110499a700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd8e0 0x7f11000cf0b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c700 tx=0x7f10f000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:30:49.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f10f5ffb700 1 -- 172.21.15.102:0/993475555 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0005d70 con 0x7f11000cd8e0 2023-03-03T07:30:49.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f10f5ffb700 1 -- 172.21.15.102:0/993475555 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f00070d0 con 0x7f11000cd8e0 2023-03-03T07:30:49.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11000ccde0 con 0x7f11000cd8e0 2023-03-03T07:30:49.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.796+0000 7f10f5ffb700 1 -- 172.21.15.102:0/993475555 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0007440 con 0x7f11000cd8e0 2023-03-03T07:30:49.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.797+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110015d160 con 0x7f11000cd8e0 2023-03-03T07:30:49.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.798+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11000cd0c0 con 0x7f11000cd8e0 2023-03-03T07:30:49.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.799+0000 7f10f5ffb700 1 -- 172.21.15.102:0/993475555 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f10f0019030 con 0x7f11000cd8e0 2023-03-03T07:30:49.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.800+0000 7f10f5ffb700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e40682f0 0x7f10e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:30:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.800+0000 7f10f5ffb700 1 -- 172.21.15.102:0/993475555 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f10f0084fb0 con 0x7f11000cd8e0 2023-03-03T07:30:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.801+0000 7f110519b700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e40682f0 0x7f10e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:30:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.801+0000 7f110519b700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e40682f0 0x7f10e406a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f10fc001110 tx=0x7f10fc007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:30:49.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:49.804+0000 7f10f5ffb700 1 -- 172.21.15.102:0/993475555 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10f004b5e0 con 0x7f11000cd8e0 2023-03-03T07:30:50.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.041+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11001e4da0 con 0x7f11000cd8e0 2023-03-03T07:30:50.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.041+0000 7f10f5ffb700 1 -- 172.21.15.102:0/993475555 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f10f0017030 con 0x7f11000cd8e0 2023-03-03T07:30:50.044 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:30:50.044 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:30:50.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e40682f0 msgr2=0x7f10e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:30:50.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e40682f0 0x7f10e406a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f10fc001110 tx=0x7f10fc007e30 comp rx=0 tx=0).stop 2023-03-03T07:30:50.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd8e0 msgr2=0x7f11000cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:30:50.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd8e0 0x7f11000cf0b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c700 tx=0x7f10f000ca10 comp rx=0 tx=0).stop 2023-03-03T07:30:50.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 shutdown_connections 2023-03-03T07:30:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e40682f0 0x7f10e406a7a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:30:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1100153290 0x7f11001f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:30:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11000cd390 0x7f1100153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:30:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 --2- 172.21.15.102:0/993475555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11000cd8e0 0x7f11000cf0b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:30:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 >> 172.21.15.102:0/993475555 conn(0x7f1100065fb0 msgr2=0x7f1100158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:30:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 shutdown_connections 2023-03-03T07:30:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:30:50.044+0000 7f11073ff700 1 -- 172.21.15.102:0/993475555 wait complete. 2023-03-03T07:30:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:30:50.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 106 secs before thrashing 2023-03-03T07:31:06.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:31:06.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:31:36.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:31:36.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:06.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:06.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:36.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:32:36.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.324+0000 7fddf1fff700 1 Processor -- start 2023-03-03T07:32:36.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.325+0000 7fddf1fff700 1 -- start start 2023-03-03T07:32:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddf1fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:32:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddf1fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fddec0cd390 0x7fddec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:32:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddf1fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddec153660 con 0x7fddec153290 2023-03-03T07:32:36.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddf1fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddec06c360 con 0x7fddec0cd390 2023-03-03T07:32:36.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddf1fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec15cea0 con 0x7fddec0cd8e0 2023-03-03T07:32:36.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddebfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fddec0cd390 0x7fddec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:32:36.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddeb7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fddec153290 0x7fddec158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42464/0 (socket says 172.21.15.102:42464) 2023-03-03T07:32:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddeb7fe700 1 -- 172.21.15.102:0/1121863531 learned_addr learned my addr 172.21.15.102:0/1121863531 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:32:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.326+0000 7fddeaffd700 1 --2- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:32:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.327+0000 7fddebfff700 1 -- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 msgr2=0x7fddec0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:32:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.327+0000 7fddebfff700 1 --2- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.327+0000 7fddebfff700 1 -- 172.21.15.102:0/1121863531 >> v1:172.21.15.172:6789/0 conn(0x7fddec153290 legacy=0x7fddec158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:32:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.327+0000 7fddebfff700 1 -- 172.21.15.102:0/1121863531 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec06c7f0 con 0x7fddec0cd390 2023-03-03T07:32:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.327+0000 7fddea7fc700 1 -- 172.21.15.102:0/1121863531 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966122239 0 0) 0x7fddec153660 con 0x7fddec153290 2023-03-03T07:32:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.327+0000 7fddebfff700 1 --2- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec0ce360 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009d90 tx=0x7fdde400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f30e1d68fb2506cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:32:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.327+0000 7fddea7fc700 1 -- 172.21.15.102:0/1121863531 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde400e070 con 0x7fddec0cd390 2023-03-03T07:32:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddea7fc700 1 -- 172.21.15.102:0/1121863531 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde400bb20 con 0x7fddec0cd390 2023-03-03T07:32:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddea7fc700 1 -- 172.21.15.102:0/1121863531 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4012430 con 0x7fddec0cd390 2023-03-03T07:32:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 -- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 msgr2=0x7fddec0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:32:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 --2- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec0ce360 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009d90 tx=0x7fdde400b330 comp rx=0 tx=0).stop 2023-03-03T07:32:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 -- 172.21.15.102:0/1121863531 shutdown_connections 2023-03-03T07:32:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 --2- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec0ce360 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 --2- 172.21.15.102:0/1121863531 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 -- 172.21.15.102:0/1121863531 >> 172.21.15.102:0/1121863531 conn(0x7fddec065fb0 msgr2=0x7fddec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:32:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 -- 172.21.15.102:0/1121863531 shutdown_connections 2023-03-03T07:32:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.328+0000 7fddf1fff700 1 -- 172.21.15.102:0/1121863531 wait complete. 2023-03-03T07:32:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.330+0000 7fddf1fff700 1 Processor -- start 2023-03-03T07:32:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.331+0000 7fddf1fff700 1 -- start start 2023-03-03T07:32:36.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddf1fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec14f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:32:36.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddf1fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec1e41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:32:36.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddf1fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddec153290 0x7fddec1e4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:32:36.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddf1fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec0c7250 con 0x7fddec153290 2023-03-03T07:32:36.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddf1fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec0c7660 con 0x7fddec0cd390 2023-03-03T07:32:36.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddf1fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec0c6f70 con 0x7fddec0cd8e0 2023-03-03T07:32:36.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddeb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec14f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:32:36.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddeaffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec1e41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:32:36.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddeb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec14f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38814/0 (socket says 172.21.15.102:38814) 2023-03-03T07:32:36.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddeaffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec1e41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45142/0 (socket says 172.21.15.102:45142) 2023-03-03T07:32:36.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddeb7fe700 1 -- 172.21.15.102:0/3541937656 learned_addr learned my addr 172.21.15.102:0/3541937656 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:32:36.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.332+0000 7fddebfff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddec153290 0x7fddec1e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:32:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddeaffd700 1 -- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 msgr2=0x7fddec14f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:32:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddeaffd700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec14f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddeaffd700 1 -- 172.21.15.102:0/3541937656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddec153290 msgr2=0x7fddec1e4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:32:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddeaffd700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddec153290 0x7fddec1e4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddeaffd700 1 -- 172.21.15.102:0/3541937656 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4009a70 con 0x7fddec0cd8e0 2023-03-03T07:32:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddeaffd700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec1e41d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdde000e900 tx=0x7fdde000ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:32:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddd3fff700 1 -- 172.21.15.102:0/3541937656 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0013680 con 0x7fddec0cd8e0 2023-03-03T07:32:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddd3fff700 1 -- 172.21.15.102:0/3541937656 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00137e0 con 0x7fddec0cd8e0 2023-03-03T07:32:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddec0ccde0 con 0x7fddec0cd8e0 2023-03-03T07:32:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.333+0000 7fddd3fff700 1 -- 172.21.15.102:0/3541937656 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0013b50 con 0x7fddec0cd8e0 2023-03-03T07:32:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.334+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddec15d160 con 0x7fddec0cd8e0 2023-03-03T07:32:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.335+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddec0cd0c0 con 0x7fddec0cd8e0 2023-03-03T07:32:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.336+0000 7fddd3fff700 1 -- 172.21.15.102:0/3541937656 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdde0019030 con 0x7fddec0cd8e0 2023-03-03T07:32:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.337+0000 7fddd3fff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddd80682f0 0x7fddd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:32:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.337+0000 7fddd3fff700 1 -- 172.21.15.102:0/3541937656 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdde0085f00 con 0x7fddec0cd8e0 2023-03-03T07:32:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.337+0000 7fddeb7fe700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddd80682f0 0x7fddd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:32:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.338+0000 7fddeb7fe700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddd80682f0 0x7fddd806a7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdddc000f60 tx=0x7fdddc007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:32:36.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.341+0000 7fddd3fff700 1 -- 172.21.15.102:0/3541937656 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdde0052610 con 0x7fddec0cd8e0 2023-03-03T07:32:36.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.576+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddec04e6e0 con 0x7fddec0cd8e0 2023-03-03T07:32:36.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.577+0000 7fddd3fff700 1 -- 172.21.15.102:0/3541937656 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdde0017020 con 0x7fddec0cd8e0 2023-03-03T07:32:36.578 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:32:36.578 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:32:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddd80682f0 msgr2=0x7fddd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:32:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddd80682f0 0x7fddd806a7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdddc000f60 tx=0x7fdddc007ea0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 msgr2=0x7fddec1e41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:32:36.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec1e41d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdde000e900 tx=0x7fdde000ec10 comp rx=0 tx=0).stop 2023-03-03T07:32:36.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 shutdown_connections 2023-03-03T07:32:36.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddec153290 0x7fddec1e4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddec0cd390 0x7fddec14f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddd80682f0 0x7fddd806a7a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 --2- 172.21.15.102:0/3541937656 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddec0cd8e0 0x7fddec1e41d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:32:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 >> 172.21.15.102:0/3541937656 conn(0x7fddec065fb0 msgr2=0x7fddec158280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:32:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.579+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 shutdown_connections 2023-03-03T07:32:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:32:36.580+0000 7fddf1fff700 1 -- 172.21.15.102:0/3541937656 wait complete. 2023-03-03T07:32:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:32:36.608 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-03-03T07:32:36.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:36.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:05.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:33:05.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.862+0000 7f74a2dff700 1 Processor -- start 2023-03-03T07:33:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.863+0000 7f74a2dff700 1 -- start start 2023-03-03T07:33:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f74a2dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f749c153290 0x7f749c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f74a2dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c0cd8e0 0x7f749c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f74a2dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c153660 con 0x7f749c1593c0 2023-03-03T07:33:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f74a2dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c06c360 con 0x7f749c153290 2023-03-03T07:33:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f74a2dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c15cea0 con 0x7f749c0cd8e0 2023-03-03T07:33:05.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f749bfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f749c1593c0 0x7f749c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49094/0 (socket says 172.21.15.102:49094) 2023-03-03T07:33:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f749bfff700 1 -- 172.21.15.102:0/1012753658 learned_addr learned my addr 172.21.15.102:0/1012753658 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:33:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f74a0b9b700 1 --2- 172.21.15.102:0/1012753658 >> v2:172.21.15.172:3300/0 conn(0x7f749c153290 0x7f749c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.864+0000 7f74a139c700 1 --2- 172.21.15.102:0/1012753658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c0cd8e0 0x7f749c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f749b7fe700 1 -- 172.21.15.102:0/1012753658 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427182008 0 0) 0x7f749c153660 con 0x7f749c1593c0 2023-03-03T07:33:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f749b7fe700 1 -- 172.21.15.102:0/1012753658 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f748c0035b0 con 0x7f749c1593c0 2023-03-03T07:33:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f74a139c700 1 -- 172.21.15.102:0/1012753658 >> v2:172.21.15.172:3300/0 conn(0x7f749c153290 msgr2=0x7f749c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:33:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f74a139c700 1 --2- 172.21.15.102:0/1012753658 >> v2:172.21.15.172:3300/0 conn(0x7f749c153290 0x7f749c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f74a139c700 1 -- 172.21.15.102:0/1012753658 >> v1:172.21.15.172:6789/0 conn(0x7f749c1593c0 legacy=0x7f749c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f74a139c700 1 -- 172.21.15.102:0/1012753658 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c06c7f0 con 0x7f749c0cd8e0 2023-03-03T07:33:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f74a0b9b700 1 --2- 172.21.15.102:0/1012753658 >> v2:172.21.15.172:3300/0 conn(0x7f749c153290 0x7f749c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:33:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f74a139c700 1 --2- 172.21.15.102:0/1012753658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c0cd8e0 0x7f749c0ce530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7494009d90 tx=0x7f749400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c562e296a8a4bde server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:33:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.865+0000 7f749b7fe700 1 -- 172.21.15.102:0/1012753658 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749400e070 con 0x7f749c0cd8e0 2023-03-03T07:33:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f749b7fe700 1 -- 172.21.15.102:0/1012753658 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749400bb20 con 0x7f749c0cd8e0 2023-03-03T07:33:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f749b7fe700 1 -- 172.21.15.102:0/1012753658 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494012430 con 0x7f749c0cd8e0 2023-03-03T07:33:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 -- 172.21.15.102:0/1012753658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c0cd8e0 msgr2=0x7f749c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:05.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 --2- 172.21.15.102:0/1012753658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c0cd8e0 0x7f749c0ce530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7494009d90 tx=0x7f749400b330 comp rx=0 tx=0).stop 2023-03-03T07:33:05.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 -- 172.21.15.102:0/1012753658 shutdown_connections 2023-03-03T07:33:05.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 --2- 172.21.15.102:0/1012753658 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c0cd8e0 0x7f749c0ce530 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:05.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 --2- 172.21.15.102:0/1012753658 >> v2:172.21.15.172:3300/0 conn(0x7f749c153290 0x7f749c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:05.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 -- 172.21.15.102:0/1012753658 >> 172.21.15.102:0/1012753658 conn(0x7f749c065fb0 msgr2=0x7f749c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:33:05.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 -- 172.21.15.102:0/1012753658 shutdown_connections 2023-03-03T07:33:05.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.866+0000 7f74a2dff700 1 -- 172.21.15.102:0/1012753658 wait complete. 2023-03-03T07:33:05.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.868+0000 7f74a2dff700 1 Processor -- start 2023-03-03T07:33:05.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.868+0000 7f74a2dff700 1 -- start start 2023-03-03T07:33:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.869+0000 7f74a2dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749c0cd8e0 0x7f749c155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.869+0000 7f74a2dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749c153290 0x7f749c0c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:05.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.869+0000 7f74a2dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c1593c0 0x7f749c1efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:05.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a2dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c155fe0 con 0x7f749c153290 2023-03-03T07:33:05.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a2dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c151580 con 0x7f749c0cd8e0 2023-03-03T07:33:05.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a2dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c154ea0 con 0x7f749c1593c0 2023-03-03T07:33:05.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749c0cd8e0 0x7f749c155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:05.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749c0cd8e0 0x7f749c155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39524/0 (socket says 172.21.15.102:39524) 2023-03-03T07:33:05.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 -- 172.21.15.102:0/647859671 learned_addr learned my addr 172.21.15.102:0/647859671 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:33:05.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f749bfff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749c153290 0x7f749c0c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a139c700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c1593c0 0x7f749c1efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 -- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c1593c0 msgr2=0x7f749c1efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c1593c0 0x7f749c1efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 -- 172.21.15.102:0/647859671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749c153290 msgr2=0x7f749c0c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:05.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749c153290 0x7f749c0c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.870+0000 7f74a0b9b700 1 -- 172.21.15.102:0/647859671 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900094a0 con 0x7f749c0cd8e0 2023-03-03T07:33:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.871+0000 7f74a0b9b700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749c0cd8e0 0x7f749c155aa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f749000ec10 tx=0x7f749000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:33:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.871+0000 7f74997fa700 1 -- 172.21.15.102:0/647859671 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490009760 con 0x7f749c0cd8e0 2023-03-03T07:33:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.871+0000 7f74997fa700 1 -- 172.21.15.102:0/647859671 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749000c850 con 0x7f749c0cd8e0 2023-03-03T07:33:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.871+0000 7f74997fa700 1 -- 172.21.15.102:0/647859671 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749000cbe0 con 0x7f749c0cd8e0 2023-03-03T07:33:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.871+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7494009a70 con 0x7f749c0cd8e0 2023-03-03T07:33:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.871+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c0ccde0 con 0x7f749c0cd8e0 2023-03-03T07:33:05.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.872+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c15d160 con 0x7f749c0cd8e0 2023-03-03T07:33:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.874+0000 7f74997fa700 1 -- 172.21.15.102:0/647859671 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7490019030 con 0x7f749c0cd8e0 2023-03-03T07:33:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.875+0000 7f74997fa700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f74800682f0 0x7f748006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.875+0000 7f74997fa700 1 -- 172.21.15.102:0/647859671 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7490085c00 con 0x7f749c0cd8e0 2023-03-03T07:33:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.875+0000 7f749bfff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f74800682f0 0x7f748006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:05.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.876+0000 7f749bfff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f74800682f0 0x7f748006a7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f749c1572c0 tx=0x7f7488007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:33:05.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:05.879+0000 7f74997fa700 1 -- 172.21.15.102:0/647859671 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7490052310 con 0x7f749c0cd8e0 2023-03-03T07:33:06.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.113+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f749c0cd0c0 con 0x7f749c0cd8e0 2023-03-03T07:33:06.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.114+0000 7f74997fa700 1 -- 172.21.15.102:0/647859671 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7490055930 con 0x7f749c0cd8e0 2023-03-03T07:33:06.116 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:33:06.116 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:33:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f74800682f0 msgr2=0x7f748006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f74800682f0 0x7f748006a7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f749c1572c0 tx=0x7f7488007dc0 comp rx=0 tx=0).stop 2023-03-03T07:33:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749c0cd8e0 msgr2=0x7f749c155aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749c0cd8e0 0x7f749c155aa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f749000ec10 tx=0x7f749000ef20 comp rx=0 tx=0).stop 2023-03-03T07:33:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 shutdown_connections 2023-03-03T07:33:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f74800682f0 0x7f748006a7a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749c1593c0 0x7f749c1efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749c0cd8e0 0x7f749c155aa0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 --2- 172.21.15.102:0/647859671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749c153290 0x7f749c0c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 >> 172.21.15.102:0/647859671 conn(0x7f749c065fb0 msgr2=0x7f749c15ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:33:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 shutdown_connections 2023-03-03T07:33:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:06.117+0000 7f74a2dff700 1 -- 172.21.15.102:0/647859671 wait complete. 2023-03-03T07:33:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:33:06.146 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-03-03T07:33:06.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:06.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:13.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:33:13.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.413+0000 7f00caf9d700 1 Processor -- start 2023-03-03T07:33:13.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.413+0000 7f00caf9d700 1 -- start start 2023-03-03T07:33:13.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.415+0000 7f00caf9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f00c40ced00 0x7f00c40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:13.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.415+0000 7f00caf9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c40c3d20 con 0x7f00c40b3970 2023-03-03T07:33:13.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.415+0000 7f00caf9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c415cba0 con 0x7f00c40c41b0 2023-03-03T07:33:13.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.415+0000 7f00caf9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c40c4930 con 0x7f00c40ced00 2023-03-03T07:33:13.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.415+0000 7f00c9f9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f00c40ced00 0x7f00c40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:13.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.415+0000 7f00c979a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f00c40b3970 0x7f00c4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53316/0 (socket says 172.21.15.102:53316) 2023-03-03T07:33:13.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.415+0000 7f00c979a700 1 -- 172.21.15.102:0/2951133825 learned_addr learned my addr 172.21.15.102:0/2951133825 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:33:13.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.416+0000 7f00c8f99700 1 -- 172.21.15.102:0/2951133825 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786546592 0 0) 0x7f00c415cba0 con 0x7f00c40c41b0 2023-03-03T07:33:13.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.416+0000 7f00c9f9b700 1 -- 172.21.15.102:0/2951133825 >> v1:172.21.15.172:6789/0 conn(0x7f00c40c41b0 legacy=0x7f00c4163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:13.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.416+0000 7f00c9f9b700 1 -- 172.21.15.102:0/2951133825 >> v1:172.21.15.102:6789/0 conn(0x7f00c40b3970 legacy=0x7f00c4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:13.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.416+0000 7f00c9f9b700 1 -- 172.21.15.102:0/2951133825 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c415ce80 con 0x7f00c40ced00 2023-03-03T07:33:13.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.416+0000 7f00c9f9b700 1 --2- 172.21.15.102:0/2951133825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40ced00 0x7f00c40c5a90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009840 tx=0x7f00c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c951c7450ccf59 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:33:13.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.416+0000 7f00c8f99700 1 -- 172.21.15.102:0/2951133825 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0004230 con 0x7f00c40ced00 2023-03-03T07:33:13.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00c8f99700 1 -- 172.21.15.102:0/2951133825 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c0004390 con 0x7f00c40ced00 2023-03-03T07:33:13.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00c8f99700 1 -- 172.21.15.102:0/2951133825 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c00036a0 con 0x7f00c40ced00 2023-03-03T07:33:13.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00caf9d700 1 -- 172.21.15.102:0/2951133825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40ced00 msgr2=0x7f00c40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:13.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00caf9d700 1 --2- 172.21.15.102:0/2951133825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40ced00 0x7f00c40c5a90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009840 tx=0x7f00c0009b50 comp rx=0 tx=0).stop 2023-03-03T07:33:13.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00caf9d700 1 -- 172.21.15.102:0/2951133825 shutdown_connections 2023-03-03T07:33:13.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00caf9d700 1 --2- 172.21.15.102:0/2951133825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40ced00 0x7f00c40c5a90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:13.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00caf9d700 1 -- 172.21.15.102:0/2951133825 >> 172.21.15.102:0/2951133825 conn(0x7f00c408b1c0 msgr2=0x7f00c4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:33:13.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00caf9d700 1 -- 172.21.15.102:0/2951133825 shutdown_connections 2023-03-03T07:33:13.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.417+0000 7f00caf9d700 1 -- 172.21.15.102:0/2951133825 wait complete. 2023-03-03T07:33:13.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.419+0000 7f00caf9d700 1 Processor -- start 2023-03-03T07:33:13.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.419+0000 7f00caf9d700 1 -- start start 2023-03-03T07:33:13.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.420+0000 7f00caf9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00c40b3970 0x7f00c415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:13.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.420+0000 7f00caf9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40c41b0 0x7f00c415b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:13.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.420+0000 7f00caf9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00c40ced00 0x7f00c414a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:13.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.420+0000 7f00caf9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40b1860 con 0x7f00c40c41b0 2023-03-03T07:33:13.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.420+0000 7f00caf9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c414c0e0 con 0x7f00c40b3970 2023-03-03T07:33:13.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.420+0000 7f00caf9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c414c380 con 0x7f00c40ced00 2023-03-03T07:33:13.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40c41b0 0x7f00c415b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:13.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40c41b0 0x7f00c415b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38868/0 (socket says 172.21.15.102:38868) 2023-03-03T07:33:13.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 -- 172.21.15.102:0/886845986 learned_addr learned my addr 172.21.15.102:0/886845986 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:33:13.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c9f9b700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00c40b3970 0x7f00c415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:13.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00ca79c700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00c40ced00 0x7f00c414a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:13.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 -- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00c40ced00 msgr2=0x7f00c414a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:13.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00c40ced00 0x7f00c414a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:13.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 -- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00c40b3970 msgr2=0x7f00c415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:13.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00c40b3970 0x7f00c415b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:13.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.421+0000 7f00c979a700 1 -- 172.21.15.102:0/886845986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c00094a0 con 0x7f00c40c41b0 2023-03-03T07:33:13.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.422+0000 7f00c979a700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40c41b0 0x7f00c415b600 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f00b400c750 tx=0x7f00b400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:33:13.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.422+0000 7f00ba7fc700 1 -- 172.21.15.102:0/886845986 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4004c80 con 0x7f00c40c41b0 2023-03-03T07:33:13.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.422+0000 7f00ba7fc700 1 -- 172.21.15.102:0/886845986 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b4004de0 con 0x7f00c40c41b0 2023-03-03T07:33:13.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.422+0000 7f00ba7fc700 1 -- 172.21.15.102:0/886845986 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40072d0 con 0x7f00c40c41b0 2023-03-03T07:33:13.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.422+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c406c200 con 0x7f00c40c41b0 2023-03-03T07:33:13.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.422+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c406bf20 con 0x7f00c40c41b0 2023-03-03T07:33:13.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.423+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c415c890 con 0x7f00c40c41b0 2023-03-03T07:33:13.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.424+0000 7f00ba7fc700 1 -- 172.21.15.102:0/886845986 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f00b401a020 con 0x7f00c40c41b0 2023-03-03T07:33:13.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.425+0000 7f00ba7fc700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00a8068320 0x7f00a806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:33:13.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.426+0000 7f00ba7fc700 1 -- 172.21.15.102:0/886845986 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f00b4085960 con 0x7f00c40c41b0 2023-03-03T07:33:13.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.426+0000 7f00c9f9b700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00a8068320 0x7f00a806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:33:13.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.427+0000 7f00c9f9b700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00a8068320 0x7f00a806a7d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f00c00090b0 tx=0x7f00c0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:33:13.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.430+0000 7f00ba7fc700 1 -- 172.21.15.102:0/886845986 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f00b4051fe0 con 0x7f00c40c41b0 2023-03-03T07:33:13.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.660+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00c404e6e0 con 0x7f00c40c41b0 2023-03-03T07:33:13.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.661+0000 7f00ba7fc700 1 -- 172.21.15.102:0/886845986 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f00b4055600 con 0x7f00c40c41b0 2023-03-03T07:33:13.664 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:33:13.664 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:33:13.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.663+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00a8068320 msgr2=0x7f00a806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:13.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.663+0000 7f00caf9d700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00a8068320 0x7f00a806a7d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f00c00090b0 tx=0x7f00c0008030 comp rx=0 tx=0).stop 2023-03-03T07:33:13.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.663+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40c41b0 msgr2=0x7f00c415b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:33:13.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.663+0000 7f00caf9d700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40c41b0 0x7f00c415b600 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f00b400c750 tx=0x7f00b400ca60 comp rx=0 tx=0).stop 2023-03-03T07:33:13.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 shutdown_connections 2023-03-03T07:33:13.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00a8068320 0x7f00a806a7d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:13.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00c40ced00 0x7f00c414a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:13.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00c40b3970 0x7f00c415b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:13.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 --2- 172.21.15.102:0/886845986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c40c41b0 0x7f00c415b600 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:33:13.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 >> 172.21.15.102:0/886845986 conn(0x7f00c408b1c0 msgr2=0x7f00c41e1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:33:13.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 shutdown_connections 2023-03-03T07:33:13.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:33:13.664+0000 7f00caf9d700 1 -- 172.21.15.102:0/886845986 wait complete. 2023-03-03T07:33:13.670 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:33:13.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T07:33:37.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:37.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:01.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:34:01.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.945+0000 7f25643ff700 1 Processor -- start 2023-03-03T07:34:01.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.945+0000 7f25643ff700 1 -- start start 2023-03-03T07:34:01.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.946+0000 7f25643ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f255c153290 0x7f255c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:01.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.946+0000 7f25643ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f255c1593c0 0x7f255c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:01.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f25643ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f255c153660 con 0x7f255c0cd8e0 2023-03-03T07:34:01.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f25643ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255c06c360 con 0x7f255c1593c0 2023-03-03T07:34:01.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f25643ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255c15cea0 con 0x7f255c153290 2023-03-03T07:34:01.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f256199a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f255c1593c0 0x7f255c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:01.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f256199a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f255c1593c0 0x7f255c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56028/0 (socket says 172.21.15.102:56028) 2023-03-03T07:34:01.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f256199a700 1 -- 172.21.15.102:0/1743673997 learned_addr learned my addr 172.21.15.102:0/1743673997 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:34:01.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f256219b700 1 --2- 172.21.15.102:0/1743673997 >> v2:172.21.15.172:3300/0 conn(0x7f255c153290 0x7f255c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:01.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.947+0000 7f2561199700 1 -- 172.21.15.102:0/1743673997 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453126245 0 0) 0x7f255c153660 con 0x7f255c0cd8e0 2023-03-03T07:34:01.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f2561199700 1 -- 172.21.15.102:0/1743673997 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25480035b0 con 0x7f255c0cd8e0 2023-03-03T07:34:01.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f256199a700 1 -- 172.21.15.102:0/1743673997 >> v2:172.21.15.172:3300/0 conn(0x7f255c153290 msgr2=0x7f255c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:34:01.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f256199a700 1 --2- 172.21.15.102:0/1743673997 >> v2:172.21.15.172:3300/0 conn(0x7f255c153290 0x7f255c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:01.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f256199a700 1 -- 172.21.15.102:0/1743673997 >> v1:172.21.15.172:6789/0 conn(0x7f255c0cd8e0 legacy=0x7f255c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:01.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f256199a700 1 -- 172.21.15.102:0/1743673997 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c06c7f0 con 0x7f255c1593c0 2023-03-03T07:34:01.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f256219b700 1 --2- 172.21.15.102:0/1743673997 >> v2:172.21.15.172:3300/0 conn(0x7f255c153290 0x7f255c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:34:01.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f256199a700 1 --2- 172.21.15.102:0/1743673997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c1593c0 0x7f255c0cde20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f255c08cc70 tx=0x7f254c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd26dd1e108a42f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:34:01.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f2561199700 1 -- 172.21.15.102:0/1743673997 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c011830 con 0x7f255c1593c0 2023-03-03T07:34:01.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f2561199700 1 -- 172.21.15.102:0/1743673997 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254c011990 con 0x7f255c1593c0 2023-03-03T07:34:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.948+0000 7f2561199700 1 -- 172.21.15.102:0/1743673997 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c011d20 con 0x7f255c1593c0 2023-03-03T07:34:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 -- 172.21.15.102:0/1743673997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c1593c0 msgr2=0x7f255c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 --2- 172.21.15.102:0/1743673997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c1593c0 0x7f255c0cde20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f255c08cc70 tx=0x7f254c009bd0 comp rx=0 tx=0).stop 2023-03-03T07:34:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 -- 172.21.15.102:0/1743673997 shutdown_connections 2023-03-03T07:34:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 --2- 172.21.15.102:0/1743673997 >> v2:172.21.15.172:3300/0 conn(0x7f255c153290 0x7f255c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 --2- 172.21.15.102:0/1743673997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c1593c0 0x7f255c0cde20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 -- 172.21.15.102:0/1743673997 >> 172.21.15.102:0/1743673997 conn(0x7f255c065fb0 msgr2=0x7f255c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:34:01.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 -- 172.21.15.102:0/1743673997 shutdown_connections 2023-03-03T07:34:01.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.949+0000 7f25643ff700 1 -- 172.21.15.102:0/1743673997 wait complete. 2023-03-03T07:34:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.952+0000 7f25643ff700 1 Processor -- start 2023-03-03T07:34:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.952+0000 7f25643ff700 1 -- start start 2023-03-03T07:34:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.953+0000 7f25643ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f255c0cd8e0 0x7f255c14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.953+0000 7f25643ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c153290 0x7f255c1f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.953+0000 7f25643ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f255c1593c0 0x7f255c14d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.953+0000 7f25643ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c1500c0 con 0x7f255c153290 2023-03-03T07:34:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.953+0000 7f25643ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c0b3790 con 0x7f255c0cd8e0 2023-03-03T07:34:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.953+0000 7f25643ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255c15bad0 con 0x7f255c1593c0 2023-03-03T07:34:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f255c0cd8e0 0x7f255c14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f255c0cd8e0 0x7f255c14de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50736/0 (socket says 172.21.15.102:50736) 2023-03-03T07:34:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 -- 172.21.15.102:0/2135947067 learned_addr learned my addr 172.21.15.102:0/2135947067 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:34:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256299c700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f255c1593c0 0x7f255c14d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256199a700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c153290 0x7f255c1f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 -- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f255c1593c0 msgr2=0x7f255c14d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f255c1593c0 0x7f255c14d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 -- 172.21.15.102:0/2135947067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c153290 msgr2=0x7f255c1f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c153290 0x7f255c1f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.954+0000 7f256219b700 1 -- 172.21.15.102:0/2135947067 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580094a0 con 0x7f255c0cd8e0 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.955+0000 7f256219b700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f255c0cd8e0 0x7f255c14de10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f255800ec10 tx=0x7f255800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.955+0000 7f2552ffd700 1 -- 172.21.15.102:0/2135947067 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800c850 con 0x7f255c0cd8e0 2023-03-03T07:34:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.955+0000 7f2552ffd700 1 -- 172.21.15.102:0/2135947067 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255800c9b0 con 0x7f255c0cd8e0 2023-03-03T07:34:01.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.955+0000 7f2552ffd700 1 -- 172.21.15.102:0/2135947067 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800cd20 con 0x7f255c0cd8e0 2023-03-03T07:34:01.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.955+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c009550 con 0x7f255c0cd8e0 2023-03-03T07:34:01.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.955+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255c0ccde0 con 0x7f255c0cd8e0 2023-03-03T07:34:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.956+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255c15d160 con 0x7f255c0cd8e0 2023-03-03T07:34:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.957+0000 7f2552ffd700 1 -- 172.21.15.102:0/2135947067 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2558019030 con 0x7f255c0cd8e0 2023-03-03T07:34:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.959+0000 7f2552ffd700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2540068320 0x7f254006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.959+0000 7f2552ffd700 1 -- 172.21.15.102:0/2135947067 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2558085d30 con 0x7f255c0cd8e0 2023-03-03T07:34:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.959+0000 7f256199a700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2540068320 0x7f254006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.960+0000 7f256199a700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2540068320 0x7f254006a7d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f254c009c10 tx=0x7f254c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:34:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:01.963+0000 7f2552ffd700 1 -- 172.21.15.102:0/2135947067 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25580523b0 con 0x7f255c0cd8e0 2023-03-03T07:34:02.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.195+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f255c0cd0c0 con 0x7f255c0cd8e0 2023-03-03T07:34:02.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.196+0000 7f2552ffd700 1 -- 172.21.15.102:0/2135947067 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2558021090 con 0x7f255c0cd8e0 2023-03-03T07:34:02.201 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:34:02.202 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:34:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2540068320 msgr2=0x7f254006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2540068320 0x7f254006a7d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f254c009c10 tx=0x7f254c008030 comp rx=0 tx=0).stop 2023-03-03T07:34:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f255c0cd8e0 msgr2=0x7f255c14de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f255c0cd8e0 0x7f255c14de10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f255800ec10 tx=0x7f255800ef20 comp rx=0 tx=0).stop 2023-03-03T07:34:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 shutdown_connections 2023-03-03T07:34:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2540068320 0x7f254006a7d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f255c1593c0 0x7f255c14d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f255c0cd8e0 0x7f255c14de10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:02.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 --2- 172.21.15.102:0/2135947067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f255c153290 0x7f255c1f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:02.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 >> 172.21.15.102:0/2135947067 conn(0x7f255c065fb0 msgr2=0x7f255c15b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:34:02.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 shutdown_connections 2023-03-03T07:34:02.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:02.199+0000 7f25643ff700 1 -- 172.21.15.102:0/2135947067 wait complete. 2023-03-03T07:34:02.205 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:34:02.229 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2023-03-03T07:34:07.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:07.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:16.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:34:16.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.482+0000 7f82191ff700 1 Processor -- start 2023-03-03T07:34:16.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.483+0000 7f82191ff700 1 -- start start 2023-03-03T07:34:16.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f82191ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8214153290 0x7f8214158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f82191ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82140cd8e0 0x7f82140ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f82191ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8214153660 con 0x7f82141593c0 2023-03-03T07:34:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f82191ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821406c360 con 0x7f82140cd8e0 2023-03-03T07:34:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f82191ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821415cea0 con 0x7f8214153290 2023-03-03T07:34:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f8212d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8214153290 0x7f8214158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f821259c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f82141593c0 0x7f82140cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46624/0 (socket says 172.21.15.102:46624) 2023-03-03T07:34:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f821259c700 1 -- 172.21.15.102:0/397129639 learned_addr learned my addr 172.21.15.102:0/397129639 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:34:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.485+0000 7f821359e700 1 --2- 172.21.15.102:0/397129639 >> v2:172.21.15.102:3300/0 conn(0x7f82140cd8e0 0x7f82140ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8211d9b700 1 -- 172.21.15.102:0/397129639 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891314871 0 0) 0x7f8214153660 con 0x7f82141593c0 2023-03-03T07:34:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8212d9d700 1 -- 172.21.15.102:0/397129639 >> v2:172.21.15.102:3300/0 conn(0x7f82140cd8e0 msgr2=0x7f82140ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:34:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8212d9d700 1 --2- 172.21.15.102:0/397129639 >> v2:172.21.15.102:3300/0 conn(0x7f82140cd8e0 0x7f82140ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8212d9d700 1 -- 172.21.15.102:0/397129639 >> v1:172.21.15.172:6789/0 conn(0x7f82141593c0 legacy=0x7f82140cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8212d9d700 1 -- 172.21.15.102:0/397129639 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821406c7f0 con 0x7f8214153290 2023-03-03T07:34:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f821359e700 1 --2- 172.21.15.102:0/397129639 >> v2:172.21.15.102:3300/0 conn(0x7f82140cd8e0 0x7f82140ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:34:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8212d9d700 1 --2- 172.21.15.102:0/397129639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8214153290 0x7f8214158e80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8204009840 tx=0x7f8204009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a279eb1b5aa0678 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:34:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8211d9b700 1 -- 172.21.15.102:0/397129639 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204011830 con 0x7f8214153290 2023-03-03T07:34:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8211d9b700 1 -- 172.21.15.102:0/397129639 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8204011990 con 0x7f8214153290 2023-03-03T07:34:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.486+0000 7f8211d9b700 1 -- 172.21.15.102:0/397129639 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204011d20 con 0x7f8214153290 2023-03-03T07:34:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 -- 172.21.15.102:0/397129639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8214153290 msgr2=0x7f8214158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 --2- 172.21.15.102:0/397129639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8214153290 0x7f8214158e80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8204009840 tx=0x7f8204009b50 comp rx=0 tx=0).stop 2023-03-03T07:34:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 -- 172.21.15.102:0/397129639 shutdown_connections 2023-03-03T07:34:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 --2- 172.21.15.102:0/397129639 >> v2:172.21.15.102:3300/0 conn(0x7f82140cd8e0 0x7f82140ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 --2- 172.21.15.102:0/397129639 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8214153290 0x7f8214158e80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 -- 172.21.15.102:0/397129639 >> 172.21.15.102:0/397129639 conn(0x7f8214065fb0 msgr2=0x7f82140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:34:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 -- 172.21.15.102:0/397129639 shutdown_connections 2023-03-03T07:34:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.487+0000 7f82191ff700 1 -- 172.21.15.102:0/397129639 wait complete. 2023-03-03T07:34:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.490+0000 7f82191ff700 1 Processor -- start 2023-03-03T07:34:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.490+0000 7f82191ff700 1 -- start start 2023-03-03T07:34:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.491+0000 7f82191ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82140cd8e0 0x7f82140b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.491+0000 7f82191ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8214153290 0x7f82141f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.491+0000 7f82191ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82141593c0 0x7f8214150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.491+0000 7f82191ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8214153ea0 con 0x7f82140cd8e0 2023-03-03T07:34:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.491+0000 7f82191ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82141f2350 con 0x7f8214153290 2023-03-03T07:34:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.491+0000 7f82191ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82141f25d0 con 0x7f82141593c0 2023-03-03T07:34:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82140cd8e0 0x7f82140b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82140cd8e0 0x7f82140b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55186/0 (socket says 172.21.15.102:55186) 2023-03-03T07:34:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 -- 172.21.15.102:0/2687184352 learned_addr learned my addr 172.21.15.102:0/2687184352 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:34:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f821259c700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8214153290 0x7f82141f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f821359e700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82141593c0 0x7f8214150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 -- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82141593c0 msgr2=0x7f8214150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82141593c0 0x7f8214150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 -- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8214153290 msgr2=0x7f82141f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8214153290 0x7f82141f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.492+0000 7f8212d9d700 1 -- 172.21.15.102:0/2687184352 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040094a0 con 0x7f82140cd8e0 2023-03-03T07:34:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.493+0000 7f8212d9d700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82140cd8e0 0x7f82140b31d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f82040090b0 tx=0x7f8204004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:34:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.493+0000 7f81fb7fe700 1 -- 172.21.15.102:0/2687184352 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040036a0 con 0x7f82140cd8e0 2023-03-03T07:34:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.493+0000 7f81fb7fe700 1 -- 172.21.15.102:0/2687184352 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820401e070 con 0x7f82140cd8e0 2023-03-03T07:34:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.493+0000 7f81fb7fe700 1 -- 172.21.15.102:0/2687184352 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204003a80 con 0x7f82140cd8e0 2023-03-03T07:34:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.493+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82140ccde0 con 0x7f82140cd8e0 2023-03-03T07:34:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.493+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821415d160 con 0x7f82140cd8e0 2023-03-03T07:34:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.494+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82140cd0c0 con 0x7f82140cd8e0 2023-03-03T07:34:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.495+0000 7f81fb7fe700 1 -- 172.21.15.102:0/2687184352 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f820400f050 con 0x7f82140cd8e0 2023-03-03T07:34:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.497+0000 7f81fb7fe700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f82000682f0 0x7f820006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:34:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.498+0000 7f81fb7fe700 1 -- 172.21.15.102:0/2687184352 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8204086cf0 con 0x7f82140cd8e0 2023-03-03T07:34:16.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.498+0000 7f821259c700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f82000682f0 0x7f820006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:34:16.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.498+0000 7f821259c700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f82000682f0 0x7f820006a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f821415a580 tx=0x7f8208007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:34:16.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.502+0000 7f81fb7fe700 1 -- 172.21.15.102:0/2687184352 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8204053400 con 0x7f82140cd8e0 2023-03-03T07:34:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.733+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f821404e6e0 con 0x7f82140cd8e0 2023-03-03T07:34:16.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.734+0000 7f81fb7fe700 1 -- 172.21.15.102:0/2687184352 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8204020020 con 0x7f82140cd8e0 2023-03-03T07:34:16.735 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:34:16.736 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:34:16.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.736+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f82000682f0 msgr2=0x7f820006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:16.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.736+0000 7f82191ff700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f82000682f0 0x7f820006a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f821415a580 tx=0x7f8208007de0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.736+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82140cd8e0 msgr2=0x7f82140b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:34:16.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.736+0000 7f82191ff700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82140cd8e0 0x7f82140b31d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f82040090b0 tx=0x7f8204004750 comp rx=0 tx=0).stop 2023-03-03T07:34:16.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 shutdown_connections 2023-03-03T07:34:16.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f82000682f0 0x7f820006a7a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f82141593c0 0x7f8214150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82140cd8e0 0x7f82140b31d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 --2- 172.21.15.102:0/2687184352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8214153290 0x7f82141f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:34:16.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 >> 172.21.15.102:0/2687184352 conn(0x7f8214065fb0 msgr2=0x7f821415abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:34:16.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 shutdown_connections 2023-03-03T07:34:16.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:34:16.737+0000 7f82191ff700 1 -- 172.21.15.102:0/2687184352 wait complete. 2023-03-03T07:34:16.739 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:34:16.764 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T07:34:37.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:37.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:07.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:07.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:37.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:37.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:43.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:35:44.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.014+0000 7f6be93ff700 1 Processor -- start 2023-03-03T07:35:44.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.014+0000 7f6be93ff700 1 -- start start 2023-03-03T07:35:44.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.015+0000 7f6be93ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6be4153290 0x7f6be4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:35:44.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.015+0000 7f6be93ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be40cd8e0 0x7f6be40ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:35:44.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.016+0000 7f6be93ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be4153660 con 0x7f6be41593c0 2023-03-03T07:35:44.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.016+0000 7f6be93ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be406c360 con 0x7f6be4153290 2023-03-03T07:35:44.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.016+0000 7f6be93ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be415cea0 con 0x7f6be40cd8e0 2023-03-03T07:35:44.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.016+0000 7f6be37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be40cd8e0 0x7f6be40ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:35:44.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.016+0000 7f6be2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6be4153290 0x7f6be4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:35:44.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.016+0000 7f6be27fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6be41593c0 0x7f6be40cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33480/0 (socket says 172.21.15.102:33480) 2023-03-03T07:35:44.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.016+0000 7f6be27fc700 1 -- 172.21.15.102:0/765093801 learned_addr learned my addr 172.21.15.102:0/765093801 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:35:44.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be2ffd700 1 -- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be40cd8e0 msgr2=0x7f6be40ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:35:44.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be2ffd700 1 --2- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be40cd8e0 0x7f6be40ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be2ffd700 1 -- 172.21.15.102:0/765093801 >> v1:172.21.15.172:6789/0 conn(0x7f6be41593c0 legacy=0x7f6be40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:35:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be2ffd700 1 -- 172.21.15.102:0/765093801 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be406c7f0 con 0x7f6be4153290 2023-03-03T07:35:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be37fe700 1 --2- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be40cd8e0 0x7f6be40ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:35:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be1ffb700 1 -- 172.21.15.102:0/765093801 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929828501 0 0) 0x7f6be4153660 con 0x7f6be41593c0 2023-03-03T07:35:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be2ffd700 1 --2- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4158e80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009840 tx=0x7f6bd4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1175519814aab17f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:35:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be1ffb700 1 -- 172.21.15.102:0/765093801 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4011830 con 0x7f6be4153290 2023-03-03T07:35:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be1ffb700 1 -- 172.21.15.102:0/765093801 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4011990 con 0x7f6be4153290 2023-03-03T07:35:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.017+0000 7f6be1ffb700 1 -- 172.21.15.102:0/765093801 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4011d20 con 0x7f6be4153290 2023-03-03T07:35:44.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 -- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 msgr2=0x7f6be4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:35:44.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 --2- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4158e80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009840 tx=0x7f6bd4009b50 comp rx=0 tx=0).stop 2023-03-03T07:35:44.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 -- 172.21.15.102:0/765093801 shutdown_connections 2023-03-03T07:35:44.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 --2- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be40cd8e0 0x7f6be40ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 --2- 172.21.15.102:0/765093801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4158e80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 -- 172.21.15.102:0/765093801 >> 172.21.15.102:0/765093801 conn(0x7f6be4065fb0 msgr2=0x7f6be40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:35:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 -- 172.21.15.102:0/765093801 shutdown_connections 2023-03-03T07:35:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.018+0000 7f6be93ff700 1 -- 172.21.15.102:0/765093801 wait complete. 2023-03-03T07:35:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.020+0000 7f6be93ff700 1 Processor -- start 2023-03-03T07:35:44.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.020+0000 7f6be93ff700 1 -- start start 2023-03-03T07:35:44.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be93ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6be40cd8e0 0x7f6be40b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:35:44.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be93ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:35:44.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be93ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be41593c0 0x7f6be40b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:35:44.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be93ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be4155100 con 0x7f6be40cd8e0 2023-03-03T07:35:44.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be93ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be4158580 con 0x7f6be4153290 2023-03-03T07:35:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be93ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be41eeae0 con 0x7f6be41593c0 2023-03-03T07:35:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6be40cd8e0 0x7f6be40b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:35:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be27fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:35:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be41593c0 0x7f6be40b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:35:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be27fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52010/0 (socket says 172.21.15.102:52010) 2023-03-03T07:35:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be41593c0 0x7f6be40b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37858/0 (socket says 172.21.15.102:37858) 2023-03-03T07:35:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.021+0000 7f6be27fc700 1 -- 172.21.15.102:0/458659003 learned_addr learned my addr 172.21.15.102:0/458659003 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:35:44.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6be27fc700 1 -- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be41593c0 msgr2=0x7f6be40b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:35:44.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6be27fc700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be41593c0 0x7f6be40b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6be27fc700 1 -- 172.21.15.102:0/458659003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6be40cd8e0 msgr2=0x7f6be40b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:35:44.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6be27fc700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6be40cd8e0 0x7f6be40b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6be27fc700 1 -- 172.21.15.102:0/458659003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc009a70 con 0x7f6be4153290 2023-03-03T07:35:44.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6be27fc700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4157900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80011e0 tx=0x7f6bd800ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:35:44.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6bcb7fe700 1 -- 172.21.15.102:0/458659003 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8004c80 con 0x7f6be4153290 2023-03-03T07:35:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6bcb7fe700 1 -- 172.21.15.102:0/458659003 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8004de0 con 0x7f6be4153290 2023-03-03T07:35:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd40094a0 con 0x7f6be4153290 2023-03-03T07:35:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.022+0000 7f6bcb7fe700 1 -- 172.21.15.102:0/458659003 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8007310 con 0x7f6be4153290 2023-03-03T07:35:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.023+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be40ccde0 con 0x7f6be4153290 2023-03-03T07:35:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.024+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be415d160 con 0x7f6be4153290 2023-03-03T07:35:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.025+0000 7f6bcb7fe700 1 -- 172.21.15.102:0/458659003 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6bd801a020 con 0x7f6be4153290 2023-03-03T07:35:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.026+0000 7f6bcb7fe700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6bd0068320 0x7f6bd006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:35:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.026+0000 7f6bcb7fe700 1 -- 172.21.15.102:0/458659003 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6bd80858f0 con 0x7f6be4153290 2023-03-03T07:35:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.026+0000 7f6be2ffd700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6bd0068320 0x7f6bd006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:35:44.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.028+0000 7f6be2ffd700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6bd0068320 0x7f6bd006a7d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40090b0 tx=0x7f6bd401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:35:44.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.030+0000 7f6bcb7fe700 1 -- 172.21.15.102:0/458659003 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6bd8052020 con 0x7f6be4153290 2023-03-03T07:35:44.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.269+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be40cd0c0 con 0x7f6be4153290 2023-03-03T07:35:44.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.270+0000 7f6bcb7fe700 1 -- 172.21.15.102:0/458659003 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6bd8055640 con 0x7f6be4153290 2023-03-03T07:35:44.271 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:35:44.271 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:35:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6bd0068320 msgr2=0x7f6bd006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:35:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6bd0068320 0x7f6bd006a7d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40090b0 tx=0x7f6bd401c040 comp rx=0 tx=0).stop 2023-03-03T07:35:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 msgr2=0x7f6be4157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:35:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4157900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80011e0 tx=0x7f6bd800ca30 comp rx=0 tx=0).stop 2023-03-03T07:35:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 shutdown_connections 2023-03-03T07:35:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6bd0068320 0x7f6bd006a7d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6be41593c0 0x7f6be40b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6be40cd8e0 0x7f6be40b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 --2- 172.21.15.102:0/458659003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6be4153290 0x7f6be4157900 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:35:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 >> 172.21.15.102:0/458659003 conn(0x7f6be4065fb0 msgr2=0x7f6be415ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:35:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 shutdown_connections 2023-03-03T07:35:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:35:44.273+0000 7f6be93ff700 1 -- 172.21.15.102:0/458659003 wait complete. 2023-03-03T07:35:44.276 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:35:44.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 63 secs before thrashing 2023-03-03T07:36:07.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:07.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:38.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:38.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:47.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:36:47.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.552+0000 7f537b5ff700 1 Processor -- start 2023-03-03T07:36:47.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.553+0000 7f537b5ff700 1 -- start start 2023-03-03T07:36:47.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f537b5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5374153260 0x7f53740ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:36:47.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f537b5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53740cf940 con 0x7f53740cac20 2023-03-03T07:36:47.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f537b5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53740cfb90 con 0x7f53740c9cb0 2023-03-03T07:36:47.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f537b5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53740cfe70 con 0x7f5374153260 2023-03-03T07:36:47.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f5379b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f53740cac20 0x7f5374163610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48208/0 (socket says 172.21.15.102:48208) 2023-03-03T07:36:47.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f5379b9c700 1 -- 172.21.15.102:0/1895827480 learned_addr learned my addr 172.21.15.102:0/1895827480 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:36:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f5378b9a700 1 --1- 172.21.15.102:0/1895827480 >> v1:172.21.15.172:6789/0 conn(0x7f53740c9cb0 0x7f53740cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40720/0 (socket says 172.21.15.102:40720) 2023-03-03T07:36:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.554+0000 7f537939b700 1 --2- 172.21.15.102:0/1895827480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5374153260 0x7f53740ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:36:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f536bfff700 1 -- 172.21.15.102:0/1895827480 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377949005 0 0) 0x7f53740cfb90 con 0x7f53740c9cb0 2023-03-03T07:36:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f536bfff700 1 -- 172.21.15.102:0/1895827480 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53600035b0 con 0x7f53740c9cb0 2023-03-03T07:36:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f536bfff700 1 -- 172.21.15.102:0/1895827480 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944898337 0 0) 0x7f53740cf940 con 0x7f53740cac20 2023-03-03T07:36:47.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f536bfff700 1 -- 172.21.15.102:0/1895827480 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53740cfb90 con 0x7f53740cac20 2023-03-03T07:36:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f537939b700 1 -- 172.21.15.102:0/1895827480 >> v1:172.21.15.172:6789/0 conn(0x7f53740c9cb0 legacy=0x7f53740cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:36:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f537939b700 1 -- 172.21.15.102:0/1895827480 >> v1:172.21.15.102:6789/0 conn(0x7f53740cac20 legacy=0x7f5374163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:36:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f537939b700 1 -- 172.21.15.102:0/1895827480 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5374158f80 con 0x7f5374153260 2023-03-03T07:36:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.555+0000 7f537939b700 1 --2- 172.21.15.102:0/1895827480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5374153260 0x7f53740ca6e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5370009870 tx=0x7f5370009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ee3bea5d8656581 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:36:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f536bfff700 1 -- 172.21.15.102:0/1895827480 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370011830 con 0x7f5374153260 2023-03-03T07:36:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f536bfff700 1 -- 172.21.15.102:0/1895827480 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5370011990 con 0x7f5374153260 2023-03-03T07:36:47.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f536bfff700 1 -- 172.21.15.102:0/1895827480 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370011d20 con 0x7f5374153260 2023-03-03T07:36:47.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f537b5ff700 1 -- 172.21.15.102:0/1895827480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5374153260 msgr2=0x7f53740ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:36:47.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f537b5ff700 1 --2- 172.21.15.102:0/1895827480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5374153260 0x7f53740ca6e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5370009870 tx=0x7f5370009b80 comp rx=0 tx=0).stop 2023-03-03T07:36:47.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f537b5ff700 1 -- 172.21.15.102:0/1895827480 shutdown_connections 2023-03-03T07:36:47.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f537b5ff700 1 --2- 172.21.15.102:0/1895827480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5374153260 0x7f53740ca6e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f537b5ff700 1 -- 172.21.15.102:0/1895827480 >> 172.21.15.102:0/1895827480 conn(0x7f5374066c00 msgr2=0x7f53740ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:36:47.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f537b5ff700 1 -- 172.21.15.102:0/1895827480 shutdown_connections 2023-03-03T07:36:47.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.556+0000 7f537b5ff700 1 -- 172.21.15.102:0/1895827480 wait complete. 2023-03-03T07:36:47.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.558+0000 7f537b5ff700 1 Processor -- start 2023-03-03T07:36:47.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.558+0000 7f537b5ff700 1 -- start start 2023-03-03T07:36:47.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.559+0000 7f537b5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53740cac20 0x7f53740b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:36:47.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.559+0000 7f537b5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5374153260 0x7f5374150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:36:47.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.559+0000 7f537b5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53741506a0 0x7f5374149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:36:47.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.559+0000 7f537b5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537414eda0 con 0x7f5374153260 2023-03-03T07:36:47.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.559+0000 7f537b5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537414c300 con 0x7f53741506a0 2023-03-03T07:36:47.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.559+0000 7f537b5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537414a8f0 con 0x7f53740cac20 2023-03-03T07:36:47.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53740cac20 0x7f53740b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:36:47.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53740cac20 0x7f53740b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:43056/0 (socket says 172.21.15.102:43056) 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 -- 172.21.15.102:0/3258845269 learned_addr learned my addr 172.21.15.102:0/3258845269 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f5379b9c700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53741506a0 0x7f5374149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f5378b9a700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5374153260 0x7f5374150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 -- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53741506a0 msgr2=0x7f5374149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53741506a0 0x7f5374149920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 -- 172.21.15.102:0/3258845269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5374153260 msgr2=0x7f5374150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5374153260 0x7f5374150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.560+0000 7f537939b700 1 -- 172.21.15.102:0/3258845269 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53700094a0 con 0x7f53740cac20 2023-03-03T07:36:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.561+0000 7f537939b700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53740cac20 0x7f53740b4510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f53700097c0 tx=0x7f5370004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:36:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.561+0000 7f5369ffb700 1 -- 172.21.15.102:0/3258845269 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537001a970 con 0x7f53740cac20 2023-03-03T07:36:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.561+0000 7f5369ffb700 1 -- 172.21.15.102:0/3258845269 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537001aad0 con 0x7f53740cac20 2023-03-03T07:36:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.561+0000 7f5369ffb700 1 -- 172.21.15.102:0/3258845269 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537001ae60 con 0x7f53740cac20 2023-03-03T07:36:47.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.561+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5374159540 con 0x7f53740cac20 2023-03-03T07:36:47.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.561+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5374159260 con 0x7f53740cac20 2023-03-03T07:36:47.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.562+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5374159820 con 0x7f53740cac20 2023-03-03T07:36:47.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.563+0000 7f5369ffb700 1 -- 172.21.15.102:0/3258845269 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f537000f050 con 0x7f53740cac20 2023-03-03T07:36:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.565+0000 7f5369ffb700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53580682f0 0x7f535806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:36:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.565+0000 7f5369ffb700 1 -- 172.21.15.102:0/3258845269 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5370086fd0 con 0x7f53740cac20 2023-03-03T07:36:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.565+0000 7f5378b9a700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53580682f0 0x7f535806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:36:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.566+0000 7f5378b9a700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53580682f0 0x7f535806a7a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5374149840 tx=0x7f5364007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:36:47.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.569+0000 7f5369ffb700 1 -- 172.21.15.102:0/3258845269 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53700536e0 con 0x7f53740cac20 2023-03-03T07:36:47.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.803+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f537406cbf0 con 0x7f53740cac20 2023-03-03T07:36:47.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.804+0000 7f5369ffb700 1 -- 172.21.15.102:0/3258845269 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5370056d00 con 0x7f53740cac20 2023-03-03T07:36:47.805 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:36:47.806 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:36:47.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53580682f0 msgr2=0x7f535806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:36:47.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53580682f0 0x7f535806a7a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5374149840 tx=0x7f5364007be0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53740cac20 msgr2=0x7f53740b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:36:47.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53740cac20 0x7f53740b4510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f53700097c0 tx=0x7f5370004750 comp rx=0 tx=0).stop 2023-03-03T07:36:47.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 shutdown_connections 2023-03-03T07:36:47.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53580682f0 0x7f535806a7a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53740cac20 0x7f53740b4510 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53741506a0 0x7f5374149920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 --2- 172.21.15.102:0/3258845269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5374153260 0x7f5374150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:36:47.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 >> 172.21.15.102:0/3258845269 conn(0x7f53740c9cb0 msgr2=0x7f53740ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:36:47.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 shutdown_connections 2023-03-03T07:36:47.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:36:47.807+0000 7f537b5ff700 1 -- 172.21.15.102:0/3258845269 wait complete. 2023-03-03T07:36:47.810 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:36:47.835 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-03-03T07:37:08.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:08.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:38.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:38.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:53.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:37:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.084+0000 7f465539d700 1 Processor -- start 2023-03-03T07:37:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.085+0000 7f465539d700 1 -- start start 2023-03-03T07:37:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f465539d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f46501564b0 0x7f46500cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:37:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f465539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46500cab50 0x7f46500cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:37:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f465539d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46500c9ca0 con 0x7f46500c9610 2023-03-03T07:37:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f465539d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46500b4920 con 0x7f46501564b0 2023-03-03T07:37:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f465539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46500b4c00 con 0x7f46500cab50 2023-03-03T07:37:54.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f4647fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f46500c9610 0x7f46500ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55860/0 (socket says 172.21.15.102:55860) 2023-03-03T07:37:54.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f4654b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46500cab50 0x7f46500cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:37:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f4647fff700 1 -- 172.21.15.102:0/1222028860 learned_addr learned my addr 172.21.15.102:0/1222028860 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:37:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.086+0000 7f464ffff700 1 --2- 172.21.15.102:0/1222028860 >> v2:172.21.15.172:3300/0 conn(0x7f46501564b0 0x7f46500cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:37:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f464ffff700 1 -- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46500cab50 msgr2=0x7f46500cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:37:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f464ffff700 1 --2- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46500cab50 0x7f46500cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f464ffff700 1 -- 172.21.15.102:0/1222028860 >> v1:172.21.15.102:6789/0 conn(0x7f46500c9610 legacy=0x7f46500ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:37:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f464ffff700 1 -- 172.21.15.102:0/1222028860 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46500b46f0 con 0x7f46501564b0 2023-03-03T07:37:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f4654b9c700 1 --2- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46500cab50 0x7f46500cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:37:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f464f7fe700 1 -- 172.21.15.102:0/1222028860 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783863515 0 0) 0x7f46500c9ca0 con 0x7f46500c9610 2023-03-03T07:37:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f464ffff700 1 --2- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46501564b0 0x7f46500cb660 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4640009840 tx=0x7f4640009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdb99db7f36022b3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:37:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.087+0000 7f464f7fe700 1 -- 172.21.15.102:0/1222028860 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640011830 con 0x7f46501564b0 2023-03-03T07:37:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f464f7fe700 1 -- 172.21.15.102:0/1222028860 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640011990 con 0x7f46501564b0 2023-03-03T07:37:54.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f464f7fe700 1 -- 172.21.15.102:0/1222028860 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640011d20 con 0x7f46501564b0 2023-03-03T07:37:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 -- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46501564b0 msgr2=0x7f46500cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:37:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 --2- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46501564b0 0x7f46500cb660 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4640009840 tx=0x7f4640009b50 comp rx=0 tx=0).stop 2023-03-03T07:37:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 -- 172.21.15.102:0/1222028860 shutdown_connections 2023-03-03T07:37:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 --2- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46500cab50 0x7f46500cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 --2- 172.21.15.102:0/1222028860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46501564b0 0x7f46500cb660 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 -- 172.21.15.102:0/1222028860 >> 172.21.15.102:0/1222028860 conn(0x7f4650065fb0 msgr2=0x7f46500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:37:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 -- 172.21.15.102:0/1222028860 shutdown_connections 2023-03-03T07:37:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.088+0000 7f465539d700 1 -- 172.21.15.102:0/1222028860 wait complete. 2023-03-03T07:37:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.090+0000 7f465539d700 1 Processor -- start 2023-03-03T07:37:54.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.090+0000 7f465539d700 1 -- start start 2023-03-03T07:37:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.091+0000 7f465539d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46500c9610 0x7f46501554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:37:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.091+0000 7f465539d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46500cab50 0x7f4650154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:37:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.091+0000 7f465539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46501564b0 0x7f465014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:37:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.091+0000 7f465539d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465015b4b0 con 0x7f46500cab50 2023-03-03T07:37:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.091+0000 7f465539d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4650159f60 con 0x7f46500c9610 2023-03-03T07:37:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.091+0000 7f465539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465015aae0 con 0x7f46501564b0 2023-03-03T07:37:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46500cab50 0x7f4650154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:37:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f464ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46500c9610 0x7f46501554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:37:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46500cab50 0x7f4650154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54462/0 (socket says 172.21.15.102:54462) 2023-03-03T07:37:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 -- 172.21.15.102:0/4157261918 learned_addr learned my addr 172.21.15.102:0/4157261918 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:37:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4654b9c700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46501564b0 0x7f465014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:37:54.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 -- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46501564b0 msgr2=0x7f465014e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:37:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46501564b0 0x7f465014e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 -- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46500c9610 msgr2=0x7f46501554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:37:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46500c9610 0x7f46501554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f4647fff700 1 -- 172.21.15.102:0/4157261918 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648009a70 con 0x7f46500cab50 2023-03-03T07:37:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.092+0000 7f464ffff700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46500c9610 0x7f46501554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:37:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.093+0000 7f4647fff700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46500cab50 0x7f4650154740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f463800c550 tx=0x7f463800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:37:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.093+0000 7f464d7fa700 1 -- 172.21.15.102:0/4157261918 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638004c80 con 0x7f46500cab50 2023-03-03T07:37:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.093+0000 7f464d7fa700 1 -- 172.21.15.102:0/4157261918 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638004de0 con 0x7f46500cab50 2023-03-03T07:37:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.093+0000 7f464d7fa700 1 -- 172.21.15.102:0/4157261918 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46380072d0 con 0x7f46500cab50 2023-03-03T07:37:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.093+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46400094a0 con 0x7f46500cab50 2023-03-03T07:37:54.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.093+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46500b2210 con 0x7f46500cab50 2023-03-03T07:37:54.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.094+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46500b1f30 con 0x7f46500cab50 2023-03-03T07:37:54.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.095+0000 7f464d7fa700 1 -- 172.21.15.102:0/4157261918 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f463801a020 con 0x7f46500cab50 2023-03-03T07:37:54.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.097+0000 7f464d7fa700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46300682f0 0x7f463006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:37:54.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.097+0000 7f464d7fa700 1 -- 172.21.15.102:0/4157261918 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4638085890 con 0x7f46500cab50 2023-03-03T07:37:54.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.097+0000 7f464ffff700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46300682f0 0x7f463006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:37:54.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.098+0000 7f464ffff700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46300682f0 0x7f463006a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f46400090b0 tx=0x7f4640008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:37:54.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.101+0000 7f464d7fa700 1 -- 172.21.15.102:0/4157261918 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4638052050 con 0x7f46500cab50 2023-03-03T07:37:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.338+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4650156880 con 0x7f46500cab50 2023-03-03T07:37:54.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.338+0000 7f464d7fa700 1 -- 172.21.15.102:0/4157261918 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4638055670 con 0x7f46500cab50 2023-03-03T07:37:54.342 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:37:54.343 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:37:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46300682f0 msgr2=0x7f463006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:37:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46300682f0 0x7f463006a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f46400090b0 tx=0x7f4640008030 comp rx=0 tx=0).stop 2023-03-03T07:37:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46500cab50 msgr2=0x7f4650154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:37:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46500cab50 0x7f4650154740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f463800c550 tx=0x7f463800c860 comp rx=0 tx=0).stop 2023-03-03T07:37:54.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 shutdown_connections 2023-03-03T07:37:54.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46300682f0 0x7f463006a7a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46501564b0 0x7f465014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46500c9610 0x7f46501554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 --2- 172.21.15.102:0/4157261918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46500cab50 0x7f4650154740 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:37:54.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 >> 172.21.15.102:0/4157261918 conn(0x7f4650065fb0 msgr2=0x7f4650157360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:37:54.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 shutdown_connections 2023-03-03T07:37:54.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:37:54.341+0000 7f465539d700 1 -- 172.21.15.102:0/4157261918 wait complete. 2023-03-03T07:37:54.348 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:37:54.370 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T07:38:08.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:38:08.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:38:39.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:38:39.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:09.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:09.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:13.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:39:13.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.621+0000 7f9745fff700 1 Processor -- start 2023-03-03T07:39:13.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.621+0000 7f9745fff700 1 -- start start 2023-03-03T07:39:13.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.622+0000 7f9745fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9740153290 0x7f9740158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:39:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.622+0000 7f9745fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f97401593c0 0x7f97400cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:39:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f9745fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9740153660 con 0x7f97400cd8e0 2023-03-03T07:39:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f9745fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974006c360 con 0x7f9740153290 2023-03-03T07:39:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f9745fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974015cea0 con 0x7f97401593c0 2023-03-03T07:39:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f973f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9740153290 0x7f9740158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:39:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f973f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9740153290 0x7f9740158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48518/0 (socket says 172.21.15.102:48518) 2023-03-03T07:39:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f973f7fe700 1 -- 172.21.15.102:0/2797956634 learned_addr learned my addr 172.21.15.102:0/2797956634 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:39:13.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f9737fff700 1 --2- 172.21.15.102:0/2797956634 >> v2:172.21.15.172:3300/0 conn(0x7f97401593c0 0x7f97400cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:39:13.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f973f7fe700 1 -- 172.21.15.102:0/2797956634 >> v2:172.21.15.172:3300/0 conn(0x7f97401593c0 msgr2=0x7f97400cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:39:13.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f973f7fe700 1 --2- 172.21.15.102:0/2797956634 >> v2:172.21.15.172:3300/0 conn(0x7f97401593c0 0x7f97400cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f973f7fe700 1 -- 172.21.15.102:0/2797956634 >> v1:172.21.15.102:6789/0 conn(0x7f97400cd8e0 legacy=0x7f97400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:39:13.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.623+0000 7f973f7fe700 1 -- 172.21.15.102:0/2797956634 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974006c7f0 con 0x7f9740153290 2023-03-03T07:39:13.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.624+0000 7f973f7fe700 1 --2- 172.21.15.102:0/2797956634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740158e80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9740159fb0 tx=0x7f9730009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba10e0566bafe88d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:39:13.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.624+0000 7f973effd700 1 -- 172.21.15.102:0/2797956634 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730011830 con 0x7f9740153290 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.624+0000 7f973effd700 1 -- 172.21.15.102:0/2797956634 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9730011990 con 0x7f9740153290 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.624+0000 7f973effd700 1 -- 172.21.15.102:0/2797956634 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730011d20 con 0x7f9740153290 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 -- 172.21.15.102:0/2797956634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 msgr2=0x7f9740158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 --2- 172.21.15.102:0/2797956634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740158e80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9740159fb0 tx=0x7f9730009bd0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 -- 172.21.15.102:0/2797956634 shutdown_connections 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 --2- 172.21.15.102:0/2797956634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740158e80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 --2- 172.21.15.102:0/2797956634 >> v2:172.21.15.172:3300/0 conn(0x7f97401593c0 0x7f97400cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 -- 172.21.15.102:0/2797956634 >> 172.21.15.102:0/2797956634 conn(0x7f9740065fb0 msgr2=0x7f97400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:39:13.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 -- 172.21.15.102:0/2797956634 shutdown_connections 2023-03-03T07:39:13.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.625+0000 7f9745fff700 1 -- 172.21.15.102:0/2797956634 wait complete. 2023-03-03T07:39:13.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.627+0000 7f9745fff700 1 Processor -- start 2023-03-03T07:39:13.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.627+0000 7f9745fff700 1 -- start start 2023-03-03T07:39:13.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9745fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97400cd8e0 0x7f974014c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:39:13.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9745fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:39:13.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9745fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97401593c0 0x7f9740156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:39:13.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9745fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97401ee930 con 0x7f9740153290 2023-03-03T07:39:13.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9745fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97400b45f0 con 0x7f97400cd8e0 2023-03-03T07:39:13.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9745fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97400b1d10 con 0x7f97401593c0 2023-03-03T07:39:13.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9737fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:39:13.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9737fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48520/0 (socket says 172.21.15.102:48520) 2023-03-03T07:39:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f9737fff700 1 -- 172.21.15.102:0/3456680128 learned_addr learned my addr 172.21.15.102:0/3456680128 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:39:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f973f7fe700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97400cd8e0 0x7f974014c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:39:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.628+0000 7f973ffff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97401593c0 0x7f9740156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:39:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.629+0000 7f973f7fe700 1 -- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97401593c0 msgr2=0x7f9740156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:39:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.629+0000 7f973f7fe700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97401593c0 0x7f9740156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.629+0000 7f973f7fe700 1 -- 172.21.15.102:0/3456680128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 msgr2=0x7f9740155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:39:13.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.629+0000 7f973f7fe700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.629+0000 7f973f7fe700 1 -- 172.21.15.102:0/3456680128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9730009550 con 0x7f97400cd8e0 2023-03-03T07:39:13.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.629+0000 7f973f7fe700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97400cd8e0 0x7f974014c770 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9730016000 tx=0x7f9730004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:39:13.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.629+0000 7f973cff9700 1 -- 172.21.15.102:0/3456680128 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730004380 con 0x7f97400cd8e0 2023-03-03T07:39:13.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.630+0000 7f973cff9700 1 -- 172.21.15.102:0/3456680128 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973001f070 con 0x7f97400cd8e0 2023-03-03T07:39:13.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.630+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97400ccde0 con 0x7f97400cd8e0 2023-03-03T07:39:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.630+0000 7f973cff9700 1 -- 172.21.15.102:0/3456680128 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97300036a0 con 0x7f97400cd8e0 2023-03-03T07:39:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.630+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974015d160 con 0x7f97400cd8e0 2023-03-03T07:39:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.631+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97400cd0c0 con 0x7f97400cd8e0 2023-03-03T07:39:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.632+0000 7f973cff9700 1 -- 172.21.15.102:0/3456680128 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f973002a020 con 0x7f97400cd8e0 2023-03-03T07:39:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.633+0000 7f973cff9700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97200682f0 0x7f972006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:39:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.633+0000 7f973cff9700 1 -- 172.21.15.102:0/3456680128 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9730087210 con 0x7f97400cd8e0 2023-03-03T07:39:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.634+0000 7f9737fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97200682f0 0x7f972006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:39:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.634+0000 7f9737fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97200682f0 0x7f972006a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f974015a4d0 tx=0x7f9728009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:39:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.638+0000 7f973cff9700 1 -- 172.21.15.102:0/3456680128 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9730053920 con 0x7f97400cd8e0 2023-03-03T07:39:13.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.874+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f974004e6e0 con 0x7f97400cd8e0 2023-03-03T07:39:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.875+0000 7f973cff9700 1 -- 172.21.15.102:0/3456680128 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9730056f40 con 0x7f97400cd8e0 2023-03-03T07:39:13.877 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:39:13.877 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:39:13.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97200682f0 msgr2=0x7f972006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:39:13.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97200682f0 0x7f972006a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f974015a4d0 tx=0x7f9728009210 comp rx=0 tx=0).stop 2023-03-03T07:39:13.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97400cd8e0 msgr2=0x7f974014c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:39:13.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97400cd8e0 0x7f974014c770 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9730016000 tx=0x7f9730004750 comp rx=0 tx=0).stop 2023-03-03T07:39:13.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 shutdown_connections 2023-03-03T07:39:13.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97200682f0 0x7f972006a7a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97401593c0 0x7f9740156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97400cd8e0 0x7f974014c770 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 --2- 172.21.15.102:0/3456680128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9740153290 0x7f9740155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:39:13.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 >> 172.21.15.102:0/3456680128 conn(0x7f9740065fb0 msgr2=0x7f97400cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:39:13.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 shutdown_connections 2023-03-03T07:39:13.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:39:13.878+0000 7f9745fff700 1 -- 172.21.15.102:0/3456680128 wait complete. 2023-03-03T07:39:13.882 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:39:13.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-03-03T07:39:39.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:39.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T07:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:40:09.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:09.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:19.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:40:20.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.174+0000 7f7f591ff700 1 Processor -- start 2023-03-03T07:40:20.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.174+0000 7f7f591ff700 1 -- start start 2023-03-03T07:40:20.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.176+0000 7f7f591ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7f5414ffb0 0x7f7f540c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:20.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.176+0000 7f7f591ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540c3c80 con 0x7f7f540716c0 2023-03-03T07:40:20.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.176+0000 7f7f591ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540c41a0 con 0x7f7f5414fbe0 2023-03-03T07:40:20.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.176+0000 7f7f591ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f54071cd0 con 0x7f7f5414ffb0 2023-03-03T07:40:20.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.176+0000 7f7f52d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7f5414fbe0 0x7f7f540c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34346/0 (socket says 172.21.15.102:34346) 2023-03-03T07:40:20.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.176+0000 7f7f52d9d700 1 -- 172.21.15.102:0/536919775 learned_addr learned my addr 172.21.15.102:0/536919775 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:40:20.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.176+0000 7f7f5259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7f5414ffb0 0x7f7f540c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:20.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.177+0000 7f7f51d9b700 1 -- 172.21.15.102:0/536919775 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996634580 0 0) 0x7f7f540c41a0 con 0x7f7f5414fbe0 2023-03-03T07:40:20.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.177+0000 7f7f51d9b700 1 -- 172.21.15.102:0/536919775 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3c0035b0 con 0x7f7f5414fbe0 2023-03-03T07:40:20.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.177+0000 7f7f5259c700 1 -- 172.21.15.102:0/536919775 >> v1:172.21.15.172:6789/0 conn(0x7f7f5414fbe0 legacy=0x7f7f540c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:20.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.177+0000 7f7f5259c700 1 -- 172.21.15.102:0/536919775 >> v1:172.21.15.102:6789/0 conn(0x7f7f540716c0 legacy=0x7f7f540706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:20.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.177+0000 7f7f51d9b700 1 -- 172.21.15.102:0/536919775 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107564694 0 0) 0x7f7f4c001490 con 0x7f7f540716c0 2023-03-03T07:40:20.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.177+0000 7f7f5259c700 1 -- 172.21.15.102:0/536919775 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f540c3ec0 con 0x7f7f5414ffb0 2023-03-03T07:40:20.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.177+0000 7f7f5259c700 1 --2- 172.21.15.102:0/536919775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 0x7f7f540c3740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009840 tx=0x7f7f48009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fe944eff7b71c89 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:40:20.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f51d9b700 1 -- 172.21.15.102:0/536919775 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48011830 con 0x7f7f5414ffb0 2023-03-03T07:40:20.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f51d9b700 1 -- 172.21.15.102:0/536919775 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f48011990 con 0x7f7f5414ffb0 2023-03-03T07:40:20.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f51d9b700 1 -- 172.21.15.102:0/536919775 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48011d00 con 0x7f7f5414ffb0 2023-03-03T07:40:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f591ff700 1 -- 172.21.15.102:0/536919775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 msgr2=0x7f7f540c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f591ff700 1 --2- 172.21.15.102:0/536919775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 0x7f7f540c3740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009840 tx=0x7f7f48009b50 comp rx=0 tx=0).stop 2023-03-03T07:40:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f591ff700 1 -- 172.21.15.102:0/536919775 shutdown_connections 2023-03-03T07:40:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f591ff700 1 --2- 172.21.15.102:0/536919775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 0x7f7f540c3740 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f591ff700 1 -- 172.21.15.102:0/536919775 >> 172.21.15.102:0/536919775 conn(0x7f7f5406af90 msgr2=0x7f7f5406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:40:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.178+0000 7f7f591ff700 1 -- 172.21.15.102:0/536919775 shutdown_connections 2023-03-03T07:40:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.179+0000 7f7f591ff700 1 -- 172.21.15.102:0/536919775 wait complete. 2023-03-03T07:40:20.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.180+0000 7f7f591ff700 1 Processor -- start 2023-03-03T07:40:20.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.181+0000 7f7f591ff700 1 -- start start 2023-03-03T07:40:20.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f591ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f540716c0 0x7f7f54150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:20.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f591ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f5414fbe0 0x7f7f541529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:20.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f591ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 0x7f7f54152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:20.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f591ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f541518d0 con 0x7f7f5414fbe0 2023-03-03T07:40:20.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f591ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f540c8370 con 0x7f7f5414ffb0 2023-03-03T07:40:20.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f591ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f54153450 con 0x7f7f540716c0 2023-03-03T07:40:20.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f5259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f5414fbe0 0x7f7f541529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:20.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f5259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f5414fbe0 0x7f7f541529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49124/0 (socket says 172.21.15.102:49124) 2023-03-03T07:40:20.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f5259c700 1 -- 172.21.15.102:0/2202687995 learned_addr learned my addr 172.21.15.102:0/2202687995 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:40:20.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f52d9d700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f540716c0 0x7f7f54150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:20.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f5359e700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 0x7f7f54152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:20.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f5259c700 1 -- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f540716c0 msgr2=0x7f7f54150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:20.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.182+0000 7f7f5259c700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f540716c0 0x7f7f54150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f5259c700 1 -- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 msgr2=0x7f7f54152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f5259c700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 0x7f7f54152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f5259c700 1 -- 172.21.15.102:0/2202687995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f480094a0 con 0x7f7f5414fbe0 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f5259c700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f5414fbe0 0x7f7f541529d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7f480090b0 tx=0x7f7f48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f3b7fe700 1 -- 172.21.15.102:0/2202687995 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48004550 con 0x7f7f5414fbe0 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f3b7fe700 1 -- 172.21.15.102:0/2202687995 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4801e070 con 0x7f7f5414fbe0 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f3b7fe700 1 -- 172.21.15.102:0/2202687995 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f480038c0 con 0x7f7f5414fbe0 2023-03-03T07:40:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f54072250 con 0x7f7f5414fbe0 2023-03-03T07:40:20.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.183+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f54071f70 con 0x7f7f5414fbe0 2023-03-03T07:40:20.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.185+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f54072530 con 0x7f7f5414fbe0 2023-03-03T07:40:20.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.186+0000 7f7f3b7fe700 1 -- 172.21.15.102:0/2202687995 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f4800f050 con 0x7f7f5414fbe0 2023-03-03T07:40:20.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.187+0000 7f7f3b7fe700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f400682f0 0x7f7f4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:20.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.187+0000 7f7f3b7fe700 1 -- 172.21.15.102:0/2202687995 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7f48086d40 con 0x7f7f5414fbe0 2023-03-03T07:40:20.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.187+0000 7f7f52d9d700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f400682f0 0x7f7f4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:20.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.188+0000 7f7f52d9d700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f400682f0 0x7f7f4006a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7f44007fd0 tx=0x7f7f44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:40:20.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.191+0000 7f7f3b7fe700 1 -- 172.21.15.102:0/2202687995 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f48053450 con 0x7f7f5414fbe0 2023-03-03T07:40:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.424+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5404e6e0 con 0x7f7f5414fbe0 2023-03-03T07:40:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.425+0000 7f7f3b7fe700 1 -- 172.21.15.102:0/2202687995 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7f48015020 con 0x7f7f5414fbe0 2023-03-03T07:40:20.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:40:20.427 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:40:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.427+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f400682f0 msgr2=0x7f7f4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.427+0000 7f7f591ff700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f400682f0 0x7f7f4006a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7f44007fd0 tx=0x7f7f44009040 comp rx=0 tx=0).stop 2023-03-03T07:40:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.427+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f5414fbe0 msgr2=0x7f7f541529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.427+0000 7f7f591ff700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f5414fbe0 0x7f7f541529d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7f480090b0 tx=0x7f7f48004750 comp rx=0 tx=0).stop 2023-03-03T07:40:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 shutdown_connections 2023-03-03T07:40:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f400682f0 0x7f7f4006a7a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f540716c0 0x7f7f54150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f5414ffb0 0x7f7f54152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 --2- 172.21.15.102:0/2202687995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f5414fbe0 0x7f7f541529d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 >> 172.21.15.102:0/2202687995 conn(0x7f7f5406af90 msgr2=0x7f7f540caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:40:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 shutdown_connections 2023-03-03T07:40:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:20.428+0000 7f7f591ff700 1 -- 172.21.15.102:0/2202687995 wait complete. 2023-03-03T07:40:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:40:20.457 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-03-03T07:40:30.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:40:30.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.724+0000 7fd3789ff700 1 Processor -- start 2023-03-03T07:40:30.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.725+0000 7fd3789ff700 1 -- start start 2023-03-03T07:40:30.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd3789ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd374158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:30.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd3789ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3741593c0 0x7fd3740cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:30.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd3789ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd374153660 con 0x7fd3740cd8e0 2023-03-03T07:40:30.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd3789ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37406c360 con 0x7fd3741593c0 2023-03-03T07:40:30.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd3789ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37415cea0 con 0x7fd374153290 2023-03-03T07:40:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd371d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3741593c0 0x7fd3740cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd37259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd374158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd371d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3741593c0 0x7fd3740cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55598/0 (socket says 172.21.15.102:55598) 2023-03-03T07:40:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd371d9b700 1 -- 172.21.15.102:0/4249444058 learned_addr learned my addr 172.21.15.102:0/4249444058 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:40:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.727+0000 7fd372d9d700 1 --1- 172.21.15.102:0/4249444058 >> v1:172.21.15.102:6789/0 conn(0x7fd3740cd8e0 0x7fd3740ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35054/0 (socket says 172.21.15.102:35054) 2023-03-03T07:40:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd371d9b700 1 -- 172.21.15.102:0/4249444058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 msgr2=0x7fd374158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:40:30.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd371d9b700 1 --2- 172.21.15.102:0/4249444058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd374158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd371d9b700 1 -- 172.21.15.102:0/4249444058 >> v1:172.21.15.102:6789/0 conn(0x7fd3740cd8e0 legacy=0x7fd3740ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:30.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd371d9b700 1 -- 172.21.15.102:0/4249444058 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37406c7f0 con 0x7fd3741593c0 2023-03-03T07:40:30.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd371d9b700 1 --2- 172.21.15.102:0/4249444058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3741593c0 0x7fd3740cde20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd3680098f0 tx=0x7fd368009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a285b4d8e633c3fa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:40:30.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd37159a700 1 -- 172.21.15.102:0/4249444058 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368011830 con 0x7fd3741593c0 2023-03-03T07:40:30.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd37159a700 1 -- 172.21.15.102:0/4249444058 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368011990 con 0x7fd3741593c0 2023-03-03T07:40:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.728+0000 7fd37159a700 1 -- 172.21.15.102:0/4249444058 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368011d20 con 0x7fd3741593c0 2023-03-03T07:40:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 -- 172.21.15.102:0/4249444058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3741593c0 msgr2=0x7fd3740cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 --2- 172.21.15.102:0/4249444058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3741593c0 0x7fd3740cde20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd3680098f0 tx=0x7fd368009c00 comp rx=0 tx=0).stop 2023-03-03T07:40:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 -- 172.21.15.102:0/4249444058 shutdown_connections 2023-03-03T07:40:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 --2- 172.21.15.102:0/4249444058 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd374158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 --2- 172.21.15.102:0/4249444058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3741593c0 0x7fd3740cde20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 -- 172.21.15.102:0/4249444058 >> 172.21.15.102:0/4249444058 conn(0x7fd374065fb0 msgr2=0x7fd3740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:40:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 -- 172.21.15.102:0/4249444058 shutdown_connections 2023-03-03T07:40:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.729+0000 7fd3789ff700 1 -- 172.21.15.102:0/4249444058 wait complete. 2023-03-03T07:40:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.731+0000 7fd3789ff700 1 Processor -- start 2023-03-03T07:40:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.731+0000 7fd3789ff700 1 -- start start 2023-03-03T07:40:30.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.732+0000 7fd3789ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3740cd8e0 0x7fd3741e48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.732+0000 7fd3789ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd3740c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.732+0000 7fd3789ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3741593c0 0x7fd3741e5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.732+0000 7fd3789ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740c6c20 con 0x7fd3741593c0 2023-03-03T07:40:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.732+0000 7fd3789ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740b3ea0 con 0x7fd3740cd8e0 2023-03-03T07:40:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.732+0000 7fd3789ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741e4370 con 0x7fd374153290 2023-03-03T07:40:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3740cd8e0 0x7fd3741e48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd371d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd3740c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd372d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3741593c0 0x7fd3741e5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3740cd8e0 0x7fd3741e48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55606/0 (socket says 172.21.15.102:55606) 2023-03-03T07:40:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 -- 172.21.15.102:0/3232337230 learned_addr learned my addr 172.21.15.102:0/3232337230 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:40:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 -- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 msgr2=0x7fd3740c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd3740c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 -- 172.21.15.102:0/3232337230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3741593c0 msgr2=0x7fd3741e5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3741593c0 0x7fd3741e5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 -- 172.21.15.102:0/3232337230 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368009550 con 0x7fd3740cd8e0 2023-03-03T07:40:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.733+0000 7fd37259c700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3740cd8e0 0x7fd3741e48c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd36400e8a0 tx=0x7fd36400ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:40:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.734+0000 7fd362ffd700 1 -- 172.21.15.102:0/3232337230 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364009590 con 0x7fd3740cd8e0 2023-03-03T07:40:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.734+0000 7fd362ffd700 1 -- 172.21.15.102:0/3232337230 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd364013680 con 0x7fd3740cd8e0 2023-03-03T07:40:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.734+0000 7fd362ffd700 1 -- 172.21.15.102:0/3232337230 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3640139f0 con 0x7fd3740cd8e0 2023-03-03T07:40:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.734+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3740ccde0 con 0x7fd3740cd8e0 2023-03-03T07:40:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.734+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37415d160 con 0x7fd3740cd8e0 2023-03-03T07:40:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.735+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3740cd0c0 con 0x7fd3740cd8e0 2023-03-03T07:40:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.736+0000 7fd362ffd700 1 -- 172.21.15.102:0/3232337230 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd364019030 con 0x7fd3740cd8e0 2023-03-03T07:40:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.738+0000 7fd362ffd700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3540682f0 0x7fd35406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:40:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.738+0000 7fd362ffd700 1 -- 172.21.15.102:0/3232337230 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd364085ae0 con 0x7fd3740cd8e0 2023-03-03T07:40:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.738+0000 7fd371d9b700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3540682f0 0x7fd35406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:40:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.739+0000 7fd371d9b700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3540682f0 0x7fd35406a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd368009c40 tx=0x7fd368008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:40:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.742+0000 7fd362ffd700 1 -- 172.21.15.102:0/3232337230 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd3640521f0 con 0x7fd3740cd8e0 2023-03-03T07:40:30.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.973+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3741e4700 con 0x7fd3740cd8e0 2023-03-03T07:40:30.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.974+0000 7fd362ffd700 1 -- 172.21.15.102:0/3232337230 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd364055810 con 0x7fd3740cd8e0 2023-03-03T07:40:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:40:30.976 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:40:30.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3540682f0 msgr2=0x7fd35406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:30.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3540682f0 0x7fd35406a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd368009c40 tx=0x7fd368008030 comp rx=0 tx=0).stop 2023-03-03T07:40:30.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3740cd8e0 msgr2=0x7fd3741e48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:40:30.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3740cd8e0 0x7fd3741e48c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd36400e8a0 tx=0x7fd36400ebb0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 shutdown_connections 2023-03-03T07:40:30.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3741593c0 0x7fd3741e5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3740cd8e0 0x7fd3741e48c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3540682f0 0x7fd35406a7a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 --2- 172.21.15.102:0/3232337230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd374153290 0x7fd3740c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:40:30.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 >> 172.21.15.102:0/3232337230 conn(0x7fd374065fb0 msgr2=0x7fd3740cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:40:30.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 shutdown_connections 2023-03-03T07:40:30.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:40:30.976+0000 7fd3789ff700 1 -- 172.21.15.102:0/3232337230 wait complete. 2023-03-03T07:40:30.981 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:40:31.005 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2023-03-03T07:40:39.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:39.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:09.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:09.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:40.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:40.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:10.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:10.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:27.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:42:27.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.262+0000 7f64739ff700 1 Processor -- start 2023-03-03T07:42:27.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.262+0000 7f64739ff700 1 -- start start 2023-03-03T07:42:27.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.265+0000 7f64739ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c14fbe0 0x7f646c0c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:42:27.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.265+0000 7f64739ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c0c3c80 con 0x7f646c14ffb0 2023-03-03T07:42:27.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.265+0000 7f64739ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c0c41a0 con 0x7f646c0716c0 2023-03-03T07:42:27.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.265+0000 7f64739ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c071cd0 con 0x7f646c14fbe0 2023-03-03T07:42:27.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.265+0000 7f647179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c14fbe0 0x7f646c0c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:42:27.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.265+0000 7f6470f9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f646c14ffb0 0x7f646c0c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43470/0 (socket says 172.21.15.102:43470) 2023-03-03T07:42:27.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.265+0000 7f6470f9a700 1 -- 172.21.15.102:0/1299561175 learned_addr learned my addr 172.21.15.102:0/1299561175 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:42:27.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f6463fff700 1 -- 172.21.15.102:0/1299561175 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697759867 0 0) 0x7f645c001110 con 0x7f646c14ffb0 2023-03-03T07:42:27.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f647179b700 1 -- 172.21.15.102:0/1299561175 >> v1:172.21.15.172:6789/0 conn(0x7f646c0716c0 legacy=0x7f646c0706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:42:27.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f647179b700 1 -- 172.21.15.102:0/1299561175 >> v1:172.21.15.102:6789/0 conn(0x7f646c14ffb0 legacy=0x7f646c0c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:42:27.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f647179b700 1 -- 172.21.15.102:0/1299561175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c0c3ec0 con 0x7f646c14fbe0 2023-03-03T07:42:27.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f647179b700 1 --2- 172.21.15.102:0/1299561175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c14fbe0 0x7f646c0c3740 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f64680097f0 tx=0x7f6468009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26ff52cfed32a29 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:42:27.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f6463fff700 1 -- 172.21.15.102:0/1299561175 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468004230 con 0x7f646c14fbe0 2023-03-03T07:42:27.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f6463fff700 1 -- 172.21.15.102:0/1299561175 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468004390 con 0x7f646c14fbe0 2023-03-03T07:42:27.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.266+0000 7f6463fff700 1 -- 172.21.15.102:0/1299561175 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680036a0 con 0x7f646c14fbe0 2023-03-03T07:42:27.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.267+0000 7f64739ff700 1 -- 172.21.15.102:0/1299561175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c14fbe0 msgr2=0x7f646c0c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:42:27.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.267+0000 7f64739ff700 1 --2- 172.21.15.102:0/1299561175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c14fbe0 0x7f646c0c3740 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f64680097f0 tx=0x7f6468009b00 comp rx=0 tx=0).stop 2023-03-03T07:42:27.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.267+0000 7f64739ff700 1 -- 172.21.15.102:0/1299561175 shutdown_connections 2023-03-03T07:42:27.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.267+0000 7f64739ff700 1 --2- 172.21.15.102:0/1299561175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c14fbe0 0x7f646c0c3740 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:42:27.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.267+0000 7f64739ff700 1 -- 172.21.15.102:0/1299561175 >> 172.21.15.102:0/1299561175 conn(0x7f646c06af90 msgr2=0x7f646c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:42:27.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.267+0000 7f64739ff700 1 -- 172.21.15.102:0/1299561175 shutdown_connections 2023-03-03T07:42:27.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.267+0000 7f64739ff700 1 -- 172.21.15.102:0/1299561175 wait complete. 2023-03-03T07:42:27.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.269+0000 7f64739ff700 1 Processor -- start 2023-03-03T07:42:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.269+0000 7f64739ff700 1 -- start start 2023-03-03T07:42:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.270+0000 7f64739ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c0716c0 0x7f646c14a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.270+0000 7f64739ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f646c14fbe0 0x7f646c1ef720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f64739ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f646c14ffb0 0x7f646c143680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f64739ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c14a0f0 con 0x7f646c14ffb0 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f64739ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c1e37a0 con 0x7f646c14fbe0 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f64739ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c1f0050 con 0x7f646c0716c0 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f6470f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f646c14fbe0 0x7f646c1ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c0716c0 0x7f646c14a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:42:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c0716c0 0x7f646c14a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59514/0 (socket says 172.21.15.102:59514) 2023-03-03T07:42:27.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f6470f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f646c14fbe0 0x7f646c1ef720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36978/0 (socket says 172.21.15.102:36978) 2023-03-03T07:42:27.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 -- 172.21.15.102:0/2321891519 learned_addr learned my addr 172.21.15.102:0/2321891519 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:42:27.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f6471f9c700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f646c14ffb0 0x7f646c143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:42:27.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 -- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f646c14fbe0 msgr2=0x7f646c1ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:42:27.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f646c14fbe0 0x7f646c1ef720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:42:27.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 -- 172.21.15.102:0/2321891519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f646c14ffb0 msgr2=0x7f646c143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:42:27.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f646c14ffb0 0x7f646c143680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:42:27.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.271+0000 7f647179b700 1 -- 172.21.15.102:0/2321891519 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680094a0 con 0x7f646c0716c0 2023-03-03T07:42:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.272+0000 7f647179b700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c0716c0 0x7f646c14a540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f64680090b0 tx=0x7f6468004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:42:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.272+0000 7f6461ffb700 1 -- 172.21.15.102:0/2321891519 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468011630 con 0x7f646c0716c0 2023-03-03T07:42:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.272+0000 7f6461ffb700 1 -- 172.21.15.102:0/2321891519 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646801f070 con 0x7f646c0716c0 2023-03-03T07:42:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.272+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c072250 con 0x7f646c0716c0 2023-03-03T07:42:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.272+0000 7f6461ffb700 1 -- 172.21.15.102:0/2321891519 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680119d0 con 0x7f646c0716c0 2023-03-03T07:42:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.272+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c071f70 con 0x7f646c0716c0 2023-03-03T07:42:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.273+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c072530 con 0x7f646c0716c0 2023-03-03T07:42:27.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.274+0000 7f6461ffb700 1 -- 172.21.15.102:0/2321891519 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6468017020 con 0x7f646c0716c0 2023-03-03T07:42:27.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.276+0000 7f6461ffb700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64500682f0 0x7f645006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:42:27.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.276+0000 7f6461ffb700 1 -- 172.21.15.102:0/2321891519 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6468021070 con 0x7f646c0716c0 2023-03-03T07:42:27.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.276+0000 7f6470f9a700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64500682f0 0x7f645006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:42:27.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.277+0000 7f6470f9a700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64500682f0 0x7f645006a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f646c0c5510 tx=0x7f645c004a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:42:27.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.280+0000 7f6461ffb700 1 -- 172.21.15.102:0/2321891519 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6468053fc0 con 0x7f646c0716c0 2023-03-03T07:42:27.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.514+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f646c14a390 con 0x7f646c0716c0 2023-03-03T07:42:27.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.515+0000 7f6461ffb700 1 -- 172.21.15.102:0/2321891519 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f64680575e0 con 0x7f646c0716c0 2023-03-03T07:42:27.516 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:42:27.516 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:42:27.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.517+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64500682f0 msgr2=0x7f645006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:42:27.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.517+0000 7f64739ff700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64500682f0 0x7f645006a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f646c0c5510 tx=0x7f645c004a50 comp rx=0 tx=0).stop 2023-03-03T07:42:27.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.517+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c0716c0 msgr2=0x7f646c14a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:42:27.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.517+0000 7f64739ff700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c0716c0 0x7f646c14a540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f64680090b0 tx=0x7f6468004750 comp rx=0 tx=0).stop 2023-03-03T07:42:27.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 shutdown_connections 2023-03-03T07:42:27.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64500682f0 0x7f645006a7a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:42:27.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f646c0716c0 0x7f646c14a540 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:42:27.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f646c14ffb0 0x7f646c143680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:42:27.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 --2- 172.21.15.102:0/2321891519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f646c14fbe0 0x7f646c1ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:42:27.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 >> 172.21.15.102:0/2321891519 conn(0x7f646c06af90 msgr2=0x7f646c155c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:42:27.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 shutdown_connections 2023-03-03T07:42:27.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:42:27.518+0000 7f64739ff700 1 -- 172.21.15.102:0/2321891519 wait complete. 2023-03-03T07:42:27.522 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:42:27.546 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T07:42:40.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:40.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:10.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:10.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:11.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:43:11.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.796+0000 7fd9f539d700 1 Processor -- start 2023-03-03T07:43:11.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.796+0000 7fd9f539d700 1 -- start start 2023-03-03T07:43:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9f539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f00cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9f539d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9f00c9610 0x7fd9f00cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9f539d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f00c9ca0 con 0x7fd9f00ca440 2023-03-03T07:43:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9f539d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f00b4920 con 0x7fd9f00c9610 2023-03-03T07:43:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9f539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00b4c00 con 0x7fd9f01564b0 2023-03-03T07:43:11.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9ef7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9f00c9610 0x7fd9f00cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:11.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9f4b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd9f00ca440 0x7fd9f00ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44306/0 (socket says 172.21.15.102:44306) 2023-03-03T07:43:11.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9f4b9c700 1 -- 172.21.15.102:0/3844107593 learned_addr learned my addr 172.21.15.102:0/3844107593 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:43:11.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.798+0000 7fd9effff700 1 --2- 172.21.15.102:0/3844107593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f00cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:11.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9ef7fe700 1 -- 172.21.15.102:0/3844107593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 msgr2=0x7fd9f00cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:43:11.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9ef7fe700 1 --2- 172.21.15.102:0/3844107593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f00cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:11.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9ef7fe700 1 -- 172.21.15.102:0/3844107593 >> v1:172.21.15.102:6789/0 conn(0x7fd9f00ca440 legacy=0x7fd9f00ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:11.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9ef7fe700 1 -- 172.21.15.102:0/3844107593 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f00b46f0 con 0x7fd9f00c9610 2023-03-03T07:43:11.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9eeffd700 1 -- 172.21.15.102:0/3844107593 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86307441 0 0) 0x7fd9f00c9ca0 con 0x7fd9f00ca440 2023-03-03T07:43:11.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9ef7fe700 1 --2- 172.21.15.102:0/3844107593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f00cbba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009840 tx=0x7fd9e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d729ba110c49e91b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:43:11.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9eeffd700 1 -- 172.21.15.102:0/3844107593 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4011830 con 0x7fd9f00c9610 2023-03-03T07:43:11.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9eeffd700 1 -- 172.21.15.102:0/3844107593 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e4011990 con 0x7fd9f00c9610 2023-03-03T07:43:11.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.799+0000 7fd9eeffd700 1 -- 172.21.15.102:0/3844107593 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4011d20 con 0x7fd9f00c9610 2023-03-03T07:43:11.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 -- 172.21.15.102:0/3844107593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 msgr2=0x7fd9f00cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:11.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3844107593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f00cbba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009840 tx=0x7fd9e4009b50 comp rx=0 tx=0).stop 2023-03-03T07:43:11.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 -- 172.21.15.102:0/3844107593 shutdown_connections 2023-03-03T07:43:11.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3844107593 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f00cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:11.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3844107593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f00cbba0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:11.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 -- 172.21.15.102:0/3844107593 >> 172.21.15.102:0/3844107593 conn(0x7fd9f0065fb0 msgr2=0x7fd9f00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:43:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 -- 172.21.15.102:0/3844107593 shutdown_connections 2023-03-03T07:43:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.800+0000 7fd9f539d700 1 -- 172.21.15.102:0/3844107593 wait complete. 2023-03-03T07:43:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.802+0000 7fd9f539d700 1 Processor -- start 2023-03-03T07:43:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.802+0000 7fd9f539d700 1 -- start start 2023-03-03T07:43:11.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.803+0000 7fd9f539d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f0157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:11.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.803+0000 7fd9f539d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f00ca440 0x7fd9f0150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:11.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.803+0000 7fd9f539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f0150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:11.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.803+0000 7fd9f539d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f01ef2e0 con 0x7fd9f00c9610 2023-03-03T07:43:11.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.803+0000 7fd9f539d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f0159220 con 0x7fd9f00ca440 2023-03-03T07:43:11.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.803+0000 7fd9f539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f01ef4d0 con 0x7fd9f01564b0 2023-03-03T07:43:11.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f0157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:11.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9f4b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f0150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:11.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9ef7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f00ca440 0x7fd9f0150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:11.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f0157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35348/0 (socket says 172.21.15.102:35348) 2023-03-03T07:43:11.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 -- 172.21.15.102:0/3235580164 learned_addr learned my addr 172.21.15.102:0/3235580164 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:43:11.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 -- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 msgr2=0x7fd9f0150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:11.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f0150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:11.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 -- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f00ca440 msgr2=0x7fd9f0150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:11.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f00ca440 0x7fd9f0150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:11.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 -- 172.21.15.102:0/3235580164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40094a0 con 0x7fd9f00c9610 2023-03-03T07:43:11.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.804+0000 7fd9effff700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f0157940 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000e870 tx=0x7fd9e000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:43:11.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.805+0000 7fd9ecff9700 1 -- 172.21.15.102:0/3235580164 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0009590 con 0x7fd9f00c9610 2023-03-03T07:43:11.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.805+0000 7fd9ecff9700 1 -- 172.21.15.102:0/3235580164 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e0013680 con 0x7fd9f00c9610 2023-03-03T07:43:11.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.805+0000 7fd9ecff9700 1 -- 172.21.15.102:0/3235580164 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e00139f0 con 0x7fd9f00c9610 2023-03-03T07:43:11.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.805+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f00b2210 con 0x7fd9f00c9610 2023-03-03T07:43:11.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.805+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f00b1f30 con 0x7fd9f00c9610 2023-03-03T07:43:11.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.806+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f0156880 con 0x7fd9f00c9610 2023-03-03T07:43:11.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.807+0000 7fd9ecff9700 1 -- 172.21.15.102:0/3235580164 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd9e0023070 con 0x7fd9f00c9610 2023-03-03T07:43:11.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.808+0000 7fd9ecff9700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9dc0682f0 0x7fd9dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:11.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.809+0000 7fd9ecff9700 1 -- 172.21.15.102:0/3235580164 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd9e0085f10 con 0x7fd9f00c9610 2023-03-03T07:43:11.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.809+0000 7fd9ef7fe700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9dc0682f0 0x7fd9dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:11.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.809+0000 7fd9ef7fe700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9dc0682f0 0x7fd9dc06a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40090b0 tx=0x7fd9e4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:43:11.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:11.813+0000 7fd9ecff9700 1 -- 172.21.15.102:0/3235580164 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9e0052530 con 0x7fd9f00c9610 2023-03-03T07:43:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.047+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9f0150710 con 0x7fd9f00c9610 2023-03-03T07:43:12.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.048+0000 7fd9ecff9700 1 -- 172.21.15.102:0/3235580164 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd9e0017090 con 0x7fd9f00c9610 2023-03-03T07:43:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:43:12.050 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:43:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9dc0682f0 msgr2=0x7fd9dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9dc0682f0 0x7fd9dc06a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40090b0 tx=0x7fd9e4008030 comp rx=0 tx=0).stop 2023-03-03T07:43:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 msgr2=0x7fd9f0157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f0157940 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000e870 tx=0x7fd9e000eb80 comp rx=0 tx=0).stop 2023-03-03T07:43:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 shutdown_connections 2023-03-03T07:43:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9dc0682f0 0x7fd9dc06a7a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9f01564b0 0x7fd9f0150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9f00c9610 0x7fd9f0157940 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 --2- 172.21.15.102:0/3235580164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9f00ca440 0x7fd9f0150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 >> 172.21.15.102:0/3235580164 conn(0x7fd9f0065fb0 msgr2=0x7fd9f00cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:43:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 shutdown_connections 2023-03-03T07:43:12.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:12.051+0000 7fd9f539d700 1 -- 172.21.15.102:0/3235580164 wait complete. 2023-03-03T07:43:12.054 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:43:12.079 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-03-03T07:43:17.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:43:17.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.327+0000 7fb767bff700 1 Processor -- start 2023-03-03T07:43:17.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.327+0000 7fb767bff700 1 -- start start 2023-03-03T07:43:17.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb767bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7600c5f30 0x7fb76006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb767bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7600c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb767bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7600c5390 con 0x7fb760150c80 2023-03-03T07:43:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb767bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb760157ad0 con 0x7fb7600c5f30 2023-03-03T07:43:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb767bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7600c4a20 con 0x7fb7600c2020 2023-03-03T07:43:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb76599b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb760150c80 0x7fb7600c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52536/0 (socket says 172.21.15.102:52536) 2023-03-03T07:43:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb76599b700 1 -- 172.21.15.102:0/3795571260 learned_addr learned my addr 172.21.15.102:0/3795571260 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:43:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb76519a700 1 --2- 172.21.15.102:0/3795571260 >> v2:172.21.15.102:3300/0 conn(0x7fb7600c5f30 0x7fb76006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:17.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.329+0000 7fb76619c700 1 --2- 172.21.15.102:0/3795571260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7600c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:17.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb76519a700 1 -- 172.21.15.102:0/3795571260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 msgr2=0x7fb7600c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:43:17.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb76519a700 1 --2- 172.21.15.102:0/3795571260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7600c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb76519a700 1 -- 172.21.15.102:0/3795571260 >> v1:172.21.15.102:6789/0 conn(0x7fb760150c80 legacy=0x7fb7600c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:17.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb76519a700 1 -- 172.21.15.102:0/3795571260 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76006f010 con 0x7fb7600c5f30 2023-03-03T07:43:17.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb76519a700 1 --2- 172.21.15.102:0/3795571260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76006d100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb750009840 tx=0x7fb750009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e66aba8f386d357 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:43:17.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb764999700 1 -- 172.21.15.102:0/3795571260 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750011830 con 0x7fb7600c5f30 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb764999700 1 -- 172.21.15.102:0/3795571260 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb750011990 con 0x7fb7600c5f30 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.330+0000 7fb764999700 1 -- 172.21.15.102:0/3795571260 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750011d20 con 0x7fb7600c5f30 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 -- 172.21.15.102:0/3795571260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 msgr2=0x7fb76006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 --2- 172.21.15.102:0/3795571260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76006d100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb750009840 tx=0x7fb750009b50 comp rx=0 tx=0).stop 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 -- 172.21.15.102:0/3795571260 shutdown_connections 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 --2- 172.21.15.102:0/3795571260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7600c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 --2- 172.21.15.102:0/3795571260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76006d100 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 -- 172.21.15.102:0/3795571260 >> 172.21.15.102:0/3795571260 conn(0x7fb76006af90 msgr2=0x7fb76006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:43:17.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 -- 172.21.15.102:0/3795571260 shutdown_connections 2023-03-03T07:43:17.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.331+0000 7fb767bff700 1 -- 172.21.15.102:0/3795571260 wait complete. 2023-03-03T07:43:17.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.333+0000 7fb767bff700 1 Processor -- start 2023-03-03T07:43:17.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.333+0000 7fb767bff700 1 -- start start 2023-03-03T07:43:17.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.334+0000 7fb767bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7601e4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:17.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.334+0000 7fb767bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:17.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.334+0000 7fb767bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb760150c80 0x7fb7601521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:17.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.334+0000 7fb767bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb760151a10 con 0x7fb7600c5f30 2023-03-03T07:43:17.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.334+0000 7fb767bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7601e5150 con 0x7fb760150c80 2023-03-03T07:43:17.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.334+0000 7fb767bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb760150220 con 0x7fb7600c2020 2023-03-03T07:43:17.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76599b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7601e4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76014e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40706/0 (socket says 172.21.15.102:40706) 2023-03-03T07:43:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 -- 172.21.15.102:0/3729799163 learned_addr learned my addr 172.21.15.102:0/3729799163 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:43:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76619c700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb760150c80 0x7fb7601521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 -- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 msgr2=0x7fb7601e4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7601e4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 -- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb760150c80 msgr2=0x7fb7601521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:17.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb760150c80 0x7fb7601521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 -- 172.21.15.102:0/3729799163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7500094a0 con 0x7fb7600c5f30 2023-03-03T07:43:17.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.335+0000 7fb76519a700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76014e940 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb750000c00 tx=0x7fb750004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:43:17.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.336+0000 7fb7567fc700 1 -- 172.21.15.102:0/3729799163 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500036a0 con 0x7fb7600c5f30 2023-03-03T07:43:17.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.336+0000 7fb7567fc700 1 -- 172.21.15.102:0/3729799163 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75001f070 con 0x7fb7600c5f30 2023-03-03T07:43:17.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.336+0000 7fb7567fc700 1 -- 172.21.15.102:0/3729799163 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750003a60 con 0x7fb7600c5f30 2023-03-03T07:43:17.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.336+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7601577f0 con 0x7fb7600c5f30 2023-03-03T07:43:17.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.336+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb760157510 con 0x7fb7600c5f30 2023-03-03T07:43:17.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.337+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7600c4cc0 con 0x7fb7600c5f30 2023-03-03T07:43:17.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.338+0000 7fb7567fc700 1 -- 172.21.15.102:0/3729799163 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb75000f050 con 0x7fb7600c5f30 2023-03-03T07:43:17.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.339+0000 7fb7567fc700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb744068320 0x7fb74406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:43:17.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.339+0000 7fb7567fc700 1 -- 172.21.15.102:0/3729799163 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb750021070 con 0x7fb7600c5f30 2023-03-03T07:43:17.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.340+0000 7fb76599b700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb744068320 0x7fb74406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:43:17.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.341+0000 7fb76599b700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb744068320 0x7fb74406a7d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005910 tx=0x7fb75c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:43:17.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.344+0000 7fb7567fc700 1 -- 172.21.15.102:0/3729799163 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb750053580 con 0x7fb7600c5f30 2023-03-03T07:43:17.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.577+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb76004e6e0 con 0x7fb7600c5f30 2023-03-03T07:43:17.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.578+0000 7fb7567fc700 1 -- 172.21.15.102:0/3729799163 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb750015070 con 0x7fb7600c5f30 2023-03-03T07:43:17.579 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:43:17.579 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:43:17.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb744068320 msgr2=0x7fb74406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:17.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb744068320 0x7fb74406a7d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005910 tx=0x7fb75c009040 comp rx=0 tx=0).stop 2023-03-03T07:43:17.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 msgr2=0x7fb76014e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:43:17.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76014e940 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb750000c00 tx=0x7fb750004750 comp rx=0 tx=0).stop 2023-03-03T07:43:17.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 shutdown_connections 2023-03-03T07:43:17.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb744068320 0x7fb74406a7d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7600c2020 0x7fb7601e4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb760150c80 0x7fb7601521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 --2- 172.21.15.102:0/3729799163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7600c5f30 0x7fb76014e940 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:43:17.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.580+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 >> 172.21.15.102:0/3729799163 conn(0x7fb76006af90 msgr2=0x7fb7600c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:43:17.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.581+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 shutdown_connections 2023-03-03T07:43:17.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:43:17.581+0000 7fb767bff700 1 -- 172.21.15.102:0/3729799163 wait complete. 2023-03-03T07:43:17.583 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:43:17.608 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T07:43:40.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:40.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:11.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:11.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:41.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:41.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:41.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:44:41.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.851+0000 7f113a99d700 1 Processor -- start 2023-03-03T07:44:41.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.851+0000 7f113a99d700 1 -- start start 2023-03-03T07:44:41.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.853+0000 7f113a99d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11340ced00 0x7f11340c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:44:41.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.853+0000 7f113a99d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340c3d20 con 0x7f11340c41b0 2023-03-03T07:44:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113a99d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113415cba0 con 0x7f11340b3970 2023-03-03T07:44:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113a99d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11340c4930 con 0x7f11340ced00 2023-03-03T07:44:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113999b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11340ced00 0x7f11340c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:44:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113a19c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f11340c41b0 0x7f1134163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36090/0 (socket says 172.21.15.102:36090) 2023-03-03T07:44:41.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113a19c700 1 -- 172.21.15.102:0/2343938517 learned_addr learned my addr 172.21.15.102:0/2343938517 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:44:41.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f1138999700 1 -- 172.21.15.102:0/2343938517 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055280208 0 0) 0x7f113415cba0 con 0x7f11340b3970 2023-03-03T07:44:41.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113999b700 1 -- 172.21.15.102:0/2343938517 >> v1:172.21.15.172:6789/0 conn(0x7f11340b3970 legacy=0x7f1134158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:44:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113999b700 1 -- 172.21.15.102:0/2343938517 >> v1:172.21.15.102:6789/0 conn(0x7f11340c41b0 legacy=0x7f1134163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:44:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.854+0000 7f113999b700 1 -- 172.21.15.102:0/2343938517 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113415ce80 con 0x7f11340ced00 2023-03-03T07:44:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.855+0000 7f113999b700 1 --2- 172.21.15.102:0/2343938517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340ced00 0x7f11340c5a90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1130009840 tx=0x7f1130009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385f55db147e9f6d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:44:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.855+0000 7f1138999700 1 -- 172.21.15.102:0/2343938517 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1130004230 con 0x7f11340ced00 2023-03-03T07:44:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.855+0000 7f1138999700 1 -- 172.21.15.102:0/2343938517 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1130004390 con 0x7f11340ced00 2023-03-03T07:44:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.855+0000 7f1138999700 1 -- 172.21.15.102:0/2343938517 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11300036a0 con 0x7f11340ced00 2023-03-03T07:44:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.855+0000 7f113a99d700 1 -- 172.21.15.102:0/2343938517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340ced00 msgr2=0x7f11340c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:44:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.855+0000 7f113a99d700 1 --2- 172.21.15.102:0/2343938517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340ced00 0x7f11340c5a90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1130009840 tx=0x7f1130009b50 comp rx=0 tx=0).stop 2023-03-03T07:44:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.856+0000 7f113a99d700 1 -- 172.21.15.102:0/2343938517 shutdown_connections 2023-03-03T07:44:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.856+0000 7f113a99d700 1 --2- 172.21.15.102:0/2343938517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340ced00 0x7f11340c5a90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:44:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.856+0000 7f113a99d700 1 -- 172.21.15.102:0/2343938517 >> 172.21.15.102:0/2343938517 conn(0x7f113408b1c0 msgr2=0x7f1134071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:44:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.856+0000 7f113a99d700 1 -- 172.21.15.102:0/2343938517 shutdown_connections 2023-03-03T07:44:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.856+0000 7f113a99d700 1 -- 172.21.15.102:0/2343938517 wait complete. 2023-03-03T07:44:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.858+0000 7f113a99d700 1 Processor -- start 2023-03-03T07:44:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.858+0000 7f113a99d700 1 -- start start 2023-03-03T07:44:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113a99d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11340b3970 0x7f11340c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:44:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113a99d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340c41b0 0x7f11340ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:44:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113a99d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11340ced00 0x7f11340ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:44:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113a99d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11340ce9b0 con 0x7f11340c41b0 2023-03-03T07:44:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113a99d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11340c6070 con 0x7f11340ced00 2023-03-03T07:44:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113a99d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11340c6330 con 0x7f11340b3970 2023-03-03T07:44:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113999b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11340b3970 0x7f11340c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:44:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113919a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340c41b0 0x7f11340ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:44:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113a19c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11340ced00 0x7f11340ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:44:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113999b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11340b3970 0x7f11340c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44976/0 (socket says 172.21.15.102:44976) 2023-03-03T07:44:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.859+0000 7f113999b700 1 -- 172.21.15.102:0/1367294646 learned_addr learned my addr 172.21.15.102:0/1367294646 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:44:41.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113919a700 1 -- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11340b3970 msgr2=0x7f11340c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:44:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113919a700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11340b3970 0x7f11340c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:44:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113919a700 1 -- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11340ced00 msgr2=0x7f11340ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:44:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113919a700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11340ced00 0x7f11340ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:44:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113919a700 1 -- 172.21.15.102:0/1367294646 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11300094a0 con 0x7f11340c41b0 2023-03-03T07:44:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113999b700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11340b3970 0x7f11340c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:44:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113a19c700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11340ced00 0x7f11340ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:44:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113919a700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340c41b0 0x7f11340ce470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f60 tx=0x7f112400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:44:41.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f112a7fc700 1 -- 172.21.15.102:0/1367294646 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11240052d0 con 0x7f11340c41b0 2023-03-03T07:44:41.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f112a7fc700 1 -- 172.21.15.102:0/1367294646 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1124005450 con 0x7f11340c41b0 2023-03-03T07:44:41.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f112a7fc700 1 -- 172.21.15.102:0/1367294646 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11240038c0 con 0x7f11340c41b0 2023-03-03T07:44:41.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.860+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113406c200 con 0x7f11340c41b0 2023-03-03T07:44:41.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.861+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113406bf20 con 0x7f11340c41b0 2023-03-03T07:44:41.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.862+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113415c890 con 0x7f11340c41b0 2023-03-03T07:44:41.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.863+0000 7f112a7fc700 1 -- 172.21.15.102:0/1367294646 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1124019030 con 0x7f11340c41b0 2023-03-03T07:44:41.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.865+0000 7f112a7fc700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11180682f0 0x7f111806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:44:41.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.865+0000 7f112a7fc700 1 -- 172.21.15.102:0/1367294646 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1124084e10 con 0x7f11340c41b0 2023-03-03T07:44:41.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.865+0000 7f113999b700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11180682f0 0x7f111806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:44:41.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.866+0000 7f113999b700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11180682f0 0x7f111806a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1130000c00 tx=0x7f1130008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:44:41.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:41.869+0000 7f112a7fc700 1 -- 172.21.15.102:0/1367294646 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1124051610 con 0x7f11340c41b0 2023-03-03T07:44:42.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.104+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f113404e6e0 con 0x7f11340c41b0 2023-03-03T07:44:42.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.104+0000 7f112a7fc700 1 -- 172.21.15.102:0/1367294646 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1124017090 con 0x7f11340c41b0 2023-03-03T07:44:42.105 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:44:42.106 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:44:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11180682f0 msgr2=0x7f111806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:44:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11180682f0 0x7f111806a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1130000c00 tx=0x7f1130008030 comp rx=0 tx=0).stop 2023-03-03T07:44:42.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340c41b0 msgr2=0x7f11340ce470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:44:42.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340c41b0 0x7f11340ce470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f60 tx=0x7f112400c6a0 comp rx=0 tx=0).stop 2023-03-03T07:44:42.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 shutdown_connections 2023-03-03T07:44:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11180682f0 0x7f111806a7a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:44:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11340b3970 0x7f11340c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:44:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11340ced00 0x7f11340ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:44:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 --2- 172.21.15.102:0/1367294646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11340c41b0 0x7f11340ce470 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:44:42.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 >> 172.21.15.102:0/1367294646 conn(0x7f113408b1c0 msgr2=0x7f1134071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:44:42.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 shutdown_connections 2023-03-03T07:44:42.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:44:42.107+0000 7f113a99d700 1 -- 172.21.15.102:0/1367294646 wait complete. 2023-03-03T07:44:42.111 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:44:42.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T07:45:11.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:45:11.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:45:40.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:45:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.387+0000 7f20d1d9d700 1 Processor -- start 2023-03-03T07:45:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.387+0000 7f20d1d9d700 1 -- start start 2023-03-03T07:45:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d1d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f20cc0c4650 0x7f20cc06b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:45:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d1d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f20cc06b7e0 0x7f20cc15c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:45:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d1d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc1597b0 con 0x7f20cc152960 2023-03-03T07:45:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d1d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc0c9cb0 con 0x7f20cc06b7e0 2023-03-03T07:45:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d1d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc152d30 con 0x7f20cc0c4650 2023-03-03T07:45:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d0d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f20cc152960 0x7f20cc0cbd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33712/0 (socket says 172.21.15.102:33712) 2023-03-03T07:45:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d0d9b700 1 -- 172.21.15.102:0/2904475704 learned_addr learned my addr 172.21.15.102:0/2904475704 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:45:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20d159c700 1 --2- 172.21.15.102:0/2904475704 >> v2:172.21.15.102:3300/0 conn(0x7f20cc06b7e0 0x7f20cc15c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:45:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.389+0000 7f20cbfff700 1 --2- 172.21.15.102:0/2904475704 >> v2:172.21.15.172:3300/0 conn(0x7f20cc0c4650 0x7f20cc06b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:45:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20d159c700 1 -- 172.21.15.102:0/2904475704 >> v2:172.21.15.172:3300/0 conn(0x7f20cc0c4650 msgr2=0x7f20cc06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:45:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20d159c700 1 --2- 172.21.15.102:0/2904475704 >> v2:172.21.15.172:3300/0 conn(0x7f20cc0c4650 0x7f20cc06b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20d159c700 1 -- 172.21.15.102:0/2904475704 >> v1:172.21.15.102:6789/0 conn(0x7f20cc152960 legacy=0x7f20cc0cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:45:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20d159c700 1 -- 172.21.15.102:0/2904475704 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc158950 con 0x7f20cc06b7e0 2023-03-03T07:45:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20d159c700 1 --2- 172.21.15.102:0/2904475704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc06b7e0 0x7f20cc15c110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0cfca0 tx=0x7f20c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9659fc8985fc47 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:45:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20cb7fe700 1 -- 172.21.15.102:0/2904475704 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c400e070 con 0x7f20cc06b7e0 2023-03-03T07:45:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20cb7fe700 1 -- 172.21.15.102:0/2904475704 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c4003a20 con 0x7f20cc06b7e0 2023-03-03T07:45:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.390+0000 7f20cb7fe700 1 -- 172.21.15.102:0/2904475704 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4003dd0 con 0x7f20cc06b7e0 2023-03-03T07:45:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904475704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc06b7e0 msgr2=0x7f20cc15c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:45:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904475704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc06b7e0 0x7f20cc15c110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0cfca0 tx=0x7f20c400b330 comp rx=0 tx=0).stop 2023-03-03T07:45:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904475704 shutdown_connections 2023-03-03T07:45:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904475704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc06b7e0 0x7f20cc15c110 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904475704 >> v2:172.21.15.172:3300/0 conn(0x7f20cc0c4650 0x7f20cc06b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904475704 >> 172.21.15.102:0/2904475704 conn(0x7f20cc06c1a0 msgr2=0x7f20cc06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:45:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904475704 shutdown_connections 2023-03-03T07:45:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.391+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904475704 wait complete. 2023-03-03T07:45:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.393+0000 7f20d1d9d700 1 Processor -- start 2023-03-03T07:45:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.393+0000 7f20d1d9d700 1 -- start start 2023-03-03T07:45:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.394+0000 7f20d1d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20cc06b7e0 0x7f20cc0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:45:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.394+0000 7f20d1d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc0c4650 0x7f20cc0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:45:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.394+0000 7f20d1d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc152960 0x7f20cc13c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:45:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.394+0000 7f20d1d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc157760 con 0x7f20cc0c4650 2023-03-03T07:45:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.394+0000 7f20d1d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0b2980 con 0x7f20cc152960 2023-03-03T07:45:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.394+0000 7f20d1d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc1e4c90 con 0x7f20cc06b7e0 2023-03-03T07:45:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20cc06b7e0 0x7f20cc0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:45:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d159c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc152960 0x7f20cc13c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:45:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20cc06b7e0 0x7f20cc0cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:50086/0 (socket says 172.21.15.102:50086) 2023-03-03T07:45:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 -- 172.21.15.102:0/2904565318 learned_addr learned my addr 172.21.15.102:0/2904565318 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:45:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20cbfff700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc0c4650 0x7f20cc0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:45:40.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 -- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc152960 msgr2=0x7f20cc13c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:45:40.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc152960 0x7f20cc13c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 -- 172.21.15.102:0/2904565318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc0c4650 msgr2=0x7f20cc0cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:45:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc0c4650 0x7f20cc0cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d0d9b700 1 -- 172.21.15.102:0/2904565318 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4009a70 con 0x7f20cc06b7e0 2023-03-03T07:45:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.395+0000 7f20d159c700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc152960 0x7f20cc13c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:45:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.396+0000 7f20d0d9b700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20cc06b7e0 0x7f20cc0cd060 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c7b0 tx=0x7f20c000cac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:45:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.396+0000 7f20c97fa700 1 -- 172.21.15.102:0/2904565318 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0004c80 con 0x7f20cc06b7e0 2023-03-03T07:45:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.396+0000 7f20c97fa700 1 -- 172.21.15.102:0/2904565318 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0004de0 con 0x7f20cc06b7e0 2023-03-03T07:45:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.396+0000 7f20c97fa700 1 -- 172.21.15.102:0/2904565318 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c00072f0 con 0x7f20cc06b7e0 2023-03-03T07:45:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.396+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc158f10 con 0x7f20cc06b7e0 2023-03-03T07:45:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.396+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc158c30 con 0x7f20cc06b7e0 2023-03-03T07:45:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.397+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20cc1591f0 con 0x7f20cc06b7e0 2023-03-03T07:45:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.398+0000 7f20c97fa700 1 -- 172.21.15.102:0/2904565318 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f20c001a020 con 0x7f20cc06b7e0 2023-03-03T07:45:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.400+0000 7f20c97fa700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20b00682f0 0x7f20b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:45:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.400+0000 7f20c97fa700 1 -- 172.21.15.102:0/2904565318 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f20c0085980 con 0x7f20cc06b7e0 2023-03-03T07:45:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.400+0000 7f20cbfff700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20b00682f0 0x7f20b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:45:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.401+0000 7f20cbfff700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20b00682f0 0x7f20b006a7a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0cb570 tx=0x7f20b8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:45:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.404+0000 7f20c97fa700 1 -- 172.21.15.102:0/2904565318 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f20c0052090 con 0x7f20cc06b7e0 2023-03-03T07:45:40.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.640+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20cc0677f0 con 0x7f20cc06b7e0 2023-03-03T07:45:40.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.641+0000 7f20c97fa700 1 -- 172.21.15.102:0/2904565318 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f20c00556b0 con 0x7f20cc06b7e0 2023-03-03T07:45:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:45:40.646 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:45:40.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20b00682f0 msgr2=0x7f20b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:45:40.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20b00682f0 0x7f20b006a7a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0cb570 tx=0x7f20b8009210 comp rx=0 tx=0).stop 2023-03-03T07:45:40.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20cc06b7e0 msgr2=0x7f20cc0cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:45:40.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20cc06b7e0 0x7f20cc0cd060 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c7b0 tx=0x7f20c000cac0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 shutdown_connections 2023-03-03T07:45:40.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20b00682f0 0x7f20b006a7a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20cc06b7e0 0x7f20cc0cd060 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc152960 0x7f20cc13c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 --2- 172.21.15.102:0/2904565318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20cc0c4650 0x7f20cc0cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:45:40.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 >> 172.21.15.102:0/2904565318 conn(0x7f20cc06c1a0 msgr2=0x7f20cc0b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:45:40.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 shutdown_connections 2023-03-03T07:45:40.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:45:40.644+0000 7f20d1d9d700 1 -- 172.21.15.102:0/2904565318 wait complete. 2023-03-03T07:45:40.651 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:45:40.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2023-03-03T07:45:41.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:45:41.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:11.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:11.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:42.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:42.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:12.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:12.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:37.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:47:37.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.939+0000 7ff2c91ff700 1 Processor -- start 2023-03-03T07:47:37.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.939+0000 7ff2c91ff700 1 -- start start 2023-03-03T07:47:37.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.940+0000 7ff2c91ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2c4153260 0x7ff2c40ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:47:37.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.940+0000 7ff2c91ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff2c40c9cb0 0x7ff2c40cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:47:37.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c91ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c40cf940 con 0x7ff2c40cac20 2023-03-03T07:47:37.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c91ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c40cfe70 con 0x7ff2c4153260 2023-03-03T07:47:37.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c91ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c4158d70 con 0x7ff2c40c9cb0 2023-03-03T07:47:37.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c2d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2c4153260 0x7ff2c40ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:47:37.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff2c40c9cb0 0x7ff2c40cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:47:37.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c359e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff2c40cac20 0x7ff2c415c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42902/0 (socket says 172.21.15.102:42902) 2023-03-03T07:47:37.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c359e700 1 -- 172.21.15.102:0/1097573640 learned_addr learned my addr 172.21.15.102:0/1097573640 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:47:37.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c2d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2c4153260 0x7ff2c40ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48272/0 (socket says 172.21.15.102:48272) 2023-03-03T07:47:37.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c259c700 1 -- 172.21.15.102:0/1097573640 >> v2:172.21.15.102:3300/0 conn(0x7ff2c4153260 msgr2=0x7ff2c40ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:47:37.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c259c700 1 --2- 172.21.15.102:0/1097573640 >> v2:172.21.15.102:3300/0 conn(0x7ff2c4153260 0x7ff2c40ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c259c700 1 -- 172.21.15.102:0/1097573640 >> v1:172.21.15.172:6789/0 conn(0x7ff2c40cac20 legacy=0x7ff2c415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:47:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c259c700 1 -- 172.21.15.102:0/1097573640 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c40cfb90 con 0x7ff2c40c9cb0 2023-03-03T07:47:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c2d9d700 1 --2- 172.21.15.102:0/1097573640 >> v2:172.21.15.102:3300/0 conn(0x7ff2c4153260 0x7ff2c40ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:47:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.941+0000 7ff2c1d9b700 1 -- 172.21.15.102:0/1097573640 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445450835 0 0) 0x7ff2c40cf940 con 0x7ff2c40cac20 2023-03-03T07:47:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.942+0000 7ff2c259c700 1 --2- 172.21.15.102:0/1097573640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40c9cb0 0x7ff2c40cb880 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff2c408cd80 tx=0x7ff2b8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e23f53f3ce5da6e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:47:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.942+0000 7ff2c1d9b700 1 -- 172.21.15.102:0/1097573640 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8011830 con 0x7ff2c40c9cb0 2023-03-03T07:47:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.942+0000 7ff2c1d9b700 1 -- 172.21.15.102:0/1097573640 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b8011990 con 0x7ff2c40c9cb0 2023-03-03T07:47:37.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.942+0000 7ff2c1d9b700 1 -- 172.21.15.102:0/1097573640 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8011d20 con 0x7ff2c40c9cb0 2023-03-03T07:47:37.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1097573640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40c9cb0 msgr2=0x7ff2c40cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:47:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1097573640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40c9cb0 0x7ff2c40cb880 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff2c408cd80 tx=0x7ff2b8009b20 comp rx=0 tx=0).stop 2023-03-03T07:47:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1097573640 shutdown_connections 2023-03-03T07:47:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1097573640 >> v2:172.21.15.102:3300/0 conn(0x7ff2c4153260 0x7ff2c40ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1097573640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40c9cb0 0x7ff2c40cb880 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1097573640 >> 172.21.15.102:0/1097573640 conn(0x7ff2c4066c00 msgr2=0x7ff2c40ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:47:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1097573640 shutdown_connections 2023-03-03T07:47:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.943+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1097573640 wait complete. 2023-03-03T07:47:37.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.945+0000 7ff2c91ff700 1 Processor -- start 2023-03-03T07:47:37.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.945+0000 7ff2c91ff700 1 -- start start 2023-03-03T07:47:37.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c91ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40cac20 0x7ff2c40b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:47:37.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c91ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4153260 0x7ff2c41e3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:47:37.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c91ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2c414c670 0x7ff2c41e3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:47:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c91ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c41e4f80 con 0x7ff2c4153260 2023-03-03T07:47:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c91ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c414e1c0 con 0x7ff2c40cac20 2023-03-03T07:47:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c91ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c414e420 con 0x7ff2c414c670 2023-03-03T07:47:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c2d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40cac20 0x7ff2c40b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:47:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c2d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40cac20 0x7ff2c40b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37526/0 (socket says 172.21.15.102:37526) 2023-03-03T07:47:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c2d9d700 1 -- 172.21.15.102:0/1231188607 learned_addr learned my addr 172.21.15.102:0/1231188607 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:47:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c259c700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4153260 0x7ff2c41e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:47:37.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c359e700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2c414c670 0x7ff2c41e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:47:37.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.946+0000 7ff2c2d9d700 1 -- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2c414c670 msgr2=0x7ff2c41e3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:47:37.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c2d9d700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2c414c670 0x7ff2c41e3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:37.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c2d9d700 1 -- 172.21.15.102:0/1231188607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4153260 msgr2=0x7ff2c41e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:47:37.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c2d9d700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4153260 0x7ff2c41e3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:37.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c2d9d700 1 -- 172.21.15.102:0/1231188607 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b40094a0 con 0x7ff2c40cac20 2023-03-03T07:47:37.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c359e700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2c414c670 0x7ff2c41e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:47:37.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c2d9d700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40cac20 0x7ff2c40b4250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400ebe0 tx=0x7ff2b400eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:47:37.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2ab7fe700 1 -- 172.21.15.102:0/1231188607 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4009800 con 0x7ff2c40cac20 2023-03-03T07:47:37.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2ab7fe700 1 -- 172.21.15.102:0/1231188607 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b400c910 con 0x7ff2c40cac20 2023-03-03T07:47:37.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2ab7fe700 1 -- 172.21.15.102:0/1231188607 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b400cc80 con 0x7ff2c40cac20 2023-03-03T07:47:37.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b80094a0 con 0x7ff2c40cac20 2023-03-03T07:47:37.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.947+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c4159260 con 0x7ff2c40cac20 2023-03-03T07:47:37.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.948+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c4158f80 con 0x7ff2c40cac20 2023-03-03T07:47:37.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.950+0000 7ff2ab7fe700 1 -- 172.21.15.102:0/1231188607 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff2b4019030 con 0x7ff2c40cac20 2023-03-03T07:47:37.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.951+0000 7ff2ab7fe700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2b00682f0 0x7ff2b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:47:37.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.951+0000 7ff2ab7fe700 1 -- 172.21.15.102:0/1231188607 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff2b4085ea0 con 0x7ff2c40cac20 2023-03-03T07:47:37.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.951+0000 7ff2c259c700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2b00682f0 0x7ff2b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:47:37.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.952+0000 7ff2c259c700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2b00682f0 0x7ff2b006a7a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80090b0 tx=0x7ff2b801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:47:37.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:37.954+0000 7ff2ab7fe700 1 -- 172.21.15.102:0/1231188607 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff2b40525a0 con 0x7ff2c40cac20 2023-03-03T07:47:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.191+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2c4159540 con 0x7ff2c40cac20 2023-03-03T07:47:38.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.192+0000 7ff2ab7fe700 1 -- 172.21.15.102:0/1231188607 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff2b4021090 con 0x7ff2c40cac20 2023-03-03T07:47:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:47:38.194 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:47:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2b00682f0 msgr2=0x7ff2b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:47:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2b00682f0 0x7ff2b006a7a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80090b0 tx=0x7ff2b801c040 comp rx=0 tx=0).stop 2023-03-03T07:47:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40cac20 msgr2=0x7ff2c40b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:47:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40cac20 0x7ff2c40b4250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400ebe0 tx=0x7ff2b400eef0 comp rx=0 tx=0).stop 2023-03-03T07:47:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 shutdown_connections 2023-03-03T07:47:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2b00682f0 0x7ff2b006a7a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2c414c670 0x7ff2c41e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2c40cac20 0x7ff2c40b4250 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 --2- 172.21.15.102:0/1231188607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2c4153260 0x7ff2c41e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:47:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.194+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 >> 172.21.15.102:0/1231188607 conn(0x7ff2c40c9cb0 msgr2=0x7ff2c40b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:47:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.195+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 shutdown_connections 2023-03-03T07:47:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:47:38.195+0000 7ff2c91ff700 1 -- 172.21.15.102:0/1231188607 wait complete. 2023-03-03T07:47:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:47:38.223 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T07:47:42.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:42.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:10.224 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:48:10.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.473+0000 7fa780d9d700 1 Processor -- start 2023-03-03T07:48:10.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.473+0000 7fa780d9d700 1 -- start start 2023-03-03T07:48:10.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa780d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa77c0ced00 0x7fa77c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:48:10.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa780d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa77c0c41b0 0x7fa77c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:48:10.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa780d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0c3d20 con 0x7fa77c0b3970 2023-03-03T07:48:10.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa780d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c0c4930 con 0x7fa77c0ced00 2023-03-03T07:48:10.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa780d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c0cf610 con 0x7fa77c0c41b0 2023-03-03T07:48:10.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa77b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa77c0ced00 0x7fa77c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:48:10.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa77bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa77c0c41b0 0x7fa77c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:48:10.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa77b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa77c0ced00 0x7fa77c0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50382/0 (socket says 172.21.15.102:50382) 2023-03-03T07:48:10.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa77b7fe700 1 -- 172.21.15.102:0/1786863705 learned_addr learned my addr 172.21.15.102:0/1786863705 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:48:10.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.475+0000 7fa77affd700 1 --1- 172.21.15.102:0/1786863705 >> v1:172.21.15.102:6789/0 conn(0x7fa77c0b3970 0x7fa77c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41596/0 (socket says 172.21.15.102:41596) 2023-03-03T07:48:10.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77b7fe700 1 -- 172.21.15.102:0/1786863705 >> v2:172.21.15.172:3300/0 conn(0x7fa77c0c41b0 msgr2=0x7fa77c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:48:10.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77b7fe700 1 --2- 172.21.15.102:0/1786863705 >> v2:172.21.15.172:3300/0 conn(0x7fa77c0c41b0 0x7fa77c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77b7fe700 1 -- 172.21.15.102:0/1786863705 >> v1:172.21.15.102:6789/0 conn(0x7fa77c0b3970 legacy=0x7fa77c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:48:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77b7fe700 1 -- 172.21.15.102:0/1786863705 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c15cba0 con 0x7fa77c0ced00 2023-03-03T07:48:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77a7fc700 1 -- 172.21.15.102:0/1786863705 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172264800 0 0) 0x7fa77c0c3d20 con 0x7fa77c0b3970 2023-03-03T07:48:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77b7fe700 1 --2- 172.21.15.102:0/1786863705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0ced00 0x7fa77c0c5a90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0ca080 tx=0x7fa76c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15874240703c81a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:48:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77a7fc700 1 -- 172.21.15.102:0/1786863705 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c011830 con 0x7fa77c0ced00 2023-03-03T07:48:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77a7fc700 1 -- 172.21.15.102:0/1786863705 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c011990 con 0x7fa77c0ced00 2023-03-03T07:48:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.476+0000 7fa77a7fc700 1 -- 172.21.15.102:0/1786863705 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c011d20 con 0x7fa77c0ced00 2023-03-03T07:48:10.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 -- 172.21.15.102:0/1786863705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0ced00 msgr2=0x7fa77c0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:48:10.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1786863705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0ced00 0x7fa77c0c5a90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0ca080 tx=0x7fa76c009c00 comp rx=0 tx=0).stop 2023-03-03T07:48:10.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 -- 172.21.15.102:0/1786863705 shutdown_connections 2023-03-03T07:48:10.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1786863705 >> v2:172.21.15.172:3300/0 conn(0x7fa77c0c41b0 0x7fa77c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1786863705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0ced00 0x7fa77c0c5a90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 -- 172.21.15.102:0/1786863705 >> 172.21.15.102:0/1786863705 conn(0x7fa77c08b1c0 msgr2=0x7fa77c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:48:10.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 -- 172.21.15.102:0/1786863705 shutdown_connections 2023-03-03T07:48:10.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.477+0000 7fa780d9d700 1 -- 172.21.15.102:0/1786863705 wait complete. 2023-03-03T07:48:10.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.479+0000 7fa780d9d700 1 Processor -- start 2023-03-03T07:48:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.479+0000 7fa780d9d700 1 -- start start 2023-03-03T07:48:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.480+0000 7fa780d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0b3970 0x7fa77c0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:48:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa780d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa77c0c41b0 0x7fa77c0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:48:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa780d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa77c0ced00 0x7fa77c14d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:48:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa780d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0c8460 con 0x7fa77c0b3970 2023-03-03T07:48:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa780d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0b2540 con 0x7fa77c0ced00 2023-03-03T07:48:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa780d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0b1f30 con 0x7fa77c0c41b0 2023-03-03T07:48:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0b3970 0x7fa77c0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:48:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa77c0c41b0 0x7fa77c0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:48:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0b3970 0x7fa77c0c6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50398/0 (socket says 172.21.15.102:50398) 2023-03-03T07:48:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa77c0ced00 0x7fa77c14d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:48:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77b7fe700 1 -- 172.21.15.102:0/1458976411 learned_addr learned my addr 172.21.15.102:0/1458976411 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:48:10.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77bfff700 1 -- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa77c0c41b0 msgr2=0x7fa77c0c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:48:10.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77bfff700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa77c0c41b0 0x7fa77c0c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77bfff700 1 -- 172.21.15.102:0/1458976411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0b3970 msgr2=0x7fa77c0c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:48:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77bfff700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0b3970 0x7fa77c0c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77bfff700 1 -- 172.21.15.102:0/1458976411 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c009550 con 0x7fa77c0ced00 2023-03-03T07:48:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.481+0000 7fa77b7fe700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0b3970 0x7fa77c0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:48:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.482+0000 7fa77bfff700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa77c0ced00 0x7fa77c14d9b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa77400eba0 tx=0x7fa77400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:48:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.482+0000 7fa763fff700 1 -- 172.21.15.102:0/1458976411 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774009a80 con 0x7fa77c0ced00 2023-03-03T07:48:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.482+0000 7fa763fff700 1 -- 172.21.15.102:0/1458976411 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77400f040 con 0x7fa77c0ced00 2023-03-03T07:48:10.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.482+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c06bf20 con 0x7fa77c0ced00 2023-03-03T07:48:10.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.482+0000 7fa763fff700 1 -- 172.21.15.102:0/1458976411 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77400cc40 con 0x7fa77c0ced00 2023-03-03T07:48:10.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.482+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c15ce80 con 0x7fa77c0ced00 2023-03-03T07:48:10.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.483+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c06c200 con 0x7fa77c0ced00 2023-03-03T07:48:10.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.484+0000 7fa763fff700 1 -- 172.21.15.102:0/1458976411 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa774024070 con 0x7fa77c0ced00 2023-03-03T07:48:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.486+0000 7fa763fff700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa7680682f0 0x7fa76806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:48:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.486+0000 7fa763fff700 1 -- 172.21.15.102:0/1458976411 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa774086750 con 0x7fa77c0ced00 2023-03-03T07:48:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.486+0000 7fa77b7fe700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa7680682f0 0x7fa76806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:48:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.487+0000 7fa77b7fe700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa7680682f0 0x7fa76806a7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009130 tx=0x7fa76c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:48:10.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.490+0000 7fa763fff700 1 -- 172.21.15.102:0/1458976411 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa774052e00 con 0x7fa77c0ced00 2023-03-03T07:48:10.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.725+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa77c0c9ef0 con 0x7fa77c0ced00 2023-03-03T07:48:10.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.726+0000 7fa763fff700 1 -- 172.21.15.102:0/1458976411 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa774056420 con 0x7fa77c0ced00 2023-03-03T07:48:10.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:48:10.728 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:48:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.728+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa7680682f0 msgr2=0x7fa76806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:48:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.728+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa7680682f0 0x7fa76806a7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009130 tx=0x7fa76c008030 comp rx=0 tx=0).stop 2023-03-03T07:48:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.728+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa77c0ced00 msgr2=0x7fa77c14d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:48:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.728+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa77c0ced00 0x7fa77c14d9b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa77400eba0 tx=0x7fa77400eeb0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 shutdown_connections 2023-03-03T07:48:10.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa77c0ced00 0x7fa77c14d9b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa77c0b3970 0x7fa77c0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa7680682f0 0x7fa76806a7a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 --2- 172.21.15.102:0/1458976411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa77c0c41b0 0x7fa77c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:48:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 >> 172.21.15.102:0/1458976411 conn(0x7fa77c08b1c0 msgr2=0x7fa77c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:48:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 shutdown_connections 2023-03-03T07:48:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:48:10.729+0000 7fa780d9d700 1 -- 172.21.15.102:0/1458976411 wait complete. 2023-03-03T07:48:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:48:10.756 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-03-03T07:48:12.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:12.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:42.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:42.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:12.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:12.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:43.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:43.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:45.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:49:46.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.004+0000 7fdfa53ff700 1 Processor -- start 2023-03-03T07:49:46.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.004+0000 7fdfa53ff700 1 -- start start 2023-03-03T07:49:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.005+0000 7fdfa53ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdfa0153290 0x7fdfa0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:49:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdfa53ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdfa01593c0 0x7fdfa00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:49:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdfa53ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa00cd8e0 0x7fdfa00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:49:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdfa53ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa0153660 con 0x7fdfa0153290 2023-03-03T07:49:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdfa53ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa006c360 con 0x7fdfa01593c0 2023-03-03T07:49:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdfa53ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa015cea0 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdfa0153290 0x7fdfa0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:49:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdfa0153290 0x7fdfa0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53818/0 (socket says 172.21.15.102:53818) 2023-03-03T07:49:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 -- 172.21.15.102:0/2544295803 learned_addr learned my addr 172.21.15.102:0/2544295803 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:49:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9e7fc700 1 --2- 172.21.15.102:0/2544295803 >> v2:172.21.15.172:3300/0 conn(0x7fdfa01593c0 0x7fdfa00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:49:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9f7fe700 1 --2- 172.21.15.102:0/2544295803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa00cd8e0 0x7fdfa00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 -- 172.21.15.102:0/2544295803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa00cd8e0 msgr2=0x7fdfa00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 --2- 172.21.15.102:0/2544295803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa00cd8e0 0x7fdfa00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 -- 172.21.15.102:0/2544295803 >> v2:172.21.15.172:3300/0 conn(0x7fdfa01593c0 msgr2=0x7fdfa00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 --2- 172.21.15.102:0/2544295803 >> v2:172.21.15.172:3300/0 conn(0x7fdfa01593c0 0x7fdfa00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.006+0000 7fdf9effd700 1 -- 172.21.15.102:0/2544295803 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa006c7f0 con 0x7fdfa0153290 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.007+0000 7fdf9effd700 1 --2- 172.21.15.102:0/2544295803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa0153290 0x7fdfa0158e80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0159fb0 tx=0x7fdf90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563ef2a8245f7ba0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.007+0000 7fdf9dffb700 1 -- 172.21.15.102:0/2544295803 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90011830 con 0x7fdfa0153290 2023-03-03T07:49:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.007+0000 7fdf9dffb700 1 -- 172.21.15.102:0/2544295803 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf90011990 con 0x7fdfa0153290 2023-03-03T07:49:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.007+0000 7fdf9dffb700 1 -- 172.21.15.102:0/2544295803 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90011d20 con 0x7fdfa0153290 2023-03-03T07:49:46.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 -- 172.21.15.102:0/2544295803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa0153290 msgr2=0x7fdfa0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:49:46.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/2544295803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa0153290 0x7fdfa0158e80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0159fb0 tx=0x7fdf90009bd0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 -- 172.21.15.102:0/2544295803 shutdown_connections 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/2544295803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa00cd8e0 0x7fdfa00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/2544295803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa0153290 0x7fdfa0158e80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/2544295803 >> v2:172.21.15.172:3300/0 conn(0x7fdfa01593c0 0x7fdfa00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 -- 172.21.15.102:0/2544295803 >> 172.21.15.102:0/2544295803 conn(0x7fdfa0065fb0 msgr2=0x7fdfa00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 -- 172.21.15.102:0/2544295803 shutdown_connections 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.008+0000 7fdfa53ff700 1 -- 172.21.15.102:0/2544295803 wait complete. 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.010+0000 7fdfa53ff700 1 Processor -- start 2023-03-03T07:49:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.010+0000 7fdfa53ff700 1 -- start start 2023-03-03T07:49:46.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdfa53ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfa00cd8e0 0x7fdfa01539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:49:46.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdfa53ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa0153290 0x7fdfa0155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:49:46.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdfa53ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa01593c0 0x7fdfa0155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:49:46.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdfa53ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c9090 con 0x7fdfa01593c0 2023-03-03T07:49:46.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdfa53ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c8ad0 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdfa53ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c71e0 con 0x7fdfa0153290 2023-03-03T07:49:46.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdf9effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfa00cd8e0 0x7fdfa01539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:49:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.011+0000 7fdf9effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfa00cd8e0 0x7fdfa01539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:51884/0 (socket says 172.21.15.102:51884) 2023-03-03T07:49:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9effd700 1 -- 172.21.15.102:0/1494752850 learned_addr learned my addr 172.21.15.102:0/1494752850 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:49:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9e7fc700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa0153290 0x7fdfa0155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:49:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9f7fe700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa01593c0 0x7fdfa0155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:49:46.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9effd700 1 -- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa0153290 msgr2=0x7fdfa0155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:49:46.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9effd700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa0153290 0x7fdfa0155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9effd700 1 -- 172.21.15.102:0/1494752850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa01593c0 msgr2=0x7fdfa0155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:49:46.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9effd700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa01593c0 0x7fdfa0155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9effd700 1 -- 172.21.15.102:0/1494752850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf90009550 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.012+0000 7fdf9effd700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfa00cd8e0 0x7fdfa01539a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009c10 tx=0x7fdf90004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:49:46.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.013+0000 7fdf877fe700 1 -- 172.21.15.102:0/1494752850 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90004450 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.013+0000 7fdf877fe700 1 -- 172.21.15.102:0/1494752850 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9001e070 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.013+0000 7fdf877fe700 1 -- 172.21.15.102:0/1494752850 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf900038e0 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.013+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa00ccde0 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.013+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa015d160 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.014+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa00cd0c0 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.015+0000 7fdf877fe700 1 -- 172.21.15.102:0/1494752850 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf9000f050 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.016+0000 7fdf877fe700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf8c0682f0 0x7fdf8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:49:46.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.017+0000 7fdf877fe700 1 -- 172.21.15.102:0/1494752850 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdf90086a60 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.017+0000 7fdf9e7fc700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf8c0682f0 0x7fdf8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:49:46.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.017+0000 7fdf9e7fc700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf8c0682f0 0x7fdf8c06a7a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdfa006d230 tx=0x7fdf94009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:49:46.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.021+0000 7fdf877fe700 1 -- 172.21.15.102:0/1494752850 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf90053170 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.258+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa01537f0 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.259+0000 7fdf877fe700 1 -- 172.21.15.102:0/1494752850 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdf90020090 con 0x7fdfa00cd8e0 2023-03-03T07:49:46.261 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:49:46.262 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:49:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf8c0682f0 msgr2=0x7fdf8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:49:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf8c0682f0 0x7fdf8c06a7a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdfa006d230 tx=0x7fdf94009210 comp rx=0 tx=0).stop 2023-03-03T07:49:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfa00cd8e0 msgr2=0x7fdfa01539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:49:46.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfa00cd8e0 0x7fdfa01539a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009c10 tx=0x7fdf90004750 comp rx=0 tx=0).stop 2023-03-03T07:49:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 shutdown_connections 2023-03-03T07:49:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfa01593c0 0x7fdfa0155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfa00cd8e0 0x7fdfa01539a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf8c0682f0 0x7fdf8c06a7a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 --2- 172.21.15.102:0/1494752850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdfa0153290 0x7fdfa0155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:49:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 >> 172.21.15.102:0/1494752850 conn(0x7fdfa0065fb0 msgr2=0x7fdfa00cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:49:46.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 shutdown_connections 2023-03-03T07:49:46.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:49:46.262+0000 7fdfa53ff700 1 -- 172.21.15.102:0/1494752850 wait complete. 2023-03-03T07:49:46.266 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:49:46.291 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-03-03T07:50:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T07:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:50:13.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:50:13.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:50:43.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:50:43.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:13.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:13.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:33.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:51:33.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.542+0000 7f59e51ff700 1 Processor -- start 2023-03-03T07:51:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.543+0000 7f59e51ff700 1 -- start start 2023-03-03T07:51:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59e51ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e0153290 0x7f59e0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:51:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59e51ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f59e01593c0 0x7f59e00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:51:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59e51ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e0153660 con 0x7f59e00cd8e0 2023-03-03T07:51:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59e51ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e006c360 con 0x7f59e01593c0 2023-03-03T07:51:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59e51ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e015cea0 con 0x7f59e0153290 2023-03-03T07:51:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59de59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f59e01593c0 0x7f59e00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:51:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59df59e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f59e00cd8e0 0x7f59e00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38974/0 (socket says 172.21.15.102:38974) 2023-03-03T07:51:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59df59e700 1 -- 172.21.15.102:0/1437183324 learned_addr learned my addr 172.21.15.102:0/1437183324 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:51:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.544+0000 7f59ded9d700 1 --2- 172.21.15.102:0/1437183324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e0153290 0x7f59e0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:51:33.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.545+0000 7f59de59c700 1 -- 172.21.15.102:0/1437183324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e0153290 msgr2=0x7f59e0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:51:33.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.545+0000 7f59de59c700 1 --2- 172.21.15.102:0/1437183324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e0153290 0x7f59e0158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.545+0000 7f59de59c700 1 -- 172.21.15.102:0/1437183324 >> v1:172.21.15.102:6789/0 conn(0x7f59e00cd8e0 legacy=0x7f59e00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:51:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.545+0000 7f59de59c700 1 -- 172.21.15.102:0/1437183324 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e006c7f0 con 0x7f59e01593c0 2023-03-03T07:51:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.545+0000 7f59de59c700 1 --2- 172.21.15.102:0/1437183324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 0x7f59e00cde20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009840 tx=0x7f59d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dfff309e564e07 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:51:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.545+0000 7f59ddd9b700 1 -- 172.21.15.102:0/1437183324 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4011830 con 0x7f59e01593c0 2023-03-03T07:51:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.545+0000 7f59ddd9b700 1 -- 172.21.15.102:0/1437183324 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d4011990 con 0x7f59e01593c0 2023-03-03T07:51:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59ddd9b700 1 -- 172.21.15.102:0/1437183324 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4011d20 con 0x7f59e01593c0 2023-03-03T07:51:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 -- 172.21.15.102:0/1437183324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 msgr2=0x7f59e00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:51:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 --2- 172.21.15.102:0/1437183324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 0x7f59e00cde20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009840 tx=0x7f59d4009b50 comp rx=0 tx=0).stop 2023-03-03T07:51:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 -- 172.21.15.102:0/1437183324 shutdown_connections 2023-03-03T07:51:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 --2- 172.21.15.102:0/1437183324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e0153290 0x7f59e0158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 --2- 172.21.15.102:0/1437183324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 0x7f59e00cde20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 -- 172.21.15.102:0/1437183324 >> 172.21.15.102:0/1437183324 conn(0x7f59e0065fb0 msgr2=0x7f59e00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:51:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 -- 172.21.15.102:0/1437183324 shutdown_connections 2023-03-03T07:51:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.546+0000 7f59e51ff700 1 -- 172.21.15.102:0/1437183324 wait complete. 2023-03-03T07:51:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.548+0000 7f59e51ff700 1 Processor -- start 2023-03-03T07:51:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.548+0000 7f59e51ff700 1 -- start start 2023-03-03T07:51:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.549+0000 7f59e51ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e00cd8e0 0x7f59e00b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:51:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.549+0000 7f59e51ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59e0153290 0x7f59e00b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:51:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.549+0000 7f59e51ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 0x7f59e00b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:51:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.549+0000 7f59e51ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e01ee500 con 0x7f59e01593c0 2023-03-03T07:51:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.549+0000 7f59e51ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00b4750 con 0x7f59e0153290 2023-03-03T07:51:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.549+0000 7f59e51ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00b4a10 con 0x7f59e00cd8e0 2023-03-03T07:51:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e00cd8e0 0x7f59e00b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:51:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e00cd8e0 0x7f59e00b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:33920/0 (socket says 172.21.15.102:33920) 2023-03-03T07:51:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59de59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59e0153290 0x7f59e00b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:51:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 -- 172.21.15.102:0/12615017 learned_addr learned my addr 172.21.15.102:0/12615017 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:51:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59df59e700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 0x7f59e00b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:51:33.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 -- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59e0153290 msgr2=0x7f59e00b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:51:33.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59e0153290 0x7f59e00b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 -- 172.21.15.102:0/12615017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 msgr2=0x7f59e00b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:51:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 0x7f59e00b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 -- 172.21.15.102:0/12615017 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d40094a0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.550+0000 7f59ded9d700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e00cd8e0 0x7f59e00b4130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f59d000e9b0 tx=0x7f59d000ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:51:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.551+0000 7f59c77fe700 1 -- 172.21.15.102:0/12615017 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0009710 con 0x7f59e00cd8e0 2023-03-03T07:51:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.551+0000 7f59c77fe700 1 -- 172.21.15.102:0/12615017 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d000ee50 con 0x7f59e00cd8e0 2023-03-03T07:51:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.551+0000 7f59c77fe700 1 -- 172.21.15.102:0/12615017 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d000c9f0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.551+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e00ccde0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.551+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e015d160 con 0x7f59e00cd8e0 2023-03-03T07:51:33.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.552+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e00cd0c0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.554+0000 7f59c77fe700 1 -- 172.21.15.102:0/12615017 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f59d0019030 con 0x7f59e00cd8e0 2023-03-03T07:51:33.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.555+0000 7f59c77fe700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f59cc068320 0x7f59cc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:51:33.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.555+0000 7f59c77fe700 1 -- 172.21.15.102:0/12615017 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f59d0085bb0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.555+0000 7f59de59c700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f59cc068320 0x7f59cc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:51:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.556+0000 7f59de59c700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f59cc068320 0x7f59cc06a7d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:51:33.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.559+0000 7f59c77fe700 1 -- 172.21.15.102:0/12615017 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f59d00522c0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.793+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e004e6e0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.794+0000 7f59c77fe700 1 -- 172.21.15.102:0/12615017 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f59d00558e0 con 0x7f59e00cd8e0 2023-03-03T07:51:33.796 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:51:33.797 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:51:33.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f59cc068320 msgr2=0x7f59cc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:51:33.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f59cc068320 0x7f59cc06a7d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d4008030 comp rx=0 tx=0).stop 2023-03-03T07:51:33.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e00cd8e0 msgr2=0x7f59e00b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:51:33.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e00cd8e0 0x7f59e00b4130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f59d000e9b0 tx=0x7f59d000ecc0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 shutdown_connections 2023-03-03T07:51:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f59cc068320 0x7f59cc06a7d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59e00cd8e0 0x7f59e00b4130 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59e01593c0 0x7f59e00b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 --2- 172.21.15.102:0/12615017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59e0153290 0x7f59e00b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:51:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 >> 172.21.15.102:0/12615017 conn(0x7f59e0065fb0 msgr2=0x7f59e015c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:51:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 shutdown_connections 2023-03-03T07:51:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:51:33.797+0000 7f59e51ff700 1 -- 172.21.15.102:0/12615017 wait complete. 2023-03-03T07:51:33.801 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:51:33.826 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T07:51:43.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:43.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:14.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:14.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:44.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:44.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:11.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:53:12.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.080+0000 7fad0bbff700 1 Processor -- start 2023-03-03T07:53:12.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.080+0000 7fad0bbff700 1 -- start start 2023-03-03T07:53:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.081+0000 7fad0bbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fad04153290 0x7fad04158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.081+0000 7fad0bbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fad041593c0 0x7fad040cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0bbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad04153660 con 0x7fad040cd8e0 2023-03-03T07:53:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0bbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0406c360 con 0x7fad04153290 2023-03-03T07:53:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0bbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0415cea0 con 0x7fad041593c0 2023-03-03T07:53:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0999b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fad04153290 0x7fad04158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0919a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fad041593c0 0x7fad040cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0919a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fad041593c0 0x7fad040cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60156/0 (socket says 172.21.15.102:60156) 2023-03-03T07:53:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0919a700 1 -- 172.21.15.102:0/1254681369 learned_addr learned my addr 172.21.15.102:0/1254681369 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:53:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0a19c700 1 --1- 172.21.15.102:0/1254681369 >> v1:172.21.15.102:6789/0 conn(0x7fad040cd8e0 0x7fad040ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47250/0 (socket says 172.21.15.102:47250) 2023-03-03T07:53:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0919a700 1 -- 172.21.15.102:0/1254681369 >> v2:172.21.15.102:3300/0 conn(0x7fad04153290 msgr2=0x7fad04158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0919a700 1 --2- 172.21.15.102:0/1254681369 >> v2:172.21.15.102:3300/0 conn(0x7fad04153290 0x7fad04158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0919a700 1 -- 172.21.15.102:0/1254681369 >> v1:172.21.15.102:6789/0 conn(0x7fad040cd8e0 legacy=0x7fad040ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0919a700 1 -- 172.21.15.102:0/1254681369 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0406c7f0 con 0x7fad041593c0 2023-03-03T07:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad0999b700 1 --2- 172.21.15.102:0/1254681369 >> v2:172.21.15.102:3300/0 conn(0x7fad04153290 0x7fad04158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.082+0000 7fad08999700 1 -- 172.21.15.102:0/1254681369 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639973667 0 0) 0x7fad04153660 con 0x7fad040cd8e0 2023-03-03T07:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.083+0000 7fad0919a700 1 --2- 172.21.15.102:0/1254681369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad041593c0 0x7fad040cde20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fad04159fb0 tx=0x7facf4009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fab986efe9b1a4b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.083+0000 7fad08999700 1 -- 172.21.15.102:0/1254681369 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4011830 con 0x7fad041593c0 2023-03-03T07:53:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.083+0000 7fad08999700 1 -- 172.21.15.102:0/1254681369 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf4011990 con 0x7fad041593c0 2023-03-03T07:53:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.083+0000 7fad08999700 1 -- 172.21.15.102:0/1254681369 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4011d20 con 0x7fad041593c0 2023-03-03T07:53:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 -- 172.21.15.102:0/1254681369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad041593c0 msgr2=0x7fad040cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1254681369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad041593c0 0x7fad040cde20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fad04159fb0 tx=0x7facf4009bd0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 -- 172.21.15.102:0/1254681369 shutdown_connections 2023-03-03T07:53:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1254681369 >> v2:172.21.15.102:3300/0 conn(0x7fad04153290 0x7fad04158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1254681369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad041593c0 0x7fad040cde20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 -- 172.21.15.102:0/1254681369 >> 172.21.15.102:0/1254681369 conn(0x7fad04065fb0 msgr2=0x7fad040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:53:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 -- 172.21.15.102:0/1254681369 shutdown_connections 2023-03-03T07:53:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.084+0000 7fad0bbff700 1 -- 172.21.15.102:0/1254681369 wait complete. 2023-03-03T07:53:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.086+0000 7fad0bbff700 1 Processor -- start 2023-03-03T07:53:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.086+0000 7fad0bbff700 1 -- start start 2023-03-03T07:53:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0bbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad040cd8e0 0x7fad040c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0bbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad04153290 0x7fad041542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0bbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad041593c0 0x7fad041f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0bbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad040c6140 con 0x7fad041593c0 2023-03-03T07:53:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0bbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad040c75e0 con 0x7fad04153290 2023-03-03T07:53:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0bbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad040c7fb0 con 0x7fad040cd8e0 2023-03-03T07:53:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0a19c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad041593c0 0x7fad041f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0a19c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad041593c0 0x7fad041f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57656/0 (socket says 172.21.15.102:57656) 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0a19c700 1 -- 172.21.15.102:0/1115182149 learned_addr learned my addr 172.21.15.102:0/1115182149 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.087+0000 7fad0919a700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad04153290 0x7fad041542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0999b700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad040cd8e0 0x7fad040c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0a19c700 1 -- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad040cd8e0 msgr2=0x7fad040c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0a19c700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad040cd8e0 0x7fad040c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0a19c700 1 -- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad04153290 msgr2=0x7fad041542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0a19c700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad04153290 0x7fad041542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0a19c700 1 -- 172.21.15.102:0/1115182149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000094a0 con 0x7fad041593c0 2023-03-03T07:53:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0a19c700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad041593c0 0x7fad041f1a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7facfc00cba0 tx=0x7facfc00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:53:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7facfa7fc700 1 -- 172.21.15.102:0/1115182149 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc00aaa0 con 0x7fad041593c0 2023-03-03T07:53:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7facfa7fc700 1 -- 172.21.15.102:0/1115182149 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc00ac20 con 0x7fad041593c0 2023-03-03T07:53:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7facfa7fc700 1 -- 172.21.15.102:0/1115182149 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc0070d0 con 0x7fad041593c0 2023-03-03T07:53:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.088+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf4009550 con 0x7fad041593c0 2023-03-03T07:53:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.089+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad040ccde0 con 0x7fad041593c0 2023-03-03T07:53:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.090+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0415d160 con 0x7fad041593c0 2023-03-03T07:53:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.091+0000 7facfa7fc700 1 -- 172.21.15.102:0/1115182149 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7facfc010070 con 0x7fad041593c0 2023-03-03T07:53:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.092+0000 7facfa7fc700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7face80682f0 0x7face806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.093+0000 7facfa7fc700 1 -- 172.21.15.102:0/1115182149 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7facfc085730 con 0x7fad041593c0 2023-03-03T07:53:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.093+0000 7fad0999b700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7face80682f0 0x7face806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.093+0000 7fad0999b700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7face80682f0 0x7face806a7a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fad00009d70 tx=0x7fad00009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:53:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.097+0000 7facfa7fc700 1 -- 172.21.15.102:0/1115182149 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facfc051e40 con 0x7fad041593c0 2023-03-03T07:53:12.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.333+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad040cd0c0 con 0x7fad041593c0 2023-03-03T07:53:12.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.334+0000 7facfa7fc700 1 -- 172.21.15.102:0/1115182149 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7facfc018090 con 0x7fad041593c0 2023-03-03T07:53:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:53:12.335 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:53:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7face80682f0 msgr2=0x7face806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7face80682f0 0x7face806a7a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fad00009d70 tx=0x7fad00009210 comp rx=0 tx=0).stop 2023-03-03T07:53:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad041593c0 msgr2=0x7fad041f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad041593c0 0x7fad041f1a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7facfc00cba0 tx=0x7facfc00ceb0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 shutdown_connections 2023-03-03T07:53:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7face80682f0 0x7face806a7a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fad040cd8e0 0x7fad040c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad041593c0 0x7fad041f1a60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 --2- 172.21.15.102:0/1115182149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad04153290 0x7fad041542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:12.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 >> 172.21.15.102:0/1115182149 conn(0x7fad04065fb0 msgr2=0x7fad0415ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:53:12.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 shutdown_connections 2023-03-03T07:53:12.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:12.337+0000 7fad0bbff700 1 -- 172.21.15.102:0/1115182149 wait complete. 2023-03-03T07:53:12.340 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:53:12.366 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T07:53:14.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:14.393 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:21.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:53:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.621+0000 7efc07bff700 1 Processor -- start 2023-03-03T07:53:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.622+0000 7efc07bff700 1 -- start start 2023-03-03T07:53:21.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.623+0000 7efc07bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.623+0000 7efc07bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efc000cd390 0x7efc000ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.623+0000 7efc07bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc00153660 con 0x7efc00153290 2023-03-03T07:53:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.623+0000 7efc07bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0006c360 con 0x7efc000cd390 2023-03-03T07:53:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.623+0000 7efc07bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0015cea0 con 0x7efc000cd8e0 2023-03-03T07:53:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.623+0000 7efc0599b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc00153290 0x7efc00158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46398/0 (socket says 172.21.15.102:46398) 2023-03-03T07:53:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.623+0000 7efc0599b700 1 -- 172.21.15.102:0/412835207 learned_addr learned my addr 172.21.15.102:0/412835207 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:53:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0619c700 1 --2- 172.21.15.102:0/412835207 >> v2:172.21.15.172:3300/0 conn(0x7efc000cd390 0x7efc000ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:21.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0519a700 1 --2- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:21.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc04999700 1 -- 172.21.15.102:0/412835207 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934780228 0 0) 0x7efc00153660 con 0x7efc00153290 2023-03-03T07:53:21.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc04999700 1 -- 172.21.15.102:0/412835207 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbec0035b0 con 0x7efc00153290 2023-03-03T07:53:21.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0619c700 1 -- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 msgr2=0x7efc000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:53:21.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0619c700 1 --2- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc000cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0619c700 1 -- 172.21.15.102:0/412835207 >> v1:172.21.15.102:6789/0 conn(0x7efc00153290 legacy=0x7efc00158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:21.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0619c700 1 -- 172.21.15.102:0/412835207 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0006c7f0 con 0x7efc000cd390 2023-03-03T07:53:21.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0519a700 1 --2- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:53:21.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.624+0000 7efc0619c700 1 --2- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000ce360 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d90 tx=0x7efbf800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be72bc43c3840833 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:53:21.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc04999700 1 -- 172.21.15.102:0/412835207 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800e070 con 0x7efc000cd390 2023-03-03T07:53:21.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc04999700 1 -- 172.21.15.102:0/412835207 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf800bb20 con 0x7efc000cd390 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc04999700 1 -- 172.21.15.102:0/412835207 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800be90 con 0x7efc000cd390 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc07bff700 1 -- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 msgr2=0x7efc000ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc07bff700 1 --2- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000ce360 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d90 tx=0x7efbf800b330 comp rx=0 tx=0).stop 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc07bff700 1 -- 172.21.15.102:0/412835207 shutdown_connections 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc07bff700 1 --2- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000ce360 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc07bff700 1 --2- 172.21.15.102:0/412835207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc07bff700 1 -- 172.21.15.102:0/412835207 >> 172.21.15.102:0/412835207 conn(0x7efc00065fb0 msgr2=0x7efc000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:53:21.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.625+0000 7efc07bff700 1 -- 172.21.15.102:0/412835207 shutdown_connections 2023-03-03T07:53:21.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.626+0000 7efc07bff700 1 -- 172.21.15.102:0/412835207 wait complete. 2023-03-03T07:53:21.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.628+0000 7efc07bff700 1 Processor -- start 2023-03-03T07:53:21.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.628+0000 7efc07bff700 1 -- start start 2023-03-03T07:53:21.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc07bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:21.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc07bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc000cd8e0 0x7efc001f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:21.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc07bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc00153290 0x7efc000c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:21.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc07bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc000c61e0 con 0x7efc000cd8e0 2023-03-03T07:53:21.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc07bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc000c2590 con 0x7efc00153290 2023-03-03T07:53:21.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc07bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc000c2870 con 0x7efc000cd390 2023-03-03T07:53:21.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc0519a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc000cd8e0 0x7efc001f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:21.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc0619c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc00153290 0x7efc000c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc0599b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc0519a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc000cd8e0 0x7efc001f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37728/0 (socket says 172.21.15.102:37728) 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.629+0000 7efc0519a700 1 -- 172.21.15.102:0/2190892300 learned_addr learned my addr 172.21.15.102:0/2190892300 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efc0519a700 1 -- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 msgr2=0x7efc000c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efc0519a700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efc0519a700 1 -- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc00153290 msgr2=0x7efc000c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efc0519a700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc00153290 0x7efc000c6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efc0519a700 1 -- 172.21.15.102:0/2190892300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf00094a0 con 0x7efc000cd8e0 2023-03-03T07:53:21.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efc0519a700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc000cd8e0 0x7efc001f2200 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efbf000ebe0 tx=0x7efbf000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:53:21.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efbf67fc700 1 -- 172.21.15.102:0/2190892300 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000c850 con 0x7efc000cd8e0 2023-03-03T07:53:21.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efbf67fc700 1 -- 172.21.15.102:0/2190892300 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000c9b0 con 0x7efc000cd8e0 2023-03-03T07:53:21.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efbf67fc700 1 -- 172.21.15.102:0/2190892300 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000cd40 con 0x7efc000cd8e0 2023-03-03T07:53:21.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.630+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf8009a70 con 0x7efc000cd8e0 2023-03-03T07:53:21.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.631+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc000ccde0 con 0x7efc000cd8e0 2023-03-03T07:53:21.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.632+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0015d160 con 0x7efc000cd8e0 2023-03-03T07:53:21.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.633+0000 7efbf67fc700 1 -- 172.21.15.102:0/2190892300 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efbf0023070 con 0x7efc000cd8e0 2023-03-03T07:53:21.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.635+0000 7efbf67fc700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbe40682f0 0x7efbe406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:53:21.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.635+0000 7efbf67fc700 1 -- 172.21.15.102:0/2190892300 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7efbf0086110 con 0x7efc000cd8e0 2023-03-03T07:53:21.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.635+0000 7efc0599b700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbe40682f0 0x7efbe406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:53:21.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.636+0000 7efc0599b700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbe40682f0 0x7efbe406a7a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7efbfc007fd0 tx=0x7efbfc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:53:21.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.639+0000 7efbf67fc700 1 -- 172.21.15.102:0/2190892300 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efbf0052820 con 0x7efc000cd8e0 2023-03-03T07:53:21.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.871+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc000cd0c0 con 0x7efc000cd8e0 2023-03-03T07:53:21.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.872+0000 7efbf67fc700 1 -- 172.21.15.102:0/2190892300 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7efbf000cea0 con 0x7efc000cd8e0 2023-03-03T07:53:21.873 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:53:21.873 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:53:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.874+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbe40682f0 msgr2=0x7efbe406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.874+0000 7efc07bff700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbe40682f0 0x7efbe406a7a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7efbfc007fd0 tx=0x7efbfc009040 comp rx=0 tx=0).stop 2023-03-03T07:53:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.874+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc000cd8e0 msgr2=0x7efc001f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:53:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.874+0000 7efc07bff700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc000cd8e0 0x7efc001f2200 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efbf000ebe0 tx=0x7efbf000eef0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 shutdown_connections 2023-03-03T07:53:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbe40682f0 0x7efbe406a7a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc00153290 0x7efc000c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 --2- 172.21.15.102:0/2190892300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc000cd8e0 0x7efc001f2200 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:53:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 >> 172.21.15.102:0/2190892300 conn(0x7efc00065fb0 msgr2=0x7efc000cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:53:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 shutdown_connections 2023-03-03T07:53:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:53:21.875+0000 7efc07bff700 1 -- 172.21.15.102:0/2190892300 wait complete. 2023-03-03T07:53:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:53:21.902 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-03-03T07:53:44.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:44.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:14.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:14.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:44.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:44.938 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:56.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:54:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.153+0000 7ff0bd9ff700 1 Processor -- start 2023-03-03T07:54:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.153+0000 7ff0bd9ff700 1 -- start start 2023-03-03T07:54:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.154+0000 7ff0bd9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:54:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0bd9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0b80716c0 0x7ff0b80706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:54:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0bd9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b80c3c80 con 0x7ff0b814ffb0 2023-03-03T07:54:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0bd9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b80c41a0 con 0x7ff0b80716c0 2023-03-03T07:54:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0bd9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8071cd0 con 0x7ff0b814fbe0 2023-03-03T07:54:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0b77fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0b80716c0 0x7ff0b80706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:54:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0b67fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff0b814ffb0 0x7ff0b80c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:58484/0 (socket says 172.21.15.102:58484) 2023-03-03T07:54:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0b67fc700 1 -- 172.21.15.102:0/2848407397 learned_addr learned my addr 172.21.15.102:0/2848407397 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:54:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0b6ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:54:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0b5ffb700 1 -- 172.21.15.102:0/2848407397 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023660901 0 0) 0x7ff0ac001110 con 0x7ff0b814ffb0 2023-03-03T07:54:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.155+0000 7ff0b77fe700 1 -- 172.21.15.102:0/2848407397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 msgr2=0x7ff0b80c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:54:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b77fe700 1 --2- 172.21.15.102:0/2848407397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b77fe700 1 -- 172.21.15.102:0/2848407397 >> v1:172.21.15.172:6789/0 conn(0x7ff0b814ffb0 legacy=0x7ff0b80c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:54:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b77fe700 1 -- 172.21.15.102:0/2848407397 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b80c3ec0 con 0x7ff0b80716c0 2023-03-03T07:54:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b6ffd700 1 --2- 172.21.15.102:0/2848407397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T07:54:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b77fe700 1 --2- 172.21.15.102:0/2848407397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b80706f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009d90 tx=0x7ff0b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689c60f3c2d2546f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:54:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b5ffb700 1 -- 172.21.15.102:0/2848407397 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b000e070 con 0x7ff0b80716c0 2023-03-03T07:54:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b5ffb700 1 -- 172.21.15.102:0/2848407397 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b000bb20 con 0x7ff0b80716c0 2023-03-03T07:54:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.156+0000 7ff0b5ffb700 1 -- 172.21.15.102:0/2848407397 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0012430 con 0x7ff0b80716c0 2023-03-03T07:54:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/2848407397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 msgr2=0x7ff0b80706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:54:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/2848407397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b80706f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009d90 tx=0x7ff0b000b330 comp rx=0 tx=0).stop 2023-03-03T07:54:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/2848407397 shutdown_connections 2023-03-03T07:54:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/2848407397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b80706f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/2848407397 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/2848407397 >> 172.21.15.102:0/2848407397 conn(0x7ff0b806af90 msgr2=0x7ff0b806c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:54:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/2848407397 shutdown_connections 2023-03-03T07:54:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.157+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/2848407397 wait complete. 2023-03-03T07:54:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.159+0000 7ff0bd9ff700 1 Processor -- start 2023-03-03T07:54:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.159+0000 7ff0bd9ff700 1 -- start start 2023-03-03T07:54:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0bd9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b81ef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:54:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0bd9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:54:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0bd9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b814ffb0 0x7ff0b8137f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:54:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0bd9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b813d8d0 con 0x7ff0b80716c0 2023-03-03T07:54:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0bd9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8139800 con 0x7ff0b814ffb0 2023-03-03T07:54:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0bd9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8145a00 con 0x7ff0b814fbe0 2023-03-03T07:54:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b81ef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:54:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b81ef3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56632/0 (socket says 172.21.15.102:56632) 2023-03-03T07:54:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0b6ffd700 1 -- 172.21.15.102:0/1377057525 learned_addr learned my addr 172.21.15.102:0/1377057525 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:54:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0b77fe700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b814ffb0 0x7ff0b8137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:54:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.160+0000 7ff0b67fc700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:54:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff0b6ffd700 1 -- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 msgr2=0x7ff0b80cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:54:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff0b6ffd700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff0b6ffd700 1 -- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b814ffb0 msgr2=0x7ff0b8137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:54:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff0b6ffd700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b814ffb0 0x7ff0b8137f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff0b6ffd700 1 -- 172.21.15.102:0/1377057525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a80094a0 con 0x7ff0b80716c0 2023-03-03T07:54:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff0b6ffd700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b81ef3c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800ebe0 tx=0x7ff0a800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:54:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff09ffff700 1 -- 172.21.15.102:0/1377057525 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800c850 con 0x7ff0b80716c0 2023-03-03T07:54:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff09ffff700 1 -- 172.21.15.102:0/1377057525 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a800c9b0 con 0x7ff0b80716c0 2023-03-03T07:54:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0009a70 con 0x7ff0b80716c0 2023-03-03T07:54:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.161+0000 7ff09ffff700 1 -- 172.21.15.102:0/1377057525 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800cd40 con 0x7ff0b80716c0 2023-03-03T07:54:57.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.162+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b8072250 con 0x7ff0b80716c0 2023-03-03T07:54:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.162+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b8071f70 con 0x7ff0b80716c0 2023-03-03T07:54:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.164+0000 7ff09ffff700 1 -- 172.21.15.102:0/1377057525 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff0a8023070 con 0x7ff0b80716c0 2023-03-03T07:54:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.165+0000 7ff09ffff700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0a40682f0 0x7ff0a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:54:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.165+0000 7ff09ffff700 1 -- 172.21.15.102:0/1377057525 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff0a80860e0 con 0x7ff0b80716c0 2023-03-03T07:54:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.165+0000 7ff0b67fc700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0a40682f0 0x7ff0a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:54:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.166+0000 7ff0b67fc700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0a40682f0 0x7ff0a406a7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff0b806cb90 tx=0x7ff0ac007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:54:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.169+0000 7ff09ffff700 1 -- 172.21.15.102:0/1377057525 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0a80527f0 con 0x7ff0b80716c0 2023-03-03T07:54:57.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.403+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0b804e760 con 0x7ff0b80716c0 2023-03-03T07:54:57.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.403+0000 7ff09ffff700 1 -- 172.21.15.102:0/1377057525 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff0a800cea0 con 0x7ff0b80716c0 2023-03-03T07:54:57.405 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:54:57.405 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:54:57.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0a40682f0 msgr2=0x7ff0a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:54:57.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0a40682f0 0x7ff0a406a7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff0b806cb90 tx=0x7ff0ac007e70 comp rx=0 tx=0).stop 2023-03-03T07:54:57.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 msgr2=0x7ff0b81ef3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:54:57.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b81ef3c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800ebe0 tx=0x7ff0a800eef0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 shutdown_connections 2023-03-03T07:54:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0b814ffb0 0x7ff0b8137f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0b80716c0 0x7ff0b81ef3c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0a40682f0 0x7ff0a406a7a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 --2- 172.21.15.102:0/1377057525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0b814fbe0 0x7ff0b80cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:54:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 >> 172.21.15.102:0/1377057525 conn(0x7ff0b806af90 msgr2=0x7ff0b806ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:54:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 shutdown_connections 2023-03-03T07:54:57.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:54:57.406+0000 7ff0bd9ff700 1 -- 172.21.15.102:0/1377057525 wait complete. 2023-03-03T07:54:57.409 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:54:57.434 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2023-03-03T07:55:15.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:15.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:45.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:45.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:15.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:15.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:45.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:45.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:53.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:56:53.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.684+0000 7fcc429ff700 1 Processor -- start 2023-03-03T07:56:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.684+0000 7fcc429ff700 1 -- start start 2023-03-03T07:56:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.686+0000 7fcc429ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc3c0cd390 0x7fcc3c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.686+0000 7fcc429ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c153660 con 0x7fcc3c153290 2023-03-03T07:56:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.686+0000 7fcc429ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c06c360 con 0x7fcc3c0cd8e0 2023-03-03T07:56:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.686+0000 7fcc429ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3c15cea0 con 0x7fcc3c0cd390 2023-03-03T07:56:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.686+0000 7fcc3bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcc3c153290 0x7fcc3c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49082/0 (socket says 172.21.15.102:49082) 2023-03-03T07:56:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.686+0000 7fcc3bfff700 1 -- 172.21.15.102:0/4261220100 learned_addr learned my addr 172.21.15.102:0/4261220100 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:56:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.686+0000 7fcc40f9c700 1 --2- 172.21.15.102:0/4261220100 >> v2:172.21.15.172:3300/0 conn(0x7fcc3c0cd390 0x7fcc3c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc3affd700 1 -- 172.21.15.102:0/4261220100 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202303369 0 0) 0x7fcc3c06c360 con 0x7fcc3c0cd8e0 2023-03-03T07:56:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc3affd700 1 -- 172.21.15.102:0/4261220100 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc240035b0 con 0x7fcc3c0cd8e0 2023-03-03T07:56:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc3affd700 1 -- 172.21.15.102:0/4261220100 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764180321 0 0) 0x7fcc3c153660 con 0x7fcc3c153290 2023-03-03T07:56:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc3affd700 1 -- 172.21.15.102:0/4261220100 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c06c360 con 0x7fcc3c153290 2023-03-03T07:56:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc40f9c700 1 -- 172.21.15.102:0/4261220100 >> v1:172.21.15.172:6789/0 conn(0x7fcc3c0cd8e0 legacy=0x7fcc3c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc40f9c700 1 -- 172.21.15.102:0/4261220100 >> v1:172.21.15.102:6789/0 conn(0x7fcc3c153290 legacy=0x7fcc3c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc40f9c700 1 -- 172.21.15.102:0/4261220100 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c06c7f0 con 0x7fcc3c0cd390 2023-03-03T07:56:53.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc40f9c700 1 --2- 172.21.15.102:0/4261220100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c0cd390 0x7fcc3c0ce530 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009d90 tx=0x7fcc3400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c08cd6313a545329 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.687+0000 7fcc3affd700 1 -- 172.21.15.102:0/4261220100 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400e070 con 0x7fcc3c0cd390 2023-03-03T07:56:53.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc3affd700 1 -- 172.21.15.102:0/4261220100 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3400bb20 con 0x7fcc3c0cd390 2023-03-03T07:56:53.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc3affd700 1 -- 172.21.15.102:0/4261220100 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34012430 con 0x7fcc3c0cd390 2023-03-03T07:56:53.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc429ff700 1 -- 172.21.15.102:0/4261220100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c0cd390 msgr2=0x7fcc3c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc429ff700 1 --2- 172.21.15.102:0/4261220100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c0cd390 0x7fcc3c0ce530 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009d90 tx=0x7fcc3400b330 comp rx=0 tx=0).stop 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc429ff700 1 -- 172.21.15.102:0/4261220100 shutdown_connections 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc429ff700 1 --2- 172.21.15.102:0/4261220100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c0cd390 0x7fcc3c0ce530 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc429ff700 1 -- 172.21.15.102:0/4261220100 >> 172.21.15.102:0/4261220100 conn(0x7fcc3c065fb0 msgr2=0x7fcc3c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc429ff700 1 -- 172.21.15.102:0/4261220100 shutdown_connections 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.688+0000 7fcc429ff700 1 -- 172.21.15.102:0/4261220100 wait complete. 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.690+0000 7fcc429ff700 1 Processor -- start 2023-03-03T07:56:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.690+0000 7fcc429ff700 1 -- start start 2023-03-03T07:56:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.691+0000 7fcc429ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc3c0cd390 0x7fcc3c153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.691+0000 7fcc429ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3c0cd8e0 0x7fcc3c0cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.691+0000 7fcc429ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c153290 0x7fcc3c1f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.691+0000 7fcc429ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0c5cf0 con 0x7fcc3c0cd8e0 2023-03-03T07:56:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.691+0000 7fcc429ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c156120 con 0x7fcc3c153290 2023-03-03T07:56:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.691+0000 7fcc429ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c1e4790 con 0x7fcc3c0cd390 2023-03-03T07:56:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c153290 0x7fcc3c1f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc3bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc3c0cd390 0x7fcc3c153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c153290 0x7fcc3c1f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56714/0 (socket says 172.21.15.102:56714) 2023-03-03T07:56:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 -- 172.21.15.102:0/3397384462 learned_addr learned my addr 172.21.15.102:0/3397384462 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:56:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc3b7fe700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3c0cd8e0 0x7fcc3c0cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 -- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc3c0cd390 msgr2=0x7fcc3c153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc3c0cd390 0x7fcc3c153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 -- 172.21.15.102:0/3397384462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3c0cd8e0 msgr2=0x7fcc3c0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3c0cd8e0 0x7fcc3c0cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 -- 172.21.15.102:0/3397384462 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc34009a70 con 0x7fcc3c153290 2023-03-03T07:56:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc3b7fe700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3c0cd8e0 0x7fcc3c0cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:56:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.692+0000 7fcc40f9c700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c153290 0x7fcc3c1f2090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400b680 tx=0x7fcc34004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.693+0000 7fcc38ff9700 1 -- 172.21.15.102:0/3397384462 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3401c810 con 0x7fcc3c153290 2023-03-03T07:56:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.693+0000 7fcc38ff9700 1 -- 172.21.15.102:0/3397384462 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc34020070 con 0x7fcc3c153290 2023-03-03T07:56:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.693+0000 7fcc38ff9700 1 -- 172.21.15.102:0/3397384462 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3401cb90 con 0x7fcc3c153290 2023-03-03T07:56:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.693+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c0ccde0 con 0x7fcc3c153290 2023-03-03T07:56:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.693+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c15d160 con 0x7fcc3c153290 2023-03-03T07:56:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.694+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c0cd0c0 con 0x7fcc3c153290 2023-03-03T07:56:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.696+0000 7fcc38ff9700 1 -- 172.21.15.102:0/3397384462 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcc3402e070 con 0x7fcc3c153290 2023-03-03T07:56:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.697+0000 7fcc38ff9700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc280682f0 0x7fcc2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:56:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.697+0000 7fcc38ff9700 1 -- 172.21.15.102:0/3397384462 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcc34022070 con 0x7fcc3c153290 2023-03-03T07:56:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.697+0000 7fcc3bfff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc280682f0 0x7fcc2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:56:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.698+0000 7fcc3bfff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc280682f0 0x7fcc2806a7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c007fd0 tx=0x7fcc2c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:56:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.700+0000 7fcc38ff9700 1 -- 172.21.15.102:0/3397384462 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc340544b0 con 0x7fcc3c153290 2023-03-03T07:56:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.930+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc3c0c5ef0 con 0x7fcc3c153290 2023-03-03T07:56:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.931+0000 7fcc38ff9700 1 -- 172.21.15.102:0/3397384462 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcc3402c070 con 0x7fcc3c153290 2023-03-03T07:56:53.933 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:56:53.933 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:56:53.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc280682f0 msgr2=0x7fcc2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc280682f0 0x7fcc2806a7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c007fd0 tx=0x7fcc2c007dc0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c153290 msgr2=0x7fcc3c1f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:56:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c153290 0x7fcc3c1f2090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400b680 tx=0x7fcc34004ad0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 shutdown_connections 2023-03-03T07:56:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc280682f0 0x7fcc2806a7a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc3c0cd390 0x7fcc3c153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc3c153290 0x7fcc3c1f2090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 --2- 172.21.15.102:0/3397384462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3c0cd8e0 0x7fcc3c0cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:56:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 >> 172.21.15.102:0/3397384462 conn(0x7fcc3c065fb0 msgr2=0x7fcc3c158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:56:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 shutdown_connections 2023-03-03T07:56:53.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:56:53.934+0000 7fcc429ff700 1 -- 172.21.15.102:0/3397384462 wait complete. 2023-03-03T07:56:53.939 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:56:53.964 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T07:57:15.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:15.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:21.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:57:22.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.232+0000 7fd40d9ff700 1 Processor -- start 2023-03-03T07:57:22.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.232+0000 7fd40d9ff700 1 -- start start 2023-03-03T07:57:22.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.233+0000 7fd40d9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd408153290 0x7fd408158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd40d9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd4080cd8e0 0x7fd4080ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd40d9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd408153660 con 0x7fd4081593c0 2023-03-03T07:57:22.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd40d9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40806c360 con 0x7fd408153290 2023-03-03T07:57:22.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd40d9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40815cea0 con 0x7fd4080cd8e0 2023-03-03T07:57:22.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd406ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd408153290 0x7fd408158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd4067fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd4081593c0 0x7fd4080cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59304/0 (socket says 172.21.15.102:59304) 2023-03-03T07:57:22.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd4067fc700 1 -- 172.21.15.102:0/4236056153 learned_addr learned my addr 172.21.15.102:0/4236056153 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T07:57:22.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd4077fe700 1 --2- 172.21.15.102:0/4236056153 >> v2:172.21.15.172:3300/0 conn(0x7fd4080cd8e0 0x7fd4080ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd405ffb700 1 -- 172.21.15.102:0/4236056153 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509370075 0 0) 0x7fd408153660 con 0x7fd4081593c0 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd405ffb700 1 -- 172.21.15.102:0/4236056153 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f00035b0 con 0x7fd4081593c0 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd406ffd700 1 -- 172.21.15.102:0/4236056153 >> v2:172.21.15.172:3300/0 conn(0x7fd4080cd8e0 msgr2=0x7fd4080ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd406ffd700 1 --2- 172.21.15.102:0/4236056153 >> v2:172.21.15.172:3300/0 conn(0x7fd4080cd8e0 0x7fd4080ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.234+0000 7fd406ffd700 1 -- 172.21.15.102:0/4236056153 >> v1:172.21.15.172:6789/0 conn(0x7fd4081593c0 legacy=0x7fd4080cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.235+0000 7fd406ffd700 1 -- 172.21.15.102:0/4236056153 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40806c7f0 con 0x7fd408153290 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.235+0000 7fd406ffd700 1 --2- 172.21.15.102:0/4236056153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd408153290 0x7fd408158e80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd4080c9390 tx=0x7fd3f8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885d869a46ec0def server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.235+0000 7fd405ffb700 1 -- 172.21.15.102:0/4236056153 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8011830 con 0x7fd408153290 2023-03-03T07:57:22.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.235+0000 7fd405ffb700 1 -- 172.21.15.102:0/4236056153 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f8011990 con 0x7fd408153290 2023-03-03T07:57:22.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.235+0000 7fd405ffb700 1 -- 172.21.15.102:0/4236056153 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8011d00 con 0x7fd408153290 2023-03-03T07:57:22.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 -- 172.21.15.102:0/4236056153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd408153290 msgr2=0x7fd408158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/4236056153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd408153290 0x7fd408158e80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd4080c9390 tx=0x7fd3f8009b20 comp rx=0 tx=0).stop 2023-03-03T07:57:22.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 -- 172.21.15.102:0/4236056153 shutdown_connections 2023-03-03T07:57:22.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/4236056153 >> v2:172.21.15.172:3300/0 conn(0x7fd4080cd8e0 0x7fd4080ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/4236056153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd408153290 0x7fd408158e80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 -- 172.21.15.102:0/4236056153 >> 172.21.15.102:0/4236056153 conn(0x7fd408065fb0 msgr2=0x7fd4080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:22.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 -- 172.21.15.102:0/4236056153 shutdown_connections 2023-03-03T07:57:22.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.236+0000 7fd40d9ff700 1 -- 172.21.15.102:0/4236056153 wait complete. 2023-03-03T07:57:22.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.238+0000 7fd40d9ff700 1 Processor -- start 2023-03-03T07:57:22.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.238+0000 7fd40d9ff700 1 -- start start 2023-03-03T07:57:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd40d9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4080cd8e0 0x7fd4081e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd40d9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd408153290 0x7fd4080b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd40d9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4081593c0 0x7fd4080b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd40d9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081e5f00 con 0x7fd4080cd8e0 2023-03-03T07:57:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd40d9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b3660 con 0x7fd408153290 2023-03-03T07:57:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd40d9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b3900 con 0x7fd4081593c0 2023-03-03T07:57:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd406ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4080cd8e0 0x7fd4081e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd406ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4080cd8e0 0x7fd4081e3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58438/0 (socket says 172.21.15.102:58438) 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd406ffd700 1 -- 172.21.15.102:0/122410872 learned_addr learned my addr 172.21.15.102:0/122410872 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.239+0000 7fd4067fc700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd408153290 0x7fd4080b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd4077fe700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4081593c0 0x7fd4080b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd406ffd700 1 -- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4081593c0 msgr2=0x7fd4080b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd406ffd700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4081593c0 0x7fd4080b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd406ffd700 1 -- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd408153290 msgr2=0x7fd4080b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd406ffd700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd408153290 0x7fd4080b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd406ffd700 1 -- 172.21.15.102:0/122410872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f80094a0 con 0x7fd4080cd8e0 2023-03-03T07:57:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd4067fc700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd408153290 0x7fd4080b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T07:57:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd406ffd700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4080cd8e0 0x7fd4081e3c80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8009b60 tx=0x7fd3f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd3effff700 1 -- 172.21.15.102:0/122410872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f80036a0 con 0x7fd4080cd8e0 2023-03-03T07:57:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd3effff700 1 -- 172.21.15.102:0/122410872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f801f070 con 0x7fd4080cd8e0 2023-03-03T07:57:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd3effff700 1 -- 172.21.15.102:0/122410872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8003a40 con 0x7fd4080cd8e0 2023-03-03T07:57:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.240+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4080ccde0 con 0x7fd4080cd8e0 2023-03-03T07:57:22.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.241+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40815d160 con 0x7fd4080cd8e0 2023-03-03T07:57:22.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.242+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4080cd0c0 con 0x7fd4080cd8e0 2023-03-03T07:57:22.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.243+0000 7fd3effff700 1 -- 172.21.15.102:0/122410872 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd3f802d070 con 0x7fd4080cd8e0 2023-03-03T07:57:22.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.244+0000 7fd3effff700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3f4068320 0x7fd3f406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:57:22.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.244+0000 7fd3effff700 1 -- 172.21.15.102:0/122410872 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd3f8021070 con 0x7fd4080cd8e0 2023-03-03T07:57:22.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.244+0000 7fd4067fc700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3f4068320 0x7fd3f406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:57:22.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.245+0000 7fd4067fc700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3f4068320 0x7fd3f406a7d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd40815b5e0 tx=0x7fd3fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:57:22.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.248+0000 7fd3effff700 1 -- 172.21.15.102:0/122410872 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd3f80539b0 con 0x7fd4080cd8e0 2023-03-03T07:57:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.488+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4080ca950 con 0x7fd4080cd8e0 2023-03-03T07:57:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.488+0000 7fd3effff700 1 -- 172.21.15.102:0/122410872 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd3f8056fd0 con 0x7fd4080cd8e0 2023-03-03T07:57:22.490 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:57:22.490 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:57:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3f4068320 msgr2=0x7fd3f406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3f4068320 0x7fd3f406a7d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd40815b5e0 tx=0x7fd3fc009040 comp rx=0 tx=0).stop 2023-03-03T07:57:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4080cd8e0 msgr2=0x7fd4081e3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:57:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4080cd8e0 0x7fd4081e3c80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8009b60 tx=0x7fd3f8004750 comp rx=0 tx=0).stop 2023-03-03T07:57:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 shutdown_connections 2023-03-03T07:57:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3f4068320 0x7fd3f406a7d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd4081593c0 0x7fd4080b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4080cd8e0 0x7fd4081e3c80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 --2- 172.21.15.102:0/122410872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd408153290 0x7fd4080b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:57:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 >> 172.21.15.102:0/122410872 conn(0x7fd408065fb0 msgr2=0x7fd40815bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:57:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 shutdown_connections 2023-03-03T07:57:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:57:22.491+0000 7fd40d9ff700 1 -- 172.21.15.102:0/122410872 wait complete. 2023-03-03T07:57:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:57:22.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T07:57:45.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:45.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:16.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:16.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:41.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:58:41.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.769+0000 7f75bedff700 1 Processor -- start 2023-03-03T07:58:41.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.770+0000 7f75bedff700 1 -- start start 2023-03-03T07:58:41.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.771+0000 7f75bedff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f75b8152e50 0x7f75b80c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:41.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.771+0000 7f75bedff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b80c9590 con 0x7f75b80c8eb0 2023-03-03T07:58:41.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.771+0000 7f75bedff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b80ca400 con 0x7f75b80c9f60 2023-03-03T07:58:41.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.771+0000 7f75bedff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b80cda80 con 0x7f75b8152e50 2023-03-03T07:58:41.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.771+0000 7f75bcb9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f75b8152e50 0x7f75b80c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:41.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.771+0000 7f75bcb9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f75b8152e50 0x7f75b80c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40894/0 (socket says 172.21.15.102:40894) 2023-03-03T07:58:41.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.772+0000 7f75bcb9b700 1 -- 172.21.15.102:0/3804063369 learned_addr learned my addr 172.21.15.102:0/3804063369 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:58:41.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.772+0000 7f75b7fff700 1 --1- 172.21.15.102:0/3804063369 >> v1:172.21.15.102:6789/0 conn(0x7f75b80c8eb0 0x7f75b80ccc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33624/0 (socket says 172.21.15.102:33624) 2023-03-03T07:58:41.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.772+0000 7f75bcb9b700 1 -- 172.21.15.102:0/3804063369 >> v1:172.21.15.172:6789/0 conn(0x7f75b80c9f60 legacy=0x7f75b80cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:41.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.772+0000 7f75bcb9b700 1 -- 172.21.15.102:0/3804063369 >> v1:172.21.15.102:6789/0 conn(0x7f75b80c8eb0 legacy=0x7f75b80ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:41.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.772+0000 7f75bcb9b700 1 -- 172.21.15.102:0/3804063369 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8159880 con 0x7f75b8152e50 2023-03-03T07:58:41.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.772+0000 7f75bcb9b700 1 --2- 172.21.15.102:0/3804063369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b8152e50 0x7f75b80c9a20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0098f0 tx=0x7f75ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3099a9abae3e8339 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:41.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75b77fe700 1 -- 172.21.15.102:0/3804063369 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00b8a0 con 0x7f75b8152e50 2023-03-03T07:58:41.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75b77fe700 1 -- 172.21.15.102:0/3804063369 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00ba00 con 0x7f75b8152e50 2023-03-03T07:58:41.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75b77fe700 1 -- 172.21.15.102:0/3804063369 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00bd90 con 0x7f75b8152e50 2023-03-03T07:58:41.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75bedff700 1 -- 172.21.15.102:0/3804063369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b8152e50 msgr2=0x7f75b80c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:41.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75bedff700 1 --2- 172.21.15.102:0/3804063369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b8152e50 0x7f75b80c9a20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0098f0 tx=0x7f75ac009c00 comp rx=0 tx=0).stop 2023-03-03T07:58:41.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75bedff700 1 -- 172.21.15.102:0/3804063369 shutdown_connections 2023-03-03T07:58:41.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75bedff700 1 --2- 172.21.15.102:0/3804063369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b8152e50 0x7f75b80c9a20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75bedff700 1 -- 172.21.15.102:0/3804063369 >> 172.21.15.102:0/3804063369 conn(0x7f75b80704c0 msgr2=0x7f75b80708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:41.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75bedff700 1 -- 172.21.15.102:0/3804063369 shutdown_connections 2023-03-03T07:58:41.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.773+0000 7f75bedff700 1 -- 172.21.15.102:0/3804063369 wait complete. 2023-03-03T07:58:41.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.775+0000 7f75bedff700 1 Processor -- start 2023-03-03T07:58:41.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.775+0000 7f75bedff700 1 -- start start 2023-03-03T07:58:41.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.776+0000 7f75bedff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b80c8eb0 0x7f75b80b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:41.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.776+0000 7f75bedff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75b80c9f60 0x7f75b80cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:41.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.776+0000 7f75bedff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75b8152e50 0x7f75b80b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:41.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.776+0000 7f75bedff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b80b2690 con 0x7f75b80c8eb0 2023-03-03T07:58:41.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.776+0000 7f75bedff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b814f650 con 0x7f75b80c9f60 2023-03-03T07:58:41.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.776+0000 7f75bedff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b814ea40 con 0x7f75b8152e50 2023-03-03T07:58:41.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bcb9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b80c8eb0 0x7f75b80b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:41.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75b7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75b80c9f60 0x7f75b80cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:41.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bd39c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75b8152e50 0x7f75b80b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:41.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75b7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75b80c9f60 0x7f75b80cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54804/0 (socket says 172.21.15.102:54804) 2023-03-03T07:58:41.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bcb9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b80c8eb0 0x7f75b80b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40900/0 (socket says 172.21.15.102:40900) 2023-03-03T07:58:41.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75b7fff700 1 -- 172.21.15.102:0/3352538272 learned_addr learned my addr 172.21.15.102:0/3352538272 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:58:41.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bcb9b700 1 -- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75b8152e50 msgr2=0x7f75b80b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:41.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bcb9b700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75b8152e50 0x7f75b80b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bcb9b700 1 -- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75b80c9f60 msgr2=0x7f75b80cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:41.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bcb9b700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75b80c9f60 0x7f75b80cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:41.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.777+0000 7f75bcb9b700 1 -- 172.21.15.102:0/3352538272 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac009550 con 0x7f75b80c8eb0 2023-03-03T07:58:41.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.778+0000 7f75bcb9b700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b80c8eb0 0x7f75b80b2150 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f75ac016000 tx=0x7f75ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:41.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.778+0000 7f75b57fa700 1 -- 172.21.15.102:0/3352538272 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac004510 con 0x7f75b80c8eb0 2023-03-03T07:58:41.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.778+0000 7f75b57fa700 1 -- 172.21.15.102:0/3352538272 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac01e070 con 0x7f75b80c8eb0 2023-03-03T07:58:41.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.778+0000 7f75b57fa700 1 -- 172.21.15.102:0/3352538272 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0038e0 con 0x7f75b80c8eb0 2023-03-03T07:58:41.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.778+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b80c7dc0 con 0x7f75b80c8eb0 2023-03-03T07:58:41.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.778+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b80c7ae0 con 0x7f75b80c8eb0 2023-03-03T07:58:41.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.779+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b80c80a0 con 0x7f75b80c8eb0 2023-03-03T07:58:41.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.780+0000 7f75b57fa700 1 -- 172.21.15.102:0/3352538272 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f75ac00f050 con 0x7f75b80c8eb0 2023-03-03T07:58:41.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.781+0000 7f75b57fa700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f759c0682f0 0x7f759c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:58:41.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.782+0000 7f75b57fa700 1 -- 172.21.15.102:0/3352538272 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f75ac020070 con 0x7f75b80c8eb0 2023-03-03T07:58:41.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.782+0000 7f75b7fff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f759c0682f0 0x7f759c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:58:41.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.783+0000 7f75b7fff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f759c0682f0 0x7f759c06a7a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75b814d580 tx=0x7f75a4007f60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:58:41.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:41.786+0000 7f75b57fa700 1 -- 172.21.15.102:0/3352538272 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f75ac0538f0 con 0x7f75b80c8eb0 2023-03-03T07:58:42.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.020+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75b814f930 con 0x7f75b80c8eb0 2023-03-03T07:58:42.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.021+0000 7f75b57fa700 1 -- 172.21.15.102:0/3352538272 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f75ac056f10 con 0x7f75b80c8eb0 2023-03-03T07:58:42.022 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:58:42.022 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:58:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.023+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f759c0682f0 msgr2=0x7f759c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.023+0000 7f75bedff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f759c0682f0 0x7f759c06a7a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75b814d580 tx=0x7f75a4007f60 comp rx=0 tx=0).stop 2023-03-03T07:58:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.023+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b80c8eb0 msgr2=0x7f75b80b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:58:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.023+0000 7f75bedff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b80c8eb0 0x7f75b80b2150 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f75ac016000 tx=0x7f75ac004750 comp rx=0 tx=0).stop 2023-03-03T07:58:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 shutdown_connections 2023-03-03T07:58:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f759c0682f0 0x7f759c06a7a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:42.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75b8152e50 0x7f75b80b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:42.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75b80c8eb0 0x7f75b80b2150 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:42.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 --2- 172.21.15.102:0/3352538272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75b80c9f60 0x7f75b80cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:58:42.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 >> 172.21.15.102:0/3352538272 conn(0x7f75b80704c0 msgr2=0x7f75b8158480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:58:42.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 shutdown_connections 2023-03-03T07:58:42.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:58:42.024+0000 7f75bedff700 1 -- 172.21.15.102:0/3352538272 wait complete. 2023-03-03T07:58:42.027 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:58:42.052 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 68 secs before thrashing 2023-03-03T07:58:46.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:46.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:16.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:16.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:46.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:46.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:50.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T07:59:50.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.307+0000 7f6e7d59d700 1 Processor -- start 2023-03-03T07:59:50.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.307+0000 7f6e7d59d700 1 -- start start 2023-03-03T07:59:50.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7d59d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6e780b3970 0x7f6e780c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:50.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7d59d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6e780c41b0 0x7f6e780cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:50.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7d59d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780c3d20 con 0x7f6e780ced00 2023-03-03T07:59:50.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7d59d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e780c4930 con 0x7f6e780c41b0 2023-03-03T07:59:50.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7d59d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e780cf610 con 0x7f6e780b3970 2023-03-03T07:59:50.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7cd9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6e780c41b0 0x7f6e780cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:50.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7cd9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6e780c41b0 0x7f6e780cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36472/0 (socket says 172.21.15.102:36472) 2023-03-03T07:59:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e7cd9c700 1 -- 172.21.15.102:0/831964117 learned_addr learned my addr 172.21.15.102:0/831964117 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:59:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e77fff700 1 --1- 172.21.15.102:0/831964117 >> v1:172.21.15.172:6789/0 conn(0x7f6e780ced00 0x7f6e78158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35288/0 (socket says 172.21.15.102:35288) 2023-03-03T07:59:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.309+0000 7f6e777fe700 1 --2- 172.21.15.102:0/831964117 >> v2:172.21.15.172:3300/0 conn(0x7f6e780b3970 0x7f6e780c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e7cd9c700 1 -- 172.21.15.102:0/831964117 >> v2:172.21.15.172:3300/0 conn(0x7f6e780b3970 msgr2=0x7f6e780c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T07:59:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e7cd9c700 1 --2- 172.21.15.102:0/831964117 >> v2:172.21.15.172:3300/0 conn(0x7f6e780b3970 0x7f6e780c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e7cd9c700 1 -- 172.21.15.102:0/831964117 >> v1:172.21.15.172:6789/0 conn(0x7f6e780ced00 legacy=0x7f6e78158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e76ffd700 1 -- 172.21.15.102:0/831964117 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57469942 0 0) 0x7f6e780c3d20 con 0x7f6e780ced00 2023-03-03T07:59:50.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e7cd9c700 1 -- 172.21.15.102:0/831964117 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7815cba0 con 0x7f6e780c41b0 2023-03-03T07:59:50.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e7cd9c700 1 --2- 172.21.15.102:0/831964117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780c41b0 0x7f6e780cf0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6e780c4ba0 tx=0x7f6e7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac451a94a8966a52 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:50.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e76ffd700 1 -- 172.21.15.102:0/831964117 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000e070 con 0x7f6e780c41b0 2023-03-03T07:59:50.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e76ffd700 1 -- 172.21.15.102:0/831964117 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7000b8d0 con 0x7f6e780c41b0 2023-03-03T07:59:50.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.310+0000 7f6e76ffd700 1 -- 172.21.15.102:0/831964117 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000bc80 con 0x7f6e780c41b0 2023-03-03T07:59:50.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 -- 172.21.15.102:0/831964117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780c41b0 msgr2=0x7f6e780cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:50.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/831964117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780c41b0 0x7f6e780cf0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6e780c4ba0 tx=0x7f6e7000b330 comp rx=0 tx=0).stop 2023-03-03T07:59:50.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 -- 172.21.15.102:0/831964117 shutdown_connections 2023-03-03T07:59:50.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/831964117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780c41b0 0x7f6e780cf0d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/831964117 >> v2:172.21.15.172:3300/0 conn(0x7f6e780b3970 0x7f6e780c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 -- 172.21.15.102:0/831964117 >> 172.21.15.102:0/831964117 conn(0x7f6e7808b1c0 msgr2=0x7f6e78071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:50.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 -- 172.21.15.102:0/831964117 shutdown_connections 2023-03-03T07:59:50.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.311+0000 7f6e7d59d700 1 -- 172.21.15.102:0/831964117 wait complete. 2023-03-03T07:59:50.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.313+0000 7f6e7d59d700 1 Processor -- start 2023-03-03T07:59:50.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.313+0000 7f6e7d59d700 1 -- start start 2023-03-03T07:59:50.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.314+0000 7f6e7d59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780b3970 0x7f6e7815bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:50.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.314+0000 7f6e7d59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e780c41b0 0x7f6e78159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:50.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.314+0000 7f6e7d59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e780ced00 0x7f6e78147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:50.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.314+0000 7f6e7d59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7815a200 con 0x7f6e780b3970 2023-03-03T07:59:50.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.314+0000 7f6e7d59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e781ef820 con 0x7f6e780c41b0 2023-03-03T07:59:50.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.314+0000 7f6e7d59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780c66f0 con 0x7f6e780ced00 2023-03-03T07:59:50.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780b3970 0x7f6e7815bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780b3970 0x7f6e7815bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36488/0 (socket says 172.21.15.102:36488) 2023-03-03T07:59:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 -- 172.21.15.102:0/2414712435 learned_addr learned my addr 172.21.15.102:0/2414712435 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T07:59:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e777fe700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e780c41b0 0x7f6e78159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e7cd9c700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e780ced00 0x7f6e78147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 -- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e780ced00 msgr2=0x7f6e78147de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e780ced00 0x7f6e78147de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 -- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e780c41b0 msgr2=0x7f6e78159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e780c41b0 0x7f6e78159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 -- 172.21.15.102:0/2414712435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70009b30 con 0x7f6e780b3970 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.315+0000 7f6e77fff700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780b3970 0x7f6e7815bd60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6e680011e0 tx=0x7f6e6800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.316+0000 7f6e74ff9700 1 -- 172.21.15.102:0/2414712435 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68004c80 con 0x7f6e780b3970 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.316+0000 7f6e74ff9700 1 -- 172.21.15.102:0/2414712435 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e68004e00 con 0x7f6e780b3970 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.316+0000 7f6e74ff9700 1 -- 172.21.15.102:0/2414712435 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68007310 con 0x7f6e780b3970 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.316+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7806bf20 con 0x7f6e780b3970 2023-03-03T07:59:50.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.316+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7815ce80 con 0x7f6e780b3970 2023-03-03T07:59:50.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.317+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7806c200 con 0x7f6e780b3970 2023-03-03T07:59:50.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.318+0000 7f6e74ff9700 1 -- 172.21.15.102:0/2414712435 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6e6801a020 con 0x7f6e780b3970 2023-03-03T07:59:50.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.320+0000 7f6e74ff9700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6e640682f0 0x7f6e6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T07:59:50.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.320+0000 7f6e74ff9700 1 -- 172.21.15.102:0/2414712435 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6e68085920 con 0x7f6e780b3970 2023-03-03T07:59:50.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.321+0000 7f6e777fe700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6e640682f0 0x7f6e6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T07:59:50.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.321+0000 7f6e777fe700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6e640682f0 0x7f6e6406a7a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009b40 tx=0x7f6e6c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T07:59:50.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.325+0000 7f6e74ff9700 1 -- 172.21.15.102:0/2414712435 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6e68051fd0 con 0x7f6e780b3970 2023-03-03T07:59:50.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.559+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e7804e6e0 con 0x7f6e780b3970 2023-03-03T07:59:50.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.559+0000 7f6e74ff9700 1 -- 172.21.15.102:0/2414712435 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6e680555f0 con 0x7f6e780b3970 2023-03-03T07:59:50.561 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T07:59:50.561 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T07:59:50.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6e640682f0 msgr2=0x7f6e6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:50.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6e640682f0 0x7f6e6406a7a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009b40 tx=0x7f6e6c009210 comp rx=0 tx=0).stop 2023-03-03T07:59:50.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780b3970 msgr2=0x7f6e7815bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T07:59:50.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780b3970 0x7f6e7815bd60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6e680011e0 tx=0x7f6e6800ca30 comp rx=0 tx=0).stop 2023-03-03T07:59:50.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 shutdown_connections 2023-03-03T07:59:50.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6e640682f0 0x7f6e6406a7a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6e780ced00 0x7f6e78147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e780b3970 0x7f6e7815bd60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 --2- 172.21.15.102:0/2414712435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e780c41b0 0x7f6e78159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T07:59:50.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 >> 172.21.15.102:0/2414712435 conn(0x7f6e7808b1c0 msgr2=0x7f6e780cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T07:59:50.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 shutdown_connections 2023-03-03T07:59:50.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T07:59:50.562+0000 7f6e7d59d700 1 -- 172.21.15.102:0/2414712435 wait complete. 2023-03-03T07:59:50.566 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T07:59:50.589 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-03-03T08:00:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T07:59:59.998+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:00:16.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:16.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:47.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:47.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:17.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:17.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:32.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:01:32.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.840+0000 7f8509fff700 1 Processor -- start 2023-03-03T08:01:32.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.840+0000 7f8509fff700 1 -- start start 2023-03-03T08:01:32.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.841+0000 7f8509fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8504150c80 0x7f850406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:32.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.841+0000 7f8509fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85040c5f30 0x7f85040c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:32.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.841+0000 7f8509fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85040c5390 con 0x7f85040c2020 2023-03-03T08:01:32.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.841+0000 7f8509fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8504157ad0 con 0x7f85040c5f30 2023-03-03T08:01:32.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.841+0000 7f8509fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85040c4a20 con 0x7f8504150c80 2023-03-03T08:01:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8503fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85040c5f30 0x7f85040c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8503fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85040c5f30 0x7f85040c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43692/0 (socket says 172.21.15.102:43692) 2023-03-03T08:01:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8502ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f85040c2020 0x7f85040c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52112/0 (socket says 172.21.15.102:52112) 2023-03-03T08:01:32.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8503fff700 1 -- 172.21.15.102:0/2932916282 learned_addr learned my addr 172.21.15.102:0/2932916282 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:01:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f85037fe700 1 --2- 172.21.15.102:0/2932916282 >> v2:172.21.15.172:3300/0 conn(0x7f8504150c80 0x7f850406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8503fff700 1 -- 172.21.15.102:0/2932916282 >> v2:172.21.15.172:3300/0 conn(0x7f8504150c80 msgr2=0x7f850406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:01:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8503fff700 1 --2- 172.21.15.102:0/2932916282 >> v2:172.21.15.172:3300/0 conn(0x7f8504150c80 0x7f850406d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8503fff700 1 -- 172.21.15.102:0/2932916282 >> v1:172.21.15.172:6789/0 conn(0x7f85040c2020 legacy=0x7f85040c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f85027fc700 1 -- 172.21.15.102:0/2932916282 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413910178 0 0) 0x7f85040c5390 con 0x7f85040c2020 2023-03-03T08:01:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.842+0000 7f8503fff700 1 -- 172.21.15.102:0/2932916282 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850406f010 con 0x7f85040c5f30 2023-03-03T08:01:32.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.843+0000 7f8503fff700 1 --2- 172.21.15.102:0/2932916282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85040c5f30 0x7f85040c2720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8504072a90 tx=0x7f84fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbff1371deb2c993 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.843+0000 7f85027fc700 1 -- 172.21.15.102:0/2932916282 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00e070 con 0x7f85040c5f30 2023-03-03T08:01:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.843+0000 7f85027fc700 1 -- 172.21.15.102:0/2932916282 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc00b910 con 0x7f85040c5f30 2023-03-03T08:01:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.843+0000 7f85027fc700 1 -- 172.21.15.102:0/2932916282 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00bcc0 con 0x7f85040c5f30 2023-03-03T08:01:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.843+0000 7f8509fff700 1 -- 172.21.15.102:0/2932916282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85040c5f30 msgr2=0x7f85040c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:32.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.843+0000 7f8509fff700 1 --2- 172.21.15.102:0/2932916282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85040c5f30 0x7f85040c2720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8504072a90 tx=0x7f84fc00b330 comp rx=0 tx=0).stop 2023-03-03T08:01:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.843+0000 7f8509fff700 1 -- 172.21.15.102:0/2932916282 shutdown_connections 2023-03-03T08:01:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.844+0000 7f8509fff700 1 --2- 172.21.15.102:0/2932916282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85040c5f30 0x7f85040c2720 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.844+0000 7f8509fff700 1 --2- 172.21.15.102:0/2932916282 >> v2:172.21.15.172:3300/0 conn(0x7f8504150c80 0x7f850406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.844+0000 7f8509fff700 1 -- 172.21.15.102:0/2932916282 >> 172.21.15.102:0/2932916282 conn(0x7f850406af90 msgr2=0x7f850406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:32.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.844+0000 7f8509fff700 1 -- 172.21.15.102:0/2932916282 shutdown_connections 2023-03-03T08:01:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.844+0000 7f8509fff700 1 -- 172.21.15.102:0/2932916282 wait complete. 2023-03-03T08:01:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.846+0000 7f8509fff700 1 Processor -- start 2023-03-03T08:01:32.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.846+0000 7f8509fff700 1 -- start start 2023-03-03T08:01:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8509fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85040c2020 0x7f85041eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8509fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85040c5f30 0x7f85041e2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:32.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8509fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8504150c80 0x7f8504154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8509fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504155e50 con 0x7f8504150c80 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8509fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040c98d0 con 0x7f85040c2020 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8509fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85041eefa0 con 0x7f85040c5f30 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f85037fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85040c2020 0x7f85041eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8503fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8504150c80 0x7f8504154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f85037fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85040c2020 0x7f85041eea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39494/0 (socket says 172.21.15.102:39494) 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f85037fe700 1 -- 172.21.15.102:0/1975707264 learned_addr learned my addr 172.21.15.102:0/1975707264 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:01:32.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.847+0000 7f8502ffd700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85040c5f30 0x7f85041e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f85037fe700 1 -- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85040c5f30 msgr2=0x7f85041e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f85037fe700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85040c5f30 0x7f85041e2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f85037fe700 1 -- 172.21.15.102:0/1975707264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8504150c80 msgr2=0x7f8504154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f85037fe700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8504150c80 0x7f8504154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f85037fe700 1 -- 172.21.15.102:0/1975707264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc009b30 con 0x7f85040c2020 2023-03-03T08:01:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f8503fff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8504150c80 0x7f8504154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:01:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f85037fe700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85040c2020 0x7f85041eea60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84f400e8a0 tx=0x7f84f400ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f84ebfff700 1 -- 172.21.15.102:0/1975707264 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4009590 con 0x7f85040c2020 2023-03-03T08:01:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f84ebfff700 1 -- 172.21.15.102:0/1975707264 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4013680 con 0x7f85040c2020 2023-03-03T08:01:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85041577f0 con 0x7f85040c2020 2023-03-03T08:01:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f84ebfff700 1 -- 172.21.15.102:0/1975707264 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f40139f0 con 0x7f85040c2020 2023-03-03T08:01:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.848+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8504157510 con 0x7f85040c2020 2023-03-03T08:01:32.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.849+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85040c4cc0 con 0x7f85040c2020 2023-03-03T08:01:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.851+0000 7f84ebfff700 1 -- 172.21.15.102:0/1975707264 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f84f4019030 con 0x7f85040c2020 2023-03-03T08:01:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.852+0000 7f84ebfff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84f00682f0 0x7f84f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:01:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.852+0000 7f84ebfff700 1 -- 172.21.15.102:0/1975707264 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f84f4085bd0 con 0x7f85040c2020 2023-03-03T08:01:32.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.852+0000 7f8502ffd700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84f00682f0 0x7f84f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:01:32.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.853+0000 7f8502ffd700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84f00682f0 0x7f84f006a7a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f85041e51b0 tx=0x7f84f8007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:01:32.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:32.856+0000 7f84ebfff700 1 -- 172.21.15.102:0/1975707264 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84f40521f0 con 0x7f85040c2020 2023-03-03T08:01:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.088+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85040c88c0 con 0x7f85040c2020 2023-03-03T08:01:33.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.089+0000 7f84ebfff700 1 -- 172.21.15.102:0/1975707264 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f84f4021090 con 0x7f85040c2020 2023-03-03T08:01:33.091 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:01:33.091 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:01:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84f00682f0 msgr2=0x7f84f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84f00682f0 0x7f84f006a7a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f85041e51b0 tx=0x7f84f8007de0 comp rx=0 tx=0).stop 2023-03-03T08:01:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85040c2020 msgr2=0x7f85041eea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:01:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85040c2020 0x7f85041eea60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84f400e8a0 tx=0x7f84f400ebb0 comp rx=0 tx=0).stop 2023-03-03T08:01:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 shutdown_connections 2023-03-03T08:01:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8504150c80 0x7f8504154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85040c2020 0x7f85041eea60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84f00682f0 0x7f84f006a7a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 --2- 172.21.15.102:0/1975707264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f85040c5f30 0x7f85041e2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:01:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 >> 172.21.15.102:0/1975707264 conn(0x7f850406af90 msgr2=0x7f850414d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:01:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 shutdown_connections 2023-03-03T08:01:33.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:01:33.092+0000 7f8509fff700 1 -- 172.21.15.102:0/1975707264 wait complete. 2023-03-03T08:01:33.096 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:01:33.121 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T08:01:47.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:47.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:17.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:02:17.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.369+0000 7fb834dff700 1 Processor -- start 2023-03-03T08:02:17.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.369+0000 7fb834dff700 1 -- start start 2023-03-03T08:02:17.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb834dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb830153290 0x7fb830158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:17.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb834dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8300cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:17.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb834dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8300cd8e0 0x7fb8300ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:17.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb834dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb830153660 con 0x7fb830153290 2023-03-03T08:02:17.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb834dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83006c360 con 0x7fb8300cd8e0 2023-03-03T08:02:17.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb834dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83015cea0 con 0x7fb8301593c0 2023-03-03T08:02:17.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb82e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb830153290 0x7fb830158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:17.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.370+0000 7fb82ed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8300cd8e0 0x7fb8300ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:17.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8300cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:17.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8300cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51610/0 (socket says 172.21.15.102:51610) 2023-03-03T08:02:17.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82ed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb8300cd8e0 0x7fb8300ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35858/0 (socket says 172.21.15.102:35858) 2023-03-03T08:02:17.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb830153290 0x7fb830158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47624/0 (socket says 172.21.15.102:47624) 2023-03-03T08:02:17.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82ed9d700 1 -- 172.21.15.102:0/4007138291 learned_addr learned my addr 172.21.15.102:0/4007138291 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:02:17.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82e59c700 1 -- 172.21.15.102:0/4007138291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 msgr2=0x7fb8300cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:17.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82e59c700 1 --2- 172.21.15.102:0/4007138291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8300cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82e59c700 1 -- 172.21.15.102:0/4007138291 >> v2:172.21.15.172:3300/0 conn(0x7fb8300cd8e0 msgr2=0x7fb8300ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:02:17.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82e59c700 1 --2- 172.21.15.102:0/4007138291 >> v2:172.21.15.172:3300/0 conn(0x7fb8300cd8e0 0x7fb8300ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82e59c700 1 -- 172.21.15.102:0/4007138291 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83006c7f0 con 0x7fb830153290 2023-03-03T08:02:17.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82dd9b700 1 --2- 172.21.15.102:0/4007138291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8300cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:17.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82ed9d700 1 --2- 172.21.15.102:0/4007138291 >> v2:172.21.15.172:3300/0 conn(0x7fb8300cd8e0 0x7fb8300ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:02:17.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.371+0000 7fb82e59c700 1 --2- 172.21.15.102:0/4007138291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb830153290 0x7fb830158e80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb83006d0e0 tx=0x7fb820009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf364add6e39560 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:17.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb82d59a700 1 -- 172.21.15.102:0/4007138291 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820011830 con 0x7fb830153290 2023-03-03T08:02:17.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb82d59a700 1 -- 172.21.15.102:0/4007138291 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb820011990 con 0x7fb830153290 2023-03-03T08:02:17.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb82d59a700 1 -- 172.21.15.102:0/4007138291 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820011d20 con 0x7fb830153290 2023-03-03T08:02:17.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb834dff700 1 -- 172.21.15.102:0/4007138291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb830153290 msgr2=0x7fb830158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:17.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb834dff700 1 --2- 172.21.15.102:0/4007138291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb830153290 0x7fb830158e80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb83006d0e0 tx=0x7fb820009b20 comp rx=0 tx=0).stop 2023-03-03T08:02:17.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb834dff700 1 -- 172.21.15.102:0/4007138291 shutdown_connections 2023-03-03T08:02:17.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb834dff700 1 --2- 172.21.15.102:0/4007138291 >> v2:172.21.15.172:3300/0 conn(0x7fb8300cd8e0 0x7fb8300ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb834dff700 1 --2- 172.21.15.102:0/4007138291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb830153290 0x7fb830158e80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb834dff700 1 --2- 172.21.15.102:0/4007138291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8300cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.372+0000 7fb834dff700 1 -- 172.21.15.102:0/4007138291 >> 172.21.15.102:0/4007138291 conn(0x7fb830065fb0 msgr2=0x7fb8300663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:17.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.373+0000 7fb834dff700 1 -- 172.21.15.102:0/4007138291 shutdown_connections 2023-03-03T08:02:17.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.373+0000 7fb834dff700 1 -- 172.21.15.102:0/4007138291 wait complete. 2023-03-03T08:02:17.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.375+0000 7fb834dff700 1 Processor -- start 2023-03-03T08:02:17.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.375+0000 7fb834dff700 1 -- start start 2023-03-03T08:02:17.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb834dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8300cd8e0 0x7fb8301ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:17.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb834dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb830153290 0x7fb8300b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:17.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb834dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8301557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:17.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb834dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8301ef840 con 0x7fb8300cd8e0 2023-03-03T08:02:17.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb834dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8301568c0 con 0x7fb830153290 2023-03-03T08:02:17.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb834dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb830153920 con 0x7fb8301593c0 2023-03-03T08:02:17.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8300cd8e0 0x7fb8301ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8300cd8e0 0x7fb8301ef300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47632/0 (socket says 172.21.15.102:47632) 2023-03-03T08:02:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 -- 172.21.15.102:0/2971786547 learned_addr learned my addr 172.21.15.102:0/2971786547 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:02:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82dd9b700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb830153290 0x7fb8300b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82ed9d700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8301557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 -- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 msgr2=0x7fb8301557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8301557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 -- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb830153290 msgr2=0x7fb8300b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:17.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb830153290 0x7fb8300b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82e59c700 1 -- 172.21.15.102:0/2971786547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828009b30 con 0x7fb8300cd8e0 2023-03-03T08:02:17.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.376+0000 7fb82ed9d700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8301557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:02:17.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.377+0000 7fb82e59c700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8300cd8e0 0x7fb8301ef300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb820000c00 tx=0x7fb820004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:17.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.377+0000 7fb81effd700 1 -- 172.21.15.102:0/2971786547 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820004510 con 0x7fb8300cd8e0 2023-03-03T08:02:17.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.377+0000 7fb81effd700 1 -- 172.21.15.102:0/2971786547 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82001e070 con 0x7fb8300cd8e0 2023-03-03T08:02:17.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.377+0000 7fb81effd700 1 -- 172.21.15.102:0/2971786547 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8200038e0 con 0x7fb8300cd8e0 2023-03-03T08:02:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.377+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8240094a0 con 0x7fb8300cd8e0 2023-03-03T08:02:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.377+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8200094a0 con 0x7fb8300cd8e0 2023-03-03T08:02:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.378+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8300ccde0 con 0x7fb8300cd8e0 2023-03-03T08:02:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.379+0000 7fb81effd700 1 -- 172.21.15.102:0/2971786547 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb82000f050 con 0x7fb8300cd8e0 2023-03-03T08:02:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.381+0000 7fb81effd700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8100682f0 0x7fb81006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:02:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.381+0000 7fb81effd700 1 -- 172.21.15.102:0/2971786547 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb820086a70 con 0x7fb8300cd8e0 2023-03-03T08:02:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.381+0000 7fb82dd9b700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8100682f0 0x7fb81006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:02:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.382+0000 7fb82dd9b700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8100682f0 0x7fb81006a7a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb8300cd5c0 tx=0x7fb824009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:02:17.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.385+0000 7fb81effd700 1 -- 172.21.15.102:0/2971786547 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb820053180 con 0x7fb8300cd8e0 2023-03-03T08:02:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.621+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8300cd0c0 con 0x7fb8300cd8e0 2023-03-03T08:02:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.621+0000 7fb81effd700 1 -- 172.21.15.102:0/2971786547 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb820020090 con 0x7fb8300cd8e0 2023-03-03T08:02:17.624 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:02:17.624 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:02:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8100682f0 msgr2=0x7fb81006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8100682f0 0x7fb81006a7a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb8300cd5c0 tx=0x7fb824009210 comp rx=0 tx=0).stop 2023-03-03T08:02:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8300cd8e0 msgr2=0x7fb8301ef300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:02:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8300cd8e0 0x7fb8301ef300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb820000c00 tx=0x7fb820004750 comp rx=0 tx=0).stop 2023-03-03T08:02:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 shutdown_connections 2023-03-03T08:02:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8100682f0 0x7fb81006a7a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8301593c0 0x7fb8301557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8300cd8e0 0x7fb8301ef300 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 --2- 172.21.15.102:0/2971786547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb830153290 0x7fb8300b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:02:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 >> 172.21.15.102:0/2971786547 conn(0x7fb830065fb0 msgr2=0x7fb83015a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:02:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 shutdown_connections 2023-03-03T08:02:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:02:17.624+0000 7fb834dff700 1 -- 172.21.15.102:0/2971786547 wait complete. 2023-03-03T08:02:17.629 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:02:17.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:17.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:17.657 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2023-03-03T08:02:47.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:47.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:14.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:03:14.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.925+0000 7f900a3ff700 1 Processor -- start 2023-03-03T08:03:14.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.925+0000 7f900a3ff700 1 -- start start 2023-03-03T08:03:14.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.926+0000 7f900a3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90040cd8e0 0x7f90040cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.926+0000 7f900a3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90040cd390 0x7f90040ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.926+0000 7f900a3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004153660 con 0x7f9004153290 2023-03-03T08:03:14.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.926+0000 7f900a3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900406c360 con 0x7f90040cd390 2023-03-03T08:03:14.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.926+0000 7f900a3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900415cea0 con 0x7f90040cd8e0 2023-03-03T08:03:14.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f900899c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90040cd390 0x7f90040ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f9003fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9004153290 0x7f9004158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39632/0 (socket says 172.21.15.102:39632) 2023-03-03T08:03:14.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f9003fff700 1 -- 172.21.15.102:0/1766845399 learned_addr learned my addr 172.21.15.102:0/1766845399 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f90037fe700 1 --2- 172.21.15.102:0/1766845399 >> v2:172.21.15.172:3300/0 conn(0x7f90040cd8e0 0x7f90040cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f900899c700 1 -- 172.21.15.102:0/1766845399 >> v2:172.21.15.172:3300/0 conn(0x7f90040cd8e0 msgr2=0x7f90040cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f900899c700 1 --2- 172.21.15.102:0/1766845399 >> v2:172.21.15.172:3300/0 conn(0x7f90040cd8e0 0x7f90040cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f900899c700 1 -- 172.21.15.102:0/1766845399 >> v1:172.21.15.102:6789/0 conn(0x7f9004153290 legacy=0x7f9004158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f900899c700 1 -- 172.21.15.102:0/1766845399 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900406c7f0 con 0x7f90040cd390 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.927+0000 7f9002ffd700 1 -- 172.21.15.102:0/1766845399 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334264412 0 0) 0x7f9004153660 con 0x7f9004153290 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.928+0000 7f900899c700 1 --2- 172.21.15.102:0/1766845399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd390 0x7f90040ce360 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9004159fb0 tx=0x7f8ffc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941d2e543f270566 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.928+0000 7f9002ffd700 1 -- 172.21.15.102:0/1766845399 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00e070 con 0x7f90040cd390 2023-03-03T08:03:14.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.928+0000 7f9002ffd700 1 -- 172.21.15.102:0/1766845399 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc00bb20 con 0x7f90040cd390 2023-03-03T08:03:14.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.928+0000 7f9002ffd700 1 -- 172.21.15.102:0/1766845399 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc012430 con 0x7f90040cd390 2023-03-03T08:03:14.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 -- 172.21.15.102:0/1766845399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd390 msgr2=0x7f90040ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 --2- 172.21.15.102:0/1766845399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd390 0x7f90040ce360 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9004159fb0 tx=0x7f8ffc00b330 comp rx=0 tx=0).stop 2023-03-03T08:03:14.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 -- 172.21.15.102:0/1766845399 shutdown_connections 2023-03-03T08:03:14.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 --2- 172.21.15.102:0/1766845399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd390 0x7f90040ce360 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 --2- 172.21.15.102:0/1766845399 >> v2:172.21.15.172:3300/0 conn(0x7f90040cd8e0 0x7f90040cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 -- 172.21.15.102:0/1766845399 >> 172.21.15.102:0/1766845399 conn(0x7f9004065fb0 msgr2=0x7f90040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:14.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 -- 172.21.15.102:0/1766845399 shutdown_connections 2023-03-03T08:03:14.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.929+0000 7f900a3ff700 1 -- 172.21.15.102:0/1766845399 wait complete. 2023-03-03T08:03:14.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.931+0000 7f900a3ff700 1 Processor -- start 2023-03-03T08:03:14.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.931+0000 7f900a3ff700 1 -- start start 2023-03-03T08:03:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f900a3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90040cd390 0x7f9004150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f900a3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd8e0 0x7f9004157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f900a3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9004153290 0x7f90041efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f900a3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040cfd20 con 0x7f90040cd8e0 2023-03-03T08:03:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f900a3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041585b0 con 0x7f9004153290 2023-03-03T08:03:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f900a3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90041582d0 con 0x7f90040cd390 2023-03-03T08:03:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f90037fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd8e0 0x7f9004157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f90037fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd8e0 0x7f9004157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59484/0 (socket says 172.21.15.102:59484) 2023-03-03T08:03:14.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f90037fe700 1 -- 172.21.15.102:0/337880472 learned_addr learned my addr 172.21.15.102:0/337880472 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:03:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f9003fff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90040cd390 0x7f9004150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.932+0000 7f900899c700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9004153290 0x7f90041efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f90037fe700 1 -- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90040cd390 msgr2=0x7f9004150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f90037fe700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90040cd390 0x7f9004150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f90037fe700 1 -- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9004153290 msgr2=0x7f90041efaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f90037fe700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9004153290 0x7f90041efaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:14.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f90037fe700 1 -- 172.21.15.102:0/337880472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009a70 con 0x7f90040cd8e0 2023-03-03T08:03:14.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f90037fe700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd8e0 0x7f9004157270 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800e9b0 tx=0x7f8ff800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f9000ff9700 1 -- 172.21.15.102:0/337880472 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8009710 con 0x7f90040cd8e0 2023-03-03T08:03:14.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f9000ff9700 1 -- 172.21.15.102:0/337880472 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff800ee50 con 0x7f90040cd8e0 2023-03-03T08:03:14.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f9000ff9700 1 -- 172.21.15.102:0/337880472 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff800ca50 con 0x7f90040cd8e0 2023-03-03T08:03:14.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.933+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90040ccde0 con 0x7f90040cd8e0 2023-03-03T08:03:14.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.934+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900415d160 con 0x7f90040cd8e0 2023-03-03T08:03:14.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.935+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90040cd0c0 con 0x7f90040cd8e0 2023-03-03T08:03:14.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.936+0000 7f9000ff9700 1 -- 172.21.15.102:0/337880472 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8ff8019030 con 0x7f90040cd8e0 2023-03-03T08:03:14.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.938+0000 7f9000ff9700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ff00682f0 0x7f8ff006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:03:14.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.938+0000 7f9000ff9700 1 -- 172.21.15.102:0/337880472 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8ff8085c60 con 0x7f90040cd8e0 2023-03-03T08:03:14.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.938+0000 7f9003fff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ff00682f0 0x7f8ff006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:03:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.939+0000 7f9003fff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ff00682f0 0x7f8ff006a7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4000f60 tx=0x7f8ff4007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:03:14.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:14.942+0000 7f9000ff9700 1 -- 172.21.15.102:0/337880472 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ff8052370 con 0x7f90040cd8e0 2023-03-03T08:03:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.179+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f900415a5d0 con 0x7f90040cd8e0 2023-03-03T08:03:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.179+0000 7f9000ff9700 1 -- 172.21.15.102:0/337880472 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8ff8055990 con 0x7f90040cd8e0 2023-03-03T08:03:15.182 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:03:15.182 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:03:15.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ff00682f0 msgr2=0x7f8ff006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:15.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ff00682f0 0x7f8ff006a7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4000f60 tx=0x7f8ff4007e70 comp rx=0 tx=0).stop 2023-03-03T08:03:15.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd8e0 msgr2=0x7f9004157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:03:15.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd8e0 0x7f9004157270 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800e9b0 tx=0x7f8ff800ecc0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 shutdown_connections 2023-03-03T08:03:15.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ff00682f0 0x7f8ff006a7a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90040cd390 0x7f9004150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9004153290 0x7f90041efaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 --2- 172.21.15.102:0/337880472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90040cd8e0 0x7f9004157270 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:03:15.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 >> 172.21.15.102:0/337880472 conn(0x7f9004065fb0 msgr2=0x7f9004150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:03:15.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 shutdown_connections 2023-03-03T08:03:15.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:03:15.182+0000 7f900a3ff700 1 -- 172.21.15.102:0/337880472 wait complete. 2023-03-03T08:03:15.186 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:03:15.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-03-03T08:03:17.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:17.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:48.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:48.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:18.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:18.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:48.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:48.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:59.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:04:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.468+0000 7f35973ff700 1 Processor -- start 2023-03-03T08:04:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.468+0000 7f35973ff700 1 -- start start 2023-03-03T08:04:59.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.469+0000 7f35973ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3590153290 0x7f3590158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:04:59.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f35973ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35901593c0 0x7f35900cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:04:59.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f35973ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3590153660 con 0x7f35900cd8e0 2023-03-03T08:04:59.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f35973ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359006c360 con 0x7f3590153290 2023-03-03T08:04:59.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f35973ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359015cea0 con 0x7f35901593c0 2023-03-03T08:04:59.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f359519b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3590153290 0x7f3590158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:04:59.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f359599c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f35900cd8e0 0x7f35900ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52538/0 (socket says 172.21.15.102:52538) 2023-03-03T08:04:59.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f359519b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3590153290 0x7f3590158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38504/0 (socket says 172.21.15.102:38504) 2023-03-03T08:04:59.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f359599c700 1 -- 172.21.15.102:0/4056534502 learned_addr learned my addr 172.21.15.102:0/4056534502 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:04:59.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.470+0000 7f359499a700 1 --2- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35901593c0 0x7f35900cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:04:59.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f359519b700 1 -- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35901593c0 msgr2=0x7f35900cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:04:59.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f359519b700 1 --2- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35901593c0 0x7f35900cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f359519b700 1 -- 172.21.15.102:0/4056534502 >> v1:172.21.15.172:6789/0 conn(0x7f35900cd8e0 legacy=0x7f35900ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:04:59.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f359519b700 1 -- 172.21.15.102:0/4056534502 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359006c7f0 con 0x7f3590153290 2023-03-03T08:04:59.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f359499a700 1 --2- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35901593c0 0x7f35900cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:04:59.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f3587fff700 1 -- 172.21.15.102:0/4056534502 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879317944 0 0) 0x7f3590153660 con 0x7f35900cd8e0 2023-03-03T08:04:59.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f359519b700 1 --2- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f3590158e80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f358c009840 tx=0x7f358c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8318f5e7ab38688 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:04:59.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f3587fff700 1 -- 172.21.15.102:0/4056534502 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c011830 con 0x7f3590153290 2023-03-03T08:04:59.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f3587fff700 1 -- 172.21.15.102:0/4056534502 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c011990 con 0x7f3590153290 2023-03-03T08:04:59.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.471+0000 7f3587fff700 1 -- 172.21.15.102:0/4056534502 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c011d20 con 0x7f3590153290 2023-03-03T08:04:59.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 -- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 msgr2=0x7f3590158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:04:59.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 --2- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f3590158e80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f358c009840 tx=0x7f358c009b50 comp rx=0 tx=0).stop 2023-03-03T08:04:59.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 -- 172.21.15.102:0/4056534502 shutdown_connections 2023-03-03T08:04:59.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 --2- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f3590158e80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 --2- 172.21.15.102:0/4056534502 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35901593c0 0x7f35900cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 -- 172.21.15.102:0/4056534502 >> 172.21.15.102:0/4056534502 conn(0x7f3590065fb0 msgr2=0x7f35900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:04:59.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 -- 172.21.15.102:0/4056534502 shutdown_connections 2023-03-03T08:04:59.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.472+0000 7f35973ff700 1 -- 172.21.15.102:0/4056534502 wait complete. 2023-03-03T08:04:59.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.474+0000 7f35973ff700 1 Processor -- start 2023-03-03T08:04:59.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.474+0000 7f35973ff700 1 -- start start 2023-03-03T08:04:59.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f35973ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35900cd8e0 0x7f35900c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:04:59.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f35973ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f359014e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f35973ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35901593c0 0x7f359014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f35973ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359014dad0 con 0x7f35901593c0 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f35973ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359014c100 con 0x7f3590153290 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f35973ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359014c3a0 con 0x7f35900cd8e0 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f359499a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f359014e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f359519b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35900cd8e0 0x7f35900c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f359599c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35901593c0 0x7f359014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:04:59.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f359499a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f359014e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38510/0 (socket says 172.21.15.102:38510) 2023-03-03T08:04:59.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.475+0000 7f359499a700 1 -- 172.21.15.102:0/521572698 learned_addr learned my addr 172.21.15.102:0/521572698 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:04:59.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f359499a700 1 -- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35900cd8e0 msgr2=0x7f35900c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:04:59.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f359499a700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35900cd8e0 0x7f35900c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f359499a700 1 -- 172.21.15.102:0/521572698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35901593c0 msgr2=0x7f359014e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:04:59.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f359499a700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35901593c0 0x7f359014e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f359499a700 1 -- 172.21.15.102:0/521572698 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800094a0 con 0x7f3590153290 2023-03-03T08:04:59.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f359519b700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35900cd8e0 0x7f35900c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:04:59.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f359499a700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f359014e260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f358000ebe0 tx=0x7f358000eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:04:59.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.476+0000 7f3585ffb700 1 -- 172.21.15.102:0/521572698 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358000c850 con 0x7f3590153290 2023-03-03T08:04:59.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.477+0000 7f3585ffb700 1 -- 172.21.15.102:0/521572698 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358000c9b0 con 0x7f3590153290 2023-03-03T08:04:59.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.477+0000 7f3585ffb700 1 -- 172.21.15.102:0/521572698 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358000cd40 con 0x7f3590153290 2023-03-03T08:04:59.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.477+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358c0094a0 con 0x7f3590153290 2023-03-03T08:04:59.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.477+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35900ccde0 con 0x7f3590153290 2023-03-03T08:04:59.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.478+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359015d160 con 0x7f3590153290 2023-03-03T08:04:59.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.479+0000 7f3585ffb700 1 -- 172.21.15.102:0/521572698 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3580019030 con 0x7f3590153290 2023-03-03T08:04:59.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.480+0000 7f3585ffb700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35740682f0 0x7f357406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:04:59.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.480+0000 7f3585ffb700 1 -- 172.21.15.102:0/521572698 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3580085da0 con 0x7f3590153290 2023-03-03T08:04:59.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.481+0000 7f359519b700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35740682f0 0x7f357406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:04:59.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.481+0000 7f359519b700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35740682f0 0x7f357406a7a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f358c0090b0 tx=0x7f358c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:04:59.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.485+0000 7f3585ffb700 1 -- 172.21.15.102:0/521572698 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3580052420 con 0x7f3590153290 2023-03-03T08:04:59.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.717+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35900cd0c0 con 0x7f3590153290 2023-03-03T08:04:59.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.718+0000 7f3585ffb700 1 -- 172.21.15.102:0/521572698 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3580017090 con 0x7f3590153290 2023-03-03T08:04:59.719 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:04:59.719 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:04:59.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35740682f0 msgr2=0x7f357406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:04:59.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35740682f0 0x7f357406a7a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f358c0090b0 tx=0x7f358c008030 comp rx=0 tx=0).stop 2023-03-03T08:04:59.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 msgr2=0x7f359014e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:04:59.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f359014e260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f358000ebe0 tx=0x7f358000eef0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 shutdown_connections 2023-03-03T08:04:59.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35740682f0 0x7f357406a7a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35900cd8e0 0x7f35900c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35901593c0 0x7f359014e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 --2- 172.21.15.102:0/521572698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3590153290 0x7f359014e260 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:04:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 >> 172.21.15.102:0/521572698 conn(0x7f3590065fb0 msgr2=0x7f35900cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:04:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.720+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 shutdown_connections 2023-03-03T08:04:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:04:59.721+0000 7f35973ff700 1 -- 172.21.15.102:0/521572698 wait complete. 2023-03-03T08:04:59.724 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:04:59.748 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T08:05:18.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:18.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:48.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:48.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:57.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:05:58.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.010+0000 7f6b633ff700 1 Processor -- start 2023-03-03T08:05:58.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.011+0000 7f6b633ff700 1 -- start start 2023-03-03T08:05:58.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b633ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6b5c153290 0x7f6b5c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:58.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b633ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c1593c0 0x7f6b5c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:58.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b633ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c153660 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b633ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c06c360 con 0x7f6b5c153290 2023-03-03T08:05:58.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b633ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c15cea0 con 0x7f6b5c1593c0 2023-03-03T08:05:58.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b6119b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6b5c153290 0x7f6b5c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:58.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b6119b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6b5c153290 0x7f6b5c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42550/0 (socket says 172.21.15.102:42550) 2023-03-03T08:05:58.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b6199c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6b5c0cd8e0 0x7f6b5c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:55424/0 (socket says 172.21.15.102:55424) 2023-03-03T08:05:58.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b6119b700 1 -- 172.21.15.102:0/1068247586 learned_addr learned my addr 172.21.15.102:0/1068247586 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:05:58.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.012+0000 7f6b6099a700 1 --2- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c1593c0 0x7f6b5c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:58.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.013+0000 7f6b6119b700 1 -- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c1593c0 msgr2=0x7f6b5c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.013+0000 7f6b6119b700 1 --2- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c1593c0 0x7f6b5c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.013+0000 7f6b6119b700 1 -- 172.21.15.102:0/1068247586 >> v1:172.21.15.172:6789/0 conn(0x7f6b5c0cd8e0 legacy=0x7f6b5c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.013+0000 7f6b6119b700 1 -- 172.21.15.102:0/1068247586 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c06c7f0 con 0x7f6b5c153290 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.013+0000 7f6b6099a700 1 --2- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c1593c0 0x7f6b5c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.013+0000 7f6b6119b700 1 --2- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c153290 0x7f6b5c158e80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b580098f0 tx=0x7f6b58009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc1ee9d3329aa663 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.013+0000 7f6b53fff700 1 -- 172.21.15.102:0/1068247586 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58011830 con 0x7f6b5c153290 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b53fff700 1 -- 172.21.15.102:0/1068247586 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b58011990 con 0x7f6b5c153290 2023-03-03T08:05:58.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b53fff700 1 -- 172.21.15.102:0/1068247586 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58011d20 con 0x7f6b5c153290 2023-03-03T08:05:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 -- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c153290 msgr2=0x7f6b5c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c153290 0x7f6b5c158e80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b580098f0 tx=0x7f6b58009c00 comp rx=0 tx=0).stop 2023-03-03T08:05:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 -- 172.21.15.102:0/1068247586 shutdown_connections 2023-03-03T08:05:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c153290 0x7f6b5c158e80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1068247586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c1593c0 0x7f6b5c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 -- 172.21.15.102:0/1068247586 >> 172.21.15.102:0/1068247586 conn(0x7f6b5c065fb0 msgr2=0x7f6b5c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 -- 172.21.15.102:0/1068247586 shutdown_connections 2023-03-03T08:05:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.014+0000 7f6b633ff700 1 -- 172.21.15.102:0/1068247586 wait complete. 2023-03-03T08:05:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.016+0000 7f6b633ff700 1 Processor -- start 2023-03-03T08:05:58.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.017+0000 7f6b633ff700 1 -- start start 2023-03-03T08:05:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b633ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c0cd8e0 0x7f6b5c151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b633ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b5c153290 0x7f6b5c1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b633ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c1593c0 0x7f6b5c1ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b633ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c1f3090 con 0x7f6b5c153290 2023-03-03T08:05:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b633ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c1ef630 con 0x7f6b5c1593c0 2023-03-03T08:05:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b633ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c1f2b30 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b6119b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c0cd8e0 0x7f6b5c151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:58.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b6119b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c0cd8e0 0x7f6b5c151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38370/0 (socket says 172.21.15.102:38370) 2023-03-03T08:05:58.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b6119b700 1 -- 172.21.15.102:0/1391550395 learned_addr learned my addr 172.21.15.102:0/1391550395 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:05:58.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b6199c700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c1593c0 0x7f6b5c1ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:58.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.018+0000 7f6b6099a700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b5c153290 0x7f6b5c1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:58.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b6119b700 1 -- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c1593c0 msgr2=0x7f6b5c1ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:58.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b6119b700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c1593c0 0x7f6b5c1ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b6119b700 1 -- 172.21.15.102:0/1391550395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b5c153290 msgr2=0x7f6b5c1eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b6119b700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b5c153290 0x7f6b5c1eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b6119b700 1 -- 172.21.15.102:0/1391550395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0094a0 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b6199c700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c1593c0 0x7f6b5c1ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:05:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b6119b700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c0cd8e0 0x7f6b5c151130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009c40 tx=0x7f6b58004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:58.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b51ffb700 1 -- 172.21.15.102:0/1391550395 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58004400 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b51ffb700 1 -- 172.21.15.102:0/1391550395 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5801f070 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b51ffb700 1 -- 172.21.15.102:0/1391550395 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b580038c0 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b58009550 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.019+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c0ccde0 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.020+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c15d160 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.022+0000 7f6b51ffb700 1 -- 172.21.15.102:0/1391550395 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b5802a020 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.023+0000 7f6b51ffb700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6b400682c0 0x7f6b4006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:05:58.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.023+0000 7f6b51ffb700 1 -- 172.21.15.102:0/1391550395 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6b580873b0 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.023+0000 7f6b6099a700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6b400682c0 0x7f6b4006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:05:58.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.024+0000 7f6b6099a700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6b400682c0 0x7f6b4006a770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0b4d70 tx=0x7f6b4c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:05:58.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.028+0000 7f6b51ffb700 1 -- 172.21.15.102:0/1391550395 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b58053a40 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.265+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b5c06d430 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.266+0000 7f6b51ffb700 1 -- 172.21.15.102:0/1391550395 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6b58057060 con 0x7f6b5c0cd8e0 2023-03-03T08:05:58.267 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:05:58.267 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:05:58.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6b400682c0 msgr2=0x7f6b4006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:58.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6b400682c0 0x7f6b4006a770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0b4d70 tx=0x7f6b4c009210 comp rx=0 tx=0).stop 2023-03-03T08:05:58.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c0cd8e0 msgr2=0x7f6b5c151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:05:58.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c0cd8e0 0x7f6b5c151130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009c40 tx=0x7f6b58004750 comp rx=0 tx=0).stop 2023-03-03T08:05:58.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 shutdown_connections 2023-03-03T08:05:58.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6b400682c0 0x7f6b4006a770 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6b5c0cd8e0 0x7f6b5c151130 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5c1593c0 0x7f6b5c1ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 --2- 172.21.15.102:0/1391550395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b5c153290 0x7f6b5c1eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:05:58.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 >> 172.21.15.102:0/1391550395 conn(0x7f6b5c065fb0 msgr2=0x7f6b5c0cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:05:58.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 shutdown_connections 2023-03-03T08:05:58.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:05:58.269+0000 7f6b633ff700 1 -- 172.21.15.102:0/1391550395 wait complete. 2023-03-03T08:05:58.278 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:05:58.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2023-03-03T08:06:19.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:19.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:49.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:49.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:19.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:19.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:43.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:07:43.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.558+0000 7fa6385ff700 1 Processor -- start 2023-03-03T08:07:43.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.558+0000 7fa6385ff700 1 -- start start 2023-03-03T08:07:43.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.559+0000 7fa6385ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa630153290 0x7fa630158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:07:43.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.559+0000 7fa6385ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa630153660 con 0x7fa6300cd8e0 2023-03-03T08:07:43.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.559+0000 7fa6385ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63006c360 con 0x7fa6301593c0 2023-03-03T08:07:43.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.559+0000 7fa6385ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6300cec40 con 0x7fa630153290 2023-03-03T08:07:43.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa63639b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa630153290 0x7fa630158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:07:43.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa63639b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa630153290 0x7fa630158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45790/0 (socket says 172.21.15.102:45790) 2023-03-03T08:07:43.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa63639b700 1 -- 172.21.15.102:0/3376286217 learned_addr learned my addr 172.21.15.102:0/3376286217 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:07:43.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa636b9c700 1 --1- 172.21.15.102:0/3376286217 >> v1:172.21.15.102:6789/0 conn(0x7fa6300cd8e0 0x7fa6300ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54946/0 (socket says 172.21.15.102:54946) 2023-03-03T08:07:43.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa63639b700 1 -- 172.21.15.102:0/3376286217 >> v1:172.21.15.172:6789/0 conn(0x7fa6301593c0 legacy=0x7fa6300cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:07:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa63639b700 1 -- 172.21.15.102:0/3376286217 >> v1:172.21.15.102:6789/0 conn(0x7fa6300cd8e0 legacy=0x7fa6300ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:07:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa63639b700 1 -- 172.21.15.102:0/3376286217 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63015cea0 con 0x7fa630153290 2023-03-03T08:07:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa635399700 1 -- 172.21.15.102:0/3376286217 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989750616 0 0) 0x7fa63006c360 con 0x7fa6301593c0 2023-03-03T08:07:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.560+0000 7fa63639b700 1 --2- 172.21.15.102:0/3376286217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa630153290 0x7fa630158e80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0098f0 tx=0x7fa62c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0b1452309d9afc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:07:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa635399700 1 -- 172.21.15.102:0/3376286217 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00b8a0 con 0x7fa630153290 2023-03-03T08:07:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa635399700 1 -- 172.21.15.102:0/3376286217 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c00ba00 con 0x7fa630153290 2023-03-03T08:07:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa635399700 1 -- 172.21.15.102:0/3376286217 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00bd90 con 0x7fa630153290 2023-03-03T08:07:43.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa6385ff700 1 -- 172.21.15.102:0/3376286217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa630153290 msgr2=0x7fa630158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:07:43.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3376286217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa630153290 0x7fa630158e80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0098f0 tx=0x7fa62c009c00 comp rx=0 tx=0).stop 2023-03-03T08:07:43.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa6385ff700 1 -- 172.21.15.102:0/3376286217 shutdown_connections 2023-03-03T08:07:43.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3376286217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa630153290 0x7fa630158e80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.561+0000 7fa6385ff700 1 -- 172.21.15.102:0/3376286217 >> 172.21.15.102:0/3376286217 conn(0x7fa630065fb0 msgr2=0x7fa6300663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:07:43.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.562+0000 7fa6385ff700 1 -- 172.21.15.102:0/3376286217 shutdown_connections 2023-03-03T08:07:43.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.562+0000 7fa6385ff700 1 -- 172.21.15.102:0/3376286217 wait complete. 2023-03-03T08:07:43.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.563+0000 7fa6385ff700 1 Processor -- start 2023-03-03T08:07:43.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.564+0000 7fa6385ff700 1 -- start start 2023-03-03T08:07:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa6385ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6300cd8e0 0x7fa6301e60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:07:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa6385ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa630153290 0x7fa6301502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:07:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa6385ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6301593c0 0x7fa6301e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:07:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa6385ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63014fe70 con 0x7fa6300cd8e0 2023-03-03T08:07:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa6385ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300c57d0 con 0x7fa6301593c0 2023-03-03T08:07:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa6385ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300c5a70 con 0x7fa630153290 2023-03-03T08:07:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa63639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6300cd8e0 0x7fa6301e60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:07:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa636b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6301593c0 0x7fa6301e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:07:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa635b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa630153290 0x7fa6301502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:07:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa636b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6301593c0 0x7fa6301e5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48324/0 (socket says 172.21.15.102:48324) 2023-03-03T08:07:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa636b9c700 1 -- 172.21.15.102:0/3236851265 learned_addr learned my addr 172.21.15.102:0/3236851265 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:07:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.565+0000 7fa636b9c700 1 -- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa630153290 msgr2=0x7fa6301502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:07:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa636b9c700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa630153290 0x7fa6301502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa636b9c700 1 -- 172.21.15.102:0/3236851265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6300cd8e0 msgr2=0x7fa6301e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:07:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa636b9c700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6300cd8e0 0x7fa6301e60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa636b9c700 1 -- 172.21.15.102:0/3236851265 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c009550 con 0x7fa6301593c0 2023-03-03T08:07:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa63639b700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6300cd8e0 0x7fa6301e60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:07:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa635b9a700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa630153290 0x7fa6301502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:07:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa636b9c700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6301593c0 0x7fa6301e5720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa62800cbc0 tx=0x7fa62800ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:07:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa626ffd700 1 -- 172.21.15.102:0/3236851265 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62800a9e0 con 0x7fa6301593c0 2023-03-03T08:07:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa626ffd700 1 -- 172.21.15.102:0/3236851265 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62800ab40 con 0x7fa6301593c0 2023-03-03T08:07:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63015d160 con 0x7fa6301593c0 2023-03-03T08:07:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa626ffd700 1 -- 172.21.15.102:0/3236851265 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628007070 con 0x7fa6301593c0 2023-03-03T08:07:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.566+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63006c7f0 con 0x7fa6301593c0 2023-03-03T08:07:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.567+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6300ccde0 con 0x7fa6301593c0 2023-03-03T08:07:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.569+0000 7fa626ffd700 1 -- 172.21.15.102:0/3236851265 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa628019030 con 0x7fa6301593c0 2023-03-03T08:07:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.570+0000 7fa626ffd700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa6140682f0 0x7fa61406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:07:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.570+0000 7fa626ffd700 1 -- 172.21.15.102:0/3236851265 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa6280858b0 con 0x7fa6301593c0 2023-03-03T08:07:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.570+0000 7fa63639b700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa6140682f0 0x7fa61406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:07:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.571+0000 7fa63639b700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa6140682f0 0x7fa61406a7a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa62c014040 tx=0x7fa62c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:07:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.574+0000 7fa626ffd700 1 -- 172.21.15.102:0/3236851265 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa62804b5e0 con 0x7fa6301593c0 2023-03-03T08:07:43.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.810+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa63014f960 con 0x7fa6301593c0 2023-03-03T08:07:43.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.810+0000 7fa626ffd700 1 -- 172.21.15.102:0/3236851265 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa63014f960 con 0x7fa6301593c0 2023-03-03T08:07:43.813 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:07:43.813 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:07:43.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa6140682f0 msgr2=0x7fa61406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:07:43.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa6140682f0 0x7fa61406a7a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa62c014040 tx=0x7fa62c01c040 comp rx=0 tx=0).stop 2023-03-03T08:07:43.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6301593c0 msgr2=0x7fa6301e5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:07:43.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6301593c0 0x7fa6301e5720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa62800cbc0 tx=0x7fa62800ced0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 shutdown_connections 2023-03-03T08:07:43.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6301593c0 0x7fa6301e5720 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6300cd8e0 0x7fa6301e60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa6140682f0 0x7fa61406a7a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 --2- 172.21.15.102:0/3236851265 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa630153290 0x7fa6301502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:07:43.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.813+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 >> 172.21.15.102:0/3236851265 conn(0x7fa630065fb0 msgr2=0x7fa630073680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:07:43.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.814+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 shutdown_connections 2023-03-03T08:07:43.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:07:43.814+0000 7fa6385ff700 1 -- 172.21.15.102:0/3236851265 wait complete. 2023-03-03T08:07:43.817 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:07:43.844 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T08:07:49.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:49.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:20.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:20.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:41.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:08:42.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.095+0000 7fc77c99d700 1 Processor -- start 2023-03-03T08:08:42.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.095+0000 7fc77c99d700 1 -- start start 2023-03-03T08:08:42.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.097+0000 7fc77c99d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7780b3970 0x7fc7780c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:08:42.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.097+0000 7fc77c99d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780c3d20 con 0x7fc7780ced00 2023-03-03T08:08:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.097+0000 7fc77c99d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77815cba0 con 0x7fc7780c41b0 2023-03-03T08:08:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.097+0000 7fc77c99d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7780c4930 con 0x7fc7780b3970 2023-03-03T08:08:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.097+0000 7fc776ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7780b3970 0x7fc7780c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:08:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.097+0000 7fc777fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc7780c41b0 0x7fc778163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:48090/0 (socket says 172.21.15.102:48090) 2023-03-03T08:08:42.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.097+0000 7fc777fff700 1 -- 172.21.15.102:0/3793217947 learned_addr learned my addr 172.21.15.102:0/3793217947 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:08:42.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.098+0000 7fc776ffd700 1 -- 172.21.15.102:0/3793217947 >> v1:172.21.15.172:6789/0 conn(0x7fc7780c41b0 legacy=0x7fc778163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:08:42.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.098+0000 7fc776ffd700 1 -- 172.21.15.102:0/3793217947 >> v1:172.21.15.102:6789/0 conn(0x7fc7780ced00 legacy=0x7fc778158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:08:42.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.098+0000 7fc776ffd700 1 -- 172.21.15.102:0/3793217947 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77815ce80 con 0x7fc7780b3970 2023-03-03T08:08:42.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.098+0000 7fc7767fc700 1 -- 172.21.15.102:0/3793217947 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905073662 0 0) 0x7fc77815cba0 con 0x7fc7780c41b0 2023-03-03T08:08:42.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.098+0000 7fc776ffd700 1 --2- 172.21.15.102:0/3793217947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7780c5a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009840 tx=0x7fc76c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ccde781b1269d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:08:42.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.098+0000 7fc7767fc700 1 -- 172.21.15.102:0/3793217947 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c011830 con 0x7fc7780b3970 2023-03-03T08:08:42.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.098+0000 7fc7767fc700 1 -- 172.21.15.102:0/3793217947 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76c011990 con 0x7fc7780b3970 2023-03-03T08:08:42.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc7767fc700 1 -- 172.21.15.102:0/3793217947 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c011d20 con 0x7fc7780b3970 2023-03-03T08:08:42.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc77c99d700 1 -- 172.21.15.102:0/3793217947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 msgr2=0x7fc7780c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:08:42.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc77c99d700 1 --2- 172.21.15.102:0/3793217947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7780c5a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009840 tx=0x7fc76c009b50 comp rx=0 tx=0).stop 2023-03-03T08:08:42.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc77c99d700 1 -- 172.21.15.102:0/3793217947 shutdown_connections 2023-03-03T08:08:42.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc77c99d700 1 --2- 172.21.15.102:0/3793217947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7780c5a90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc77c99d700 1 -- 172.21.15.102:0/3793217947 >> 172.21.15.102:0/3793217947 conn(0x7fc77808b1c0 msgr2=0x7fc778071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:08:42.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc77c99d700 1 -- 172.21.15.102:0/3793217947 shutdown_connections 2023-03-03T08:08:42.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.099+0000 7fc77c99d700 1 -- 172.21.15.102:0/3793217947 wait complete. 2023-03-03T08:08:42.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.101+0000 7fc77c99d700 1 Processor -- start 2023-03-03T08:08:42.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.101+0000 7fc77c99d700 1 -- start start 2023-03-03T08:08:42.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.102+0000 7fc77c99d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7781e5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:08:42.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.102+0000 7fc77c99d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780c41b0 0x7fc7781ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:08:42.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.102+0000 7fc77c99d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7780ced00 0x7fc7781eece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:08:42.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.102+0000 7fc77c99d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77814d540 con 0x7fc7780b3970 2023-03-03T08:08:42.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.102+0000 7fc77c99d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7781f27c0 con 0x7fc7780ced00 2023-03-03T08:08:42.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.102+0000 7fc77c99d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7781f2aa0 con 0x7fc7780c41b0 2023-03-03T08:08:42.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7781e5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:08:42.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7781e5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47994/0 (socket says 172.21.15.102:47994) 2023-03-03T08:08:42.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 -- 172.21.15.102:0/1427727972 learned_addr learned my addr 172.21.15.102:0/1427727972 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:08:42.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc777fff700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7780ced00 0x7fc7781eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:08:42.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc776ffd700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780c41b0 0x7fc7781ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:08:42.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 -- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780c41b0 msgr2=0x7fc7781ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:08:42.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780c41b0 0x7fc7781ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 -- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7780ced00 msgr2=0x7fc7781eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:08:42.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7780ced00 0x7fc7781eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 -- 172.21.15.102:0/1427727972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c0094a0 con 0x7fc7780b3970 2023-03-03T08:08:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.103+0000 7fc7777fe700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7781e5aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c4e0 tx=0x7fc76800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:08:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.104+0000 7fc75ffff700 1 -- 172.21.15.102:0/1427727972 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680052d0 con 0x7fc7780b3970 2023-03-03T08:08:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.104+0000 7fc75ffff700 1 -- 172.21.15.102:0/1427727972 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc768005430 con 0x7fc7780b3970 2023-03-03T08:08:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.104+0000 7fc75ffff700 1 -- 172.21.15.102:0/1427727972 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680038c0 con 0x7fc7780b3970 2023-03-03T08:08:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.104+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77806c200 con 0x7fc7780b3970 2023-03-03T08:08:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.104+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77806bf20 con 0x7fc7780b3970 2023-03-03T08:08:42.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.105+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77815c890 con 0x7fc7780b3970 2023-03-03T08:08:42.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.106+0000 7fc75ffff700 1 -- 172.21.15.102:0/1427727972 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc768019030 con 0x7fc7780b3970 2023-03-03T08:08:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.107+0000 7fc75ffff700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7640682f0 0x7fc76406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:08:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.108+0000 7fc75ffff700 1 -- 172.21.15.102:0/1427727972 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc768084fe0 con 0x7fc7780b3970 2023-03-03T08:08:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.108+0000 7fc776ffd700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7640682f0 0x7fc76406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:08:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.109+0000 7fc776ffd700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7640682f0 0x7fc76406a7a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0090b0 tx=0x7fc76c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:08:42.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.112+0000 7fc75ffff700 1 -- 172.21.15.102:0/1427727972 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc768051750 con 0x7fc7780b3970 2023-03-03T08:08:42.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.349+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7780cda50 con 0x7fc7780b3970 2023-03-03T08:08:42.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.349+0000 7fc75ffff700 1 -- 172.21.15.102:0/1427727972 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc768017090 con 0x7fc7780b3970 2023-03-03T08:08:42.352 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:08:42.352 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:08:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7640682f0 msgr2=0x7fc76406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:08:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7640682f0 0x7fc76406a7a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0090b0 tx=0x7fc76c01c040 comp rx=0 tx=0).stop 2023-03-03T08:08:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 msgr2=0x7fc7781e5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:08:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7781e5aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c4e0 tx=0x7fc76800c7f0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 shutdown_connections 2023-03-03T08:08:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7780ced00 0x7fc7781eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780b3970 0x7fc7781e5aa0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7640682f0 0x7fc76406a7a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 --2- 172.21.15.102:0/1427727972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780c41b0 0x7fc7781ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:08:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 >> 172.21.15.102:0/1427727972 conn(0x7fc77808b1c0 msgr2=0x7fc7780cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:08:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 shutdown_connections 2023-03-03T08:08:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:08:42.352+0000 7fc77c99d700 1 -- 172.21.15.102:0/1427727972 wait complete. 2023-03-03T08:08:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:08:42.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-03-03T08:08:50.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:50.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:20.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:20.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:50.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:50.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T08:10:00.000+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:10:21.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:21.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:30.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:10:30.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.634+0000 7fd79e19d700 1 Processor -- start 2023-03-03T08:10:30.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.635+0000 7fd79e19d700 1 -- start start 2023-03-03T08:10:30.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.636+0000 7fd79e19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7981564b0 0x7fd7980cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:10:30.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.636+0000 7fd79e19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7980cab50 0x7fd7980cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:10:30.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.636+0000 7fd79e19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7980c9ca0 con 0x7fd7980c9610 2023-03-03T08:10:30.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.636+0000 7fd79e19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7980b4920 con 0x7fd7980cab50 2023-03-03T08:10:30.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.636+0000 7fd79e19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980b4c00 con 0x7fd7981564b0 2023-03-03T08:10:30.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d99c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7980cab50 0x7fd7980cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:10:30.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79c99a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd7980c9610 0x7fd7980ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59338/0 (socket says 172.21.15.102:59338) 2023-03-03T08:10:30.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79c99a700 1 -- 172.21.15.102:0/2261432449 learned_addr learned my addr 172.21.15.102:0/2261432449 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:10:30.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d19b700 1 --2- 172.21.15.102:0/2261432449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7981564b0 0x7fd7980cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:10:30.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d99c700 1 -- 172.21.15.102:0/2261432449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7981564b0 msgr2=0x7fd7980cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:10:30.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d99c700 1 --2- 172.21.15.102:0/2261432449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7981564b0 0x7fd7980cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d99c700 1 -- 172.21.15.102:0/2261432449 >> v1:172.21.15.172:6789/0 conn(0x7fd7980c9610 legacy=0x7fd7980ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:10:30.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd78ffff700 1 -- 172.21.15.102:0/2261432449 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184704706 0 0) 0x7fd7980c9ca0 con 0x7fd7980c9610 2023-03-03T08:10:30.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d99c700 1 -- 172.21.15.102:0/2261432449 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7980b46f0 con 0x7fd7980cab50 2023-03-03T08:10:30.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d19b700 1 --2- 172.21.15.102:0/2261432449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7981564b0 0x7fd7980cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:10:30.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.637+0000 7fd79d99c700 1 --2- 172.21.15.102:0/2261432449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 0x7fd7980cbba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd79000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8796b8e15261f2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:10:30.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd78ffff700 1 -- 172.21.15.102:0/2261432449 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79000e070 con 0x7fd7980cab50 2023-03-03T08:10:30.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd78ffff700 1 -- 172.21.15.102:0/2261432449 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79000bb20 con 0x7fd7980cab50 2023-03-03T08:10:30.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd78ffff700 1 -- 172.21.15.102:0/2261432449 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790012430 con 0x7fd7980cab50 2023-03-03T08:10:30.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd79e19d700 1 -- 172.21.15.102:0/2261432449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 msgr2=0x7fd7980cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:10:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd79e19d700 1 --2- 172.21.15.102:0/2261432449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 0x7fd7980cbba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd79000b330 comp rx=0 tx=0).stop 2023-03-03T08:10:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd79e19d700 1 -- 172.21.15.102:0/2261432449 shutdown_connections 2023-03-03T08:10:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd79e19d700 1 --2- 172.21.15.102:0/2261432449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 0x7fd7980cbba0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd79e19d700 1 --2- 172.21.15.102:0/2261432449 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7981564b0 0x7fd7980cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd79e19d700 1 -- 172.21.15.102:0/2261432449 >> 172.21.15.102:0/2261432449 conn(0x7fd798065fb0 msgr2=0x7fd7980663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:10:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.638+0000 7fd79e19d700 1 -- 172.21.15.102:0/2261432449 shutdown_connections 2023-03-03T08:10:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.639+0000 7fd79e19d700 1 -- 172.21.15.102:0/2261432449 wait complete. 2023-03-03T08:10:30.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.641+0000 7fd79e19d700 1 Processor -- start 2023-03-03T08:10:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.641+0000 7fd79e19d700 1 -- start start 2023-03-03T08:10:30.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79e19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7980c9610 0x7fd798156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:10:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79e19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 0x7fd7981e3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:10:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79e19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7981564b0 0x7fd7981e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:10:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79e19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980cd7a0 con 0x7fd7980cab50 2023-03-03T08:10:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79e19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980c7ee0 con 0x7fd7981564b0 2023-03-03T08:10:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79e19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980cea60 con 0x7fd7980c9610 2023-03-03T08:10:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79d19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7980c9610 0x7fd798156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:10:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79d99c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7981564b0 0x7fd7981e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:10:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79d19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7980c9610 0x7fd798156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:50306/0 (socket says 172.21.15.102:50306) 2023-03-03T08:10:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79d19b700 1 -- 172.21.15.102:0/4256596243 learned_addr learned my addr 172.21.15.102:0/4256596243 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:10:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.642+0000 7fd79c99a700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 0x7fd7981e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:10:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79d99c700 1 -- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7980c9610 msgr2=0x7fd798156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:10:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79d99c700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7980c9610 0x7fd798156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79d99c700 1 -- 172.21.15.102:0/4256596243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 msgr2=0x7fd7981e3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:10:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79d99c700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 0x7fd7981e3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79d99c700 1 -- 172.21.15.102:0/4256596243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940094a0 con 0x7fd7981564b0 2023-03-03T08:10:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79d19b700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7980c9610 0x7fd798156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:10:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79d99c700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7981564b0 0x7fd7981e3eb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd79808c6b0 tx=0x7fd790004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:10:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd78dffb700 1 -- 172.21.15.102:0/4256596243 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790012430 con 0x7fd7981564b0 2023-03-03T08:10:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd78dffb700 1 -- 172.21.15.102:0/4256596243 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd790020070 con 0x7fd7981564b0 2023-03-03T08:10:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.643+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd790009a70 con 0x7fd7981564b0 2023-03-03T08:10:30.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.644+0000 7fd78dffb700 1 -- 172.21.15.102:0/4256596243 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900098f0 con 0x7fd7981564b0 2023-03-03T08:10:30.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.644+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7980b2210 con 0x7fd7981564b0 2023-03-03T08:10:30.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.645+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7980b1f30 con 0x7fd7981564b0 2023-03-03T08:10:30.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.646+0000 7fd78dffb700 1 -- 172.21.15.102:0/4256596243 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd790004040 con 0x7fd7981564b0 2023-03-03T08:10:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.647+0000 7fd78dffb700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd77c0682f0 0x7fd77c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:10:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.647+0000 7fd78dffb700 1 -- 172.21.15.102:0/4256596243 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd790022070 con 0x7fd7981564b0 2023-03-03T08:10:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.647+0000 7fd79d19b700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd77c0682f0 0x7fd77c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:10:30.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.648+0000 7fd79d19b700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd77c0682f0 0x7fd77c06a7a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd794009fd0 tx=0x7fd794009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:10:30.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.652+0000 7fd78dffb700 1 -- 172.21.15.102:0/4256596243 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7900546e0 con 0x7fd7981564b0 2023-03-03T08:10:30.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.882+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd798156880 con 0x7fd7981564b0 2023-03-03T08:10:30.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.883+0000 7fd78dffb700 1 -- 172.21.15.102:0/4256596243 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd790057d00 con 0x7fd7981564b0 2023-03-03T08:10:30.885 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:10:30.885 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:10:30.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd77c0682f0 msgr2=0x7fd77c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:10:30.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd77c0682f0 0x7fd77c06a7a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd794009fd0 tx=0x7fd794009210 comp rx=0 tx=0).stop 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7981564b0 msgr2=0x7fd7981e3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7981564b0 0x7fd7981e3eb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd79808c6b0 tx=0x7fd790004ad0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 shutdown_connections 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd77c0682f0 0x7fd77c06a7a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd7980c9610 0x7fd798156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7981564b0 0x7fd7981e3eb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 --2- 172.21.15.102:0/4256596243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7980cab50 0x7fd7981e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:10:30.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 >> 172.21.15.102:0/4256596243 conn(0x7fd798065fb0 msgr2=0x7fd7980b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:10:30.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 shutdown_connections 2023-03-03T08:10:30.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:10:30.886+0000 7fd79e19d700 1 -- 172.21.15.102:0/4256596243 wait complete. 2023-03-03T08:10:30.889 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:10:30.914 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T08:10:51.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:51.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:21.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:21.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:51.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:51.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:08.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:12:09.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.165+0000 7f18e11ff700 1 Processor -- start 2023-03-03T08:12:09.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.165+0000 7f18e11ff700 1 -- start start 2023-03-03T08:12:09.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18e11ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f18dc0cea00 0x7f18dc06cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:12:09.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18e11ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc0cd040 0x7f18dc15c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:12:09.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18e11ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc15cd00 con 0x7f18dc152eb0 2023-03-03T08:12:09.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18e11ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18dc0ce5b0 con 0x7f18dc0cea00 2023-03-03T08:12:09.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18e11ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc15cfa0 con 0x7f18dc0cd040 2023-03-03T08:12:09.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18d3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f18dc0cea00 0x7f18dc06cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:12:09.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18db59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc0cd040 0x7f18dc15c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:12:09.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18db59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc0cd040 0x7f18dc15c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41308/0 (socket says 172.21.15.102:41308) 2023-03-03T08:12:09.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18db59e700 1 -- 172.21.15.102:0/729204411 learned_addr learned my addr 172.21.15.102:0/729204411 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:12:09.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18d3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f18dc0cea00 0x7f18dc06cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38798/0 (socket says 172.21.15.102:38798) 2023-03-03T08:12:09.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18db59e700 1 -- 172.21.15.102:0/729204411 >> v2:172.21.15.172:3300/0 conn(0x7f18dc0cea00 msgr2=0x7f18dc06cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:12:09.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.167+0000 7f18db59e700 1 --2- 172.21.15.102:0/729204411 >> v2:172.21.15.172:3300/0 conn(0x7f18dc0cea00 0x7f18dc06cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.168+0000 7f18db59e700 1 -- 172.21.15.102:0/729204411 >> v1:172.21.15.172:6789/0 conn(0x7f18dc152eb0 legacy=0x7f18dc06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:12:09.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.168+0000 7f18db59e700 1 -- 172.21.15.102:0/729204411 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0cf750 con 0x7f18dc0cd040 2023-03-03T08:12:09.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.168+0000 7f18db59e700 1 --2- 172.21.15.102:0/729204411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc0cd040 0x7f18dc15c160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009e50 tx=0x7f18d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13e98630c12c333e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:12:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.168+0000 7f18da59c700 1 -- 172.21.15.102:0/729204411 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d400e070 con 0x7f18dc0cd040 2023-03-03T08:12:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.168+0000 7f18da59c700 1 -- 172.21.15.102:0/729204411 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d400b910 con 0x7f18dc0cd040 2023-03-03T08:12:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.168+0000 7f18da59c700 1 -- 172.21.15.102:0/729204411 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d400bcc0 con 0x7f18dc0cd040 2023-03-03T08:12:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 -- 172.21.15.102:0/729204411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc0cd040 msgr2=0x7f18dc15c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:12:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 --2- 172.21.15.102:0/729204411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc0cd040 0x7f18dc15c160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009e50 tx=0x7f18d400b330 comp rx=0 tx=0).stop 2023-03-03T08:12:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 -- 172.21.15.102:0/729204411 shutdown_connections 2023-03-03T08:12:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 --2- 172.21.15.102:0/729204411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc0cd040 0x7f18dc15c160 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 --2- 172.21.15.102:0/729204411 >> v2:172.21.15.172:3300/0 conn(0x7f18dc0cea00 0x7f18dc06cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 -- 172.21.15.102:0/729204411 >> 172.21.15.102:0/729204411 conn(0x7f18dc066c40 msgr2=0x7f18dc0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:12:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 -- 172.21.15.102:0/729204411 shutdown_connections 2023-03-03T08:12:09.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.169+0000 7f18e11ff700 1 -- 172.21.15.102:0/729204411 wait complete. 2023-03-03T08:12:09.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.171+0000 7f18e11ff700 1 Processor -- start 2023-03-03T08:12:09.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.171+0000 7f18e11ff700 1 -- start start 2023-03-03T08:12:09.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18e11ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18dc0cd040 0x7f18dc1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:12:09.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18e11ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18dc0cea00 0x7f18dc1f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:12:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18e11ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc152eb0 0x7f18dc1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:12:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18e11ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1ee420 con 0x7f18dc0cd040 2023-03-03T08:12:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18e11ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0b1400 con 0x7f18dc0cea00 2023-03-03T08:12:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18e11ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc150a50 con 0x7f18dc152eb0 2023-03-03T08:12:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18dad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18dc0cd040 0x7f18dc1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:12:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18dad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18dc0cd040 0x7f18dc1516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41548/0 (socket says 172.21.15.102:41548) 2023-03-03T08:12:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18db59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc152eb0 0x7f18dc1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18dad9d700 1 -- 172.21.15.102:0/2880247862 learned_addr learned my addr 172.21.15.102:0/2880247862 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.172+0000 7f18d3fff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18dc0cea00 0x7f18dc1f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18db59e700 1 -- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18dc0cea00 msgr2=0x7f18dc1f1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18db59e700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18dc0cea00 0x7f18dc1f1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18db59e700 1 -- 172.21.15.102:0/2880247862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18dc0cd040 msgr2=0x7f18dc1516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18db59e700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18dc0cd040 0x7f18dc1516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18db59e700 1 -- 172.21.15.102:0/2880247862 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4009b30 con 0x7f18dc152eb0 2023-03-03T08:12:09.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18d3fff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18dc0cea00 0x7f18dc1f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:12:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18db59e700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc152eb0 0x7f18dc1f2520 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f18d4015000 tx=0x7f18d4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:12:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18d37fe700 1 -- 172.21.15.102:0/2880247862 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4004780 con 0x7f18dc152eb0 2023-03-03T08:12:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.173+0000 7f18d37fe700 1 -- 172.21.15.102:0/2880247862 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d40092e0 con 0x7f18dc152eb0 2023-03-03T08:12:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.174+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc153280 con 0x7f18dc152eb0 2023-03-03T08:12:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.174+0000 7f18d37fe700 1 -- 172.21.15.102:0/2880247862 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d401c680 con 0x7f18dc152eb0 2023-03-03T08:12:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.174+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc0cfa30 con 0x7f18dc152eb0 2023-03-03T08:12:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.175+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc0cfd10 con 0x7f18dc152eb0 2023-03-03T08:12:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.176+0000 7f18d37fe700 1 -- 172.21.15.102:0/2880247862 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18d4022030 con 0x7f18dc152eb0 2023-03-03T08:12:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.177+0000 7f18d37fe700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18bc0682f0 0x7f18bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:12:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.177+0000 7f18d37fe700 1 -- 172.21.15.102:0/2880247862 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f18d40874a0 con 0x7f18dc152eb0 2023-03-03T08:12:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.177+0000 7f18dad9d700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18bc0682f0 0x7f18bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:12:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.178+0000 7f18dad9d700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18bc0682f0 0x7f18bc06a7a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f18cc001880 tx=0x7f18cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:12:09.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.181+0000 7f18d37fe700 1 -- 172.21.15.102:0/2880247862 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18d4053bb0 con 0x7f18dc152eb0 2023-03-03T08:12:09.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.414+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18dc0c5d60 con 0x7f18dc152eb0 2023-03-03T08:12:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.415+0000 7f18d37fe700 1 -- 172.21.15.102:0/2880247862 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f18d40571d0 con 0x7f18dc152eb0 2023-03-03T08:12:09.417 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:12:09.417 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:12:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18bc0682f0 msgr2=0x7f18bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:12:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18bc0682f0 0x7f18bc06a7a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f18cc001880 tx=0x7f18cc009040 comp rx=0 tx=0).stop 2023-03-03T08:12:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc152eb0 msgr2=0x7f18dc1f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc152eb0 0x7f18dc1f2520 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f18d4015000 tx=0x7f18d4004ad0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 shutdown_connections 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18bc0682f0 0x7f18bc06a7a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18dc152eb0 0x7f18dc1f2520 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18dc0cd040 0x7f18dc1516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 --2- 172.21.15.102:0/2880247862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18dc0cea00 0x7f18dc1f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 >> 172.21.15.102:0/2880247862 conn(0x7f18dc066c40 msgr2=0x7f18dc0b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:12:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 shutdown_connections 2023-03-03T08:12:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:12:09.418+0000 7f18e11ff700 1 -- 172.21.15.102:0/2880247862 wait complete. 2023-03-03T08:12:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:12:09.447 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-03-03T08:12:21.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:21.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:52.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:52.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:22.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:22.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:45.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:13:45.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.713+0000 7f10e25ff700 1 Processor -- start 2023-03-03T08:13:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.713+0000 7f10e25ff700 1 -- start start 2023-03-03T08:13:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.714+0000 7f10e25ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10dc0ced80 0x7f10dc0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10e25ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc1597f0 0x7f10dc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10e25ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc0cb9f0 con 0x7f10dc0cf9a0 2023-03-03T08:13:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10e25ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc0ca750 con 0x7f10dc0ced80 2023-03-03T08:13:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10e25ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc15a230 con 0x7f10dc1597f0 2023-03-03T08:13:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10dbfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10dc0ced80 0x7f10dc0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10db7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc1597f0 0x7f10dc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10e0b9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f10dc0cf9a0 0x7f10dc15c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:55922/0 (socket says 172.21.15.102:55922) 2023-03-03T08:13:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.715+0000 7f10e0b9c700 1 -- 172.21.15.102:0/908938511 learned_addr learned my addr 172.21.15.102:0/908938511 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:13:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.716+0000 7f10dbfff700 1 -- 172.21.15.102:0/908938511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc1597f0 msgr2=0x7f10dc0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:13:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.716+0000 7f10dbfff700 1 --2- 172.21.15.102:0/908938511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc1597f0 0x7f10dc0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.716+0000 7f10dbfff700 1 -- 172.21.15.102:0/908938511 >> v1:172.21.15.172:6789/0 conn(0x7f10dc0cf9a0 legacy=0x7f10dc15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.716+0000 7f10dbfff700 1 -- 172.21.15.102:0/908938511 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc159c80 con 0x7f10dc0ced80 2023-03-03T08:13:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.716+0000 7f10dbfff700 1 --2- 172.21.15.102:0/908938511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc0ced80 0x7f10dc0cf150 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009840 tx=0x7f10cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a0d2e9f66accda2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.716+0000 7f10daffd700 1 -- 172.21.15.102:0/908938511 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc011830 con 0x7f10dc0ced80 2023-03-03T08:13:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.716+0000 7f10daffd700 1 -- 172.21.15.102:0/908938511 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc011990 con 0x7f10dc0ced80 2023-03-03T08:13:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10daffd700 1 -- 172.21.15.102:0/908938511 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc011d20 con 0x7f10dc0ced80 2023-03-03T08:13:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 -- 172.21.15.102:0/908938511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc0ced80 msgr2=0x7f10dc0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 --2- 172.21.15.102:0/908938511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc0ced80 0x7f10dc0cf150 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009840 tx=0x7f10cc009b50 comp rx=0 tx=0).stop 2023-03-03T08:13:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 -- 172.21.15.102:0/908938511 shutdown_connections 2023-03-03T08:13:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 --2- 172.21.15.102:0/908938511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc0ced80 0x7f10dc0cf150 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 --2- 172.21.15.102:0/908938511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc1597f0 0x7f10dc0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 -- 172.21.15.102:0/908938511 >> 172.21.15.102:0/908938511 conn(0x7f10dc065fb0 msgr2=0x7f10dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:13:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 -- 172.21.15.102:0/908938511 shutdown_connections 2023-03-03T08:13:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.717+0000 7f10e25ff700 1 -- 172.21.15.102:0/908938511 wait complete. 2023-03-03T08:13:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.719+0000 7f10e25ff700 1 Processor -- start 2023-03-03T08:13:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.720+0000 7f10e25ff700 1 -- start start 2023-03-03T08:13:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e25ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10dc0ced80 0x7f10dc15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e25ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc0cf9a0 0x7f10dc14f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e25ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc1597f0 0x7f10dc14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e25ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc14df10 con 0x7f10dc0ced80 2023-03-03T08:13:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e25ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc150cc0 con 0x7f10dc1597f0 2023-03-03T08:13:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e25ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc150830 con 0x7f10dc0cf9a0 2023-03-03T08:13:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e0b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc1597f0 0x7f10dc14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10db7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc0cf9a0 0x7f10dc14f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10dbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10dc0ced80 0x7f10dc15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e0b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc1597f0 0x7f10dc14f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:32914/0 (socket says 172.21.15.102:32914) 2023-03-03T08:13:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.721+0000 7f10e0b9c700 1 -- 172.21.15.102:0/365912957 learned_addr learned my addr 172.21.15.102:0/365912957 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:13:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e0b9c700 1 -- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc0cf9a0 msgr2=0x7f10dc14f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e0b9c700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc0cf9a0 0x7f10dc14f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e0b9c700 1 -- 172.21.15.102:0/365912957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10dc0ced80 msgr2=0x7f10dc15a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e0b9c700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10dc0ced80 0x7f10dc15a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e0b9c700 1 -- 172.21.15.102:0/365912957 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc0094a0 con 0x7f10dc1597f0 2023-03-03T08:13:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10db7fe700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc0cf9a0 0x7f10dc14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:13:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10dbfff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10dc0ced80 0x7f10dc15a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:13:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e0b9c700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc1597f0 0x7f10dc14f890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f10d400ca80 tx=0x7f10d400cd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10d8ff9700 1 -- 172.21.15.102:0/365912957 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400a9e0 con 0x7f10dc1597f0 2023-03-03T08:13:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10d8ff9700 1 -- 172.21.15.102:0/365912957 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d400ab40 con 0x7f10dc1597f0 2023-03-03T08:13:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10d8ff9700 1 -- 172.21.15.102:0/365912957 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4012400 con 0x7f10dc1597f0 2023-03-03T08:13:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc0cb0c0 con 0x7f10dc1597f0 2023-03-03T08:13:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.722+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc0cade0 con 0x7f10dc1597f0 2023-03-03T08:13:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.723+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc0cb3a0 con 0x7f10dc1597f0 2023-03-03T08:13:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.725+0000 7f10d8ff9700 1 -- 172.21.15.102:0/365912957 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f10d4019030 con 0x7f10dc1597f0 2023-03-03T08:13:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.726+0000 7f10d8ff9700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10c80682f0 0x7f10c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.726+0000 7f10d8ff9700 1 -- 172.21.15.102:0/365912957 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f10d4085a70 con 0x7f10dc1597f0 2023-03-03T08:13:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.726+0000 7f10dbfff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10c80682f0 0x7f10c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.727+0000 7f10dbfff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10c80682f0 0x7f10c806a7a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.729+0000 7f10d8ff9700 1 -- 172.21.15.102:0/365912957 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10d40522b0 con 0x7f10dc1597f0 2023-03-03T08:13:45.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.966+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10dc14f070 con 0x7f10dc1597f0 2023-03-03T08:13:45.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.967+0000 7f10d8ff9700 1 -- 172.21.15.102:0/365912957 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f10dc14f070 con 0x7f10dc1597f0 2023-03-03T08:13:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:13:45.969 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:13:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10c80682f0 msgr2=0x7f10c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10c80682f0 0x7f10c806a7a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc01c040 comp rx=0 tx=0).stop 2023-03-03T08:13:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc1597f0 msgr2=0x7f10dc14f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc1597f0 0x7f10dc14f890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f10d400ca80 tx=0x7f10d400cd90 comp rx=0 tx=0).stop 2023-03-03T08:13:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 shutdown_connections 2023-03-03T08:13:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10dc1597f0 0x7f10dc14f890 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10dc0ced80 0x7f10dc15a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10c80682f0 0x7f10c806a7a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 --2- 172.21.15.102:0/365912957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10dc0cf9a0 0x7f10dc14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 >> 172.21.15.102:0/365912957 conn(0x7f10dc065fb0 msgr2=0x7f10dc0cbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:13:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 shutdown_connections 2023-03-03T08:13:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:45.970+0000 7f10e25ff700 1 -- 172.21.15.102:0/365912957 wait complete. 2023-03-03T08:13:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:13:45.998 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-03-03T08:13:52.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:52.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:52.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:13:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.250+0000 7fc77d5ff700 1 Processor -- start 2023-03-03T08:13:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.250+0000 7fc77d5ff700 1 -- start start 2023-03-03T08:13:53.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.251+0000 7fc77d5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc778153290 0x7fc778158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:53.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.251+0000 7fc77d5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc7780cd8e0 0x7fc7780ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:53.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.251+0000 7fc77d5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778153660 con 0x7fc7781593c0 2023-03-03T08:13:53.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.251+0000 7fc77d5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc77806c360 con 0x7fc778153290 2023-03-03T08:13:53.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.251+0000 7fc77d5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc77815cea0 con 0x7fc7780cd8e0 2023-03-03T08:13:53.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.251+0000 7fc776ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc778153290 0x7fc778158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:53.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc7777fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc7780cd8e0 0x7fc7780ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:53.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc7767fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc7781593c0 0x7fc7780cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44120/0 (socket says 172.21.15.102:44120) 2023-03-03T08:13:53.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc7767fc700 1 -- 172.21.15.102:0/2005370499 learned_addr learned my addr 172.21.15.102:0/2005370499 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:13:53.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc776ffd700 1 -- 172.21.15.102:0/2005370499 >> v2:172.21.15.172:3300/0 conn(0x7fc7780cd8e0 msgr2=0x7fc7780ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:13:53.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc776ffd700 1 --2- 172.21.15.102:0/2005370499 >> v2:172.21.15.172:3300/0 conn(0x7fc7780cd8e0 0x7fc7780ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc776ffd700 1 -- 172.21.15.102:0/2005370499 >> v1:172.21.15.102:6789/0 conn(0x7fc7781593c0 legacy=0x7fc7780cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc776ffd700 1 -- 172.21.15.102:0/2005370499 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77806c7f0 con 0x7fc778153290 2023-03-03T08:13:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc775ffb700 1 -- 172.21.15.102:0/2005370499 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837182737 0 0) 0x7fc778153660 con 0x7fc7781593c0 2023-03-03T08:13:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.252+0000 7fc776ffd700 1 --2- 172.21.15.102:0/2005370499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc778158e80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc7780c9390 tx=0x7fc768009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60890b52089cef1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc775ffb700 1 -- 172.21.15.102:0/2005370499 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768011830 con 0x7fc778153290 2023-03-03T08:13:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc775ffb700 1 -- 172.21.15.102:0/2005370499 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc768011990 con 0x7fc778153290 2023-03-03T08:13:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc775ffb700 1 -- 172.21.15.102:0/2005370499 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768011d20 con 0x7fc778153290 2023-03-03T08:13:53.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 -- 172.21.15.102:0/2005370499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 msgr2=0x7fc778158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:53.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/2005370499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc778158e80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc7780c9390 tx=0x7fc768009b20 comp rx=0 tx=0).stop 2023-03-03T08:13:53.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 -- 172.21.15.102:0/2005370499 shutdown_connections 2023-03-03T08:13:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/2005370499 >> v2:172.21.15.172:3300/0 conn(0x7fc7780cd8e0 0x7fc7780ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/2005370499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc778158e80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 -- 172.21.15.102:0/2005370499 >> 172.21.15.102:0/2005370499 conn(0x7fc778065fb0 msgr2=0x7fc7780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:13:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 -- 172.21.15.102:0/2005370499 shutdown_connections 2023-03-03T08:13:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.253+0000 7fc77d5ff700 1 -- 172.21.15.102:0/2005370499 wait complete. 2023-03-03T08:13:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.255+0000 7fc77d5ff700 1 Processor -- start 2023-03-03T08:13:53.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.256+0000 7fc77d5ff700 1 -- start start 2023-03-03T08:13:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc77d5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780cd8e0 0x7fc7780b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc77d5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc7780c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc77d5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7781593c0 0x7fc77814dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc77d5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7781eef20 con 0x7fc778153290 2023-03-03T08:13:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc77d5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7781ee650 con 0x7fc7781593c0 2023-03-03T08:13:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc77d5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc778156130 con 0x7fc7780cd8e0 2023-03-03T08:13:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc7767fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc7780c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc776ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780cd8e0 0x7fc7780b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc7767fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc7780c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41632/0 (socket says 172.21.15.102:41632) 2023-03-03T08:13:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc7767fc700 1 -- 172.21.15.102:0/1607573410 learned_addr learned my addr 172.21.15.102:0/1607573410 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:13:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc7777fe700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7781593c0 0x7fc77814dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc776ffd700 1 -- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7781593c0 msgr2=0x7fc77814dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc776ffd700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7781593c0 0x7fc77814dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc776ffd700 1 -- 172.21.15.102:0/1607573410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 msgr2=0x7fc7780c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:53.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.257+0000 7fc776ffd700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc7780c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.258+0000 7fc776ffd700 1 -- 172.21.15.102:0/1607573410 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680094a0 con 0x7fc7780cd8e0 2023-03-03T08:13:53.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.258+0000 7fc776ffd700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780cd8e0 0x7fc7780b2cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc768016000 tx=0x7fc768004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:53.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.258+0000 7fc75f7fe700 1 -- 172.21.15.102:0/1607573410 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768004550 con 0x7fc7780cd8e0 2023-03-03T08:13:53.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.258+0000 7fc75f7fe700 1 -- 172.21.15.102:0/1607573410 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76801f070 con 0x7fc7780cd8e0 2023-03-03T08:13:53.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.258+0000 7fc75f7fe700 1 -- 172.21.15.102:0/1607573410 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680038e0 con 0x7fc7780cd8e0 2023-03-03T08:13:53.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.258+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7780ccde0 con 0x7fc7780cd8e0 2023-03-03T08:13:53.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.258+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77815d160 con 0x7fc7780cd8e0 2023-03-03T08:13:53.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.259+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7780cd0c0 con 0x7fc7780cd8e0 2023-03-03T08:13:53.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.261+0000 7fc75f7fe700 1 -- 172.21.15.102:0/1607573410 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc76800f050 con 0x7fc7780cd8e0 2023-03-03T08:13:53.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.262+0000 7fc75f7fe700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.262+0000 7fc75f7fe700 1 -- 172.21.15.102:0/1607573410 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc768021070 con 0x7fc7780cd8e0 2023-03-03T08:13:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.262+0000 7fc7767fc700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.263+0000 7fc7767fc700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc778157320 tx=0x7fc76c007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:53.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.266+0000 7fc75f7fe700 1 -- 172.21.15.102:0/1607573410 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc768053c80 con 0x7fc7780cd8e0 2023-03-03T08:13:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.499+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc778157370 con 0x7fc7780cd8e0 2023-03-03T08:13:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.499+0000 7fc75f7fe700 1 -- 172.21.15.102:0/1607573410 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc768015090 con 0x7fc7780cd8e0 2023-03-03T08:13:53.501 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:13:53.501 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:13:53.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 msgr2=0x7fc76406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:53.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc778157320 tx=0x7fc76c007ea0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780cd8e0 msgr2=0x7fc7780b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780cd8e0 0x7fc7780b2cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc768016000 tx=0x7fc768004750 comp rx=0 tx=0).stop 2023-03-03T08:13:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 shutdown_connections 2023-03-03T08:13:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7780cd8e0 0x7fc7780b2cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7781593c0 0x7fc77814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 --2- 172.21.15.102:0/1607573410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc778153290 0x7fc7780c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 >> 172.21.15.102:0/1607573410 conn(0x7fc778065fb0 msgr2=0x7fc778150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:13:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 shutdown_connections 2023-03-03T08:13:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:53.502+0000 7fc77d5ff700 1 -- 172.21.15.102:0/1607573410 wait complete. 2023-03-03T08:13:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:13:53.530 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-03-03T08:13:56.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:13:56.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.779+0000 7f78509fe700 1 Processor -- start 2023-03-03T08:13:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.779+0000 7f78509fe700 1 -- start start 2023-03-03T08:13:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f78509fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f784c1564b0 0x7f784c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f78509fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f784c0c9610 0x7f784c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f78509fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0ca440 0x7f784c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f78509fe700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784c0c9ca0 con 0x7f784c0c9610 2023-03-03T08:13:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f78509fe700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784c0b46f0 con 0x7f784c1564b0 2023-03-03T08:13:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f78509fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0b4920 con 0x7f784c0ca440 2023-03-03T08:13:56.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f784a59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f784c0c9610 0x7f784c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:56.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f784ad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f784c1564b0 0x7f784c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:56.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f784a59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f784c0c9610 0x7f784c0cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44824/0 (socket says 172.21.15.102:44824) 2023-03-03T08:13:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f784a59c700 1 -- 172.21.15.102:0/1324828487 learned_addr learned my addr 172.21.15.102:0/1324828487 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:13:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.781+0000 7f784b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0ca440 0x7f784c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f784a59c700 1 -- 172.21.15.102:0/1324828487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0ca440 msgr2=0x7f784c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:13:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f784a59c700 1 --2- 172.21.15.102:0/1324828487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0ca440 0x7f784c0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f784a59c700 1 -- 172.21.15.102:0/1324828487 >> v2:172.21.15.172:3300/0 conn(0x7f784c1564b0 msgr2=0x7f784c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:13:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f784a59c700 1 --2- 172.21.15.102:0/1324828487 >> v2:172.21.15.172:3300/0 conn(0x7f784c1564b0 0x7f784c0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f784a59c700 1 -- 172.21.15.102:0/1324828487 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c0b4c00 con 0x7f784c0c9610 2023-03-03T08:13:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f784a59c700 1 --2- 172.21.15.102:0/1324828487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0c9610 0x7f784c0cbba0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f784c08c600 tx=0x7f7840009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870b8b84f1bdb983 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f7849d9b700 1 -- 172.21.15.102:0/1324828487 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7840011830 con 0x7f784c0c9610 2023-03-03T08:13:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f7849d9b700 1 -- 172.21.15.102:0/1324828487 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7840011990 con 0x7f784c0c9610 2023-03-03T08:13:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.782+0000 7f7849d9b700 1 -- 172.21.15.102:0/1324828487 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7840011d20 con 0x7f784c0c9610 2023-03-03T08:13:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 -- 172.21.15.102:0/1324828487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0c9610 msgr2=0x7f784c0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 --2- 172.21.15.102:0/1324828487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0c9610 0x7f784c0cbba0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f784c08c600 tx=0x7f7840009bd0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 -- 172.21.15.102:0/1324828487 shutdown_connections 2023-03-03T08:13:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 --2- 172.21.15.102:0/1324828487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0ca440 0x7f784c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 --2- 172.21.15.102:0/1324828487 >> v2:172.21.15.172:3300/0 conn(0x7f784c1564b0 0x7f784c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 --2- 172.21.15.102:0/1324828487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0c9610 0x7f784c0cbba0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 -- 172.21.15.102:0/1324828487 >> 172.21.15.102:0/1324828487 conn(0x7f784c065fb0 msgr2=0x7f784c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:13:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 -- 172.21.15.102:0/1324828487 shutdown_connections 2023-03-03T08:13:56.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.783+0000 7f78509fe700 1 -- 172.21.15.102:0/1324828487 wait complete. 2023-03-03T08:13:56.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.785+0000 7f78509fe700 1 Processor -- start 2023-03-03T08:13:56.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.785+0000 7f78509fe700 1 -- start start 2023-03-03T08:13:56.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.786+0000 7f78509fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0c9610 0x7f784c158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:56.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.786+0000 7f78509fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0ca440 0x7f784c15c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.786+0000 7f78509fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f784c1564b0 0x7f784c15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.786+0000 7f78509fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c15a0a0 con 0x7f784c0ca440 2023-03-03T08:13:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.786+0000 7f78509fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0cdea0 con 0x7f784c1564b0 2023-03-03T08:13:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.786+0000 7f78509fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c15a2a0 con 0x7f784c0c9610 2023-03-03T08:13:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0c9610 0x7f784c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f784c1564b0 0x7f784c15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0ca440 0x7f784c15c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:56.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0c9610 0x7f784c158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48562/0 (socket says 172.21.15.102:48562) 2023-03-03T08:13:56.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784ad9d700 1 -- 172.21.15.102:0/561744677 learned_addr learned my addr 172.21.15.102:0/561744677 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:13:56.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f784c1564b0 0x7f784c15cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50308/0 (socket says 172.21.15.102:50308) 2023-03-03T08:13:56.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784a59c700 1 -- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0c9610 msgr2=0x7f784c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:56.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784a59c700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0c9610 0x7f784c158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784a59c700 1 -- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f784c1564b0 msgr2=0x7f784c15cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:56.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784a59c700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f784c1564b0 0x7f784c15cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:56.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784a59c700 1 -- 172.21.15.102:0/561744677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7840009550 con 0x7f784c0ca440 2023-03-03T08:13:56.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.787+0000 7f784a59c700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0ca440 0x7f784c15c980 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f784c15d410 tx=0x7f7840004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:56.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.788+0000 7f78337fe700 1 -- 172.21.15.102:0/561744677 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7840004450 con 0x7f784c0ca440 2023-03-03T08:13:56.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.788+0000 7f78337fe700 1 -- 172.21.15.102:0/561744677 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78400036a0 con 0x7f784c0ca440 2023-03-03T08:13:56.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.788+0000 7f78337fe700 1 -- 172.21.15.102:0/561744677 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78400045f0 con 0x7f784c0ca440 2023-03-03T08:13:56.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.788+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784c0b2210 con 0x7f784c0ca440 2023-03-03T08:13:56.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.788+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784c0b1f30 con 0x7f784c0ca440 2023-03-03T08:13:56.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.789+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784c156880 con 0x7f784c0ca440 2023-03-03T08:13:56.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.790+0000 7f78337fe700 1 -- 172.21.15.102:0/561744677 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f784000f050 con 0x7f784c0ca440 2023-03-03T08:13:56.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.792+0000 7f78337fe700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7838068320 0x7f783806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:13:56.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.792+0000 7f78337fe700 1 -- 172.21.15.102:0/561744677 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7840021030 con 0x7f784c0ca440 2023-03-03T08:13:56.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.792+0000 7f784ad9d700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7838068320 0x7f783806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:13:56.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.793+0000 7f784ad9d700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7838068320 0x7f783806a7d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f783c009c00 tx=0x7f783c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:13:56.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:56.797+0000 7f78337fe700 1 -- 172.21.15.102:0/561744677 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7840053cb0 con 0x7f784c0ca440 2023-03-03T08:13:57.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.034+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f784c0677f0 con 0x7f784c0ca440 2023-03-03T08:13:57.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.035+0000 7f78337fe700 1 -- 172.21.15.102:0/561744677 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f78400572d0 con 0x7f784c0ca440 2023-03-03T08:13:57.036 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:13:57.036 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:13:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7838068320 msgr2=0x7f783806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7838068320 0x7f783806a7d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f783c009c00 tx=0x7f783c009210 comp rx=0 tx=0).stop 2023-03-03T08:13:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0ca440 msgr2=0x7f784c15c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:13:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0ca440 0x7f784c15c980 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f784c15d410 tx=0x7f7840004750 comp rx=0 tx=0).stop 2023-03-03T08:13:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 shutdown_connections 2023-03-03T08:13:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7838068320 0x7f783806a7d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f784c0c9610 0x7f784c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f784c1564b0 0x7f784c15cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 --2- 172.21.15.102:0/561744677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f784c0ca440 0x7f784c15c980 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:13:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 >> 172.21.15.102:0/561744677 conn(0x7f784c065fb0 msgr2=0x7f784c155590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:13:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.037+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 shutdown_connections 2023-03-03T08:13:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:13:57.038+0000 7f78509fe700 1 -- 172.21.15.102:0/561744677 wait complete. 2023-03-03T08:13:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:13:57.065 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-03-03T08:14:22.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:22.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:52.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:52.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:18.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:15:18.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.313+0000 7fd546dff700 1 Processor -- start 2023-03-03T08:15:18.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.314+0000 7fd546dff700 1 -- start start 2023-03-03T08:15:18.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.315+0000 7fd546dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd540150c80 0x7fd54006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:15:18.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.315+0000 7fd546dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd5400c5f30 0x7fd5400c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:15:18.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.315+0000 7fd546dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5400c5390 con 0x7fd5400c2020 2023-03-03T08:15:18.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.315+0000 7fd546dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd540157ad0 con 0x7fd540150c80 2023-03-03T08:15:18.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.315+0000 7fd546dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5400c4a20 con 0x7fd5400c5f30 2023-03-03T08:15:18.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd544b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd540150c80 0x7fd54006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:15:18.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd54539c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd5400c5f30 0x7fd5400c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:15:18.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd544b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd540150c80 0x7fd54006d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38110/0 (socket says 172.21.15.102:38110) 2023-03-03T08:15:18.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd54539c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd5400c5f30 0x7fd5400c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46498/0 (socket says 172.21.15.102:46498) 2023-03-03T08:15:18.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd544b9b700 1 -- 172.21.15.102:0/1872708498 learned_addr learned my addr 172.21.15.102:0/1872708498 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd544b9b700 1 -- 172.21.15.102:0/1872708498 >> v2:172.21.15.172:3300/0 conn(0x7fd5400c5f30 msgr2=0x7fd5400c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd544b9b700 1 --2- 172.21.15.102:0/1872708498 >> v2:172.21.15.172:3300/0 conn(0x7fd5400c5f30 0x7fd5400c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd544b9b700 1 -- 172.21.15.102:0/1872708498 >> v1:172.21.15.102:6789/0 conn(0x7fd5400c2020 legacy=0x7fd5400c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd544b9b700 1 -- 172.21.15.102:0/1872708498 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54006f010 con 0x7fd540150c80 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.316+0000 7fd53f7fe700 1 -- 172.21.15.102:0/1872708498 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234770462 0 0) 0x7fd5400c5390 con 0x7fd5400c2020 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.317+0000 7fd544b9b700 1 --2- 172.21.15.102:0/1872708498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd54006d100 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd5400cbac0 tx=0x7fd534009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dabc2b218491631 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.317+0000 7fd53f7fe700 1 -- 172.21.15.102:0/1872708498 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534011830 con 0x7fd540150c80 2023-03-03T08:15:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.317+0000 7fd53f7fe700 1 -- 172.21.15.102:0/1872708498 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534011990 con 0x7fd540150c80 2023-03-03T08:15:18.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.317+0000 7fd53f7fe700 1 -- 172.21.15.102:0/1872708498 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534011d20 con 0x7fd540150c80 2023-03-03T08:15:18.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.317+0000 7fd546dff700 1 -- 172.21.15.102:0/1872708498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 msgr2=0x7fd54006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:15:18.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.317+0000 7fd546dff700 1 --2- 172.21.15.102:0/1872708498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd54006d100 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd5400cbac0 tx=0x7fd534009bd0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.318+0000 7fd546dff700 1 -- 172.21.15.102:0/1872708498 shutdown_connections 2023-03-03T08:15:18.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.318+0000 7fd546dff700 1 --2- 172.21.15.102:0/1872708498 >> v2:172.21.15.172:3300/0 conn(0x7fd5400c5f30 0x7fd5400c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.318+0000 7fd546dff700 1 --2- 172.21.15.102:0/1872708498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd54006d100 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.318+0000 7fd546dff700 1 -- 172.21.15.102:0/1872708498 >> 172.21.15.102:0/1872708498 conn(0x7fd54006af90 msgr2=0x7fd54006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:15:18.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.318+0000 7fd546dff700 1 -- 172.21.15.102:0/1872708498 shutdown_connections 2023-03-03T08:15:18.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.318+0000 7fd546dff700 1 -- 172.21.15.102:0/1872708498 wait complete. 2023-03-03T08:15:18.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.320+0000 7fd546dff700 1 Processor -- start 2023-03-03T08:15:18.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.320+0000 7fd546dff700 1 -- start start 2023-03-03T08:15:18.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd546dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400c2020 0x7fd5401f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:15:18.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd546dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5400c5f30 0x7fd540154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:15:18.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd546dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd540149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:15:18.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd546dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd540156480 con 0x7fd540150c80 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd546dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400cb400 con 0x7fd5400c5f30 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd546dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5401e5f90 con 0x7fd5400c2020 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd544b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400c2020 0x7fd5401f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd54539c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd540149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd53ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5400c5f30 0x7fd540154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd544b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400c2020 0x7fd5401f1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57476/0 (socket says 172.21.15.102:57476) 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.321+0000 7fd544b9b700 1 -- 172.21.15.102:0/3215762479 learned_addr learned my addr 172.21.15.102:0/3215762479 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:15:18.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd544b9b700 1 -- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5400c5f30 msgr2=0x7fd540154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:15:18.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd544b9b700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5400c5f30 0x7fd540154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd544b9b700 1 -- 172.21.15.102:0/3215762479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 msgr2=0x7fd540149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:15:18.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd544b9b700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd540149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd544b9b700 1 -- 172.21.15.102:0/3215762479 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534009550 con 0x7fd5400c2020 2023-03-03T08:15:18.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd53ffff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5400c5f30 0x7fd540154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:15:18.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd54539c700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd540149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:15:18.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd544b9b700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400c2020 0x7fd5401f1790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd534014040 tx=0x7fd534004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:15:18.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd53d7fa700 1 -- 172.21.15.102:0/3215762479 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5340044d0 con 0x7fd5400c2020 2023-03-03T08:15:18.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd53d7fa700 1 -- 172.21.15.102:0/3215762479 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53401e070 con 0x7fd5400c2020 2023-03-03T08:15:18.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.322+0000 7fd53d7fa700 1 -- 172.21.15.102:0/3215762479 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5340038e0 con 0x7fd5400c2020 2023-03-03T08:15:18.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.323+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5401577f0 con 0x7fd5400c2020 2023-03-03T08:15:18.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.323+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd540157510 con 0x7fd5400c2020 2023-03-03T08:15:18.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.324+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5400c4cc0 con 0x7fd5400c2020 2023-03-03T08:15:18.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.325+0000 7fd53d7fa700 1 -- 172.21.15.102:0/3215762479 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd53400f050 con 0x7fd5400c2020 2023-03-03T08:15:18.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.326+0000 7fd53d7fa700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd5240682f0 0x7fd52406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:15:18.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.326+0000 7fd53d7fa700 1 -- 172.21.15.102:0/3215762479 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd534086ca0 con 0x7fd5400c2020 2023-03-03T08:15:18.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.326+0000 7fd53ffff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd5240682f0 0x7fd52406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:15:18.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.327+0000 7fd53ffff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd5240682f0 0x7fd52406a7a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd54014ad40 tx=0x7fd52c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:15:18.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.331+0000 7fd53d7fa700 1 -- 172.21.15.102:0/3215762479 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd5340533b0 con 0x7fd5400c2020 2023-03-03T08:15:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.568+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54004e6e0 con 0x7fd5400c2020 2023-03-03T08:15:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.569+0000 7fd53d7fa700 1 -- 172.21.15.102:0/3215762479 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd534020090 con 0x7fd5400c2020 2023-03-03T08:15:18.570 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:15:18.571 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:15:18.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd5240682f0 msgr2=0x7fd52406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:15:18.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd5240682f0 0x7fd52406a7a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd54014ad40 tx=0x7fd52c009040 comp rx=0 tx=0).stop 2023-03-03T08:15:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400c2020 msgr2=0x7fd5401f1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:15:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400c2020 0x7fd5401f1790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd534014040 tx=0x7fd534004750 comp rx=0 tx=0).stop 2023-03-03T08:15:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 shutdown_connections 2023-03-03T08:15:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd5240682f0 0x7fd52406a7a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd5400c2020 0x7fd5401f1790 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd540150c80 0x7fd540149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 --2- 172.21.15.102:0/3215762479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5400c5f30 0x7fd540154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:15:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 >> 172.21.15.102:0/3215762479 conn(0x7fd54006af90 msgr2=0x7fd54014cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:15:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 shutdown_connections 2023-03-03T08:15:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:15:18.571+0000 7fd546dff700 1 -- 172.21.15.102:0/3215762479 wait complete. 2023-03-03T08:15:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:15:18.600 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-03-03T08:15:23.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:23.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:53.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:53.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:23.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:23.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:24.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:16:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.852+0000 7fc318bff700 1 Processor -- start 2023-03-03T08:16:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.852+0000 7fc318bff700 1 -- start start 2023-03-03T08:16:24.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.853+0000 7fc318bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc3140cd8e0 0x7fc3140cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:16:24.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.853+0000 7fc318bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3140cd390 0x7fc3140ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:16:24.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.853+0000 7fc318bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc314153660 con 0x7fc314153290 2023-03-03T08:16:24.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc318bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc31406c360 con 0x7fc3140cd8e0 2023-03-03T08:16:24.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc318bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31415cea0 con 0x7fc3140cd390 2023-03-03T08:16:24.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc312d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3140cd390 0x7fc3140ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:16:24.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc311d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc3140cd8e0 0x7fc3140cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:16:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc312d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3140cd390 0x7fc3140ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:50400/0 (socket says 172.21.15.102:50400) 2023-03-03T08:16:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc312d9d700 1 -- 172.21.15.102:0/2589696402 learned_addr learned my addr 172.21.15.102:0/2589696402 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:16:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc31259c700 1 --1- 172.21.15.102:0/2589696402 >> v1:172.21.15.172:6789/0 conn(0x7fc314153290 0x7fc314158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:58746/0 (socket says 172.21.15.102:58746) 2023-03-03T08:16:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc31159a700 1 -- 172.21.15.102:0/2589696402 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535875514 0 0) 0x7fc314153660 con 0x7fc314153290 2023-03-03T08:16:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.854+0000 7fc31159a700 1 -- 172.21.15.102:0/2589696402 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2fc0035b0 con 0x7fc314153290 2023-03-03T08:16:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc311d9b700 1 -- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3140cd390 msgr2=0x7fc3140ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:16:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc311d9b700 1 --2- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3140cd390 0x7fc3140ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc311d9b700 1 -- 172.21.15.102:0/2589696402 >> v1:172.21.15.172:6789/0 conn(0x7fc314153290 legacy=0x7fc314158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc311d9b700 1 -- 172.21.15.102:0/2589696402 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31406c7f0 con 0x7fc3140cd8e0 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc312d9d700 1 --2- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3140cd390 0x7fc3140ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc311d9b700 1 --2- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd8e0 0x7fc3140cde20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc308009840 tx=0x7fc308009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d0e8fd0d9d240f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc31159a700 1 -- 172.21.15.102:0/2589696402 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc308011830 con 0x7fc3140cd8e0 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc31159a700 1 -- 172.21.15.102:0/2589696402 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc308011990 con 0x7fc3140cd8e0 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.855+0000 7fc31159a700 1 -- 172.21.15.102:0/2589696402 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc308011d20 con 0x7fc3140cd8e0 2023-03-03T08:16:24.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 -- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd8e0 msgr2=0x7fc3140cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:16:24.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 --2- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd8e0 0x7fc3140cde20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc308009840 tx=0x7fc308009b50 comp rx=0 tx=0).stop 2023-03-03T08:16:24.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 -- 172.21.15.102:0/2589696402 shutdown_connections 2023-03-03T08:16:24.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 --2- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3140cd390 0x7fc3140ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 --2- 172.21.15.102:0/2589696402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd8e0 0x7fc3140cde20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 -- 172.21.15.102:0/2589696402 >> 172.21.15.102:0/2589696402 conn(0x7fc314065fb0 msgr2=0x7fc3140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:16:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 -- 172.21.15.102:0/2589696402 shutdown_connections 2023-03-03T08:16:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.856+0000 7fc318bff700 1 -- 172.21.15.102:0/2589696402 wait complete. 2023-03-03T08:16:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.858+0000 7fc318bff700 1 Processor -- start 2023-03-03T08:16:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.858+0000 7fc318bff700 1 -- start start 2023-03-03T08:16:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc318bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd390 0x7fc3140b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:16:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc318bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3140cd8e0 0x7fc314158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:16:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc318bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc314153290 0x7fc314155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:16:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc318bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140b36d0 con 0x7fc3140cd8e0 2023-03-03T08:16:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc318bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3141550b0 con 0x7fc3140cd390 2023-03-03T08:16:24.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc318bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140b3ee0 con 0x7fc314153290 2023-03-03T08:16:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc31259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd390 0x7fc3140b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:16:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.859+0000 7fc312d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc314153290 0x7fc314155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:16:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc311d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3140cd8e0 0x7fc314158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:16:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc314153290 0x7fc314155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:50412/0 (socket says 172.21.15.102:50412) 2023-03-03T08:16:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc31259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd390 0x7fc3140b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50566/0 (socket says 172.21.15.102:50566) 2023-03-03T08:16:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 -- 172.21.15.102:0/4195565270 learned_addr learned my addr 172.21.15.102:0/4195565270 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:16:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 -- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd390 msgr2=0x7fc3140b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd390 0x7fc3140b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 -- 172.21.15.102:0/4195565270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3140cd8e0 msgr2=0x7fc314158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3140cd8e0 0x7fc314158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 -- 172.21.15.102:0/4195565270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c009b30 con 0x7fc314153290 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc312d9d700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc314153290 0x7fc314155540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009e70 tx=0x7fc30c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.860+0000 7fc302ffd700 1 -- 172.21.15.102:0/4195565270 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c018070 con 0x7fc314153290 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.861+0000 7fc302ffd700 1 -- 172.21.15.102:0/4195565270 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c00f040 con 0x7fc314153290 2023-03-03T08:16:24.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.861+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3080094a0 con 0x7fc314153290 2023-03-03T08:16:24.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.861+0000 7fc302ffd700 1 -- 172.21.15.102:0/4195565270 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c003c40 con 0x7fc314153290 2023-03-03T08:16:24.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.861+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3140ccde0 con 0x7fc314153290 2023-03-03T08:16:24.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.862+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31415d160 con 0x7fc314153290 2023-03-03T08:16:24.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.863+0000 7fc302ffd700 1 -- 172.21.15.102:0/4195565270 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc30c00c880 con 0x7fc314153290 2023-03-03T08:16:24.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.864+0000 7fc302ffd700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2f40682f0 0x7fc2f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:16:24.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.864+0000 7fc302ffd700 1 -- 172.21.15.102:0/4195565270 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc30c085fe0 con 0x7fc314153290 2023-03-03T08:16:24.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.865+0000 7fc31259c700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2f40682f0 0x7fc2f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:16:24.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.865+0000 7fc31259c700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2f40682f0 0x7fc2f406a7a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc304007fd0 tx=0x7fc304007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:16:24.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:24.868+0000 7fc302ffd700 1 -- 172.21.15.102:0/4195565270 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc30c052710 con 0x7fc314153290 2023-03-03T08:16:25.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.104+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3140cd0c0 con 0x7fc314153290 2023-03-03T08:16:25.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.105+0000 7fc302ffd700 1 -- 172.21.15.102:0/4195565270 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc30c055d30 con 0x7fc314153290 2023-03-03T08:16:25.107 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:16:25.108 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:16:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2f40682f0 msgr2=0x7fc2f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:16:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2f40682f0 0x7fc2f406a7a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc304007fd0 tx=0x7fc304007dc0 comp rx=0 tx=0).stop 2023-03-03T08:16:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc314153290 msgr2=0x7fc314155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:16:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc314153290 0x7fc314155540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009e70 tx=0x7fc30c00c3a0 comp rx=0 tx=0).stop 2023-03-03T08:16:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 shutdown_connections 2023-03-03T08:16:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2f40682f0 0x7fc2f406a7a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc314153290 0x7fc314155540 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3140cd390 0x7fc3140b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 --2- 172.21.15.102:0/4195565270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3140cd8e0 0x7fc314158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:16:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 >> 172.21.15.102:0/4195565270 conn(0x7fc314065fb0 msgr2=0x7fc314073810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:16:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 shutdown_connections 2023-03-03T08:16:25.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:16:25.108+0000 7fc318bff700 1 -- 172.21.15.102:0/4195565270 wait complete. 2023-03-03T08:16:25.112 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:16:25.137 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2023-03-03T08:16:53.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:53.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:23.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:23.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:54.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:54.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:59.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:17:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.387+0000 7fb3c339d700 1 Processor -- start 2023-03-03T08:17:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.387+0000 7fb3c339d700 1 -- start start 2023-03-03T08:17:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c339d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc153030 0x7fb3bc0c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:17:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c339d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3bc0b1e90 0x7fb3bc15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:17:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c339d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3bc0b3f30 0x7fb3bc0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:17:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c339d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc06c780 con 0x7fb3bc0b1e90 2023-03-03T08:17:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c339d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc15c620 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c339d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc15c050 con 0x7fb3bc153030 2023-03-03T08:17:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c239b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc153030 0x7fb3bc0c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:17:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c1b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3bc0b1e90 0x7fb3bc15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:17:59.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c2b9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3bc0b3f30 0x7fb3bc0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:17:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c239b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc153030 0x7fb3bc0c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48266/0 (socket says 172.21.15.102:48266) 2023-03-03T08:17:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.389+0000 7fb3c239b700 1 -- 172.21.15.102:0/1840190327 learned_addr learned my addr 172.21.15.102:0/1840190327 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:17:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c2b9c700 1 -- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc153030 msgr2=0x7fb3bc0c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:17:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c2b9c700 1 --2- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc153030 0x7fb3bc0c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c2b9c700 1 -- 172.21.15.102:0/1840190327 >> v2:172.21.15.102:3300/0 conn(0x7fb3bc0b1e90 msgr2=0x7fb3bc15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:17:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c2b9c700 1 --2- 172.21.15.102:0/1840190327 >> v2:172.21.15.102:3300/0 conn(0x7fb3bc0b1e90 0x7fb3bc15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c2b9c700 1 -- 172.21.15.102:0/1840190327 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc06bee0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c239b700 1 --2- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc153030 0x7fb3bc0c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c1b9a700 1 --2- 172.21.15.102:0/1840190327 >> v2:172.21.15.102:3300/0 conn(0x7fb3bc0b1e90 0x7fb3bc15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c2b9c700 1 --2- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc0cece0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc06cd30 tx=0x7fb3b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f8362b6779ad5a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c1399700 1 -- 172.21.15.102:0/1840190327 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400e070 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c1399700 1 -- 172.21.15.102:0/1840190327 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b400bae0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.390+0000 7fb3c1399700 1 -- 172.21.15.102:0/1840190327 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400be90 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 -- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 msgr2=0x7fb3bc0cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:17:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 --2- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc0cece0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc06cd30 tx=0x7fb3b400b330 comp rx=0 tx=0).stop 2023-03-03T08:17:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 -- 172.21.15.102:0/1840190327 shutdown_connections 2023-03-03T08:17:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 --2- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc0cece0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 --2- 172.21.15.102:0/1840190327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc153030 0x7fb3bc0c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 --2- 172.21.15.102:0/1840190327 >> v2:172.21.15.102:3300/0 conn(0x7fb3bc0b1e90 0x7fb3bc15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 -- 172.21.15.102:0/1840190327 >> 172.21.15.102:0/1840190327 conn(0x7fb3bc08b1c0 msgr2=0x7fb3bc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:17:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 -- 172.21.15.102:0/1840190327 shutdown_connections 2023-03-03T08:17:59.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.391+0000 7fb3c339d700 1 -- 172.21.15.102:0/1840190327 wait complete. 2023-03-03T08:17:59.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.393+0000 7fb3c339d700 1 Processor -- start 2023-03-03T08:17:59.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.394+0000 7fb3c339d700 1 -- start start 2023-03-03T08:17:59.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.394+0000 7fb3c339d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc0b1e90 0x7fb3bc1499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:17:59.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c339d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:17:59.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c339d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3bc153030 0x7fb3bc0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:17:59.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c339d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc1ee0b0 con 0x7fb3bc153030 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c339d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc14eff0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c339d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc1f17c0 con 0x7fb3bc0b1e90 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c2b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3bc153030 0x7fb3bc0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c239b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc0b1e90 0x7fb3bc1499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c1b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c239b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc0b1e90 0x7fb3bc1499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48276/0 (socket says 172.21.15.102:48276) 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c239b700 1 -- 172.21.15.102:0/2941290216 learned_addr learned my addr 172.21.15.102:0/2941290216 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:17:59.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c1b9a700 1 -- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc0b1e90 msgr2=0x7fb3bc1499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:17:59.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c1b9a700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc0b1e90 0x7fb3bc1499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c1b9a700 1 -- 172.21.15.102:0/2941290216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3bc153030 msgr2=0x7fb3bc0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:17:59.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c1b9a700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3bc153030 0x7fb3bc0cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c1b9a700 1 -- 172.21.15.102:0/2941290216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac0094a0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.395+0000 7fb3c2b9c700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3bc153030 0x7fb3bc0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:17:59.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.396+0000 7fb3c1b9a700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc153480 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e980 tx=0x7fb3ac00ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:17:59.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.396+0000 7fb3b2ffd700 1 -- 172.21.15.102:0/2941290216 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac004030 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.396+0000 7fb3b2ffd700 1 -- 172.21.15.102:0/2941290216 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac0041b0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.396+0000 7fb3b2ffd700 1 -- 172.21.15.102:0/2941290216 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac004560 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.396+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b8009550 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.396+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b4009a70 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.397+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc06c4a0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.399+0000 7fb3b2ffd700 1 -- 172.21.15.102:0/2941290216 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb3ac019070 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.400+0000 7fb3b2ffd700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3a00682f0 0x7fb3a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:17:59.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.400+0000 7fb3b2ffd700 1 -- 172.21.15.102:0/2941290216 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb3ac085a30 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.400+0000 7fb3c239b700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3a00682f0 0x7fb3a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:17:59.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.401+0000 7fb3c239b700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3a00682f0 0x7fb3a006a7a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009d80 tx=0x7fb3b80092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:17:59.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.403+0000 7fb3b2ffd700 1 -- 172.21.15.102:0/2941290216 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb3ac0521c0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.641+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3bc15c240 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.641+0000 7fb3b2ffd700 1 -- 172.21.15.102:0/2941290216 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb3ac0557e0 con 0x7fb3bc0b3f30 2023-03-03T08:17:59.644 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:17:59.644 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:17:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.644+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3a00682f0 msgr2=0x7fb3a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:17:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.644+0000 7fb3c339d700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3a00682f0 0x7fb3a006a7a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009d80 tx=0x7fb3b80092c0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.644+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 msgr2=0x7fb3bc153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:17:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.644+0000 7fb3c339d700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc153480 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e980 tx=0x7fb3ac00ec90 comp rx=0 tx=0).stop 2023-03-03T08:17:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 shutdown_connections 2023-03-03T08:17:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3a00682f0 0x7fb3a006a7a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3bc0b1e90 0x7fb3bc1499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3bc153030 0x7fb3bc0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 --2- 172.21.15.102:0/2941290216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3bc0b3f30 0x7fb3bc153480 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:17:59.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 >> 172.21.15.102:0/2941290216 conn(0x7fb3bc08b1c0 msgr2=0x7fb3bc1509e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:17:59.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 shutdown_connections 2023-03-03T08:17:59.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:17:59.645+0000 7fb3c339d700 1 -- 172.21.15.102:0/2941290216 wait complete. 2023-03-03T08:17:59.649 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:17:59.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 82 secs before thrashing 2023-03-03T08:18:24.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:24.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:54.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:54.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:21.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:19:21.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.925+0000 7fadf33ff700 1 Processor -- start 2023-03-03T08:19:21.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.925+0000 7fadf33ff700 1 -- start start 2023-03-03T08:19:21.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.926+0000 7fadf33ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fadec0cd8e0 0x7fadec0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:21.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.927+0000 7fadf33ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec153660 con 0x7fadec153290 2023-03-03T08:19:21.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.927+0000 7fadf33ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec06c360 con 0x7fadec0cd390 2023-03-03T08:19:21.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.927+0000 7fadf33ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadec15cea0 con 0x7fadec0cd8e0 2023-03-03T08:19:21.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.927+0000 7fadf099a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fadec0cd8e0 0x7fadec0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:21.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.927+0000 7fadf099a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fadec0cd8e0 0x7fadec0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59442/0 (socket says 172.21.15.102:59442) 2023-03-03T08:19:21.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.927+0000 7fadf099a700 1 -- 172.21.15.102:0/585366681 learned_addr learned my addr 172.21.15.102:0/585366681 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:19:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.927+0000 7fade3fff700 1 -- 172.21.15.102:0/585366681 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368824174 0 0) 0x7fadec06c360 con 0x7fadec0cd390 2023-03-03T08:19:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fade3fff700 1 -- 172.21.15.102:0/585366681 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd80035b0 con 0x7fadec0cd390 2023-03-03T08:19:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fadf099a700 1 -- 172.21.15.102:0/585366681 >> v1:172.21.15.172:6789/0 conn(0x7fadec0cd390 legacy=0x7fadec0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fadf099a700 1 -- 172.21.15.102:0/585366681 >> v1:172.21.15.102:6789/0 conn(0x7fadec153290 legacy=0x7fadec158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fadf099a700 1 -- 172.21.15.102:0/585366681 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec06c7f0 con 0x7fadec0cd8e0 2023-03-03T08:19:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fadf099a700 1 --2- 172.21.15.102:0/585366681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0cde20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faddc0098f0 tx=0x7faddc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0c445478855819 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:19:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fade3fff700 1 -- 172.21.15.102:0/585366681 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc011830 con 0x7fadec0cd8e0 2023-03-03T08:19:21.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fade3fff700 1 -- 172.21.15.102:0/585366681 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc011990 con 0x7fadec0cd8e0 2023-03-03T08:19:21.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.928+0000 7fade3fff700 1 -- 172.21.15.102:0/585366681 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc011d20 con 0x7fadec0cd8e0 2023-03-03T08:19:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.929+0000 7fadf33ff700 1 -- 172.21.15.102:0/585366681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 msgr2=0x7fadec0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.929+0000 7fadf33ff700 1 --2- 172.21.15.102:0/585366681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0cde20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faddc0098f0 tx=0x7faddc009c00 comp rx=0 tx=0).stop 2023-03-03T08:19:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.929+0000 7fadf33ff700 1 -- 172.21.15.102:0/585366681 shutdown_connections 2023-03-03T08:19:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.929+0000 7fadf33ff700 1 --2- 172.21.15.102:0/585366681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0cde20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.929+0000 7fadf33ff700 1 -- 172.21.15.102:0/585366681 >> 172.21.15.102:0/585366681 conn(0x7fadec065fb0 msgr2=0x7fadec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:19:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.929+0000 7fadf33ff700 1 -- 172.21.15.102:0/585366681 shutdown_connections 2023-03-03T08:19:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.929+0000 7fadf33ff700 1 -- 172.21.15.102:0/585366681 wait complete. 2023-03-03T08:19:21.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.931+0000 7fadf33ff700 1 Processor -- start 2023-03-03T08:19:21.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.931+0000 7fadf33ff700 1 -- start start 2023-03-03T08:19:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.932+0000 7fadf33ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadec0cd390 0x7fadec1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.932+0000 7fadf33ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.932+0000 7fadf33ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadec153290 0x7fadec0b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.932+0000 7fadf33ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec1ef1a0 con 0x7fadec0cd8e0 2023-03-03T08:19:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.932+0000 7fadf33ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0b2900 con 0x7fadec153290 2023-03-03T08:19:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.932+0000 7fadf33ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec1ef690 con 0x7fadec0cd390 2023-03-03T08:19:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.932+0000 7fadf099a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:21.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59450/0 (socket says 172.21.15.102:59450) 2023-03-03T08:19:21.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 -- 172.21.15.102:0/2537956636 learned_addr learned my addr 172.21.15.102:0/2537956636 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:19:21.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf199c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadec153290 0x7fadec0b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:21.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf119b700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadec0cd390 0x7fadec1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:21.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 -- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadec0cd390 msgr2=0x7fadec1516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:21.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadec0cd390 0x7fadec1516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:21.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 -- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadec153290 msgr2=0x7fadec0b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:21.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadec153290 0x7fadec0b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 -- 172.21.15.102:0/2537956636 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc009550 con 0x7fadec0cd8e0 2023-03-03T08:19:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.933+0000 7fadf099a700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0b3320 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faddc009c40 tx=0x7faddc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:19:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.934+0000 7fade1ffb700 1 -- 172.21.15.102:0/2537956636 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc004550 con 0x7fadec0cd8e0 2023-03-03T08:19:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.934+0000 7fade1ffb700 1 -- 172.21.15.102:0/2537956636 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc01f070 con 0x7fadec0cd8e0 2023-03-03T08:19:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.934+0000 7fade1ffb700 1 -- 172.21.15.102:0/2537956636 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc0038e0 con 0x7fadec0cd8e0 2023-03-03T08:19:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.934+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec0ccde0 con 0x7fadec0cd8e0 2023-03-03T08:19:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.934+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec15d160 con 0x7fadec0cd8e0 2023-03-03T08:19:21.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.935+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadec0cd0c0 con 0x7fadec0cd8e0 2023-03-03T08:19:21.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.936+0000 7fade1ffb700 1 -- 172.21.15.102:0/2537956636 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faddc00f050 con 0x7fadec0cd8e0 2023-03-03T08:19:21.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.937+0000 7fade1ffb700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadd0068320 0x7fadd006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:21.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.937+0000 7fade1ffb700 1 -- 172.21.15.102:0/2537956636 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7faddc021070 con 0x7fadec0cd8e0 2023-03-03T08:19:21.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.938+0000 7fadf119b700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadd0068320 0x7fadd006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:21.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.938+0000 7fadf119b700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadd0068320 0x7fadd006a7d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fade8007fd0 tx=0x7fade8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:19:21.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:21.942+0000 7fade1ffb700 1 -- 172.21.15.102:0/2537956636 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faddc053d60 con 0x7fadec0cd8e0 2023-03-03T08:19:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.172+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadec0cc6c0 con 0x7fadec0cd8e0 2023-03-03T08:19:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.172+0000 7fade1ffb700 1 -- 172.21.15.102:0/2537956636 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7faddc02b090 con 0x7fadec0cd8e0 2023-03-03T08:19:22.174 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:19:22.174 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:19:22.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadd0068320 msgr2=0x7fadd006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:22.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadd0068320 0x7fadd006a7d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fade8007fd0 tx=0x7fade8009040 comp rx=0 tx=0).stop 2023-03-03T08:19:22.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 msgr2=0x7fadec0b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0b3320 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faddc009c40 tx=0x7faddc004750 comp rx=0 tx=0).stop 2023-03-03T08:19:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 shutdown_connections 2023-03-03T08:19:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fadd0068320 0x7fadd006a7d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fadec0cd390 0x7fadec1516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadec153290 0x7fadec0b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 --2- 172.21.15.102:0/2537956636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadec0cd8e0 0x7fadec0b3320 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 >> 172.21.15.102:0/2537956636 conn(0x7fadec065fb0 msgr2=0x7fadec0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:19:22.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 shutdown_connections 2023-03-03T08:19:22.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:22.175+0000 7fadf33ff700 1 -- 172.21.15.102:0/2537956636 wait complete. 2023-03-03T08:19:22.178 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:19:22.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T08:19:24.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:24.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:50.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:19:50.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.453+0000 7efc069ff700 1 Processor -- start 2023-03-03T08:19:50.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.454+0000 7efc069ff700 1 -- start start 2023-03-03T08:19:50.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.455+0000 7efc069ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:50.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.455+0000 7efc069ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc00153660 con 0x7efc00153290 2023-03-03T08:19:50.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.455+0000 7efc069ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0006c360 con 0x7efc000cd8e0 2023-03-03T08:19:50.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.455+0000 7efc069ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0015cea0 con 0x7efc000cd390 2023-03-03T08:19:50.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.455+0000 7efbfffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc00153290 0x7efc00158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44246/0 (socket says 172.21.15.102:44246) 2023-03-03T08:19:50.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.455+0000 7efbfffff700 1 -- 172.21.15.102:0/1627999282 learned_addr learned my addr 172.21.15.102:0/1627999282 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:19:50.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.455+0000 7efc04f9c700 1 --2- 172.21.15.102:0/1627999282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:50.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efbfeffd700 1 -- 172.21.15.102:0/1627999282 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749736905 0 0) 0x7efc00153660 con 0x7efc00153290 2023-03-03T08:19:50.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efbfeffd700 1 -- 172.21.15.102:0/1627999282 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe80035b0 con 0x7efc00153290 2023-03-03T08:19:50.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efbfeffd700 1 -- 172.21.15.102:0/1627999282 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443245391 0 0) 0x7efc0006c360 con 0x7efc000cd8e0 2023-03-03T08:19:50.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efbfeffd700 1 -- 172.21.15.102:0/1627999282 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc00153660 con 0x7efc000cd8e0 2023-03-03T08:19:50.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efc04f9c700 1 -- 172.21.15.102:0/1627999282 >> v1:172.21.15.172:6789/0 conn(0x7efc000cd8e0 legacy=0x7efc000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:50.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efc04f9c700 1 -- 172.21.15.102:0/1627999282 >> v1:172.21.15.102:6789/0 conn(0x7efc00153290 legacy=0x7efc00158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:50.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efc04f9c700 1 -- 172.21.15.102:0/1627999282 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0006c7f0 con 0x7efc000cd390 2023-03-03T08:19:50.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efc04f9c700 1 --2- 172.21.15.102:0/1627999282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000ce530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009dc0 tx=0x7efbf800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afa6bc9818b70d4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:19:50.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.456+0000 7efbfeffd700 1 -- 172.21.15.102:0/1627999282 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800e070 con 0x7efc000cd390 2023-03-03T08:19:50.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efbfeffd700 1 -- 172.21.15.102:0/1627999282 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf800bb20 con 0x7efc000cd390 2023-03-03T08:19:50.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efbfeffd700 1 -- 172.21.15.102:0/1627999282 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8012430 con 0x7efc000cd390 2023-03-03T08:19:50.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efc069ff700 1 -- 172.21.15.102:0/1627999282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 msgr2=0x7efc000ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:50.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efc069ff700 1 --2- 172.21.15.102:0/1627999282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000ce530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009dc0 tx=0x7efbf800b330 comp rx=0 tx=0).stop 2023-03-03T08:19:50.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efc069ff700 1 -- 172.21.15.102:0/1627999282 shutdown_connections 2023-03-03T08:19:50.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efc069ff700 1 --2- 172.21.15.102:0/1627999282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd390 0x7efc000ce530 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efc069ff700 1 -- 172.21.15.102:0/1627999282 >> 172.21.15.102:0/1627999282 conn(0x7efc00065fb0 msgr2=0x7efc000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:19:50.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efc069ff700 1 -- 172.21.15.102:0/1627999282 shutdown_connections 2023-03-03T08:19:50.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.457+0000 7efc069ff700 1 -- 172.21.15.102:0/1627999282 wait complete. 2023-03-03T08:19:50.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.460+0000 7efc069ff700 1 Processor -- start 2023-03-03T08:19:50.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.460+0000 7efc069ff700 1 -- start start 2023-03-03T08:19:50.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc069ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:50.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc069ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc0014e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:50.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc069ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc00153290 0x7efc000c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:50.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc069ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00154360 con 0x7efc00153290 2023-03-03T08:19:50.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc069ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0014ea20 con 0x7efc000cd390 2023-03-03T08:19:50.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc069ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc000c91c0 con 0x7efc000cd8e0 2023-03-03T08:19:50.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efbff7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc0014e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:50.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efbfffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:50.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc04f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc00153290 0x7efc000c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:50.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efbfffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000c7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33790/0 (socket says 172.21.15.102:33790) 2023-03-03T08:19:50.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efc04f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc00153290 0x7efc000c72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55332/0 (socket says 172.21.15.102:55332) 2023-03-03T08:19:50.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efbff7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc0014e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:35922/0 (socket says 172.21.15.102:35922) 2023-03-03T08:19:50.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.462+0000 7efbfffff700 1 -- 172.21.15.102:0/3216247858 learned_addr learned my addr 172.21.15.102:0/3216247858 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbff7fe700 1 -- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 msgr2=0x7efc000c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbff7fe700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbff7fe700 1 -- 172.21.15.102:0/3216247858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc00153290 msgr2=0x7efc000c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbff7fe700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc00153290 0x7efc000c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbff7fe700 1 -- 172.21.15.102:0/3216247858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8009a70 con 0x7efc000cd8e0 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbff7fe700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc0014e4e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efbf4001050 tx=0x7efbf400c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbfcff9700 1 -- 172.21.15.102:0/3216247858 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf40050d0 con 0x7efc000cd8e0 2023-03-03T08:19:50.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbfcff9700 1 -- 172.21.15.102:0/3216247858 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf4005250 con 0x7efc000cd8e0 2023-03-03T08:19:50.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc000ccde0 con 0x7efc000cd8e0 2023-03-03T08:19:50.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efbfcff9700 1 -- 172.21.15.102:0/3216247858 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4003680 con 0x7efc000cd8e0 2023-03-03T08:19:50.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.463+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0015d160 con 0x7efc000cd8e0 2023-03-03T08:19:50.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.464+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc000cd0c0 con 0x7efc000cd8e0 2023-03-03T08:19:50.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.466+0000 7efbfcff9700 1 -- 172.21.15.102:0/3216247858 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efbf4019030 con 0x7efc000cd8e0 2023-03-03T08:19:50.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.467+0000 7efbfcff9700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbec0682f0 0x7efbec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:19:50.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.467+0000 7efbfcff9700 1 -- 172.21.15.102:0/3216247858 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7efbf4085740 con 0x7efc000cd8e0 2023-03-03T08:19:50.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.467+0000 7efbfffff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbec0682f0 0x7efbec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:19:50.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.468+0000 7efbfffff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbec0682f0 0x7efbec06a7a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7efbf0000f60 tx=0x7efbf0007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:19:50.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.471+0000 7efbfcff9700 1 -- 172.21.15.102:0/3216247858 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efbf4051e40 con 0x7efc000cd8e0 2023-03-03T08:19:50.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.707+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc0004e6e0 con 0x7efc000cd8e0 2023-03-03T08:19:50.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.712+0000 7efbfcff9700 1 -- 172.21.15.102:0/3216247858 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7efbf4017090 con 0x7efc000cd8e0 2023-03-03T08:19:50.713 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:19:50.713 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:19:50.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbec0682f0 msgr2=0x7efbec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:50.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbec0682f0 0x7efbec06a7a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7efbf0000f60 tx=0x7efbf0007ea0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 msgr2=0x7efc0014e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:19:50.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc0014e4e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efbf4001050 tx=0x7efbf400c730 comp rx=0 tx=0).stop 2023-03-03T08:19:50.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 shutdown_connections 2023-03-03T08:19:50.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc00153290 0x7efc000c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc000cd390 0x7efc000c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efbec0682f0 0x7efbec06a7a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 --2- 172.21.15.102:0/3216247858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc000cd8e0 0x7efc0014e4e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:19:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 >> 172.21.15.102:0/3216247858 conn(0x7efc00065fb0 msgr2=0x7efc001585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:19:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 shutdown_connections 2023-03-03T08:19:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:19:50.714+0000 7efc069ff700 1 -- 172.21.15.102:0/3216247858 wait complete. 2023-03-03T08:19:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:19:50.744 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 16 secs before thrashing 2023-03-03T08:19:54.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:54.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T08:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:20:06.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:20:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.016+0000 7f3a815ff700 1 Processor -- start 2023-03-03T08:20:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.016+0000 7f3a815ff700 1 -- start start 2023-03-03T08:20:07.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.017+0000 7f3a815ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c150c80 0x7f3a7c06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.017+0000 7f3a815ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c0c5390 con 0x7f3a7c0c2020 2023-03-03T08:20:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.017+0000 7f3a815ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c0c4a20 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a815ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c157ad0 con 0x7f3a7c150c80 2023-03-03T08:20:07.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a73fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3a7c0c2020 0x7f3a7c0c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39008/0 (socket says 172.21.15.102:39008) 2023-03-03T08:20:07.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a73fff700 1 -- 172.21.15.102:0/4268594741 learned_addr learned my addr 172.21.15.102:0/4268594741 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:20:07.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7affd700 1 --2- 172.21.15.102:0/4268594741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c150c80 0x7f3a7c06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:07.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7a7fc700 1 -- 172.21.15.102:0/4268594741 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925071315 0 0) 0x7f3a7c0c4a20 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7a7fc700 1 -- 172.21.15.102:0/4268594741 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a680035b0 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7a7fc700 1 -- 172.21.15.102:0/4268594741 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720601946 0 0) 0x7f3a7c0c5390 con 0x7f3a7c0c2020 2023-03-03T08:20:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7a7fc700 1 -- 172.21.15.102:0/4268594741 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a7c0c4a20 con 0x7f3a7c0c2020 2023-03-03T08:20:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7affd700 1 -- 172.21.15.102:0/4268594741 >> v1:172.21.15.172:6789/0 conn(0x7f3a7c0c5f30 legacy=0x7f3a7c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7affd700 1 -- 172.21.15.102:0/4268594741 >> v1:172.21.15.102:6789/0 conn(0x7f3a7c0c2020 legacy=0x7f3a7c0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.018+0000 7f3a7affd700 1 -- 172.21.15.102:0/4268594741 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c06f010 con 0x7f3a7c150c80 2023-03-03T08:20:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.019+0000 7f3a7affd700 1 --2- 172.21.15.102:0/4268594741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c150c80 0x7f3a7c06d100 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009870 tx=0x7f3a6c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a54d896b75be7253 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:20:07.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.019+0000 7f3a7a7fc700 1 -- 172.21.15.102:0/4268594741 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c011830 con 0x7f3a7c150c80 2023-03-03T08:20:07.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.019+0000 7f3a7a7fc700 1 -- 172.21.15.102:0/4268594741 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6c011990 con 0x7f3a7c150c80 2023-03-03T08:20:07.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.019+0000 7f3a7a7fc700 1 -- 172.21.15.102:0/4268594741 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c011d20 con 0x7f3a7c150c80 2023-03-03T08:20:07.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.020+0000 7f3a815ff700 1 -- 172.21.15.102:0/4268594741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c150c80 msgr2=0x7f3a7c06d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:07.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.020+0000 7f3a815ff700 1 --2- 172.21.15.102:0/4268594741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c150c80 0x7f3a7c06d100 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009870 tx=0x7f3a6c009b80 comp rx=0 tx=0).stop 2023-03-03T08:20:07.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.020+0000 7f3a815ff700 1 -- 172.21.15.102:0/4268594741 shutdown_connections 2023-03-03T08:20:07.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.020+0000 7f3a815ff700 1 --2- 172.21.15.102:0/4268594741 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c150c80 0x7f3a7c06d100 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.020+0000 7f3a815ff700 1 -- 172.21.15.102:0/4268594741 >> 172.21.15.102:0/4268594741 conn(0x7f3a7c06af90 msgr2=0x7f3a7c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:20:07.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.020+0000 7f3a815ff700 1 -- 172.21.15.102:0/4268594741 shutdown_connections 2023-03-03T08:20:07.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.020+0000 7f3a815ff700 1 -- 172.21.15.102:0/4268594741 wait complete. 2023-03-03T08:20:07.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.022+0000 7f3a815ff700 1 Processor -- start 2023-03-03T08:20:07.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.022+0000 7f3a815ff700 1 -- start start 2023-03-03T08:20:07.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.023+0000 7f3a815ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a7c0c2020 0x7f3a7c0c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:07.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.023+0000 7f3a815ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c0c5f30 0x7f3a7c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:07.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.023+0000 7f3a815ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a7c150c80 0x7f3a7c152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:07.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.023+0000 7f3a815ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1521f0 con 0x7f3a7c0c2020 2023-03-03T08:20:07.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.023+0000 7f3a815ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c14aed0 con 0x7f3a7c150c80 2023-03-03T08:20:07.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.023+0000 7f3a815ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c14a350 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.023+0000 7f3a7affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a7c0c2020 0x7f3a7c0c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c0c5f30 0x7f3a7c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a7b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a7c150c80 0x7f3a7c152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a7affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a7c0c2020 0x7f3a7c0c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43848/0 (socket says 172.21.15.102:43848) 2023-03-03T08:20:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c0c5f30 0x7f3a7c14fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48690/0 (socket says 172.21.15.102:48690) 2023-03-03T08:20:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a7affd700 1 -- 172.21.15.102:0/3325350161 learned_addr learned my addr 172.21.15.102:0/3325350161 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:20:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 -- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a7c150c80 msgr2=0x7f3a7c152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a7c150c80 0x7f3a7c152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 -- 172.21.15.102:0/3325350161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a7c0c2020 msgr2=0x7f3a7c0c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a7c0c2020 0x7f3a7c0c7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 -- 172.21.15.102:0/3325350161 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0094a0 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a7affd700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a7c0c2020 0x7f3a7c0c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.024+0000 7f3a73fff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c0c5f30 0x7f3a7c14fd60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400c490 tx=0x7f3a6400c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.025+0000 7f3a737fe700 1 -- 172.21.15.102:0/3325350161 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64005d70 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.025+0000 7f3a737fe700 1 -- 172.21.15.102:0/3325350161 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a640070d0 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.025+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c1577f0 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.025+0000 7f3a737fe700 1 -- 172.21.15.102:0/3325350161 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64007440 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.025+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c157510 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.026+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c0c4cc0 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.027+0000 7f3a737fe700 1 -- 172.21.15.102:0/3325350161 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3a64010070 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.028+0000 7f3a737fe700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3a5c0682f0 0x7f3a5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.028+0000 7f3a737fe700 1 -- 172.21.15.102:0/3325350161 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3a64084bc0 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.029+0000 7f3a7affd700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3a5c0682f0 0x7f3a5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.030+0000 7f3a7affd700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3a5c0682f0 0x7f3a5c06a7a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c000c00 tx=0x7f3a6c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:20:07.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.032+0000 7f3a737fe700 1 -- 172.21.15.102:0/3325350161 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3a6404a5e0 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.267+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a7c0c5470 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.268+0000 7f3a737fe700 1 -- 172.21.15.102:0/3325350161 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3a64051380 con 0x7f3a7c0c5f30 2023-03-03T08:20:07.269 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:20:07.269 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:20:07.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3a5c0682f0 msgr2=0x7f3a5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:07.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3a5c0682f0 0x7f3a5c06a7a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c000c00 tx=0x7f3a6c01c040 comp rx=0 tx=0).stop 2023-03-03T08:20:07.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c0c5f30 msgr2=0x7f3a7c14fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:07.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c0c5f30 0x7f3a7c14fd60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400c490 tx=0x7f3a6400c7a0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 shutdown_connections 2023-03-03T08:20:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a7c150c80 0x7f3a7c152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a7c0c2020 0x7f3a7c0c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3a5c0682f0 0x7f3a5c06a7a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 --2- 172.21.15.102:0/3325350161 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a7c0c5f30 0x7f3a7c14fd60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 >> 172.21.15.102:0/3325350161 conn(0x7f3a7c06af90 msgr2=0x7f3a7c0c9150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:20:07.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 shutdown_connections 2023-03-03T08:20:07.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:07.271+0000 7f3a815ff700 1 -- 172.21.15.102:0/3325350161 wait complete. 2023-03-03T08:20:07.274 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:20:07.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T08:20:25.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:25.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:45.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:20:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.563+0000 7fcfe159d700 1 Processor -- start 2023-03-03T08:20:45.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.563+0000 7fcfe159d700 1 -- start start 2023-03-03T08:20:45.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfe159d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfdc0b3970 0x7fcfdc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfe159d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0c41b0 0x7fcfdc0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfe159d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfdc0c3d20 con 0x7fcfdc0ced00 2023-03-03T08:20:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfe159d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfdc0c4930 con 0x7fcfdc0b3970 2023-03-03T08:20:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfe159d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc0cf610 con 0x7fcfdc0c41b0 2023-03-03T08:20:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfdbfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcfdc0ced00 0x7fcfdc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35434/0 (socket says 172.21.15.102:35434) 2023-03-03T08:20:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfdb7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfdc0b3970 0x7fcfdc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfdbfff700 1 -- 172.21.15.102:0/1228125607 learned_addr learned my addr 172.21.15.102:0/1228125607 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:20:45.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.565+0000 7fcfe0d9c700 1 --2- 172.21.15.102:0/1228125607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0c41b0 0x7fcfdc0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfdb7fe700 1 -- 172.21.15.102:0/1228125607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0c41b0 msgr2=0x7fcfdc0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:20:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfdb7fe700 1 --2- 172.21.15.102:0/1228125607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0c41b0 0x7fcfdc0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfdb7fe700 1 -- 172.21.15.102:0/1228125607 >> v1:172.21.15.172:6789/0 conn(0x7fcfdc0ced00 legacy=0x7fcfdc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:45.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfdaffd700 1 -- 172.21.15.102:0/1228125607 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383885560 0 0) 0x7fcfdc0c3d20 con 0x7fcfdc0ced00 2023-03-03T08:20:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfdb7fe700 1 -- 172.21.15.102:0/1228125607 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfdc15cba0 con 0x7fcfdc0b3970 2023-03-03T08:20:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfe0d9c700 1 --2- 172.21.15.102:0/1228125607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0c41b0 0x7fcfdc0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:20:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfdb7fe700 1 --2- 172.21.15.102:0/1228125607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc0c5a90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009870 tx=0x7fcfd0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfca30c3e78a09a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:20:45.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.566+0000 7fcfdaffd700 1 -- 172.21.15.102:0/1228125607 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0011830 con 0x7fcfdc0b3970 2023-03-03T08:20:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfdaffd700 1 -- 172.21.15.102:0/1228125607 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd0011990 con 0x7fcfdc0b3970 2023-03-03T08:20:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfdaffd700 1 -- 172.21.15.102:0/1228125607 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0011d20 con 0x7fcfdc0b3970 2023-03-03T08:20:45.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 -- 172.21.15.102:0/1228125607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 msgr2=0x7fcfdc0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 --2- 172.21.15.102:0/1228125607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc0c5a90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009870 tx=0x7fcfd0009b80 comp rx=0 tx=0).stop 2023-03-03T08:20:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 -- 172.21.15.102:0/1228125607 shutdown_connections 2023-03-03T08:20:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 --2- 172.21.15.102:0/1228125607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0c41b0 0x7fcfdc0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 --2- 172.21.15.102:0/1228125607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc0c5a90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 -- 172.21.15.102:0/1228125607 >> 172.21.15.102:0/1228125607 conn(0x7fcfdc08b1c0 msgr2=0x7fcfdc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:20:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 -- 172.21.15.102:0/1228125607 shutdown_connections 2023-03-03T08:20:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.567+0000 7fcfe159d700 1 -- 172.21.15.102:0/1228125607 wait complete. 2023-03-03T08:20:45.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.569+0000 7fcfe159d700 1 Processor -- start 2023-03-03T08:20:45.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.569+0000 7fcfe159d700 1 -- start start 2023-03-03T08:20:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.570+0000 7fcfe159d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc1e5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:45.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.570+0000 7fcfe159d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfdc0c41b0 0x7fcfdc159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.570+0000 7fcfe159d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0ced00 0x7fcfdc1e4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.570+0000 7fcfe159d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc1e40b0 con 0x7fcfdc0b3970 2023-03-03T08:20:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.570+0000 7fcfe159d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc15b1e0 con 0x7fcfdc0c41b0 2023-03-03T08:20:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.570+0000 7fcfe159d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc1e31c0 con 0x7fcfdc0ced00 2023-03-03T08:20:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc1e5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfdc0c41b0 0x7fcfdc159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:45.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc1e5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54456/0 (socket says 172.21.15.102:54456) 2023-03-03T08:20:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 -- 172.21.15.102:0/992126955 learned_addr learned my addr 172.21.15.102:0/992126955 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:20:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfe0d9c700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0ced00 0x7fcfdc1e4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 -- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0ced00 msgr2=0x7fcfdc1e4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:45.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0ced00 0x7fcfdc1e4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 -- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfdc0c41b0 msgr2=0x7fcfdc159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfdc0c41b0 0x7fcfdc159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdbfff700 1 -- 172.21.15.102:0/992126955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4009a70 con 0x7fcfdc0b3970 2023-03-03T08:20:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.571+0000 7fcfdb7fe700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfdc0c41b0 0x7fcfdc159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:20:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.572+0000 7fcfdbfff700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc1e5970 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0011e0 tx=0x7fcfcc00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:20:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.572+0000 7fcfd8ff9700 1 -- 172.21.15.102:0/992126955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc004c80 con 0x7fcfdc0b3970 2023-03-03T08:20:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.572+0000 7fcfd8ff9700 1 -- 172.21.15.102:0/992126955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfcc004de0 con 0x7fcfdc0b3970 2023-03-03T08:20:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.572+0000 7fcfd8ff9700 1 -- 172.21.15.102:0/992126955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc007310 con 0x7fcfdc0b3970 2023-03-03T08:20:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.572+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd00094a0 con 0x7fcfdc0b3970 2023-03-03T08:20:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.572+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfdc06bf20 con 0x7fcfdc0b3970 2023-03-03T08:20:45.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.573+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfdc15ce80 con 0x7fcfdc0b3970 2023-03-03T08:20:45.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.574+0000 7fcfd8ff9700 1 -- 172.21.15.102:0/992126955 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcfcc01a020 con 0x7fcfdc0b3970 2023-03-03T08:20:45.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.576+0000 7fcfd8ff9700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfc80682f0 0x7fcfc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:20:45.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.576+0000 7fcfd8ff9700 1 -- 172.21.15.102:0/992126955 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcfcc0859a0 con 0x7fcfdc0b3970 2023-03-03T08:20:45.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.576+0000 7fcfdb7fe700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfc80682f0 0x7fcfc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:20:45.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.577+0000 7fcfdb7fe700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfc80682f0 0x7fcfc806a7a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00097c0 tx=0x7fcfd001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:20:45.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.581+0000 7fcfd8ff9700 1 -- 172.21.15.102:0/992126955 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcfcc052020 con 0x7fcfdc0b3970 2023-03-03T08:20:45.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.816+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfdc06c200 con 0x7fcfdc0b3970 2023-03-03T08:20:45.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.817+0000 7fcfd8ff9700 1 -- 172.21.15.102:0/992126955 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcfcc055640 con 0x7fcfdc0b3970 2023-03-03T08:20:45.818 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:20:45.818 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:20:45.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.819+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfc80682f0 msgr2=0x7fcfc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:45.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.819+0000 7fcfe159d700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfc80682f0 0x7fcfc806a7a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00097c0 tx=0x7fcfd001c040 comp rx=0 tx=0).stop 2023-03-03T08:20:45.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.819+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 msgr2=0x7fcfdc1e5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:20:45.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.819+0000 7fcfe159d700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc1e5970 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0011e0 tx=0x7fcfcc00ca30 comp rx=0 tx=0).stop 2023-03-03T08:20:45.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 shutdown_connections 2023-03-03T08:20:45.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcfc80682f0 0x7fcfc806a7a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfdc0ced00 0x7fcfdc1e4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfdc0b3970 0x7fcfdc1e5970 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 --2- 172.21.15.102:0/992126955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfdc0c41b0 0x7fcfdc159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:20:45.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 >> 172.21.15.102:0/992126955 conn(0x7fcfdc08b1c0 msgr2=0x7fcfdc1e1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:20:45.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 shutdown_connections 2023-03-03T08:20:45.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:20:45.820+0000 7fcfe159d700 1 -- 172.21.15.102:0/992126955 wait complete. 2023-03-03T08:20:45.823 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:20:45.847 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2023-03-03T08:20:55.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:55.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:25.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:25.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:34.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:21:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.096+0000 7fe2cbbff700 1 Processor -- start 2023-03-03T08:21:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.096+0000 7fe2cbbff700 1 -- start start 2023-03-03T08:21:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.097+0000 7fe2cbbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2c41597f0 0x7fe2c40cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:21:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.097+0000 7fe2cbbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40cf9a0 0x7fe2c4163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:21:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.097+0000 7fe2cbbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c40cb9f0 con 0x7fe2c40ced80 2023-03-03T08:21:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.097+0000 7fe2cbbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c40ca750 con 0x7fe2c41597f0 2023-03-03T08:21:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.097+0000 7fe2cbbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c415a230 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2ca19c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40cf9a0 0x7fe2c4163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:21:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c919a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2c41597f0 0x7fe2c40cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:21:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c999b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe2c40ced80 0x7fe2c40cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56512/0 (socket says 172.21.15.102:56512) 2023-03-03T08:21:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c999b700 1 -- 172.21.15.102:0/2028955099 learned_addr learned my addr 172.21.15.102:0/2028955099 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:21:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c8999700 1 -- 172.21.15.102:0/2028955099 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206074020 0 0) 0x7fe2c40cb9f0 con 0x7fe2c40ced80 2023-03-03T08:21:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c8999700 1 -- 172.21.15.102:0/2028955099 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b00035b0 con 0x7fe2c40ced80 2023-03-03T08:21:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c919a700 1 -- 172.21.15.102:0/2028955099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40cf9a0 msgr2=0x7fe2c4163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:21:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c919a700 1 --2- 172.21.15.102:0/2028955099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40cf9a0 0x7fe2c4163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c919a700 1 -- 172.21.15.102:0/2028955099 >> v1:172.21.15.172:6789/0 conn(0x7fe2c40ced80 legacy=0x7fe2c40cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:21:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2c919a700 1 -- 172.21.15.102:0/2028955099 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c4159c80 con 0x7fe2c41597f0 2023-03-03T08:21:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.098+0000 7fe2ca19c700 1 --2- 172.21.15.102:0/2028955099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40cf9a0 0x7fe2c4163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:21:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.099+0000 7fe2c919a700 1 --2- 172.21.15.102:0/2028955099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c41597f0 0x7fe2c40cf150 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009840 tx=0x7fe2b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28385c78f79d5288 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:21:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.099+0000 7fe2c8999700 1 -- 172.21.15.102:0/2028955099 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4011830 con 0x7fe2c41597f0 2023-03-03T08:21:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.099+0000 7fe2c8999700 1 -- 172.21.15.102:0/2028955099 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b4011990 con 0x7fe2c41597f0 2023-03-03T08:21:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.099+0000 7fe2c8999700 1 -- 172.21.15.102:0/2028955099 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4011d20 con 0x7fe2c41597f0 2023-03-03T08:21:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.099+0000 7fe2cbbff700 1 -- 172.21.15.102:0/2028955099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c41597f0 msgr2=0x7fe2c40cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:21:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.099+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/2028955099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c41597f0 0x7fe2c40cf150 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009840 tx=0x7fe2b4009b50 comp rx=0 tx=0).stop 2023-03-03T08:21:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.100+0000 7fe2cbbff700 1 -- 172.21.15.102:0/2028955099 shutdown_connections 2023-03-03T08:21:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.100+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/2028955099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40cf9a0 0x7fe2c4163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.100+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/2028955099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c41597f0 0x7fe2c40cf150 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.100+0000 7fe2cbbff700 1 -- 172.21.15.102:0/2028955099 >> 172.21.15.102:0/2028955099 conn(0x7fe2c4065fb0 msgr2=0x7fe2c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:21:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.100+0000 7fe2cbbff700 1 -- 172.21.15.102:0/2028955099 shutdown_connections 2023-03-03T08:21:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.100+0000 7fe2cbbff700 1 -- 172.21.15.102:0/2028955099 wait complete. 2023-03-03T08:21:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.102+0000 7fe2cbbff700 1 Processor -- start 2023-03-03T08:21:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.102+0000 7fe2cbbff700 1 -- start start 2023-03-03T08:21:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2cbbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40ced80 0x7fe2c40c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:21:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2cbbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c40cf9a0 0x7fe2c41e4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:21:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2cbbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2c41597f0 0x7fe2c41514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:21:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2cbbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40b2fd0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2cbbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40b49a0 con 0x7fe2c41597f0 2023-03-03T08:21:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2cbbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c414f6f0 con 0x7fe2c40ced80 2023-03-03T08:21:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2c919a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c40cf9a0 0x7fe2c41e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:21:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2c919a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c40cf9a0 0x7fe2c41e4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43648/0 (socket says 172.21.15.102:43648) 2023-03-03T08:21:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2c919a700 1 -- 172.21.15.102:0/1643938702 learned_addr learned my addr 172.21.15.102:0/1643938702 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:21:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2ca19c700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2c41597f0 0x7fe2c41514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.103+0000 7fe2c999b700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40ced80 0x7fe2c40c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2c919a700 1 -- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40ced80 msgr2=0x7fe2c40c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2c919a700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40ced80 0x7fe2c40c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2c919a700 1 -- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2c41597f0 msgr2=0x7fe2c41514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2c919a700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2c41597f0 0x7fe2c41514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2c919a700 1 -- 172.21.15.102:0/1643938702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0094a0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2ca19c700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2c41597f0 0x7fe2c41514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:21:35.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2c919a700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c40cf9a0 0x7fe2c41e4390 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40090b0 tx=0x7fe2b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:21:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2ba7fc700 1 -- 172.21.15.102:0/1643938702 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4004550 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2ba7fc700 1 -- 172.21.15.102:0/1643938702 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b401e070 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2ba7fc700 1 -- 172.21.15.102:0/1643938702 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b40038e0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b40094a0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.104+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c40cb0c0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.105+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c40cade0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.107+0000 7fe2ba7fc700 1 -- 172.21.15.102:0/1643938702 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2b400f050 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.109+0000 7fe2ba7fc700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2a80682f0 0x7fe2a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:21:35.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.109+0000 7fe2ba7fc700 1 -- 172.21.15.102:0/1643938702 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe2b4086bc0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.109+0000 7fe2c999b700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2a80682f0 0x7fe2a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:21:35.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.110+0000 7fe2c999b700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2a80682f0 0x7fe2a806a7a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0001560 tx=0x7fe2c00019f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:21:35.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.113+0000 7fe2ba7fc700 1 -- 172.21.15.102:0/1643938702 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2b40532d0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.349+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2c40cb3a0 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.350+0000 7fe2ba7fc700 1 -- 172.21.15.102:0/1643938702 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe2b4015090 con 0x7fe2c40cf9a0 2023-03-03T08:21:35.351 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:21:35.351 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:21:35.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2a80682f0 msgr2=0x7fe2a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:21:35.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2a80682f0 0x7fe2a806a7a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0001560 tx=0x7fe2c00019f0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c40cf9a0 msgr2=0x7fe2c41e4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:21:35.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c40cf9a0 0x7fe2c41e4390 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40090b0 tx=0x7fe2b4004750 comp rx=0 tx=0).stop 2023-03-03T08:21:35.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 shutdown_connections 2023-03-03T08:21:35.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2a80682f0 0x7fe2a806a7a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2c40ced80 0x7fe2c40c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2c41597f0 0x7fe2c41514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 --2- 172.21.15.102:0/1643938702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2c40cf9a0 0x7fe2c41e4390 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:21:35.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.352+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 >> 172.21.15.102:0/1643938702 conn(0x7fe2c4065fb0 msgr2=0x7fe2c415a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:21:35.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.353+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 shutdown_connections 2023-03-03T08:21:35.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:21:35.353+0000 7fe2cbbff700 1 -- 172.21.15.102:0/1643938702 wait complete. 2023-03-03T08:21:35.355 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:21:35.380 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-03-03T08:21:55.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:55.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:25.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:25.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:56.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:56.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:17.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:23:17.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.629+0000 7f45f73ff700 1 Processor -- start 2023-03-03T08:23:17.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.629+0000 7f45f73ff700 1 -- start start 2023-03-03T08:23:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.630+0000 7f45f73ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f00c5f30 0x7f45f006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.630+0000 7f45f73ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45f00c2020 0x7f45f00c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.630+0000 7f45f73ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00c5390 con 0x7f45f0150c80 2023-03-03T08:23:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.630+0000 7f45f73ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f0157ad0 con 0x7f45f00c2020 2023-03-03T08:23:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.630+0000 7f45f73ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00c4a20 con 0x7f45f00c5f30 2023-03-03T08:23:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f519b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f45f0150c80 0x7f45f00c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37572/0 (socket says 172.21.15.102:37572) 2023-03-03T08:23:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f519b700 1 -- 172.21.15.102:0/3719236617 learned_addr learned my addr 172.21.15.102:0/3719236617 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:23:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f499a700 1 --2- 172.21.15.102:0/3719236617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f00c5f30 0x7f45f006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f599c700 1 --2- 172.21.15.102:0/3719236617 >> v2:172.21.15.102:3300/0 conn(0x7f45f00c2020 0x7f45f00c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f499a700 1 -- 172.21.15.102:0/3719236617 >> v2:172.21.15.102:3300/0 conn(0x7f45f00c2020 msgr2=0x7f45f00c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f499a700 1 --2- 172.21.15.102:0/3719236617 >> v2:172.21.15.102:3300/0 conn(0x7f45f00c2020 0x7f45f00c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f499a700 1 -- 172.21.15.102:0/3719236617 >> v1:172.21.15.102:6789/0 conn(0x7f45f0150c80 legacy=0x7f45f00c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.631+0000 7f45f499a700 1 -- 172.21.15.102:0/3719236617 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f006f010 con 0x7f45f00c5f30 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.632+0000 7f45f499a700 1 --2- 172.21.15.102:0/3719236617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f00c5f30 0x7f45f006d100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009840 tx=0x7f45e0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=745aeee42a897980 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.632+0000 7f45e7fff700 1 -- 172.21.15.102:0/3719236617 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0011830 con 0x7f45f00c5f30 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.632+0000 7f45e7fff700 1 -- 172.21.15.102:0/3719236617 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e0011990 con 0x7f45f00c5f30 2023-03-03T08:23:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.632+0000 7f45e7fff700 1 -- 172.21.15.102:0/3719236617 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0011d20 con 0x7f45f00c5f30 2023-03-03T08:23:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.632+0000 7f45f73ff700 1 -- 172.21.15.102:0/3719236617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f00c5f30 msgr2=0x7f45f006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.632+0000 7f45f73ff700 1 --2- 172.21.15.102:0/3719236617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f00c5f30 0x7f45f006d100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009840 tx=0x7f45e0009b50 comp rx=0 tx=0).stop 2023-03-03T08:23:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.633+0000 7f45f73ff700 1 -- 172.21.15.102:0/3719236617 shutdown_connections 2023-03-03T08:23:17.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.633+0000 7f45f73ff700 1 --2- 172.21.15.102:0/3719236617 >> v2:172.21.15.102:3300/0 conn(0x7f45f00c2020 0x7f45f00c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.633+0000 7f45f73ff700 1 --2- 172.21.15.102:0/3719236617 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f00c5f30 0x7f45f006d100 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.633+0000 7f45f73ff700 1 -- 172.21.15.102:0/3719236617 >> 172.21.15.102:0/3719236617 conn(0x7f45f006af90 msgr2=0x7f45f006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:23:17.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.633+0000 7f45f73ff700 1 -- 172.21.15.102:0/3719236617 shutdown_connections 2023-03-03T08:23:17.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.633+0000 7f45f73ff700 1 -- 172.21.15.102:0/3719236617 wait complete. 2023-03-03T08:23:17.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.635+0000 7f45f73ff700 1 Processor -- start 2023-03-03T08:23:17.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.635+0000 7f45f73ff700 1 -- start start 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f73ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f00c2020 0x7f45f01f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f73ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00c5f30 0x7f45f00c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f73ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f0150c80 0x7f45f01554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f73ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f0070e60 con 0x7f45f00c2020 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f73ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f0071120 con 0x7f45f00c5f30 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f73ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f0071400 con 0x7f45f0150c80 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f519b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f00c2020 0x7f45f01f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f519b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f00c2020 0x7f45f01f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59504/0 (socket says 172.21.15.102:59504) 2023-03-03T08:23:17.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f519b700 1 -- 172.21.15.102:0/1408185800 learned_addr learned my addr 172.21.15.102:0/1408185800 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:23:17.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f599c700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f0150c80 0x7f45f01554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:17.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.636+0000 7f45f499a700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00c5f30 0x7f45f00c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:17.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f519b700 1 -- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f0150c80 msgr2=0x7f45f01554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:17.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f519b700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f0150c80 0x7f45f01554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f519b700 1 -- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00c5f30 msgr2=0x7f45f00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f519b700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00c5f30 0x7f45f00c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f519b700 1 -- 172.21.15.102:0/1408185800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e00094a0 con 0x7f45f00c2020 2023-03-03T08:23:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f599c700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f0150c80 0x7f45f01554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:23:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f499a700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00c5f30 0x7f45f00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:23:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45f519b700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f00c2020 0x7f45f01f28a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c480 tx=0x7f45ec00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:23:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45e5ffb700 1 -- 172.21.15.102:0/1408185800 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec005d70 con 0x7f45f00c2020 2023-03-03T08:23:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45e5ffb700 1 -- 172.21.15.102:0/1408185800 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec00ce90 con 0x7f45f00c2020 2023-03-03T08:23:17.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.637+0000 7f45e5ffb700 1 -- 172.21.15.102:0/1408185800 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec0072d0 con 0x7f45f00c2020 2023-03-03T08:23:17.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.638+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f01577f0 con 0x7f45f00c2020 2023-03-03T08:23:17.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.638+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f0157510 con 0x7f45f00c2020 2023-03-03T08:23:17.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.639+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f00c4cc0 con 0x7f45f00c2020 2023-03-03T08:23:17.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.640+0000 7f45e5ffb700 1 -- 172.21.15.102:0/1408185800 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f45ec019030 con 0x7f45f00c2020 2023-03-03T08:23:17.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.642+0000 7f45e5ffb700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45d40682f0 0x7f45d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:17.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.642+0000 7f45e5ffb700 1 -- 172.21.15.102:0/1408185800 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f45ec084e80 con 0x7f45f00c2020 2023-03-03T08:23:17.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.642+0000 7f45f499a700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45d40682f0 0x7f45d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:17.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.643+0000 7f45f499a700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45d40682f0 0x7f45d406a7a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:23:17.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.647+0000 7f45e5ffb700 1 -- 172.21.15.102:0/1408185800 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f45ec04b5e0 con 0x7f45f00c2020 2023-03-03T08:23:17.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.881+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45f004e6e0 con 0x7f45f00c2020 2023-03-03T08:23:17.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.881+0000 7f45e5ffb700 1 -- 172.21.15.102:0/1408185800 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f45ec017090 con 0x7f45f00c2020 2023-03-03T08:23:17.883 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:23:17.883 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:23:17.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45d40682f0 msgr2=0x7f45d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:17.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45d40682f0 0x7f45d406a7a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0008030 comp rx=0 tx=0).stop 2023-03-03T08:23:17.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f00c2020 msgr2=0x7f45f01f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f00c2020 0x7f45f01f28a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c480 tx=0x7f45ec00c790 comp rx=0 tx=0).stop 2023-03-03T08:23:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 shutdown_connections 2023-03-03T08:23:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45d40682f0 0x7f45d406a7a0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45f0150c80 0x7f45f01554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f00c2020 0x7f45f01f28a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 --2- 172.21.15.102:0/1408185800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45f00c5f30 0x7f45f00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 >> 172.21.15.102:0/1408185800 conn(0x7f45f006af90 msgr2=0x7f45f014e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:23:17.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 shutdown_connections 2023-03-03T08:23:17.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:17.884+0000 7f45f73ff700 1 -- 172.21.15.102:0/1408185800 wait complete. 2023-03-03T08:23:17.887 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:23:17.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2023-03-03T08:23:26.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:26.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:34.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:23:35.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.163+0000 7fd29ddff700 1 Processor -- start 2023-03-03T08:23:35.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.163+0000 7fd29ddff700 1 -- start start 2023-03-03T08:23:35.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd29ddff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd2980cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:35.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd29ddff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2980cf9a0 0x7fd298163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:35.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd29ddff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2980cb9f0 con 0x7fd2980ced80 2023-03-03T08:23:35.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd29ddff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2980ca750 con 0x7fd2980cf9a0 2023-03-03T08:23:35.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd29ddff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29815a230 con 0x7fd2981597f0 2023-03-03T08:23:35.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd28ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd2980cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:35.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd2977fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd2980ced80 0x7fd2980cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33690/0 (socket says 172.21.15.102:33690) 2023-03-03T08:23:35.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd297fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2980cf9a0 0x7fd298163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.165+0000 7fd2977fe700 1 -- 172.21.15.102:0/3143556056 learned_addr learned my addr 172.21.15.102:0/3143556056 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:23:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd28ffff700 1 -- 172.21.15.102:0/3143556056 >> v2:172.21.15.172:3300/0 conn(0x7fd2980cf9a0 msgr2=0x7fd298163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:23:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd28ffff700 1 --2- 172.21.15.102:0/3143556056 >> v2:172.21.15.172:3300/0 conn(0x7fd2980cf9a0 0x7fd298163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd28ffff700 1 -- 172.21.15.102:0/3143556056 >> v1:172.21.15.172:6789/0 conn(0x7fd2980ced80 legacy=0x7fd2980cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd28ffff700 1 -- 172.21.15.102:0/3143556056 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298159c80 con 0x7fd2981597f0 2023-03-03T08:23:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd28ffff700 1 --2- 172.21.15.102:0/3143556056 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd2980cf150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd280009840 tx=0x7fd280009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8e9dde9202b6cdd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:23:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd296ffd700 1 -- 172.21.15.102:0/3143556056 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280011830 con 0x7fd2981597f0 2023-03-03T08:23:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd296ffd700 1 -- 172.21.15.102:0/3143556056 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd280011990 con 0x7fd2981597f0 2023-03-03T08:23:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.166+0000 7fd296ffd700 1 -- 172.21.15.102:0/3143556056 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280011d20 con 0x7fd2981597f0 2023-03-03T08:23:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 -- 172.21.15.102:0/3143556056 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 msgr2=0x7fd2980cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 --2- 172.21.15.102:0/3143556056 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd2980cf150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd280009840 tx=0x7fd280009b50 comp rx=0 tx=0).stop 2023-03-03T08:23:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 -- 172.21.15.102:0/3143556056 shutdown_connections 2023-03-03T08:23:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 --2- 172.21.15.102:0/3143556056 >> v2:172.21.15.172:3300/0 conn(0x7fd2980cf9a0 0x7fd298163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 --2- 172.21.15.102:0/3143556056 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd2980cf150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 -- 172.21.15.102:0/3143556056 >> 172.21.15.102:0/3143556056 conn(0x7fd298065fb0 msgr2=0x7fd2980663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:23:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 -- 172.21.15.102:0/3143556056 shutdown_connections 2023-03-03T08:23:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.167+0000 7fd29ddff700 1 -- 172.21.15.102:0/3143556056 wait complete. 2023-03-03T08:23:35.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.169+0000 7fd29ddff700 1 Processor -- start 2023-03-03T08:23:35.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.169+0000 7fd29ddff700 1 -- start start 2023-03-03T08:23:35.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.170+0000 7fd29ddff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980ced80 0x7fd2980b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:35.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.170+0000 7fd29ddff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2980cf9a0 0x7fd29814f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:35.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.170+0000 7fd29ddff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd29814f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:35.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.170+0000 7fd29ddff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29814db30 con 0x7fd2980ced80 2023-03-03T08:23:35.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.170+0000 7fd29ddff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29814ed50 con 0x7fd2980cf9a0 2023-03-03T08:23:35.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.170+0000 7fd29ddff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2980b2a90 con 0x7fd2981597f0 2023-03-03T08:23:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.170+0000 7fd2977fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980ced80 0x7fd2980b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd2977fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980ced80 0x7fd2980b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42118/0 (socket says 172.21.15.102:42118) 2023-03-03T08:23:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd2977fe700 1 -- 172.21.15.102:0/4243373355 learned_addr learned my addr 172.21.15.102:0/4243373355 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:23:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd28ffff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2980cf9a0 0x7fd29814f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd297fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd29814f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd28ffff700 1 -- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 msgr2=0x7fd29814f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd28ffff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd29814f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd28ffff700 1 -- 172.21.15.102:0/4243373355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980ced80 msgr2=0x7fd2980b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd28ffff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980ced80 0x7fd2980b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd28ffff700 1 -- 172.21.15.102:0/4243373355 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2800094a0 con 0x7fd2980cf9a0 2023-03-03T08:23:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd2977fe700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980ced80 0x7fd2980b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:23:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.171+0000 7fd28ffff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2980cf9a0 0x7fd29814f200 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd280016000 tx=0x7fd280004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:23:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.172+0000 7fd294ff9700 1 -- 172.21.15.102:0/4243373355 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2800036a0 con 0x7fd2980cf9a0 2023-03-03T08:23:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.172+0000 7fd294ff9700 1 -- 172.21.15.102:0/4243373355 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28001e070 con 0x7fd2980cf9a0 2023-03-03T08:23:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.172+0000 7fd294ff9700 1 -- 172.21.15.102:0/4243373355 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280003a40 con 0x7fd2980cf9a0 2023-03-03T08:23:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.172+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2980cb0c0 con 0x7fd2980cf9a0 2023-03-03T08:23:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.172+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2980cade0 con 0x7fd2980cf9a0 2023-03-03T08:23:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.173+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2980cb3a0 con 0x7fd2980cf9a0 2023-03-03T08:23:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.174+0000 7fd294ff9700 1 -- 172.21.15.102:0/4243373355 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd28000f050 con 0x7fd2980cf9a0 2023-03-03T08:23:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.175+0000 7fd294ff9700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2780682f0 0x7fd27806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:23:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.175+0000 7fd294ff9700 1 -- 172.21.15.102:0/4243373355 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd280087860 con 0x7fd2980cf9a0 2023-03-03T08:23:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.176+0000 7fd2977fe700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2780682f0 0x7fd27806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:23:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.176+0000 7fd2977fe700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2780682f0 0x7fd27806a7a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd288005910 tx=0x7fd288009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:23:35.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.178+0000 7fd294ff9700 1 -- 172.21.15.102:0/4243373355 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd280053ff0 con 0x7fd2980cf9a0 2023-03-03T08:23:35.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.415+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd29814eee0 con 0x7fd2980cf9a0 2023-03-03T08:23:35.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.416+0000 7fd294ff9700 1 -- 172.21.15.102:0/4243373355 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd280057610 con 0x7fd2980cf9a0 2023-03-03T08:23:35.418 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:23:35.418 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:23:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2780682f0 msgr2=0x7fd27806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2780682f0 0x7fd27806a7a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd288005910 tx=0x7fd288009040 comp rx=0 tx=0).stop 2023-03-03T08:23:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2980cf9a0 msgr2=0x7fd29814f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:23:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2980cf9a0 0x7fd29814f200 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd280016000 tx=0x7fd280004750 comp rx=0 tx=0).stop 2023-03-03T08:23:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 shutdown_connections 2023-03-03T08:23:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2780682f0 0x7fd27806a7a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2981597f0 0x7fd29814f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.418+0000 7fd29ddff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980ced80 0x7fd2980b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.419+0000 7fd29ddff700 1 --2- 172.21.15.102:0/4243373355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2980cf9a0 0x7fd29814f200 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:23:35.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.419+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 >> 172.21.15.102:0/4243373355 conn(0x7fd298065fb0 msgr2=0x7fd2980cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:23:35.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.419+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 shutdown_connections 2023-03-03T08:23:35.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:23:35.419+0000 7fd29ddff700 1 -- 172.21.15.102:0/4243373355 wait complete. 2023-03-03T08:23:35.423 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:23:35.447 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T08:23:56.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:56.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:15.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:24:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.718+0000 7f55f19ff700 1 Processor -- start 2023-03-03T08:24:15.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.719+0000 7f55f19ff700 1 -- start start 2023-03-03T08:24:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55f19ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd8e0 0x7f55ec0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55f19ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ec153660 con 0x7f55ec0cd390 2023-03-03T08:24:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55f19ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ec06c360 con 0x7f55ec153290 2023-03-03T08:24:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55f19ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec15cea0 con 0x7f55ec0cd8e0 2023-03-03T08:24:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55eb7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f55ec0cd390 0x7f55ec0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42514/0 (socket says 172.21.15.102:42514) 2023-03-03T08:24:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55eb7fe700 1 -- 172.21.15.102:0/492349586 learned_addr learned my addr 172.21.15.102:0/492349586 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:24:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55ea7fc700 1 --2- 172.21.15.102:0/492349586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd8e0 0x7f55ec0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.720+0000 7f55eaffd700 1 --1- 172.21.15.102:0/492349586 >> v1:172.21.15.172:6789/0 conn(0x7f55ec153290 0x7f55ec158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46100/0 (socket says 172.21.15.102:46100) 2023-03-03T08:24:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55ea7fc700 1 -- 172.21.15.102:0/492349586 >> v1:172.21.15.172:6789/0 conn(0x7f55ec153290 legacy=0x7f55ec158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55ea7fc700 1 -- 172.21.15.102:0/492349586 >> v1:172.21.15.102:6789/0 conn(0x7f55ec0cd390 legacy=0x7f55ec0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55e9ffb700 1 -- 172.21.15.102:0/492349586 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584575749 0 0) 0x7f55ec153660 con 0x7f55ec0cd390 2023-03-03T08:24:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55ea7fc700 1 -- 172.21.15.102:0/492349586 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec06c7f0 con 0x7f55ec0cd8e0 2023-03-03T08:24:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55ea7fc700 1 --2- 172.21.15.102:0/492349586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd8e0 0x7f55ec0cde20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55e00097f0 tx=0x7f55e0009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e61f92542d050677 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:24:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55e9ffb700 1 -- 172.21.15.102:0/492349586 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0004230 con 0x7f55ec0cd8e0 2023-03-03T08:24:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55e9ffb700 1 -- 172.21.15.102:0/492349586 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e0004390 con 0x7f55ec0cd8e0 2023-03-03T08:24:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.721+0000 7f55e9ffb700 1 -- 172.21.15.102:0/492349586 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e00036a0 con 0x7f55ec0cd8e0 2023-03-03T08:24:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.722+0000 7f55f19ff700 1 -- 172.21.15.102:0/492349586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd8e0 msgr2=0x7f55ec0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.722+0000 7f55f19ff700 1 --2- 172.21.15.102:0/492349586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd8e0 0x7f55ec0cde20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55e00097f0 tx=0x7f55e0009b00 comp rx=0 tx=0).stop 2023-03-03T08:24:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.722+0000 7f55f19ff700 1 -- 172.21.15.102:0/492349586 shutdown_connections 2023-03-03T08:24:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.722+0000 7f55f19ff700 1 --2- 172.21.15.102:0/492349586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd8e0 0x7f55ec0cde20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.722+0000 7f55f19ff700 1 -- 172.21.15.102:0/492349586 >> 172.21.15.102:0/492349586 conn(0x7f55ec065fb0 msgr2=0x7f55ec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:24:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.722+0000 7f55f19ff700 1 -- 172.21.15.102:0/492349586 shutdown_connections 2023-03-03T08:24:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.722+0000 7f55f19ff700 1 -- 172.21.15.102:0/492349586 wait complete. 2023-03-03T08:24:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.724+0000 7f55f19ff700 1 Processor -- start 2023-03-03T08:24:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.724+0000 7f55f19ff700 1 -- start start 2023-03-03T08:24:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.725+0000 7f55f19ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd390 0x7f55ec0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.725+0000 7f55f19ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55ec0cd8e0 0x7f55ec0b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.725+0000 7f55f19ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55ec153290 0x7f55ec0b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.725+0000 7f55f19ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec1e5fc0 con 0x7f55ec0cd8e0 2023-03-03T08:24:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.725+0000 7f55f19ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec153860 con 0x7f55ec153290 2023-03-03T08:24:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.725+0000 7f55f19ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0b30c0 con 0x7f55ec0cd390 2023-03-03T08:24:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eaffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd390 0x7f55ec0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55ec153290 0x7f55ec0b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55ea7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55ec0cd8e0 0x7f55ec0b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55ec153290 0x7f55ec0b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54654/0 (socket says 172.21.15.102:54654) 2023-03-03T08:24:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 -- 172.21.15.102:0/2112365237 learned_addr learned my addr 172.21.15.102:0/2112365237 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:24:15.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 -- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd390 msgr2=0x7f55ec0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:15.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd390 0x7f55ec0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 -- 172.21.15.102:0/2112365237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55ec0cd8e0 msgr2=0x7f55ec0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:15.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55ec0cd8e0 0x7f55ec0b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 -- 172.21.15.102:0/2112365237 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e00094a0 con 0x7f55ec153290 2023-03-03T08:24:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eaffd700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd390 0x7f55ec0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:24:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.726+0000 7f55eb7fe700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55ec153290 0x7f55ec0b40e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f55e400cc10 tx=0x7f55e400cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:24:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.727+0000 7f55d3fff700 1 -- 172.21.15.102:0/2112365237 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e400a9e0 con 0x7f55ec153290 2023-03-03T08:24:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.727+0000 7f55d3fff700 1 -- 172.21.15.102:0/2112365237 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e400ab40 con 0x7f55ec153290 2023-03-03T08:24:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.727+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec0ccde0 con 0x7f55ec153290 2023-03-03T08:24:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.727+0000 7f55d3fff700 1 -- 172.21.15.102:0/2112365237 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e40070d0 con 0x7f55ec153290 2023-03-03T08:24:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.727+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec15d160 con 0x7f55ec153290 2023-03-03T08:24:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.728+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec0cd0c0 con 0x7f55ec153290 2023-03-03T08:24:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.729+0000 7f55d3fff700 1 -- 172.21.15.102:0/2112365237 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f55e4010070 con 0x7f55ec153290 2023-03-03T08:24:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.730+0000 7f55d3fff700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55d80682f0 0x7f55d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:15.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.731+0000 7f55d3fff700 1 -- 172.21.15.102:0/2112365237 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f55e4085620 con 0x7f55ec153290 2023-03-03T08:24:15.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.731+0000 7f55eaffd700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55d80682f0 0x7f55d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:15.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.731+0000 7f55eaffd700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55d80682f0 0x7f55d806a7a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f55dc000f60 tx=0x7f55dc000ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:24:15.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.734+0000 7f55d3fff700 1 -- 172.21.15.102:0/2112365237 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f55e4051cd0 con 0x7f55ec153290 2023-03-03T08:24:15.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.973+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55ec1e61f0 con 0x7f55ec153290 2023-03-03T08:24:15.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.975+0000 7f55d3fff700 1 -- 172.21.15.102:0/2112365237 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f55e40552f0 con 0x7f55ec153290 2023-03-03T08:24:15.976 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:24:15.976 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:24:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.976+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55d80682f0 msgr2=0x7f55d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55d80682f0 0x7f55d806a7a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f55dc000f60 tx=0x7f55dc000ef0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55ec153290 msgr2=0x7f55ec0b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55ec153290 0x7f55ec0b40e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f55e400cc10 tx=0x7f55e400cf20 comp rx=0 tx=0).stop 2023-03-03T08:24:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 shutdown_connections 2023-03-03T08:24:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55d80682f0 0x7f55d806a7a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55ec0cd390 0x7f55ec0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55ec153290 0x7f55ec0b40e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 --2- 172.21.15.102:0/2112365237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55ec0cd8e0 0x7f55ec0b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 >> 172.21.15.102:0/2112365237 conn(0x7f55ec065fb0 msgr2=0x7f55ec150ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:24:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 shutdown_connections 2023-03-03T08:24:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:15.977+0000 7f55f19ff700 1 -- 172.21.15.102:0/2112365237 wait complete. 2023-03-03T08:24:15.981 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:24:16.006 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-03-03T08:24:26.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:26.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:39.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:24:39.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.257+0000 7faeb57ff700 1 Processor -- start 2023-03-03T08:24:39.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.257+0000 7faeb57ff700 1 -- start start 2023-03-03T08:24:39.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.258+0000 7faeb57ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb0153290 0x7faeb0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:39.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeb57ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faeb01593c0 0x7faeb00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:39.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeb57ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb0153660 con 0x7faeb00cd8e0 2023-03-03T08:24:39.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeb57ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb006c360 con 0x7faeb01593c0 2023-03-03T08:24:39.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeb57ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb015cea0 con 0x7faeb0153290 2023-03-03T08:24:39.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeaf7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7faeb00cd8e0 0x7faeb00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33104/0 (socket says 172.21.15.102:33104) 2023-03-03T08:24:39.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeaeffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb0153290 0x7faeb0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:39.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeaf7fe700 1 -- 172.21.15.102:0/917440629 learned_addr learned my addr 172.21.15.102:0/917440629 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:24:39.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeae7fc700 1 --2- 172.21.15.102:0/917440629 >> v2:172.21.15.172:3300/0 conn(0x7faeb01593c0 0x7faeb00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:39.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeadffb700 1 -- 172.21.15.102:0/917440629 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497370568 0 0) 0x7faeb0153660 con 0x7faeb00cd8e0 2023-03-03T08:24:39.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeadffb700 1 -- 172.21.15.102:0/917440629 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae980035b0 con 0x7faeb00cd8e0 2023-03-03T08:24:39.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeaeffd700 1 -- 172.21.15.102:0/917440629 >> v2:172.21.15.172:3300/0 conn(0x7faeb01593c0 msgr2=0x7faeb00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:24:39.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.259+0000 7faeaeffd700 1 --2- 172.21.15.102:0/917440629 >> v2:172.21.15.172:3300/0 conn(0x7faeb01593c0 0x7faeb00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.260+0000 7faeaeffd700 1 -- 172.21.15.102:0/917440629 >> v1:172.21.15.172:6789/0 conn(0x7faeb00cd8e0 legacy=0x7faeb00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:39.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.260+0000 7faeaeffd700 1 -- 172.21.15.102:0/917440629 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb006c7f0 con 0x7faeb0153290 2023-03-03T08:24:39.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.260+0000 7faeae7fc700 1 --2- 172.21.15.102:0/917440629 >> v2:172.21.15.172:3300/0 conn(0x7faeb01593c0 0x7faeb00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:24:39.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.260+0000 7faeaeffd700 1 --2- 172.21.15.102:0/917440629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb0153290 0x7faeb0158e80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faea0009840 tx=0x7faea0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fae57d52f57b867c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:24:39.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.260+0000 7faeadffb700 1 -- 172.21.15.102:0/917440629 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0011830 con 0x7faeb0153290 2023-03-03T08:24:39.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.260+0000 7faeadffb700 1 -- 172.21.15.102:0/917440629 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea0011990 con 0x7faeb0153290 2023-03-03T08:24:39.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.260+0000 7faeadffb700 1 -- 172.21.15.102:0/917440629 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0011d20 con 0x7faeb0153290 2023-03-03T08:24:39.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 -- 172.21.15.102:0/917440629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb0153290 msgr2=0x7faeb0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:39.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 --2- 172.21.15.102:0/917440629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb0153290 0x7faeb0158e80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faea0009840 tx=0x7faea0009b50 comp rx=0 tx=0).stop 2023-03-03T08:24:39.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 -- 172.21.15.102:0/917440629 shutdown_connections 2023-03-03T08:24:39.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 --2- 172.21.15.102:0/917440629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb0153290 0x7faeb0158e80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 --2- 172.21.15.102:0/917440629 >> v2:172.21.15.172:3300/0 conn(0x7faeb01593c0 0x7faeb00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 -- 172.21.15.102:0/917440629 >> 172.21.15.102:0/917440629 conn(0x7faeb0065fb0 msgr2=0x7faeb00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:24:39.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 -- 172.21.15.102:0/917440629 shutdown_connections 2023-03-03T08:24:39.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.261+0000 7faeb57ff700 1 -- 172.21.15.102:0/917440629 wait complete. 2023-03-03T08:24:39.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.264+0000 7faeb57ff700 1 Processor -- start 2023-03-03T08:24:39.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.264+0000 7faeb57ff700 1 -- start start 2023-03-03T08:24:39.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.265+0000 7faeb57ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb00cd8e0 0x7faeb00caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:39.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.265+0000 7faeb57ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb0153290 0x7faeb00b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:39.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.265+0000 7faeb57ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb01593c0 0x7faeb00b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:39.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.265+0000 7faeb57ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb00c7080 con 0x7faeb01593c0 2023-03-03T08:24:39.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.265+0000 7faeb57ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb00c8260 con 0x7faeb0153290 2023-03-03T08:24:39.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.265+0000 7faeb57ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb00c6140 con 0x7faeb00cd8e0 2023-03-03T08:24:39.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaeffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb00cd8e0 0x7faeb00caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:39.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeae7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb0153290 0x7faeb00b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:39.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeae7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb0153290 0x7faeb00b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42222/0 (socket says 172.21.15.102:42222) 2023-03-03T08:24:39.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeae7fc700 1 -- 172.21.15.102:0/1152612487 learned_addr learned my addr 172.21.15.102:0/1152612487 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:24:39.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaf7fe700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb01593c0 0x7faeb00b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:39.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaeffd700 1 -- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb0153290 msgr2=0x7faeb00b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:39.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaeffd700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb0153290 0x7faeb00b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaeffd700 1 -- 172.21.15.102:0/1152612487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb01593c0 msgr2=0x7faeb00b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:39.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaeffd700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb01593c0 0x7faeb00b22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaeffd700 1 -- 172.21.15.102:0/1152612487 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea40094a0 con 0x7faeb00cd8e0 2023-03-03T08:24:39.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.266+0000 7faeaeffd700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb00cd8e0 0x7faeb00caf70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faea0016000 tx=0x7faea0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:24:39.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.267+0000 7fae977fe700 1 -- 172.21.15.102:0/1152612487 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0004590 con 0x7faeb00cd8e0 2023-03-03T08:24:39.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.267+0000 7fae977fe700 1 -- 172.21.15.102:0/1152612487 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea001e070 con 0x7faeb00cd8e0 2023-03-03T08:24:39.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.267+0000 7fae977fe700 1 -- 172.21.15.102:0/1152612487 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea00038c0 con 0x7faeb00cd8e0 2023-03-03T08:24:39.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.267+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea00094a0 con 0x7faeb00cd8e0 2023-03-03T08:24:39.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.267+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb00ccde0 con 0x7faeb00cd8e0 2023-03-03T08:24:39.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.268+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb015d160 con 0x7faeb00cd8e0 2023-03-03T08:24:39.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.269+0000 7fae977fe700 1 -- 172.21.15.102:0/1152612487 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faea000f050 con 0x7faeb00cd8e0 2023-03-03T08:24:39.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.271+0000 7fae977fe700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae9c068320 0x7fae9c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:24:39.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.271+0000 7fae977fe700 1 -- 172.21.15.102:0/1152612487 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7faea0086e10 con 0x7faeb00cd8e0 2023-03-03T08:24:39.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.271+0000 7faeae7fc700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae9c068320 0x7fae9c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:24:39.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.272+0000 7faeae7fc700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae9c068320 0x7fae9c06a7d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faeb008c980 tx=0x7faea4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:24:39.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.275+0000 7fae977fe700 1 -- 172.21.15.102:0/1152612487 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faea008a030 con 0x7faeb00cd8e0 2023-03-03T08:24:39.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.508+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeb00cd0c0 con 0x7faeb00cd8e0 2023-03-03T08:24:39.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.509+0000 7fae977fe700 1 -- 172.21.15.102:0/1152612487 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7faea0020090 con 0x7faeb00cd8e0 2023-03-03T08:24:39.510 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:24:39.511 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:24:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae9c068320 msgr2=0x7fae9c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae9c068320 0x7fae9c06a7d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faeb008c980 tx=0x7faea4009210 comp rx=0 tx=0).stop 2023-03-03T08:24:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb00cd8e0 msgr2=0x7faeb00caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:24:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb00cd8e0 0x7faeb00caf70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faea0016000 tx=0x7faea0004750 comp rx=0 tx=0).stop 2023-03-03T08:24:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 shutdown_connections 2023-03-03T08:24:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae9c068320 0x7fae9c06a7d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb00cd8e0 0x7faeb00caf70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb01593c0 0x7faeb00b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 --2- 172.21.15.102:0/1152612487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb0153290 0x7faeb00b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:24:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 >> 172.21.15.102:0/1152612487 conn(0x7faeb0065fb0 msgr2=0x7faeb015ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:24:39.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 shutdown_connections 2023-03-03T08:24:39.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:24:39.512+0000 7faeb57ff700 1 -- 172.21.15.102:0/1152612487 wait complete. 2023-03-03T08:24:39.515 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:24:39.540 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-03-03T08:24:56.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:56.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:27.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:27.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:57.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:57.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:27.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:27.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:32.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:26:32.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.789+0000 7f233e79d700 1 Processor -- start 2023-03-03T08:26:32.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.789+0000 7f233e79d700 1 -- start start 2023-03-03T08:26:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f233e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380c5a90 0x7f23380c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:26:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f233e79d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380c3d20 con 0x7f23380ced00 2023-03-03T08:26:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f233e79d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233815cba0 con 0x7f23380b3970 2023-03-03T08:26:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f233e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380c4930 con 0x7f23380c5a90 2023-03-03T08:26:32.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f233d79b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f23380ced00 0x7f2338158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49010/0 (socket says 172.21.15.102:49010) 2023-03-03T08:26:32.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f233d79b700 1 -- 172.21.15.102:0/2831964171 learned_addr learned my addr 172.21.15.102:0/2831964171 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:26:32.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f233df9c700 1 --2- 172.21.15.102:0/2831964171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380c5a90 0x7f23380c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:26:32.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.791+0000 7f232ffff700 1 -- 172.21.15.102:0/2831964171 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249053273 0 0) 0x7f23380c3d20 con 0x7f23380ced00 2023-03-03T08:26:32.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f232ffff700 1 -- 172.21.15.102:0/2831964171 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23240035b0 con 0x7f23380ced00 2023-03-03T08:26:32.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f232ffff700 1 -- 172.21.15.102:0/2831964171 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939599996 0 0) 0x7f233815cba0 con 0x7f23380b3970 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f232ffff700 1 -- 172.21.15.102:0/2831964171 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23380c3d20 con 0x7f23380b3970 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f233df9c700 1 -- 172.21.15.102:0/2831964171 >> v1:172.21.15.172:6789/0 conn(0x7f23380b3970 legacy=0x7f2338161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f233df9c700 1 -- 172.21.15.102:0/2831964171 >> v1:172.21.15.102:6789/0 conn(0x7f23380ced00 legacy=0x7f2338158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f233df9c700 1 -- 172.21.15.102:0/2831964171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233815ce80 con 0x7f23380c5a90 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f233df9c700 1 --2- 172.21.15.102:0/2831964171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380c5a90 0x7f23380c41b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2330009dc0 tx=0x7f233000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e7c7a31e2843f8f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f232ffff700 1 -- 172.21.15.102:0/2831964171 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233000e070 con 0x7f23380c5a90 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f232ffff700 1 -- 172.21.15.102:0/2831964171 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233000bb20 con 0x7f23380c5a90 2023-03-03T08:26:32.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.792+0000 7f232ffff700 1 -- 172.21.15.102:0/2831964171 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330012430 con 0x7f23380c5a90 2023-03-03T08:26:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.793+0000 7f233e79d700 1 -- 172.21.15.102:0/2831964171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380c5a90 msgr2=0x7f23380c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:26:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.793+0000 7f233e79d700 1 --2- 172.21.15.102:0/2831964171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380c5a90 0x7f23380c41b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2330009dc0 tx=0x7f233000b330 comp rx=0 tx=0).stop 2023-03-03T08:26:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.793+0000 7f233e79d700 1 -- 172.21.15.102:0/2831964171 shutdown_connections 2023-03-03T08:26:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.793+0000 7f233e79d700 1 --2- 172.21.15.102:0/2831964171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380c5a90 0x7f23380c41b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:26:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.793+0000 7f233e79d700 1 -- 172.21.15.102:0/2831964171 >> 172.21.15.102:0/2831964171 conn(0x7f233808b1c0 msgr2=0x7f2338071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:26:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.793+0000 7f233e79d700 1 -- 172.21.15.102:0/2831964171 shutdown_connections 2023-03-03T08:26:32.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.793+0000 7f233e79d700 1 -- 172.21.15.102:0/2831964171 wait complete. 2023-03-03T08:26:32.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.795+0000 7f233e79d700 1 Processor -- start 2023-03-03T08:26:32.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.796+0000 7f233e79d700 1 -- start start 2023-03-03T08:26:32.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380b3970 0x7f23381511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:26:32.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233e79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23380c5a90 0x7f2338159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:26:32.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233e79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23380ced00 0x7f233814ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:26:32.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233e79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380c7bf0 con 0x7f23380c5a90 2023-03-03T08:26:32.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233e79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233814f5a0 con 0x7f23380ced00 2023-03-03T08:26:32.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233814fb20 con 0x7f23380b3970 2023-03-03T08:26:32.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233d79b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380b3970 0x7f23381511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:26:32.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233cf9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23380c5a90 0x7f2338159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:26:32.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233d79b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380b3970 0x7f23381511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46958/0 (socket says 172.21.15.102:46958) 2023-03-03T08:26:32.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233d79b700 1 -- 172.21.15.102:0/2828648186 learned_addr learned my addr 172.21.15.102:0/2828648186 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:26:32.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.797+0000 7f233df9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23380ced00 0x7f233814ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:26:32.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233d79b700 1 -- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23380ced00 msgr2=0x7f233814ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:26:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233d79b700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23380ced00 0x7f233814ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:26:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233d79b700 1 -- 172.21.15.102:0/2828648186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23380c5a90 msgr2=0x7f2338159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:26:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233d79b700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23380c5a90 0x7f2338159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:26:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233d79b700 1 -- 172.21.15.102:0/2828648186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2330009a70 con 0x7f23380b3970 2023-03-03T08:26:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233df9c700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23380ced00 0x7f233814ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:26:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233cf9a700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23380c5a90 0x7f2338159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:26:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233d79b700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380b3970 0x7f23381511f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f233400c700 tx=0x7f233400ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:26:32.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f232dffb700 1 -- 172.21.15.102:0/2828648186 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334005d70 con 0x7f23380b3970 2023-03-03T08:26:32.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f232dffb700 1 -- 172.21.15.102:0/2828648186 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23340070d0 con 0x7f23380b3970 2023-03-03T08:26:32.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f232dffb700 1 -- 172.21.15.102:0/2828648186 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334007440 con 0x7f23380b3970 2023-03-03T08:26:32.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233806c200 con 0x7f23380b3970 2023-03-03T08:26:32.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.798+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233806bf20 con 0x7f23380b3970 2023-03-03T08:26:32.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.799+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233815c890 con 0x7f23380b3970 2023-03-03T08:26:32.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.801+0000 7f232dffb700 1 -- 172.21.15.102:0/2828648186 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2334019030 con 0x7f23380b3970 2023-03-03T08:26:32.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.802+0000 7f232dffb700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f231c068320 0x7f231c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:26:32.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.802+0000 7f232dffb700 1 -- 172.21.15.102:0/2828648186 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2334084f30 con 0x7f23380b3970 2023-03-03T08:26:32.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.802+0000 7f233cf9a700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f231c068320 0x7f231c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:26:32.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.803+0000 7f233cf9a700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f231c068320 0x7f231c06a7d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f23380c4430 tx=0x7f2328009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:26:32.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:32.806+0000 7f232dffb700 1 -- 172.21.15.102:0/2828648186 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f233404b5e0 con 0x7f23380b3970 2023-03-03T08:26:33.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.040+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233814fe00 con 0x7f23380b3970 2023-03-03T08:26:33.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.041+0000 7f232dffb700 1 -- 172.21.15.102:0/2828648186 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2334017070 con 0x7f23380b3970 2023-03-03T08:26:33.043 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:26:33.043 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:26:33.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f231c068320 msgr2=0x7f231c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:26:33.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f231c068320 0x7f231c06a7d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f23380c4430 tx=0x7f2328009040 comp rx=0 tx=0).stop 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380b3970 msgr2=0x7f23381511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380b3970 0x7f23381511f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f233400c700 tx=0x7f233400ca10 comp rx=0 tx=0).stop 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 shutdown_connections 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f231c068320 0x7f231c06a7d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f23380b3970 0x7f23381511f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23380ced00 0x7f233814ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 --2- 172.21.15.102:0/2828648186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23380c5a90 0x7f2338159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:26:33.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 >> 172.21.15.102:0/2828648186 conn(0x7f233808b1c0 msgr2=0x7f23380cdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:26:33.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 shutdown_connections 2023-03-03T08:26:33.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:26:33.044+0000 7f233e79d700 1 -- 172.21.15.102:0/2828648186 wait complete. 2023-03-03T08:26:33.047 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:26:33.072 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T08:26:57.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:57.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:27.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:27.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:40.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:27:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.341+0000 7f6a3adff700 1 Processor -- start 2023-03-03T08:27:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.341+0000 7f6a3adff700 1 -- start start 2023-03-03T08:27:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.342+0000 7f6a3adff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a340cd8e0 0x7f6a340cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:27:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a3adff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a34153660 con 0x7f6a34153290 2023-03-03T08:27:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a3adff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a3406c360 con 0x7f6a340cd390 2023-03-03T08:27:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a3adff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3415cea0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a38b9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6a34153290 0x7f6a34158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51512/0 (socket says 172.21.15.102:51512) 2023-03-03T08:27:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a38b9b700 1 -- 172.21.15.102:0/690200485 learned_addr learned my addr 172.21.15.102:0/690200485 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:27:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a33fff700 1 --2- 172.21.15.102:0/690200485 >> v2:172.21.15.102:3300/0 conn(0x7f6a340cd8e0 0x7f6a340cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:27:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a337fe700 1 -- 172.21.15.102:0/690200485 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063842920 0 0) 0x7f6a3406c360 con 0x7f6a340cd390 2023-03-03T08:27:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a337fe700 1 -- 172.21.15.102:0/690200485 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a240035b0 con 0x7f6a340cd390 2023-03-03T08:27:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a33fff700 1 -- 172.21.15.102:0/690200485 >> v1:172.21.15.172:6789/0 conn(0x7f6a340cd390 legacy=0x7f6a340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:27:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.343+0000 7f6a33fff700 1 -- 172.21.15.102:0/690200485 >> v1:172.21.15.102:6789/0 conn(0x7f6a34153290 legacy=0x7f6a34158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:27:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.344+0000 7f6a337fe700 1 -- 172.21.15.102:0/690200485 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781872257 0 0) 0x7f6a34153660 con 0x7f6a34153290 2023-03-03T08:27:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.344+0000 7f6a33fff700 1 -- 172.21.15.102:0/690200485 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3406c7f0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.344+0000 7f6a33fff700 1 --2- 172.21.15.102:0/690200485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd8e0 0x7f6a340cde20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009840 tx=0x7f6a20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2f14f7495bc0b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:27:40.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.344+0000 7f6a337fe700 1 -- 172.21.15.102:0/690200485 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20011830 con 0x7f6a340cd8e0 2023-03-03T08:27:40.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.344+0000 7f6a337fe700 1 -- 172.21.15.102:0/690200485 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a20011990 con 0x7f6a340cd8e0 2023-03-03T08:27:40.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.344+0000 7f6a337fe700 1 -- 172.21.15.102:0/690200485 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20011d20 con 0x7f6a340cd8e0 2023-03-03T08:27:40.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.345+0000 7f6a3adff700 1 -- 172.21.15.102:0/690200485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd8e0 msgr2=0x7f6a340cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:27:40.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.345+0000 7f6a3adff700 1 --2- 172.21.15.102:0/690200485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd8e0 0x7f6a340cde20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009840 tx=0x7f6a20009b50 comp rx=0 tx=0).stop 2023-03-03T08:27:40.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.345+0000 7f6a3adff700 1 -- 172.21.15.102:0/690200485 shutdown_connections 2023-03-03T08:27:40.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.345+0000 7f6a3adff700 1 --2- 172.21.15.102:0/690200485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd8e0 0x7f6a340cde20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:27:40.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.345+0000 7f6a3adff700 1 -- 172.21.15.102:0/690200485 >> 172.21.15.102:0/690200485 conn(0x7f6a34065fb0 msgr2=0x7f6a340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:27:40.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.345+0000 7f6a3adff700 1 -- 172.21.15.102:0/690200485 shutdown_connections 2023-03-03T08:27:40.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.345+0000 7f6a3adff700 1 -- 172.21.15.102:0/690200485 wait complete. 2023-03-03T08:27:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.347+0000 7f6a3adff700 1 Processor -- start 2023-03-03T08:27:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.347+0000 7f6a3adff700 1 -- start start 2023-03-03T08:27:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.348+0000 7f6a3adff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd390 0x7f6a341e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:27:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.348+0000 7f6a3adff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a340cd8e0 0x7f6a341500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:27:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.348+0000 7f6a3adff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a34153290 0x7f6a34150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:27:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.348+0000 7f6a3adff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a34158a80 con 0x7f6a340cd390 2023-03-03T08:27:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.348+0000 7f6a3adff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3414f520 con 0x7f6a340cd8e0 2023-03-03T08:27:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.348+0000 7f6a3adff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a340c57c0 con 0x7f6a34153290 2023-03-03T08:27:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a340cd8e0 0x7f6a341500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:27:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a3939c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a34153290 0x7f6a34150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:27:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a340cd8e0 0x7f6a341500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:51940/0 (socket says 172.21.15.102:51940) 2023-03-03T08:27:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 -- 172.21.15.102:0/2821710332 learned_addr learned my addr 172.21.15.102:0/2821710332 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:27:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a38b9b700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd390 0x7f6a341e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:27:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 -- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a34153290 msgr2=0x7f6a34150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:27:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a34153290 0x7f6a34150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:27:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 -- 172.21.15.102:0/2821710332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd390 msgr2=0x7f6a341e54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:27:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd390 0x7f6a341e54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:27:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a33fff700 1 -- 172.21.15.102:0/2821710332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200094a0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.349+0000 7f6a3939c700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a34153290 0x7f6a34150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:27:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.350+0000 7f6a33fff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a340cd8e0 0x7f6a341500c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6a200090b0 tx=0x7f6a20004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:27:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.350+0000 7f6a317fa700 1 -- 172.21.15.102:0/2821710332 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2001a7e0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.350+0000 7f6a317fa700 1 -- 172.21.15.102:0/2821710332 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2001a940 con 0x7f6a340cd8e0 2023-03-03T08:27:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.350+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a340ccde0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.350+0000 7f6a317fa700 1 -- 172.21.15.102:0/2821710332 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2001acd0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.350+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3415d160 con 0x7f6a340cd8e0 2023-03-03T08:27:40.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.351+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a340cd0c0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.353+0000 7f6a317fa700 1 -- 172.21.15.102:0/2821710332 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a2002b030 con 0x7f6a340cd8e0 2023-03-03T08:27:40.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.354+0000 7f6a317fa700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a180682f0 0x7f6a1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:27:40.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.354+0000 7f6a317fa700 1 -- 172.21.15.102:0/2821710332 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6a20087600 con 0x7f6a340cd8e0 2023-03-03T08:27:40.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.354+0000 7f6a38b9b700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a180682f0 0x7f6a1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:27:40.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.355+0000 7f6a38b9b700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a180682f0 0x7f6a1806a7a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6a28000f60 tx=0x7f6a28007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:27:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.357+0000 7f6a317fa700 1 -- 172.21.15.102:0/2821710332 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a20053d90 con 0x7f6a340cd8e0 2023-03-03T08:27:40.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.590+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a341ef2e0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.591+0000 7f6a317fa700 1 -- 172.21.15.102:0/2821710332 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6a200573b0 con 0x7f6a340cd8e0 2023-03-03T08:27:40.594 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:27:40.594 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:27:40.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a180682f0 msgr2=0x7f6a1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:27:40.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a180682f0 0x7f6a1806a7a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6a28000f60 tx=0x7f6a28007e70 comp rx=0 tx=0).stop 2023-03-03T08:27:40.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a340cd8e0 msgr2=0x7f6a341500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:27:40.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a340cd8e0 0x7f6a341500c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6a200090b0 tx=0x7f6a20004750 comp rx=0 tx=0).stop 2023-03-03T08:27:40.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 shutdown_connections 2023-03-03T08:27:40.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a180682f0 0x7f6a1806a7a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:27:40.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a34153290 0x7f6a34150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:27:40.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a340cd390 0x7f6a341e54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:27:40.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 --2- 172.21.15.102:0/2821710332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a340cd8e0 0x7f6a341500c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:27:40.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 >> 172.21.15.102:0/2821710332 conn(0x7f6a34065fb0 msgr2=0x7f6a34156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:27:40.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 shutdown_connections 2023-03-03T08:27:40.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:27:40.594+0000 7f6a3adff700 1 -- 172.21.15.102:0/2821710332 wait complete. 2023-03-03T08:27:40.598 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:27:40.623 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T08:27:58.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:58.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:08.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:28:08.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.874+0000 7f8ac8dff700 1 Processor -- start 2023-03-03T08:28:08.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.874+0000 7f8ac8dff700 1 -- start start 2023-03-03T08:28:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.876+0000 7f8ac8dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac40c2020 0x7f8ac406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:28:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.876+0000 7f8ac8dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac40c5390 con 0x7f8ac4150c80 2023-03-03T08:28:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.876+0000 7f8ac8dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac40c4a20 con 0x7f8ac40c5f30 2023-03-03T08:28:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.876+0000 7f8ac8dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac4157ad0 con 0x7f8ac40c2020 2023-03-03T08:28:08.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.876+0000 7f8ac259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8ac4150c80 0x7f8ac40c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59398/0 (socket says 172.21.15.102:59398) 2023-03-03T08:28:08.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.876+0000 7f8ac259c700 1 -- 172.21.15.102:0/3550483567 learned_addr learned my addr 172.21.15.102:0/3550483567 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:28:08.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.876+0000 7f8ac2d9d700 1 --2- 172.21.15.102:0/3550483567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac40c2020 0x7f8ac406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:28:08.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac159a700 1 -- 172.21.15.102:0/3550483567 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695708039 0 0) 0x7f8ac40c5390 con 0x7f8ac4150c80 2023-03-03T08:28:08.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac159a700 1 -- 172.21.15.102:0/3550483567 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aac0035b0 con 0x7f8ac4150c80 2023-03-03T08:28:08.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac2d9d700 1 -- 172.21.15.102:0/3550483567 >> v1:172.21.15.172:6789/0 conn(0x7f8ac40c5f30 legacy=0x7f8ac40c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:28:08.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac2d9d700 1 -- 172.21.15.102:0/3550483567 >> v1:172.21.15.102:6789/0 conn(0x7f8ac4150c80 legacy=0x7f8ac40c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:28:08.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac2d9d700 1 -- 172.21.15.102:0/3550483567 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac406f010 con 0x7f8ac40c2020 2023-03-03T08:28:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac2d9d700 1 --2- 172.21.15.102:0/3550483567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac40c2020 0x7f8ac406d100 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009dc0 tx=0x7f8abc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca95c99b35b67d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:28:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac159a700 1 -- 172.21.15.102:0/3550483567 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc00e070 con 0x7f8ac40c2020 2023-03-03T08:28:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac159a700 1 -- 172.21.15.102:0/3550483567 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc00bb20 con 0x7f8ac40c2020 2023-03-03T08:28:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.877+0000 7f8ac159a700 1 -- 172.21.15.102:0/3550483567 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc012430 con 0x7f8ac40c2020 2023-03-03T08:28:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.878+0000 7f8ac8dff700 1 -- 172.21.15.102:0/3550483567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac40c2020 msgr2=0x7f8ac406d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:28:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.878+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/3550483567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac40c2020 0x7f8ac406d100 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009dc0 tx=0x7f8abc00b330 comp rx=0 tx=0).stop 2023-03-03T08:28:08.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.878+0000 7f8ac8dff700 1 -- 172.21.15.102:0/3550483567 shutdown_connections 2023-03-03T08:28:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.878+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/3550483567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac40c2020 0x7f8ac406d100 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:28:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.878+0000 7f8ac8dff700 1 -- 172.21.15.102:0/3550483567 >> 172.21.15.102:0/3550483567 conn(0x7f8ac406af90 msgr2=0x7f8ac406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:28:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.878+0000 7f8ac8dff700 1 -- 172.21.15.102:0/3550483567 shutdown_connections 2023-03-03T08:28:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.878+0000 7f8ac8dff700 1 -- 172.21.15.102:0/3550483567 wait complete. 2023-03-03T08:28:08.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.880+0000 7f8ac8dff700 1 Processor -- start 2023-03-03T08:28:08.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.880+0000 7f8ac8dff700 1 -- start start 2023-03-03T08:28:08.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.881+0000 7f8ac8dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac40c2020 0x7f8ac4156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:28:08.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.881+0000 7f8ac8dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac40c5f30 0x7f8ac41e0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:28:08.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.881+0000 7f8ac8dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac4150c80 0x7f8ac4156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:28:08.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.881+0000 7f8ac8dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac41f2020 con 0x7f8ac40c5f30 2023-03-03T08:28:08.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.881+0000 7f8ac8dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac41f2550 con 0x7f8ac40c2020 2023-03-03T08:28:08.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.881+0000 7f8ac8dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac4155bf0 con 0x7f8ac4150c80 2023-03-03T08:28:08.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac40c2020 0x7f8ac4156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:28:08.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac4150c80 0x7f8ac4156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:28:08.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac40c2020 0x7f8ac4156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52298/0 (socket says 172.21.15.102:52298) 2023-03-03T08:28:08.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac259c700 1 -- 172.21.15.102:0/1173306186 learned_addr learned my addr 172.21.15.102:0/1173306186 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:28:08.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac1d9b700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac40c5f30 0x7f8ac41e0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:28:08.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac2d9d700 1 -- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac40c2020 msgr2=0x7f8ac4156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:28:08.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac2d9d700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac40c2020 0x7f8ac4156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:28:08.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac2d9d700 1 -- 172.21.15.102:0/1173306186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac40c5f30 msgr2=0x7f8ac41e0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:28:08.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac2d9d700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac40c5f30 0x7f8ac41e0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:28:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac2d9d700 1 -- 172.21.15.102:0/1173306186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc009a70 con 0x7f8ac4150c80 2023-03-03T08:28:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac1d9b700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac40c5f30 0x7f8ac41e0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:28:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.882+0000 7f8ac2d9d700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac4150c80 0x7f8ac4156080 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8abc016000 tx=0x7f8abc004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:28:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.883+0000 7f8ab2ffd700 1 -- 172.21.15.102:0/1173306186 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc012430 con 0x7f8ac4150c80 2023-03-03T08:28:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.883+0000 7f8ab2ffd700 1 -- 172.21.15.102:0/1173306186 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc020070 con 0x7f8ac4150c80 2023-03-03T08:28:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.883+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac41577f0 con 0x7f8ac4150c80 2023-03-03T08:28:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.883+0000 7f8ab2ffd700 1 -- 172.21.15.102:0/1173306186 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc004280 con 0x7f8ac4150c80 2023-03-03T08:28:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.883+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac4157510 con 0x7f8ac4150c80 2023-03-03T08:28:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.884+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac40c4cc0 con 0x7f8ac4150c80 2023-03-03T08:28:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.885+0000 7f8ab2ffd700 1 -- 172.21.15.102:0/1173306186 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8abc0043e0 con 0x7f8ac4150c80 2023-03-03T08:28:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.886+0000 7f8ab2ffd700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8aa4068320 0x7f8aa406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:28:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.887+0000 7f8ab2ffd700 1 -- 172.21.15.102:0/1173306186 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8abc022070 con 0x7f8ac4150c80 2023-03-03T08:28:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.887+0000 7f8ac259c700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8aa4068320 0x7f8aa406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:28:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.887+0000 7f8ac259c700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8aa4068320 0x7f8aa406a7d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40011e0 tx=0x7f8ab4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:28:08.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:08.891+0000 7f8ab2ffd700 1 -- 172.21.15.102:0/1173306186 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8abc054750 con 0x7f8ac4150c80 2023-03-03T08:28:09.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.125+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ac404e6e0 con 0x7f8ac4150c80 2023-03-03T08:28:09.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.126+0000 7f8ab2ffd700 1 -- 172.21.15.102:0/1173306186 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8abc057d70 con 0x7f8ac4150c80 2023-03-03T08:28:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:28:09.128 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:28:09.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8aa4068320 msgr2=0x7f8aa406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:28:09.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8aa4068320 0x7f8aa406a7d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40011e0 tx=0x7f8ab4009040 comp rx=0 tx=0).stop 2023-03-03T08:28:09.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac4150c80 msgr2=0x7f8ac4156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:28:09.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac4150c80 0x7f8ac4156080 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8abc016000 tx=0x7f8abc004ad0 comp rx=0 tx=0).stop 2023-03-03T08:28:09.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 shutdown_connections 2023-03-03T08:28:09.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8aa4068320 0x7f8aa406a7d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:28:09.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ac4150c80 0x7f8ac4156080 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:28:09.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac40c2020 0x7f8ac4156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:28:09.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 --2- 172.21.15.102:0/1173306186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac40c5f30 0x7f8ac41e0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:28:09.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.128+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 >> 172.21.15.102:0/1173306186 conn(0x7f8ac406af90 msgr2=0x7f8ac414da10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:28:09.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.129+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 shutdown_connections 2023-03-03T08:28:09.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:28:09.129+0000 7f8ac8dff700 1 -- 172.21.15.102:0/1173306186 wait complete. 2023-03-03T08:28:09.133 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:28:09.161 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-03-03T08:28:28.259 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:28.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:58.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:58.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:02.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:29:02.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.415+0000 7f4b3f59e700 1 Processor -- start 2023-03-03T08:29:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.415+0000 7f4b3f59e700 1 -- start start 2023-03-03T08:29:02.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3f59e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b400ced00 0x7f4b400c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:02.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3f59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4b400c41b0 0x7f4b400cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:02.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3f59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400c3d20 con 0x7f4b400b3970 2023-03-03T08:29:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3f59e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b400c4930 con 0x7f4b400ced00 2023-03-03T08:29:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3f59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b400cf610 con 0x7f4b400c41b0 2023-03-03T08:29:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b400ced00 0x7f4b400c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b400ced00 0x7f4b400c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47314/0 (socket says 172.21.15.102:47314) 2023-03-03T08:29:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4b400b3970 0x7f4b40158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40516/0 (socket says 172.21.15.102:40516) 2023-03-03T08:29:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3e59c700 1 -- 172.21.15.102:0/3910237346 learned_addr learned my addr 172.21.15.102:0/3910237346 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:29:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.417+0000 7f4b3ed9d700 1 --2- 172.21.15.102:0/3910237346 >> v2:172.21.15.172:3300/0 conn(0x7f4b400c41b0 0x7f4b400cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3e59c700 1 -- 172.21.15.102:0/3910237346 >> v2:172.21.15.172:3300/0 conn(0x7f4b400c41b0 msgr2=0x7f4b400cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:29:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3e59c700 1 --2- 172.21.15.102:0/3910237346 >> v2:172.21.15.172:3300/0 conn(0x7f4b400c41b0 0x7f4b400cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3e59c700 1 -- 172.21.15.102:0/3910237346 >> v1:172.21.15.102:6789/0 conn(0x7f4b400b3970 legacy=0x7f4b40158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3e59c700 1 -- 172.21.15.102:0/3910237346 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4015cba0 con 0x7f4b400ced00 2023-03-03T08:29:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3e59c700 1 --2- 172.21.15.102:0/3910237346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b400c5a90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4b400ca080 tx=0x7f4b30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9619bd3e933c74 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:29:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3d59a700 1 -- 172.21.15.102:0/3910237346 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30011830 con 0x7f4b400ced00 2023-03-03T08:29:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3d59a700 1 -- 172.21.15.102:0/3910237346 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b30011990 con 0x7f4b400ced00 2023-03-03T08:29:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.418+0000 7f4b3d59a700 1 -- 172.21.15.102:0/3910237346 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30011d20 con 0x7f4b400ced00 2023-03-03T08:29:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 -- 172.21.15.102:0/3910237346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 msgr2=0x7f4b400c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:02.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/3910237346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b400c5a90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4b400ca080 tx=0x7f4b30009c00 comp rx=0 tx=0).stop 2023-03-03T08:29:02.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 -- 172.21.15.102:0/3910237346 shutdown_connections 2023-03-03T08:29:02.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/3910237346 >> v2:172.21.15.172:3300/0 conn(0x7f4b400c41b0 0x7f4b400cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/3910237346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b400c5a90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 -- 172.21.15.102:0/3910237346 >> 172.21.15.102:0/3910237346 conn(0x7f4b4008b1c0 msgr2=0x7f4b40071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:29:02.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 -- 172.21.15.102:0/3910237346 shutdown_connections 2023-03-03T08:29:02.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.419+0000 7f4b3f59e700 1 -- 172.21.15.102:0/3910237346 wait complete. 2023-03-03T08:29:02.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.421+0000 7f4b3f59e700 1 Processor -- start 2023-03-03T08:29:02.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.421+0000 7f4b3f59e700 1 -- start start 2023-03-03T08:29:02.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.422+0000 7f4b3f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b400b3970 0x7f4b400b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:02.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.422+0000 7f4b3f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b400c41b0 0x7f4b401ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:02.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.422+0000 7f4b3f59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b401ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:02.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.422+0000 7f4b3f59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4015a020 con 0x7f4b400ced00 2023-03-03T08:29:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.422+0000 7f4b3f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b401f22c0 con 0x7f4b400b3970 2023-03-03T08:29:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.422+0000 7f4b3f59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b400cf1b0 con 0x7f4b400c41b0 2023-03-03T08:29:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b400b3970 0x7f4b400b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b400b3970 0x7f4b400b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34994/0 (socket says 172.21.15.102:34994) 2023-03-03T08:29:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b401ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3e59c700 1 -- 172.21.15.102:0/2897603535 learned_addr learned my addr 172.21.15.102:0/2897603535 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:29:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3dd9b700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b400c41b0 0x7f4b401ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3ed9d700 1 -- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b400c41b0 msgr2=0x7f4b401ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3ed9d700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b400c41b0 0x7f4b401ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3ed9d700 1 -- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b400b3970 msgr2=0x7f4b400b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3ed9d700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b400b3970 0x7f4b400b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3ed9d700 1 -- 172.21.15.102:0/2897603535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b30009550 con 0x7f4b400ced00 2023-03-03T08:29:02.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3e59c700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b400b3970 0x7f4b400b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:29:02.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.423+0000 7f4b3ed9d700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b401ee6e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800eba0 tx=0x7f4b3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:29:02.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.424+0000 7f4b2effd700 1 -- 172.21.15.102:0/2897603535 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38009a80 con 0x7f4b400ced00 2023-03-03T08:29:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.424+0000 7f4b2effd700 1 -- 172.21.15.102:0/2897603535 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3800f040 con 0x7f4b400ced00 2023-03-03T08:29:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.424+0000 7f4b2effd700 1 -- 172.21.15.102:0/2897603535 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3800cc40 con 0x7f4b400ced00 2023-03-03T08:29:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.424+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4006bf20 con 0x7f4b400ced00 2023-03-03T08:29:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.424+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4015ce80 con 0x7f4b400ced00 2023-03-03T08:29:02.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.425+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4006c200 con 0x7f4b400ced00 2023-03-03T08:29:02.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.426+0000 7f4b2effd700 1 -- 172.21.15.102:0/2897603535 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b38024070 con 0x7f4b400ced00 2023-03-03T08:29:02.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.428+0000 7f4b2effd700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b200682f0 0x7f4b2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:02.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.428+0000 7f4b2effd700 1 -- 172.21.15.102:0/2897603535 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4b38086750 con 0x7f4b400ced00 2023-03-03T08:29:02.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.429+0000 7f4b3e59c700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b200682f0 0x7f4b2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:02.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.429+0000 7f4b3e59c700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b200682f0 0x7f4b2006a7a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4b30009c40 tx=0x7f4b30008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:29:02.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.433+0000 7f4b2effd700 1 -- 172.21.15.102:0/2897603535 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b38052e00 con 0x7f4b400ced00 2023-03-03T08:29:02.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.665+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b401f2430 con 0x7f4b400ced00 2023-03-03T08:29:02.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.665+0000 7f4b2effd700 1 -- 172.21.15.102:0/2897603535 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4b38056420 con 0x7f4b400ced00 2023-03-03T08:29:02.668 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:29:02.668 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:29:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b200682f0 msgr2=0x7f4b2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b200682f0 0x7f4b2006a7a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4b30009c40 tx=0x7f4b30008030 comp rx=0 tx=0).stop 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 msgr2=0x7f4b401ee6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b401ee6e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800eba0 tx=0x7f4b3800eeb0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 shutdown_connections 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b400ced00 0x7f4b401ee6e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b400b3970 0x7f4b400b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b200682f0 0x7f4b2006a7a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 --2- 172.21.15.102:0/2897603535 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b400c41b0 0x7f4b401ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 >> 172.21.15.102:0/2897603535 conn(0x7f4b4008b1c0 msgr2=0x7f4b400669c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:29:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 shutdown_connections 2023-03-03T08:29:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:02.668+0000 7f4b3f59e700 1 -- 172.21.15.102:0/2897603535 wait complete. 2023-03-03T08:29:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:29:02.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-03-03T08:29:28.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:28.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:44.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:29:44.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.954+0000 7fc31f59e700 1 Processor -- start 2023-03-03T08:29:44.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.954+0000 7fc31f59e700 1 -- start start 2023-03-03T08:29:44.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc31f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3201564b0 0x7fc3200cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:44.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc31f59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3200c9ca0 con 0x7fc3200c9610 2023-03-03T08:29:44.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc31e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3201564b0 0x7fc3200cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:44.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc31f59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3200b46f0 con 0x7fc3200cab50 2023-03-03T08:29:44.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc31f59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200b4920 con 0x7fc3201564b0 2023-03-03T08:29:44.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc317fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc3200c9610 0x7fc3200ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47088/0 (socket says 172.21.15.102:47088) 2023-03-03T08:29:44.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc317fff700 1 -- 172.21.15.102:0/3168861872 learned_addr learned my addr 172.21.15.102:0/3168861872 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:29:44.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.956+0000 7fc31ed9d700 1 --1- 172.21.15.102:0/3168861872 >> v1:172.21.15.172:6789/0 conn(0x7fc3200cab50 0x7fc3201636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54572/0 (socket says 172.21.15.102:54572) 2023-03-03T08:29:44.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.957+0000 7fc31dd9b700 1 -- 172.21.15.102:0/3168861872 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692954227 0 0) 0x7fc3200b46f0 con 0x7fc3200cab50 2023-03-03T08:29:44.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.957+0000 7fc31dd9b700 1 -- 172.21.15.102:0/3168861872 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc30c0035b0 con 0x7fc3200cab50 2023-03-03T08:29:44.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.957+0000 7fc31e59c700 1 -- 172.21.15.102:0/3168861872 >> v1:172.21.15.172:6789/0 conn(0x7fc3200cab50 legacy=0x7fc3201636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:44.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.957+0000 7fc31e59c700 1 -- 172.21.15.102:0/3168861872 >> v1:172.21.15.102:6789/0 conn(0x7fc3200c9610 legacy=0x7fc3200ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:44.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.957+0000 7fc31e59c700 1 -- 172.21.15.102:0/3168861872 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3200b4c00 con 0x7fc3201564b0 2023-03-03T08:29:44.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.957+0000 7fc31e59c700 1 --2- 172.21.15.102:0/3168861872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3201564b0 0x7fc3200cb660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc310009870 tx=0x7fc310009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=198203df6790d8ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:29:44.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31dd9b700 1 -- 172.21.15.102:0/3168861872 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc310004230 con 0x7fc3201564b0 2023-03-03T08:29:44.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31dd9b700 1 -- 172.21.15.102:0/3168861872 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc310004390 con 0x7fc3201564b0 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31dd9b700 1 -- 172.21.15.102:0/3168861872 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100036a0 con 0x7fc3201564b0 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31f59e700 1 -- 172.21.15.102:0/3168861872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3201564b0 msgr2=0x7fc3200cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31f59e700 1 --2- 172.21.15.102:0/3168861872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3201564b0 0x7fc3200cb660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc310009870 tx=0x7fc310009b80 comp rx=0 tx=0).stop 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31f59e700 1 -- 172.21.15.102:0/3168861872 shutdown_connections 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31f59e700 1 --2- 172.21.15.102:0/3168861872 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3201564b0 0x7fc3200cb660 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.958+0000 7fc31f59e700 1 -- 172.21.15.102:0/3168861872 >> 172.21.15.102:0/3168861872 conn(0x7fc320065fb0 msgr2=0x7fc3200663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.959+0000 7fc31f59e700 1 -- 172.21.15.102:0/3168861872 shutdown_connections 2023-03-03T08:29:44.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.959+0000 7fc31f59e700 1 -- 172.21.15.102:0/3168861872 wait complete. 2023-03-03T08:29:44.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.961+0000 7fc31f59e700 1 Processor -- start 2023-03-03T08:29:44.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.961+0000 7fc31f59e700 1 -- start start 2023-03-03T08:29:44.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3200c9610 0x7fc3201ef390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:44.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3200cab50 0x7fc3200c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:44.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31f59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3201564b0 0x7fc320159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:44.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31f59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32014e800 con 0x7fc3201564b0 2023-03-03T08:29:44.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc320155bd0 con 0x7fc3200c9610 2023-03-03T08:29:44.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31f59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32014ec20 con 0x7fc3200cab50 2023-03-03T08:29:44.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc317fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3200cab50 0x7fc3200c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:44.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3201564b0 0x7fc320159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:44.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3200c9610 0x7fc3201ef390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:44.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3201564b0 0x7fc320159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41092/0 (socket says 172.21.15.102:41092) 2023-03-03T08:29:44.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.962+0000 7fc31ed9d700 1 -- 172.21.15.102:0/2685394402 learned_addr learned my addr 172.21.15.102:0/2685394402 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:29:44.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc317fff700 1 -- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3200c9610 msgr2=0x7fc3201ef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:44.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc317fff700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3200c9610 0x7fc3201ef390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:44.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc317fff700 1 -- 172.21.15.102:0/2685394402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3201564b0 msgr2=0x7fc320159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:44.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc317fff700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3201564b0 0x7fc320159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:44.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc317fff700 1 -- 172.21.15.102:0/2685394402 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3100094a0 con 0x7fc3200cab50 2023-03-03T08:29:44.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc31ed9d700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3201564b0 0x7fc320159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:29:44.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc317fff700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3200cab50 0x7fc3200c71f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc30800c780 tx=0x7fc30800ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:29:44.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.963+0000 7fc316ffd700 1 -- 172.21.15.102:0/2685394402 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc308004c80 con 0x7fc3200cab50 2023-03-03T08:29:44.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.964+0000 7fc316ffd700 1 -- 172.21.15.102:0/2685394402 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc308004de0 con 0x7fc3200cab50 2023-03-03T08:29:44.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.964+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3200b2210 con 0x7fc3200cab50 2023-03-03T08:29:44.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.964+0000 7fc316ffd700 1 -- 172.21.15.102:0/2685394402 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc308007310 con 0x7fc3200cab50 2023-03-03T08:29:44.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.964+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3200b1f30 con 0x7fc3200cab50 2023-03-03T08:29:44.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.965+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc320156880 con 0x7fc3200cab50 2023-03-03T08:29:44.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.966+0000 7fc316ffd700 1 -- 172.21.15.102:0/2685394402 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc30801a020 con 0x7fc3200cab50 2023-03-03T08:29:44.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.967+0000 7fc316ffd700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3000682f0 0x7fc30006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:29:44.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.967+0000 7fc316ffd700 1 -- 172.21.15.102:0/2685394402 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc3080859a0 con 0x7fc3200cab50 2023-03-03T08:29:44.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.968+0000 7fc31e59c700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3000682f0 0x7fc30006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:29:44.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.968+0000 7fc31e59c700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3000682f0 0x7fc30006a7a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc3100090b0 tx=0x7fc310008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:29:44.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:44.971+0000 7fc316ffd700 1 -- 172.21.15.102:0/2685394402 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3080520b0 con 0x7fc3200cab50 2023-03-03T08:29:45.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.204+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc32014e940 con 0x7fc3200cab50 2023-03-03T08:29:45.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.204+0000 7fc316ffd700 1 -- 172.21.15.102:0/2685394402 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc32014e940 con 0x7fc3200cab50 2023-03-03T08:29:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:29:45.207 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:29:45.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3000682f0 msgr2=0x7fc30006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3000682f0 0x7fc30006a7a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc3100090b0 tx=0x7fc310008030 comp rx=0 tx=0).stop 2023-03-03T08:29:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3200cab50 msgr2=0x7fc3200c71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:29:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3200cab50 0x7fc3200c71f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc30800c780 tx=0x7fc30800ca90 comp rx=0 tx=0).stop 2023-03-03T08:29:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 shutdown_connections 2023-03-03T08:29:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3201564b0 0x7fc320159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3200c9610 0x7fc3201ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3000682f0 0x7fc30006a7a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:45.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 --2- 172.21.15.102:0/2685394402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3200cab50 0x7fc3200c71f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:29:45.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 >> 172.21.15.102:0/2685394402 conn(0x7fc320065fb0 msgr2=0x7fc3200cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:29:45.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 shutdown_connections 2023-03-03T08:29:45.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:29:45.207+0000 7fc31f59e700 1 -- 172.21.15.102:0/2685394402 wait complete. 2023-03-03T08:29:45.211 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:29:45.236 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 101 secs before thrashing 2023-03-03T08:29:58.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:58.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T08:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:30:29.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:29.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:59.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:59.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:26.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:31:26.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.484+0000 7f8cc099d700 1 Processor -- start 2023-03-03T08:31:26.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.484+0000 7f8cc099d700 1 -- start start 2023-03-03T08:31:26.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.486+0000 7f8cc099d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0c5a90 0x7f8cbc0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:26.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.486+0000 7f8cc099d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0c3d20 con 0x7f8cbc0ced00 2023-03-03T08:31:26.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.486+0000 7f8cc099d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc15cba0 con 0x7f8cbc0b3970 2023-03-03T08:31:26.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.486+0000 7f8cc099d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc0c4930 con 0x7f8cbc0c5a90 2023-03-03T08:31:26.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.486+0000 7f8cbb7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8cbc0ced00 0x7f8cbc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55000/0 (socket says 172.21.15.102:55000) 2023-03-03T08:31:26.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.486+0000 7f8cbb7fe700 1 -- 172.21.15.102:0/1433932398 learned_addr learned my addr 172.21.15.102:0/1433932398 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:31:26.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.486+0000 7f8cbbfff700 1 --2- 172.21.15.102:0/1433932398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0c5a90 0x7f8cbc0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cbbfff700 1 -- 172.21.15.102:0/1433932398 >> v1:172.21.15.172:6789/0 conn(0x7f8cbc0b3970 legacy=0x7f8cbc161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cba7fc700 1 -- 172.21.15.102:0/1433932398 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182370722 0 0) 0x7f8cbc0c3d20 con 0x7f8cbc0ced00 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cbbfff700 1 -- 172.21.15.102:0/1433932398 >> v1:172.21.15.102:6789/0 conn(0x7f8cbc0ced00 legacy=0x7f8cbc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cbbfff700 1 -- 172.21.15.102:0/1433932398 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc15ce80 con 0x7f8cbc0c5a90 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cbbfff700 1 --2- 172.21.15.102:0/1433932398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0c5a90 0x7f8cbc0c41b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009dc0 tx=0x7f8cb400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d10955423373075 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cba7fc700 1 -- 172.21.15.102:0/1433932398 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400e070 con 0x7f8cbc0c5a90 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cba7fc700 1 -- 172.21.15.102:0/1433932398 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb400bad0 con 0x7f8cbc0c5a90 2023-03-03T08:31:26.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.487+0000 7f8cba7fc700 1 -- 172.21.15.102:0/1433932398 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400be80 con 0x7f8cbc0c5a90 2023-03-03T08:31:26.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.488+0000 7f8cc099d700 1 -- 172.21.15.102:0/1433932398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0c5a90 msgr2=0x7f8cbc0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:26.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.488+0000 7f8cc099d700 1 --2- 172.21.15.102:0/1433932398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0c5a90 0x7f8cbc0c41b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009dc0 tx=0x7f8cb400b330 comp rx=0 tx=0).stop 2023-03-03T08:31:26.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.488+0000 7f8cc099d700 1 -- 172.21.15.102:0/1433932398 shutdown_connections 2023-03-03T08:31:26.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.488+0000 7f8cc099d700 1 --2- 172.21.15.102:0/1433932398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0c5a90 0x7f8cbc0c41b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:26.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.488+0000 7f8cc099d700 1 -- 172.21.15.102:0/1433932398 >> 172.21.15.102:0/1433932398 conn(0x7f8cbc08b1c0 msgr2=0x7f8cbc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:31:26.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.488+0000 7f8cc099d700 1 -- 172.21.15.102:0/1433932398 shutdown_connections 2023-03-03T08:31:26.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.488+0000 7f8cc099d700 1 -- 172.21.15.102:0/1433932398 wait complete. 2023-03-03T08:31:26.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.490+0000 7f8cc099d700 1 Processor -- start 2023-03-03T08:31:26.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.490+0000 7f8cc099d700 1 -- start start 2023-03-03T08:31:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cc099d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cbc0b3970 0x7f8cbc15bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cc099d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0c5a90 0x7f8cbc0ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cc099d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0ced00 0x7f8cbc1f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cc099d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc14d9e0 con 0x7f8cbc0b3970 2023-03-03T08:31:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cc099d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc14e2e0 con 0x7f8cbc0c5a90 2023-03-03T08:31:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cc099d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc1e5230 con 0x7f8cbc0ced00 2023-03-03T08:31:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cbb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cbc0b3970 0x7f8cbc15bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cbb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cbc0b3970 0x7f8cbc15bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51516/0 (socket says 172.21.15.102:51516) 2023-03-03T08:31:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.491+0000 7f8cbb7fe700 1 -- 172.21.15.102:0/2282892561 learned_addr learned my addr 172.21.15.102:0/2282892561 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:31:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbbfff700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0ced00 0x7f8cbc1f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbaffd700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0c5a90 0x7f8cbc0ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:26.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbb7fe700 1 -- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0ced00 msgr2=0x7f8cbc1f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:26.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbb7fe700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0ced00 0x7f8cbc1f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:26.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbb7fe700 1 -- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0c5a90 msgr2=0x7f8cbc0ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:26.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbb7fe700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0c5a90 0x7f8cbc0ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:26.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbb7fe700 1 -- 172.21.15.102:0/2282892561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb4009a70 con 0x7f8cbc0b3970 2023-03-03T08:31:26.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbbfff700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0ced00 0x7f8cbc1f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:31:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.492+0000 7f8cbb7fe700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cbc0b3970 0x7f8cbc15bef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c580 tx=0x7f8cac00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:31:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.493+0000 7f8ca3fff700 1 -- 172.21.15.102:0/2282892561 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac004c80 con 0x7f8cbc0b3970 2023-03-03T08:31:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.493+0000 7f8ca3fff700 1 -- 172.21.15.102:0/2282892561 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac004e00 con 0x7f8cbc0b3970 2023-03-03T08:31:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.493+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc06c200 con 0x7f8cbc0b3970 2023-03-03T08:31:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.493+0000 7f8ca3fff700 1 -- 172.21.15.102:0/2282892561 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac007310 con 0x7f8cbc0b3970 2023-03-03T08:31:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.493+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc06bf20 con 0x7f8cbc0b3970 2023-03-03T08:31:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.494+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc15c890 con 0x7f8cbc0b3970 2023-03-03T08:31:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.495+0000 7f8ca3fff700 1 -- 172.21.15.102:0/2282892561 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8cac01a020 con 0x7f8cbc0b3970 2023-03-03T08:31:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.496+0000 7f8ca3fff700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ca80682f0 0x7f8ca806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.496+0000 7f8ca3fff700 1 -- 172.21.15.102:0/2282892561 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8cac0858a0 con 0x7f8cbc0b3970 2023-03-03T08:31:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.497+0000 7f8cbaffd700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ca80682f0 0x7f8ca806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.497+0000 7f8cbaffd700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ca80682f0 0x7f8ca806a7a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00012e0 tx=0x7f8cb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:31:26.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.501+0000 7f8ca3fff700 1 -- 172.21.15.102:0/2282892561 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8cac051f20 con 0x7f8cbc0b3970 2023-03-03T08:31:26.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.737+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cbc04e6e0 con 0x7f8cbc0b3970 2023-03-03T08:31:26.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.737+0000 7f8ca3fff700 1 -- 172.21.15.102:0/2282892561 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8cac055540 con 0x7f8cbc0b3970 2023-03-03T08:31:26.740 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:31:26.740 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:31:26.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ca80682f0 msgr2=0x7f8ca806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:26.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ca80682f0 0x7f8ca806a7a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00012e0 tx=0x7f8cb0009040 comp rx=0 tx=0).stop 2023-03-03T08:31:26.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cbc0b3970 msgr2=0x7f8cbc15bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:26.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cbc0b3970 0x7f8cbc15bef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c580 tx=0x7f8cac00c890 comp rx=0 tx=0).stop 2023-03-03T08:31:26.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 shutdown_connections 2023-03-03T08:31:26.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ca80682f0 0x7f8ca806a7a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:26.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8cbc0ced00 0x7f8cbc1f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:26.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cbc0b3970 0x7f8cbc15bef0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:26.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 --2- 172.21.15.102:0/2282892561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0c5a90 0x7f8cbc0ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:26.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 >> 172.21.15.102:0/2282892561 conn(0x7f8cbc08b1c0 msgr2=0x7f8cbc06b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:31:26.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 shutdown_connections 2023-03-03T08:31:26.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:26.740+0000 7f8cc099d700 1 -- 172.21.15.102:0/2282892561 wait complete. 2023-03-03T08:31:26.744 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:31:26.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:31:27.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.010+0000 7fc0931ff700 1 Processor -- start 2023-03-03T08:31:27.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.010+0000 7fc0931ff700 1 -- start start 2023-03-03T08:31:27.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.011+0000 7fc0931ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c153260 0x7fc08c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:27.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc0931ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc08c0c9cb0 0x7fc08c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:27.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc0931ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0cf940 con 0x7fc08c0cac20 2023-03-03T08:31:27.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc0931ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c0cfe70 con 0x7fc08c0c9cb0 2023-03-03T08:31:27.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc0931ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c158d70 con 0x7fc08c153260 2023-03-03T08:31:27.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc08bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc08c0c9cb0 0x7fc08c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc08bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc08c0c9cb0 0x7fc08c0cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50874/0 (socket says 172.21.15.102:50874) 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc08bfff700 1 -- 172.21.15.102:0/1627930608 learned_addr learned my addr 172.21.15.102:0/1627930608 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc090f9b700 1 --2- 172.21.15.102:0/1627930608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c153260 0x7fc08c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc08bfff700 1 -- 172.21.15.102:0/1627930608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c153260 msgr2=0x7fc08c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc08bfff700 1 --2- 172.21.15.102:0/1627930608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c153260 0x7fc08c0ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.012+0000 7fc08bfff700 1 -- 172.21.15.102:0/1627930608 >> v1:172.21.15.172:6789/0 conn(0x7fc08c0cac20 legacy=0x7fc08c15c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.013+0000 7fc08bfff700 1 -- 172.21.15.102:0/1627930608 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c0cfb90 con 0x7fc08c0c9cb0 2023-03-03T08:31:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.013+0000 7fc08b7fe700 1 -- 172.21.15.102:0/1627930608 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122124568 0 0) 0x7fc08c0cf940 con 0x7fc08c0cac20 2023-03-03T08:31:27.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.013+0000 7fc08bfff700 1 --2- 172.21.15.102:0/1627930608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c0c9cb0 0x7fc08c0cb880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc0780098f0 tx=0x7fc078009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23297b0063f2cc8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:31:27.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.013+0000 7fc08b7fe700 1 -- 172.21.15.102:0/1627930608 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078011830 con 0x7fc08c0c9cb0 2023-03-03T08:31:27.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.013+0000 7fc08b7fe700 1 -- 172.21.15.102:0/1627930608 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc078011990 con 0x7fc08c0c9cb0 2023-03-03T08:31:27.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.013+0000 7fc08b7fe700 1 -- 172.21.15.102:0/1627930608 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078011d20 con 0x7fc08c0c9cb0 2023-03-03T08:31:27.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 -- 172.21.15.102:0/1627930608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c0c9cb0 msgr2=0x7fc08c0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:27.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 --2- 172.21.15.102:0/1627930608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c0c9cb0 0x7fc08c0cb880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc0780098f0 tx=0x7fc078009c00 comp rx=0 tx=0).stop 2023-03-03T08:31:27.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 -- 172.21.15.102:0/1627930608 shutdown_connections 2023-03-03T08:31:27.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 --2- 172.21.15.102:0/1627930608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c153260 0x7fc08c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 --2- 172.21.15.102:0/1627930608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c0c9cb0 0x7fc08c0cb880 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 -- 172.21.15.102:0/1627930608 >> 172.21.15.102:0/1627930608 conn(0x7fc08c066c00 msgr2=0x7fc08c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:31:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 -- 172.21.15.102:0/1627930608 shutdown_connections 2023-03-03T08:31:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.014+0000 7fc0931ff700 1 -- 172.21.15.102:0/1627930608 wait complete. 2023-03-03T08:31:27.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.017+0000 7fc0931ff700 1 Processor -- start 2023-03-03T08:31:27.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.017+0000 7fc0931ff700 1 -- start start 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.018+0000 7fc0931ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc08c0cac20 0x7fc08c1557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.018+0000 7fc0931ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c153260 0x7fc08c0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.018+0000 7fc0931ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c0c63f0 0x7fc08c153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.018+0000 7fc0931ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0c7f50 con 0x7fc08c153260 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.018+0000 7fc0931ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0c8fe0 con 0x7fc08c0cac20 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.018+0000 7fc0931ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c153af0 con 0x7fc08c0c63f0 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc08c0cac20 0x7fc08c1557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:27.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc08c0cac20 0x7fc08c1557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38150/0 (socket says 172.21.15.102:38150) 2023-03-03T08:31:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 -- 172.21.15.102:0/2223428135 learned_addr learned my addr 172.21.15.102:0/2223428135 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:31:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc08bfff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c153260 0x7fc08c0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc09179c700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c0c63f0 0x7fc08c153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 -- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c0c63f0 msgr2=0x7fc08c153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c0c63f0 0x7fc08c153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 -- 172.21.15.102:0/2223428135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c153260 msgr2=0x7fc08c0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c153260 0x7fc08c0c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 -- 172.21.15.102:0/2223428135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc078009550 con 0x7fc08c0cac20 2023-03-03T08:31:27.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc08bfff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c153260 0x7fc08c0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:31:27.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.019+0000 7fc090f9b700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc08c0cac20 0x7fc08c1557e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc08000e9e0 tx=0x7fc08000ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:31:27.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.020+0000 7fc0897fa700 1 -- 172.21.15.102:0/2223428135 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080009710 con 0x7fc08c0cac20 2023-03-03T08:31:27.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.020+0000 7fc0897fa700 1 -- 172.21.15.102:0/2223428135 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08000c7f0 con 0x7fc08c0cac20 2023-03-03T08:31:27.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.020+0000 7fc0897fa700 1 -- 172.21.15.102:0/2223428135 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000cb60 con 0x7fc08c0cac20 2023-03-03T08:31:27.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.020+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c159260 con 0x7fc08c0cac20 2023-03-03T08:31:27.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.020+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c158f80 con 0x7fc08c0cac20 2023-03-03T08:31:27.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.021+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c159540 con 0x7fc08c0cac20 2023-03-03T08:31:27.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.022+0000 7fc0897fa700 1 -- 172.21.15.102:0/2223428135 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc080023070 con 0x7fc08c0cac20 2023-03-03T08:31:27.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.024+0000 7fc0897fa700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc070068320 0x7fc07006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:31:27.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.024+0000 7fc0897fa700 1 -- 172.21.15.102:0/2223428135 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc080085f70 con 0x7fc08c0cac20 2023-03-03T08:31:27.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.024+0000 7fc08bfff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc070068320 0x7fc07006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:31:27.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.025+0000 7fc08bfff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc070068320 0x7fc07006a7d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc078014040 tx=0x7fc078008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:31:27.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.028+0000 7fc0897fa700 1 -- 172.21.15.102:0/2223428135 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc080052680 con 0x7fc08c0cac20 2023-03-03T08:31:27.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.265+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc08c04e6e0 con 0x7fc08c0cac20 2023-03-03T08:31:27.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.267+0000 7fc0897fa700 1 -- 172.21.15.102:0/2223428135 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc080055ca0 con 0x7fc08c0cac20 2023-03-03T08:31:27.268 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:31:27.268 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:31:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.268+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc070068320 msgr2=0x7fc07006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.268+0000 7fc0931ff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc070068320 0x7fc07006a7d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc078014040 tx=0x7fc078008030 comp rx=0 tx=0).stop 2023-03-03T08:31:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.268+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc08c0cac20 msgr2=0x7fc08c1557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:31:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.268+0000 7fc0931ff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc08c0cac20 0x7fc08c1557e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc08000e9e0 tx=0x7fc08000ecf0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 shutdown_connections 2023-03-03T08:31:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc070068320 0x7fc07006a7d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc08c0c63f0 0x7fc08c153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc08c0cac20 0x7fc08c1557e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 --2- 172.21.15.102:0/2223428135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc08c153260 0x7fc08c0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:31:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 >> 172.21.15.102:0/2223428135 conn(0x7fc08c0c9cb0 msgr2=0x7fc08c1580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:31:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 shutdown_connections 2023-03-03T08:31:27.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:31:27.269+0000 7fc0931ff700 1 -- 172.21.15.102:0/2223428135 wait complete. 2023-03-03T08:31:27.273 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:31:27.297 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2023-03-03T08:31:29.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:29.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:59.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:59.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:29.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:29.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:31.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:32:31.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.547+0000 7fd37d19d700 1 Processor -- start 2023-03-03T08:32:31.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.547+0000 7fd37d19d700 1 -- start start 2023-03-03T08:32:31.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd37d19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3780b3970 0x7fd3780c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:32:31.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd37d19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780c41b0 0x7fd3780cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:32:31.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd37d19d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3780c3d20 con 0x7fd3780ced00 2023-03-03T08:32:31.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd37d19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780c4930 con 0x7fd3780b3970 2023-03-03T08:32:31.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd37d19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780cf610 con 0x7fd3780c41b0 2023-03-03T08:32:31.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd377fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd3780ced00 0x7fd378158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37558/0 (socket says 172.21.15.102:37558) 2023-03-03T08:32:31.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd377fff700 1 -- 172.21.15.102:0/2274153436 learned_addr learned my addr 172.21.15.102:0/2274153436 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:32:31.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd3777fe700 1 --2- 172.21.15.102:0/2274153436 >> v2:172.21.15.172:3300/0 conn(0x7fd3780b3970 0x7fd3780c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:32:31.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.549+0000 7fd37c99c700 1 --2- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780c41b0 0x7fd3780cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:32:31.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd3777fe700 1 -- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780c41b0 msgr2=0x7fd3780cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:32:31.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd3777fe700 1 --2- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780c41b0 0x7fd3780cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd3777fe700 1 -- 172.21.15.102:0/2274153436 >> v1:172.21.15.102:6789/0 conn(0x7fd3780ced00 legacy=0x7fd378158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:32:31.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd3777fe700 1 -- 172.21.15.102:0/2274153436 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37815cba0 con 0x7fd3780b3970 2023-03-03T08:32:31.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd376ffd700 1 -- 172.21.15.102:0/2274153436 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204783770 0 0) 0x7fd3780c3d20 con 0x7fd3780ced00 2023-03-03T08:32:31.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd3777fe700 1 --2- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780b3970 0x7fd3780c5a90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009870 tx=0x7fd36c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1bb3e18f0ea25d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:32:31.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd376ffd700 1 -- 172.21.15.102:0/2274153436 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c011830 con 0x7fd3780b3970 2023-03-03T08:32:31.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd376ffd700 1 -- 172.21.15.102:0/2274153436 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c011990 con 0x7fd3780b3970 2023-03-03T08:32:31.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.550+0000 7fd376ffd700 1 -- 172.21.15.102:0/2274153436 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c011d20 con 0x7fd3780b3970 2023-03-03T08:32:31.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 -- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780b3970 msgr2=0x7fd3780c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:32:31.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 --2- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780b3970 0x7fd3780c5a90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009870 tx=0x7fd36c009b80 comp rx=0 tx=0).stop 2023-03-03T08:32:31.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 -- 172.21.15.102:0/2274153436 shutdown_connections 2023-03-03T08:32:31.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 --2- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780c41b0 0x7fd3780cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 --2- 172.21.15.102:0/2274153436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780b3970 0x7fd3780c5a90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 -- 172.21.15.102:0/2274153436 >> 172.21.15.102:0/2274153436 conn(0x7fd37808b1c0 msgr2=0x7fd378071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:32:31.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 -- 172.21.15.102:0/2274153436 shutdown_connections 2023-03-03T08:32:31.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.551+0000 7fd37d19d700 1 -- 172.21.15.102:0/2274153436 wait complete. 2023-03-03T08:32:31.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.553+0000 7fd37d19d700 1 Processor -- start 2023-03-03T08:32:31.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.553+0000 7fd37d19d700 1 -- start start 2023-03-03T08:32:31.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.554+0000 7fd37d19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780b3970 0x7fd3780c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:32:31.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.554+0000 7fd37d19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780c41b0 0x7fd3780c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:32:31.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.554+0000 7fd37d19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ced00 0x7fd37815b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:32:31.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.554+0000 7fd37d19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780c7270 con 0x7fd3780ced00 2023-03-03T08:32:31.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.554+0000 7fd37d19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781e2fc0 con 0x7fd3780c41b0 2023-03-03T08:32:31.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.554+0000 7fd37d19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781e2ce0 con 0x7fd3780b3970 2023-03-03T08:32:31.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780c41b0 0x7fd3780c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:32:31.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd377fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780b3970 0x7fd3780c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:32:31.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780c41b0 0x7fd3780c6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:58588/0 (socket says 172.21.15.102:58588) 2023-03-03T08:32:31.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 -- 172.21.15.102:0/994082877 learned_addr learned my addr 172.21.15.102:0/994082877 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:32:31.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd37c99c700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ced00 0x7fd37815b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:32:31.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 -- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780b3970 msgr2=0x7fd3780c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780b3970 0x7fd3780c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 -- 172.21.15.102:0/994082877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ced00 msgr2=0x7fd37815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ced00 0x7fd37815b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 -- 172.21.15.102:0/994082877 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c0094a0 con 0x7fd3780c41b0 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd377fff700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780b3970 0x7fd3780c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd37c99c700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ced00 0x7fd37815b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.555+0000 7fd3777fe700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780c41b0 0x7fd3780c6820 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd36c014040 tx=0x7fd36c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:32:31.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.556+0000 7fd374ff9700 1 -- 172.21.15.102:0/994082877 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c0036a0 con 0x7fd3780c41b0 2023-03-03T08:32:31.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.556+0000 7fd374ff9700 1 -- 172.21.15.102:0/994082877 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c01e070 con 0x7fd3780c41b0 2023-03-03T08:32:31.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.556+0000 7fd374ff9700 1 -- 172.21.15.102:0/994082877 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c003a00 con 0x7fd3780c41b0 2023-03-03T08:32:31.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.556+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37806bf20 con 0x7fd3780c41b0 2023-03-03T08:32:31.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.556+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37815ce80 con 0x7fd3780c41b0 2023-03-03T08:32:31.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.557+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37806c200 con 0x7fd3780c41b0 2023-03-03T08:32:31.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.558+0000 7fd374ff9700 1 -- 172.21.15.102:0/994082877 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd36c00f050 con 0x7fd3780c41b0 2023-03-03T08:32:31.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.559+0000 7fd374ff9700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3640682f0 0x7fd36406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:32:31.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.559+0000 7fd374ff9700 1 -- 172.21.15.102:0/994082877 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd36c086cd0 con 0x7fd3780c41b0 2023-03-03T08:32:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.560+0000 7fd377fff700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3640682f0 0x7fd36406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:32:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.560+0000 7fd377fff700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3640682f0 0x7fd36406a7a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd368007fd0 tx=0x7fd368007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:32:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.564+0000 7fd374ff9700 1 -- 172.21.15.102:0/994082877 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd36c0533e0 con 0x7fd3780c41b0 2023-03-03T08:32:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.797+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd37804e6e0 con 0x7fd3780c41b0 2023-03-03T08:32:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.798+0000 7fd374ff9700 1 -- 172.21.15.102:0/994082877 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd36c020090 con 0x7fd3780c41b0 2023-03-03T08:32:31.799 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:32:31.799 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:32:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.800+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3640682f0 msgr2=0x7fd36406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:32:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.800+0000 7fd37d19d700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3640682f0 0x7fd36406a7a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd368007fd0 tx=0x7fd368007de0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780c41b0 msgr2=0x7fd3780c6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:32:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780c41b0 0x7fd3780c6820 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd36c014040 tx=0x7fd36c004750 comp rx=0 tx=0).stop 2023-03-03T08:32:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 shutdown_connections 2023-03-03T08:32:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd3640682f0 0x7fd36406a7a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd3780b3970 0x7fd3780c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3780ced00 0x7fd37815b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 --2- 172.21.15.102:0/994082877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3780c41b0 0x7fd3780c6820 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:32:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 >> 172.21.15.102:0/994082877 conn(0x7fd37808b1c0 msgr2=0x7fd3781520b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:32:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 shutdown_connections 2023-03-03T08:32:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:32:31.801+0000 7fd37d19d700 1 -- 172.21.15.102:0/994082877 wait complete. 2023-03-03T08:32:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:32:31.829 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-03-03T08:33:00.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:00.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:18.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:33:19.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.099+0000 7f42b3fff700 1 Processor -- start 2023-03-03T08:33:19.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.099+0000 7f42b3fff700 1 -- start start 2023-03-03T08:33:19.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f42b40ced00 0x7f42b40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:19.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42b40b3970 0x7f42b40c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40cf0d0 0x7f42b4158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b3fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b40c4930 con 0x7f42b40b3970 2023-03-03T08:33:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b40c3d20 con 0x7f42b40ced00 2023-03-03T08:33:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40cf570 con 0x7f42b40cf0d0 2023-03-03T08:33:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b27fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42b40b3970 0x7f42b40c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:19.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f42b40ced00 0x7f42b40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:19.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40cf0d0 0x7f42b4158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:19.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b27fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42b40b3970 0x7f42b40c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38708/0 (socket says 172.21.15.102:38708) 2023-03-03T08:33:19.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b27fc700 1 -- 172.21.15.102:0/372678411 learned_addr learned my addr 172.21.15.102:0/372678411 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:33:19.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.101+0000 7f42b2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f42b40ced00 0x7f42b40c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39370/0 (socket says 172.21.15.102:39370) 2023-03-03T08:33:19.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b2ffd700 1 -- 172.21.15.102:0/372678411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40cf0d0 msgr2=0x7f42b4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:33:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b2ffd700 1 --2- 172.21.15.102:0/372678411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40cf0d0 0x7f42b4158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b2ffd700 1 -- 172.21.15.102:0/372678411 >> v2:172.21.15.102:3300/0 conn(0x7f42b40b3970 msgr2=0x7f42b40c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:33:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b2ffd700 1 --2- 172.21.15.102:0/372678411 >> v2:172.21.15.102:3300/0 conn(0x7f42b40b3970 0x7f42b40c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b2ffd700 1 -- 172.21.15.102:0/372678411 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b415cba0 con 0x7f42b40ced00 2023-03-03T08:33:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b27fc700 1 --2- 172.21.15.102:0/372678411 >> v2:172.21.15.102:3300/0 conn(0x7f42b40b3970 0x7f42b40c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:33:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b2ffd700 1 --2- 172.21.15.102:0/372678411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b40c5a90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f42b40b1b40 tx=0x7f42a4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fef7e3ee8d67646 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b1ffb700 1 -- 172.21.15.102:0/372678411 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4011830 con 0x7f42b40ced00 2023-03-03T08:33:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b1ffb700 1 -- 172.21.15.102:0/372678411 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a4011990 con 0x7f42b40ced00 2023-03-03T08:33:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.102+0000 7f42b1ffb700 1 -- 172.21.15.102:0/372678411 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4011d20 con 0x7f42b40ced00 2023-03-03T08:33:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 -- 172.21.15.102:0/372678411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 msgr2=0x7f42b40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 --2- 172.21.15.102:0/372678411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b40c5a90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f42b40b1b40 tx=0x7f42a4009b20 comp rx=0 tx=0).stop 2023-03-03T08:33:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 -- 172.21.15.102:0/372678411 shutdown_connections 2023-03-03T08:33:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 --2- 172.21.15.102:0/372678411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40cf0d0 0x7f42b4158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 --2- 172.21.15.102:0/372678411 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b40c5a90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 --2- 172.21.15.102:0/372678411 >> v2:172.21.15.102:3300/0 conn(0x7f42b40b3970 0x7f42b40c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 -- 172.21.15.102:0/372678411 >> 172.21.15.102:0/372678411 conn(0x7f42b408b1c0 msgr2=0x7f42b4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:33:19.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 -- 172.21.15.102:0/372678411 shutdown_connections 2023-03-03T08:33:19.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.103+0000 7f42b3fff700 1 -- 172.21.15.102:0/372678411 wait complete. 2023-03-03T08:33:19.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.105+0000 7f42b3fff700 1 Processor -- start 2023-03-03T08:33:19.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.105+0000 7f42b3fff700 1 -- start start 2023-03-03T08:33:19.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.106+0000 7f42b3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40b3970 0x7f42b4150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:19.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.106+0000 7f42b3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b4147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:19.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42b40cf0d0 0x7f42b4148500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:19.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b415b5a0 con 0x7f42b40cf0d0 2023-03-03T08:33:19.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b414c9f0 con 0x7f42b40ced00 2023-03-03T08:33:19.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b414c430 con 0x7f42b40b3970 2023-03-03T08:33:19.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40b3970 0x7f42b4150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:19.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b27fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b4147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:19.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42b40cf0d0 0x7f42b4148500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:19.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40b3970 0x7f42b4150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46398/0 (socket says 172.21.15.102:46398) 2023-03-03T08:33:19.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 -- 172.21.15.102:0/135683813 learned_addr learned my addr 172.21.15.102:0/135683813 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:33:19.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 -- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 msgr2=0x7f42b4147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:19.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b4147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 -- 172.21.15.102:0/135683813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42b40cf0d0 msgr2=0x7f42b4148500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:19.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42b40cf0d0 0x7f42b4148500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b2ffd700 1 -- 172.21.15.102:0/135683813 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a8009550 con 0x7f42b40b3970 2023-03-03T08:33:19.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b27fc700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b4147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:33:19.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.107+0000 7f42b37fe700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42b40cf0d0 0x7f42b4148500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:33:19.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.108+0000 7f42b2ffd700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40b3970 0x7f42b4150730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009b60 tx=0x7f42a4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:19.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.108+0000 7f429bfff700 1 -- 172.21.15.102:0/135683813 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4004450 con 0x7f42b40b3970 2023-03-03T08:33:19.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.108+0000 7f429bfff700 1 -- 172.21.15.102:0/135683813 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a401e070 con 0x7f42b40b3970 2023-03-03T08:33:19.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.108+0000 7f429bfff700 1 -- 172.21.15.102:0/135683813 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a40038e0 con 0x7f42b40b3970 2023-03-03T08:33:19.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.108+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a40094a0 con 0x7f42b40b3970 2023-03-03T08:33:19.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.108+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b406bf20 con 0x7f42b40b3970 2023-03-03T08:33:19.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.109+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b415ce80 con 0x7f42b40b3970 2023-03-03T08:33:19.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.110+0000 7f429bfff700 1 -- 172.21.15.102:0/135683813 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f42a400f050 con 0x7f42b40b3970 2023-03-03T08:33:19.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.112+0000 7f429bfff700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a00682f0 0x7f42a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:19.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.112+0000 7f429bfff700 1 -- 172.21.15.102:0/135683813 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f42a4086b30 con 0x7f42b40b3970 2023-03-03T08:33:19.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.112+0000 7f42b27fc700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a00682f0 0x7f42a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:19.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.113+0000 7f42b27fc700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a00682f0 0x7f42a006a7a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f42b40c9560 tx=0x7f42a80092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:19.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.116+0000 7f429bfff700 1 -- 172.21.15.102:0/135683813 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42a4053300 con 0x7f42b40b3970 2023-03-03T08:33:19.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.354+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42b406c200 con 0x7f42b40b3970 2023-03-03T08:33:19.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.355+0000 7f429bfff700 1 -- 172.21.15.102:0/135683813 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f42a4020090 con 0x7f42b40b3970 2023-03-03T08:33:19.357 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:33:19.357 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:33:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a00682f0 msgr2=0x7f42a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a00682f0 0x7f42a006a7a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f42b40c9560 tx=0x7f42a80092c0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40b3970 msgr2=0x7f42b4150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40b3970 0x7f42b4150730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009b60 tx=0x7f42a4004750 comp rx=0 tx=0).stop 2023-03-03T08:33:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 shutdown_connections 2023-03-03T08:33:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a00682f0 0x7f42a006a7a0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42b40b3970 0x7f42b4150730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42b40cf0d0 0x7f42b4148500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 --2- 172.21.15.102:0/135683813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42b40ced00 0x7f42b4147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 >> 172.21.15.102:0/135683813 conn(0x7f42b408b1c0 msgr2=0x7f42b40b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:33:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 shutdown_connections 2023-03-03T08:33:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:19.358+0000 7f42b3fff700 1 -- 172.21.15.102:0/135683813 wait complete. 2023-03-03T08:33:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:33:19.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before thrashing 2023-03-03T08:33:23.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:33:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.637+0000 7fdbe5dff700 1 Processor -- start 2023-03-03T08:33:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.638+0000 7fdbe5dff700 1 -- start start 2023-03-03T08:33:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbe5dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbe0153290 0x7fdbe0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbe5dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe01593c0 0x7fdbe00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbe5dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbe00cd8e0 0x7fdbe00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbe5dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe0153660 con 0x7fdbe00cd8e0 2023-03-03T08:33:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbe5dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe006c360 con 0x7fdbe0153290 2023-03-03T08:33:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbe5dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe015cea0 con 0x7fdbe01593c0 2023-03-03T08:33:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe01593c0 0x7fdbe00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbdffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbe00cd8e0 0x7fdbe00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbdf7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbe0153290 0x7fdbe0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbdf7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbe0153290 0x7fdbe0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39392/0 (socket says 172.21.15.102:39392) 2023-03-03T08:33:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdf7fe700 1 -- 172.21.15.102:0/1027296628 learned_addr learned my addr 172.21.15.102:0/1027296628 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:33:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.639+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe01593c0 0x7fdbe00cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46404/0 (socket says 172.21.15.102:46404) 2023-03-03T08:33:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbe00cd8e0 0x7fdbe00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38734/0 (socket says 172.21.15.102:38734) 2023-03-03T08:33:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdf7fe700 1 -- 172.21.15.102:0/1027296628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe01593c0 msgr2=0x7fdbe00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:33:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdf7fe700 1 --2- 172.21.15.102:0/1027296628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe01593c0 0x7fdbe00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdf7fe700 1 -- 172.21.15.102:0/1027296628 >> v2:172.21.15.102:3300/0 conn(0x7fdbe00cd8e0 msgr2=0x7fdbe00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:33:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdf7fe700 1 --2- 172.21.15.102:0/1027296628 >> v2:172.21.15.102:3300/0 conn(0x7fdbe00cd8e0 0x7fdbe00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdf7fe700 1 -- 172.21.15.102:0/1027296628 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe006c7f0 con 0x7fdbe0153290 2023-03-03T08:33:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdffff700 1 --2- 172.21.15.102:0/1027296628 >> v2:172.21.15.102:3300/0 conn(0x7fdbe00cd8e0 0x7fdbe00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:33:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.640+0000 7fdbdf7fe700 1 --2- 172.21.15.102:0/1027296628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 0x7fdbe0158e80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdbe008cc70 tx=0x7fdbd0009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eefb4bf42c8597cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbde7fc700 1 -- 172.21.15.102:0/1027296628 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0011830 con 0x7fdbe0153290 2023-03-03T08:33:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbde7fc700 1 -- 172.21.15.102:0/1027296628 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd0011990 con 0x7fdbe0153290 2023-03-03T08:33:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbde7fc700 1 -- 172.21.15.102:0/1027296628 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0011d20 con 0x7fdbe0153290 2023-03-03T08:33:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1027296628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 msgr2=0x7fdbe0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1027296628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 0x7fdbe0158e80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdbe008cc70 tx=0x7fdbd0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1027296628 shutdown_connections 2023-03-03T08:33:23.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1027296628 >> v2:172.21.15.102:3300/0 conn(0x7fdbe00cd8e0 0x7fdbe00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1027296628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 0x7fdbe0158e80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1027296628 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe01593c0 0x7fdbe00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1027296628 >> 172.21.15.102:0/1027296628 conn(0x7fdbe0065fb0 msgr2=0x7fdbe00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:33:23.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1027296628 shutdown_connections 2023-03-03T08:33:23.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.641+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1027296628 wait complete. 2023-03-03T08:33:23.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.643+0000 7fdbe5dff700 1 Processor -- start 2023-03-03T08:33:23.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.644+0000 7fdbe5dff700 1 -- start start 2023-03-03T08:33:23.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.644+0000 7fdbe5dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe00cd8e0 0x7fdbe01503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:23.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.644+0000 7fdbe5dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 0x7fdbe01e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:23.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbe5dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbe01593c0 0x7fdbe01ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:23.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbe5dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe01559c0 con 0x7fdbe01593c0 2023-03-03T08:33:23.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbe5dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe01f25f0 con 0x7fdbe0153290 2023-03-03T08:33:23.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbe5dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe0150900 con 0x7fdbe00cd8e0 2023-03-03T08:33:23.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbe01593c0 0x7fdbe01ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:23.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe00cd8e0 0x7fdbe01503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:23.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbe01593c0 0x7fdbe01ef900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38744/0 (socket says 172.21.15.102:38744) 2023-03-03T08:33:23.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 -- 172.21.15.102:0/1097124335 learned_addr learned my addr 172.21.15.102:0/1097124335 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:33:23.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdeffd700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 0x7fdbe01e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:23.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 -- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe00cd8e0 msgr2=0x7fdbe01503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:23.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe00cd8e0 0x7fdbe01503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 -- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 msgr2=0x7fdbe01e5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:23.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 0x7fdbe01e5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdffff700 1 -- 172.21.15.102:0/1097124335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd8009a70 con 0x7fdbe01593c0 2023-03-03T08:33:23.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.645+0000 7fdbdf7fe700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe00cd8e0 0x7fdbe01503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:33:23.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.646+0000 7fdbdffff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbe01593c0 0x7fdbe01ef900 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009e60 tx=0x7fdbd800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:23.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.646+0000 7fdbc7fff700 1 -- 172.21.15.102:0/1097124335 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8009cd0 con 0x7fdbe01593c0 2023-03-03T08:33:23.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.646+0000 7fdbc7fff700 1 -- 172.21.15.102:0/1097124335 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd800f040 con 0x7fdbe01593c0 2023-03-03T08:33:23.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.646+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd0009550 con 0x7fdbe01593c0 2023-03-03T08:33:23.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.646+0000 7fdbc7fff700 1 -- 172.21.15.102:0/1097124335 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd800ccb0 con 0x7fdbe01593c0 2023-03-03T08:33:23.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.646+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe00ccde0 con 0x7fdbe01593c0 2023-03-03T08:33:23.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.647+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe015d160 con 0x7fdbe01593c0 2023-03-03T08:33:23.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.648+0000 7fdbc7fff700 1 -- 172.21.15.102:0/1097124335 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdbd8014480 con 0x7fdbe01593c0 2023-03-03T08:33:23.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.649+0000 7fdbc7fff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbcc0682f0 0x7fdbcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:23.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.650+0000 7fdbc7fff700 1 -- 172.21.15.102:0/1097124335 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdbd8086570 con 0x7fdbe01593c0 2023-03-03T08:33:23.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.650+0000 7fdbdf7fe700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbcc0682f0 0x7fdbcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:23.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.651+0000 7fdbdf7fe700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbcc0682f0 0x7fdbcc06a7a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0014040 tx=0x7fdbd0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:23.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.654+0000 7fdbc7fff700 1 -- 172.21.15.102:0/1097124335 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdbd8052b90 con 0x7fdbe01593c0 2023-03-03T08:33:23.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.887+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbe00cd0c0 con 0x7fdbe01593c0 2023-03-03T08:33:23.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.887+0000 7fdbc7fff700 1 -- 172.21.15.102:0/1097124335 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdbd80561b0 con 0x7fdbe01593c0 2023-03-03T08:33:23.889 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:33:23.889 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:33:23.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbcc0682f0 msgr2=0x7fdbcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:23.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbcc0682f0 0x7fdbcc06a7a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0014040 tx=0x7fdbd0008030 comp rx=0 tx=0).stop 2023-03-03T08:33:23.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbe01593c0 msgr2=0x7fdbe01ef900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:23.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbe01593c0 0x7fdbe01ef900 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009e60 tx=0x7fdbd800c3a0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 shutdown_connections 2023-03-03T08:33:23.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbcc0682f0 0x7fdbcc06a7a0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbe00cd8e0 0x7fdbe01503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbe01593c0 0x7fdbe01ef900 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 --2- 172.21.15.102:0/1097124335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbe0153290 0x7fdbe01e5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 >> 172.21.15.102:0/1097124335 conn(0x7fdbe0065fb0 msgr2=0x7fdbe015c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:33:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 shutdown_connections 2023-03-03T08:33:23.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:23.890+0000 7fdbe5dff700 1 -- 172.21.15.102:0/1097124335 wait complete. 2023-03-03T08:33:23.897 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:33:23.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2023-03-03T08:33:30.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:30.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:37.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:33:38.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.171+0000 7f3e145ff700 1 Processor -- start 2023-03-03T08:33:38.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.171+0000 7f3e145ff700 1 -- start start 2023-03-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.172+0000 7f3e145ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c153290 0x7f3e0c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.172+0000 7f3e145ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e0c1593c0 0x7f3e0c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.172+0000 7f3e145ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3e0c0cd8e0 0x7f3e0c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.172+0000 7f3e145ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0c153660 con 0x7f3e0c1593c0 2023-03-03T08:33:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.172+0000 7f3e145ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0c06c360 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.172+0000 7f3e145ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c15cea0 con 0x7f3e0c153290 2023-03-03T08:33:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e0c1593c0 0x7f3e0c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e1239b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c153290 0x7f3e0c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e0c1593c0 0x7f3e0c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45358/0 (socket says 172.21.15.102:45358) 2023-03-03T08:33:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 -- 172.21.15.102:0/2024874694 learned_addr learned my addr 172.21.15.102:0/2024874694 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:33:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e12b9c700 1 --2- 172.21.15.102:0/2024874694 >> v2:172.21.15.172:3300/0 conn(0x7f3e0c0cd8e0 0x7f3e0c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 -- 172.21.15.102:0/2024874694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c153290 msgr2=0x7f3e0c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:33:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 --2- 172.21.15.102:0/2024874694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c153290 0x7f3e0c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 -- 172.21.15.102:0/2024874694 >> v2:172.21.15.172:3300/0 conn(0x7f3e0c0cd8e0 msgr2=0x7f3e0c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:33:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 --2- 172.21.15.102:0/2024874694 >> v2:172.21.15.172:3300/0 conn(0x7f3e0c0cd8e0 0x7f3e0c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 -- 172.21.15.102:0/2024874694 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c06c7f0 con 0x7f3e0c1593c0 2023-03-03T08:33:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.173+0000 7f3e11b9a700 1 --2- 172.21.15.102:0/2024874694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1593c0 0x7f3e0c0cde20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c06d0e0 tx=0x7f3dfc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67cc66af99cd3e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e11399700 1 -- 172.21.15.102:0/2024874694 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc011830 con 0x7f3e0c1593c0 2023-03-03T08:33:38.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e11399700 1 -- 172.21.15.102:0/2024874694 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc011990 con 0x7f3e0c1593c0 2023-03-03T08:33:38.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e11399700 1 -- 172.21.15.102:0/2024874694 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc011d20 con 0x7f3e0c1593c0 2023-03-03T08:33:38.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e145ff700 1 -- 172.21.15.102:0/2024874694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1593c0 msgr2=0x7f3e0c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:38.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2024874694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1593c0 0x7f3e0c0cde20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c06d0e0 tx=0x7f3dfc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e145ff700 1 -- 172.21.15.102:0/2024874694 shutdown_connections 2023-03-03T08:33:38.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2024874694 >> v2:172.21.15.172:3300/0 conn(0x7f3e0c0cd8e0 0x7f3e0c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2024874694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c153290 0x7f3e0c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2024874694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1593c0 0x7f3e0c0cde20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.174+0000 7f3e145ff700 1 -- 172.21.15.102:0/2024874694 >> 172.21.15.102:0/2024874694 conn(0x7f3e0c065fb0 msgr2=0x7f3e0c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:33:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.175+0000 7f3e145ff700 1 -- 172.21.15.102:0/2024874694 shutdown_connections 2023-03-03T08:33:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.175+0000 7f3e145ff700 1 -- 172.21.15.102:0/2024874694 wait complete. 2023-03-03T08:33:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.176+0000 7f3e145ff700 1 Processor -- start 2023-03-03T08:33:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.177+0000 7f3e145ff700 1 -- start start 2023-03-03T08:33:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e145ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0cd8e0 0x7f3e0c0b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e145ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0c153290 0x7f3e0c0cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e145ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c1593c0 0x7f3e0c15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e145ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c0c90a0 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e145ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c0b2df0 con 0x7f3e0c153290 2023-03-03T08:33:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e145ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c0c6c50 con 0x7f3e0c1593c0 2023-03-03T08:33:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e1239b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0cd8e0 0x7f3e0c0b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e11b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0c153290 0x7f3e0c0cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e11b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0c153290 0x7f3e0c0cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38154/0 (socket says 172.21.15.102:38154) 2023-03-03T08:33:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e11b9a700 1 -- 172.21.15.102:0/2786624926 learned_addr learned my addr 172.21.15.102:0/2786624926 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:33:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e12b9c700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c1593c0 0x7f3e0c15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e1239b700 1 -- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c1593c0 msgr2=0x7f3e0c15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e1239b700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c1593c0 0x7f3e0c15ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e1239b700 1 -- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0c153290 msgr2=0x7f3e0c0cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e1239b700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0c153290 0x7f3e0c0cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.178+0000 7f3e1239b700 1 -- 172.21.15.102:0/2786624926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc009550 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.179+0000 7f3e1239b700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0cd8e0 0x7f3e0c0b28b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e870 tx=0x7f3e0800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.179+0000 7f3e02ffd700 1 -- 172.21.15.102:0/2786624926 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08013680 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.179+0000 7f3e02ffd700 1 -- 172.21.15.102:0/2786624926 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e080137e0 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.179+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c0ccde0 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.179+0000 7f3e02ffd700 1 -- 172.21.15.102:0/2786624926 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08013b90 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.179+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c15d160 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.180+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0c0cd0c0 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.181+0000 7f3e02ffd700 1 -- 172.21.15.102:0/2786624926 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3e08019030 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.182+0000 7f3e02ffd700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3df00682f0 0x7f3df006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:33:38.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.183+0000 7f3e02ffd700 1 -- 172.21.15.102:0/2786624926 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3e08085940 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.183+0000 7f3e11b9a700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3df00682f0 0x7f3df006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:33:38.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.184+0000 7f3e11b9a700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3df00682f0 0x7f3df006a7a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c151b40 tx=0x7f3dfc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:33:38.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.187+0000 7f3e02ffd700 1 -- 172.21.15.102:0/2786624926 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3e08052050 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.420+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e0c0c7210 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.421+0000 7f3e02ffd700 1 -- 172.21.15.102:0/2786624926 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3e08055670 con 0x7f3e0c0cd8e0 2023-03-03T08:33:38.422 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:33:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:33:38.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3df00682f0 msgr2=0x7f3df006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3df00682f0 0x7f3df006a7a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c151b40 tx=0x7f3dfc008030 comp rx=0 tx=0).stop 2023-03-03T08:33:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0cd8e0 msgr2=0x7f3e0c0b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:33:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0cd8e0 0x7f3e0c0b28b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e870 tx=0x7f3e0800eb80 comp rx=0 tx=0).stop 2023-03-03T08:33:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 shutdown_connections 2023-03-03T08:33:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3df00682f0 0x7f3df006a7a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3e0c1593c0 0x7f3e0c15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0cd8e0 0x7f3e0c0b28b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 --2- 172.21.15.102:0/2786624926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3e0c153290 0x7f3e0c0cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:33:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 >> 172.21.15.102:0/2786624926 conn(0x7f3e0c065fb0 msgr2=0x7f3e0c156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:33:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.423+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 shutdown_connections 2023-03-03T08:33:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:33:38.424+0000 7f3e145ff700 1 -- 172.21.15.102:0/2786624926 wait complete. 2023-03-03T08:33:38.426 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:33:38.452 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-03-03T08:34:00.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:00.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:30.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:30.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:00.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:00.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:03.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:35:03.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.703+0000 7ff3bc3ff700 1 Processor -- start 2023-03-03T08:35:03.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.704+0000 7ff3bc3ff700 1 -- start start 2023-03-03T08:35:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3bc3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4153290 0x7ff3b4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:35:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3bc3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff3b41593c0 0x7ff3b40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:35:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3bc3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b40cd8e0 0x7ff3b40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:35:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3bc3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b4153660 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3bc3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b406c360 con 0x7ff3b41593c0 2023-03-03T08:35:03.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3bc3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b415cea0 con 0x7ff3b4153290 2023-03-03T08:35:03.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3ba99c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b40cd8e0 0x7ff3b40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:35:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3b999a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff3b41593c0 0x7ff3b40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:35:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3ba19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4153290 0x7ff3b4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:35:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3ba99c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b40cd8e0 0x7ff3b40ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55692/0 (socket says 172.21.15.102:55692) 2023-03-03T08:35:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.705+0000 7ff3ba99c700 1 -- 172.21.15.102:0/579678013 learned_addr learned my addr 172.21.15.102:0/579678013 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:35:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3ba99c700 1 -- 172.21.15.102:0/579678013 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4153290 msgr2=0x7ff3b4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:35:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3ba99c700 1 --2- 172.21.15.102:0/579678013 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4153290 0x7ff3b4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3ba99c700 1 -- 172.21.15.102:0/579678013 >> v2:172.21.15.172:3300/0 conn(0x7ff3b41593c0 msgr2=0x7ff3b40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:35:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3ba99c700 1 --2- 172.21.15.102:0/579678013 >> v2:172.21.15.172:3300/0 conn(0x7ff3b41593c0 0x7ff3b40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3ba99c700 1 -- 172.21.15.102:0/579678013 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b406c7f0 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3b999a700 1 --2- 172.21.15.102:0/579678013 >> v2:172.21.15.172:3300/0 conn(0x7ff3b41593c0 0x7ff3b40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:35:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3ba99c700 1 --2- 172.21.15.102:0/579678013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40cd8e0 0x7ff3b40ce360 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40b1970 tx=0x7ff3ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37f995a2e5e2753 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:35:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3b9199700 1 -- 172.21.15.102:0/579678013 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac00e070 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3b9199700 1 -- 172.21.15.102:0/579678013 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac00b910 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.706+0000 7ff3b9199700 1 -- 172.21.15.102:0/579678013 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac00bcc0 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/579678013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40cd8e0 msgr2=0x7ff3b40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:35:03.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/579678013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40cd8e0 0x7ff3b40ce360 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40b1970 tx=0x7ff3ac00b330 comp rx=0 tx=0).stop 2023-03-03T08:35:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/579678013 shutdown_connections 2023-03-03T08:35:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/579678013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b40cd8e0 0x7ff3b40ce360 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/579678013 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b4153290 0x7ff3b4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/579678013 >> v2:172.21.15.172:3300/0 conn(0x7ff3b41593c0 0x7ff3b40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/579678013 >> 172.21.15.102:0/579678013 conn(0x7ff3b4065fb0 msgr2=0x7ff3b40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:35:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/579678013 shutdown_connections 2023-03-03T08:35:03.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.707+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/579678013 wait complete. 2023-03-03T08:35:03.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.710+0000 7ff3bc3ff700 1 Processor -- start 2023-03-03T08:35:03.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.710+0000 7ff3bc3ff700 1 -- start start 2023-03-03T08:35:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3bc3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cd8e0 0x7ff3b41e5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:35:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3bc3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b4153290 0x7ff3b40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:35:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3bc3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b41593c0 0x7ff3b40b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:35:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3bc3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40c7c60 con 0x7ff3b4153290 2023-03-03T08:35:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3bc3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40b1c70 con 0x7ff3b41593c0 2023-03-03T08:35:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3bc3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40b2510 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cd8e0 0x7ff3b41e5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:35:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cd8e0 0x7ff3b41e5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34940/0 (socket says 172.21.15.102:34940) 2023-03-03T08:35:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 -- 172.21.15.102:0/3821834521 learned_addr learned my addr 172.21.15.102:0/3821834521 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:35:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3b999a700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b4153290 0x7ff3b40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:35:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba99c700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b41593c0 0x7ff3b40b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:35:03.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 -- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b41593c0 msgr2=0x7ff3b40b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:35:03.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b41593c0 0x7ff3b40b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 -- 172.21.15.102:0/3821834521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b4153290 msgr2=0x7ff3b40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:35:03.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b4153290 0x7ff3b40b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3ba19b700 1 -- 172.21.15.102:0/3821834521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a40094a0 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.712+0000 7ff3b999a700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b4153290 0x7ff3b40b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.713+0000 7ff3ba19b700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cd8e0 0x7ff3b41e5560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000e8a0 tx=0x7ff3b000ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.713+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3821834521 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0009590 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.713+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3821834521 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b0013680 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.713+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3821834521 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b00139f0 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.713+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac009b30 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.713+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b40ccde0 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.714+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b415d160 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.716+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3821834521 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff3b0019030 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.717+0000 7ff3aaffd700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff398068320 0x7ff39806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:35:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.717+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3821834521 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff3b0085a40 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.717+0000 7ff3b999a700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff398068320 0x7ff39806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:35:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.718+0000 7ff3b999a700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff398068320 0x7ff39806a7d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40cfd60 tx=0x7ff3a4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:35:03.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.721+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3821834521 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff3b0052150 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.955+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3b40cd0c0 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.956+0000 7ff3aaffd700 1 -- 172.21.15.102:0/3821834521 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff3b0055770 con 0x7ff3b40cd8e0 2023-03-03T08:35:03.957 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:35:03.957 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:35:03.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.958+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff398068320 msgr2=0x7ff39806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:35:03.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.958+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff398068320 0x7ff39806a7d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40cfd60 tx=0x7ff3a4009210 comp rx=0 tx=0).stop 2023-03-03T08:35:03.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.958+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cd8e0 msgr2=0x7ff3b41e5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:35:03.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.958+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cd8e0 0x7ff3b41e5560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000e8a0 tx=0x7ff3b000ebb0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 shutdown_connections 2023-03-03T08:35:03.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff398068320 0x7ff39806a7d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff3b40cd8e0 0x7ff3b41e5560 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b41593c0 0x7ff3b40b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 --2- 172.21.15.102:0/3821834521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b4153290 0x7ff3b40b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:35:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 >> 172.21.15.102:0/3821834521 conn(0x7ff3b4065fb0 msgr2=0x7ff3b4156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:35:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 shutdown_connections 2023-03-03T08:35:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:35:03.959+0000 7ff3bc3ff700 1 -- 172.21.15.102:0/3821834521 wait complete. 2023-03-03T08:35:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:35:03.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T08:35:31.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:31.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:01.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:01.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:31.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:31.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:58.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:36:59.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.240+0000 7f13b1f9d700 1 Processor -- start 2023-03-03T08:36:59.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.240+0000 7f13b1f9d700 1 -- start start 2023-03-03T08:36:59.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13b1f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13ac0ced00 0x7f13ac0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:36:59.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13b1f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13ac0b3970 0x7f13ac0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:36:59.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13b1f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac0c3d20 con 0x7f13ac0cf0d0 2023-03-03T08:36:59.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13b1f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac0c4930 con 0x7f13ac0b3970 2023-03-03T08:36:59.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13b1f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac0cf570 con 0x7f13ac0ced00 2023-03-03T08:36:59.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13abfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13ac0b3970 0x7f13ac0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:36:59.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13b0f9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13ac0ced00 0x7f13ac0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:36:59.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13abfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13ac0b3970 0x7f13ac0c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48930/0 (socket says 172.21.15.102:48930) 2023-03-03T08:36:59.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13b179c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f13ac0cf0d0 0x7f13ac158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59334/0 (socket says 172.21.15.102:59334) 2023-03-03T08:36:59.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.242+0000 7f13abfff700 1 -- 172.21.15.102:0/687544454 learned_addr learned my addr 172.21.15.102:0/687544454 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:36:59.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13abfff700 1 -- 172.21.15.102:0/687544454 >> v2:172.21.15.172:3300/0 conn(0x7f13ac0ced00 msgr2=0x7f13ac0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:36:59.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13abfff700 1 --2- 172.21.15.102:0/687544454 >> v2:172.21.15.172:3300/0 conn(0x7f13ac0ced00 0x7f13ac0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13ab7fe700 1 -- 172.21.15.102:0/687544454 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048597717 0 0) 0x7f13ac0c3d20 con 0x7f13ac0cf0d0 2023-03-03T08:36:59.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13abfff700 1 -- 172.21.15.102:0/687544454 >> v1:172.21.15.172:6789/0 conn(0x7f13ac0cf0d0 legacy=0x7f13ac158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:36:59.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13abfff700 1 -- 172.21.15.102:0/687544454 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac15cba0 con 0x7f13ac0b3970 2023-03-03T08:36:59.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13abfff700 1 --2- 172.21.15.102:0/687544454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac0c41b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0b1b40 tx=0x7f1398009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cef0d3df11e581 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:36:59.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13ab7fe700 1 -- 172.21.15.102:0/687544454 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398011830 con 0x7f13ac0b3970 2023-03-03T08:36:59.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13ab7fe700 1 -- 172.21.15.102:0/687544454 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1398011990 con 0x7f13ac0b3970 2023-03-03T08:36:59.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.243+0000 7f13ab7fe700 1 -- 172.21.15.102:0/687544454 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398011d20 con 0x7f13ac0b3970 2023-03-03T08:36:59.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 -- 172.21.15.102:0/687544454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 msgr2=0x7f13ac0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:36:59.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/687544454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac0c41b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0b1b40 tx=0x7f1398009bd0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 -- 172.21.15.102:0/687544454 shutdown_connections 2023-03-03T08:36:59.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/687544454 >> v2:172.21.15.172:3300/0 conn(0x7f13ac0ced00 0x7f13ac0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/687544454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac0c41b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 -- 172.21.15.102:0/687544454 >> 172.21.15.102:0/687544454 conn(0x7f13ac08b1c0 msgr2=0x7f13ac071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:36:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 -- 172.21.15.102:0/687544454 shutdown_connections 2023-03-03T08:36:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.244+0000 7f13b1f9d700 1 -- 172.21.15.102:0/687544454 wait complete. 2023-03-03T08:36:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.247+0000 7f13b1f9d700 1 Processor -- start 2023-03-03T08:36:59.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.247+0000 7f13b1f9d700 1 -- start start 2023-03-03T08:36:59.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.248+0000 7f13b1f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:36:59.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.248+0000 7f13b1f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13ac0ced00 0x7f13ac1f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:36:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b1f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ac0cf0d0 0x7f13ac14e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:36:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b1f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac14ee90 con 0x7f13ac0b3970 2023-03-03T08:36:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b1f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac151d80 con 0x7f13ac0cf0d0 2023-03-03T08:36:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b1f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac1f2960 con 0x7f13ac0ced00 2023-03-03T08:36:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:36:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13abfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13ac0ced00 0x7f13ac1f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:36:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b179c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ac0cf0d0 0x7f13ac14e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:36:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac15a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48944/0 (socket says 172.21.15.102:48944) 2023-03-03T08:36:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 -- 172.21.15.102:0/1906171569 learned_addr learned my addr 172.21.15.102:0/1906171569 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:36:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13abfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13ac0ced00 0x7f13ac1f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34194/0 (socket says 172.21.15.102:34194) 2023-03-03T08:36:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 -- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13ac0ced00 msgr2=0x7f13ac1f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:36:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13ac0ced00 0x7f13ac1f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 -- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ac0cf0d0 msgr2=0x7f13ac14e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:36:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ac0cf0d0 0x7f13ac14e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b0f9b700 1 -- 172.21.15.102:0/1906171569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1398009550 con 0x7f13ac0b3970 2023-03-03T08:36:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.249+0000 7f13b179c700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ac0cf0d0 0x7f13ac14e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:36:59.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.250+0000 7f13b0f9b700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac15a4b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f13a000e8a0 tx=0x7f13a000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:36:59.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.250+0000 7f13a97fa700 1 -- 172.21.15.102:0/1906171569 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0009590 con 0x7f13ac0b3970 2023-03-03T08:36:59.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.250+0000 7f13a97fa700 1 -- 172.21.15.102:0/1906171569 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0013680 con 0x7f13ac0b3970 2023-03-03T08:36:59.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.250+0000 7f13a97fa700 1 -- 172.21.15.102:0/1906171569 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a00139f0 con 0x7f13ac0b3970 2023-03-03T08:36:59.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.250+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac06bf20 con 0x7f13ac0b3970 2023-03-03T08:36:59.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.250+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac15ce80 con 0x7f13ac0b3970 2023-03-03T08:36:59.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.251+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac06c200 con 0x7f13ac0b3970 2023-03-03T08:36:59.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.252+0000 7f13a97fa700 1 -- 172.21.15.102:0/1906171569 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13a0019030 con 0x7f13ac0b3970 2023-03-03T08:36:59.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.254+0000 7f13a97fa700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1390068320 0x7f139006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:36:59.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.254+0000 7f13a97fa700 1 -- 172.21.15.102:0/1906171569 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f13a0085b20 con 0x7f13ac0b3970 2023-03-03T08:36:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.254+0000 7f13abfff700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1390068320 0x7f139006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:36:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.255+0000 7f13abfff700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1390068320 0x7f139006a7d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0b1b40 tx=0x7f1398008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:36:59.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.258+0000 7f13a97fa700 1 -- 172.21.15.102:0/1906171569 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13a00521a0 con 0x7f13ac0b3970 2023-03-03T08:36:59.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.496+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13ac04e6e0 con 0x7f13ac0b3970 2023-03-03T08:36:59.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.497+0000 7f13a97fa700 1 -- 172.21.15.102:0/1906171569 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f13a0021090 con 0x7f13ac0b3970 2023-03-03T08:36:59.499 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:36:59.499 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:36:59.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1390068320 msgr2=0x7f139006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:36:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1390068320 0x7f139006a7d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0b1b40 tx=0x7f1398008030 comp rx=0 tx=0).stop 2023-03-03T08:36:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 msgr2=0x7f13ac15a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:36:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac15a4b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f13a000e8a0 tx=0x7f13a000ebb0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 shutdown_connections 2023-03-03T08:36:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ac0cf0d0 0x7f13ac14e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac0b3970 0x7f13ac15a4b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1390068320 0x7f139006a7d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 --2- 172.21.15.102:0/1906171569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13ac0ced00 0x7f13ac1f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:36:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.499+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 >> 172.21.15.102:0/1906171569 conn(0x7f13ac08b1c0 msgr2=0x7f13ac0ca0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:36:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.500+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 shutdown_connections 2023-03-03T08:36:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:36:59.500+0000 7f13b1f9d700 1 -- 172.21.15.102:0/1906171569 wait complete. 2023-03-03T08:36:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:36:59.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 76 secs before thrashing 2023-03-03T08:37:01.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:01.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:32.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:32.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:02.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:02.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:15.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:38:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.781+0000 7f9f801ff700 1 Processor -- start 2023-03-03T08:38:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.781+0000 7f9f801ff700 1 -- start start 2023-03-03T08:38:15.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.782+0000 7f9f801ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f78153290 0x7f9f78158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.782+0000 7f9f801ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f781593c0 0x7f9f780cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.782+0000 7f9f801ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f780ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.782+0000 7f9f801ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f78153660 con 0x7f9f781593c0 2023-03-03T08:38:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.782+0000 7f9f801ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7806c360 con 0x7f9f78153290 2023-03-03T08:38:15.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.782+0000 7f9f801ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7815cea0 con 0x7f9f780cd8e0 2023-03-03T08:38:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f781593c0 0x7f9f780cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7df9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f78153290 0x7f9f78158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f781593c0 0x7f9f780cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57310/0 (socket says 172.21.15.102:57310) 2023-03-03T08:38:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 -- 172.21.15.102:0/2684931492 learned_addr learned my addr 172.21.15.102:0/2684931492 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:38:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7e79c700 1 --2- 172.21.15.102:0/2684931492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f780ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 -- 172.21.15.102:0/2684931492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 msgr2=0x7f9f780ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 --2- 172.21.15.102:0/2684931492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f780ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 -- 172.21.15.102:0/2684931492 >> v2:172.21.15.172:3300/0 conn(0x7f9f78153290 msgr2=0x7f9f78158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 --2- 172.21.15.102:0/2684931492 >> v2:172.21.15.172:3300/0 conn(0x7f9f78153290 0x7f9f78158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7d79a700 1 -- 172.21.15.102:0/2684931492 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7806c7f0 con 0x7f9f781593c0 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.783+0000 7f9f7df9b700 1 --2- 172.21.15.102:0/2684931492 >> v2:172.21.15.172:3300/0 conn(0x7f9f78153290 0x7f9f78158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.784+0000 7f9f7d79a700 1 --2- 172.21.15.102:0/2684931492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f780cde20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9f7808cc70 tx=0x7f9f68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5746af2e690daa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:38:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.784+0000 7f9f7cf99700 1 -- 172.21.15.102:0/2684931492 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68011830 con 0x7f9f781593c0 2023-03-03T08:38:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.784+0000 7f9f7cf99700 1 -- 172.21.15.102:0/2684931492 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f68011990 con 0x7f9f781593c0 2023-03-03T08:38:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.784+0000 7f9f7cf99700 1 -- 172.21.15.102:0/2684931492 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68011d20 con 0x7f9f781593c0 2023-03-03T08:38:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 -- 172.21.15.102:0/2684931492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 msgr2=0x7f9f780cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 --2- 172.21.15.102:0/2684931492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f780cde20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9f7808cc70 tx=0x7f9f68009bd0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 -- 172.21.15.102:0/2684931492 shutdown_connections 2023-03-03T08:38:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 --2- 172.21.15.102:0/2684931492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f780ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 --2- 172.21.15.102:0/2684931492 >> v2:172.21.15.172:3300/0 conn(0x7f9f78153290 0x7f9f78158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 --2- 172.21.15.102:0/2684931492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f780cde20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 -- 172.21.15.102:0/2684931492 >> 172.21.15.102:0/2684931492 conn(0x7f9f78065fb0 msgr2=0x7f9f780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:38:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 -- 172.21.15.102:0/2684931492 shutdown_connections 2023-03-03T08:38:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.785+0000 7f9f801ff700 1 -- 172.21.15.102:0/2684931492 wait complete. 2023-03-03T08:38:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.787+0000 7f9f801ff700 1 Processor -- start 2023-03-03T08:38:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.787+0000 7f9f801ff700 1 -- start start 2023-03-03T08:38:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f801ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f781efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f801ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f78153290 0x7f9f780b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f801ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f781512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f801ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780b3170 con 0x7f9f781593c0 2023-03-03T08:38:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f801ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780b4370 con 0x7f9f78153290 2023-03-03T08:38:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f801ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781ef410 con 0x7f9f780cd8e0 2023-03-03T08:38:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f7e79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f781512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f7df9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f781efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f7d79a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f78153290 0x7f9f780b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f7e79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f781512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57324/0 (socket says 172.21.15.102:57324) 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.788+0000 7f9f7e79c700 1 -- 172.21.15.102:0/3758464564 learned_addr learned my addr 172.21.15.102:0/3758464564 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f7e79c700 1 -- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 msgr2=0x7f9f781efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f7e79c700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f781efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f7e79c700 1 -- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f78153290 msgr2=0x7f9f780b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f7e79c700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f78153290 0x7f9f780b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f7e79c700 1 -- 172.21.15.102:0/3758464564 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f740094a0 con 0x7f9f781593c0 2023-03-03T08:38:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f7e79c700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f781512c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000efc0 tx=0x7f9f7000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:38:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f6e7fc700 1 -- 172.21.15.102:0/3758464564 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000ca10 con 0x7f9f781593c0 2023-03-03T08:38:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f6e7fc700 1 -- 172.21.15.102:0/3758464564 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7000f040 con 0x7f9f781593c0 2023-03-03T08:38:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f6e7fc700 1 -- 172.21.15.102:0/3758464564 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000cdb0 con 0x7f9f781593c0 2023-03-03T08:38:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.789+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f68009550 con 0x7f9f781593c0 2023-03-03T08:38:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.790+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f780ccde0 con 0x7f9f781593c0 2023-03-03T08:38:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.791+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7815d160 con 0x7f9f781593c0 2023-03-03T08:38:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.792+0000 7f9f6e7fc700 1 -- 172.21.15.102:0/3758464564 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9f70014480 con 0x7f9f781593c0 2023-03-03T08:38:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.794+0000 7f9f6e7fc700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f5c0682f0 0x7f9f5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.794+0000 7f9f6e7fc700 1 -- 172.21.15.102:0/3758464564 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9f700865f0 con 0x7f9f781593c0 2023-03-03T08:38:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.794+0000 7f9f7df9b700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f5c0682f0 0x7f9f5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.795+0000 7f9f7df9b700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f5c0682f0 0x7f9f5c06a7a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009c50 tx=0x7f9f74009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:38:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:15.798+0000 7f9f6e7fc700 1 -- 172.21.15.102:0/3758464564 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9f70052ca0 con 0x7f9f781593c0 2023-03-03T08:38:16.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.033+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f780cd0c0 con 0x7f9f781593c0 2023-03-03T08:38:16.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.033+0000 7f9f6e7fc700 1 -- 172.21.15.102:0/3758464564 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9f700562c0 con 0x7f9f781593c0 2023-03-03T08:38:16.035 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:38:16.035 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:38:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f5c0682f0 msgr2=0x7f9f5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f5c0682f0 0x7f9f5c06a7a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009c50 tx=0x7f9f74009210 comp rx=0 tx=0).stop 2023-03-03T08:38:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 msgr2=0x7f9f781512c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f781512c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000efc0 tx=0x7f9f7000c0c0 comp rx=0 tx=0).stop 2023-03-03T08:38:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 shutdown_connections 2023-03-03T08:38:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f5c0682f0 0x7f9f5c06a7a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f780cd8e0 0x7f9f781efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f781593c0 0x7f9f781512c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 --2- 172.21.15.102:0/3758464564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f78153290 0x7f9f780b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 >> 172.21.15.102:0/3758464564 conn(0x7f9f78065fb0 msgr2=0x7f9f7815c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:38:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 shutdown_connections 2023-03-03T08:38:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:16.036+0000 7f9f801ff700 1 -- 172.21.15.102:0/3758464564 wait complete. 2023-03-03T08:38:16.040 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:38:16.064 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-03-03T08:38:28.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:38:28.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.314+0000 7f1c5639d700 1 Processor -- start 2023-03-03T08:38:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.314+0000 7f1c5639d700 1 -- start start 2023-03-03T08:38:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c5639d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c500c9610 0x7f1c500ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c5639d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1c500ca980 0x7f1c501636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c5639d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c500c9ca0 con 0x7f1c501564b0 2023-03-03T08:38:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c5639d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c500b4920 con 0x7f1c500ca980 2023-03-03T08:38:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c5639d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c500b4c00 con 0x7f1c500c9610 2023-03-03T08:38:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c5539b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1c501564b0 0x7f1c500cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43778/0 (socket says 172.21.15.102:43778) 2023-03-03T08:38:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c5539b700 1 -- 172.21.15.102:0/2132157701 learned_addr learned my addr 172.21.15.102:0/2132157701 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:38:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c55b9c700 1 --2- 172.21.15.102:0/2132157701 >> v2:172.21.15.102:3300/0 conn(0x7f1c500ca980 0x7f1c501636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.316+0000 7f1c54b9a700 1 --2- 172.21.15.102:0/2132157701 >> v2:172.21.15.172:3300/0 conn(0x7f1c500c9610 0x7f1c500ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c47fff700 1 -- 172.21.15.102:0/2132157701 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161664011 0 0) 0x7f1c500c9ca0 con 0x7f1c501564b0 2023-03-03T08:38:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c55b9c700 1 -- 172.21.15.102:0/2132157701 >> v2:172.21.15.172:3300/0 conn(0x7f1c500c9610 msgr2=0x7f1c500ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:38:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c55b9c700 1 --2- 172.21.15.102:0/2132157701 >> v2:172.21.15.172:3300/0 conn(0x7f1c500c9610 0x7f1c500ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c55b9c700 1 -- 172.21.15.102:0/2132157701 >> v1:172.21.15.102:6789/0 conn(0x7f1c501564b0 legacy=0x7f1c500cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c55b9c700 1 -- 172.21.15.102:0/2132157701 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c500b46f0 con 0x7f1c500ca980 2023-03-03T08:38:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c55b9c700 1 --2- 172.21.15.102:0/2132157701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500ca980 0x7f1c501636a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c5015d410 tx=0x7f1c4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c80ac49d55c6cc0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:38:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c47fff700 1 -- 172.21.15.102:0/2132157701 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4800e070 con 0x7f1c500ca980 2023-03-03T08:38:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c47fff700 1 -- 172.21.15.102:0/2132157701 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4800bb20 con 0x7f1c500ca980 2023-03-03T08:38:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.317+0000 7f1c47fff700 1 -- 172.21.15.102:0/2132157701 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48012430 con 0x7f1c500ca980 2023-03-03T08:38:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 -- 172.21.15.102:0/2132157701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500ca980 msgr2=0x7f1c501636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 --2- 172.21.15.102:0/2132157701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500ca980 0x7f1c501636a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c5015d410 tx=0x7f1c4800b330 comp rx=0 tx=0).stop 2023-03-03T08:38:28.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 -- 172.21.15.102:0/2132157701 shutdown_connections 2023-03-03T08:38:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 --2- 172.21.15.102:0/2132157701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500ca980 0x7f1c501636a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 --2- 172.21.15.102:0/2132157701 >> v2:172.21.15.172:3300/0 conn(0x7f1c500c9610 0x7f1c500ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 -- 172.21.15.102:0/2132157701 >> 172.21.15.102:0/2132157701 conn(0x7f1c50065fb0 msgr2=0x7f1c500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:38:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 -- 172.21.15.102:0/2132157701 shutdown_connections 2023-03-03T08:38:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.318+0000 7f1c5639d700 1 -- 172.21.15.102:0/2132157701 wait complete. 2023-03-03T08:38:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.320+0000 7f1c5639d700 1 Processor -- start 2023-03-03T08:38:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.320+0000 7f1c5639d700 1 -- start start 2023-03-03T08:38:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.321+0000 7f1c5639d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500c9610 0x7f1c50158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.321+0000 7f1c5639d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c500ca980 0x7f1c50154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.321+0000 7f1c5639d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c501564b0 0x7f1c500c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.321+0000 7f1c5639d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50159310 con 0x7f1c500c9610 2023-03-03T08:38:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.321+0000 7f1c5639d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c501595d0 con 0x7f1c501564b0 2023-03-03T08:38:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.321+0000 7f1c5639d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50156aa0 con 0x7f1c500ca980 2023-03-03T08:38:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c5539b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500c9610 0x7f1c50158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c54b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c500ca980 0x7f1c50154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c501564b0 0x7f1c500c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c501564b0 0x7f1c500c7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34102/0 (socket says 172.21.15.102:34102) 2023-03-03T08:38:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c54b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c500ca980 0x7f1c50154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:35984/0 (socket says 172.21.15.102:35984) 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 -- 172.21.15.102:0/3500635848 learned_addr learned my addr 172.21.15.102:0/3500635848 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 -- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c500ca980 msgr2=0x7f1c50154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c500ca980 0x7f1c50154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 -- 172.21.15.102:0/3500635848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500c9610 msgr2=0x7f1c50158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500c9610 0x7f1c50158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 -- 172.21.15.102:0/3500635848 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48009a70 con 0x7f1c501564b0 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.322+0000 7f1c55b9c700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c501564b0 0x7f1c500c7390 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1c5006b2a0 tx=0x7f1c48004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:38:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.323+0000 7f1c45ffb700 1 -- 172.21.15.102:0/3500635848 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48012430 con 0x7f1c501564b0 2023-03-03T08:38:28.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.323+0000 7f1c45ffb700 1 -- 172.21.15.102:0/3500635848 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c48020070 con 0x7f1c501564b0 2023-03-03T08:38:28.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.323+0000 7f1c45ffb700 1 -- 172.21.15.102:0/3500635848 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48004280 con 0x7f1c501564b0 2023-03-03T08:38:28.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.323+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c500b2210 con 0x7f1c501564b0 2023-03-03T08:38:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.323+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c500b1f30 con 0x7f1c501564b0 2023-03-03T08:38:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.324+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c50156880 con 0x7f1c501564b0 2023-03-03T08:38:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.325+0000 7f1c45ffb700 1 -- 172.21.15.102:0/3500635848 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1c4802e070 con 0x7f1c501564b0 2023-03-03T08:38:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.326+0000 7f1c45ffb700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c340682f0 0x7f1c3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:38:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.326+0000 7f1c45ffb700 1 -- 172.21.15.102:0/3500635848 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1c48022070 con 0x7f1c501564b0 2023-03-03T08:38:28.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.327+0000 7f1c5539b700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c340682f0 0x7f1c3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:38:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.327+0000 7f1c5539b700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c340682f0 0x7f1c3406a7a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c000f60 tx=0x7f1c4c007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:38:28.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.331+0000 7f1c45ffb700 1 -- 172.21.15.102:0/3500635848 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1c480540b0 con 0x7f1c501564b0 2023-03-03T08:38:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.563+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c500c70b0 con 0x7f1c501564b0 2023-03-03T08:38:28.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.565+0000 7f1c45ffb700 1 -- 172.21.15.102:0/3500635848 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1c500c70b0 con 0x7f1c501564b0 2023-03-03T08:38:28.567 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:38:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:38:28.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c340682f0 msgr2=0x7f1c3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:28.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c340682f0 0x7f1c3406a7a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c000f60 tx=0x7f1c4c007e70 comp rx=0 tx=0).stop 2023-03-03T08:38:28.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c501564b0 msgr2=0x7f1c500c7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:38:28.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c501564b0 0x7f1c500c7390 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1c5006b2a0 tx=0x7f1c48004ad0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 shutdown_connections 2023-03-03T08:38:28.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c501564b0 0x7f1c500c7390 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c500c9610 0x7f1c50158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c340682f0 0x7f1c3406a7a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 --2- 172.21.15.102:0/3500635848 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c500ca980 0x7f1c50154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:38:28.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 >> 172.21.15.102:0/3500635848 conn(0x7f1c50065fb0 msgr2=0x7f1c50151bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:38:28.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 shutdown_connections 2023-03-03T08:38:28.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:38:28.567+0000 7f1c5639d700 1 -- 172.21.15.102:0/3500635848 wait complete. 2023-03-03T08:38:28.571 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:38:28.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T08:38:32.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:32.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:02.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:39:02.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:02.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:02.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.855+0000 7fe7e85ff700 1 Processor -- start 2023-03-03T08:39:02.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.855+0000 7fe7e85ff700 1 -- start start 2023-03-03T08:39:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.857+0000 7fe7e85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:39:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.857+0000 7fe7e85ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e0153660 con 0x7fe7e00cd390 2023-03-03T08:39:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.857+0000 7fe7e85ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e006c360 con 0x7fe7e0153290 2023-03-03T08:39:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.857+0000 7fe7e85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e015cea0 con 0x7fe7e00cd8e0 2023-03-03T08:39:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.857+0000 7fe7e6b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe7e00cd390 0x7fe7e00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37190/0 (socket says 172.21.15.102:37190) 2023-03-03T08:39:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.857+0000 7fe7e6b9c700 1 -- 172.21.15.102:0/2939853084 learned_addr learned my addr 172.21.15.102:0/2939853084 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:39:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.857+0000 7fe7e5b9a700 1 --2- 172.21.15.102:0/2939853084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:39:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5399700 1 -- 172.21.15.102:0/2939853084 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17162796 0 0) 0x7fe7e0153660 con 0x7fe7e00cd390 2023-03-03T08:39:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5399700 1 -- 172.21.15.102:0/2939853084 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7cc0035b0 con 0x7fe7e00cd390 2023-03-03T08:39:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5399700 1 -- 172.21.15.102:0/2939853084 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205073121 0 0) 0x7fe7e006c360 con 0x7fe7e0153290 2023-03-03T08:39:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5399700 1 -- 172.21.15.102:0/2939853084 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e0153660 con 0x7fe7e0153290 2023-03-03T08:39:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5b9a700 1 -- 172.21.15.102:0/2939853084 >> v1:172.21.15.172:6789/0 conn(0x7fe7e0153290 legacy=0x7fe7e0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:39:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5b9a700 1 -- 172.21.15.102:0/2939853084 >> v1:172.21.15.102:6789/0 conn(0x7fe7e00cd390 legacy=0x7fe7e00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:39:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5b9a700 1 -- 172.21.15.102:0/2939853084 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e006c7f0 con 0x7fe7e00cd8e0 2023-03-03T08:39:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5b9a700 1 --2- 172.21.15.102:0/2939853084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e00cde20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18113ea8c80cf231 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:39:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.858+0000 7fe7e5399700 1 -- 172.21.15.102:0/2939853084 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0011830 con 0x7fe7e00cd8e0 2023-03-03T08:39:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e5399700 1 -- 172.21.15.102:0/2939853084 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d0011990 con 0x7fe7e00cd8e0 2023-03-03T08:39:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e5399700 1 -- 172.21.15.102:0/2939853084 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0011d00 con 0x7fe7e00cd8e0 2023-03-03T08:39:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e85ff700 1 -- 172.21.15.102:0/2939853084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 msgr2=0x7fe7e00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:39:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/2939853084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e00cde20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d0009b80 comp rx=0 tx=0).stop 2023-03-03T08:39:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e85ff700 1 -- 172.21.15.102:0/2939853084 shutdown_connections 2023-03-03T08:39:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/2939853084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e00cde20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:39:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e85ff700 1 -- 172.21.15.102:0/2939853084 >> 172.21.15.102:0/2939853084 conn(0x7fe7e0065fb0 msgr2=0x7fe7e00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:39:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e85ff700 1 -- 172.21.15.102:0/2939853084 shutdown_connections 2023-03-03T08:39:02.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.859+0000 7fe7e85ff700 1 -- 172.21.15.102:0/2939853084 wait complete. 2023-03-03T08:39:02.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.861+0000 7fe7e85ff700 1 Processor -- start 2023-03-03T08:39:02.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.862+0000 7fe7e85ff700 1 -- start start 2023-03-03T08:39:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e85ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e00cd390 0x7fe7e00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:39:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e014f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:39:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e85ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e0153290 0x7fe7e01481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:39:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e85ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e01f2870 con 0x7fe7e00cd390 2023-03-03T08:39:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e85ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e01f2da0 con 0x7fe7e0153290 2023-03-03T08:39:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e014e100 con 0x7fe7e00cd8e0 2023-03-03T08:39:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e00cd390 0x7fe7e00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:39:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e00cd390 0x7fe7e00cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40824/0 (socket says 172.21.15.102:40824) 2023-03-03T08:39:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e639b700 1 -- 172.21.15.102:0/936410384 learned_addr learned my addr 172.21.15.102:0/936410384 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:39:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e5b9a700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e014f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:39:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.863+0000 7fe7e6b9c700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e0153290 0x7fe7e01481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:39:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e639b700 1 -- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 msgr2=0x7fe7e014f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:39:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e639b700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e014f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:39:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e639b700 1 -- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e0153290 msgr2=0x7fe7e01481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:39:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e639b700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e0153290 0x7fe7e01481a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:39:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e639b700 1 -- 172.21.15.102:0/936410384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d00094a0 con 0x7fe7e00cd390 2023-03-03T08:39:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e5b9a700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e014f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:39:02.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e639b700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e00cd390 0x7fe7e00cf0b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00c500 tx=0x7fe7dc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:39:02.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7d6ffd700 1 -- 172.21.15.102:0/936410384 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0052d0 con 0x7fe7e00cd390 2023-03-03T08:39:02.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7d6ffd700 1 -- 172.21.15.102:0/936410384 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc005450 con 0x7fe7e00cd390 2023-03-03T08:39:02.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e00ccde0 con 0x7fe7e00cd390 2023-03-03T08:39:02.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.864+0000 7fe7d6ffd700 1 -- 172.21.15.102:0/936410384 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0038c0 con 0x7fe7e00cd390 2023-03-03T08:39:02.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.865+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e015d160 con 0x7fe7e00cd390 2023-03-03T08:39:02.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.866+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e00cd0c0 con 0x7fe7e00cd390 2023-03-03T08:39:02.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.867+0000 7fe7d6ffd700 1 -- 172.21.15.102:0/936410384 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe7dc019030 con 0x7fe7e00cd390 2023-03-03T08:39:02.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.868+0000 7fe7d6ffd700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7c40682f0 0x7fe7c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:39:02.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.868+0000 7fe7d6ffd700 1 -- 172.21.15.102:0/936410384 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe7dc084ff0 con 0x7fe7e00cd390 2023-03-03T08:39:02.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.869+0000 7fe7e5b9a700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7c40682f0 0x7fe7c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:39:02.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.869+0000 7fe7e5b9a700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7c40682f0 0x7fe7c406a7a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00090b0 tx=0x7fe7d0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:39:02.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:02.873+0000 7fe7d6ffd700 1 -- 172.21.15.102:0/936410384 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7dc051700 con 0x7fe7e00cd390 2023-03-03T08:39:03.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.108+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7e0067ce0 con 0x7fe7e00cd390 2023-03-03T08:39:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.109+0000 7fe7d6ffd700 1 -- 172.21.15.102:0/936410384 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe7dc017090 con 0x7fe7e00cd390 2023-03-03T08:39:03.110 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:39:03.111 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:39:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.111+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7c40682f0 msgr2=0x7fe7c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:39:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.111+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7c40682f0 0x7fe7c406a7a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00090b0 tx=0x7fe7d0008030 comp rx=0 tx=0).stop 2023-03-03T08:39:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e00cd390 msgr2=0x7fe7e00cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:39:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e00cd390 0x7fe7e00cf0b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00c500 tx=0x7fe7dc00c810 comp rx=0 tx=0).stop 2023-03-03T08:39:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 shutdown_connections 2023-03-03T08:39:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7e0153290 0x7fe7e01481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:39:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e00cd390 0x7fe7e00cf0b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:39:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7c40682f0 0x7fe7c406a7a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:39:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 --2- 172.21.15.102:0/936410384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7e00cd8e0 0x7fe7e014f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:39:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 >> 172.21.15.102:0/936410384 conn(0x7fe7e0065fb0 msgr2=0x7fe7e0156bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:39:03.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 shutdown_connections 2023-03-03T08:39:03.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:39:03.112+0000 7fe7e85ff700 1 -- 172.21.15.102:0/936410384 wait complete. 2023-03-03T08:39:03.115 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:39:03.140 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-03T08:39:32.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:32.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T08:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:40:03.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:03.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:14.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:40:14.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.390+0000 7f01191ff700 1 Processor -- start 2023-03-03T08:40:14.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.390+0000 7f01191ff700 1 -- start start 2023-03-03T08:40:14.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.391+0000 7f01191ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0114153290 0x7f0114158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:40:14.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.391+0000 7f01191ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f01141593c0 0x7f01140cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:40:14.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.391+0000 7f01191ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114153660 con 0x7f01140cd8e0 2023-03-03T08:40:14.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.391+0000 7f01191ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011406c360 con 0x7f0114153290 2023-03-03T08:40:14.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.391+0000 7f01191ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011415cea0 con 0x7f01141593c0 2023-03-03T08:40:14.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f011259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f01141593c0 0x7f01140cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:40:14.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f011359e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f01140cd8e0 0x7f01140ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:48262/0 (socket says 172.21.15.102:48262) 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f011359e700 1 -- 172.21.15.102:0/1072720451 learned_addr learned my addr 172.21.15.102:0/1072720451 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f0112d9d700 1 --2- 172.21.15.102:0/1072720451 >> v2:172.21.15.102:3300/0 conn(0x7f0114153290 0x7f0114158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f0111d9b700 1 -- 172.21.15.102:0/1072720451 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495615904 0 0) 0x7f0114153660 con 0x7f01140cd8e0 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f0111d9b700 1 -- 172.21.15.102:0/1072720451 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc0035b0 con 0x7f01140cd8e0 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f011259c700 1 -- 172.21.15.102:0/1072720451 >> v2:172.21.15.102:3300/0 conn(0x7f0114153290 msgr2=0x7f0114158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f011259c700 1 --2- 172.21.15.102:0/1072720451 >> v2:172.21.15.102:3300/0 conn(0x7f0114153290 0x7f0114158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f011259c700 1 -- 172.21.15.102:0/1072720451 >> v1:172.21.15.172:6789/0 conn(0x7f01140cd8e0 legacy=0x7f01140ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:40:14.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.392+0000 7f011259c700 1 -- 172.21.15.102:0/1072720451 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011406c7f0 con 0x7f01141593c0 2023-03-03T08:40:14.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f011259c700 1 --2- 172.21.15.102:0/1072720451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140cde20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0114159fb0 tx=0x7f0108009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7078ef97e591c58d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:40:14.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f0111d9b700 1 -- 172.21.15.102:0/1072720451 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011830 con 0x7f01141593c0 2023-03-03T08:40:14.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f0111d9b700 1 -- 172.21.15.102:0/1072720451 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108011990 con 0x7f01141593c0 2023-03-03T08:40:14.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f0111d9b700 1 -- 172.21.15.102:0/1072720451 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011d20 con 0x7f01141593c0 2023-03-03T08:40:14.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f01191ff700 1 -- 172.21.15.102:0/1072720451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 msgr2=0x7f01140cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:40:14.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f01191ff700 1 --2- 172.21.15.102:0/1072720451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140cde20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0114159fb0 tx=0x7f0108009b20 comp rx=0 tx=0).stop 2023-03-03T08:40:14.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f01191ff700 1 -- 172.21.15.102:0/1072720451 shutdown_connections 2023-03-03T08:40:14.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f01191ff700 1 --2- 172.21.15.102:0/1072720451 >> v2:172.21.15.102:3300/0 conn(0x7f0114153290 0x7f0114158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.393+0000 7f01191ff700 1 --2- 172.21.15.102:0/1072720451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140cde20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.394+0000 7f01191ff700 1 -- 172.21.15.102:0/1072720451 >> 172.21.15.102:0/1072720451 conn(0x7f0114065fb0 msgr2=0x7f01140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:40:14.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.394+0000 7f01191ff700 1 -- 172.21.15.102:0/1072720451 shutdown_connections 2023-03-03T08:40:14.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.394+0000 7f01191ff700 1 -- 172.21.15.102:0/1072720451 wait complete. 2023-03-03T08:40:14.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.396+0000 7f01191ff700 1 Processor -- start 2023-03-03T08:40:14.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.396+0000 7f01191ff700 1 -- start start 2023-03-03T08:40:14.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f01191ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140cd8e0 0x7f01141e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:40:14.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f01191ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0114153290 0x7f01141f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:40:14.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f01191ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:40:14.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f01191ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01141e3900 con 0x7f0114153290 2023-03-03T08:40:14.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f01191ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01141e6130 con 0x7f01141593c0 2023-03-03T08:40:14.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f01191ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c81e0 con 0x7f01140cd8e0 2023-03-03T08:40:14.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0114153290 0x7f01141f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0114153290 0x7f01141f1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51326/0 (socket says 172.21.15.102:51326) 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 -- 172.21.15.102:0/3155564919 learned_addr learned my addr 172.21.15.102:0/3155564919 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f0112d9d700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140cd8e0 0x7f01141e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011359e700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 -- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140cd8e0 msgr2=0x7f01141e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140cd8e0 0x7f01141e5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 -- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 msgr2=0x7f01140c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:40:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011259c700 1 -- 172.21.15.102:0/3155564919 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080094a0 con 0x7f0114153290 2023-03-03T08:40:14.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f0112d9d700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140cd8e0 0x7f01141e5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:40:14.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.397+0000 7f011359e700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:40:14.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.398+0000 7f011259c700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0114153290 0x7f01141f1d00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f01080090b0 tx=0x7f0108004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:40:14.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.398+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3155564919 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01080036a0 con 0x7f0114153290 2023-03-03T08:40:14.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.398+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3155564919 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010801f070 con 0x7f0114153290 2023-03-03T08:40:14.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.398+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3155564919 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108003a40 con 0x7f0114153290 2023-03-03T08:40:14.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.398+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01140ccde0 con 0x7f0114153290 2023-03-03T08:40:14.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.398+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011415d160 con 0x7f0114153290 2023-03-03T08:40:14.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.399+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01140cd0c0 con 0x7f0114153290 2023-03-03T08:40:14.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.400+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3155564919 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f010802d070 con 0x7f0114153290 2023-03-03T08:40:14.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.402+0000 7f00fb7fe700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0100068320 0x7f010006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:40:14.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.402+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3155564919 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0108021070 con 0x7f0114153290 2023-03-03T08:40:14.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.402+0000 7f0112d9d700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0100068320 0x7f010006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:40:14.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.403+0000 7f0112d9d700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0100068320 0x7f010006a7d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0104009c30 tx=0x7f0104008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:40:14.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.406+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3155564919 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0108053960 con 0x7f0114153290 2023-03-03T08:40:14.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.638+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01140c8790 con 0x7f0114153290 2023-03-03T08:40:14.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.638+0000 7f00fb7fe700 1 -- 172.21.15.102:0/3155564919 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0108056f80 con 0x7f0114153290 2023-03-03T08:40:14.641 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:40:14.641 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:40:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0100068320 msgr2=0x7f010006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:40:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0100068320 0x7f010006a7d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0104009c30 tx=0x7f0104008040 comp rx=0 tx=0).stop 2023-03-03T08:40:14.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0114153290 msgr2=0x7f01141f1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:40:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0114153290 0x7f01141f1d00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f01080090b0 tx=0x7f0108004750 comp rx=0 tx=0).stop 2023-03-03T08:40:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 shutdown_connections 2023-03-03T08:40:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0100068320 0x7f010006a7d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01140cd8e0 0x7f01141e5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01141593c0 0x7f01140c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 --2- 172.21.15.102:0/3155564919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0114153290 0x7f01141f1d00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:40:14.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 >> 172.21.15.102:0/3155564919 conn(0x7f0114065fb0 msgr2=0x7f01141510e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:40:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 shutdown_connections 2023-03-03T08:40:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:40:14.641+0000 7f01191ff700 1 -- 172.21.15.102:0/3155564919 wait complete. 2023-03-03T08:40:14.645 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:40:14.670 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2023-03-03T08:40:33.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:33.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:03.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:03.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:11.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:41:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.922+0000 7fb72ffff700 1 Processor -- start 2023-03-03T08:41:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.922+0000 7fb72ffff700 1 -- start start 2023-03-03T08:41:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb728153290 0x7fb728158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:41:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb7281593c0 0x7fb7280cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:41:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb728153660 con 0x7fb7280cd8e0 2023-03-03T08:41:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72ffff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72806c360 con 0x7fb728153290 2023-03-03T08:41:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72ffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72815cea0 con 0x7fb7281593c0 2023-03-03T08:41:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72dd9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb728153290 0x7fb728158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:41:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72dd9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb728153290 0x7fb728158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34370/0 (socket says 172.21.15.102:34370) 2023-03-03T08:41:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72dd9b700 1 -- 172.21.15.102:0/1985633164 learned_addr learned my addr 172.21.15.102:0/1985633164 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:41:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72d59a700 1 --2- 172.21.15.102:0/1985633164 >> v2:172.21.15.172:3300/0 conn(0x7fb7281593c0 0x7fb7280cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:41:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72dd9b700 1 -- 172.21.15.102:0/1985633164 >> v2:172.21.15.172:3300/0 conn(0x7fb7281593c0 msgr2=0x7fb7280cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:41:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72dd9b700 1 --2- 172.21.15.102:0/1985633164 >> v2:172.21.15.172:3300/0 conn(0x7fb7281593c0 0x7fb7280cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:11.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.924+0000 7fb72dd9b700 1 -- 172.21.15.102:0/1985633164 >> v1:172.21.15.172:6789/0 conn(0x7fb7280cd8e0 legacy=0x7fb7280ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:41:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.925+0000 7fb72dd9b700 1 -- 172.21.15.102:0/1985633164 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72806c7f0 con 0x7fb728153290 2023-03-03T08:41:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.925+0000 7fb72dd9b700 1 --2- 172.21.15.102:0/1985633164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb728153290 0x7fb728158e80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb728159fb0 tx=0x7fb724009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f98d7ce389e23ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:41:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.925+0000 7fb72cd99700 1 -- 172.21.15.102:0/1985633164 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011830 con 0x7fb728153290 2023-03-03T08:41:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.925+0000 7fb72cd99700 1 -- 172.21.15.102:0/1985633164 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724011990 con 0x7fb728153290 2023-03-03T08:41:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.925+0000 7fb72cd99700 1 -- 172.21.15.102:0/1985633164 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011d20 con 0x7fb728153290 2023-03-03T08:41:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 -- 172.21.15.102:0/1985633164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb728153290 msgr2=0x7fb728158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:41:11.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 --2- 172.21.15.102:0/1985633164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb728153290 0x7fb728158e80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb728159fb0 tx=0x7fb724009bd0 comp rx=0 tx=0).stop 2023-03-03T08:41:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 -- 172.21.15.102:0/1985633164 shutdown_connections 2023-03-03T08:41:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 --2- 172.21.15.102:0/1985633164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb728153290 0x7fb728158e80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 --2- 172.21.15.102:0/1985633164 >> v2:172.21.15.172:3300/0 conn(0x7fb7281593c0 0x7fb7280cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 -- 172.21.15.102:0/1985633164 >> 172.21.15.102:0/1985633164 conn(0x7fb728065fb0 msgr2=0x7fb7280663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:41:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 -- 172.21.15.102:0/1985633164 shutdown_connections 2023-03-03T08:41:11.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.926+0000 7fb72ffff700 1 -- 172.21.15.102:0/1985633164 wait complete. 2023-03-03T08:41:11.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.929+0000 7fb72ffff700 1 Processor -- start 2023-03-03T08:41:11.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.929+0000 7fb72ffff700 1 -- start start 2023-03-03T08:41:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.930+0000 7fb72ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7280cd8e0 0x7fb7280c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:41:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.930+0000 7fb72ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb728153290 0x7fb728151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:41:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.930+0000 7fb72ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7281593c0 0x7fb7280c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:41:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.930+0000 7fb72ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7280c31f0 con 0x7fb7281593c0 2023-03-03T08:41:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.930+0000 7fb72ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7281f2bd0 con 0x7fb7280cd8e0 2023-03-03T08:41:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.930+0000 7fb72ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7281f2680 con 0x7fb728153290 2023-03-03T08:41:11.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7280cd8e0 0x7fb7280c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:41:11.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72d59a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb728153290 0x7fb728151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:41:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7280cd8e0 0x7fb7280c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48580/0 (socket says 172.21.15.102:48580) 2023-03-03T08:41:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 -- 172.21.15.102:0/191791830 learned_addr learned my addr 172.21.15.102:0/191791830 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:41:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72e59c700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7281593c0 0x7fb7280c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:41:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 -- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb728153290 msgr2=0x7fb728151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:41:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb728153290 0x7fb728151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 -- 172.21.15.102:0/191791830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7281593c0 msgr2=0x7fb7280c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:41:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7281593c0 0x7fb7280c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 -- 172.21.15.102:0/191791830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb724009550 con 0x7fb7280cd8e0 2023-03-03T08:41:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.931+0000 7fb72dd9b700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7280cd8e0 0x7fb7280c24f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb724000c00 tx=0x7fb724004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:41:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.932+0000 7fb71e7fc700 1 -- 172.21.15.102:0/191791830 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724004550 con 0x7fb7280cd8e0 2023-03-03T08:41:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.932+0000 7fb71e7fc700 1 -- 172.21.15.102:0/191791830 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72401f070 con 0x7fb7280cd8e0 2023-03-03T08:41:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.932+0000 7fb71e7fc700 1 -- 172.21.15.102:0/191791830 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7240038e0 con 0x7fb7280cd8e0 2023-03-03T08:41:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.932+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7280ccde0 con 0x7fb7280cd8e0 2023-03-03T08:41:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.932+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72815d160 con 0x7fb7280cd8e0 2023-03-03T08:41:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.933+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7280cd0c0 con 0x7fb7280cd8e0 2023-03-03T08:41:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.934+0000 7fb71e7fc700 1 -- 172.21.15.102:0/191791830 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb72400f050 con 0x7fb7280cd8e0 2023-03-03T08:41:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.935+0000 7fb71e7fc700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c068320 0x7fb70c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:41:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.936+0000 7fb71e7fc700 1 -- 172.21.15.102:0/191791830 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb724021070 con 0x7fb7280cd8e0 2023-03-03T08:41:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.936+0000 7fb72d59a700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c068320 0x7fb70c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:41:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.936+0000 7fb72d59a700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c068320 0x7fb70c06a7d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb7281599c0 tx=0x7fb718009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:41:11.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:11.940+0000 7fb71e7fc700 1 -- 172.21.15.102:0/191791830 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb724053c90 con 0x7fb7280cd8e0 2023-03-03T08:41:12.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.176+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb72804e6e0 con 0x7fb7280cd8e0 2023-03-03T08:41:12.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.177+0000 7fb71e7fc700 1 -- 172.21.15.102:0/191791830 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb724015090 con 0x7fb7280cd8e0 2023-03-03T08:41:12.179 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:41:12.179 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:41:12.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c068320 msgr2=0x7fb70c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:41:12.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c068320 0x7fb70c06a7d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb7281599c0 tx=0x7fb718009210 comp rx=0 tx=0).stop 2023-03-03T08:41:12.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7280cd8e0 msgr2=0x7fb7280c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:41:12.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7280cd8e0 0x7fb7280c24f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb724000c00 tx=0x7fb724004750 comp rx=0 tx=0).stop 2023-03-03T08:41:12.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 shutdown_connections 2023-03-03T08:41:12.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7281593c0 0x7fb7280c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:12.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7280cd8e0 0x7fb7280c24f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:12.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c068320 0x7fb70c06a7d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:12.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 --2- 172.21.15.102:0/191791830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb728153290 0x7fb728151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:41:12.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 >> 172.21.15.102:0/191791830 conn(0x7fb728065fb0 msgr2=0x7fb7280cc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:41:12.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.180+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 shutdown_connections 2023-03-03T08:41:12.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:41:12.181+0000 7fb72ffff700 1 -- 172.21.15.102:0/191791830 wait complete. 2023-03-03T08:41:12.184 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:41:12.209 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T08:41:33.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:33.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:04.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:04.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:14.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:42:14.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.478+0000 7fb92e1ff700 1 Processor -- start 2023-03-03T08:42:14.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.478+0000 7fb92e1ff700 1 -- start start 2023-03-03T08:42:14.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb92e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd8e0 0x7fb9280cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb92e1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb9280cd390 0x7fb9280ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb92e1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb928153660 con 0x7fb928153290 2023-03-03T08:42:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb92e1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92806c360 con 0x7fb9280cd390 2023-03-03T08:42:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb92e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92815cea0 con 0x7fb9280cd8e0 2023-03-03T08:42:14.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb9277fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb928153290 0x7fb928158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45580/0 (socket says 172.21.15.102:45580) 2023-03-03T08:42:14.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb9277fe700 1 -- 172.21.15.102:0/3609462005 learned_addr learned my addr 172.21.15.102:0/3609462005 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:42:14.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb927fff700 1 --2- 172.21.15.102:0/3609462005 >> v2:172.21.15.172:3300/0 conn(0x7fb9280cd390 0x7fb9280ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:14.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.480+0000 7fb926ffd700 1 --2- 172.21.15.102:0/3609462005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd8e0 0x7fb9280cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:14.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.481+0000 7fb9267fc700 1 -- 172.21.15.102:0/3609462005 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509749198 0 0) 0x7fb928153660 con 0x7fb928153290 2023-03-03T08:42:14.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.481+0000 7fb9267fc700 1 -- 172.21.15.102:0/3609462005 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9100035b0 con 0x7fb928153290 2023-03-03T08:42:14.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.481+0000 7fb926ffd700 1 -- 172.21.15.102:0/3609462005 >> v2:172.21.15.172:3300/0 conn(0x7fb9280cd390 msgr2=0x7fb9280ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:42:14.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.481+0000 7fb926ffd700 1 --2- 172.21.15.102:0/3609462005 >> v2:172.21.15.172:3300/0 conn(0x7fb9280cd390 0x7fb9280ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.481+0000 7fb926ffd700 1 -- 172.21.15.102:0/3609462005 >> v1:172.21.15.102:6789/0 conn(0x7fb928153290 legacy=0x7fb928158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.481+0000 7fb926ffd700 1 -- 172.21.15.102:0/3609462005 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92806c7f0 con 0x7fb9280cd8e0 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.481+0000 7fb926ffd700 1 --2- 172.21.15.102:0/3609462005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd8e0 0x7fb9280cde20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009840 tx=0x7fb91c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce8246f95bfd94f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb9267fc700 1 -- 172.21.15.102:0/3609462005 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c011830 con 0x7fb9280cd8e0 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb9267fc700 1 -- 172.21.15.102:0/3609462005 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91c011990 con 0x7fb9280cd8e0 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb9267fc700 1 -- 172.21.15.102:0/3609462005 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c011d00 con 0x7fb9280cd8e0 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 -- 172.21.15.102:0/3609462005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd8e0 msgr2=0x7fb9280cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/3609462005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd8e0 0x7fb9280cde20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009840 tx=0x7fb91c009b50 comp rx=0 tx=0).stop 2023-03-03T08:42:14.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 -- 172.21.15.102:0/3609462005 shutdown_connections 2023-03-03T08:42:14.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/3609462005 >> v2:172.21.15.172:3300/0 conn(0x7fb9280cd390 0x7fb9280ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/3609462005 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd8e0 0x7fb9280cde20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 -- 172.21.15.102:0/3609462005 >> 172.21.15.102:0/3609462005 conn(0x7fb928065fb0 msgr2=0x7fb9280663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:42:14.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 -- 172.21.15.102:0/3609462005 shutdown_connections 2023-03-03T08:42:14.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.482+0000 7fb92e1ff700 1 -- 172.21.15.102:0/3609462005 wait complete. 2023-03-03T08:42:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.484+0000 7fb92e1ff700 1 Processor -- start 2023-03-03T08:42:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.485+0000 7fb92e1ff700 1 -- start start 2023-03-03T08:42:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb92e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd390 0x7fb9281e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb92e1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9280cd8e0 0x7fb9280c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb92e1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb928153290 0x7fb9280c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb92e1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281efa50 con 0x7fb928153290 2023-03-03T08:42:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb92e1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280c6ad0 con 0x7fb9280cd8e0 2023-03-03T08:42:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb92e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280c61e0 con 0x7fb9280cd390 2023-03-03T08:42:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd390 0x7fb9281e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb927fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb928153290 0x7fb9280c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb926ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9280cd8e0 0x7fb9280c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd390 0x7fb9281e4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45394/0 (socket says 172.21.15.102:45394) 2023-03-03T08:42:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 -- 172.21.15.102:0/4090260138 learned_addr learned my addr 172.21.15.102:0/4090260138 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 -- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9280cd8e0 msgr2=0x7fb9280c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9280cd8e0 0x7fb9280c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 -- 172.21.15.102:0/4090260138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb928153290 msgr2=0x7fb9280c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb928153290 0x7fb9280c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.486+0000 7fb9277fe700 1 -- 172.21.15.102:0/4090260138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c0094a0 con 0x7fb9280cd390 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.487+0000 7fb9277fe700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd390 0x7fb9281e4440 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb91800c4d0 tx=0x7fb91800c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.487+0000 7fb90ffff700 1 -- 172.21.15.102:0/4090260138 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918005d70 con 0x7fb9280cd390 2023-03-03T08:42:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.487+0000 7fb90ffff700 1 -- 172.21.15.102:0/4090260138 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9180070d0 con 0x7fb9280cd390 2023-03-03T08:42:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.487+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9280ccde0 con 0x7fb9280cd390 2023-03-03T08:42:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.487+0000 7fb90ffff700 1 -- 172.21.15.102:0/4090260138 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918007440 con 0x7fb9280cd390 2023-03-03T08:42:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.487+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92815d160 con 0x7fb9280cd390 2023-03-03T08:42:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.488+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9280cd0c0 con 0x7fb9280cd390 2023-03-03T08:42:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.490+0000 7fb90ffff700 1 -- 172.21.15.102:0/4090260138 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb918019030 con 0x7fb9280cd390 2023-03-03T08:42:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.491+0000 7fb90ffff700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9140682f0 0x7fb91406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.491+0000 7fb90ffff700 1 -- 172.21.15.102:0/4090260138 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb918084f70 con 0x7fb9280cd390 2023-03-03T08:42:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.491+0000 7fb926ffd700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9140682f0 0x7fb91406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.492+0000 7fb926ffd700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9140682f0 0x7fb91406a7a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb91c000c00 tx=0x7fb91c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:42:14.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.495+0000 7fb90ffff700 1 -- 172.21.15.102:0/4090260138 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb91804b5e0 con 0x7fb9280cd390 2023-03-03T08:42:14.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.726+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9280ca320 con 0x7fb9280cd390 2023-03-03T08:42:14.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.727+0000 7fb90ffff700 1 -- 172.21.15.102:0/4090260138 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb918017020 con 0x7fb9280cd390 2023-03-03T08:42:14.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:42:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:42:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9140682f0 msgr2=0x7fb91406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9140682f0 0x7fb91406a7a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb91c000c00 tx=0x7fb91c01c040 comp rx=0 tx=0).stop 2023-03-03T08:42:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd390 msgr2=0x7fb9281e4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd390 0x7fb9281e4440 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb91800c4d0 tx=0x7fb91800c7e0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 shutdown_connections 2023-03-03T08:42:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9140682f0 0x7fb91406a7a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9280cd390 0x7fb9281e4440 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb928153290 0x7fb9280c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 --2- 172.21.15.102:0/4090260138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9280cd8e0 0x7fb9280c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 >> 172.21.15.102:0/4090260138 conn(0x7fb928065fb0 msgr2=0x7fb9280cbd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:42:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 shutdown_connections 2023-03-03T08:42:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:14.729+0000 7fb92e1ff700 1 -- 172.21.15.102:0/4090260138 wait complete. 2023-03-03T08:42:14.732 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:42:14.757 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T08:42:34.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:34.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:58.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:42:59.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.007+0000 7f9a3b59d700 1 Processor -- start 2023-03-03T08:42:59.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.007+0000 7f9a3b59d700 1 -- start start 2023-03-03T08:42:59.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3b59d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a340ced00 0x7f9a340c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:59.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3b59d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a340b3970 0x7f9a340c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3b59d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a340c3d20 con 0x7f9a340cf0d0 2023-03-03T08:42:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3b59d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a340c4930 con 0x7f9a340b3970 2023-03-03T08:42:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3b59d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a340cf570 con 0x7f9a340ced00 2023-03-03T08:42:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3a59b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a340ced00 0x7f9a340c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a39d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a340b3970 0x7f9a340c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3a59b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a340ced00 0x7f9a340c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43464/0 (socket says 172.21.15.102:43464) 2023-03-03T08:42:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.009+0000 7f9a3a59b700 1 -- 172.21.15.102:0/1500306083 learned_addr learned my addr 172.21.15.102:0/1500306083 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:42:59.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a3a59b700 1 -- 172.21.15.102:0/1500306083 >> v2:172.21.15.102:3300/0 conn(0x7f9a340b3970 msgr2=0x7f9a340c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:42:59.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a3a59b700 1 --2- 172.21.15.102:0/1500306083 >> v2:172.21.15.102:3300/0 conn(0x7f9a340b3970 0x7f9a340c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a3a59b700 1 -- 172.21.15.102:0/1500306083 >> v1:172.21.15.102:6789/0 conn(0x7f9a340cf0d0 legacy=0x7f9a34158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:59.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a3a59b700 1 -- 172.21.15.102:0/1500306083 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3415cba0 con 0x7f9a340ced00 2023-03-03T08:42:59.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a3a59b700 1 --2- 172.21.15.102:0/1500306083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340ced00 0x7f9a340c5a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9a340b1b40 tx=0x7f9a30009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2d5f2638ce4b613 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:42:59.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a39599700 1 -- 172.21.15.102:0/1500306083 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30011830 con 0x7f9a340ced00 2023-03-03T08:42:59.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a39599700 1 -- 172.21.15.102:0/1500306083 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a30011990 con 0x7f9a340ced00 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.010+0000 7f9a39599700 1 -- 172.21.15.102:0/1500306083 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30011d20 con 0x7f9a340ced00 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 -- 172.21.15.102:0/1500306083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340ced00 msgr2=0x7f9a340c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/1500306083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340ced00 0x7f9a340c5a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9a340b1b40 tx=0x7f9a30009bd0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 -- 172.21.15.102:0/1500306083 shutdown_connections 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/1500306083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340ced00 0x7f9a340c5a90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/1500306083 >> v2:172.21.15.102:3300/0 conn(0x7f9a340b3970 0x7f9a340c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 -- 172.21.15.102:0/1500306083 >> 172.21.15.102:0/1500306083 conn(0x7f9a3408b1c0 msgr2=0x7f9a34071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:42:59.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 -- 172.21.15.102:0/1500306083 shutdown_connections 2023-03-03T08:42:59.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.011+0000 7f9a3b59d700 1 -- 172.21.15.102:0/1500306083 wait complete. 2023-03-03T08:42:59.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.013+0000 7f9a3b59d700 1 Processor -- start 2023-03-03T08:42:59.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.013+0000 7f9a3b59d700 1 -- start start 2023-03-03T08:42:59.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.014+0000 7f9a3b59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340b3970 0x7f9a3414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:59.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.014+0000 7f9a3b59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a340ced00 0x7f9a341f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:59.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.014+0000 7f9a3b59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a340cf0d0 0x7f9a34150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:59.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.014+0000 7f9a3b59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a34159cc0 con 0x7f9a340cf0d0 2023-03-03T08:42:59.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.014+0000 7f9a3b59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a34159eb0 con 0x7f9a340b3970 2023-03-03T08:42:59.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.014+0000 7f9a3b59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3415a450 con 0x7f9a340ced00 2023-03-03T08:42:59.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a3a59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340b3970 0x7f9a3414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:59.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a39d9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a340ced00 0x7f9a341f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:59.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a3a59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340b3970 0x7f9a3414cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43468/0 (socket says 172.21.15.102:43468) 2023-03-03T08:42:59.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a3a59b700 1 -- 172.21.15.102:0/2033980577 learned_addr learned my addr 172.21.15.102:0/2033980577 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:42:59.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a3ad9c700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a340cf0d0 0x7f9a34150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:59.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a39d9a700 1 -- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340b3970 msgr2=0x7f9a3414cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:59.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a39d9a700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340b3970 0x7f9a3414cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a39d9a700 1 -- 172.21.15.102:0/2033980577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a340cf0d0 msgr2=0x7f9a34150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:59.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a39d9a700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a340cf0d0 0x7f9a34150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a39d9a700 1 -- 172.21.15.102:0/2033980577 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a30009550 con 0x7f9a340ced00 2023-03-03T08:42:59.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a3a59b700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340b3970 0x7f9a3414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:42:59.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.015+0000 7f9a3ad9c700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a340cf0d0 0x7f9a34150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:42:59.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.016+0000 7f9a39d9a700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a340ced00 0x7f9a341f1b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400e840 tx=0x7f9a2400eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:42:59.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.016+0000 7f9a2affd700 1 -- 172.21.15.102:0/2033980577 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24013680 con 0x7f9a340ced00 2023-03-03T08:42:59.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.016+0000 7f9a2affd700 1 -- 172.21.15.102:0/2033980577 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a240137e0 con 0x7f9a340ced00 2023-03-03T08:42:59.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.016+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3406bf20 con 0x7f9a340ced00 2023-03-03T08:42:59.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.016+0000 7f9a2affd700 1 -- 172.21.15.102:0/2033980577 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24013b50 con 0x7f9a340ced00 2023-03-03T08:42:59.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.016+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3415ce80 con 0x7f9a340ced00 2023-03-03T08:42:59.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.017+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3406c200 con 0x7f9a340ced00 2023-03-03T08:42:59.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.019+0000 7f9a2affd700 1 -- 172.21.15.102:0/2033980577 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a24023070 con 0x7f9a340ced00 2023-03-03T08:42:59.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.020+0000 7f9a2affd700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a180682f0 0x7f9a1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:42:59.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.020+0000 7f9a2affd700 1 -- 172.21.15.102:0/2033980577 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9a24085e00 con 0x7f9a340ced00 2023-03-03T08:42:59.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.020+0000 7f9a3a59b700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a180682f0 0x7f9a1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:42:59.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.021+0000 7f9a3a59b700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a180682f0 0x7f9a1806a7a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009c10 tx=0x7f9a30008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:42:59.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.024+0000 7f9a2affd700 1 -- 172.21.15.102:0/2033980577 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a24052420 con 0x7f9a340ced00 2023-03-03T08:42:59.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.257+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a340c55f0 con 0x7f9a340ced00 2023-03-03T08:42:59.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.258+0000 7f9a2affd700 1 -- 172.21.15.102:0/2033980577 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9a24017090 con 0x7f9a340ced00 2023-03-03T08:42:59.260 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:42:59.260 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:42:59.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.261+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a180682f0 msgr2=0x7f9a1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:59.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.261+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a180682f0 0x7f9a1806a7a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009c10 tx=0x7f9a30008030 comp rx=0 tx=0).stop 2023-03-03T08:42:59.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.261+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a340ced00 msgr2=0x7f9a341f1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:42:59.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.261+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a340ced00 0x7f9a341f1b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400e840 tx=0x7f9a2400eb50 comp rx=0 tx=0).stop 2023-03-03T08:42:59.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 shutdown_connections 2023-03-03T08:42:59.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a340cf0d0 0x7f9a34150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a340b3970 0x7f9a3414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a180682f0 0x7f9a1806a7a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 --2- 172.21.15.102:0/2033980577 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a340ced00 0x7f9a341f1b30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:42:59.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 >> 172.21.15.102:0/2033980577 conn(0x7f9a3408b1c0 msgr2=0x7f9a34065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:42:59.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 shutdown_connections 2023-03-03T08:42:59.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:42:59.262+0000 7f9a3b59d700 1 -- 172.21.15.102:0/2033980577 wait complete. 2023-03-03T08:42:59.265 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:42:59.290 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T08:43:04.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:04.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:34.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:34.429 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:04.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:04.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:34.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:34.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:37.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:37.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.553+0000 7fb09adff700 1 Processor -- start 2023-03-03T08:44:37.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.553+0000 7fb09adff700 1 -- start start 2023-03-03T08:44:37.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.555+0000 7fb09adff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb0940c9cb0 0x7fb0940cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.555+0000 7fb09adff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940cf940 con 0x7fb0940ca190 2023-03-03T08:44:37.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.555+0000 7fb09adff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940cfb90 con 0x7fb094153260 2023-03-03T08:44:37.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.555+0000 7fb09adff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0940cfe70 con 0x7fb0940c9cb0 2023-03-03T08:44:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.556+0000 7fb098b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb094153260 0x7fb0940ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33596/0 (socket says 172.21.15.102:33596) 2023-03-03T08:44:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.556+0000 7fb098b9b700 1 -- 172.21.15.102:0/3309863074 learned_addr learned my addr 172.21.15.102:0/3309863074 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:44:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.556+0000 7fb093fff700 1 --2- 172.21.15.102:0/3309863074 >> v2:172.21.15.172:3300/0 conn(0x7fb0940c9cb0 0x7fb0940cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.556+0000 7fb093fff700 1 -- 172.21.15.102:0/3309863074 >> v1:172.21.15.172:6789/0 conn(0x7fb094153260 legacy=0x7fb0940ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.556+0000 7fb093fff700 1 -- 172.21.15.102:0/3309863074 >> v1:172.21.15.102:6789/0 conn(0x7fb0940ca190 legacy=0x7fb094163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.556+0000 7fb093fff700 1 -- 172.21.15.102:0/3309863074 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094158f80 con 0x7fb0940c9cb0 2023-03-03T08:44:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.557+0000 7fb093fff700 1 --2- 172.21.15.102:0/3309863074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940c9cb0 0x7fb0940cb880 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb080009840 tx=0x7fb080009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5f05766b43612a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.557+0000 7fb0937fe700 1 -- 172.21.15.102:0/3309863074 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080011830 con 0x7fb0940c9cb0 2023-03-03T08:44:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.557+0000 7fb0937fe700 1 -- 172.21.15.102:0/3309863074 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080011990 con 0x7fb0940c9cb0 2023-03-03T08:44:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.557+0000 7fb0937fe700 1 -- 172.21.15.102:0/3309863074 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080011d20 con 0x7fb0940c9cb0 2023-03-03T08:44:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.557+0000 7fb09adff700 1 -- 172.21.15.102:0/3309863074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940c9cb0 msgr2=0x7fb0940cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.557+0000 7fb09adff700 1 --2- 172.21.15.102:0/3309863074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940c9cb0 0x7fb0940cb880 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb080009840 tx=0x7fb080009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.558+0000 7fb09adff700 1 -- 172.21.15.102:0/3309863074 shutdown_connections 2023-03-03T08:44:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.558+0000 7fb09adff700 1 --2- 172.21.15.102:0/3309863074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940c9cb0 0x7fb0940cb880 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.558+0000 7fb09adff700 1 -- 172.21.15.102:0/3309863074 >> 172.21.15.102:0/3309863074 conn(0x7fb094066c00 msgr2=0x7fb0940ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.558+0000 7fb09adff700 1 -- 172.21.15.102:0/3309863074 shutdown_connections 2023-03-03T08:44:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.558+0000 7fb09adff700 1 -- 172.21.15.102:0/3309863074 wait complete. 2023-03-03T08:44:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.560+0000 7fb09adff700 1 Processor -- start 2023-03-03T08:44:37.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.560+0000 7fb09adff700 1 -- start start 2023-03-03T08:44:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.561+0000 7fb09adff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940ca190 0x7fb0941ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.561+0000 7fb09adff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb094153260 0x7fb0941e5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.561+0000 7fb09adff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb094159ed0 0x7fb094155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.561+0000 7fb09adff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094154c40 con 0x7fb094159ed0 2023-03-03T08:44:37.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.561+0000 7fb09adff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940c99e0 con 0x7fb0940ca190 2023-03-03T08:44:37.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.561+0000 7fb09adff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941536e0 con 0x7fb094153260 2023-03-03T08:44:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.561+0000 7fb098b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940ca190 0x7fb0941ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb093fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb094153260 0x7fb0941e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb098b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940ca190 0x7fb0941ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52742/0 (socket says 172.21.15.102:52742) 2023-03-03T08:44:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb098b9b700 1 -- 172.21.15.102:0/824524716 learned_addr learned my addr 172.21.15.102:0/824524716 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:44:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb09939c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb094159ed0 0x7fb094155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb09939c700 1 -- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb094153260 msgr2=0x7fb0941e5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb09939c700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb094153260 0x7fb0941e5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb09939c700 1 -- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940ca190 msgr2=0x7fb0941ee890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb09939c700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940ca190 0x7fb0941ee890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb09939c700 1 -- 172.21.15.102:0/824524716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0800094a0 con 0x7fb094159ed0 2023-03-03T08:44:37.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb098b9b700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940ca190 0x7fb0941ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb093fff700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb094153260 0x7fb0941e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.562+0000 7fb09939c700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb094159ed0 0x7fb094155090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00cbc0 tx=0x7fb08c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.563+0000 7fb0917fa700 1 -- 172.21.15.102:0/824524716 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c003c00 con 0x7fb094159ed0 2023-03-03T08:44:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.563+0000 7fb0917fa700 1 -- 172.21.15.102:0/824524716 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08c003d60 con 0x7fb094159ed0 2023-03-03T08:44:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.563+0000 7fb0917fa700 1 -- 172.21.15.102:0/824524716 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00aa40 con 0x7fb094159ed0 2023-03-03T08:44:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.563+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094159540 con 0x7fb094159ed0 2023-03-03T08:44:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.563+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094159260 con 0x7fb094159ed0 2023-03-03T08:44:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.564+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094159820 con 0x7fb094159ed0 2023-03-03T08:44:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.565+0000 7fb0917fa700 1 -- 172.21.15.102:0/824524716 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb08c010050 con 0x7fb094159ed0 2023-03-03T08:44:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.566+0000 7fb0917fa700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb0780682f0 0x7fb07806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.566+0000 7fb0917fa700 1 -- 172.21.15.102:0/824524716 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb08c085c30 con 0x7fb094159ed0 2023-03-03T08:44:37.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.567+0000 7fb098b9b700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb0780682f0 0x7fb07806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.567+0000 7fb098b9b700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb0780682f0 0x7fb07806a7a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb088005910 tx=0x7fb088009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:37.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.571+0000 7fb0917fa700 1 -- 172.21.15.102:0/824524716 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb08c0522b0 con 0x7fb094159ed0 2023-03-03T08:44:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.802+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb094154d80 con 0x7fb094159ed0 2023-03-03T08:44:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.803+0000 7fb0917fa700 1 -- 172.21.15.102:0/824524716 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb08c0558d0 con 0x7fb094159ed0 2023-03-03T08:44:37.804 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:44:37.804 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:44:37.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb0780682f0 msgr2=0x7fb07806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb0780682f0 0x7fb07806a7a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb088005910 tx=0x7fb088009040 comp rx=0 tx=0).stop 2023-03-03T08:44:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb094159ed0 msgr2=0x7fb094155090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb094159ed0 0x7fb094155090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00cbc0 tx=0x7fb08c00ced0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 shutdown_connections 2023-03-03T08:44:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb094159ed0 0x7fb094155090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0940ca190 0x7fb0941ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb0780682f0 0x7fb07806a7a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 --2- 172.21.15.102:0/824524716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb094153260 0x7fb0941e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.805+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 >> 172.21.15.102:0/824524716 conn(0x7fb0940c9cb0 msgr2=0x7fb0940cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.806+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 shutdown_connections 2023-03-03T08:44:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:44:37.806+0000 7fb09adff700 1 -- 172.21.15.102:0/824524716 wait complete. 2023-03-03T08:44:37.809 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:44:37.834 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T08:45:05.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:05.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:16.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:17.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.098+0000 7f5a249ff700 1 Processor -- start 2023-03-03T08:45:17.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.098+0000 7f5a249ff700 1 -- start start 2023-03-03T08:45:17.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.099+0000 7f5a249ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a200cd8e0 0x7f5a200cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.099+0000 7f5a249ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd390 0x7f5a200ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.099+0000 7f5a249ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a20153660 con 0x7f5a20153290 2023-03-03T08:45:17.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.099+0000 7f5a249ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2006c360 con 0x7f5a200cd8e0 2023-03-03T08:45:17.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.099+0000 7f5a249ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2015cea0 con 0x7f5a200cd390 2023-03-03T08:45:17.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a1e59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5a20153290 0x7f5a20158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60282/0 (socket says 172.21.15.102:60282) 2023-03-03T08:45:17.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a1e59c700 1 -- 172.21.15.102:0/3478542862 learned_addr learned my addr 172.21.15.102:0/3478542862 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:45:17.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a1ed9d700 1 --2- 172.21.15.102:0/3478542862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd390 0x7f5a200ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a17fff700 1 --2- 172.21.15.102:0/3478542862 >> v2:172.21.15.102:3300/0 conn(0x7f5a200cd8e0 0x7f5a200cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a1dd9b700 1 -- 172.21.15.102:0/3478542862 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65074513 0 0) 0x7f5a20153660 con 0x7f5a20153290 2023-03-03T08:45:17.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a1dd9b700 1 -- 172.21.15.102:0/3478542862 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a0c0035b0 con 0x7f5a20153290 2023-03-03T08:45:17.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a17fff700 1 -- 172.21.15.102:0/3478542862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd390 msgr2=0x7f5a200ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:17.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a17fff700 1 --2- 172.21.15.102:0/3478542862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd390 0x7f5a200ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a17fff700 1 -- 172.21.15.102:0/3478542862 >> v1:172.21.15.102:6789/0 conn(0x7f5a20153290 legacy=0x7f5a20158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a17fff700 1 -- 172.21.15.102:0/3478542862 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2006c7f0 con 0x7f5a200cd8e0 2023-03-03T08:45:17.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.100+0000 7f5a1ed9d700 1 --2- 172.21.15.102:0/3478542862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd390 0x7f5a200ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:45:17.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.101+0000 7f5a17fff700 1 --2- 172.21.15.102:0/3478542862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd8e0 0x7f5a200cde20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009840 tx=0x7f5a08009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d3f299b86c0a55 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:17.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.101+0000 7f5a1dd9b700 1 -- 172.21.15.102:0/3478542862 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011830 con 0x7f5a200cd8e0 2023-03-03T08:45:17.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.101+0000 7f5a1dd9b700 1 -- 172.21.15.102:0/3478542862 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08011990 con 0x7f5a200cd8e0 2023-03-03T08:45:17.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.101+0000 7f5a1dd9b700 1 -- 172.21.15.102:0/3478542862 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011d20 con 0x7f5a200cd8e0 2023-03-03T08:45:17.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.101+0000 7f5a249ff700 1 -- 172.21.15.102:0/3478542862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd8e0 msgr2=0x7f5a200cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.101+0000 7f5a249ff700 1 --2- 172.21.15.102:0/3478542862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd8e0 0x7f5a200cde20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009840 tx=0x7f5a08009b50 comp rx=0 tx=0).stop 2023-03-03T08:45:17.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.102+0000 7f5a249ff700 1 -- 172.21.15.102:0/3478542862 shutdown_connections 2023-03-03T08:45:17.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.102+0000 7f5a249ff700 1 --2- 172.21.15.102:0/3478542862 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd390 0x7f5a200ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.102+0000 7f5a249ff700 1 --2- 172.21.15.102:0/3478542862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd8e0 0x7f5a200cde20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.102+0000 7f5a249ff700 1 -- 172.21.15.102:0/3478542862 >> 172.21.15.102:0/3478542862 conn(0x7f5a20065fb0 msgr2=0x7f5a200663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:17.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.102+0000 7f5a249ff700 1 -- 172.21.15.102:0/3478542862 shutdown_connections 2023-03-03T08:45:17.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.102+0000 7f5a249ff700 1 -- 172.21.15.102:0/3478542862 wait complete. 2023-03-03T08:45:17.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.104+0000 7f5a249ff700 1 Processor -- start 2023-03-03T08:45:17.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.104+0000 7f5a249ff700 1 -- start start 2023-03-03T08:45:17.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.105+0000 7f5a249ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd390 0x7f5a200b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.105+0000 7f5a249ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd8e0 0x7f5a20158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.105+0000 7f5a249ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a20153290 0x7f5a20155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.105+0000 7f5a249ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a200b36d0 con 0x7f5a200cd390 2023-03-03T08:45:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.105+0000 7f5a249ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a201550b0 con 0x7f5a20153290 2023-03-03T08:45:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.105+0000 7f5a249ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a200b3ee0 con 0x7f5a200cd8e0 2023-03-03T08:45:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.105+0000 7f5a1e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd390 0x7f5a200b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd390 0x7f5a200b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43620/0 (socket says 172.21.15.102:43620) 2023-03-03T08:45:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 -- 172.21.15.102:0/1977956675 learned_addr learned my addr 172.21.15.102:0/1977956675 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:45:17.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1ed9d700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a20153290 0x7f5a20155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a17fff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd8e0 0x7f5a20158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 -- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd8e0 msgr2=0x7f5a20158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd8e0 0x7f5a20158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 -- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a20153290 msgr2=0x7f5a20155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a20153290 0x7f5a20155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 -- 172.21.15.102:0/1977956675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18009a70 con 0x7f5a200cd390 2023-03-03T08:45:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.106+0000 7f5a1e59c700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd390 0x7f5a200b4650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c730 tx=0x7f5a1000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.107+0000 7f5a16ffd700 1 -- 172.21.15.102:0/1977956675 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10005d70 con 0x7f5a200cd390 2023-03-03T08:45:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.107+0000 7f5a16ffd700 1 -- 172.21.15.102:0/1977956675 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a100070d0 con 0x7f5a200cd390 2023-03-03T08:45:17.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.107+0000 7f5a16ffd700 1 -- 172.21.15.102:0/1977956675 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10007460 con 0x7f5a200cd390 2023-03-03T08:45:17.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.107+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a080094a0 con 0x7f5a200cd390 2023-03-03T08:45:17.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.107+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a200ccde0 con 0x7f5a200cd390 2023-03-03T08:45:17.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.108+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2015d160 con 0x7f5a200cd390 2023-03-03T08:45:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.109+0000 7f5a16ffd700 1 -- 172.21.15.102:0/1977956675 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a10019030 con 0x7f5a200cd390 2023-03-03T08:45:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.110+0000 7f5a16ffd700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a000682f0 0x7f5a0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.111+0000 7f5a16ffd700 1 -- 172.21.15.102:0/1977956675 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5a10085120 con 0x7f5a200cd390 2023-03-03T08:45:17.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.111+0000 7f5a17fff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a000682f0 0x7f5a0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.111+0000 7f5a17fff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a000682f0 0x7f5a0006a7a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5a08000c00 tx=0x7f5a0801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:17.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.115+0000 7f5a16ffd700 1 -- 172.21.15.102:0/1977956675 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a10051830 con 0x7f5a200cd390 2023-03-03T08:45:17.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.349+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a200cd0c0 con 0x7f5a200cd390 2023-03-03T08:45:17.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.349+0000 7f5a16ffd700 1 -- 172.21.15.102:0/1977956675 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5a10054e50 con 0x7f5a200cd390 2023-03-03T08:45:17.351 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:45:17.351 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:45:17.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a000682f0 msgr2=0x7f5a0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a000682f0 0x7f5a0006a7a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5a08000c00 tx=0x7f5a0801c040 comp rx=0 tx=0).stop 2023-03-03T08:45:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd390 msgr2=0x7f5a200b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd390 0x7f5a200b4650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c730 tx=0x7f5a1000ca40 comp rx=0 tx=0).stop 2023-03-03T08:45:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 shutdown_connections 2023-03-03T08:45:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a20153290 0x7f5a20155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200cd390 0x7f5a200b4650 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a000682f0 0x7f5a0006a7a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 --2- 172.21.15.102:0/1977956675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5a200cd8e0 0x7f5a20158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 >> 172.21.15.102:0/1977956675 conn(0x7f5a20065fb0 msgr2=0x7f5a20073810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:17.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 shutdown_connections 2023-03-03T08:45:17.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:45:17.352+0000 7f5a249ff700 1 -- 172.21.15.102:0/1977956675 wait complete. 2023-03-03T08:45:17.356 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:45:17.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T08:45:35.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:35.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:05.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:05.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:05.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:46:05.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.640+0000 7fa5de5ff700 1 Processor -- start 2023-03-03T08:46:05.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.640+0000 7fa5de5ff700 1 -- start start 2023-03-03T08:46:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5de5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa5d81531c0 0x7fa5d815c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5de5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa5d80c66d0 0x7fa5d80c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5de5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80723d0 con 0x7fa5d815c890 2023-03-03T08:46:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5de5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d8153590 con 0x7fa5d81531c0 2023-03-03T08:46:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5de5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d8072860 con 0x7fa5d80c66d0 2023-03-03T08:46:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5d7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa5d81531c0 0x7fa5d815c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:05.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5d7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa5d81531c0 0x7fa5d815c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37466/0 (socket says 172.21.15.102:37466) 2023-03-03T08:46:05.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5d7fff700 1 -- 172.21.15.102:0/563367689 learned_addr learned my addr 172.21.15.102:0/563367689 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:46:05.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.642+0000 7fa5dcb9c700 1 --2- 172.21.15.102:0/563367689 >> v2:172.21.15.172:3300/0 conn(0x7fa5d80c66d0 0x7fa5d80c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:05.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d7fff700 1 -- 172.21.15.102:0/563367689 >> v2:172.21.15.172:3300/0 conn(0x7fa5d80c66d0 msgr2=0x7fa5d80c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:46:05.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d7fff700 1 --2- 172.21.15.102:0/563367689 >> v2:172.21.15.172:3300/0 conn(0x7fa5d80c66d0 0x7fa5d80c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d7fff700 1 -- 172.21.15.102:0/563367689 >> v1:172.21.15.172:6789/0 conn(0x7fa5d815c890 legacy=0x7fa5d80c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:05.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d7fff700 1 -- 172.21.15.102:0/563367689 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d80c8c50 con 0x7fa5d81531c0 2023-03-03T08:46:05.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d7fff700 1 --2- 172.21.15.102:0/563367689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 0x7fa5d815c350 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80c99b0 tx=0x7fa5c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3e5ea07276f160 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:46:05.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d6ffd700 1 -- 172.21.15.102:0/563367689 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8011830 con 0x7fa5d81531c0 2023-03-03T08:46:05.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d6ffd700 1 -- 172.21.15.102:0/563367689 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c8011990 con 0x7fa5d81531c0 2023-03-03T08:46:05.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.643+0000 7fa5d6ffd700 1 -- 172.21.15.102:0/563367689 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8011d20 con 0x7fa5d81531c0 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 -- 172.21.15.102:0/563367689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 msgr2=0x7fa5d815c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/563367689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 0x7fa5d815c350 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80c99b0 tx=0x7fa5c8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 -- 172.21.15.102:0/563367689 shutdown_connections 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/563367689 >> v2:172.21.15.172:3300/0 conn(0x7fa5d80c66d0 0x7fa5d80c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/563367689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 0x7fa5d815c350 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 -- 172.21.15.102:0/563367689 >> 172.21.15.102:0/563367689 conn(0x7fa5d806b410 msgr2=0x7fa5d806b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 -- 172.21.15.102:0/563367689 shutdown_connections 2023-03-03T08:46:05.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.644+0000 7fa5de5ff700 1 -- 172.21.15.102:0/563367689 wait complete. 2023-03-03T08:46:05.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.646+0000 7fa5de5ff700 1 Processor -- start 2023-03-03T08:46:05.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.646+0000 7fa5de5ff700 1 -- start start 2023-03-03T08:46:05.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.647+0000 7fa5de5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5d80c66d0 0x7fa5d81eefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:05.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.647+0000 7fa5de5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 0x7fa5d81ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:05.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.647+0000 7fa5de5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5d815c890 0x7fa5d81f1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:05.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.647+0000 7fa5de5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81ee510 con 0x7fa5d81531c0 2023-03-03T08:46:05.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.647+0000 7fa5de5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81f2220 con 0x7fa5d80c66d0 2023-03-03T08:46:05.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.647+0000 7fa5de5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81f2500 con 0x7fa5d815c890 2023-03-03T08:46:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5d80c66d0 0x7fa5d81eefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5d80c66d0 0x7fa5d81eefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55336/0 (socket says 172.21.15.102:55336) 2023-03-03T08:46:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 -- 172.21.15.102:0/3829051129 learned_addr learned my addr 172.21.15.102:0/3829051129 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:46:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5dcb9c700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5d815c890 0x7fa5d81f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d77fe700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 0x7fa5d81ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 -- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5d815c890 msgr2=0x7fa5d81f1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5d815c890 0x7fa5d81f1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 -- 172.21.15.102:0/3829051129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 msgr2=0x7fa5d81ef870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 0x7fa5d81ef870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5d7fff700 1 -- 172.21.15.102:0/3829051129 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8009550 con 0x7fa5d80c66d0 2023-03-03T08:46:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.648+0000 7fa5dcb9c700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5d815c890 0x7fa5d81f1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:46:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.649+0000 7fa5d7fff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5d80c66d0 0x7fa5d81eefb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8000c00 tx=0x7fa5c8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:46:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.649+0000 7fa5d4ff9700 1 -- 172.21.15.102:0/3829051129 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8004380 con 0x7fa5d80c66d0 2023-03-03T08:46:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.649+0000 7fa5d4ff9700 1 -- 172.21.15.102:0/3829051129 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c801f070 con 0x7fa5d80c66d0 2023-03-03T08:46:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.649+0000 7fa5d4ff9700 1 -- 172.21.15.102:0/3829051129 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80036a0 con 0x7fa5d80c66d0 2023-03-03T08:46:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.649+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d8071aa0 con 0x7fa5d80c66d0 2023-03-03T08:46:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.649+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d80c8f30 con 0x7fa5d80c66d0 2023-03-03T08:46:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.650+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d8071d80 con 0x7fa5d80c66d0 2023-03-03T08:46:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.651+0000 7fa5d4ff9700 1 -- 172.21.15.102:0/3829051129 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa5c802a020 con 0x7fa5d80c66d0 2023-03-03T08:46:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.653+0000 7fa5d4ff9700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5c40682f0 0x7fa5c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.653+0000 7fa5d4ff9700 1 -- 172.21.15.102:0/3829051129 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa5c80872f0 con 0x7fa5d80c66d0 2023-03-03T08:46:05.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.653+0000 7fa5d77fe700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5c40682f0 0x7fa5c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:05.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.653+0000 7fa5d77fe700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5c40682f0 0x7fa5c406a7a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa5d814fc80 tx=0x7fa5cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:46:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.657+0000 7fa5d4ff9700 1 -- 172.21.15.102:0/3829051129 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5c8053a00 con 0x7fa5d80c66d0 2023-03-03T08:46:05.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.890+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5d80c5f50 con 0x7fa5d80c66d0 2023-03-03T08:46:05.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.891+0000 7fa5d4ff9700 1 -- 172.21.15.102:0/3829051129 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa5c8006d80 con 0x7fa5d80c66d0 2023-03-03T08:46:05.893 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:46:05.893 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:46:05.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5c40682f0 msgr2=0x7fa5c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:05.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5c40682f0 0x7fa5c406a7a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa5d814fc80 tx=0x7fa5cc009040 comp rx=0 tx=0).stop 2023-03-03T08:46:05.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5d80c66d0 msgr2=0x7fa5d81eefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:05.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5d80c66d0 0x7fa5d81eefb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8000c00 tx=0x7fa5c8004750 comp rx=0 tx=0).stop 2023-03-03T08:46:05.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 shutdown_connections 2023-03-03T08:46:05.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5c40682f0 0x7fa5c406a7a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5d815c890 0x7fa5d81f1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5d80c66d0 0x7fa5d81eefb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 --2- 172.21.15.102:0/3829051129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5d81531c0 0x7fa5d81ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:05.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 >> 172.21.15.102:0/3829051129 conn(0x7fa5d806b410 msgr2=0x7fa5d80c37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:46:05.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 shutdown_connections 2023-03-03T08:46:05.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:46:05.894+0000 7fa5de5ff700 1 -- 172.21.15.102:0/3829051129 wait complete. 2023-03-03T08:46:05.897 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:46:05.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 65 secs before thrashing 2023-03-03T08:46:35.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:35.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:05.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:05.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:10.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:47:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.188+0000 7fa4f85ff700 1 Processor -- start 2023-03-03T08:47:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.188+0000 7fa4f85ff700 1 -- start start 2023-03-03T08:47:11.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.189+0000 7fa4f85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:11.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.189+0000 7fa4f85ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa4f00cd390 0x7fa4f00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:11.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.189+0000 7fa4f85ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f0153660 con 0x7fa4f0153290 2023-03-03T08:47:11.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.189+0000 7fa4f85ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f006c360 con 0x7fa4f00cd390 2023-03-03T08:47:11.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.189+0000 7fa4f85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f015cea0 con 0x7fa4f00cd8e0 2023-03-03T08:47:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f639b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa4f0153290 0x7fa4f0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49492/0 (socket says 172.21.15.102:49492) 2023-03-03T08:47:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f639b700 1 -- 172.21.15.102:0/2590061802 learned_addr learned my addr 172.21.15.102:0/2590061802 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:47:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f5b9a700 1 --2- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f6b9c700 1 --2- 172.21.15.102:0/2590061802 >> v2:172.21.15.172:3300/0 conn(0x7fa4f00cd390 0x7fa4f00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f5399700 1 -- 172.21.15.102:0/2590061802 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755872270 0 0) 0x7fa4f0153660 con 0x7fa4f0153290 2023-03-03T08:47:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f5399700 1 -- 172.21.15.102:0/2590061802 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc0035b0 con 0x7fa4f0153290 2023-03-03T08:47:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f6b9c700 1 -- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 msgr2=0x7fa4f00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:47:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f6b9c700 1 --2- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f6b9c700 1 -- 172.21.15.102:0/2590061802 >> v1:172.21.15.172:6789/0 conn(0x7fa4f0153290 legacy=0x7fa4f0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.190+0000 7fa4f6b9c700 1 -- 172.21.15.102:0/2590061802 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f006c7f0 con 0x7fa4f00cd390 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f6b9c700 1 --2- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f00ce360 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009d90 tx=0x7fa4e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2cad1fad8179cad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f5399700 1 -- 172.21.15.102:0/2590061802 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e800e070 con 0x7fa4f00cd390 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f5399700 1 -- 172.21.15.102:0/2590061802 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e800bb20 con 0x7fa4f00cd390 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f5399700 1 -- 172.21.15.102:0/2590061802 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8012430 con 0x7fa4f00cd390 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f85ff700 1 -- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 msgr2=0x7fa4f00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f00ce360 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009d90 tx=0x7fa4e800b330 comp rx=0 tx=0).stop 2023-03-03T08:47:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f85ff700 1 -- 172.21.15.102:0/2590061802 shutdown_connections 2023-03-03T08:47:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f00ce360 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/2590061802 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.191+0000 7fa4f85ff700 1 -- 172.21.15.102:0/2590061802 >> 172.21.15.102:0/2590061802 conn(0x7fa4f0065fb0 msgr2=0x7fa4f00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:47:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.192+0000 7fa4f85ff700 1 -- 172.21.15.102:0/2590061802 shutdown_connections 2023-03-03T08:47:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.192+0000 7fa4f85ff700 1 -- 172.21.15.102:0/2590061802 wait complete. 2023-03-03T08:47:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.194+0000 7fa4f85ff700 1 Processor -- start 2023-03-03T08:47:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.194+0000 7fa4f85ff700 1 -- start start 2023-03-03T08:47:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f85ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f01ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f01efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f85ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4f0153290 0x7fa4f0150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f85ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f00c8660 con 0x7fa4f0153290 2023-03-03T08:47:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f85ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f00c6f70 con 0x7fa4f00cd390 2023-03-03T08:47:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f00c7250 con 0x7fa4f00cd8e0 2023-03-03T08:47:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f5b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f01efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f639b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f01ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f6b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4f0153290 0x7fa4f0150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f5b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f01efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48442/0 (socket says 172.21.15.102:48442) 2023-03-03T08:47:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.195+0000 7fa4f5b9a700 1 -- 172.21.15.102:0/1823654145 learned_addr learned my addr 172.21.15.102:0/1823654145 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:47:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f639b700 1 -- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 msgr2=0x7fa4f01efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f639b700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f01efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f639b700 1 -- 172.21.15.102:0/1823654145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4f0153290 msgr2=0x7fa4f0150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f639b700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4f0153290 0x7fa4f0150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f639b700 1 -- 172.21.15.102:0/1823654145 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e8009a70 con 0x7fa4f00cd390 2023-03-03T08:47:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f5b9a700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f01efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:47:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f6b9c700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4f0153290 0x7fa4f0150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:47:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f639b700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f01ef820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c490 tx=0x7fa4ec00c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:11.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4e6ffd700 1 -- 172.21.15.102:0/1823654145 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec005d70 con 0x7fa4f00cd390 2023-03-03T08:47:11.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4e6ffd700 1 -- 172.21.15.102:0/1823654145 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec0070d0 con 0x7fa4f00cd390 2023-03-03T08:47:11.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4e6ffd700 1 -- 172.21.15.102:0/1823654145 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec007440 con 0x7fa4f00cd390 2023-03-03T08:47:11.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f00ccde0 con 0x7fa4f00cd390 2023-03-03T08:47:11.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.196+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f015d160 con 0x7fa4f00cd390 2023-03-03T08:47:11.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.197+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f00cd0c0 con 0x7fa4f00cd390 2023-03-03T08:47:11.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.199+0000 7fa4e6ffd700 1 -- 172.21.15.102:0/1823654145 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa4ec010070 con 0x7fa4f00cd390 2023-03-03T08:47:11.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.200+0000 7fa4e6ffd700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4d4068320 0x7fa4d406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:11.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.200+0000 7fa4e6ffd700 1 -- 172.21.15.102:0/1823654145 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa4ec084bb0 con 0x7fa4f00cd390 2023-03-03T08:47:11.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.200+0000 7fa4f5b9a700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4d4068320 0x7fa4d406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:11.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.201+0000 7fa4f5b9a700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4d4068320 0x7fa4d406a7d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0073980 tx=0x7fa4e0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:11.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.204+0000 7fa4e6ffd700 1 -- 172.21.15.102:0/1823654145 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa4ec04a5e0 con 0x7fa4f00cd390 2023-03-03T08:47:11.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.437+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4f004e6e0 con 0x7fa4f00cd390 2023-03-03T08:47:11.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.439+0000 7fa4e6ffd700 1 -- 172.21.15.102:0/1823654145 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa4ec051370 con 0x7fa4f00cd390 2023-03-03T08:47:11.441 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:47:11.441 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:47:11.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4d4068320 msgr2=0x7fa4d406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4d4068320 0x7fa4d406a7d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0073980 tx=0x7fa4e0009210 comp rx=0 tx=0).stop 2023-03-03T08:47:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 msgr2=0x7fa4f01ef820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f01ef820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c490 tx=0x7fa4ec00c7a0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 shutdown_connections 2023-03-03T08:47:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4f0153290 0x7fa4f0150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4f00cd390 0x7fa4f01ef820 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4d4068320 0x7fa4d406a7d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 --2- 172.21.15.102:0/1823654145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4f00cd8e0 0x7fa4f01efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:11.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.441+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 >> 172.21.15.102:0/1823654145 conn(0x7fa4f0065fb0 msgr2=0x7fa4f0157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:47:11.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.442+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 shutdown_connections 2023-03-03T08:47:11.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:47:11.442+0000 7fa4f85ff700 1 -- 172.21.15.102:0/1823654145 wait complete. 2023-03-03T08:47:11.445 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:47:11.471 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 54 secs before thrashing 2023-03-03T08:47:35.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:35.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:05.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:48:05.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.718+0000 7f4792fff700 1 Processor -- start 2023-03-03T08:48:05.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.718+0000 7f4792fff700 1 -- start start 2023-03-03T08:48:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.719+0000 7f4792fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f478c153290 0x7f478c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:05.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.719+0000 7f4792fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c0cd8e0 0x7f478c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.719+0000 7f4792fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c153660 con 0x7f478c1593c0 2023-03-03T08:48:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.719+0000 7f4792fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478c06c360 con 0x7f478c153290 2023-03-03T08:48:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.719+0000 7f4792fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c15cea0 con 0x7f478c0cd8e0 2023-03-03T08:48:05.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f4790d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f478c153290 0x7f478c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f4790d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f478c153290 0x7f478c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35588/0 (socket says 172.21.15.102:35588) 2023-03-03T08:48:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f479159c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c0cd8e0 0x7f478c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f4790d9b700 1 -- 172.21.15.102:0/1331299562 learned_addr learned my addr 172.21.15.102:0/1331299562 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:48:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f4790d9b700 1 -- 172.21.15.102:0/1331299562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c0cd8e0 msgr2=0x7f478c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:48:05.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f4790d9b700 1 --2- 172.21.15.102:0/1331299562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c0cd8e0 0x7f478c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f4790d9b700 1 -- 172.21.15.102:0/1331299562 >> v1:172.21.15.102:6789/0 conn(0x7f478c1593c0 legacy=0x7f478c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.720+0000 7f4790d9b700 1 -- 172.21.15.102:0/1331299562 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c06c7f0 con 0x7f478c153290 2023-03-03T08:48:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.721+0000 7f4790d9b700 1 --2- 172.21.15.102:0/1331299562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c158e80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f47800098f0 tx=0x7f4780009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f819cda8ebce148 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:48:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.721+0000 7f478b7fe700 1 -- 172.21.15.102:0/1331299562 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780011830 con 0x7f478c153290 2023-03-03T08:48:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.721+0000 7f478b7fe700 1 -- 172.21.15.102:0/1331299562 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4780011990 con 0x7f478c153290 2023-03-03T08:48:05.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.721+0000 7f478b7fe700 1 -- 172.21.15.102:0/1331299562 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780011d20 con 0x7f478c153290 2023-03-03T08:48:05.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.721+0000 7f4792fff700 1 -- 172.21.15.102:0/1331299562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 msgr2=0x7f478c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:05.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.721+0000 7f4792fff700 1 --2- 172.21.15.102:0/1331299562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c158e80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f47800098f0 tx=0x7f4780009c00 comp rx=0 tx=0).stop 2023-03-03T08:48:05.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.721+0000 7f4792fff700 1 -- 172.21.15.102:0/1331299562 shutdown_connections 2023-03-03T08:48:05.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.722+0000 7f4792fff700 1 --2- 172.21.15.102:0/1331299562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c0cd8e0 0x7f478c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.722+0000 7f4792fff700 1 --2- 172.21.15.102:0/1331299562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c158e80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.722+0000 7f4792fff700 1 -- 172.21.15.102:0/1331299562 >> 172.21.15.102:0/1331299562 conn(0x7f478c065fb0 msgr2=0x7f478c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:48:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.722+0000 7f4792fff700 1 -- 172.21.15.102:0/1331299562 shutdown_connections 2023-03-03T08:48:05.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.722+0000 7f4792fff700 1 -- 172.21.15.102:0/1331299562 wait complete. 2023-03-03T08:48:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.724+0000 7f4792fff700 1 Processor -- start 2023-03-03T08:48:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.724+0000 7f4792fff700 1 -- start start 2023-03-03T08:48:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f4792fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f478c0cd8e0 0x7f478c156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:05.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f4792fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c15b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f4792fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c1593c0 0x7f478c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f4792fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c0b44e0 con 0x7f478c153290 2023-03-03T08:48:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f4792fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c1ee5d0 con 0x7f478c0cd8e0 2023-03-03T08:48:05.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f4792fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c1ee8b0 con 0x7f478c1593c0 2023-03-03T08:48:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f478bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c15b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:05.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f4790d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f478c0cd8e0 0x7f478c156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f479159c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c1593c0 0x7f478c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f478bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c15b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35600/0 (socket says 172.21.15.102:35600) 2023-03-03T08:48:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.725+0000 7f478bfff700 1 -- 172.21.15.102:0/2039002839 learned_addr learned my addr 172.21.15.102:0/2039002839 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:48:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f4790d9b700 1 -- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c1593c0 msgr2=0x7f478c150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f4790d9b700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c1593c0 0x7f478c150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f4790d9b700 1 -- 172.21.15.102:0/2039002839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 msgr2=0x7f478c15b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f4790d9b700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c15b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f4790d9b700 1 -- 172.21.15.102:0/2039002839 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780009550 con 0x7f478c0cd8e0 2023-03-03T08:48:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f4790d9b700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f478c0cd8e0 0x7f478c156290 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4780016000 tx=0x7f4780004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:48:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f47897fa700 1 -- 172.21.15.102:0/2039002839 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780004510 con 0x7f478c0cd8e0 2023-03-03T08:48:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f47897fa700 1 -- 172.21.15.102:0/2039002839 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478001e070 con 0x7f478c0cd8e0 2023-03-03T08:48:05.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.726+0000 7f47897fa700 1 -- 172.21.15.102:0/2039002839 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47800038c0 con 0x7f478c0cd8e0 2023-03-03T08:48:05.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.727+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c0ccde0 con 0x7f478c0cd8e0 2023-03-03T08:48:05.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.727+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c15d160 con 0x7f478c0cd8e0 2023-03-03T08:48:05.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.728+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c0cd0c0 con 0x7f478c0cd8e0 2023-03-03T08:48:05.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.729+0000 7f47897fa700 1 -- 172.21.15.102:0/2039002839 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f478002a020 con 0x7f478c0cd8e0 2023-03-03T08:48:05.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.730+0000 7f47897fa700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47700682f0 0x7f477006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:05.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.730+0000 7f47897fa700 1 -- 172.21.15.102:0/2039002839 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4780086d60 con 0x7f478c0cd8e0 2023-03-03T08:48:05.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.731+0000 7f478bfff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47700682f0 0x7f477006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:05.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.731+0000 7f478bfff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47700682f0 0x7f477006a7a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f478c15b890 tx=0x7f4778009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:48:05.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.735+0000 7f47897fa700 1 -- 172.21.15.102:0/2039002839 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47800533f0 con 0x7f478c0cd8e0 2023-03-03T08:48:05.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.969+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f478c1ee710 con 0x7f478c0cd8e0 2023-03-03T08:48:05.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.971+0000 7f47897fa700 1 -- 172.21.15.102:0/2039002839 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f478c1ee710 con 0x7f478c0cd8e0 2023-03-03T08:48:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:48:05.972 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:48:05.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47700682f0 msgr2=0x7f477006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:05.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47700682f0 0x7f477006a7a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f478c15b890 tx=0x7f4778009040 comp rx=0 tx=0).stop 2023-03-03T08:48:05.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f478c0cd8e0 msgr2=0x7f478c156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:05.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f478c0cd8e0 0x7f478c156290 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4780016000 tx=0x7f4780004750 comp rx=0 tx=0).stop 2023-03-03T08:48:05.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 shutdown_connections 2023-03-03T08:48:05.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47700682f0 0x7f477006a7a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f478c1593c0 0x7f478c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f478c0cd8e0 0x7f478c156290 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 --2- 172.21.15.102:0/2039002839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c153290 0x7f478c15b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:05.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 >> 172.21.15.102:0/2039002839 conn(0x7f478c065fb0 msgr2=0x7f478c156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:48:05.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 shutdown_connections 2023-03-03T08:48:05.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:48:05.973+0000 7f4792fff700 1 -- 172.21.15.102:0/2039002839 wait complete. 2023-03-03T08:48:05.977 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:48:06.002 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 78 secs before thrashing 2023-03-03T08:48:06.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:06.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:36.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:36.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:06.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:06.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:24.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:49:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.253+0000 7fc9cb9ff700 1 Processor -- start 2023-03-03T08:49:24.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.253+0000 7fc9cb9ff700 1 -- start start 2023-03-03T08:49:24.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.254+0000 7fc9cb9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd8e0 0x7fc9c40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:24.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.254+0000 7fc9cb9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9c40cd390 0x7fc9c40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:24.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.254+0000 7fc9cb9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c4153660 con 0x7fc9c4153290 2023-03-03T08:49:24.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.254+0000 7fc9cb9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c406c360 con 0x7fc9c40cd390 2023-03-03T08:49:24.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.254+0000 7fc9cb9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c415cea0 con 0x7fc9c40cd8e0 2023-03-03T08:49:24.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c979b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc9c4153290 0x7fc9c4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50160/0 (socket says 172.21.15.102:50160) 2023-03-03T08:49:24.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c979b700 1 -- 172.21.15.102:0/302290455 learned_addr learned my addr 172.21.15.102:0/302290455 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:49:24.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c8f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd8e0 0x7fc9c40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c9f9c700 1 --2- 172.21.15.102:0/302290455 >> v2:172.21.15.102:3300/0 conn(0x7fc9c40cd390 0x7fc9c40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9bbfff700 1 -- 172.21.15.102:0/302290455 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545418989 0 0) 0x7fc9c4153660 con 0x7fc9c4153290 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9bbfff700 1 -- 172.21.15.102:0/302290455 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b00035b0 con 0x7fc9c4153290 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c8f9a700 1 -- 172.21.15.102:0/302290455 >> v2:172.21.15.102:3300/0 conn(0x7fc9c40cd390 msgr2=0x7fc9c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c8f9a700 1 --2- 172.21.15.102:0/302290455 >> v2:172.21.15.102:3300/0 conn(0x7fc9c40cd390 0x7fc9c40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c8f9a700 1 -- 172.21.15.102:0/302290455 >> v1:172.21.15.172:6789/0 conn(0x7fc9c4153290 legacy=0x7fc9c4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.255+0000 7fc9c8f9a700 1 -- 172.21.15.102:0/302290455 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c406c7f0 con 0x7fc9c40cd8e0 2023-03-03T08:49:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.256+0000 7fc9c8f9a700 1 --2- 172.21.15.102:0/302290455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd8e0 0x7fc9c40cde20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009840 tx=0x7fc9b4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee01f2aecbc379dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:49:24.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.256+0000 7fc9bbfff700 1 -- 172.21.15.102:0/302290455 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4011830 con 0x7fc9c40cd8e0 2023-03-03T08:49:24.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.256+0000 7fc9bbfff700 1 -- 172.21.15.102:0/302290455 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b4011990 con 0x7fc9c40cd8e0 2023-03-03T08:49:24.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.256+0000 7fc9bbfff700 1 -- 172.21.15.102:0/302290455 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4011d20 con 0x7fc9c40cd8e0 2023-03-03T08:49:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/302290455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd8e0 msgr2=0x7fc9c40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/302290455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd8e0 0x7fc9c40cde20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009840 tx=0x7fc9b4009b50 comp rx=0 tx=0).stop 2023-03-03T08:49:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/302290455 shutdown_connections 2023-03-03T08:49:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/302290455 >> v2:172.21.15.102:3300/0 conn(0x7fc9c40cd390 0x7fc9c40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/302290455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd8e0 0x7fc9c40cde20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/302290455 >> 172.21.15.102:0/302290455 conn(0x7fc9c4065fb0 msgr2=0x7fc9c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:49:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/302290455 shutdown_connections 2023-03-03T08:49:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.257+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/302290455 wait complete. 2023-03-03T08:49:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.259+0000 7fc9cb9ff700 1 Processor -- start 2023-03-03T08:49:24.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.259+0000 7fc9cb9ff700 1 -- start start 2023-03-03T08:49:24.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.260+0000 7fc9cb9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd390 0x7fc9c40c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:24.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.260+0000 7fc9cb9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c40cd8e0 0x7fc9c414e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:24.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.260+0000 7fc9cb9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c4153290 0x7fc9c414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:24.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.260+0000 7fc9cb9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c414ebc0 con 0x7fc9c40cd8e0 2023-03-03T08:49:24.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.260+0000 7fc9cb9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4154850 con 0x7fc9c4153290 2023-03-03T08:49:24.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.260+0000 7fc9cb9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40b40a0 con 0x7fc9c40cd390 2023-03-03T08:49:24.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c979b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd390 0x7fc9c40c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:24.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c4153290 0x7fc9c414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:24.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.260+0000 7fc9c8f9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c40cd8e0 0x7fc9c414e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:24.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c4153290 0x7fc9c414e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33238/0 (socket says 172.21.15.102:33238) 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 -- 172.21.15.102:0/3916771826 learned_addr learned my addr 172.21.15.102:0/3916771826 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c979b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd390 0x7fc9c40c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36864/0 (socket says 172.21.15.102:36864) 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 -- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd390 msgr2=0x7fc9c40c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd390 0x7fc9c40c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 -- 172.21.15.102:0/3916771826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c40cd8e0 msgr2=0x7fc9c414e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c40cd8e0 0x7fc9c414e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 -- 172.21.15.102:0/3916771826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b40094a0 con 0x7fc9c4153290 2023-03-03T08:49:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c8f9a700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c40cd8e0 0x7fc9c414e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:49:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.261+0000 7fc9c9f9c700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c4153290 0x7fc9c414e680 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009d30 tx=0x7fc9bc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:49:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.262+0000 7fc9b9ffb700 1 -- 172.21.15.102:0/3916771826 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc00cc90 con 0x7fc9c4153290 2023-03-03T08:49:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.262+0000 7fc9b9ffb700 1 -- 172.21.15.102:0/3916771826 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc00f040 con 0x7fc9c4153290 2023-03-03T08:49:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.262+0000 7fc9b9ffb700 1 -- 172.21.15.102:0/3916771826 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc009e80 con 0x7fc9c4153290 2023-03-03T08:49:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.262+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c40ccde0 con 0x7fc9c4153290 2023-03-03T08:49:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.262+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c415d160 con 0x7fc9c4153290 2023-03-03T08:49:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.263+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c40cd0c0 con 0x7fc9c4153290 2023-03-03T08:49:24.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.264+0000 7fc9b9ffb700 1 -- 172.21.15.102:0/3916771826 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc9bc014480 con 0x7fc9c4153290 2023-03-03T08:49:24.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.265+0000 7fc9b9ffb700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a80682f0 0x7fc9a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:24.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.265+0000 7fc9b9ffb700 1 -- 172.21.15.102:0/3916771826 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc9bc086710 con 0x7fc9c4153290 2023-03-03T08:49:24.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.266+0000 7fc9c979b700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a80682f0 0x7fc9a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:24.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.267+0000 7fc9c979b700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a80682f0 0x7fc9a806a7a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0007fd0 tx=0x7fc9c0007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:49:24.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.270+0000 7fc9b9ffb700 1 -- 172.21.15.102:0/3916771826 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc9bc052dc0 con 0x7fc9c4153290 2023-03-03T08:49:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.505+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9c404e6e0 con 0x7fc9c4153290 2023-03-03T08:49:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.506+0000 7fc9b9ffb700 1 -- 172.21.15.102:0/3916771826 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc9bc0563e0 con 0x7fc9c4153290 2023-03-03T08:49:24.507 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:49:24.508 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:49:24.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a80682f0 msgr2=0x7fc9a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:24.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a80682f0 0x7fc9a806a7a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0007fd0 tx=0x7fc9c0007dc0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c4153290 msgr2=0x7fc9c414e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:24.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c4153290 0x7fc9c414e680 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009d30 tx=0x7fc9bc00c3a0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 shutdown_connections 2023-03-03T08:49:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a80682f0 0x7fc9a806a7a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c40cd390 0x7fc9c40c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c4153290 0x7fc9c414e680 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 --2- 172.21.15.102:0/3916771826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c40cd8e0 0x7fc9c414e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 >> 172.21.15.102:0/3916771826 conn(0x7fc9c4065fb0 msgr2=0x7fc9c4151100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:49:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 shutdown_connections 2023-03-03T08:49:24.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:49:24.509+0000 7fc9cb9ff700 1 -- 172.21.15.102:0/3916771826 wait complete. 2023-03-03T08:49:24.512 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:49:24.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T08:49:36.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:36.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:00.002 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T08:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:50:06.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:06.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:12.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:50:12.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.803+0000 7f9e9edff700 1 Processor -- start 2023-03-03T08:50:12.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.803+0000 7f9e9edff700 1 -- start start 2023-03-03T08:50:12.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9edff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9e981597f0 0x7f9e980cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:12.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9edff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9e980cf9a0 0x7f9e98163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:12.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9edff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e980cb9f0 con 0x7f9e980ced80 2023-03-03T08:50:12.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9edff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e980ca750 con 0x7f9e981597f0 2023-03-03T08:50:12.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9edff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9815a230 con 0x7f9e980cf9a0 2023-03-03T08:50:12.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9cb9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9e980ced80 0x7f9e980cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:43008/0 (socket says 172.21.15.102:43008) 2023-03-03T08:50:12.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9cb9b700 1 -- 172.21.15.102:0/469884172 learned_addr learned my addr 172.21.15.102:0/469884172 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:50:12.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e9d39c700 1 --2- 172.21.15.102:0/469884172 >> v2:172.21.15.172:3300/0 conn(0x7f9e980cf9a0 0x7f9e98163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:12.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e97fff700 1 --2- 172.21.15.102:0/469884172 >> v2:172.21.15.102:3300/0 conn(0x7f9e981597f0 0x7f9e980cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:12.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.805+0000 7f9e977fe700 1 -- 172.21.15.102:0/469884172 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186908360 0 0) 0x7f9e980cb9f0 con 0x7f9e980ced80 2023-03-03T08:50:12.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e977fe700 1 -- 172.21.15.102:0/469884172 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e880035b0 con 0x7f9e980ced80 2023-03-03T08:50:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e9d39c700 1 -- 172.21.15.102:0/469884172 >> v2:172.21.15.102:3300/0 conn(0x7f9e981597f0 msgr2=0x7f9e980cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:50:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e9d39c700 1 --2- 172.21.15.102:0/469884172 >> v2:172.21.15.102:3300/0 conn(0x7f9e981597f0 0x7f9e980cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e9d39c700 1 -- 172.21.15.102:0/469884172 >> v1:172.21.15.172:6789/0 conn(0x7f9e980ced80 legacy=0x7f9e980cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e9d39c700 1 -- 172.21.15.102:0/469884172 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98159c80 con 0x7f9e980cf9a0 2023-03-03T08:50:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e97fff700 1 --2- 172.21.15.102:0/469884172 >> v2:172.21.15.102:3300/0 conn(0x7f9e981597f0 0x7f9e980cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:50:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e9d39c700 1 --2- 172.21.15.102:0/469884172 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980cf9a0 0x7f9e98163660 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9e9815d3d0 tx=0x7f9e9000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d248f24d1d16d972 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e977fe700 1 -- 172.21.15.102:0/469884172 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9000e070 con 0x7f9e980cf9a0 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e977fe700 1 -- 172.21.15.102:0/469884172 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9000bb20 con 0x7f9e980cf9a0 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.806+0000 7f9e977fe700 1 -- 172.21.15.102:0/469884172 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90012430 con 0x7f9e980cf9a0 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 -- 172.21.15.102:0/469884172 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980cf9a0 msgr2=0x7f9e98163660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 --2- 172.21.15.102:0/469884172 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980cf9a0 0x7f9e98163660 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9e9815d3d0 tx=0x7f9e9000b330 comp rx=0 tx=0).stop 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 -- 172.21.15.102:0/469884172 shutdown_connections 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 --2- 172.21.15.102:0/469884172 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980cf9a0 0x7f9e98163660 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 --2- 172.21.15.102:0/469884172 >> v2:172.21.15.102:3300/0 conn(0x7f9e981597f0 0x7f9e980cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:12.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 -- 172.21.15.102:0/469884172 >> 172.21.15.102:0/469884172 conn(0x7f9e98065fb0 msgr2=0x7f9e980663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:12.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 -- 172.21.15.102:0/469884172 shutdown_connections 2023-03-03T08:50:12.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.807+0000 7f9e9edff700 1 -- 172.21.15.102:0/469884172 wait complete. 2023-03-03T08:50:12.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.809+0000 7f9e9edff700 1 Processor -- start 2023-03-03T08:50:12.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.809+0000 7f9e9edff700 1 -- start start 2023-03-03T08:50:12.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.810+0000 7f9e9edff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980ced80 0x7f9e980c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:12.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.810+0000 7f9e9edff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e980cf9a0 0x7f9e9814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:12.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.810+0000 7f9e9edff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e981597f0 0x7f9e9814e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:12.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.810+0000 7f9e9edff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e980b4a80 con 0x7f9e981597f0 2023-03-03T08:50:12.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.810+0000 7f9e9edff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e980ce1f0 con 0x7f9e980ced80 2023-03-03T08:50:12.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.810+0000 7f9e9edff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e980cdf50 con 0x7f9e980cf9a0 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980ced80 0x7f9e980c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980ced80 0x7f9e980c7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53446/0 (socket says 172.21.15.102:53446) 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 -- 172.21.15.102:0/547036866 learned_addr learned my addr 172.21.15.102:0/547036866 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e97fff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e980cf9a0 0x7f9e9814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9d39c700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e981597f0 0x7f9e9814e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 -- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e980cf9a0 msgr2=0x7f9e9814e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e980cf9a0 0x7f9e9814e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:12.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 -- 172.21.15.102:0/547036866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e981597f0 msgr2=0x7f9e9814e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:12.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e981597f0 0x7f9e9814e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:12.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 -- 172.21.15.102:0/547036866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e840094a0 con 0x7f9e980ced80 2023-03-03T08:50:12.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.811+0000 7f9e9cb9b700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980ced80 0x7f9e980c7b50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c001eb0 tx=0x7f9e8c00c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:12.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.812+0000 7f9e957fa700 1 -- 172.21.15.102:0/547036866 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c005d70 con 0x7f9e980ced80 2023-03-03T08:50:12.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.812+0000 7f9e957fa700 1 -- 172.21.15.102:0/547036866 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c0070d0 con 0x7f9e980ced80 2023-03-03T08:50:12.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.812+0000 7f9e957fa700 1 -- 172.21.15.102:0/547036866 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c007460 con 0x7f9e980ced80 2023-03-03T08:50:12.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.812+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e90009a70 con 0x7f9e980ced80 2023-03-03T08:50:12.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.812+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e980cb0c0 con 0x7f9e980ced80 2023-03-03T08:50:12.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.813+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e980cade0 con 0x7f9e980ced80 2023-03-03T08:50:12.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.814+0000 7f9e957fa700 1 -- 172.21.15.102:0/547036866 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9e8c019030 con 0x7f9e980ced80 2023-03-03T08:50:12.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.815+0000 7f9e957fa700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9e7c0682f0 0x7f9e7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:12.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.816+0000 7f9e957fa700 1 -- 172.21.15.102:0/547036866 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9e8c085070 con 0x7f9e980ced80 2023-03-03T08:50:12.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.816+0000 7f9e97fff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9e7c0682f0 0x7f9e7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:12.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.817+0000 7f9e97fff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9e7c0682f0 0x7f9e7c06a7a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9e980c9620 tx=0x7f9e84009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:12.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:12.820+0000 7f9e957fa700 1 -- 172.21.15.102:0/547036866 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9e8c051780 con 0x7f9e980ced80 2023-03-03T08:50:13.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.054+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e980cb3a0 con 0x7f9e980ced80 2023-03-03T08:50:13.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.055+0000 7f9e957fa700 1 -- 172.21.15.102:0/547036866 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9e8c017090 con 0x7f9e980ced80 2023-03-03T08:50:13.057 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:50:13.057 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:50:13.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9e7c0682f0 msgr2=0x7f9e7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:13.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9e7c0682f0 0x7f9e7c06a7a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9e980c9620 tx=0x7f9e84009210 comp rx=0 tx=0).stop 2023-03-03T08:50:13.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980ced80 msgr2=0x7f9e980c7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:13.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980ced80 0x7f9e980c7b50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c001eb0 tx=0x7f9e8c00c7b0 comp rx=0 tx=0).stop 2023-03-03T08:50:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 shutdown_connections 2023-03-03T08:50:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e981597f0 0x7f9e9814e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e980ced80 0x7f9e980c7b50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9e7c0682f0 0x7f9e7c06a7a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 --2- 172.21.15.102:0/547036866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9e980cf9a0 0x7f9e9814e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 >> 172.21.15.102:0/547036866 conn(0x7f9e98065fb0 msgr2=0x7f9e980cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 shutdown_connections 2023-03-03T08:50:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:50:13.058+0000 7f9e9edff700 1 -- 172.21.15.102:0/547036866 wait complete. 2023-03-03T08:50:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:50:13.087 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 72 secs before thrashing 2023-03-03T08:50:36.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:36.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:07.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:07.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:25.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:51:25.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.339+0000 7f0e37fff700 1 Processor -- start 2023-03-03T08:51:25.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.339+0000 7f0e37fff700 1 -- start start 2023-03-03T08:51:25.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e37fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e381564b0 0x7f0e380cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:25.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e37fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380c9ca0 con 0x7f0e380c9610 2023-03-03T08:51:25.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e37fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380b46f0 con 0x7f0e380cab50 2023-03-03T08:51:25.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e37fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380b4920 con 0x7f0e381564b0 2023-03-03T08:51:25.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e377fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0e380cab50 0x7f0e381636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56390/0 (socket says 172.21.15.102:56390) 2023-03-03T08:51:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e377fe700 1 -- 172.21.15.102:0/3973374467 learned_addr learned my addr 172.21.15.102:0/3973374467 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:51:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e367fc700 1 --1- 172.21.15.102:0/3973374467 >> v1:172.21.15.102:6789/0 conn(0x7f0e380c9610 0x7f0e380ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60268/0 (socket says 172.21.15.102:60268) 2023-03-03T08:51:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.341+0000 7f0e36ffd700 1 --2- 172.21.15.102:0/3973374467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e381564b0 0x7f0e380cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e35ffb700 1 -- 172.21.15.102:0/3973374467 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865514548 0 0) 0x7f0e380b46f0 con 0x7f0e380cab50 2023-03-03T08:51:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e35ffb700 1 -- 172.21.15.102:0/3973374467 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e200035b0 con 0x7f0e380cab50 2023-03-03T08:51:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e36ffd700 1 -- 172.21.15.102:0/3973374467 >> v1:172.21.15.172:6789/0 conn(0x7f0e380cab50 legacy=0x7f0e381636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:25.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e35ffb700 1 -- 172.21.15.102:0/3973374467 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851035356 0 0) 0x7f0e380c9ca0 con 0x7f0e380c9610 2023-03-03T08:51:25.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e36ffd700 1 -- 172.21.15.102:0/3973374467 >> v1:172.21.15.102:6789/0 conn(0x7f0e380c9610 legacy=0x7f0e380ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:25.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e36ffd700 1 -- 172.21.15.102:0/3973374467 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e380b4c00 con 0x7f0e381564b0 2023-03-03T08:51:25.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e36ffd700 1 --2- 172.21.15.102:0/3973374467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e381564b0 0x7f0e380cb660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e28009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=724789a0a41ca75c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:25.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e35ffb700 1 -- 172.21.15.102:0/3973374467 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28011830 con 0x7f0e381564b0 2023-03-03T08:51:25.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e35ffb700 1 -- 172.21.15.102:0/3973374467 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e28011990 con 0x7f0e381564b0 2023-03-03T08:51:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.342+0000 7f0e35ffb700 1 -- 172.21.15.102:0/3973374467 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28011d20 con 0x7f0e381564b0 2023-03-03T08:51:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.343+0000 7f0e37fff700 1 -- 172.21.15.102:0/3973374467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e381564b0 msgr2=0x7f0e380cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.343+0000 7f0e37fff700 1 --2- 172.21.15.102:0/3973374467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e381564b0 0x7f0e380cb660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e28009b80 comp rx=0 tx=0).stop 2023-03-03T08:51:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.343+0000 7f0e37fff700 1 -- 172.21.15.102:0/3973374467 shutdown_connections 2023-03-03T08:51:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.343+0000 7f0e37fff700 1 --2- 172.21.15.102:0/3973374467 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e381564b0 0x7f0e380cb660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.343+0000 7f0e37fff700 1 -- 172.21.15.102:0/3973374467 >> 172.21.15.102:0/3973374467 conn(0x7f0e38065fb0 msgr2=0x7f0e380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:51:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.343+0000 7f0e37fff700 1 -- 172.21.15.102:0/3973374467 shutdown_connections 2023-03-03T08:51:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.343+0000 7f0e37fff700 1 -- 172.21.15.102:0/3973374467 wait complete. 2023-03-03T08:51:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.345+0000 7f0e37fff700 1 Processor -- start 2023-03-03T08:51:25.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.345+0000 7f0e37fff700 1 -- start start 2023-03-03T08:51:25.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e37fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e380c9610 0x7f0e38157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:25.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e37fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e380cab50 0x7f0e381546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:25.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e37fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e381564b0 0x7f0e38150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:25.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e37fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3815c5a0 con 0x7f0e381564b0 2023-03-03T08:51:25.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e37fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3815c880 con 0x7f0e380cab50 2023-03-03T08:51:25.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e37fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3815cb60 con 0x7f0e380c9610 2023-03-03T08:51:25.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e377fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e381564b0 0x7f0e38150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:25.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e36ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e380c9610 0x7f0e38157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:25.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.346+0000 7f0e367fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e380cab50 0x7f0e381546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:25.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e380c9610 0x7f0e38157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42666/0 (socket says 172.21.15.102:42666) 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 -- 172.21.15.102:0/2682997919 learned_addr learned my addr 172.21.15.102:0/2682997919 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 -- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e380cab50 msgr2=0x7f0e381546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e380cab50 0x7f0e381546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 -- 172.21.15.102:0/2682997919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e381564b0 msgr2=0x7f0e38150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e381564b0 0x7f0e38150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 -- 172.21.15.102:0/2682997919 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280094a0 con 0x7f0e380c9610 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.347+0000 7f0e36ffd700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e380c9610 0x7f0e38157730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009bc0 tx=0x7f0e28004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.348+0000 7f0e1f7fe700 1 -- 172.21.15.102:0/2682997919 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2801a7e0 con 0x7f0e380c9610 2023-03-03T08:51:25.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.348+0000 7f0e1f7fe700 1 -- 172.21.15.102:0/2682997919 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2801a960 con 0x7f0e380c9610 2023-03-03T08:51:25.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.348+0000 7f0e1f7fe700 1 -- 172.21.15.102:0/2682997919 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2801ad10 con 0x7f0e380c9610 2023-03-03T08:51:25.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.348+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e380b2210 con 0x7f0e380c9610 2023-03-03T08:51:25.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.348+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e380b1f30 con 0x7f0e380c9610 2023-03-03T08:51:25.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.349+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e38156880 con 0x7f0e380c9610 2023-03-03T08:51:25.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.350+0000 7f0e1f7fe700 1 -- 172.21.15.102:0/2682997919 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0e2800f050 con 0x7f0e380c9610 2023-03-03T08:51:25.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.351+0000 7f0e1f7fe700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e240682f0 0x7f0e2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:25.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.351+0000 7f0e1f7fe700 1 -- 172.21.15.102:0/2682997919 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0e28086ca0 con 0x7f0e380c9610 2023-03-03T08:51:25.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.352+0000 7f0e367fc700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e240682f0 0x7f0e2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:25.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.352+0000 7f0e367fc700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e240682f0 0x7f0e2406a7a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0e381eba00 tx=0x7f0e2c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:25.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.356+0000 7f0e1f7fe700 1 -- 172.21.15.102:0/2682997919 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e280533b0 con 0x7f0e380c9610 2023-03-03T08:51:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.589+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e380677f0 con 0x7f0e380c9610 2023-03-03T08:51:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.590+0000 7f0e1f7fe700 1 -- 172.21.15.102:0/2682997919 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0e28015090 con 0x7f0e380c9610 2023-03-03T08:51:25.592 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:51:25.592 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:51:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e240682f0 msgr2=0x7f0e2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e240682f0 0x7f0e2406a7a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0e381eba00 tx=0x7f0e2c007de0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e380c9610 msgr2=0x7f0e38157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e380c9610 0x7f0e38157730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009bc0 tx=0x7f0e28004750 comp rx=0 tx=0).stop 2023-03-03T08:51:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 shutdown_connections 2023-03-03T08:51:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e240682f0 0x7f0e2406a7a0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e380c9610 0x7f0e38157730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e381564b0 0x7f0e38150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 --2- 172.21.15.102:0/2682997919 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e380cab50 0x7f0e381546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 >> 172.21.15.102:0/2682997919 conn(0x7f0e38065fb0 msgr2=0x7f0e38155590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:51:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 shutdown_connections 2023-03-03T08:51:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:51:25.593+0000 7f0e37fff700 1 -- 172.21.15.102:0/2682997919 wait complete. 2023-03-03T08:51:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:51:25.621 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 45 secs before thrashing 2023-03-03T08:51:37.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:37.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:07.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:07.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:10.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:52:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.872+0000 7ff5b139d700 1 Processor -- start 2023-03-03T08:52:10.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.872+0000 7ff5b139d700 1 -- start start 2023-03-03T08:52:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5b139d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff5ac0ced00 0x7ff5ac0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5b139d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff5ac0c41b0 0x7ff5ac0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5b139d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0c3d20 con 0x7ff5ac0b3970 2023-03-03T08:52:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5b139d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac0c4930 con 0x7ff5ac0ced00 2023-03-03T08:52:10.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5b139d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac0cf610 con 0x7ff5ac0c41b0 2023-03-03T08:52:10.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5abfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff5ac0ced00 0x7ff5ac0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5b0b9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff5ac0c41b0 0x7ff5ac0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5ab7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff5ac0b3970 0x7ff5ac158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40054/0 (socket says 172.21.15.102:40054) 2023-03-03T08:52:10.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.874+0000 7ff5ab7fe700 1 -- 172.21.15.102:0/3342309505 learned_addr learned my addr 172.21.15.102:0/3342309505 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:52:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5abfff700 1 -- 172.21.15.102:0/3342309505 >> v2:172.21.15.172:3300/0 conn(0x7ff5ac0c41b0 msgr2=0x7ff5ac0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:52:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5abfff700 1 --2- 172.21.15.102:0/3342309505 >> v2:172.21.15.172:3300/0 conn(0x7ff5ac0c41b0 0x7ff5ac0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5abfff700 1 -- 172.21.15.102:0/3342309505 >> v1:172.21.15.102:6789/0 conn(0x7ff5ac0b3970 legacy=0x7ff5ac158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5abfff700 1 -- 172.21.15.102:0/3342309505 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac15cba0 con 0x7ff5ac0ced00 2023-03-03T08:52:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5abfff700 1 --2- 172.21.15.102:0/3342309505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac0c5a90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0ca080 tx=0x7ff59c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef35c44928ffdf9b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5aaffd700 1 -- 172.21.15.102:0/3342309505 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c011830 con 0x7ff5ac0ced00 2023-03-03T08:52:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5aaffd700 1 -- 172.21.15.102:0/3342309505 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c011990 con 0x7ff5ac0ced00 2023-03-03T08:52:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.875+0000 7ff5aaffd700 1 -- 172.21.15.102:0/3342309505 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c011d20 con 0x7ff5ac0ced00 2023-03-03T08:52:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 -- 172.21.15.102:0/3342309505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0ced00 msgr2=0x7ff5ac0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 --2- 172.21.15.102:0/3342309505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac0c5a90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0ca080 tx=0x7ff59c009b50 comp rx=0 tx=0).stop 2023-03-03T08:52:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 -- 172.21.15.102:0/3342309505 shutdown_connections 2023-03-03T08:52:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 --2- 172.21.15.102:0/3342309505 >> v2:172.21.15.172:3300/0 conn(0x7ff5ac0c41b0 0x7ff5ac0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 --2- 172.21.15.102:0/3342309505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac0c5a90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 -- 172.21.15.102:0/3342309505 >> 172.21.15.102:0/3342309505 conn(0x7ff5ac08b1c0 msgr2=0x7ff5ac071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 -- 172.21.15.102:0/3342309505 shutdown_connections 2023-03-03T08:52:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.876+0000 7ff5b139d700 1 -- 172.21.15.102:0/3342309505 wait complete. 2023-03-03T08:52:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.878+0000 7ff5b139d700 1 Processor -- start 2023-03-03T08:52:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.878+0000 7ff5b139d700 1 -- start start 2023-03-03T08:52:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.879+0000 7ff5b139d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5ac0b3970 0x7ff5ac0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.879+0000 7ff5b139d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0c41b0 0x7ff5ac1ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.879+0000 7ff5b139d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac1ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.879+0000 7ff5b139d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac15a020 con 0x7ff5ac0c41b0 2023-03-03T08:52:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.879+0000 7ff5b139d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac1f22c0 con 0x7ff5ac0ced00 2023-03-03T08:52:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.879+0000 7ff5b139d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0cf1b0 con 0x7ff5ac0b3970 2023-03-03T08:52:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5abfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5ac0b3970 0x7ff5ac0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5ab7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0c41b0 0x7ff5ac1ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5b0b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac1ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5abfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5ac0b3970 0x7ff5ac0b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:49560/0 (socket says 172.21.15.102:49560) 2023-03-03T08:52:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5abfff700 1 -- 172.21.15.102:0/2435742644 learned_addr learned my addr 172.21.15.102:0/2435742644 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:52:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5b0b9c700 1 -- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5ac0b3970 msgr2=0x7ff5ac0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5b0b9c700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5ac0b3970 0x7ff5ac0b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5b0b9c700 1 -- 172.21.15.102:0/2435742644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0c41b0 msgr2=0x7ff5ac1ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5b0b9c700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0c41b0 0x7ff5ac1ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5b0b9c700 1 -- 172.21.15.102:0/2435742644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0094a0 con 0x7ff5ac0ced00 2023-03-03T08:52:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.880+0000 7ff5ab7fe700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0c41b0 0x7ff5ac1ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:52:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.881+0000 7ff5b0b9c700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac1ee6e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400eba0 tx=0x7ff5a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:10.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.881+0000 7ff5a8ff9700 1 -- 172.21.15.102:0/2435742644 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4009a80 con 0x7ff5ac0ced00 2023-03-03T08:52:10.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.881+0000 7ff5a8ff9700 1 -- 172.21.15.102:0/2435742644 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a400f040 con 0x7ff5ac0ced00 2023-03-03T08:52:10.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.881+0000 7ff5a8ff9700 1 -- 172.21.15.102:0/2435742644 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a400cc40 con 0x7ff5ac0ced00 2023-03-03T08:52:10.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.881+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ac06bf20 con 0x7ff5ac0ced00 2023-03-03T08:52:10.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.881+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ac15ce80 con 0x7ff5ac0ced00 2023-03-03T08:52:10.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.882+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ac06c200 con 0x7ff5ac0ced00 2023-03-03T08:52:10.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.883+0000 7ff5a8ff9700 1 -- 172.21.15.102:0/2435742644 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5a4024070 con 0x7ff5ac0ced00 2023-03-03T08:52:10.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.885+0000 7ff5a8ff9700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff5980682f0 0x7ff59806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.885+0000 7ff5a8ff9700 1 -- 172.21.15.102:0/2435742644 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff5a40866f0 con 0x7ff5ac0ced00 2023-03-03T08:52:10.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.885+0000 7ff5abfff700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff5980682f0 0x7ff59806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.886+0000 7ff5abfff700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff5980682f0 0x7ff59806a7a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009b90 tx=0x7ff59c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:10.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:10.889+0000 7ff5a8ff9700 1 -- 172.21.15.102:0/2435742644 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5a4052e00 con 0x7ff5ac0ced00 2023-03-03T08:52:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.124+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5ac04e6e0 con 0x7ff5ac0ced00 2023-03-03T08:52:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.125+0000 7ff5a8ff9700 1 -- 172.21.15.102:0/2435742644 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff5a4056420 con 0x7ff5ac0ced00 2023-03-03T08:52:11.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:52:11.127 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:52:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff5980682f0 msgr2=0x7ff59806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff5980682f0 0x7ff59806a7a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009b90 tx=0x7ff59c01c040 comp rx=0 tx=0).stop 2023-03-03T08:52:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5ac0ced00 msgr2=0x7ff5ac1ee6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac1ee6e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400eba0 tx=0x7ff5a400eeb0 comp rx=0 tx=0).stop 2023-03-03T08:52:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 shutdown_connections 2023-03-03T08:52:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff5980682f0 0x7ff59806a7a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff5ac0b3970 0x7ff5ac0b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5ac0ced00 0x7ff5ac1ee6e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 --2- 172.21.15.102:0/2435742644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5ac0c41b0 0x7ff5ac1ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.128+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 >> 172.21.15.102:0/2435742644 conn(0x7ff5ac08b1c0 msgr2=0x7ff5ac0669c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.129+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 shutdown_connections 2023-03-03T08:52:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:52:11.129+0000 7ff5b139d700 1 -- 172.21.15.102:0/2435742644 wait complete. 2023-03-03T08:52:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:52:11.158 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 63 secs before thrashing 2023-03-03T08:52:37.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:37.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:08.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:08.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:14.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:53:14.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.407+0000 7fcfa9fff700 1 Processor -- start 2023-03-03T08:53:14.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.407+0000 7fcfa9fff700 1 -- start start 2023-03-03T08:53:14.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa9fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfa4153290 0x7fcfa4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:14.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa9fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa41593c0 0x7fcfa40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:14.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa9fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa4153660 con 0x7fcfa40cd8e0 2023-03-03T08:53:14.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa9fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa406c360 con 0x7fcfa4153290 2023-03-03T08:53:14.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa9fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa415cea0 con 0x7fcfa41593c0 2023-03-03T08:53:14.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa41593c0 0x7fcfa40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:14.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa3fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcfa40cd8e0 0x7fcfa40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41196/0 (socket says 172.21.15.102:41196) 2023-03-03T08:53:14.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa3fff700 1 -- 172.21.15.102:0/348053223 learned_addr learned my addr 172.21.15.102:0/348053223 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:53:14.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa37fe700 1 --2- 172.21.15.102:0/348053223 >> v2:172.21.15.102:3300/0 conn(0x7fcfa4153290 0x7fcfa4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:14.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.409+0000 7fcfa27fc700 1 -- 172.21.15.102:0/348053223 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908880649 0 0) 0x7fcfa4153660 con 0x7fcfa40cd8e0 2023-03-03T08:53:14.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa2ffd700 1 -- 172.21.15.102:0/348053223 >> v2:172.21.15.102:3300/0 conn(0x7fcfa4153290 msgr2=0x7fcfa4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:53:14.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa2ffd700 1 --2- 172.21.15.102:0/348053223 >> v2:172.21.15.102:3300/0 conn(0x7fcfa4153290 0x7fcfa4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa2ffd700 1 -- 172.21.15.102:0/348053223 >> v1:172.21.15.172:6789/0 conn(0x7fcfa40cd8e0 legacy=0x7fcfa40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:14.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa2ffd700 1 -- 172.21.15.102:0/348053223 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa406c7f0 con 0x7fcfa41593c0 2023-03-03T08:53:14.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa2ffd700 1 --2- 172.21.15.102:0/348053223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa41593c0 0x7fcfa40cde20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009840 tx=0x7fcf98009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31335c73812889a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:14.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa27fc700 1 -- 172.21.15.102:0/348053223 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011830 con 0x7fcfa41593c0 2023-03-03T08:53:14.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa27fc700 1 -- 172.21.15.102:0/348053223 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf98011990 con 0x7fcfa41593c0 2023-03-03T08:53:14.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.410+0000 7fcfa27fc700 1 -- 172.21.15.102:0/348053223 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011d20 con 0x7fcfa41593c0 2023-03-03T08:53:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 -- 172.21.15.102:0/348053223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa41593c0 msgr2=0x7fcfa40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/348053223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa41593c0 0x7fcfa40cde20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009840 tx=0x7fcf98009b50 comp rx=0 tx=0).stop 2023-03-03T08:53:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 -- 172.21.15.102:0/348053223 shutdown_connections 2023-03-03T08:53:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/348053223 >> v2:172.21.15.102:3300/0 conn(0x7fcfa4153290 0x7fcfa4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/348053223 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa41593c0 0x7fcfa40cde20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 -- 172.21.15.102:0/348053223 >> 172.21.15.102:0/348053223 conn(0x7fcfa4065fb0 msgr2=0x7fcfa40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:53:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 -- 172.21.15.102:0/348053223 shutdown_connections 2023-03-03T08:53:14.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.411+0000 7fcfa9fff700 1 -- 172.21.15.102:0/348053223 wait complete. 2023-03-03T08:53:14.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.413+0000 7fcfa9fff700 1 Processor -- start 2023-03-03T08:53:14.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.413+0000 7fcfa9fff700 1 -- start start 2023-03-03T08:53:14.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.414+0000 7fcfa9fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfa40cd8e0 0x7fcfa414d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:14.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.414+0000 7fcfa9fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa4153290 0x7fcfa41e5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:14.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.414+0000 7fcfa9fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa41593c0 0x7fcfa41e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:14.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.414+0000 7fcfa9fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa414f570 con 0x7fcfa41593c0 2023-03-03T08:53:14.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.414+0000 7fcfa9fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4150610 con 0x7fcfa40cd8e0 2023-03-03T08:53:14.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.414+0000 7fcfa9fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa414eb50 con 0x7fcfa4153290 2023-03-03T08:53:14.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.414+0000 7fcfa3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa41593c0 0x7fcfa41e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:14.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfa40cd8e0 0x7fcfa414d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:14.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa41593c0 0x7fcfa41e5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37804/0 (socket says 172.21.15.102:37804) 2023-03-03T08:53:14.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfa40cd8e0 0x7fcfa414d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39070/0 (socket says 172.21.15.102:39070) 2023-03-03T08:53:14.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa4153290 0x7fcfa41e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:14.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa3fff700 1 -- 172.21.15.102:0/4188300554 learned_addr learned my addr 172.21.15.102:0/4188300554 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:53:14.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa2ffd700 1 -- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfa40cd8e0 msgr2=0x7fcfa414d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:14.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa2ffd700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfa40cd8e0 0x7fcfa414d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa2ffd700 1 -- 172.21.15.102:0/4188300554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa41593c0 msgr2=0x7fcfa41e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:14.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa2ffd700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa41593c0 0x7fcfa41e5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa2ffd700 1 -- 172.21.15.102:0/4188300554 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980094a0 con 0x7fcfa4153290 2023-03-03T08:53:14.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa3fff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa41593c0 0x7fcfa41e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:53:14.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.415+0000 7fcfa2ffd700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa4153290 0x7fcfa41e5870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcf98016000 tx=0x7fcf98004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:14.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.416+0000 7fcf8bfff700 1 -- 172.21.15.102:0/4188300554 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980036a0 con 0x7fcfa4153290 2023-03-03T08:53:14.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.416+0000 7fcf8bfff700 1 -- 172.21.15.102:0/4188300554 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9801f070 con 0x7fcfa4153290 2023-03-03T08:53:14.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.416+0000 7fcf8bfff700 1 -- 172.21.15.102:0/4188300554 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98003a60 con 0x7fcfa4153290 2023-03-03T08:53:14.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.416+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa40ccde0 con 0x7fcfa4153290 2023-03-03T08:53:14.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.416+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa415d160 con 0x7fcfa4153290 2023-03-03T08:53:14.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.417+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa40cd0c0 con 0x7fcfa4153290 2023-03-03T08:53:14.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.418+0000 7fcf8bfff700 1 -- 172.21.15.102:0/4188300554 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf9800f050 con 0x7fcfa4153290 2023-03-03T08:53:14.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.419+0000 7fcf8bfff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:14.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.419+0000 7fcf8bfff700 1 -- 172.21.15.102:0/4188300554 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcf98021070 con 0x7fcfa4153290 2023-03-03T08:53:14.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.420+0000 7fcfa37fe700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:14.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.420+0000 7fcfa37fe700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009b40 tx=0x7fcf94009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:14.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.424+0000 7fcf8bfff700 1 -- 172.21.15.102:0/4188300554 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcf98053cf0 con 0x7fcfa4153290 2023-03-03T08:53:14.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.656+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfa404e6e0 con 0x7fcfa4153290 2023-03-03T08:53:14.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.657+0000 7fcf8bfff700 1 -- 172.21.15.102:0/4188300554 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcf9802b090 con 0x7fcfa4153290 2023-03-03T08:53:14.658 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:53:14.659 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:53:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 msgr2=0x7fcf9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009b40 tx=0x7fcf94009210 comp rx=0 tx=0).stop 2023-03-03T08:53:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa4153290 msgr2=0x7fcfa41e5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa4153290 0x7fcfa41e5870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcf98016000 tx=0x7fcf98004750 comp rx=0 tx=0).stop 2023-03-03T08:53:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 shutdown_connections 2023-03-03T08:53:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa41593c0 0x7fcfa41e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfa40cd8e0 0x7fcfa414d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 --2- 172.21.15.102:0/4188300554 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfa4153290 0x7fcfa41e5870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 >> 172.21.15.102:0/4188300554 conn(0x7fcfa4065fb0 msgr2=0x7fcfa40cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:53:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 shutdown_connections 2023-03-03T08:53:14.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:53:14.660+0000 7fcfa9fff700 1 -- 172.21.15.102:0/4188300554 wait complete. 2023-03-03T08:53:14.663 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:53:14.688 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 82 secs before thrashing 2023-03-03T08:53:38.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:38.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:08.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:08.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:36.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:54:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.946+0000 7f1d413ff700 1 Processor -- start 2023-03-03T08:54:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.946+0000 7f1d413ff700 1 -- start start 2023-03-03T08:54:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.947+0000 7f1d413ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1d3c153290 0x7f1d3c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.947+0000 7f1d413ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d3c0cd8e0 0x7f1d3c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d413ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c153660 con 0x7f1d3c1593c0 2023-03-03T08:54:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d413ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3c06c360 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d413ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3c15cea0 con 0x7f1d3c153290 2023-03-03T08:54:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3a7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1d3c1593c0 0x7f1d3c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33692/0 (socket says 172.21.15.102:33692) 2023-03-03T08:54:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d3c0cd8e0 0x7f1d3c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3a7fc700 1 -- 172.21.15.102:0/2128150327 learned_addr learned my addr 172.21.15.102:0/2128150327 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:54:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3affd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1d3c153290 0x7f1d3c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3b7fe700 1 -- 172.21.15.102:0/2128150327 >> v2:172.21.15.172:3300/0 conn(0x7f1d3c153290 msgr2=0x7f1d3c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3b7fe700 1 --2- 172.21.15.102:0/2128150327 >> v2:172.21.15.172:3300/0 conn(0x7f1d3c153290 0x7f1d3c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3b7fe700 1 -- 172.21.15.102:0/2128150327 >> v1:172.21.15.172:6789/0 conn(0x7f1d3c1593c0 legacy=0x7f1d3c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d3b7fe700 1 -- 172.21.15.102:0/2128150327 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c06c7f0 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.948+0000 7f1d39ffb700 1 -- 172.21.15.102:0/2128150327 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680972084 0 0) 0x7f1d3c153660 con 0x7f1d3c1593c0 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.949+0000 7f1d3b7fe700 1 --2- 172.21.15.102:0/2128150327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0ce530 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c08cc70 tx=0x7f1d3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edb7112d64c1bdb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.949+0000 7f1d39ffb700 1 -- 172.21.15.102:0/2128150327 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3400e070 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.949+0000 7f1d39ffb700 1 -- 172.21.15.102:0/2128150327 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3400bb20 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.949+0000 7f1d39ffb700 1 -- 172.21.15.102:0/2128150327 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d34012430 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 -- 172.21.15.102:0/2128150327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 msgr2=0x7f1d3c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2128150327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0ce530 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c08cc70 tx=0x7f1d3400b330 comp rx=0 tx=0).stop 2023-03-03T08:54:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 -- 172.21.15.102:0/2128150327 shutdown_connections 2023-03-03T08:54:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2128150327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0ce530 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2128150327 >> v2:172.21.15.172:3300/0 conn(0x7f1d3c153290 0x7f1d3c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 -- 172.21.15.102:0/2128150327 >> 172.21.15.102:0/2128150327 conn(0x7f1d3c065fb0 msgr2=0x7f1d3c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:54:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 -- 172.21.15.102:0/2128150327 shutdown_connections 2023-03-03T08:54:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.950+0000 7f1d413ff700 1 -- 172.21.15.102:0/2128150327 wait complete. 2023-03-03T08:54:36.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.952+0000 7f1d413ff700 1 Processor -- start 2023-03-03T08:54:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.952+0000 7f1d413ff700 1 -- start start 2023-03-03T08:54:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d413ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d413ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d3c153290 0x7f1d3c14c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d413ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d3c1593c0 0x7f1d3c1480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d413ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c1e4f70 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d413ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c1e2660 con 0x7f1d3c153290 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d413ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c14cb20 con 0x7f1d3c1593c0 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d3affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d3affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54782/0 (socket says 172.21.15.102:54782) 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d3affd700 1 -- 172.21.15.102:0/2661407208 learned_addr learned my addr 172.21.15.102:0/2661407208 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d3a7fc700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d3c153290 0x7f1d3c14c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.953+0000 7f1d3b7fe700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d3c1593c0 0x7f1d3c1480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d3affd700 1 -- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d3c1593c0 msgr2=0x7f1d3c1480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d3affd700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d3c1593c0 0x7f1d3c1480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d3affd700 1 -- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d3c153290 msgr2=0x7f1d3c14c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d3affd700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d3c153290 0x7f1d3c14c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d3affd700 1 -- 172.21.15.102:0/2661407208 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d34009a70 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d3affd700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0b1c70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00e9b0 tx=0x7f1d2c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:36.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d237fe700 1 -- 172.21.15.102:0/2661407208 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c009710 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d237fe700 1 -- 172.21.15.102:0/2661407208 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c00ee50 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d237fe700 1 -- 172.21.15.102:0/2661407208 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00ca50 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.954+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c0ccde0 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.955+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c15d160 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.956+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c0cd0c0 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.957+0000 7f1d237fe700 1 -- 172.21.15.102:0/2661407208 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1d2c019030 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.958+0000 7f1d237fe700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d280682f0 0x7f1d2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.958+0000 7f1d237fe700 1 -- 172.21.15.102:0/2661407208 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1d2c085d70 con 0x7f1d3c0cd8e0 2023-03-03T08:54:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.958+0000 7f1d3a7fc700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d280682f0 0x7f1d2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.959+0000 7f1d3a7fc700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d280682f0 0x7f1d2806a7a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c159060 tx=0x7f1d30007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:36.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:36.962+0000 7f1d237fe700 1 -- 172.21.15.102:0/2661407208 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1d2c052390 con 0x7f1d3c0cd8e0 2023-03-03T08:54:37.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.194+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d3c15cbb0 con 0x7f1d3c0cd8e0 2023-03-03T08:54:37.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.195+0000 7f1d237fe700 1 -- 172.21.15.102:0/2661407208 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1d2c021090 con 0x7f1d3c0cd8e0 2023-03-03T08:54:37.196 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:54:37.197 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:54:37.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d280682f0 msgr2=0x7f1d2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:37.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d280682f0 0x7f1d2806a7a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c159060 tx=0x7f1d30007de0 comp rx=0 tx=0).stop 2023-03-03T08:54:37.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 msgr2=0x7f1d3c0b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:37.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0b1c70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00e9b0 tx=0x7f1d2c00ecc0 comp rx=0 tx=0).stop 2023-03-03T08:54:37.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 shutdown_connections 2023-03-03T08:54:37.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d280682f0 0x7f1d2806a7a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:37.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d3c1593c0 0x7f1d3c1480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:37.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d3c0cd8e0 0x7f1d3c0b1c70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:37.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 --2- 172.21.15.102:0/2661407208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d3c153290 0x7f1d3c14c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:37.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 >> 172.21.15.102:0/2661407208 conn(0x7f1d3c065fb0 msgr2=0x7f1d3c15ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:54:37.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 shutdown_connections 2023-03-03T08:54:37.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:54:37.198+0000 7f1d413ff700 1 -- 172.21.15.102:0/2661407208 wait complete. 2023-03-03T08:54:37.201 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:54:37.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2023-03-03T08:54:38.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:38.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:07.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:55:07.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.478+0000 7f053299d700 1 Processor -- start 2023-03-03T08:55:07.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.478+0000 7f053299d700 1 -- start start 2023-03-03T08:55:07.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.480+0000 7f053299d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c1646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:07.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.480+0000 7f053299d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c0c9ca0 con 0x7f052c1564b0 2023-03-03T08:55:07.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.480+0000 7f053299d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c0b46f0 con 0x7f052c0c9610 2023-03-03T08:55:07.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.480+0000 7f053299d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c0b4920 con 0x7f052c0cab50 2023-03-03T08:55:07.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.480+0000 7f053199b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f052c1564b0 0x7f052c0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48870/0 (socket says 172.21.15.102:48870) 2023-03-03T08:55:07.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.480+0000 7f053199b700 1 -- 172.21.15.102:0/2129986061 learned_addr learned my addr 172.21.15.102:0/2129986061 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:55:07.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.480+0000 7f053219c700 1 --2- 172.21.15.102:0/2129986061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c1646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:07.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f0530999700 1 -- 172.21.15.102:0/2129986061 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114666251 0 0) 0x7f052c0c9ca0 con 0x7f052c1564b0 2023-03-03T08:55:07.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f0530999700 1 -- 172.21.15.102:0/2129986061 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05180035b0 con 0x7f052c1564b0 2023-03-03T08:55:07.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f0530999700 1 -- 172.21.15.102:0/2129986061 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751973703 0 0) 0x7f052c0b46f0 con 0x7f052c0c9610 2023-03-03T08:55:07.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f053219c700 1 -- 172.21.15.102:0/2129986061 >> v1:172.21.15.172:6789/0 conn(0x7f052c0c9610 legacy=0x7f052c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:07.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f053219c700 1 -- 172.21.15.102:0/2129986061 >> v1:172.21.15.102:6789/0 conn(0x7f052c1564b0 legacy=0x7f052c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:07.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f053219c700 1 -- 172.21.15.102:0/2129986061 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c0b4c00 con 0x7f052c0cab50 2023-03-03T08:55:07.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f053219c700 1 --2- 172.21.15.102:0/2129986061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c1646b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0524009dc0 tx=0x7f052400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2592a111e4cf2d65 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:07.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f0530999700 1 -- 172.21.15.102:0/2129986061 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052400e070 con 0x7f052c0cab50 2023-03-03T08:55:07.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.481+0000 7f0530999700 1 -- 172.21.15.102:0/2129986061 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052400bb20 con 0x7f052c0cab50 2023-03-03T08:55:07.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f0530999700 1 -- 172.21.15.102:0/2129986061 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052400be90 con 0x7f052c0cab50 2023-03-03T08:55:07.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f053299d700 1 -- 172.21.15.102:0/2129986061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 msgr2=0x7f052c1646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:07.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f053299d700 1 --2- 172.21.15.102:0/2129986061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c1646b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0524009dc0 tx=0x7f052400b330 comp rx=0 tx=0).stop 2023-03-03T08:55:07.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f053299d700 1 -- 172.21.15.102:0/2129986061 shutdown_connections 2023-03-03T08:55:07.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f053299d700 1 --2- 172.21.15.102:0/2129986061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c1646b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f053299d700 1 -- 172.21.15.102:0/2129986061 >> 172.21.15.102:0/2129986061 conn(0x7f052c065fb0 msgr2=0x7f052c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:07.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f053299d700 1 -- 172.21.15.102:0/2129986061 shutdown_connections 2023-03-03T08:55:07.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.482+0000 7f053299d700 1 -- 172.21.15.102:0/2129986061 wait complete. 2023-03-03T08:55:07.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.484+0000 7f053299d700 1 Processor -- start 2023-03-03T08:55:07.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.484+0000 7f053299d700 1 -- start start 2023-03-03T08:55:07.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.485+0000 7f053299d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f052c0c9610 0x7f052c1ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:07.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.485+0000 7f053299d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c0c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:07.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.485+0000 7f053299d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f052c1564b0 0x7f052c1e3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:07.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.485+0000 7f053299d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c0b3260 con 0x7f052c1564b0 2023-03-03T08:55:07.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.485+0000 7f053299d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c1e1e90 con 0x7f052c0c9610 2023-03-03T08:55:07.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.485+0000 7f053299d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c1edd60 con 0x7f052c0cab50 2023-03-03T08:55:07.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053119a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c0c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f052c0c9610 0x7f052c1ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f052c0c9610 0x7f052c1ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47150/0 (socket says 172.21.15.102:47150) 2023-03-03T08:55:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 -- 172.21.15.102:0/3347400584 learned_addr learned my addr 172.21.15.102:0/3347400584 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:55:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053119a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c0c6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58212/0 (socket says 172.21.15.102:58212) 2023-03-03T08:55:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053219c700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f052c1564b0 0x7f052c1e3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 -- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 msgr2=0x7f052c0c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c0c6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 -- 172.21.15.102:0/3347400584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f052c1564b0 msgr2=0x7f052c1e3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f052c1564b0 0x7f052c1e3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 -- 172.21.15.102:0/3347400584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0524009a70 con 0x7f052c0c9610 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053219c700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f052c1564b0 0x7f052c1e3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053119a700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c0c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.486+0000 7f053199b700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f052c0c9610 0x7f052c1ee8c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f052800c4d0 tx=0x7f052800c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.487+0000 7f05227fc700 1 -- 172.21.15.102:0/3347400584 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528005d70 con 0x7f052c0c9610 2023-03-03T08:55:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.487+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c0b2210 con 0x7f052c0c9610 2023-03-03T08:55:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.487+0000 7f05227fc700 1 -- 172.21.15.102:0/3347400584 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05280070d0 con 0x7f052c0c9610 2023-03-03T08:55:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.487+0000 7f05227fc700 1 -- 172.21.15.102:0/3347400584 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528007440 con 0x7f052c0c9610 2023-03-03T08:55:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.487+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c0b1f30 con 0x7f052c0c9610 2023-03-03T08:55:07.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.488+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c156880 con 0x7f052c0c9610 2023-03-03T08:55:07.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.489+0000 7f05227fc700 1 -- 172.21.15.102:0/3347400584 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0528019030 con 0x7f052c0c9610 2023-03-03T08:55:07.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.490+0000 7f05227fc700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05100682f0 0x7f051006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:07.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.491+0000 7f05227fc700 1 -- 172.21.15.102:0/3347400584 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0528084fe0 con 0x7f052c0c9610 2023-03-03T08:55:07.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.491+0000 7f053119a700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05100682f0 0x7f051006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:07.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.491+0000 7f053119a700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05100682f0 0x7f051006a7a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f052c0b2ae0 tx=0x7f051c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:07.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.495+0000 7f05227fc700 1 -- 172.21.15.102:0/3347400584 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f052804b5e0 con 0x7f052c0c9610 2023-03-03T08:55:07.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.731+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f052c0677f0 con 0x7f052c0c9610 2023-03-03T08:55:07.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.733+0000 7f05227fc700 1 -- 172.21.15.102:0/3347400584 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0528017020 con 0x7f052c0c9610 2023-03-03T08:55:07.734 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:55:07.734 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:55:07.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.734+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05100682f0 msgr2=0x7f051006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:07.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.734+0000 7f053299d700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05100682f0 0x7f051006a7a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f052c0b2ae0 tx=0x7f051c009040 comp rx=0 tx=0).stop 2023-03-03T08:55:07.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.734+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f052c0c9610 msgr2=0x7f052c1ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:07.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.734+0000 7f053299d700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f052c0c9610 0x7f052c1ee8c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f052800c4d0 tx=0x7f052800c7e0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 shutdown_connections 2023-03-03T08:55:07.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f052c1564b0 0x7f052c1e3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f052c0c9610 0x7f052c1ee8c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f05100682f0 0x7f051006a7a0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 --2- 172.21.15.102:0/3347400584 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f052c0cab50 0x7f052c0c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:07.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 >> 172.21.15.102:0/3347400584 conn(0x7f052c065fb0 msgr2=0x7f052c0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:07.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 shutdown_connections 2023-03-03T08:55:07.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:07.735+0000 7f053299d700 1 -- 172.21.15.102:0/3347400584 wait complete. 2023-03-03T08:55:07.738 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:55:07.764 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-03-03T08:55:08.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:08.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:30.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:55:31.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.015+0000 7f1b4199d700 1 Processor -- start 2023-03-03T08:55:31.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.015+0000 7f1b4199d700 1 -- start start 2023-03-03T08:55:31.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b4199d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1b3c1564b0 0x7f1b3c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:31.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b4199d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b3c0c9610 0x7f1b3c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:31.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b4199d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0c9ca0 con 0x7f1b3c0ca440 2023-03-03T08:55:31.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b4199d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c0b4920 con 0x7f1b3c0c9610 2023-03-03T08:55:31.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b4199d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c0b4c00 con 0x7f1b3c1564b0 2023-03-03T08:55:31.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b3bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b3c0c9610 0x7f1b3c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:31.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b4099b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1b3c1564b0 0x7f1b3c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b3bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b3c0c9610 0x7f1b3c0cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36912/0 (socket says 172.21.15.102:36912) 2023-03-03T08:55:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b3bfff700 1 -- 172.21.15.102:0/405581197 learned_addr learned my addr 172.21.15.102:0/405581197 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:55:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.017+0000 7f1b4119c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1b3c0ca440 0x7f1b3c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36692/0 (socket says 172.21.15.102:36692) 2023-03-03T08:55:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3b7fe700 1 -- 172.21.15.102:0/405581197 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922813522 0 0) 0x7f1b3c0c9ca0 con 0x7f1b3c0ca440 2023-03-03T08:55:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3bfff700 1 -- 172.21.15.102:0/405581197 >> v2:172.21.15.172:3300/0 conn(0x7f1b3c1564b0 msgr2=0x7f1b3c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3bfff700 1 --2- 172.21.15.102:0/405581197 >> v2:172.21.15.172:3300/0 conn(0x7f1b3c1564b0 0x7f1b3c0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3bfff700 1 -- 172.21.15.102:0/405581197 >> v1:172.21.15.172:6789/0 conn(0x7f1b3c0ca440 legacy=0x7f1b3c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3bfff700 1 -- 172.21.15.102:0/405581197 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c0b46f0 con 0x7f1b3c0c9610 2023-03-03T08:55:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3bfff700 1 --2- 172.21.15.102:0/405581197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c0cbba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c08c600 tx=0x7f1b28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3e12b35985ac25 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3b7fe700 1 -- 172.21.15.102:0/405581197 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28011830 con 0x7f1b3c0c9610 2023-03-03T08:55:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3b7fe700 1 -- 172.21.15.102:0/405581197 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b28011990 con 0x7f1b3c0c9610 2023-03-03T08:55:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.018+0000 7f1b3b7fe700 1 -- 172.21.15.102:0/405581197 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28011d20 con 0x7f1b3c0c9610 2023-03-03T08:55:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 -- 172.21.15.102:0/405581197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c0c9610 msgr2=0x7f1b3c0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 --2- 172.21.15.102:0/405581197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c0cbba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c08c600 tx=0x7f1b28009bd0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 -- 172.21.15.102:0/405581197 shutdown_connections 2023-03-03T08:55:31.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 --2- 172.21.15.102:0/405581197 >> v2:172.21.15.172:3300/0 conn(0x7f1b3c1564b0 0x7f1b3c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 --2- 172.21.15.102:0/405581197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c0cbba0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 -- 172.21.15.102:0/405581197 >> 172.21.15.102:0/405581197 conn(0x7f1b3c065fb0 msgr2=0x7f1b3c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 -- 172.21.15.102:0/405581197 shutdown_connections 2023-03-03T08:55:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.019+0000 7f1b4199d700 1 -- 172.21.15.102:0/405581197 wait complete. 2023-03-03T08:55:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.022+0000 7f1b4199d700 1 Processor -- start 2023-03-03T08:55:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.022+0000 7f1b4199d700 1 -- start start 2023-03-03T08:55:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.023+0000 7f1b4199d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c1f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.023+0000 7f1b4199d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b3c0ca440 0x7f1b3c1f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.023+0000 7f1b4199d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c1564b0 0x7f1b3c0c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.023+0000 7f1b4199d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0c3f30 con 0x7f1b3c1564b0 2023-03-03T08:55:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.023+0000 7f1b4199d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0c31c0 con 0x7f1b3c0c9610 2023-03-03T08:55:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.023+0000 7f1b4199d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0c7ad0 con 0x7f1b3c0ca440 2023-03-03T08:55:31.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4119c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c1564b0 0x7f1b3c0c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4099b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c1f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b3bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b3c0ca440 0x7f1b3c1f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4099b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c1f1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56914/0 (socket says 172.21.15.102:56914) 2023-03-03T08:55:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4119c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c1564b0 0x7f1b3c0c4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36924/0 (socket says 172.21.15.102:36924) 2023-03-03T08:55:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4099b700 1 -- 172.21.15.102:0/3701775839 learned_addr learned my addr 172.21.15.102:0/3701775839 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:55:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4119c700 1 -- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b3c0ca440 msgr2=0x7f1b3c1f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4119c700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b3c0ca440 0x7f1b3c1f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4119c700 1 -- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b3c0c9610 msgr2=0x7f1b3c1f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4119c700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c1f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4119c700 1 -- 172.21.15.102:0/3701775839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b28009550 con 0x7f1b3c1564b0 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b3bfff700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b3c0ca440 0x7f1b3c1f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.024+0000 7f1b4099b700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c1f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.025+0000 7f1b4119c700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c1564b0 0x7f1b3c0c4350 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400cbe0 tx=0x7f1b3400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.025+0000 7f1b397fa700 1 -- 172.21.15.102:0/3701775839 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400aa40 con 0x7f1b3c1564b0 2023-03-03T08:55:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.025+0000 7f1b397fa700 1 -- 172.21.15.102:0/3701775839 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3400abc0 con 0x7f1b3c1564b0 2023-03-03T08:55:31.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.025+0000 7f1b397fa700 1 -- 172.21.15.102:0/3701775839 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b340070d0 con 0x7f1b3c1564b0 2023-03-03T08:55:31.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.025+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c0b2210 con 0x7f1b3c1564b0 2023-03-03T08:55:31.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.025+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c0b1f30 con 0x7f1b3c1564b0 2023-03-03T08:55:31.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.026+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c156880 con 0x7f1b3c1564b0 2023-03-03T08:55:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.027+0000 7f1b397fa700 1 -- 172.21.15.102:0/3701775839 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b34010070 con 0x7f1b3c1564b0 2023-03-03T08:55:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.029+0000 7f1b397fa700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b200682f0 0x7f1b2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.029+0000 7f1b397fa700 1 -- 172.21.15.102:0/3701775839 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1b34085600 con 0x7f1b3c1564b0 2023-03-03T08:55:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.029+0000 7f1b4099b700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b200682f0 0x7f1b2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.030+0000 7f1b4099b700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b200682f0 0x7f1b2006a7a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1b300112c0 tx=0x7f1b30009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.033+0000 7f1b397fa700 1 -- 172.21.15.102:0/3701775839 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b34051d10 con 0x7f1b3c1564b0 2023-03-03T08:55:31.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.266+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b3c0c8f10 con 0x7f1b3c1564b0 2023-03-03T08:55:31.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.266+0000 7f1b397fa700 1 -- 172.21.15.102:0/3701775839 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1b34055330 con 0x7f1b3c1564b0 2023-03-03T08:55:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:55:31.268 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:55:31.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b200682f0 msgr2=0x7f1b2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:31.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b200682f0 0x7f1b2006a7a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1b300112c0 tx=0x7f1b30009210 comp rx=0 tx=0).stop 2023-03-03T08:55:31.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c1564b0 msgr2=0x7f1b3c0c4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:31.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c1564b0 0x7f1b3c0c4350 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400cbe0 tx=0x7f1b3400cef0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 shutdown_connections 2023-03-03T08:55:31.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b3c1564b0 0x7f1b3c0c4350 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b3c0c9610 0x7f1b3c1f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b200682f0 0x7f1b2006a7a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 --2- 172.21.15.102:0/3701775839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b3c0ca440 0x7f1b3c1f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 >> 172.21.15.102:0/3701775839 conn(0x7f1b3c065fb0 msgr2=0x7f1b3c154f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 shutdown_connections 2023-03-03T08:55:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:55:31.269+0000 7f1b4199d700 1 -- 172.21.15.102:0/3701775839 wait complete. 2023-03-03T08:55:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:55:31.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2023-03-03T08:55:39.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:39.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:01.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:56:01.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.552+0000 7eff487ff700 1 Processor -- start 2023-03-03T08:56:01.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.553+0000 7eff487ff700 1 -- start start 2023-03-03T08:56:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff487ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7eff400cd8e0 0x7eff400cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff487ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7eff400cd390 0x7eff400ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff487ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff40153660 con 0x7eff40153290 2023-03-03T08:56:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff487ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4006c360 con 0x7eff400cd8e0 2023-03-03T08:56:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff487ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4015cea0 con 0x7eff400cd390 2023-03-03T08:56:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff45d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7eff400cd8e0 0x7eff400cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff4659b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7eff40153290 0x7eff40158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59968/0 (socket says 172.21.15.102:59968) 2023-03-03T08:56:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.554+0000 7eff4659b700 1 -- 172.21.15.102:0/1103379956 learned_addr learned my addr 172.21.15.102:0/1103379956 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:56:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff46d9c700 1 --2- 172.21.15.102:0/1103379956 >> v2:172.21.15.172:3300/0 conn(0x7eff400cd390 0x7eff400ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff45599700 1 -- 172.21.15.102:0/1103379956 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822877907 0 0) 0x7eff40153660 con 0x7eff40153290 2023-03-03T08:56:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff45599700 1 -- 172.21.15.102:0/1103379956 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff2c0035b0 con 0x7eff40153290 2023-03-03T08:56:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff45d9a700 1 -- 172.21.15.102:0/1103379956 >> v2:172.21.15.172:3300/0 conn(0x7eff400cd390 msgr2=0x7eff400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:56:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff45d9a700 1 --2- 172.21.15.102:0/1103379956 >> v2:172.21.15.172:3300/0 conn(0x7eff400cd390 0x7eff400ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff45d9a700 1 -- 172.21.15.102:0/1103379956 >> v1:172.21.15.172:6789/0 conn(0x7eff40153290 legacy=0x7eff40158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff45d9a700 1 -- 172.21.15.102:0/1103379956 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4006c7f0 con 0x7eff400cd8e0 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.555+0000 7eff45d9a700 1 --2- 172.21.15.102:0/1103379956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff400cde20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7eff400c9390 tx=0x7eff30009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1befd0c1cb61393 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff45599700 1 -- 172.21.15.102:0/1103379956 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30011830 con 0x7eff400cd8e0 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff45599700 1 -- 172.21.15.102:0/1103379956 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff30011990 con 0x7eff400cd8e0 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff45599700 1 -- 172.21.15.102:0/1103379956 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30011d20 con 0x7eff400cd8e0 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff487ff700 1 -- 172.21.15.102:0/1103379956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 msgr2=0x7eff400cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff487ff700 1 --2- 172.21.15.102:0/1103379956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff400cde20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7eff400c9390 tx=0x7eff30009b20 comp rx=0 tx=0).stop 2023-03-03T08:56:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff487ff700 1 -- 172.21.15.102:0/1103379956 shutdown_connections 2023-03-03T08:56:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff487ff700 1 --2- 172.21.15.102:0/1103379956 >> v2:172.21.15.172:3300/0 conn(0x7eff400cd390 0x7eff400ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff487ff700 1 --2- 172.21.15.102:0/1103379956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff400cde20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.556+0000 7eff487ff700 1 -- 172.21.15.102:0/1103379956 >> 172.21.15.102:0/1103379956 conn(0x7eff40065fb0 msgr2=0x7eff400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:56:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.557+0000 7eff487ff700 1 -- 172.21.15.102:0/1103379956 shutdown_connections 2023-03-03T08:56:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.557+0000 7eff487ff700 1 -- 172.21.15.102:0/1103379956 wait complete. 2023-03-03T08:56:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.559+0000 7eff487ff700 1 Processor -- start 2023-03-03T08:56:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.560+0000 7eff487ff700 1 -- start start 2023-03-03T08:56:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.561+0000 7eff487ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7eff400cd390 0x7eff400c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.561+0000 7eff487ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff401e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.561+0000 7eff487ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7eff40153290 0x7eff401e44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.561+0000 7eff487ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff401e3aa0 con 0x7eff400cd8e0 2023-03-03T08:56:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.561+0000 7eff487ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff401e3200 con 0x7eff400cd390 2023-03-03T08:56:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.561+0000 7eff487ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff401e34e0 con 0x7eff40153290 2023-03-03T08:56:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.561+0000 7eff45d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff401e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff401e3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49478/0 (socket says 172.21.15.102:49478) 2023-03-03T08:56:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 -- 172.21.15.102:0/2864855722 learned_addr learned my addr 172.21.15.102:0/2864855722 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:56:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff4659b700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7eff400cd390 0x7eff400c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff46d9c700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7eff40153290 0x7eff401e44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 -- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7eff40153290 msgr2=0x7eff401e44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7eff40153290 0x7eff401e44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 -- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7eff400cd390 msgr2=0x7eff400c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7eff400cd390 0x7eff400c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 -- 172.21.15.102:0/2864855722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff300094a0 con 0x7eff400cd8e0 2023-03-03T08:56:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.562+0000 7eff45d9a700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff401e3fa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7eff30009b60 tx=0x7eff30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:56:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.563+0000 7eff36ffd700 1 -- 172.21.15.102:0/2864855722 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30004590 con 0x7eff400cd8e0 2023-03-03T08:56:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.563+0000 7eff36ffd700 1 -- 172.21.15.102:0/2864855722 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3001f070 con 0x7eff400cd8e0 2023-03-03T08:56:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.563+0000 7eff36ffd700 1 -- 172.21.15.102:0/2864855722 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff300038e0 con 0x7eff400cd8e0 2023-03-03T08:56:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.563+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff400ccde0 con 0x7eff400cd8e0 2023-03-03T08:56:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.563+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4015d160 con 0x7eff400cd8e0 2023-03-03T08:56:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.564+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff400cd0c0 con 0x7eff400cd8e0 2023-03-03T08:56:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.565+0000 7eff36ffd700 1 -- 172.21.15.102:0/2864855722 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff3000f050 con 0x7eff400cd8e0 2023-03-03T08:56:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.567+0000 7eff36ffd700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff240682f0 0x7eff2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.567+0000 7eff36ffd700 1 -- 172.21.15.102:0/2864855722 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7eff30021070 con 0x7eff400cd8e0 2023-03-03T08:56:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.567+0000 7eff4659b700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff240682f0 0x7eff2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.568+0000 7eff4659b700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff240682f0 0x7eff2406a7a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7eff3c007fd0 tx=0x7eff3c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:56:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.572+0000 7eff36ffd700 1 -- 172.21.15.102:0/2864855722 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7eff30053c20 con 0x7eff400cd8e0 2023-03-03T08:56:01.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.805+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff4004e6e0 con 0x7eff400cd8e0 2023-03-03T08:56:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.806+0000 7eff36ffd700 1 -- 172.21.15.102:0/2864855722 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7eff30057240 con 0x7eff400cd8e0 2023-03-03T08:56:01.808 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:56:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:56:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff240682f0 msgr2=0x7eff2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff240682f0 0x7eff2406a7a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7eff3c007fd0 tx=0x7eff3c007dc0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 msgr2=0x7eff401e3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff401e3fa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7eff30009b60 tx=0x7eff30004750 comp rx=0 tx=0).stop 2023-03-03T08:56:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 shutdown_connections 2023-03-03T08:56:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff240682f0 0x7eff2406a7a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7eff40153290 0x7eff401e44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7eff400cd390 0x7eff400c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 --2- 172.21.15.102:0/2864855722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff400cd8e0 0x7eff401e3fa0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 >> 172.21.15.102:0/2864855722 conn(0x7eff40065fb0 msgr2=0x7eff40156ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:56:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 shutdown_connections 2023-03-03T08:56:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:56:01.809+0000 7eff487ff700 1 -- 172.21.15.102:0/2864855722 wait complete. 2023-03-03T08:56:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:56:01.838 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2023-03-03T08:56:09.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:09.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:39.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:39.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:09.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:09.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:21.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:57:22.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.085+0000 7fa2d1f9d700 1 Processor -- start 2023-03-03T08:57:22.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.085+0000 7fa2d1f9d700 1 -- start start 2023-03-03T08:57:22.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d1f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0ced00 0x7fa2cc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:22.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d1f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2cc0b3970 0x7fa2cc0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:22.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d1f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc0c3d20 con 0x7fa2cc0cf0d0 2023-03-03T08:57:22.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d1f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2cc0c4930 con 0x7fa2cc0b3970 2023-03-03T08:57:22.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d1f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc15cba0 con 0x7fa2cc0ced00 2023-03-03T08:57:22.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2cbfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2cc0b3970 0x7fa2cc0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:22.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d179c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa2cc0cf0d0 0x7fa2cc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49514/0 (socket says 172.21.15.102:49514) 2023-03-03T08:57:22.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d0f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0ced00 0x7fa2cc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:22.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.087+0000 7fa2d179c700 1 -- 172.21.15.102:0/845299408 learned_addr learned my addr 172.21.15.102:0/845299408 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:57:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cbfff700 1 -- 172.21.15.102:0/845299408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0ced00 msgr2=0x7fa2cc0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:57:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cbfff700 1 --2- 172.21.15.102:0/845299408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0ced00 0x7fa2cc0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cbfff700 1 -- 172.21.15.102:0/845299408 >> v1:172.21.15.102:6789/0 conn(0x7fa2cc0cf0d0 legacy=0x7fa2cc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cbfff700 1 -- 172.21.15.102:0/845299408 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc15ce80 con 0x7fa2cc0b3970 2023-03-03T08:57:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cb7fe700 1 -- 172.21.15.102:0/845299408 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917494174 0 0) 0x7fa2cc0c3d20 con 0x7fa2cc0cf0d0 2023-03-03T08:57:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cbfff700 1 --2- 172.21.15.102:0/845299408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0b3970 0x7fa2cc0c41b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009840 tx=0x7fa2b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1261d8f163b6b7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cb7fe700 1 -- 172.21.15.102:0/845299408 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8011830 con 0x7fa2cc0b3970 2023-03-03T08:57:22.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cb7fe700 1 -- 172.21.15.102:0/845299408 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b8011990 con 0x7fa2cc0b3970 2023-03-03T08:57:22.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.088+0000 7fa2cb7fe700 1 -- 172.21.15.102:0/845299408 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8011d20 con 0x7fa2cc0b3970 2023-03-03T08:57:22.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/845299408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0b3970 msgr2=0x7fa2cc0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:22.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/845299408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0b3970 0x7fa2cc0c41b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009840 tx=0x7fa2b8009b50 comp rx=0 tx=0).stop 2023-03-03T08:57:22.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/845299408 shutdown_connections 2023-03-03T08:57:22.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/845299408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0ced00 0x7fa2cc0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/845299408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0b3970 0x7fa2cc0c41b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/845299408 >> 172.21.15.102:0/845299408 conn(0x7fa2cc08b1c0 msgr2=0x7fa2cc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:22.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/845299408 shutdown_connections 2023-03-03T08:57:22.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.089+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/845299408 wait complete. 2023-03-03T08:57:22.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.091+0000 7fa2d1f9d700 1 Processor -- start 2023-03-03T08:57:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.091+0000 7fa2d1f9d700 1 -- start start 2023-03-03T08:57:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.092+0000 7fa2d1f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0b3970 0x7fa2cc1f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.092+0000 7fa2d1f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0ced00 0x7fa2cc0c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.092+0000 7fa2d1f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2cc0cf0d0 0x7fa2cc1489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.092+0000 7fa2d1f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc1482e0 con 0x7fa2cc0ced00 2023-03-03T08:57:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.092+0000 7fa2d1f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc1f2690 con 0x7fa2cc0cf0d0 2023-03-03T08:57:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.092+0000 7fa2d1f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc148550 con 0x7fa2cc0b3970 2023-03-03T08:57:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d179c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2cc0cf0d0 0x7fa2cc1489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0b3970 0x7fa2cc1f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0b3970 0x7fa2cc1f2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44654/0 (socket says 172.21.15.102:44654) 2023-03-03T08:57:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 -- 172.21.15.102:0/2623048849 learned_addr learned my addr 172.21.15.102:0/2623048849 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:57:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2cbfff700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0ced00 0x7fa2cc0c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 -- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2cc0cf0d0 msgr2=0x7fa2cc1489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2cc0cf0d0 0x7fa2cc1489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 -- 172.21.15.102:0/2623048849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0ced00 msgr2=0x7fa2cc0c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:22.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0ced00 0x7fa2cc0c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d0f9b700 1 -- 172.21.15.102:0/2623048849 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80094a0 con 0x7fa2cc0b3970 2023-03-03T08:57:22.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2cbfff700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0ced00 0x7fa2cc0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:57:22.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.093+0000 7fa2d179c700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2cc0cf0d0 0x7fa2cc1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:57:22.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.094+0000 7fa2d0f9b700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0b3970 0x7fa2cc1f2150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000e9e0 tx=0x7fa2c000ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.094+0000 7fa2c97fa700 1 -- 172.21.15.102:0/2623048849 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0009710 con 0x7fa2cc0b3970 2023-03-03T08:57:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.094+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2cc06c200 con 0x7fa2cc0b3970 2023-03-03T08:57:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.094+0000 7fa2c97fa700 1 -- 172.21.15.102:0/2623048849 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c000ee80 con 0x7fa2cc0b3970 2023-03-03T08:57:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.094+0000 7fa2c97fa700 1 -- 172.21.15.102:0/2623048849 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c000c9f0 con 0x7fa2cc0b3970 2023-03-03T08:57:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.094+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2cc06bf20 con 0x7fa2cc0b3970 2023-03-03T08:57:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.095+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2cc15c890 con 0x7fa2cc0b3970 2023-03-03T08:57:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.097+0000 7fa2c97fa700 1 -- 172.21.15.102:0/2623048849 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa2c0019030 con 0x7fa2cc0b3970 2023-03-03T08:57:22.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.098+0000 7fa2c97fa700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2b0068320 0x7fa2b006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:22.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.098+0000 7fa2c97fa700 1 -- 172.21.15.102:0/2623048849 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa2c0085c50 con 0x7fa2cc0b3970 2023-03-03T08:57:22.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.098+0000 7fa2cbfff700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2b0068320 0x7fa2b006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:22.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.099+0000 7fa2cbfff700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2b0068320 0x7fa2b006a7d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8000c00 tx=0x7fa2b801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:22.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.102+0000 7fa2c97fa700 1 -- 172.21.15.102:0/2623048849 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa2c00522e0 con 0x7fa2cc0b3970 2023-03-03T08:57:22.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.336+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2cc148830 con 0x7fa2cc0b3970 2023-03-03T08:57:22.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.337+0000 7fa2c97fa700 1 -- 172.21.15.102:0/2623048849 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa2c0055900 con 0x7fa2cc0b3970 2023-03-03T08:57:22.339 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:57:22.339 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:57:22.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2b0068320 msgr2=0x7fa2b006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:22.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2b0068320 0x7fa2b006a7d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8000c00 tx=0x7fa2b801c040 comp rx=0 tx=0).stop 2023-03-03T08:57:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0b3970 msgr2=0x7fa2cc1f2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0b3970 0x7fa2cc1f2150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000e9e0 tx=0x7fa2c000ecf0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 shutdown_connections 2023-03-03T08:57:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2b0068320 0x7fa2b006a7d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2cc0b3970 0x7fa2cc1f2150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2cc0cf0d0 0x7fa2cc1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 --2- 172.21.15.102:0/2623048849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2cc0ced00 0x7fa2cc0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 >> 172.21.15.102:0/2623048849 conn(0x7fa2cc08b1c0 msgr2=0x7fa2cc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 shutdown_connections 2023-03-03T08:57:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:57:22.339+0000 7fa2d1f9d700 1 -- 172.21.15.102:0/2623048849 wait complete. 2023-03-03T08:57:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:57:22.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-03-03T08:57:39.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:39.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:10.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:10.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:40.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:40.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:04.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:59:04.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.622+0000 7fc25e5ff700 1 Processor -- start 2023-03-03T08:59:04.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.622+0000 7fc25e5ff700 1 -- start start 2023-03-03T08:59:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc25e5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc258153260 0x7fc2580ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc25e5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2580cac20 0x7fc258163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc25e5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2580cf940 con 0x7fc2580c9cb0 2023-03-03T08:59:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc25e5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2580cfe70 con 0x7fc2580cac20 2023-03-03T08:59:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc25e5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc258158d70 con 0x7fc258153260 2023-03-03T08:59:04.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc2577fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc2580c9cb0 0x7fc2580cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38048/0 (socket says 172.21.15.102:38048) 2023-03-03T08:59:04.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc2577fe700 1 -- 172.21.15.102:0/2760253133 learned_addr learned my addr 172.21.15.102:0/2760253133 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:59:04.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc257fff700 1 --2- 172.21.15.102:0/2760253133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc258153260 0x7fc2580ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:04.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc25cb9c700 1 --2- 172.21.15.102:0/2760253133 >> v2:172.21.15.102:3300/0 conn(0x7fc2580cac20 0x7fc258163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.624+0000 7fc256ffd700 1 -- 172.21.15.102:0/2760253133 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904348584 0 0) 0x7fc2580cf940 con 0x7fc2580c9cb0 2023-03-03T08:59:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc257fff700 1 -- 172.21.15.102:0/2760253133 >> v2:172.21.15.102:3300/0 conn(0x7fc2580cac20 msgr2=0x7fc258163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:59:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc257fff700 1 --2- 172.21.15.102:0/2760253133 >> v2:172.21.15.102:3300/0 conn(0x7fc2580cac20 0x7fc258163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc257fff700 1 -- 172.21.15.102:0/2760253133 >> v1:172.21.15.172:6789/0 conn(0x7fc2580c9cb0 legacy=0x7fc2580cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc257fff700 1 -- 172.21.15.102:0/2760253133 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2580cfb90 con 0x7fc258153260 2023-03-03T08:59:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc257fff700 1 --2- 172.21.15.102:0/2760253133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc258153260 0x7fc2580ca6e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc248009840 tx=0x7fc248009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68c30f5e1978615 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc256ffd700 1 -- 172.21.15.102:0/2760253133 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248011830 con 0x7fc258153260 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc256ffd700 1 -- 172.21.15.102:0/2760253133 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc248011990 con 0x7fc258153260 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.625+0000 7fc256ffd700 1 -- 172.21.15.102:0/2760253133 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248011d20 con 0x7fc258153260 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 -- 172.21.15.102:0/2760253133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc258153260 msgr2=0x7fc2580ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/2760253133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc258153260 0x7fc2580ca6e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc248009840 tx=0x7fc248009b50 comp rx=0 tx=0).stop 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 -- 172.21.15.102:0/2760253133 shutdown_connections 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/2760253133 >> v2:172.21.15.102:3300/0 conn(0x7fc2580cac20 0x7fc258163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/2760253133 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc258153260 0x7fc2580ca6e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 -- 172.21.15.102:0/2760253133 >> 172.21.15.102:0/2760253133 conn(0x7fc258066c00 msgr2=0x7fc2580ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:04.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 -- 172.21.15.102:0/2760253133 shutdown_connections 2023-03-03T08:59:04.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.626+0000 7fc25e5ff700 1 -- 172.21.15.102:0/2760253133 wait complete. 2023-03-03T08:59:04.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.628+0000 7fc25e5ff700 1 Processor -- start 2023-03-03T08:59:04.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.628+0000 7fc25e5ff700 1 -- start start 2023-03-03T08:59:04.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc25e5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2580cac20 0x7fc25814e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc25e5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc258153260 0x7fc2581538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc25e5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2581549e0 0x7fc258153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc25e5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc258154800 con 0x7fc258153260 2023-03-03T08:59:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc25e5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2580b1f20 con 0x7fc2580cac20 2023-03-03T08:59:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc25e5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc258154520 con 0x7fc2581549e0 2023-03-03T08:59:04.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc2577fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc258153260 0x7fc2581538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc257fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2580cac20 0x7fc25814e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc257fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2580cac20 0x7fc25814e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53182/0 (socket says 172.21.15.102:53182) 2023-03-03T08:59:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc257fff700 1 -- 172.21.15.102:0/1360428615 learned_addr learned my addr 172.21.15.102:0/1360428615 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:59:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.629+0000 7fc25cb9c700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2581549e0 0x7fc258153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc257fff700 1 -- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2581549e0 msgr2=0x7fc258153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc257fff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2581549e0 0x7fc258153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc257fff700 1 -- 172.21.15.102:0/1360428615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc258153260 msgr2=0x7fc2581538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc257fff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc258153260 0x7fc2581538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc257fff700 1 -- 172.21.15.102:0/1360428615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480094a0 con 0x7fc2580cac20 2023-03-03T08:59:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc2577fe700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc258153260 0x7fc2581538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:59:04.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc257fff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2580cac20 0x7fc25814e840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc2480090b0 tx=0x7fc248004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:04.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc254ff9700 1 -- 172.21.15.102:0/1360428615 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2480036a0 con 0x7fc2580cac20 2023-03-03T08:59:04.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.630+0000 7fc254ff9700 1 -- 172.21.15.102:0/1360428615 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24801f070 con 0x7fc2580cac20 2023-03-03T08:59:04.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.631+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc258159260 con 0x7fc2580cac20 2023-03-03T08:59:04.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.631+0000 7fc254ff9700 1 -- 172.21.15.102:0/1360428615 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248003a60 con 0x7fc2580cac20 2023-03-03T08:59:04.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.631+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc258158f80 con 0x7fc2580cac20 2023-03-03T08:59:04.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.632+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc258159540 con 0x7fc2580cac20 2023-03-03T08:59:04.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.633+0000 7fc254ff9700 1 -- 172.21.15.102:0/1360428615 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc248048070 con 0x7fc2580cac20 2023-03-03T08:59:04.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.634+0000 7fc254ff9700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc244068320 0x7fc24406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:04.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.635+0000 7fc254ff9700 1 -- 172.21.15.102:0/1360428615 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc248021070 con 0x7fc2580cac20 2023-03-03T08:59:04.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.635+0000 7fc2577fe700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc244068320 0x7fc24406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:04.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.635+0000 7fc2577fe700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc244068320 0x7fc24406a7d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc258157180 tx=0x7fc24c007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:04.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.637+0000 7fc254ff9700 1 -- 172.21.15.102:0/1360428615 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2480548d0 con 0x7fc2580cac20 2023-03-03T08:59:04.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.870+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc25804e6e0 con 0x7fc2580cac20 2023-03-03T08:59:04.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.871+0000 7fc254ff9700 1 -- 172.21.15.102:0/1360428615 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc248057ef0 con 0x7fc2580cac20 2023-03-03T08:59:04.874 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:59:04.874 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:59:04.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc244068320 msgr2=0x7fc24406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:04.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc244068320 0x7fc24406a7d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc258157180 tx=0x7fc24c007e70 comp rx=0 tx=0).stop 2023-03-03T08:59:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2580cac20 msgr2=0x7fc25814e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2580cac20 0x7fc25814e840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc2480090b0 tx=0x7fc248004750 comp rx=0 tx=0).stop 2023-03-03T08:59:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 shutdown_connections 2023-03-03T08:59:04.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc244068320 0x7fc24406a7d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2581549e0 0x7fc258153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2580cac20 0x7fc25814e840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 --2- 172.21.15.102:0/1360428615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc258153260 0x7fc2581538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 >> 172.21.15.102:0/1360428615 conn(0x7fc2580c9cb0 msgr2=0x7fc2580b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 shutdown_connections 2023-03-03T08:59:04.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:04.874+0000 7fc25e5ff700 1 -- 172.21.15.102:0/1360428615 wait complete. 2023-03-03T08:59:04.879 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:59:04.903 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-03-03T08:59:10.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:10.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:12.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:59:13.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.151+0000 7fd87bfff700 1 Processor -- start 2023-03-03T08:59:13.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.151+0000 7fd87bfff700 1 -- start start 2023-03-03T08:59:13.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd87bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:13.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd87bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd87c0cab50 0x7fd87c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:13.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd87bfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c0c9ca0 con 0x7fd87c0c9610 2023-03-03T08:59:13.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd87bfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87c0b4920 con 0x7fd87c0cab50 2023-03-03T08:59:13.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd87bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c0b4c00 con 0x7fd87c1564b0 2023-03-03T08:59:13.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd87affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:13.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd87b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd87c0cab50 0x7fd87c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:13.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd873fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd87c0c9610 0x7fd87c0ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38080/0 (socket says 172.21.15.102:38080) 2023-03-03T08:59:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.153+0000 7fd873fff700 1 -- 172.21.15.102:0/1456186150 learned_addr learned my addr 172.21.15.102:0/1456186150 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:59:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87a7fc700 1 -- 172.21.15.102:0/1456186150 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018091021 0 0) 0x7fd87c0c9ca0 con 0x7fd87c0c9610 2023-03-03T08:59:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87affd700 1 -- 172.21.15.102:0/1456186150 >> v2:172.21.15.172:3300/0 conn(0x7fd87c0cab50 msgr2=0x7fd87c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:59:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87affd700 1 --2- 172.21.15.102:0/1456186150 >> v2:172.21.15.172:3300/0 conn(0x7fd87c0cab50 0x7fd87c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87affd700 1 -- 172.21.15.102:0/1456186150 >> v1:172.21.15.172:6789/0 conn(0x7fd87c0c9610 legacy=0x7fd87c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87affd700 1 -- 172.21.15.102:0/1456186150 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c0b46f0 con 0x7fd87c1564b0 2023-03-03T08:59:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87affd700 1 --2- 172.21.15.102:0/1456186150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c0cb660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009840 tx=0x7fd86c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af59204e4b4e433f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:13.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87a7fc700 1 -- 172.21.15.102:0/1456186150 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c011830 con 0x7fd87c1564b0 2023-03-03T08:59:13.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87a7fc700 1 -- 172.21.15.102:0/1456186150 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c011990 con 0x7fd87c1564b0 2023-03-03T08:59:13.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.154+0000 7fd87a7fc700 1 -- 172.21.15.102:0/1456186150 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c011d20 con 0x7fd87c1564b0 2023-03-03T08:59:13.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 -- 172.21.15.102:0/1456186150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 msgr2=0x7fd87c0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:13.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 --2- 172.21.15.102:0/1456186150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c0cb660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009840 tx=0x7fd86c009b50 comp rx=0 tx=0).stop 2023-03-03T08:59:13.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 -- 172.21.15.102:0/1456186150 shutdown_connections 2023-03-03T08:59:13.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 --2- 172.21.15.102:0/1456186150 >> v2:172.21.15.172:3300/0 conn(0x7fd87c0cab50 0x7fd87c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 --2- 172.21.15.102:0/1456186150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c0cb660 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 -- 172.21.15.102:0/1456186150 >> 172.21.15.102:0/1456186150 conn(0x7fd87c065fb0 msgr2=0x7fd87c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:13.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 -- 172.21.15.102:0/1456186150 shutdown_connections 2023-03-03T08:59:13.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.155+0000 7fd87bfff700 1 -- 172.21.15.102:0/1456186150 wait complete. 2023-03-03T08:59:13.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.157+0000 7fd87bfff700 1 Processor -- start 2023-03-03T08:59:13.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.157+0000 7fd87bfff700 1 -- start start 2023-03-03T08:59:13.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.158+0000 7fd87bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c0c9610 0x7fd87c0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:13.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.158+0000 7fd87bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd87c0cab50 0x7fd87c1e2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:13.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.158+0000 7fd87bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c1e2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:13.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.158+0000 7fd87bfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c1e1f40 con 0x7fd87c0cab50 2023-03-03T08:59:13.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.158+0000 7fd87bfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c1e10b0 con 0x7fd87c0c9610 2023-03-03T08:59:13.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.158+0000 7fd87bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c1e21e0 con 0x7fd87c1564b0 2023-03-03T08:59:13.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c0c9610 0x7fd87c0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:13.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c1e2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:13.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd873fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd87c0cab50 0x7fd87c1e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:13.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c0c9610 0x7fd87c0cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59558/0 (socket says 172.21.15.102:59558) 2023-03-03T08:59:13.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 -- 172.21.15.102:0/3249278170 learned_addr learned my addr 172.21.15.102:0/3249278170 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:59:13.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 -- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 msgr2=0x7fd87c1e2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:13.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c1e2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 -- 172.21.15.102:0/3249278170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd87c0cab50 msgr2=0x7fd87c1e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd87c0cab50 0x7fd87c1e2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 -- 172.21.15.102:0/3249278170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0094a0 con 0x7fd87c0c9610 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87b7fe700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c1e2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.159+0000 7fd87affd700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c0c9610 0x7fd87c0cf470 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009fd0 tx=0x7fd86c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.160+0000 7fd8737fe700 1 -- 172.21.15.102:0/3249278170 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c0036a0 con 0x7fd87c0c9610 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.160+0000 7fd8737fe700 1 -- 172.21.15.102:0/3249278170 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c01f070 con 0x7fd87c0c9610 2023-03-03T08:59:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.160+0000 7fd8737fe700 1 -- 172.21.15.102:0/3249278170 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c003a60 con 0x7fd87c0c9610 2023-03-03T08:59:13.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.160+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c0b2210 con 0x7fd87c0c9610 2023-03-03T08:59:13.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.160+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c0b1f30 con 0x7fd87c0c9610 2023-03-03T08:59:13.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.161+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c156880 con 0x7fd87c0c9610 2023-03-03T08:59:13.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.162+0000 7fd8737fe700 1 -- 172.21.15.102:0/3249278170 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd86c00f050 con 0x7fd87c0c9610 2023-03-03T08:59:13.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.164+0000 7fd8737fe700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c0682f0 0x7fd85c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:13.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.164+0000 7fd8737fe700 1 -- 172.21.15.102:0/3249278170 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd86c021070 con 0x7fd87c0c9610 2023-03-03T08:59:13.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.164+0000 7fd873fff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c0682f0 0x7fd85c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:13.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.164+0000 7fd873fff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c0682f0 0x7fd85c06a7a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0c6ab0 tx=0x7fd864007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:13.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.168+0000 7fd8737fe700 1 -- 172.21.15.102:0/3249278170 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd86c053890 con 0x7fd87c0c9610 2023-03-03T08:59:13.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.406+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd87c1e41c0 con 0x7fd87c0c9610 2023-03-03T08:59:13.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.407+0000 7fd8737fe700 1 -- 172.21.15.102:0/3249278170 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd86c015020 con 0x7fd87c0c9610 2023-03-03T08:59:13.408 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:59:13.408 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:59:13.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.409+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c0682f0 msgr2=0x7fd85c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:13.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.409+0000 7fd87bfff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c0682f0 0x7fd85c06a7a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0c6ab0 tx=0x7fd864007de0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.409+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c0c9610 msgr2=0x7fd87c0cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:13.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.409+0000 7fd87bfff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c0c9610 0x7fd87c0cf470 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009fd0 tx=0x7fd86c004750 comp rx=0 tx=0).stop 2023-03-03T08:59:13.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 shutdown_connections 2023-03-03T08:59:13.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c0682f0 0x7fd85c06a7a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd87c1564b0 0x7fd87c1e2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd87c0c9610 0x7fd87c0cf470 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 --2- 172.21.15.102:0/3249278170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd87c0cab50 0x7fd87c1e2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:13.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 >> 172.21.15.102:0/3249278170 conn(0x7fd87c065fb0 msgr2=0x7fd87c1e32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:13.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 shutdown_connections 2023-03-03T08:59:13.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:13.410+0000 7fd87bfff700 1 -- 172.21.15.102:0/3249278170 wait complete. 2023-03-03T08:59:13.413 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:59:13.437 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 15 secs before thrashing 2023-03-03T08:59:28.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:59:28.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.702+0000 7fd1c0fff700 1 Processor -- start 2023-03-03T08:59:28.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.703+0000 7fd1c0fff700 1 -- start start 2023-03-03T08:59:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.704+0000 7fd1c0fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd1bc153290 0x7fd1bc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.704+0000 7fd1c0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.704+0000 7fd1c0fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc153660 con 0x7fd1bc1593c0 2023-03-03T08:59:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.704+0000 7fd1c0fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc06c360 con 0x7fd1bc153290 2023-03-03T08:59:28.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.704+0000 7fd1c0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc15cea0 con 0x7fd1bc0cd8e0 2023-03-03T08:59:28.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.704+0000 7fd1b9d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd1bc1593c0 0x7fd1bc0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41406/0 (socket says 172.21.15.102:41406) 2023-03-03T08:59:28.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1b9d9b700 1 -- 172.21.15.102:0/124788743 learned_addr learned my addr 172.21.15.102:0/124788743 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T08:59:28.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1ba59c700 1 --2- 172.21.15.102:0/124788743 >> v2:172.21.15.172:3300/0 conn(0x7fd1bc153290 0x7fd1bc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1bad9d700 1 --2- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1ba59c700 1 -- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 msgr2=0x7fd1bc0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:59:28.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1b959a700 1 -- 172.21.15.102:0/124788743 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333431211 0 0) 0x7fd1bc153660 con 0x7fd1bc1593c0 2023-03-03T08:59:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1ba59c700 1 --2- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1ba59c700 1 -- 172.21.15.102:0/124788743 >> v1:172.21.15.102:6789/0 conn(0x7fd1bc1593c0 legacy=0x7fd1bc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1ba59c700 1 -- 172.21.15.102:0/124788743 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc06c7f0 con 0x7fd1bc153290 2023-03-03T08:59:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.705+0000 7fd1ba59c700 1 --2- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc153290 0x7fd1bc158e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009840 tx=0x7fd1ac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1fc1db35615e216 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1b959a700 1 -- 172.21.15.102:0/124788743 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac011830 con 0x7fd1bc153290 2023-03-03T08:59:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1b959a700 1 -- 172.21.15.102:0/124788743 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ac011990 con 0x7fd1bc153290 2023-03-03T08:59:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1b959a700 1 -- 172.21.15.102:0/124788743 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac011d20 con 0x7fd1bc153290 2023-03-03T08:59:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 -- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc153290 msgr2=0x7fd1bc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc153290 0x7fd1bc158e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009840 tx=0x7fd1ac009b50 comp rx=0 tx=0).stop 2023-03-03T08:59:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 -- 172.21.15.102:0/124788743 shutdown_connections 2023-03-03T08:59:28.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/124788743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc153290 0x7fd1bc158e80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 -- 172.21.15.102:0/124788743 >> 172.21.15.102:0/124788743 conn(0x7fd1bc065fb0 msgr2=0x7fd1bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:28.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 -- 172.21.15.102:0/124788743 shutdown_connections 2023-03-03T08:59:28.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.706+0000 7fd1c0fff700 1 -- 172.21.15.102:0/124788743 wait complete. 2023-03-03T08:59:28.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.708+0000 7fd1c0fff700 1 Processor -- start 2023-03-03T08:59:28.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.708+0000 7fd1c0fff700 1 -- start start 2023-03-03T08:59:28.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.709+0000 7fd1c0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.709+0000 7fd1c0fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1bc153290 0x7fd1bc0c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.709+0000 7fd1c0fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc1593c0 0x7fd1bc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.709+0000 7fd1c0fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc14e040 con 0x7fd1bc153290 2023-03-03T08:59:28.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.709+0000 7fd1c0fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0c8cc0 con 0x7fd1bc1593c0 2023-03-03T08:59:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.709+0000 7fd1c0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1e4cc0 con 0x7fd1bc0cd8e0 2023-03-03T08:59:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1bc153290 0x7fd1bc0c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1bc153290 0x7fd1bc0c8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43210/0 (socket says 172.21.15.102:43210) 2023-03-03T08:59:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1ba59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 -- 172.21.15.102:0/1728241110 learned_addr learned my addr 172.21.15.102:0/1728241110 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T08:59:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1bad9d700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc1593c0 0x7fd1bc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 -- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 msgr2=0x7fd1bc0c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 -- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc1593c0 msgr2=0x7fd1bc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc1593c0 0x7fd1bc0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1b9d9b700 1 -- 172.21.15.102:0/1728241110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ac0094a0 con 0x7fd1bc153290 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1ba59c700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.710+0000 7fd1bad9d700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc1593c0 0x7fd1bc0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.711+0000 7fd1b9d9b700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1bc153290 0x7fd1bc0c8780 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00011e0 tx=0x7fd1b000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.711+0000 7fd1aaffd700 1 -- 172.21.15.102:0/1728241110 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0004c80 con 0x7fd1bc153290 2023-03-03T08:59:28.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.711+0000 7fd1aaffd700 1 -- 172.21.15.102:0/1728241110 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b0004e00 con 0x7fd1bc153290 2023-03-03T08:59:28.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.711+0000 7fd1aaffd700 1 -- 172.21.15.102:0/1728241110 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0007310 con 0x7fd1bc153290 2023-03-03T08:59:28.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.711+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc0ccde0 con 0x7fd1bc153290 2023-03-03T08:59:28.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.711+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc15d160 con 0x7fd1bc153290 2023-03-03T08:59:28.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.712+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc0cd0c0 con 0x7fd1bc153290 2023-03-03T08:59:28.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.714+0000 7fd1aaffd700 1 -- 172.21.15.102:0/1728241110 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd1b001a020 con 0x7fd1bc153290 2023-03-03T08:59:28.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.715+0000 7fd1aaffd700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd19c0682f0 0x7fd19c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.715+0000 7fd1aaffd700 1 -- 172.21.15.102:0/1728241110 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd1b00858c0 con 0x7fd1bc153290 2023-03-03T08:59:28.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.715+0000 7fd1ba59c700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd19c0682f0 0x7fd19c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.716+0000 7fd1ba59c700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd19c0682f0 0x7fd19c06a7a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0090b0 tx=0x7fd1ac01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:28.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.719+0000 7fd1aaffd700 1 -- 172.21.15.102:0/1728241110 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd1b0051fd0 con 0x7fd1bc153290 2023-03-03T08:59:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.951+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1bc04e6e0 con 0x7fd1bc153290 2023-03-03T08:59:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.951+0000 7fd1aaffd700 1 -- 172.21.15.102:0/1728241110 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd1b00555f0 con 0x7fd1bc153290 2023-03-03T08:59:28.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T08:59:28.953 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:59:28.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd19c0682f0 msgr2=0x7fd19c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd19c0682f0 0x7fd19c06a7a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0090b0 tx=0x7fd1ac01c040 comp rx=0 tx=0).stop 2023-03-03T08:59:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1bc153290 msgr2=0x7fd1bc0c8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1bc153290 0x7fd1bc0c8780 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00011e0 tx=0x7fd1b000ca30 comp rx=0 tx=0).stop 2023-03-03T08:59:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 shutdown_connections 2023-03-03T08:59:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd19c0682f0 0x7fd19c06a7a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd1bc0cd8e0 0x7fd1bc0c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1bc1593c0 0x7fd1bc0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 --2- 172.21.15.102:0/1728241110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1bc153290 0x7fd1bc0c8780 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 >> 172.21.15.102:0/1728241110 conn(0x7fd1bc065fb0 msgr2=0x7fd1bc0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:28.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.954+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 shutdown_connections 2023-03-03T08:59:28.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T08:59:28.955+0000 7fd1c0fff700 1 -- 172.21.15.102:0/1728241110 wait complete. 2023-03-03T08:59:28.957 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T08:59:28.983 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 37 secs before thrashing 2023-03-03T08:59:40.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:40.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T08:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:00:05.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:00:06.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.259+0000 7fc206fff700 1 Processor -- start 2023-03-03T09:00:06.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.260+0000 7fc206fff700 1 -- start start 2023-03-03T09:00:06.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.261+0000 7fc206fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc2000c5f30 0x7fc20006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.261+0000 7fc206fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000c5390 con 0x7fc200150c80 2023-03-03T09:00:06.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.261+0000 7fc206fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000c4a20 con 0x7fc2000c2020 2023-03-03T09:00:06.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.261+0000 7fc206fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc200157ad0 con 0x7fc2000c5f30 2023-03-03T09:00:06.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc204d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc200150c80 0x7fc2000c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44026/0 (socket says 172.21.15.102:44026) 2023-03-03T09:00:06.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc204d9b700 1 -- 172.21.15.102:0/1491958211 learned_addr learned my addr 172.21.15.102:0/1491958211 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:00:06.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc1fffff700 1 --2- 172.21.15.102:0/1491958211 >> v2:172.21.15.172:3300/0 conn(0x7fc2000c5f30 0x7fc20006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc1ff7fe700 1 -- 172.21.15.102:0/1491958211 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616621997 0 0) 0x7fc2000c5390 con 0x7fc200150c80 2023-03-03T09:00:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc1ff7fe700 1 -- 172.21.15.102:0/1491958211 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f00035b0 con 0x7fc200150c80 2023-03-03T09:00:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc1fffff700 1 -- 172.21.15.102:0/1491958211 >> v1:172.21.15.172:6789/0 conn(0x7fc2000c2020 legacy=0x7fc2000c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc1ff7fe700 1 -- 172.21.15.102:0/1491958211 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377682695 0 0) 0x7fc2000c4a20 con 0x7fc2000c2020 2023-03-03T09:00:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc1fffff700 1 -- 172.21.15.102:0/1491958211 >> v1:172.21.15.102:6789/0 conn(0x7fc200150c80 legacy=0x7fc2000c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.262+0000 7fc1fffff700 1 -- 172.21.15.102:0/1491958211 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20006f010 con 0x7fc2000c5f30 2023-03-03T09:00:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc1fffff700 1 --2- 172.21.15.102:0/1491958211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2000c5f30 0x7fc20006d100 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009840 tx=0x7fc1ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=269b33da9466778f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc1ff7fe700 1 -- 172.21.15.102:0/1491958211 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec011830 con 0x7fc2000c5f30 2023-03-03T09:00:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc1ff7fe700 1 -- 172.21.15.102:0/1491958211 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec011990 con 0x7fc2000c5f30 2023-03-03T09:00:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc1ff7fe700 1 -- 172.21.15.102:0/1491958211 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec011d20 con 0x7fc2000c5f30 2023-03-03T09:00:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc206fff700 1 -- 172.21.15.102:0/1491958211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2000c5f30 msgr2=0x7fc20006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc206fff700 1 --2- 172.21.15.102:0/1491958211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2000c5f30 0x7fc20006d100 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009840 tx=0x7fc1ec009b50 comp rx=0 tx=0).stop 2023-03-03T09:00:06.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc206fff700 1 -- 172.21.15.102:0/1491958211 shutdown_connections 2023-03-03T09:00:06.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc206fff700 1 --2- 172.21.15.102:0/1491958211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2000c5f30 0x7fc20006d100 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.263+0000 7fc206fff700 1 -- 172.21.15.102:0/1491958211 >> 172.21.15.102:0/1491958211 conn(0x7fc20006af90 msgr2=0x7fc20006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:00:06.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.264+0000 7fc206fff700 1 -- 172.21.15.102:0/1491958211 shutdown_connections 2023-03-03T09:00:06.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.264+0000 7fc206fff700 1 -- 172.21.15.102:0/1491958211 wait complete. 2023-03-03T09:00:06.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.266+0000 7fc206fff700 1 Processor -- start 2023-03-03T09:00:06.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.266+0000 7fc206fff700 1 -- start start 2023-03-03T09:00:06.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc206fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2000c2020 0x7fc200149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc206fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2000c5f30 0x7fc200149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc206fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc200150c80 0x7fc200071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc206fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc200146a70 con 0x7fc2000c5f30 2023-03-03T09:00:06.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc206fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2001e1e40 con 0x7fc200150c80 2023-03-03T09:00:06.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc206fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc200146cc0 con 0x7fc2000c2020 2023-03-03T09:00:06.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc1fffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2000c5f30 0x7fc200149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc204d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2000c2020 0x7fc200149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc20559c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc200150c80 0x7fc200071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc1fffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2000c5f30 0x7fc200149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39072/0 (socket says 172.21.15.102:39072) 2023-03-03T09:00:06.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc204d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2000c2020 0x7fc200149100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38708/0 (socket says 172.21.15.102:38708) 2023-03-03T09:00:06.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.267+0000 7fc1fffff700 1 -- 172.21.15.102:0/174375437 learned_addr learned my addr 172.21.15.102:0/174375437 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:00:06.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc204d9b700 1 -- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc200150c80 msgr2=0x7fc200071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc204d9b700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc200150c80 0x7fc200071020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc204d9b700 1 -- 172.21.15.102:0/174375437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2000c5f30 msgr2=0x7fc200149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc204d9b700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2000c5f30 0x7fc200149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc204d9b700 1 -- 172.21.15.102:0/174375437 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec0094a0 con 0x7fc2000c2020 2023-03-03T09:00:06.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc20559c700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc200150c80 0x7fc200071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:00:06.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc204d9b700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2000c2020 0x7fc200149100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c390 tx=0x7fc1f400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:06.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc1fd7fa700 1 -- 172.21.15.102:0/174375437 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4005b40 con 0x7fc2000c2020 2023-03-03T09:00:06.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc1fd7fa700 1 -- 172.21.15.102:0/174375437 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4005ca0 con 0x7fc2000c2020 2023-03-03T09:00:06.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2001577f0 con 0x7fc2000c2020 2023-03-03T09:00:06.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.268+0000 7fc1fd7fa700 1 -- 172.21.15.102:0/174375437 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4007310 con 0x7fc2000c2020 2023-03-03T09:00:06.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.269+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc200157510 con 0x7fc2000c2020 2023-03-03T09:00:06.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.270+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2000c4cc0 con 0x7fc2000c2020 2023-03-03T09:00:06.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.271+0000 7fc1fd7fa700 1 -- 172.21.15.102:0/174375437 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1f4019030 con 0x7fc2000c2020 2023-03-03T09:00:06.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.272+0000 7fc1fd7fa700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc1e40682f0 0x7fc1e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.272+0000 7fc1fd7fa700 1 -- 172.21.15.102:0/174375437 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc1f4084f90 con 0x7fc2000c2020 2023-03-03T09:00:06.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.272+0000 7fc1fffff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc1e40682f0 0x7fc1e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.273+0000 7fc1fffff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc1e40682f0 0x7fc1e406a7a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc20014e4b0 tx=0x7fc1ec01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:06.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.277+0000 7fc1fd7fa700 1 -- 172.21.15.102:0/174375437 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1f4051610 con 0x7fc2000c2020 2023-03-03T09:00:06.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.512+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc200151050 con 0x7fc2000c2020 2023-03-03T09:00:06.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.513+0000 7fc1fd7fa700 1 -- 172.21.15.102:0/174375437 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc1f4017030 con 0x7fc2000c2020 2023-03-03T09:00:06.514 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:00:06.515 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:00:06.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc1e40682f0 msgr2=0x7fc1e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc1e40682f0 0x7fc1e406a7a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc20014e4b0 tx=0x7fc1ec01c040 comp rx=0 tx=0).stop 2023-03-03T09:00:06.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2000c2020 msgr2=0x7fc200149100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2000c2020 0x7fc200149100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c390 tx=0x7fc1f400c6a0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 shutdown_connections 2023-03-03T09:00:06.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc1e40682f0 0x7fc1e406a7a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2000c2020 0x7fc200149100 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc200150c80 0x7fc200071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 --2- 172.21.15.102:0/174375437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2000c5f30 0x7fc200149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 >> 172.21.15.102:0/174375437 conn(0x7fc20006af90 msgr2=0x7fc2000c6320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:00:06.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 shutdown_connections 2023-03-03T09:00:06.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:00:06.516+0000 7fc206fff700 1 -- 172.21.15.102:0/174375437 wait complete. 2023-03-03T09:00:06.519 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:00:06.544 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T09:00:10.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:10.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:40.879 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:40.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:11.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:11.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:41.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:41.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:57.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:01:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.810+0000 7f69abbff700 1 Processor -- start 2023-03-03T09:01:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.810+0000 7f69abbff700 1 -- start start 2023-03-03T09:01:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.811+0000 7f69abbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.811+0000 7f69abbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.811+0000 7f69abbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a4153660 con 0x7f69a4153290 2023-03-03T09:01:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.811+0000 7f69abbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a406c360 con 0x7f69a40cd8e0 2023-03-03T09:01:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.811+0000 7f69abbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a415cea0 con 0x7f69a40cd390 2023-03-03T09:01:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69a919a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69aa19c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69a919a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38648/0 (socket says 172.21.15.102:38648) 2023-03-03T09:01:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69a919a700 1 -- 172.21.15.102:0/760671877 learned_addr learned my addr 172.21.15.102:0/760671877 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:01:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69a999b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f69a4153290 0x7f69a4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33600/0 (socket says 172.21.15.102:33600) 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69aa19c700 1 -- 172.21.15.102:0/760671877 >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 msgr2=0x7f69a40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69aa19c700 1 --2- 172.21.15.102:0/760671877 >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69aa19c700 1 -- 172.21.15.102:0/760671877 >> v1:172.21.15.172:6789/0 conn(0x7f69a4153290 legacy=0x7f69a4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69aa19c700 1 -- 172.21.15.102:0/760671877 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a406c7f0 con 0x7f69a40cd390 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69a919a700 1 --2- 172.21.15.102:0/760671877 >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69a8999700 1 -- 172.21.15.102:0/760671877 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367580016 0 0) 0x7f69a4153660 con 0x7f69a4153290 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.812+0000 7f69aa19c700 1 --2- 172.21.15.102:0/760671877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40ce360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f699c009d90 tx=0x7f699c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa9418fcb79c834d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:01:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69a8999700 1 -- 172.21.15.102:0/760671877 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c00e070 con 0x7f69a40cd390 2023-03-03T09:01:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69a8999700 1 -- 172.21.15.102:0/760671877 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c00bb20 con 0x7f69a40cd390 2023-03-03T09:01:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69a8999700 1 -- 172.21.15.102:0/760671877 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c012430 con 0x7f69a40cd390 2023-03-03T09:01:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69abbff700 1 -- 172.21.15.102:0/760671877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 msgr2=0x7f69a40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69abbff700 1 --2- 172.21.15.102:0/760671877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40ce360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f699c009d90 tx=0x7f699c00b330 comp rx=0 tx=0).stop 2023-03-03T09:01:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69abbff700 1 -- 172.21.15.102:0/760671877 shutdown_connections 2023-03-03T09:01:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69abbff700 1 --2- 172.21.15.102:0/760671877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40ce360 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69abbff700 1 --2- 172.21.15.102:0/760671877 >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.813+0000 7f69abbff700 1 -- 172.21.15.102:0/760671877 >> 172.21.15.102:0/760671877 conn(0x7f69a4065fb0 msgr2=0x7f69a40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:01:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.814+0000 7f69abbff700 1 -- 172.21.15.102:0/760671877 shutdown_connections 2023-03-03T09:01:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.814+0000 7f69abbff700 1 -- 172.21.15.102:0/760671877 wait complete. 2023-03-03T09:01:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.816+0000 7f69abbff700 1 Processor -- start 2023-03-03T09:01:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.816+0000 7f69abbff700 1 -- start start 2023-03-03T09:01:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69abbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd390 0x7f69a41564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69abbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a40cd8e0 0x7f69a40c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69abbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a4153290 0x7f69a40c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69abbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a4156a00 con 0x7f69a40cd390 2023-03-03T09:01:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69abbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a40c7e10 con 0x7f69a40cd8e0 2023-03-03T09:01:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69abbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a40c86f0 con 0x7f69a4153290 2023-03-03T09:01:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69a999b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd390 0x7f69a41564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69a999b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd390 0x7f69a41564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38662/0 (socket says 172.21.15.102:38662) 2023-03-03T09:01:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69a999b700 1 -- 172.21.15.102:0/368623627 learned_addr learned my addr 172.21.15.102:0/368623627 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:01:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69aa19c700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a4153290 0x7f69a40c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.817+0000 7f69a919a700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a40cd8e0 0x7f69a40c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f69a999b700 1 -- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a4153290 msgr2=0x7f69a40c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f69a999b700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a4153290 0x7f69a40c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f69a999b700 1 -- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a40cd8e0 msgr2=0x7f69a40c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f69a999b700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a40cd8e0 0x7f69a40c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f69a999b700 1 -- 172.21.15.102:0/368623627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994009550 con 0x7f69a40cd390 2023-03-03T09:01:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f69a999b700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd390 0x7f69a41564c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f69a00011e0 tx=0x7f69a000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:01:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f699a7fc700 1 -- 172.21.15.102:0/368623627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0004c80 con 0x7f69a40cd390 2023-03-03T09:01:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f699a7fc700 1 -- 172.21.15.102:0/368623627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a0004e00 con 0x7f69a40cd390 2023-03-03T09:01:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f699a7fc700 1 -- 172.21.15.102:0/368623627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0007310 con 0x7f69a40cd390 2023-03-03T09:01:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.818+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c009a70 con 0x7f69a40cd390 2023-03-03T09:01:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.819+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a40ccde0 con 0x7f69a40cd390 2023-03-03T09:01:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.820+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a415d160 con 0x7f69a40cd390 2023-03-03T09:01:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.821+0000 7f699a7fc700 1 -- 172.21.15.102:0/368623627 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f69a001a020 con 0x7f69a40cd390 2023-03-03T09:01:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.822+0000 7f699a7fc700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6988068320 0x7f698806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.822+0000 7f699a7fc700 1 -- 172.21.15.102:0/368623627 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f69a0085930 con 0x7f69a40cd390 2023-03-03T09:01:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.822+0000 7f69a919a700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6988068320 0x7f698806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.823+0000 7f69a919a700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6988068320 0x7f698806a7d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f69a40c8f60 tx=0x7f69940092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:01:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:57.826+0000 7f699a7fc700 1 -- 172.21.15.102:0/368623627 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f69a00520f0 con 0x7f69a40cd390 2023-03-03T09:01:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.060+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a40c6d80 con 0x7f69a40cd390 2023-03-03T09:01:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.060+0000 7f699a7fc700 1 -- 172.21.15.102:0/368623627 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f69a0055710 con 0x7f69a40cd390 2023-03-03T09:01:58.062 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:01:58.062 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:01:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6988068320 msgr2=0x7f698806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6988068320 0x7f698806a7d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f69a40c8f60 tx=0x7f69940092c0 comp rx=0 tx=0).stop 2023-03-03T09:01:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd390 msgr2=0x7f69a41564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd390 0x7f69a41564c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f69a00011e0 tx=0x7f69a000ca30 comp rx=0 tx=0).stop 2023-03-03T09:01:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 shutdown_connections 2023-03-03T09:01:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6988068320 0x7f698806a7d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a4153290 0x7f69a40c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd390 0x7f69a41564c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 --2- 172.21.15.102:0/368623627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a40cd8e0 0x7f69a40c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.063+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 >> 172.21.15.102:0/368623627 conn(0x7f69a4065fb0 msgr2=0x7f69a4150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:01:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.064+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 shutdown_connections 2023-03-03T09:01:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:01:58.064+0000 7f69abbff700 1 -- 172.21.15.102:0/368623627 wait complete. 2023-03-03T09:01:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:01:58.091 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 24 secs before thrashing 2023-03-03T09:02:11.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:11.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:22.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:02:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.344+0000 7f53c67ff700 1 Processor -- start 2023-03-03T09:02:22.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.344+0000 7f53c67ff700 1 -- start start 2023-03-03T09:02:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.345+0000 7f53c67ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd8e0 0x7f53c00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.345+0000 7f53c67ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c0153660 con 0x7f53c0153290 2023-03-03T09:02:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.345+0000 7f53c67ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c006c360 con 0x7f53c00cd390 2023-03-03T09:02:22.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.345+0000 7f53c67ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c015cea0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.345+0000 7f53bffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f53c0153290 0x7f53c0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57296/0 (socket says 172.21.15.102:57296) 2023-03-03T09:02:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53bffff700 1 -- 172.21.15.102:0/177624040 learned_addr learned my addr 172.21.15.102:0/177624040 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:02:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53bf7fe700 1 --2- 172.21.15.102:0/177624040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd8e0 0x7f53c00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53beffd700 1 -- 172.21.15.102:0/177624040 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500565270 0 0) 0x7f53c0153660 con 0x7f53c0153290 2023-03-03T09:02:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53beffd700 1 -- 172.21.15.102:0/177624040 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a80035b0 con 0x7f53c0153290 2023-03-03T09:02:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53beffd700 1 -- 172.21.15.102:0/177624040 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100360931 0 0) 0x7f53c006c360 con 0x7f53c00cd390 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53beffd700 1 -- 172.21.15.102:0/177624040 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c0153660 con 0x7f53c00cd390 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53bf7fe700 1 -- 172.21.15.102:0/177624040 >> v1:172.21.15.172:6789/0 conn(0x7f53c00cd390 legacy=0x7f53c00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53bf7fe700 1 -- 172.21.15.102:0/177624040 >> v1:172.21.15.102:6789/0 conn(0x7f53c0153290 legacy=0x7f53c0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.346+0000 7f53bf7fe700 1 -- 172.21.15.102:0/177624040 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c006c7f0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53bf7fe700 1 --2- 172.21.15.102:0/177624040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd8e0 0x7f53c00cde20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009870 tx=0x7f53b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b891ab4564c2dd1a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53beffd700 1 -- 172.21.15.102:0/177624040 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4011830 con 0x7f53c00cd8e0 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53beffd700 1 -- 172.21.15.102:0/177624040 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b4011990 con 0x7f53c00cd8e0 2023-03-03T09:02:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53beffd700 1 -- 172.21.15.102:0/177624040 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4011d20 con 0x7f53c00cd8e0 2023-03-03T09:02:22.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53c67ff700 1 -- 172.21.15.102:0/177624040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd8e0 msgr2=0x7f53c00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:22.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53c67ff700 1 --2- 172.21.15.102:0/177624040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd8e0 0x7f53c00cde20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009870 tx=0x7f53b4009b80 comp rx=0 tx=0).stop 2023-03-03T09:02:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53c67ff700 1 -- 172.21.15.102:0/177624040 shutdown_connections 2023-03-03T09:02:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53c67ff700 1 --2- 172.21.15.102:0/177624040 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd8e0 0x7f53c00cde20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.347+0000 7f53c67ff700 1 -- 172.21.15.102:0/177624040 >> 172.21.15.102:0/177624040 conn(0x7f53c0065fb0 msgr2=0x7f53c00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.348+0000 7f53c67ff700 1 -- 172.21.15.102:0/177624040 shutdown_connections 2023-03-03T09:02:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.348+0000 7f53c67ff700 1 -- 172.21.15.102:0/177624040 wait complete. 2023-03-03T09:02:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.350+0000 7f53c67ff700 1 Processor -- start 2023-03-03T09:02:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.350+0000 7f53c67ff700 1 -- start start 2023-03-03T09:02:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53c67ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd390 0x7f53c01ef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53c67ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53c00cd8e0 0x7f53c014ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53c67ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53c0153290 0x7f53c00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53c67ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c01eeb70 con 0x7f53c00cd8e0 2023-03-03T09:02:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53c67ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0150500 con 0x7f53c0153290 2023-03-03T09:02:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53c67ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c01eede0 con 0x7f53c00cd390 2023-03-03T09:02:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53bf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53c00cd8e0 0x7f53c014ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53bf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53c00cd8e0 0x7f53c014ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54468/0 (socket says 172.21.15.102:54468) 2023-03-03T09:02:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53bf7fe700 1 -- 172.21.15.102:0/49937946 learned_addr learned my addr 172.21.15.102:0/49937946 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:02:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53c4d9c700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53c0153290 0x7f53c00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.351+0000 7f53bffff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd390 0x7f53c01ef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bf7fe700 1 -- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd390 msgr2=0x7f53c01ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bf7fe700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd390 0x7f53c01ef5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bf7fe700 1 -- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53c0153290 msgr2=0x7f53c00cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bf7fe700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53c0153290 0x7f53c00cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bf7fe700 1 -- 172.21.15.102:0/49937946 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40094a0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53c4d9c700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53c0153290 0x7f53c00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:02:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bffff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd390 0x7f53c01ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:02:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bf7fe700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53c00cd8e0 0x7f53c014ee30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53b40090b0 tx=0x7f53b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bcff9700 1 -- 172.21.15.102:0/49937946 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b401a860 con 0x7f53c00cd8e0 2023-03-03T09:02:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bcff9700 1 -- 172.21.15.102:0/49937946 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b401a9c0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c00ccde0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.352+0000 7f53bcff9700 1 -- 172.21.15.102:0/49937946 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b401ad30 con 0x7f53c00cd8e0 2023-03-03T09:02:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.353+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c015d160 con 0x7f53c00cd8e0 2023-03-03T09:02:22.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.354+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c00cd0c0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.355+0000 7f53bcff9700 1 -- 172.21.15.102:0/49937946 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f53b402b030 con 0x7f53c00cd8e0 2023-03-03T09:02:22.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.356+0000 7f53bcff9700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53ac0682f0 0x7f53ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:22.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.357+0000 7f53bcff9700 1 -- 172.21.15.102:0/49937946 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f53b4087130 con 0x7f53c00cd8e0 2023-03-03T09:02:22.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.357+0000 7f53bffff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53ac0682f0 0x7f53ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:22.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.358+0000 7f53bffff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53ac0682f0 0x7f53ac06a7a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f53b0000f60 tx=0x7f53b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:22.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.361+0000 7f53bcff9700 1 -- 172.21.15.102:0/49937946 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53b40537b0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.596+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53c004e6e0 con 0x7f53c00cd8e0 2023-03-03T09:02:22.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.597+0000 7f53bcff9700 1 -- 172.21.15.102:0/49937946 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f53b4029020 con 0x7f53c00cd8e0 2023-03-03T09:02:22.598 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:02:22.598 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:02:22.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53ac0682f0 msgr2=0x7f53ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:22.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53ac0682f0 0x7f53ac06a7a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f53b0000f60 tx=0x7f53b0009040 comp rx=0 tx=0).stop 2023-03-03T09:02:22.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53c00cd8e0 msgr2=0x7f53c014ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:22.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53c00cd8e0 0x7f53c014ee30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53b40090b0 tx=0x7f53b4004750 comp rx=0 tx=0).stop 2023-03-03T09:02:22.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 shutdown_connections 2023-03-03T09:02:22.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53ac0682f0 0x7f53ac06a7a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:22.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f53c00cd390 0x7f53c01ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:22.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53c0153290 0x7f53c00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:22.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 --2- 172.21.15.102:0/49937946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53c00cd8e0 0x7f53c014ee30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:22.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 >> 172.21.15.102:0/49937946 conn(0x7f53c0065fb0 msgr2=0x7f53c0151430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:22.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 shutdown_connections 2023-03-03T09:02:22.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:22.599+0000 7f53c67ff700 1 -- 172.21.15.102:0/49937946 wait complete. 2023-03-03T09:02:22.602 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:02:22.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-03-03T09:02:41.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:41.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:45.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:02:45.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.885+0000 7f7bb75ff700 1 Processor -- start 2023-03-03T09:02:45.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.885+0000 7f7bb75ff700 1 -- start start 2023-03-03T09:02:45.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.886+0000 7f7bb75ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7bb0153290 0x7f7bb0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:45.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.887+0000 7f7bb75ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb0153660 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.887+0000 7f7bb75ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb006c360 con 0x7f7bb01593c0 2023-03-03T09:02:45.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.887+0000 7f7bb75ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb00cec40 con 0x7f7bb0153290 2023-03-03T09:02:45.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.887+0000 7f7bb539b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7bb0153290 0x7f7bb0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:45.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.887+0000 7f7bb5b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7bb00cd8e0 0x7f7bb00ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43458/0 (socket says 172.21.15.102:43458) 2023-03-03T09:02:45.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.887+0000 7f7bb5b9c700 1 -- 172.21.15.102:0/661665234 learned_addr learned my addr 172.21.15.102:0/661665234 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:02:45.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.887+0000 7f7bb4b9a700 1 --1- 172.21.15.102:0/661665234 >> v1:172.21.15.172:6789/0 conn(0x7f7bb01593c0 0x7f7bb00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42994/0 (socket says 172.21.15.102:42994) 2023-03-03T09:02:45.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7bb539b700 1 -- 172.21.15.102:0/661665234 >> v1:172.21.15.172:6789/0 conn(0x7f7bb01593c0 legacy=0x7f7bb00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:45.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7bb539b700 1 -- 172.21.15.102:0/661665234 >> v1:172.21.15.102:6789/0 conn(0x7f7bb00cd8e0 legacy=0x7f7bb00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:45.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7ba7fff700 1 -- 172.21.15.102:0/661665234 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281575247 0 0) 0x7f7bb0153660 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7bb539b700 1 -- 172.21.15.102:0/661665234 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb015cea0 con 0x7f7bb0153290 2023-03-03T09:02:45.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7bb539b700 1 --2- 172.21.15.102:0/661665234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb0153290 0x7f7bb0158e80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009840 tx=0x7f7bac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=393519bf35872dd6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7ba7fff700 1 -- 172.21.15.102:0/661665234 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac011830 con 0x7f7bb0153290 2023-03-03T09:02:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7ba7fff700 1 -- 172.21.15.102:0/661665234 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac011990 con 0x7f7bb0153290 2023-03-03T09:02:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.888+0000 7f7ba7fff700 1 -- 172.21.15.102:0/661665234 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac011d20 con 0x7f7bb0153290 2023-03-03T09:02:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.889+0000 7f7bb75ff700 1 -- 172.21.15.102:0/661665234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb0153290 msgr2=0x7f7bb0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.889+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/661665234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb0153290 0x7f7bb0158e80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009840 tx=0x7f7bac009b50 comp rx=0 tx=0).stop 2023-03-03T09:02:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.889+0000 7f7bb75ff700 1 -- 172.21.15.102:0/661665234 shutdown_connections 2023-03-03T09:02:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.889+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/661665234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb0153290 0x7f7bb0158e80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.889+0000 7f7bb75ff700 1 -- 172.21.15.102:0/661665234 >> 172.21.15.102:0/661665234 conn(0x7f7bb0065fb0 msgr2=0x7f7bb00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.889+0000 7f7bb75ff700 1 -- 172.21.15.102:0/661665234 shutdown_connections 2023-03-03T09:02:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.889+0000 7f7bb75ff700 1 -- 172.21.15.102:0/661665234 wait complete. 2023-03-03T09:02:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.891+0000 7f7bb75ff700 1 Processor -- start 2023-03-03T09:02:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.891+0000 7f7bb75ff700 1 -- start start 2023-03-03T09:02:45.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb75ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb00cd8e0 0x7f7bb01effd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:45.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb75ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bb0153290 0x7f7bb01e5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:45.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb75ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bb01593c0 0x7f7bb01e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:45.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb75ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb014f800 con 0x7f7bb0153290 2023-03-03T09:02:45.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb75ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb015bb40 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb75ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0073d10 con 0x7f7bb01593c0 2023-03-03T09:02:45.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb4b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bb0153290 0x7f7bb01e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:45.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb539b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb00cd8e0 0x7f7bb01effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:45.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb539b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb00cd8e0 0x7f7bb01effd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43228/0 (socket says 172.21.15.102:43228) 2023-03-03T09:02:45.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb539b700 1 -- 172.21.15.102:0/1740774760 learned_addr learned my addr 172.21.15.102:0/1740774760 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:02:45.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.892+0000 7f7bb5b9c700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bb01593c0 0x7f7bb01e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.893+0000 7f7bb539b700 1 -- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bb01593c0 msgr2=0x7f7bb01e5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.893+0000 7f7bb539b700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bb01593c0 0x7f7bb01e5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.893+0000 7f7bb539b700 1 -- 172.21.15.102:0/1740774760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bb0153290 msgr2=0x7f7bb01e5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.893+0000 7f7bb539b700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bb0153290 0x7f7bb01e5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.893+0000 7f7bb539b700 1 -- 172.21.15.102:0/1740774760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac0094a0 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.893+0000 7f7bb539b700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb00cd8e0 0x7f7bb01effd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7bac014040 tx=0x7f7bac004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.893+0000 7f7ba5ffb700 1 -- 172.21.15.102:0/1740774760 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac01a7e0 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.894+0000 7f7ba5ffb700 1 -- 172.21.15.102:0/1740774760 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac01a940 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.894+0000 7f7ba5ffb700 1 -- 172.21.15.102:0/1740774760 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac01acb0 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.894+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb015d160 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.894+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb006c7f0 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.895+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb00ccde0 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.896+0000 7f7ba5ffb700 1 -- 172.21.15.102:0/1740774760 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7bac00f050 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.897+0000 7f7ba5ffb700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b940682f0 0x7f7b9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:45.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.897+0000 7f7ba5ffb700 1 -- 172.21.15.102:0/1740774760 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7bac086c20 con 0x7f7bb00cd8e0 2023-03-03T09:02:45.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.898+0000 7f7bb4b9a700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b940682f0 0x7f7b9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:45.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.898+0000 7f7bb4b9a700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b940682f0 0x7f7b9406a7a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00c9970 tx=0x7f7ba0007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:45.902+0000 7f7ba5ffb700 1 -- 172.21.15.102:0/1740774760 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7bac0533e0 con 0x7f7bb00cd8e0 2023-03-03T09:02:46.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.136+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb006b180 con 0x7f7bb00cd8e0 2023-03-03T09:02:46.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.137+0000 7f7ba5ffb700 1 -- 172.21.15.102:0/1740774760 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7bac015090 con 0x7f7bb00cd8e0 2023-03-03T09:02:46.139 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:02:46.139 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:02:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.140+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b940682f0 msgr2=0x7f7b9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.140+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b940682f0 0x7f7b9406a7a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00c9970 tx=0x7f7ba0007ed0 comp rx=0 tx=0).stop 2023-03-03T09:02:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.140+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb00cd8e0 msgr2=0x7f7bb01effd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.140+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb00cd8e0 0x7f7bb01effd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7bac014040 tx=0x7f7bac004750 comp rx=0 tx=0).stop 2023-03-03T09:02:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 shutdown_connections 2023-03-03T09:02:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b940682f0 0x7f7b9406a7a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7bb01593c0 0x7f7bb01e5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:46.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7bb00cd8e0 0x7f7bb01effd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:46.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 --2- 172.21.15.102:0/1740774760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bb0153290 0x7f7bb01e5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:46.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 >> 172.21.15.102:0/1740774760 conn(0x7f7bb0065fb0 msgr2=0x7f7bb00cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:46.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 shutdown_connections 2023-03-03T09:02:46.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:02:46.141+0000 7f7bb75ff700 1 -- 172.21.15.102:0/1740774760 wait complete. 2023-03-03T09:02:46.143 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:02:46.168 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T09:03:11.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:11.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:41.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:41.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:12.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:12.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:13.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:04:13.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.433+0000 7f073ffff700 1 Processor -- start 2023-03-03T09:04:13.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.433+0000 7f073ffff700 1 -- start start 2023-03-03T09:04:13.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.435+0000 7f073ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400ced00 0x7f07400c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:04:13.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.435+0000 7f073ffff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07400c3d20 con 0x7f07400c41b0 2023-03-03T09:04:13.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.435+0000 7f073ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074015cba0 con 0x7f07400b3970 2023-03-03T09:04:13.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.435+0000 7f073ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07400c4930 con 0x7f07400ced00 2023-03-03T09:04:13.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.435+0000 7f073effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400ced00 0x7f07400c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:04:13.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.435+0000 7f073e7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f07400b3970 0x7f0740158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:32842/0 (socket says 172.21.15.102:32842) 2023-03-03T09:04:13.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.435+0000 7f073e7fc700 1 -- 172.21.15.102:0/3656631734 learned_addr learned my addr 172.21.15.102:0/3656631734 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:04:13.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073effd700 1 -- 172.21.15.102:0/3656631734 >> v1:172.21.15.172:6789/0 conn(0x7f07400b3970 legacy=0x7f0740158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:04:13.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073dffb700 1 -- 172.21.15.102:0/3656631734 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488129598 0 0) 0x7f074015cba0 con 0x7f07400b3970 2023-03-03T09:04:13.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073effd700 1 -- 172.21.15.102:0/3656631734 >> v1:172.21.15.102:6789/0 conn(0x7f07400c41b0 legacy=0x7f0740163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:04:13.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073effd700 1 -- 172.21.15.102:0/3656631734 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074015ce80 con 0x7f07400ced00 2023-03-03T09:04:13.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073effd700 1 --2- 172.21.15.102:0/3656631734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400ced00 0x7f07400c5a90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0730009870 tx=0x7f0730009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b338e2c5ea88dfa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:04:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073dffb700 1 -- 172.21.15.102:0/3656631734 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730011830 con 0x7f07400ced00 2023-03-03T09:04:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073dffb700 1 -- 172.21.15.102:0/3656631734 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0730011990 con 0x7f07400ced00 2023-03-03T09:04:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.436+0000 7f073dffb700 1 -- 172.21.15.102:0/3656631734 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730011d20 con 0x7f07400ced00 2023-03-03T09:04:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.437+0000 7f073ffff700 1 -- 172.21.15.102:0/3656631734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400ced00 msgr2=0x7f07400c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:04:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.437+0000 7f073ffff700 1 --2- 172.21.15.102:0/3656631734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400ced00 0x7f07400c5a90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0730009870 tx=0x7f0730009b80 comp rx=0 tx=0).stop 2023-03-03T09:04:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.437+0000 7f073ffff700 1 -- 172.21.15.102:0/3656631734 shutdown_connections 2023-03-03T09:04:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.437+0000 7f073ffff700 1 --2- 172.21.15.102:0/3656631734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400ced00 0x7f07400c5a90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:04:13.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.437+0000 7f073ffff700 1 -- 172.21.15.102:0/3656631734 >> 172.21.15.102:0/3656631734 conn(0x7f074008b1c0 msgr2=0x7f0740071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:04:13.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.437+0000 7f073ffff700 1 -- 172.21.15.102:0/3656631734 shutdown_connections 2023-03-03T09:04:13.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.437+0000 7f073ffff700 1 -- 172.21.15.102:0/3656631734 wait complete. 2023-03-03T09:04:13.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.439+0000 7f073ffff700 1 Processor -- start 2023-03-03T09:04:13.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.439+0000 7f073ffff700 1 -- start start 2023-03-03T09:04:13.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.440+0000 7f073ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07400b3970 0x7f07401ee940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:04:13.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.440+0000 7f073ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400c41b0 0x7f07400c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:04:13.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.440+0000 7f073ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07400ced00 0x7f074014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:04:13.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.440+0000 7f073ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07401ee4b0 con 0x7f07400ced00 2023-03-03T09:04:13.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.440+0000 7f073ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07401f1b50 con 0x7f07400b3970 2023-03-03T09:04:13.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.440+0000 7f073ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07401f1e30 con 0x7f07400c41b0 2023-03-03T09:04:13.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07400b3970 0x7f07401ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:04:13.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073e7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400c41b0 0x7f07400c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:04:13.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07400b3970 0x7f07401ee940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39128/0 (socket says 172.21.15.102:39128) 2023-03-03T09:04:13.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 -- 172.21.15.102:0/3066537680 learned_addr learned my addr 172.21.15.102:0/3066537680 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:04:13.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073f7fe700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07400ced00 0x7f074014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:04:13.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 -- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400c41b0 msgr2=0x7f07400c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:04:13.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400c41b0 0x7f07400c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:04:13.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 -- 172.21.15.102:0/3066537680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07400ced00 msgr2=0x7f074014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:04:13.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07400ced00 0x7f074014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:04:13.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 -- 172.21.15.102:0/3066537680 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07300094a0 con 0x7f07400b3970 2023-03-03T09:04:13.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073e7fc700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400c41b0 0x7f07400c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:04:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.441+0000 7f073effd700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07400b3970 0x7f07401ee940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0730000c00 tx=0x7f0730004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:04:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.442+0000 7f07277fe700 1 -- 172.21.15.102:0/3066537680 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07300036a0 con 0x7f07400b3970 2023-03-03T09:04:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.442+0000 7f07277fe700 1 -- 172.21.15.102:0/3066537680 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073001f070 con 0x7f07400b3970 2023-03-03T09:04:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.442+0000 7f07277fe700 1 -- 172.21.15.102:0/3066537680 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730003a20 con 0x7f07400b3970 2023-03-03T09:04:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.442+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074006c200 con 0x7f07400b3970 2023-03-03T09:04:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.442+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074006bf20 con 0x7f07400b3970 2023-03-03T09:04:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.443+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074015c890 con 0x7f07400b3970 2023-03-03T09:04:13.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.444+0000 7f07277fe700 1 -- 172.21.15.102:0/3066537680 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f073000f050 con 0x7f07400b3970 2023-03-03T09:04:13.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.445+0000 7f07277fe700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f072c068320 0x7f072c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:04:13.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.446+0000 7f07277fe700 1 -- 172.21.15.102:0/3066537680 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0730021070 con 0x7f07400b3970 2023-03-03T09:04:13.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.446+0000 7f073e7fc700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f072c068320 0x7f072c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:04:13.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.446+0000 7f073e7fc700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f072c068320 0x7f072c06a7d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f0740066930 tx=0x7f0734007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:04:13.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.450+0000 7f07277fe700 1 -- 172.21.15.102:0/3066537680 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0730053da0 con 0x7f07400b3970 2023-03-03T09:04:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.686+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f074004e6e0 con 0x7f07400b3970 2023-03-03T09:04:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.687+0000 7f07277fe700 1 -- 172.21.15.102:0/3066537680 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f07300573c0 con 0x7f07400b3970 2023-03-03T09:04:13.688 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:04:13.688 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:04:13.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.689+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f072c068320 msgr2=0x7f072c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:04:13.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.689+0000 7f073ffff700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f072c068320 0x7f072c06a7d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f0740066930 tx=0x7f0734007e70 comp rx=0 tx=0).stop 2023-03-03T09:04:13.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.689+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07400b3970 msgr2=0x7f07401ee940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:04:13.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.689+0000 7f073ffff700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07400b3970 0x7f07401ee940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0730000c00 tx=0x7f0730004750 comp rx=0 tx=0).stop 2023-03-03T09:04:13.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 shutdown_connections 2023-03-03T09:04:13.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07400ced00 0x7f074014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:04:13.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07400b3970 0x7f07401ee940 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:04:13.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f072c068320 0x7f072c06a7d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:04:13.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 --2- 172.21.15.102:0/3066537680 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07400c41b0 0x7f07400c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:04:13.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 >> 172.21.15.102:0/3066537680 conn(0x7f074008b1c0 msgr2=0x7f0740065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:04:13.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 shutdown_connections 2023-03-03T09:04:13.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:04:13.690+0000 7f073ffff700 1 -- 172.21.15.102:0/3066537680 wait complete. 2023-03-03T09:04:13.692 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:04:13.718 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-03-03T09:04:42.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:42.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:12.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:12.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:19.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:05:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.968+0000 7f201119d700 1 Processor -- start 2023-03-03T09:05:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.968+0000 7f201119d700 1 -- start start 2023-03-03T09:05:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f201119d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f201119d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f200c0c41b0 0x7f200c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f201119d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200c0c3d20 con 0x7f200c0b3970 2023-03-03T09:05:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f201119d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200c0c4930 con 0x7f200c0c41b0 2023-03-03T09:05:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f201119d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0cf610 con 0x7f200c0ced00 2023-03-03T09:05:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f201099c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f200c0c41b0 0x7f200c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f200b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f200c0b3970 0x7f200c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54468/0 (socket says 172.21.15.102:54468) 2023-03-03T09:05:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f200b7fe700 1 -- 172.21.15.102:0/2928829360 learned_addr learned my addr 172.21.15.102:0/2928829360 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:05:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.970+0000 7f200bfff700 1 --2- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:19.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f201099c700 1 -- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 msgr2=0x7f200c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:05:19.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f201099c700 1 --2- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:19.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f201099c700 1 -- 172.21.15.102:0/2928829360 >> v1:172.21.15.102:6789/0 conn(0x7f200c0b3970 legacy=0x7f200c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:19.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f201099c700 1 -- 172.21.15.102:0/2928829360 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c15cba0 con 0x7f200c0c41b0 2023-03-03T09:05:19.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f200bfff700 1 --2- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:05:19.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f200affd700 1 -- 172.21.15.102:0/2928829360 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125529194 0 0) 0x7f200c0c3d20 con 0x7f200c0b3970 2023-03-03T09:05:19.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f201099c700 1 --2- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 0x7f200c0cf0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2004009dc0 tx=0x7f200400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7ca5aa90ce56927 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:19.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f200affd700 1 -- 172.21.15.102:0/2928829360 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400e070 con 0x7f200c0c41b0 2023-03-03T09:05:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f200affd700 1 -- 172.21.15.102:0/2928829360 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200400bae0 con 0x7f200c0c41b0 2023-03-03T09:05:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.971+0000 7f200affd700 1 -- 172.21.15.102:0/2928829360 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400be90 con 0x7f200c0c41b0 2023-03-03T09:05:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 -- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 msgr2=0x7f200c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 --2- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 0x7f200c0cf0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2004009dc0 tx=0x7f200400b330 comp rx=0 tx=0).stop 2023-03-03T09:05:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 -- 172.21.15.102:0/2928829360 shutdown_connections 2023-03-03T09:05:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 --2- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 0x7f200c0cf0d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 --2- 172.21.15.102:0/2928829360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:19.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 -- 172.21.15.102:0/2928829360 >> 172.21.15.102:0/2928829360 conn(0x7f200c08b1c0 msgr2=0x7f200c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:19.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 -- 172.21.15.102:0/2928829360 shutdown_connections 2023-03-03T09:05:19.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.972+0000 7f201119d700 1 -- 172.21.15.102:0/2928829360 wait complete. 2023-03-03T09:05:19.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.974+0000 7f201119d700 1 Processor -- start 2023-03-03T09:05:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.974+0000 7f201119d700 1 -- start start 2023-03-03T09:05:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f201119d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200c0b3970 0x7f200c148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f201119d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 0x7f200c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f201119d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f201119d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c14dab0 con 0x7f200c0b3970 2023-03-03T09:05:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f201119d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c14ea40 con 0x7f200c0c41b0 2023-03-03T09:05:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f201119d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c14d110 con 0x7f200c0ced00 2023-03-03T09:05:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f200bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200c0b3970 0x7f200c148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f200bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200c0b3970 0x7f200c148df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49100/0 (socket says 172.21.15.102:49100) 2023-03-03T09:05:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f200bfff700 1 -- 172.21.15.102:0/4091846319 learned_addr learned my addr 172.21.15.102:0/4091846319 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:05:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f200b7fe700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 0x7f200c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.975+0000 7f201099c700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f200bfff700 1 -- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 msgr2=0x7f200c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f200bfff700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c15a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f200bfff700 1 -- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 msgr2=0x7f200c159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f200bfff700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 0x7f200c159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f200bfff700 1 -- 172.21.15.102:0/4091846319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0094a0 con 0x7f200c0b3970 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f200bfff700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200c0b3970 0x7f200c148df0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00ebe0 tx=0x7f1ffc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f2008ff9700 1 -- 172.21.15.102:0/4091846319 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc00c850 con 0x7f200c0b3970 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f2008ff9700 1 -- 172.21.15.102:0/4091846319 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc00c9b0 con 0x7f200c0b3970 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f2008ff9700 1 -- 172.21.15.102:0/4091846319 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc00cd40 con 0x7f200c0b3970 2023-03-03T09:05:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.976+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2004009a70 con 0x7f200c0b3970 2023-03-03T09:05:19.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.977+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200c06bf20 con 0x7f200c0b3970 2023-03-03T09:05:19.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.978+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200c15ce80 con 0x7f200c0b3970 2023-03-03T09:05:19.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.979+0000 7f2008ff9700 1 -- 172.21.15.102:0/4091846319 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1ffc019030 con 0x7f200c0b3970 2023-03-03T09:05:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.980+0000 7f2008ff9700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ff8068320 0x7f1ff806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.980+0000 7f2008ff9700 1 -- 172.21.15.102:0/4091846319 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1ffc085cf0 con 0x7f200c0b3970 2023-03-03T09:05:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.981+0000 7f200b7fe700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ff8068320 0x7f1ff806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.981+0000 7f200b7fe700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ff8068320 0x7f1ff806a7d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f200c1e6250 tx=0x7f2000007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:19.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:19.984+0000 7f2008ff9700 1 -- 172.21.15.102:0/4091846319 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ffc052400 con 0x7f200c0b3970 2023-03-03T09:05:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.218+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f200c06c200 con 0x7f200c0b3970 2023-03-03T09:05:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.218+0000 7f2008ff9700 1 -- 172.21.15.102:0/4091846319 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1ffc00cea0 con 0x7f200c0b3970 2023-03-03T09:05:20.221 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:05:20.221 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:05:20.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ff8068320 msgr2=0x7f1ff806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:20.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ff8068320 0x7f1ff806a7d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f200c1e6250 tx=0x7f2000007e70 comp rx=0 tx=0).stop 2023-03-03T09:05:20.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200c0b3970 msgr2=0x7f200c148df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200c0b3970 0x7f200c148df0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00ebe0 tx=0x7f1ffc00eef0 comp rx=0 tx=0).stop 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 shutdown_connections 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ff8068320 0x7f1ff806a7d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f200c0ced00 0x7f200c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200c0b3970 0x7f200c148df0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 --2- 172.21.15.102:0/4091846319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f200c0c41b0 0x7f200c159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 >> 172.21.15.102:0/4091846319 conn(0x7f200c08b1c0 msgr2=0x7f200c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 shutdown_connections 2023-03-03T09:05:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:05:20.221+0000 7f201119d700 1 -- 172.21.15.102:0/4091846319 wait complete. 2023-03-03T09:05:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:05:20.251 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-03-03T09:05:42.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:42.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:12.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:12.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:42.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:42.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:02.252 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:07:02.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.518+0000 7f45b19ff700 1 Processor -- start 2023-03-03T09:07:02.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.519+0000 7f45b19ff700 1 -- start start 2023-03-03T09:07:02.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45b19ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f45ac153260 0x7f45ac0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:02.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45b19ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45ac0cac20 0x7f45ac163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:02.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45b19ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0cf940 con 0x7f45ac0c9cb0 2023-03-03T09:07:02.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45b19ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac0cfe70 con 0x7f45ac0cac20 2023-03-03T09:07:02.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45b19ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac158d70 con 0x7f45ac153260 2023-03-03T09:07:02.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45aaffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f45ac153260 0x7f45ac0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:02.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45aa7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f45ac0c9cb0 0x7f45ac0cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52368/0 (socket says 172.21.15.102:52368) 2023-03-03T09:07:02.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45ab7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45ac0cac20 0x7f45ac163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:02.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.520+0000 7f45aa7fc700 1 -- 172.21.15.102:0/636765141 learned_addr learned my addr 172.21.15.102:0/636765141 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:07:02.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45aaffd700 1 -- 172.21.15.102:0/636765141 >> v2:172.21.15.102:3300/0 conn(0x7f45ac0cac20 msgr2=0x7f45ac163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:07:02.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45aaffd700 1 --2- 172.21.15.102:0/636765141 >> v2:172.21.15.102:3300/0 conn(0x7f45ac0cac20 0x7f45ac163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45aaffd700 1 -- 172.21.15.102:0/636765141 >> v1:172.21.15.172:6789/0 conn(0x7f45ac0c9cb0 legacy=0x7f45ac0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:02.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45aaffd700 1 -- 172.21.15.102:0/636765141 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0cfb90 con 0x7f45ac153260 2023-03-03T09:07:02.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45a9ffb700 1 -- 172.21.15.102:0/636765141 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882696209 0 0) 0x7f45ac0cf940 con 0x7f45ac0c9cb0 2023-03-03T09:07:02.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45aaffd700 1 --2- 172.21.15.102:0/636765141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac153260 0x7f45ac0ca6e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f45ac08cc70 tx=0x7f459c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9602954aaf4eada server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:02.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45a9ffb700 1 -- 172.21.15.102:0/636765141 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c011830 con 0x7f45ac153260 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45a9ffb700 1 -- 172.21.15.102:0/636765141 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c011990 con 0x7f45ac153260 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.521+0000 7f45a9ffb700 1 -- 172.21.15.102:0/636765141 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c011d20 con 0x7f45ac153260 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 -- 172.21.15.102:0/636765141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac153260 msgr2=0x7f45ac0ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 --2- 172.21.15.102:0/636765141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac153260 0x7f45ac0ca6e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f45ac08cc70 tx=0x7f459c009b20 comp rx=0 tx=0).stop 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 -- 172.21.15.102:0/636765141 shutdown_connections 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 --2- 172.21.15.102:0/636765141 >> v2:172.21.15.102:3300/0 conn(0x7f45ac0cac20 0x7f45ac163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 --2- 172.21.15.102:0/636765141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac153260 0x7f45ac0ca6e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 -- 172.21.15.102:0/636765141 >> 172.21.15.102:0/636765141 conn(0x7f45ac066c00 msgr2=0x7f45ac0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:02.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 -- 172.21.15.102:0/636765141 shutdown_connections 2023-03-03T09:07:02.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.522+0000 7f45b19ff700 1 -- 172.21.15.102:0/636765141 wait complete. 2023-03-03T09:07:02.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.524+0000 7f45b19ff700 1 Processor -- start 2023-03-03T09:07:02.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.524+0000 7f45b19ff700 1 -- start start 2023-03-03T09:07:02.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.525+0000 7f45b19ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac0cac20 0x7f45ac1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:02.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.525+0000 7f45b19ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45ac153260 0x7f45ac14a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:02.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.525+0000 7f45b19ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45ac0b4240 0x7f45ac0b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:02.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.525+0000 7f45b19ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0b4c20 con 0x7f45ac153260 2023-03-03T09:07:02.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.525+0000 7f45b19ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0ccea0 con 0x7f45ac0cac20 2023-03-03T09:07:02.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.525+0000 7f45b19ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac1536e0 con 0x7f45ac0b4240 2023-03-03T09:07:02.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45ac153260 0x7f45ac14a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:02.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45ac153260 0x7f45ac14a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54748/0 (socket says 172.21.15.102:54748) 2023-03-03T09:07:02.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 -- 172.21.15.102:0/1834674710 learned_addr learned my addr 172.21.15.102:0/1834674710 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:07:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45ab7fe700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45ac0b4240 0x7f45ac0b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aaffd700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac0cac20 0x7f45ac1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 -- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45ac0b4240 msgr2=0x7f45ac0b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45ac0b4240 0x7f45ac0b46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 -- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac0cac20 msgr2=0x7f45ac1540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac0cac20 0x7f45ac1540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 -- 172.21.15.102:0/1834674710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0094a0 con 0x7f45ac153260 2023-03-03T09:07:02.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.526+0000 7f45aa7fc700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45ac153260 0x7f45ac14a2a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f45a000c550 tx=0x7f45a000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:02.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.527+0000 7f4593fff700 1 -- 172.21.15.102:0/1834674710 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0004c80 con 0x7f45ac153260 2023-03-03T09:07:02.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.527+0000 7f4593fff700 1 -- 172.21.15.102:0/1834674710 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a0004e00 con 0x7f45ac153260 2023-03-03T09:07:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.527+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac159260 con 0x7f45ac153260 2023-03-03T09:07:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.527+0000 7f4593fff700 1 -- 172.21.15.102:0/1834674710 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0007310 con 0x7f45ac153260 2023-03-03T09:07:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.527+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac158f80 con 0x7f45ac153260 2023-03-03T09:07:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.528+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac159540 con 0x7f45ac153260 2023-03-03T09:07:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.529+0000 7f4593fff700 1 -- 172.21.15.102:0/1834674710 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f45a001a020 con 0x7f45ac153260 2023-03-03T09:07:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.531+0000 7f4593fff700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45980682f0 0x7f459806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.531+0000 7f4593fff700 1 -- 172.21.15.102:0/1834674710 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f45a0085840 con 0x7f45ac153260 2023-03-03T09:07:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.532+0000 7f45aaffd700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45980682f0 0x7f459806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:02.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.532+0000 7f45aaffd700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45980682f0 0x7f459806a7a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f459c0090b0 tx=0x7f459c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:02.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.536+0000 7f4593fff700 1 -- 172.21.15.102:0/1834674710 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f45a0051f50 con 0x7f45ac153260 2023-03-03T09:07:02.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.770+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45ac04e710 con 0x7f45ac153260 2023-03-03T09:07:02.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.770+0000 7f4593fff700 1 -- 172.21.15.102:0/1834674710 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f45a0055570 con 0x7f45ac153260 2023-03-03T09:07:02.772 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:07:02.772 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:07:02.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45980682f0 msgr2=0x7f459806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:02.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45980682f0 0x7f459806a7a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f459c0090b0 tx=0x7f459c008030 comp rx=0 tx=0).stop 2023-03-03T09:07:02.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45ac153260 msgr2=0x7f45ac14a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45ac153260 0x7f45ac14a2a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f45a000c550 tx=0x7f45a000c860 comp rx=0 tx=0).stop 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 shutdown_connections 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45980682f0 0x7f459806a7a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f45ac0b4240 0x7f45ac0b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45ac0cac20 0x7f45ac1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 --2- 172.21.15.102:0/1834674710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45ac153260 0x7f45ac14a2a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 >> 172.21.15.102:0/1834674710 conn(0x7f45ac0c9cb0 msgr2=0x7f45ac0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:02.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 shutdown_connections 2023-03-03T09:07:02.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:02.773+0000 7f45b19ff700 1 -- 172.21.15.102:0/1834674710 wait complete. 2023-03-03T09:07:02.776 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:07:02.802 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 11 secs before thrashing 2023-03-03T09:07:13.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:13.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:13.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:07:14.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.067+0000 7f798a5ff700 1 Processor -- start 2023-03-03T09:07:14.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.068+0000 7f798a5ff700 1 -- start start 2023-03-03T09:07:14.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f798a5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f79840cd8e0 0x7f79840cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:14.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f798a5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79840cd390 0x7f79840ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:14.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f798a5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7984153660 con 0x7f7984153290 2023-03-03T09:07:14.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f798a5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798406c360 con 0x7f79840cd390 2023-03-03T09:07:14.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f798a5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798415cea0 con 0x7f79840cd8e0 2023-03-03T09:07:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f7983fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7984153290 0x7f7984158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57076/0 (socket says 172.21.15.102:57076) 2023-03-03T09:07:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f7983fff700 1 -- 172.21.15.102:0/1186207231 learned_addr learned my addr 172.21.15.102:0/1186207231 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:07:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f7988b9c700 1 --2- 172.21.15.102:0/1186207231 >> v2:172.21.15.102:3300/0 conn(0x7f79840cd390 0x7f79840ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.069+0000 7f79837fe700 1 --2- 172.21.15.102:0/1186207231 >> v2:172.21.15.172:3300/0 conn(0x7f79840cd8e0 0x7f79840cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.070+0000 7f7988b9c700 1 -- 172.21.15.102:0/1186207231 >> v2:172.21.15.172:3300/0 conn(0x7f79840cd8e0 msgr2=0x7f79840cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:07:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.070+0000 7f7988b9c700 1 --2- 172.21.15.102:0/1186207231 >> v2:172.21.15.172:3300/0 conn(0x7f79840cd8e0 0x7f79840cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.070+0000 7f7988b9c700 1 -- 172.21.15.102:0/1186207231 >> v1:172.21.15.102:6789/0 conn(0x7f7984153290 legacy=0x7f7984158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.070+0000 7f7988b9c700 1 -- 172.21.15.102:0/1186207231 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798406c7f0 con 0x7f79840cd390 2023-03-03T09:07:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.070+0000 7f7982ffd700 1 -- 172.21.15.102:0/1186207231 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787320302 0 0) 0x7f7984153660 con 0x7f7984153290 2023-03-03T09:07:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.070+0000 7f7988b9c700 1 --2- 172.21.15.102:0/1186207231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79840cd390 0x7f79840ce360 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7984159fb0 tx=0x7f797c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c354b963515e9877 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f7982ffd700 1 -- 172.21.15.102:0/1186207231 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c00e070 con 0x7f79840cd390 2023-03-03T09:07:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f7982ffd700 1 -- 172.21.15.102:0/1186207231 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c00bb20 con 0x7f79840cd390 2023-03-03T09:07:14.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f7982ffd700 1 -- 172.21.15.102:0/1186207231 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c012430 con 0x7f79840cd390 2023-03-03T09:07:14.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 -- 172.21.15.102:0/1186207231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79840cd390 msgr2=0x7f79840ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:14.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 --2- 172.21.15.102:0/1186207231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79840cd390 0x7f79840ce360 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7984159fb0 tx=0x7f797c00b330 comp rx=0 tx=0).stop 2023-03-03T09:07:14.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 -- 172.21.15.102:0/1186207231 shutdown_connections 2023-03-03T09:07:14.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 --2- 172.21.15.102:0/1186207231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79840cd390 0x7f79840ce360 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 --2- 172.21.15.102:0/1186207231 >> v2:172.21.15.172:3300/0 conn(0x7f79840cd8e0 0x7f79840cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 -- 172.21.15.102:0/1186207231 >> 172.21.15.102:0/1186207231 conn(0x7f7984065fb0 msgr2=0x7f79840663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:14.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 -- 172.21.15.102:0/1186207231 shutdown_connections 2023-03-03T09:07:14.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.071+0000 7f798a5ff700 1 -- 172.21.15.102:0/1186207231 wait complete. 2023-03-03T09:07:14.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.073+0000 7f798a5ff700 1 Processor -- start 2023-03-03T09:07:14.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.074+0000 7f798a5ff700 1 -- start start 2023-03-03T09:07:14.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f798a5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79840cd390 0x7f7984150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:14.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f798a5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79840cd8e0 0x7f7984157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:14.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f798a5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7984153290 0x7f79841efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:14.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f798a5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79840cfd20 con 0x7f7984153290 2023-03-03T09:07:14.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f798a5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79841585b0 con 0x7f79840cd8e0 2023-03-03T09:07:14.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f798a5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79841582d0 con 0x7f79840cd390 2023-03-03T09:07:14.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f7988b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7984153290 0x7f79841efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:14.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f79837fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79840cd8e0 0x7f7984157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f7983fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79840cd390 0x7f7984150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f79837fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79840cd8e0 0x7f7984157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56440/0 (socket says 172.21.15.102:56440) 2023-03-03T09:07:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f79837fe700 1 -- 172.21.15.102:0/984884454 learned_addr learned my addr 172.21.15.102:0/984884454 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:07:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f7988b9c700 1 -- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79840cd390 msgr2=0x7f7984150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.075+0000 7f7988b9c700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79840cd390 0x7f7984150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f7988b9c700 1 -- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79840cd8e0 msgr2=0x7f7984157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f7988b9c700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79840cd8e0 0x7f7984157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f7988b9c700 1 -- 172.21.15.102:0/984884454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c009a70 con 0x7f7984153290 2023-03-03T09:07:14.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f79837fe700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79840cd8e0 0x7f7984157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:07:14.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f7988b9c700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7984153290 0x7f79841efaf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f797c016000 tx=0x7f797c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:14.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f7980ff9700 1 -- 172.21.15.102:0/984884454 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c012430 con 0x7f7984153290 2023-03-03T09:07:14.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f7980ff9700 1 -- 172.21.15.102:0/984884454 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c020070 con 0x7f7984153290 2023-03-03T09:07:14.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f7980ff9700 1 -- 172.21.15.102:0/984884454 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c004280 con 0x7f7984153290 2023-03-03T09:07:14.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.076+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79840ccde0 con 0x7f7984153290 2023-03-03T09:07:14.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.077+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798415d160 con 0x7f7984153290 2023-03-03T09:07:14.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.078+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79840cd0c0 con 0x7f7984153290 2023-03-03T09:07:14.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.079+0000 7f7980ff9700 1 -- 172.21.15.102:0/984884454 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f797c02e070 con 0x7f7984153290 2023-03-03T09:07:14.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.080+0000 7f7980ff9700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79700682f0 0x7f797006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:14.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.080+0000 7f7980ff9700 1 -- 172.21.15.102:0/984884454 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f797c022070 con 0x7f7984153290 2023-03-03T09:07:14.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.080+0000 7f7983fff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79700682f0 0x7f797006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:14.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.081+0000 7f7983fff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79700682f0 0x7f797006a7a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7974000f60 tx=0x7f7974007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:14.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.085+0000 7f7980ff9700 1 -- 172.21.15.102:0/984884454 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f797c0540a0 con 0x7f7984153290 2023-03-03T09:07:14.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.320+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f798415a5d0 con 0x7f7984153290 2023-03-03T09:07:14.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.320+0000 7f7980ff9700 1 -- 172.21.15.102:0/984884454 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f797c0576c0 con 0x7f7984153290 2023-03-03T09:07:14.322 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:07:14.322 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:07:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79700682f0 msgr2=0x7f797006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79700682f0 0x7f797006a7a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7974000f60 tx=0x7f7974007e70 comp rx=0 tx=0).stop 2023-03-03T09:07:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7984153290 msgr2=0x7f79841efaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7984153290 0x7f79841efaf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f797c016000 tx=0x7f797c004ad0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 shutdown_connections 2023-03-03T09:07:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79700682f0 0x7f797006a7a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79840cd390 0x7f7984150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7984153290 0x7f79841efaf0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 --2- 172.21.15.102:0/984884454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79840cd8e0 0x7f7984157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 >> 172.21.15.102:0/984884454 conn(0x7f7984065fb0 msgr2=0x7f7984150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 shutdown_connections 2023-03-03T09:07:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:07:14.323+0000 7f798a5ff700 1 -- 172.21.15.102:0/984884454 wait complete. 2023-03-03T09:07:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:07:14.351 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 72 secs before thrashing 2023-03-03T09:07:43.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:43.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:13.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:13.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:26.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:08:26.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.602+0000 7fac1159d700 1 Processor -- start 2023-03-03T09:08:26.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.603+0000 7fac1159d700 1 -- start start 2023-03-03T09:08:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac1159d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fac0c14fb20 0x7fac0c0721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac1159d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fac0c071470 0x7fac0c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac1159d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c06b160 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac1159d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0c0c2810 con 0x7fac0c071470 2023-03-03T09:08:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac1159d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0c159880 con 0x7fac0c14fb20 2023-03-03T09:08:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac10d9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fac0c071470 0x7fac0c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac0bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fac0c14fb20 0x7fac0c0721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac10d9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fac0c071470 0x7fac0c0c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55294/0 (socket says 172.21.15.102:55294) 2023-03-03T09:08:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac10d9c700 1 -- 172.21.15.102:0/3293345145 learned_addr learned my addr 172.21.15.102:0/3293345145 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:08:26.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.604+0000 7fac0b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fac0c0cb2d0 0x7fac0c0c6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:39726/0 (socket says 172.21.15.102:39726) 2023-03-03T09:08:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac0affd700 1 -- 172.21.15.102:0/3293345145 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379035092 0 0) 0x7fac0c06b160 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac0affd700 1 -- 172.21.15.102:0/3293345145 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf40035b0 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac0bfff700 1 -- 172.21.15.102:0/3293345145 >> v2:172.21.15.102:3300/0 conn(0x7fac0c071470 msgr2=0x7fac0c0c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:08:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac0bfff700 1 --2- 172.21.15.102:0/3293345145 >> v2:172.21.15.102:3300/0 conn(0x7fac0c071470 0x7fac0c0c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac0bfff700 1 -- 172.21.15.102:0/3293345145 >> v1:172.21.15.172:6789/0 conn(0x7fac0c0cb2d0 legacy=0x7fac0c0c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac0bfff700 1 -- 172.21.15.102:0/3293345145 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c071d40 con 0x7fac0c14fb20 2023-03-03T09:08:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac10d9c700 1 --2- 172.21.15.102:0/3293345145 >> v2:172.21.15.102:3300/0 conn(0x7fac0c071470 0x7fac0c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:08:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.605+0000 7fac0bfff700 1 --2- 172.21.15.102:0/3293345145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c0721d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac0c070220 tx=0x7fabfc009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cde19b59f5c0049 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:08:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac0affd700 1 -- 172.21.15.102:0/3293345145 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc011830 con 0x7fac0c14fb20 2023-03-03T09:08:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac0affd700 1 -- 172.21.15.102:0/3293345145 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc011990 con 0x7fac0c14fb20 2023-03-03T09:08:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac0affd700 1 -- 172.21.15.102:0/3293345145 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc011d00 con 0x7fac0c14fb20 2023-03-03T09:08:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 -- 172.21.15.102:0/3293345145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 msgr2=0x7fac0c0721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 --2- 172.21.15.102:0/3293345145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c0721d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac0c070220 tx=0x7fabfc009b20 comp rx=0 tx=0).stop 2023-03-03T09:08:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 -- 172.21.15.102:0/3293345145 shutdown_connections 2023-03-03T09:08:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 --2- 172.21.15.102:0/3293345145 >> v2:172.21.15.102:3300/0 conn(0x7fac0c071470 0x7fac0c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 --2- 172.21.15.102:0/3293345145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c0721d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 -- 172.21.15.102:0/3293345145 >> 172.21.15.102:0/3293345145 conn(0x7fac0c0ca7c0 msgr2=0x7fac0c0cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:08:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 -- 172.21.15.102:0/3293345145 shutdown_connections 2023-03-03T09:08:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.606+0000 7fac1159d700 1 -- 172.21.15.102:0/3293345145 wait complete. 2023-03-03T09:08:26.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.608+0000 7fac1159d700 1 Processor -- start 2023-03-03T09:08:26.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.608+0000 7fac1159d700 1 -- start start 2023-03-03T09:08:26.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.609+0000 7fac1159d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac0c071470 0x7fac0c1eee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:26.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.609+0000 7fac1159d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac0c0cb2d0 0x7fac0c0bfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:26.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.609+0000 7fac1159d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:26.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.609+0000 7fac1159d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0c9f00 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.609+0000 7fac1159d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c1ef380 con 0x7fac0c14fb20 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.609+0000 7fac1159d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0c85a0 con 0x7fac0c071470 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac0c0cb2d0 0x7fac0c0bfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac10d9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac0c0cb2d0 0x7fac0c0bfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55310/0 (socket says 172.21.15.102:55310) 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 -- 172.21.15.102:0/114017620 learned_addr learned my addr 172.21.15.102:0/114017620 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0bfff700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac0c071470 0x7fac0c1eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 -- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac0c071470 msgr2=0x7fac0c1eee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac0c071470 0x7fac0c1eee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 -- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 msgr2=0x7fac0c149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac0b7fe700 1 -- 172.21.15.102:0/114017620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac04009b30 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.610+0000 7fac10d9c700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:08:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.611+0000 7fac0b7fe700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac0c0cb2d0 0x7fac0c0bfaf0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c530 tx=0x7fac0000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:08:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.611+0000 7fac08ff9700 1 -- 172.21.15.102:0/114017620 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0000cea0 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.611+0000 7fac08ff9700 1 -- 172.21.15.102:0/114017620 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac00005cb0 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.611+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc0094a0 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.611+0000 7fac08ff9700 1 -- 172.21.15.102:0/114017620 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac00007310 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.611+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c159a90 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.612+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c0706f0 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.613+0000 7fac08ff9700 1 -- 172.21.15.102:0/114017620 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac00019030 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.615+0000 7fac08ff9700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabf80682f0 0x7fabf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:26.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.615+0000 7fac08ff9700 1 -- 172.21.15.102:0/114017620 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fac00084fb0 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.615+0000 7fac0bfff700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabf80682f0 0x7fabf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:26.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.616+0000 7fac0bfff700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabf80682f0 0x7fabf806a7a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fabfc000c00 tx=0x7fabfc01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:08:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.619+0000 7fac08ff9700 1 -- 172.21.15.102:0/114017620 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac00051780 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.856+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac0c0719d0 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.856+0000 7fac08ff9700 1 -- 172.21.15.102:0/114017620 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fac00017020 con 0x7fac0c0cb2d0 2023-03-03T09:08:26.859 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:08:26.859 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:08:26.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabf80682f0 msgr2=0x7fabf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:26.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabf80682f0 0x7fabf806a7a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fabfc000c00 tx=0x7fabfc01c040 comp rx=0 tx=0).stop 2023-03-03T09:08:26.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac0c0cb2d0 msgr2=0x7fac0c0bfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:26.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac0c0cb2d0 0x7fac0c0bfaf0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c530 tx=0x7fac0000c840 comp rx=0 tx=0).stop 2023-03-03T09:08:26.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 shutdown_connections 2023-03-03T09:08:26.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabf80682f0 0x7fabf806a7a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fac0c071470 0x7fac0c1eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac0c14fb20 0x7fac0c149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 --2- 172.21.15.102:0/114017620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac0c0cb2d0 0x7fac0c0bfaf0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:26.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.859+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 >> 172.21.15.102:0/114017620 conn(0x7fac0c0ca7c0 msgr2=0x7fac0c14ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:08:26.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.860+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 shutdown_connections 2023-03-03T09:08:26.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:08:26.860+0000 7fac1159d700 1 -- 172.21.15.102:0/114017620 wait complete. 2023-03-03T09:08:26.863 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:08:26.889 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-03-03T09:08:43.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:43.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:13.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:13.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:44.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:44.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T09:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:10:08.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:10:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.145+0000 7fc62f9ff700 1 Processor -- start 2023-03-03T09:10:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.146+0000 7fc62f9ff700 1 -- start start 2023-03-03T09:10:09.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.147+0000 7fc62f9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc6280c53b0 0x7fc6280ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:09.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.147+0000 7fc62f9ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6280cb280 con 0x7fc6280cae80 2023-03-03T09:10:09.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.147+0000 7fc62f9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6281574b0 con 0x7fc62814d5d0 2023-03-03T09:10:09.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.147+0000 7fc62f9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc628070f20 con 0x7fc6280c53b0 2023-03-03T09:10:09.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc62d79b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc62814d5d0 0x7fc6280c4040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51526/0 (socket says 172.21.15.102:51526) 2023-03-03T09:10:09.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc62d79b700 1 -- 172.21.15.102:0/1914205041 learned_addr learned my addr 172.21.15.102:0/1914205041 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:10:09.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc62cf9a700 1 --2- 172.21.15.102:0/1914205041 >> v2:172.21.15.172:3300/0 conn(0x7fc6280c53b0 0x7fc6280ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:09.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc61ffff700 1 -- 172.21.15.102:0/1914205041 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415967799 0 0) 0x7fc6281574b0 con 0x7fc62814d5d0 2023-03-03T09:10:09.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc61ffff700 1 -- 172.21.15.102:0/1914205041 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6140035b0 con 0x7fc62814d5d0 2023-03-03T09:10:09.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc62cf9a700 1 -- 172.21.15.102:0/1914205041 >> v1:172.21.15.172:6789/0 conn(0x7fc62814d5d0 legacy=0x7fc6280c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:09.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc61ffff700 1 -- 172.21.15.102:0/1914205041 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179015615 0 0) 0x7fc6280cb280 con 0x7fc6280cae80 2023-03-03T09:10:09.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc62cf9a700 1 -- 172.21.15.102:0/1914205041 >> v1:172.21.15.102:6789/0 conn(0x7fc6280cae80 legacy=0x7fc6281613b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:09.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc62cf9a700 1 -- 172.21.15.102:0/1914205041 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc628157750 con 0x7fc6280c53b0 2023-03-03T09:10:09.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.148+0000 7fc62cf9a700 1 --2- 172.21.15.102:0/1914205041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6280c53b0 0x7fc6280ca940 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc618009840 tx=0x7fc618009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c82a6c9f11dc20c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc61ffff700 1 -- 172.21.15.102:0/1914205041 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618011830 con 0x7fc6280c53b0 2023-03-03T09:10:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc61ffff700 1 -- 172.21.15.102:0/1914205041 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc618011990 con 0x7fc6280c53b0 2023-03-03T09:10:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc61ffff700 1 -- 172.21.15.102:0/1914205041 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618011d20 con 0x7fc6280c53b0 2023-03-03T09:10:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1914205041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6280c53b0 msgr2=0x7fc6280ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1914205041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6280c53b0 0x7fc6280ca940 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc618009840 tx=0x7fc618009b50 comp rx=0 tx=0).stop 2023-03-03T09:10:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1914205041 shutdown_connections 2023-03-03T09:10:09.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1914205041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6280c53b0 0x7fc6280ca940 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:09.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.149+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1914205041 >> 172.21.15.102:0/1914205041 conn(0x7fc62806e310 msgr2=0x7fc62806e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:09.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.150+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1914205041 shutdown_connections 2023-03-03T09:10:09.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.150+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1914205041 wait complete. 2023-03-03T09:10:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.152+0000 7fc62f9ff700 1 Processor -- start 2023-03-03T09:10:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.152+0000 7fc62f9ff700 1 -- start start 2023-03-03T09:10:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62f9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6280c53b0 0x7fc6280c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:09.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62f9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6280cae80 0x7fc6280c21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62f9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc62814d5d0 0x7fc6280c2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62f9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6281f2470 con 0x7fc6280c53b0 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62f9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6281f15c0 con 0x7fc62814d5d0 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62f9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc628153b60 con 0x7fc6280cae80 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62d79b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6280c53b0 0x7fc6280c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62d79b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6280c53b0 0x7fc6280c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40844/0 (socket says 172.21.15.102:40844) 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62d79b700 1 -- 172.21.15.102:0/1995640669 learned_addr learned my addr 172.21.15.102:0/1995640669 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:10:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62cf9a700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6280cae80 0x7fc6280c21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:09.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.153+0000 7fc62df9c700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc62814d5d0 0x7fc6280c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:09.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc62d79b700 1 -- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6280cae80 msgr2=0x7fc6280c21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:09.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc62d79b700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6280cae80 0x7fc6280c21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:09.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc62d79b700 1 -- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc62814d5d0 msgr2=0x7fc6280c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:09.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc62d79b700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc62814d5d0 0x7fc6280c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:09.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc62d79b700 1 -- 172.21.15.102:0/1995640669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6180094a0 con 0x7fc6280c53b0 2023-03-03T09:10:09.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc62d79b700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6280c53b0 0x7fc6280c6950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc62400c500 tx=0x7fc62400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:09.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc61dffb700 1 -- 172.21.15.102:0/1995640669 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624005d70 con 0x7fc6280c53b0 2023-03-03T09:10:09.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc61dffb700 1 -- 172.21.15.102:0/1995640669 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6240070d0 con 0x7fc6280c53b0 2023-03-03T09:10:09.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc61dffb700 1 -- 172.21.15.102:0/1995640669 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624007480 con 0x7fc6280c53b0 2023-03-03T09:10:09.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.154+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6280c79e0 con 0x7fc6280c53b0 2023-03-03T09:10:09.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.155+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6280c7700 con 0x7fc6280c53b0 2023-03-03T09:10:09.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.156+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc628070720 con 0x7fc6280c53b0 2023-03-03T09:10:09.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.157+0000 7fc61dffb700 1 -- 172.21.15.102:0/1995640669 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc624019030 con 0x7fc6280c53b0 2023-03-03T09:10:09.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.158+0000 7fc61dffb700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc60c0682f0 0x7fc60c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:09.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.158+0000 7fc61dffb700 1 -- 172.21.15.102:0/1995640669 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc624085090 con 0x7fc6280c53b0 2023-03-03T09:10:09.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.158+0000 7fc62cf9a700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc60c0682f0 0x7fc60c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:09.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.159+0000 7fc62cf9a700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc60c0682f0 0x7fc60c06a7a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc618009840 tx=0x7fc61801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:09.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.162+0000 7fc61dffb700 1 -- 172.21.15.102:0/1995640669 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6240516b0 con 0x7fc6280c53b0 2023-03-03T09:10:09.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.400+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6280c2010 con 0x7fc6280c53b0 2023-03-03T09:10:09.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.401+0000 7fc61dffb700 1 -- 172.21.15.102:0/1995640669 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc624017070 con 0x7fc6280c53b0 2023-03-03T09:10:09.403 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:10:09.403 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:10:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.403+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc60c0682f0 msgr2=0x7fc60c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.403+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc60c0682f0 0x7fc60c06a7a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc618009840 tx=0x7fc61801c040 comp rx=0 tx=0).stop 2023-03-03T09:10:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.403+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6280c53b0 msgr2=0x7fc6280c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6280c53b0 0x7fc6280c6950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc62400c500 tx=0x7fc62400c810 comp rx=0 tx=0).stop 2023-03-03T09:10:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 shutdown_connections 2023-03-03T09:10:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc62814d5d0 0x7fc6280c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6280c53b0 0x7fc6280c6950 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc60c0682f0 0x7fc60c06a7a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 --2- 172.21.15.102:0/1995640669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6280cae80 0x7fc6280c21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 >> 172.21.15.102:0/1995640669 conn(0x7fc62806e310 msgr2=0x7fc6280c4040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 shutdown_connections 2023-03-03T09:10:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:09.404+0000 7fc62f9ff700 1 -- 172.21.15.102:0/1995640669 wait complete. 2023-03-03T09:10:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:10:09.433 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-03-03T09:10:14.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:14.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:44.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:44.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:55.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:10:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.691+0000 7f513b3ff700 1 Processor -- start 2023-03-03T09:10:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.691+0000 7f513b3ff700 1 -- start start 2023-03-03T09:10:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.692+0000 7f513b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5134153290 0x7f5134158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.692+0000 7f513b3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f51340cd8e0 0x7f51340ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.692+0000 7f513b3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5134153660 con 0x7f51341593c0 2023-03-03T09:10:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.692+0000 7f513b3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513406c360 con 0x7f51340cd8e0 2023-03-03T09:10:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.692+0000 7f513b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513415cea0 con 0x7f5134153290 2023-03-03T09:10:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513919b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5134153290 0x7f5134158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513899a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f51341593c0 0x7f51340cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42530/0 (socket says 172.21.15.102:42530) 2023-03-03T09:10:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513899a700 1 -- 172.21.15.102:0/3817386093 learned_addr learned my addr 172.21.15.102:0/3817386093 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:10:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513999c700 1 --2- 172.21.15.102:0/3817386093 >> v2:172.21.15.102:3300/0 conn(0x7f51340cd8e0 0x7f51340ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f512bfff700 1 -- 172.21.15.102:0/3817386093 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111953062 0 0) 0x7f5134153660 con 0x7f51341593c0 2023-03-03T09:10:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513919b700 1 -- 172.21.15.102:0/3817386093 >> v2:172.21.15.102:3300/0 conn(0x7f51340cd8e0 msgr2=0x7f51340ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:10:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513919b700 1 --2- 172.21.15.102:0/3817386093 >> v2:172.21.15.102:3300/0 conn(0x7f51340cd8e0 0x7f51340ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513919b700 1 -- 172.21.15.102:0/3817386093 >> v1:172.21.15.172:6789/0 conn(0x7f51341593c0 legacy=0x7f51340cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513919b700 1 -- 172.21.15.102:0/3817386093 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513406c7f0 con 0x7f5134153290 2023-03-03T09:10:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.693+0000 7f513919b700 1 --2- 172.21.15.102:0/3817386093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5134153290 0x7f5134158e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5130009840 tx=0x7f5130009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=216c8e48b630e00a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f512bfff700 1 -- 172.21.15.102:0/3817386093 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5130011830 con 0x7f5134153290 2023-03-03T09:10:55.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f512bfff700 1 -- 172.21.15.102:0/3817386093 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5130011990 con 0x7f5134153290 2023-03-03T09:10:55.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f512bfff700 1 -- 172.21.15.102:0/3817386093 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5130011d20 con 0x7f5134153290 2023-03-03T09:10:55.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f513b3ff700 1 -- 172.21.15.102:0/3817386093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5134153290 msgr2=0x7f5134158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:55.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f513b3ff700 1 --2- 172.21.15.102:0/3817386093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5134153290 0x7f5134158e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5130009840 tx=0x7f5130009b50 comp rx=0 tx=0).stop 2023-03-03T09:10:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f513b3ff700 1 -- 172.21.15.102:0/3817386093 shutdown_connections 2023-03-03T09:10:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f513b3ff700 1 --2- 172.21.15.102:0/3817386093 >> v2:172.21.15.102:3300/0 conn(0x7f51340cd8e0 0x7f51340ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f513b3ff700 1 --2- 172.21.15.102:0/3817386093 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5134153290 0x7f5134158e80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.694+0000 7f513b3ff700 1 -- 172.21.15.102:0/3817386093 >> 172.21.15.102:0/3817386093 conn(0x7f5134065fb0 msgr2=0x7f51340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.695+0000 7f513b3ff700 1 -- 172.21.15.102:0/3817386093 shutdown_connections 2023-03-03T09:10:55.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.695+0000 7f513b3ff700 1 -- 172.21.15.102:0/3817386093 wait complete. 2023-03-03T09:10:55.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.697+0000 7f513b3ff700 1 Processor -- start 2023-03-03T09:10:55.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.697+0000 7f513b3ff700 1 -- start start 2023-03-03T09:10:55.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513b3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51340cd8e0 0x7f513414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:55.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513b3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134153290 0x7f51340c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:55.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51341593c0 0x7f51340c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:55.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513b3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51340c6140 con 0x7f5134153290 2023-03-03T09:10:55.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513b3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134150080 con 0x7f51340cd8e0 2023-03-03T09:10:55.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51340c63e0 con 0x7f51341593c0 2023-03-03T09:10:55.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513899a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134153290 0x7f51340c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:55.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513919b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51340cd8e0 0x7f513414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:55.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513919b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51340cd8e0 0x7f513414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37338/0 (socket says 172.21.15.102:37338) 2023-03-03T09:10:55.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513919b700 1 -- 172.21.15.102:0/2961311861 learned_addr learned my addr 172.21.15.102:0/2961311861 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:10:55.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.698+0000 7f513999c700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51341593c0 0x7f51340c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:55.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513899a700 1 -- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51341593c0 msgr2=0x7f51340c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:55.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513899a700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51341593c0 0x7f51340c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513899a700 1 -- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51340cd8e0 msgr2=0x7f513414fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:55.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513899a700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51340cd8e0 0x7f513414fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513899a700 1 -- 172.21.15.102:0/2961311861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300094a0 con 0x7f5134153290 2023-03-03T09:10:55.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513919b700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51340cd8e0 0x7f513414fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:10:55.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513899a700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134153290 0x7f51340c88d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f512400c4b0 tx=0x7f512400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:55.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f5129ffb700 1 -- 172.21.15.102:0/2961311861 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124004c80 con 0x7f5134153290 2023-03-03T09:10:55.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f5129ffb700 1 -- 172.21.15.102:0/2961311861 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5124004e00 con 0x7f5134153290 2023-03-03T09:10:55.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f5129ffb700 1 -- 172.21.15.102:0/2961311861 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124007310 con 0x7f5134153290 2023-03-03T09:10:55.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51340ccde0 con 0x7f5134153290 2023-03-03T09:10:55.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.699+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513415d160 con 0x7f5134153290 2023-03-03T09:10:55.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.700+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51340cd0c0 con 0x7f5134153290 2023-03-03T09:10:55.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.702+0000 7f5129ffb700 1 -- 172.21.15.102:0/2961311861 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f512401a020 con 0x7f5134153290 2023-03-03T09:10:55.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.703+0000 7f5129ffb700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5118068320 0x7f511806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:55.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.703+0000 7f5129ffb700 1 -- 172.21.15.102:0/2961311861 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5124085890 con 0x7f5134153290 2023-03-03T09:10:55.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.703+0000 7f513919b700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5118068320 0x7f511806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:55.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.704+0000 7f513919b700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5118068320 0x7f511806a7d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f5130008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:55.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.707+0000 7f5129ffb700 1 -- 172.21.15.102:0/2961311861 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5124052080 con 0x7f5134153290 2023-03-03T09:10:55.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.943+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51340b3700 con 0x7f5134153290 2023-03-03T09:10:55.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.944+0000 7f5129ffb700 1 -- 172.21.15.102:0/2961311861 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f51340b3700 con 0x7f5134153290 2023-03-03T09:10:55.946 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:10:55.946 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:10:55.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5118068320 msgr2=0x7f511806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:55.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5118068320 0x7f511806a7d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f5130008030 comp rx=0 tx=0).stop 2023-03-03T09:10:55.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134153290 msgr2=0x7f51340c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:55.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134153290 0x7f51340c88d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f512400c4b0 tx=0x7f512400c7c0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 shutdown_connections 2023-03-03T09:10:55.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5118068320 0x7f511806a7d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51341593c0 0x7f51340c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51340cd8e0 0x7f513414fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 --2- 172.21.15.102:0/2961311861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134153290 0x7f51340c88d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:55.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 >> 172.21.15.102:0/2961311861 conn(0x7f5134065fb0 msgr2=0x7f5134156b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:55.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 shutdown_connections 2023-03-03T09:10:55.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:10:55.947+0000 7f513b3ff700 1 -- 172.21.15.102:0/2961311861 wait complete. 2023-03-03T09:10:55.951 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:10:55.976 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-03-03T09:11:14.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:14.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:42.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:11:43.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.224+0000 7f390c3ff700 1 Processor -- start 2023-03-03T09:11:43.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.225+0000 7f390c3ff700 1 -- start start 2023-03-03T09:11:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390c3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3904153290 0x7f3904158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390c3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f39041593c0 0x7f39040cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390c3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3904153660 con 0x7f39040cd8e0 2023-03-03T09:11:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390c3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390406c360 con 0x7f3904153290 2023-03-03T09:11:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390c3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390415cea0 con 0x7f39041593c0 2023-03-03T09:11:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3904153290 0x7f3904158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a99c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f39040cd8e0 0x7f39040ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48704/0 (socket says 172.21.15.102:48704) 2023-03-03T09:11:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a99c700 1 -- 172.21.15.102:0/3139280002 learned_addr learned my addr 172.21.15.102:0/3139280002 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:11:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390999a700 1 --2- 172.21.15.102:0/3139280002 >> v2:172.21.15.172:3300/0 conn(0x7f39041593c0 0x7f39040cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a19b700 1 -- 172.21.15.102:0/3139280002 >> v2:172.21.15.172:3300/0 conn(0x7f39041593c0 msgr2=0x7f39040cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:11:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a19b700 1 --2- 172.21.15.102:0/3139280002 >> v2:172.21.15.172:3300/0 conn(0x7f39041593c0 0x7f39040cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a19b700 1 -- 172.21.15.102:0/3139280002 >> v1:172.21.15.102:6789/0 conn(0x7f39040cd8e0 legacy=0x7f39040ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a19b700 1 -- 172.21.15.102:0/3139280002 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390406c7f0 con 0x7f3904153290 2023-03-03T09:11:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f390a19b700 1 --2- 172.21.15.102:0/3139280002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904158e80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3904159fb0 tx=0x7f3900009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea608503242bb95 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:11:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f3909199700 1 -- 172.21.15.102:0/3139280002 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900011830 con 0x7f3904153290 2023-03-03T09:11:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f3909199700 1 -- 172.21.15.102:0/3139280002 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3900011990 con 0x7f3904153290 2023-03-03T09:11:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.227+0000 7f3909199700 1 -- 172.21.15.102:0/3139280002 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900011d20 con 0x7f3904153290 2023-03-03T09:11:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 -- 172.21.15.102:0/3139280002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 msgr2=0x7f3904158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 --2- 172.21.15.102:0/3139280002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904158e80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3904159fb0 tx=0x7f3900009b20 comp rx=0 tx=0).stop 2023-03-03T09:11:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 -- 172.21.15.102:0/3139280002 shutdown_connections 2023-03-03T09:11:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 --2- 172.21.15.102:0/3139280002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904158e80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 --2- 172.21.15.102:0/3139280002 >> v2:172.21.15.172:3300/0 conn(0x7f39041593c0 0x7f39040cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 -- 172.21.15.102:0/3139280002 >> 172.21.15.102:0/3139280002 conn(0x7f3904065fb0 msgr2=0x7f39040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:11:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 -- 172.21.15.102:0/3139280002 shutdown_connections 2023-03-03T09:11:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.228+0000 7f390c3ff700 1 -- 172.21.15.102:0/3139280002 wait complete. 2023-03-03T09:11:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.230+0000 7f390c3ff700 1 Processor -- start 2023-03-03T09:11:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.230+0000 7f390c3ff700 1 -- start start 2023-03-03T09:11:43.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.231+0000 7f390c3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39040cd8e0 0x7f390414c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:43.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.231+0000 7f390c3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:43.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.231+0000 7f390c3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39041593c0 0x7f3904156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:43.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.231+0000 7f390c3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39041ee930 con 0x7f3904153290 2023-03-03T09:11:43.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.231+0000 7f390c3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040b45f0 con 0x7f39041593c0 2023-03-03T09:11:43.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.231+0000 7f390c3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040b1d10 con 0x7f39040cd8e0 2023-03-03T09:11:43.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39040cd8e0 0x7f390414c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:43.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a99c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39041593c0 0x7f3904156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:43.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39040cd8e0 0x7f390414c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57898/0 (socket says 172.21.15.102:57898) 2023-03-03T09:11:43.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 -- 172.21.15.102:0/2372579860 learned_addr learned my addr 172.21.15.102:0/2372579860 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390999a700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 -- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39041593c0 msgr2=0x7f3904156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39041593c0 0x7f3904156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 -- 172.21.15.102:0/2372579860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 msgr2=0x7f3904155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 -- 172.21.15.102:0/2372579860 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39000094a0 con 0x7f39040cd8e0 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390999a700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:11:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.232+0000 7f390a19b700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39040cd8e0 0x7f390414c770 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3900009b60 tx=0x7f3900004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:11:43.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.233+0000 7f38faffd700 1 -- 172.21.15.102:0/2372579860 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000036a0 con 0x7f39040cd8e0 2023-03-03T09:11:43.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.233+0000 7f38faffd700 1 -- 172.21.15.102:0/2372579860 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390001f070 con 0x7f39040cd8e0 2023-03-03T09:11:43.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.233+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39040ccde0 con 0x7f39040cd8e0 2023-03-03T09:11:43.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.233+0000 7f38faffd700 1 -- 172.21.15.102:0/2372579860 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900003a60 con 0x7f39040cd8e0 2023-03-03T09:11:43.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.233+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390415d160 con 0x7f39040cd8e0 2023-03-03T09:11:43.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.234+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39040cd0c0 con 0x7f39040cd8e0 2023-03-03T09:11:43.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.235+0000 7f38faffd700 1 -- 172.21.15.102:0/2372579860 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f390000f050 con 0x7f39040cd8e0 2023-03-03T09:11:43.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.236+0000 7f38faffd700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38e80682f0 0x7f38e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:43.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.237+0000 7f38faffd700 1 -- 172.21.15.102:0/2372579860 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3900021070 con 0x7f39040cd8e0 2023-03-03T09:11:43.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.237+0000 7f390999a700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38e80682f0 0x7f38e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:43.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.238+0000 7f390999a700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38e80682f0 0x7f38e806a7a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f390415a4d0 tx=0x7f38f4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:11:43.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.243+0000 7f38faffd700 1 -- 172.21.15.102:0/2372579860 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3900053c20 con 0x7f39040cd8e0 2023-03-03T09:11:43.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.475+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f390404e6e0 con 0x7f39040cd8e0 2023-03-03T09:11:43.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.477+0000 7f38faffd700 1 -- 172.21.15.102:0/2372579860 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3900057240 con 0x7f39040cd8e0 2023-03-03T09:11:43.478 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:11:43.478 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:11:43.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38e80682f0 msgr2=0x7f38e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:43.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38e80682f0 0x7f38e806a7a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f390415a4d0 tx=0x7f38f4009210 comp rx=0 tx=0).stop 2023-03-03T09:11:43.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39040cd8e0 msgr2=0x7f390414c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:43.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39040cd8e0 0x7f390414c770 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3900009b60 tx=0x7f3900004750 comp rx=0 tx=0).stop 2023-03-03T09:11:43.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 shutdown_connections 2023-03-03T09:11:43.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f38e80682f0 0x7f38e806a7a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39040cd8e0 0x7f390414c770 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39041593c0 0x7f3904156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 --2- 172.21.15.102:0/2372579860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3904153290 0x7f3904155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:43.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 >> 172.21.15.102:0/2372579860 conn(0x7f3904065fb0 msgr2=0x7f39040cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:11:43.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 shutdown_connections 2023-03-03T09:11:43.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:11:43.479+0000 7f390c3ff700 1 -- 172.21.15.102:0/2372579860 wait complete. 2023-03-03T09:11:43.483 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:11:43.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T09:11:44.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:44.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:14.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:14.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:31.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:12:31.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.776+0000 7f2f7f59d700 1 Processor -- start 2023-03-03T09:12:31.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.776+0000 7f2f7f59d700 1 -- start start 2023-03-03T09:12:31.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7f59d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2f780ced00 0x7f2f780c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:31.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7f59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780b3970 0x7f2f780c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:31.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7f59d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780c3d20 con 0x7f2f780cf0d0 2023-03-03T09:12:31.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7f59d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f780c4930 con 0x7f2f780ced00 2023-03-03T09:12:31.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7f59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7815cba0 con 0x7f2f780b3970 2023-03-03T09:12:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7e59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2f780ced00 0x7f2f780c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7e59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2f780ced00 0x7f2f780c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38502/0 (socket says 172.21.15.102:38502) 2023-03-03T09:12:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7e59b700 1 -- 172.21.15.102:0/2986640317 learned_addr learned my addr 172.21.15.102:0/2986640317 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:12:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7dd9a700 1 --2- 172.21.15.102:0/2986640317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780b3970 0x7f2f780c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.778+0000 7f2f7ed9c700 1 --1- 172.21.15.102:0/2986640317 >> v1:172.21.15.102:6789/0 conn(0x7f2f780cf0d0 0x7f2f78158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53422/0 (socket says 172.21.15.102:53422) 2023-03-03T09:12:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7e59b700 1 -- 172.21.15.102:0/2986640317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780b3970 msgr2=0x7f2f780c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7e59b700 1 --2- 172.21.15.102:0/2986640317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780b3970 0x7f2f780c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7e59b700 1 -- 172.21.15.102:0/2986640317 >> v1:172.21.15.102:6789/0 conn(0x7f2f780cf0d0 legacy=0x7f2f78158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7e59b700 1 -- 172.21.15.102:0/2986640317 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7815ce80 con 0x7f2f780ced00 2023-03-03T09:12:31.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7d599700 1 -- 172.21.15.102:0/2986640317 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264544287 0 0) 0x7f2f780c3d20 con 0x7f2f780cf0d0 2023-03-03T09:12:31.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7e59b700 1 --2- 172.21.15.102:0/2986640317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780ced00 0x7f2f780c5a90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2f740098f0 tx=0x7f2f74009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ada78502ee3fea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:31.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7d599700 1 -- 172.21.15.102:0/2986640317 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74011830 con 0x7f2f780ced00 2023-03-03T09:12:31.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7d599700 1 -- 172.21.15.102:0/2986640317 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f74011990 con 0x7f2f780ced00 2023-03-03T09:12:31.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.779+0000 7f2f7d599700 1 -- 172.21.15.102:0/2986640317 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74011d20 con 0x7f2f780ced00 2023-03-03T09:12:31.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 -- 172.21.15.102:0/2986640317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780ced00 msgr2=0x7f2f780c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:31.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/2986640317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780ced00 0x7f2f780c5a90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2f740098f0 tx=0x7f2f74009c00 comp rx=0 tx=0).stop 2023-03-03T09:12:31.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 -- 172.21.15.102:0/2986640317 shutdown_connections 2023-03-03T09:12:31.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/2986640317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780ced00 0x7f2f780c5a90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:31.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/2986640317 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780b3970 0x7f2f780c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:31.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 -- 172.21.15.102:0/2986640317 >> 172.21.15.102:0/2986640317 conn(0x7f2f7808b1c0 msgr2=0x7f2f78071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:31.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 -- 172.21.15.102:0/2986640317 shutdown_connections 2023-03-03T09:12:31.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.780+0000 7f2f7f59d700 1 -- 172.21.15.102:0/2986640317 wait complete. 2023-03-03T09:12:31.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.783+0000 7f2f7f59d700 1 Processor -- start 2023-03-03T09:12:31.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.783+0000 7f2f7f59d700 1 -- start start 2023-03-03T09:12:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7f59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780b3970 0x7f2f781517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7f59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780ced00 0x7f2f78150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7f59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f780cf0d0 0x7f2f7815acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7f59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7815b8a0 con 0x7f2f780b3970 2023-03-03T09:12:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7f59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f78150780 con 0x7f2f780cf0d0 2023-03-03T09:12:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7f59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7815a370 con 0x7f2f780ced00 2023-03-03T09:12:31.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780b3970 0x7f2f781517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:31.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780b3970 0x7f2f781517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38516/0 (socket says 172.21.15.102:38516) 2023-03-03T09:12:31.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 -- 172.21.15.102:0/1590523481 learned_addr learned my addr 172.21.15.102:0/1590523481 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:12:31.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7ed9c700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f780cf0d0 0x7f2f7815acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:31.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7dd9a700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780ced00 0x7f2f78150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:31.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 -- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780ced00 msgr2=0x7f2f78150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:31.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780ced00 0x7f2f78150bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 -- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f780cf0d0 msgr2=0x7f2f7815acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f780cf0d0 0x7f2f7815acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7e59b700 1 -- 172.21.15.102:0/1590523481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f74009550 con 0x7f2f780b3970 2023-03-03T09:12:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.785+0000 7f2f7dd9a700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780ced00 0x7f2f78150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:12:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.786+0000 7f2f7e59b700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780b3970 0x7f2f781517f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2f7808ba70 tx=0x7f2f74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.786+0000 7f2f6effd700 1 -- 172.21.15.102:0/1590523481 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f740045b0 con 0x7f2f780b3970 2023-03-03T09:12:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.786+0000 7f2f6effd700 1 -- 172.21.15.102:0/1590523481 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7401f070 con 0x7f2f780b3970 2023-03-03T09:12:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.786+0000 7f2f6effd700 1 -- 172.21.15.102:0/1590523481 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f740038e0 con 0x7f2f780b3970 2023-03-03T09:12:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.786+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7806c200 con 0x7f2f780b3970 2023-03-03T09:12:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.786+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7806bf20 con 0x7f2f780b3970 2023-03-03T09:12:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.787+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f780cf540 con 0x7f2f780b3970 2023-03-03T09:12:31.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.788+0000 7f2f6effd700 1 -- 172.21.15.102:0/1590523481 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f7402a020 con 0x7f2f780b3970 2023-03-03T09:12:31.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.789+0000 7f2f6effd700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f5c0682f0 0x7f2f5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:31.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.790+0000 7f2f6effd700 1 -- 172.21.15.102:0/1590523481 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2f740874c0 con 0x7f2f780b3970 2023-03-03T09:12:31.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.790+0000 7f2f7dd9a700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f5c0682f0 0x7f2f5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:31.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.791+0000 7f2f7dd9a700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f5c0682f0 0x7f2f5c06a7a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2f781498b0 tx=0x7f2f68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:31.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:31.794+0000 7f2f6effd700 1 -- 172.21.15.102:0/1590523481 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f74053bd0 con 0x7f2f780b3970 2023-03-03T09:12:32.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.028+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f7815c0f0 con 0x7f2f780b3970 2023-03-03T09:12:32.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.028+0000 7f2f6effd700 1 -- 172.21.15.102:0/1590523481 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2f740571f0 con 0x7f2f780b3970 2023-03-03T09:12:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:12:32.030 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:12:32.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f5c0682f0 msgr2=0x7f2f5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:32.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f5c0682f0 0x7f2f5c06a7a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2f781498b0 tx=0x7f2f68008040 comp rx=0 tx=0).stop 2023-03-03T09:12:32.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780b3970 msgr2=0x7f2f781517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780b3970 0x7f2f781517f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2f7808ba70 tx=0x7f2f74004750 comp rx=0 tx=0).stop 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 shutdown_connections 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f780cf0d0 0x7f2f7815acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f780b3970 0x7f2f781517f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f5c0682f0 0x7f2f5c06a7a0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 --2- 172.21.15.102:0/1590523481 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f780ced00 0x7f2f78150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 >> 172.21.15.102:0/1590523481 conn(0x7f2f7808b1c0 msgr2=0x7f2f78071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 shutdown_connections 2023-03-03T09:12:32.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:12:32.031+0000 7f2f7f59d700 1 -- 172.21.15.102:0/1590523481 wait complete. 2023-03-03T09:12:32.034 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:12:32.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 86 secs before thrashing 2023-03-03T09:12:45.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:45.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:15.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:15.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:45.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:45.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:58.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:13:58.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.308+0000 7fbe96fff700 1 Processor -- start 2023-03-03T09:13:58.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.309+0000 7fbe96fff700 1 -- start start 2023-03-03T09:13:58.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe96fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbe90153290 0x7fbe90158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:58.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe96fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbe900cd8e0 0x7fbe900ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:58.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe96fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90153660 con 0x7fbe901593c0 2023-03-03T09:13:58.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe96fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9006c360 con 0x7fbe900cd8e0 2023-03-03T09:13:58.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe96fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9015cea0 con 0x7fbe90153290 2023-03-03T09:13:58.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe94d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbe90153290 0x7fbe90158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:58.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe8ffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbe901593c0 0x7fbe900cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49372/0 (socket says 172.21.15.102:49372) 2023-03-03T09:13:58.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe8ffff700 1 -- 172.21.15.102:0/932762109 learned_addr learned my addr 172.21.15.102:0/932762109 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:13:58.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.310+0000 7fbe9559c700 1 --2- 172.21.15.102:0/932762109 >> v2:172.21.15.102:3300/0 conn(0x7fbe900cd8e0 0x7fbe900ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:58.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.311+0000 7fbe94d9b700 1 -- 172.21.15.102:0/932762109 >> v2:172.21.15.102:3300/0 conn(0x7fbe900cd8e0 msgr2=0x7fbe900ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:13:58.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.311+0000 7fbe94d9b700 1 --2- 172.21.15.102:0/932762109 >> v2:172.21.15.102:3300/0 conn(0x7fbe900cd8e0 0x7fbe900ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.311+0000 7fbe8f7fe700 1 -- 172.21.15.102:0/932762109 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893116825 0 0) 0x7fbe90153660 con 0x7fbe901593c0 2023-03-03T09:13:58.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.311+0000 7fbe94d9b700 1 -- 172.21.15.102:0/932762109 >> v1:172.21.15.172:6789/0 conn(0x7fbe901593c0 legacy=0x7fbe900cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:58.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.311+0000 7fbe94d9b700 1 -- 172.21.15.102:0/932762109 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9006c7f0 con 0x7fbe90153290 2023-03-03T09:13:58.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.311+0000 7fbe9559c700 1 --2- 172.21.15.102:0/932762109 >> v2:172.21.15.102:3300/0 conn(0x7fbe900cd8e0 0x7fbe900ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:13:58.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.311+0000 7fbe94d9b700 1 --2- 172.21.15.102:0/932762109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 0x7fbe90158e80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbe9008cc70 tx=0x7fbe84009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a88ad08bc0f137a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.312+0000 7fbe8f7fe700 1 -- 172.21.15.102:0/932762109 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84011830 con 0x7fbe90153290 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.312+0000 7fbe8f7fe700 1 -- 172.21.15.102:0/932762109 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84011990 con 0x7fbe90153290 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.312+0000 7fbe8f7fe700 1 -- 172.21.15.102:0/932762109 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84011d20 con 0x7fbe90153290 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.312+0000 7fbe96fff700 1 -- 172.21.15.102:0/932762109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 msgr2=0x7fbe90158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.312+0000 7fbe96fff700 1 --2- 172.21.15.102:0/932762109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 0x7fbe90158e80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbe9008cc70 tx=0x7fbe84009b20 comp rx=0 tx=0).stop 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.313+0000 7fbe96fff700 1 -- 172.21.15.102:0/932762109 shutdown_connections 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.313+0000 7fbe96fff700 1 --2- 172.21.15.102:0/932762109 >> v2:172.21.15.102:3300/0 conn(0x7fbe900cd8e0 0x7fbe900ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.313+0000 7fbe96fff700 1 --2- 172.21.15.102:0/932762109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 0x7fbe90158e80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.313+0000 7fbe96fff700 1 -- 172.21.15.102:0/932762109 >> 172.21.15.102:0/932762109 conn(0x7fbe90065fb0 msgr2=0x7fbe900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:13:58.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.313+0000 7fbe96fff700 1 -- 172.21.15.102:0/932762109 shutdown_connections 2023-03-03T09:13:58.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.313+0000 7fbe96fff700 1 -- 172.21.15.102:0/932762109 wait complete. 2023-03-03T09:13:58.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.315+0000 7fbe96fff700 1 Processor -- start 2023-03-03T09:13:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.315+0000 7fbe96fff700 1 -- start start 2023-03-03T09:13:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.316+0000 7fbe96fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe900cd8e0 0x7fbe900cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:58.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.316+0000 7fbe96fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 0x7fbe901ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.316+0000 7fbe96fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe901593c0 0x7fbe9014ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.316+0000 7fbe96fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe901f1b20 con 0x7fbe901593c0 2023-03-03T09:13:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.316+0000 7fbe96fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe901f23c0 con 0x7fbe90153290 2023-03-03T09:13:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.316+0000 7fbe96fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe900b3030 con 0x7fbe900cd8e0 2023-03-03T09:13:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.316+0000 7fbe9559c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe901593c0 0x7fbe9014ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 0x7fbe901ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:58.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe901593c0 0x7fbe9014ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46568/0 (socket says 172.21.15.102:46568) 2023-03-03T09:13:58.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 -- 172.21.15.102:0/695331997 learned_addr learned my addr 172.21.15.102:0/695331997 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:13:58.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe94d9b700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe900cd8e0 0x7fbe900cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 -- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe900cd8e0 msgr2=0x7fbe900cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe900cd8e0 0x7fbe900cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 -- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 msgr2=0x7fbe901ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 0x7fbe901ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 -- 172.21.15.102:0/695331997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe88009a70 con 0x7fbe901593c0 2023-03-03T09:13:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe9559c700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe901593c0 0x7fbe9014ed90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009e60 tx=0x7fbe8800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:13:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.317+0000 7fbe8d7fa700 1 -- 172.21.15.102:0/695331997 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88009cd0 con 0x7fbe901593c0 2023-03-03T09:13:58.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.318+0000 7fbe8d7fa700 1 -- 172.21.15.102:0/695331997 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8800f040 con 0x7fbe901593c0 2023-03-03T09:13:58.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.318+0000 7fbe8d7fa700 1 -- 172.21.15.102:0/695331997 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8800ccb0 con 0x7fbe901593c0 2023-03-03T09:13:58.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.318+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe840094a0 con 0x7fbe901593c0 2023-03-03T09:13:58.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.318+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe900ccde0 con 0x7fbe901593c0 2023-03-03T09:13:58.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.319+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9015d160 con 0x7fbe901593c0 2023-03-03T09:13:58.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.320+0000 7fbe8d7fa700 1 -- 172.21.15.102:0/695331997 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbe88014480 con 0x7fbe901593c0 2023-03-03T09:13:58.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.321+0000 7fbe8d7fa700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbe740682f0 0x7fbe7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:58.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.321+0000 7fbe8d7fa700 1 -- 172.21.15.102:0/695331997 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbe88086500 con 0x7fbe901593c0 2023-03-03T09:13:58.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.322+0000 7fbe94d9b700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbe740682f0 0x7fbe7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:58.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.322+0000 7fbe94d9b700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbe740682f0 0x7fbe7406a7a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbe840090b0 tx=0x7fbe84008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:13:58.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.325+0000 7fbe8d7fa700 1 -- 172.21.15.102:0/695331997 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbe88052bb0 con 0x7fbe901593c0 2023-03-03T09:13:58.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.562+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe900cd0c0 con 0x7fbe901593c0 2023-03-03T09:13:58.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.563+0000 7fbe8d7fa700 1 -- 172.21.15.102:0/695331997 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbe880561d0 con 0x7fbe901593c0 2023-03-03T09:13:58.565 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:13:58.565 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:13:58.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.565+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbe740682f0 msgr2=0x7fbe7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:58.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.565+0000 7fbe96fff700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbe740682f0 0x7fbe7406a7a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbe840090b0 tx=0x7fbe84008030 comp rx=0 tx=0).stop 2023-03-03T09:13:58.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe901593c0 msgr2=0x7fbe9014ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:58.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe901593c0 0x7fbe9014ed90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009e60 tx=0x7fbe8800c3a0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 shutdown_connections 2023-03-03T09:13:58.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbe740682f0 0x7fbe7406a7a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbe900cd8e0 0x7fbe900cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe901593c0 0x7fbe9014ed90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 --2- 172.21.15.102:0/695331997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe90153290 0x7fbe901ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:58.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 >> 172.21.15.102:0/695331997 conn(0x7fbe90065fb0 msgr2=0x7fbe900cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:13:58.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 shutdown_connections 2023-03-03T09:13:58.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:13:58.566+0000 7fbe96fff700 1 -- 172.21.15.102:0/695331997 wait complete. 2023-03-03T09:13:58.570 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:13:58.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-03-03T09:14:15.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:15.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:45.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:45.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:16.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:16.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:46.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:46.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:47.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:15:47.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.850+0000 7f324e3ff700 1 Processor -- start 2023-03-03T09:15:47.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.850+0000 7f324e3ff700 1 -- start start 2023-03-03T09:15:47.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f324e3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3248153290 0x7f3248158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:47.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f324e3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32481593c0 0x7f32480cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:47.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f324e3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3248153660 con 0x7f32480cd8e0 2023-03-03T09:15:47.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f324e3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f324806c360 con 0x7f32481593c0 2023-03-03T09:15:47.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f324e3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324815cea0 con 0x7f3248153290 2023-03-03T09:15:47.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f323ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32481593c0 0x7f32480cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:47.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f323ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32481593c0 0x7f32480cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40680/0 (socket says 172.21.15.102:40680) 2023-03-03T09:15:47.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f323ffff700 1 -- 172.21.15.102:0/2437427155 learned_addr learned my addr 172.21.15.102:0/2437427155 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:15:47.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f3247fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3248153290 0x7f3248158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:47.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f323ffff700 1 -- 172.21.15.102:0/2437427155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3248153290 msgr2=0x7f3248158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:15:47.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f323ffff700 1 --2- 172.21.15.102:0/2437427155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3248153290 0x7f3248158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:47.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.852+0000 7f323ffff700 1 -- 172.21.15.102:0/2437427155 >> v1:172.21.15.102:6789/0 conn(0x7f32480cd8e0 legacy=0x7f32480ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:47.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.853+0000 7f323ffff700 1 -- 172.21.15.102:0/2437427155 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324806c7f0 con 0x7f32481593c0 2023-03-03T09:15:47.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.853+0000 7f32477fe700 1 -- 172.21.15.102:0/2437427155 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115241973 0 0) 0x7f3248153660 con 0x7f32480cd8e0 2023-03-03T09:15:47.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.853+0000 7f323ffff700 1 --2- 172.21.15.102:0/2437427155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32481593c0 0x7f32480cde20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f32300098f0 tx=0x7f3230009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6c3f0e8f910353 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:47.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.853+0000 7f32477fe700 1 -- 172.21.15.102:0/2437427155 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230011830 con 0x7f32481593c0 2023-03-03T09:15:47.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.853+0000 7f32477fe700 1 -- 172.21.15.102:0/2437427155 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3230011990 con 0x7f32481593c0 2023-03-03T09:15:47.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.853+0000 7f32477fe700 1 -- 172.21.15.102:0/2437427155 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230011d20 con 0x7f32481593c0 2023-03-03T09:15:47.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 -- 172.21.15.102:0/2437427155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32481593c0 msgr2=0x7f32480cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:47.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 --2- 172.21.15.102:0/2437427155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32481593c0 0x7f32480cde20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f32300098f0 tx=0x7f3230009c00 comp rx=0 tx=0).stop 2023-03-03T09:15:47.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 -- 172.21.15.102:0/2437427155 shutdown_connections 2023-03-03T09:15:47.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 --2- 172.21.15.102:0/2437427155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3248153290 0x7f3248158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:47.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 --2- 172.21.15.102:0/2437427155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32481593c0 0x7f32480cde20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:47.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 -- 172.21.15.102:0/2437427155 >> 172.21.15.102:0/2437427155 conn(0x7f3248065fb0 msgr2=0x7f32480663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:15:47.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 -- 172.21.15.102:0/2437427155 shutdown_connections 2023-03-03T09:15:47.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.854+0000 7f324e3ff700 1 -- 172.21.15.102:0/2437427155 wait complete. 2023-03-03T09:15:47.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.856+0000 7f324e3ff700 1 Processor -- start 2023-03-03T09:15:47.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.856+0000 7f324e3ff700 1 -- start start 2023-03-03T09:15:47.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f324e3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32480cd8e0 0x7f32480cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:47.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f324e3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3248153290 0x7f3248154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:47.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f324e3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32481593c0 0x7f324814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:47.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f324e3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3248153f50 con 0x7f32480cd8e0 2023-03-03T09:15:47.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f324e3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32481501a0 con 0x7f3248153290 2023-03-03T09:15:47.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f324e3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32480b2740 con 0x7f32481593c0 2023-03-03T09:15:47.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f3247fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32480cd8e0 0x7f32480cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:47.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f3247fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32480cd8e0 0x7f32480cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40686/0 (socket says 172.21.15.102:40686) 2023-03-03T09:15:47.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.857+0000 7f3247fff700 1 -- 172.21.15.102:0/3202887988 learned_addr learned my addr 172.21.15.102:0/3202887988 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:15:47.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f324c99c700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32481593c0 0x7f324814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:47.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f323ffff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3248153290 0x7f3248154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:47.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f3247fff700 1 -- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32481593c0 msgr2=0x7f324814fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:47.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f3247fff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32481593c0 0x7f324814fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:47.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f3247fff700 1 -- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3248153290 msgr2=0x7f3248154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:47.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f3247fff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3248153290 0x7f3248154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:47.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f3247fff700 1 -- 172.21.15.102:0/3202887988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3230009550 con 0x7f32480cd8e0 2023-03-03T09:15:47.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f324c99c700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32481593c0 0x7f324814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:15:47.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f3247fff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32480cd8e0 0x7f32480cad00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f323800e9b0 tx=0x7f323800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:47.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f32457fa700 1 -- 172.21.15.102:0/3202887988 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3238009710 con 0x7f32480cd8e0 2023-03-03T09:15:47.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f32457fa700 1 -- 172.21.15.102:0/3202887988 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323800c850 con 0x7f32480cd8e0 2023-03-03T09:15:47.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f32457fa700 1 -- 172.21.15.102:0/3202887988 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323800cbc0 con 0x7f32480cd8e0 2023-03-03T09:15:47.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.858+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32480ccde0 con 0x7f32480cd8e0 2023-03-03T09:15:47.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.859+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324815d160 con 0x7f32480cd8e0 2023-03-03T09:15:47.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.860+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32480cd0c0 con 0x7f32480cd8e0 2023-03-03T09:15:47.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.861+0000 7f32457fa700 1 -- 172.21.15.102:0/3202887988 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3238019030 con 0x7f32480cd8e0 2023-03-03T09:15:47.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.862+0000 7f32457fa700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32280682f0 0x7f322806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.862+0000 7f32457fa700 1 -- 172.21.15.102:0/3202887988 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3238085d00 con 0x7f32480cd8e0 2023-03-03T09:15:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.863+0000 7f323ffff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32280682f0 0x7f322806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.863+0000 7f323ffff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32280682f0 0x7f322806a7a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3230009130 tx=0x7f3230008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:47.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:47.866+0000 7f32457fa700 1 -- 172.21.15.102:0/3202887988 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3238052410 con 0x7f32480cd8e0 2023-03-03T09:15:48.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.099+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f324804e6e0 con 0x7f32480cd8e0 2023-03-03T09:15:48.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.100+0000 7f32457fa700 1 -- 172.21.15.102:0/3202887988 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3238017090 con 0x7f32480cd8e0 2023-03-03T09:15:48.101 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:15:48.101 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:15:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32280682f0 msgr2=0x7f322806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32280682f0 0x7f322806a7a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3230009130 tx=0x7f3230008030 comp rx=0 tx=0).stop 2023-03-03T09:15:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32480cd8e0 msgr2=0x7f32480cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32480cd8e0 0x7f32480cad00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f323800e9b0 tx=0x7f323800ecc0 comp rx=0 tx=0).stop 2023-03-03T09:15:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 shutdown_connections 2023-03-03T09:15:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32280682f0 0x7f322806a7a0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32481593c0 0x7f324814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32480cd8e0 0x7f32480cad00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 --2- 172.21.15.102:0/3202887988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3248153290 0x7f3248154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.102+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 >> 172.21.15.102:0/3202887988 conn(0x7f3248065fb0 msgr2=0x7f32481e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:15:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.103+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 shutdown_connections 2023-03-03T09:15:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:15:48.103+0000 7f324e3ff700 1 -- 172.21.15.102:0/3202887988 wait complete. 2023-03-03T09:15:48.106 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:15:48.131 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T09:16:16.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:16.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:23.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:16:23.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.381+0000 7f040cbff700 1 Processor -- start 2023-03-03T09:16:23.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.381+0000 7f040cbff700 1 -- start start 2023-03-03T09:16:23.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.382+0000 7f040cbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0408153290 0x7f0408158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:23.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.382+0000 7f040cbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f04080cd8e0 0x7f04080ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:23.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.382+0000 7f040cbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0408153660 con 0x7f04081593c0 2023-03-03T09:16:23.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.382+0000 7f040cbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040806c360 con 0x7f04080cd8e0 2023-03-03T09:16:23.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.382+0000 7f040cbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040815cea0 con 0x7f0408153290 2023-03-03T09:16:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0408153290 0x7f0408158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f0406d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f04080cd8e0 0x7f04080ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0408153290 0x7f0408158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38912/0 (socket says 172.21.15.102:38912) 2023-03-03T09:16:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 -- 172.21.15.102:0/2916382298 learned_addr learned my addr 172.21.15.102:0/2916382298 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:16:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f0406d9d700 1 --2- 172.21.15.102:0/2916382298 >> v2:172.21.15.172:3300/0 conn(0x7f04080cd8e0 0x7f04080ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38766/0 (socket says 172.21.15.102:38766) 2023-03-03T09:16:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 -- 172.21.15.102:0/2916382298 >> v2:172.21.15.172:3300/0 conn(0x7f04080cd8e0 msgr2=0x7f04080ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:16:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 --2- 172.21.15.102:0/2916382298 >> v2:172.21.15.172:3300/0 conn(0x7f04080cd8e0 0x7f04080ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 -- 172.21.15.102:0/2916382298 >> v1:172.21.15.102:6789/0 conn(0x7f04081593c0 legacy=0x7f04080cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 -- 172.21.15.102:0/2916382298 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040806c7f0 con 0x7f0408153290 2023-03-03T09:16:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f0406d9d700 1 --2- 172.21.15.102:0/2916382298 >> v2:172.21.15.172:3300/0 conn(0x7f04080cd8e0 0x7f04080ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:16:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.383+0000 7f040659c700 1 --2- 172.21.15.102:0/2916382298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0408153290 0x7f0408158e80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f03f80098f0 tx=0x7f03f8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b61559978f10b14 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:23.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040559a700 1 -- 172.21.15.102:0/2916382298 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8011830 con 0x7f0408153290 2023-03-03T09:16:23.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040559a700 1 -- 172.21.15.102:0/2916382298 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f8011990 con 0x7f0408153290 2023-03-03T09:16:23.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040559a700 1 -- 172.21.15.102:0/2916382298 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8011d20 con 0x7f0408153290 2023-03-03T09:16:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040cbff700 1 -- 172.21.15.102:0/2916382298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0408153290 msgr2=0x7f0408158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040cbff700 1 --2- 172.21.15.102:0/2916382298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0408153290 0x7f0408158e80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f03f80098f0 tx=0x7f03f8009c00 comp rx=0 tx=0).stop 2023-03-03T09:16:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040cbff700 1 -- 172.21.15.102:0/2916382298 shutdown_connections 2023-03-03T09:16:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040cbff700 1 --2- 172.21.15.102:0/2916382298 >> v2:172.21.15.172:3300/0 conn(0x7f04080cd8e0 0x7f04080ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040cbff700 1 --2- 172.21.15.102:0/2916382298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0408153290 0x7f0408158e80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.384+0000 7f040cbff700 1 -- 172.21.15.102:0/2916382298 >> 172.21.15.102:0/2916382298 conn(0x7f0408065fb0 msgr2=0x7f04080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:16:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.385+0000 7f040cbff700 1 -- 172.21.15.102:0/2916382298 shutdown_connections 2023-03-03T09:16:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.385+0000 7f040cbff700 1 -- 172.21.15.102:0/2916382298 wait complete. 2023-03-03T09:16:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.387+0000 7f040cbff700 1 Processor -- start 2023-03-03T09:16:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.387+0000 7f040cbff700 1 -- start start 2023-03-03T09:16:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040cbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04080cd8e0 0x7f0408151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040cbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0408153290 0x7f04080b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040cbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04081593c0 0x7f040814e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040cbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04080b47f0 con 0x7f04081593c0 2023-03-03T09:16:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040cbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408150bd0 con 0x7f0408153290 2023-03-03T09:16:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040cbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04081503f0 con 0x7f04080cd8e0 2023-03-03T09:16:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f0406d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04081593c0 0x7f040814e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04080cd8e0 0x7f0408151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f0405d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0408153290 0x7f04080b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f0406d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04081593c0 0x7f040814e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45690/0 (socket says 172.21.15.102:45690) 2023-03-03T09:16:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f0406d9d700 1 -- 172.21.15.102:0/3414717140 learned_addr learned my addr 172.21.15.102:0/3414717140 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:16:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.388+0000 7f040659c700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04080cd8e0 0x7f0408151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38920/0 (socket says 172.21.15.102:38920) 2023-03-03T09:16:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f040659c700 1 -- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0408153290 msgr2=0x7f04080b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f040659c700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0408153290 0x7f04080b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f040659c700 1 -- 172.21.15.102:0/3414717140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04081593c0 msgr2=0x7f040814e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f040659c700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04081593c0 0x7f040814e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f040659c700 1 -- 172.21.15.102:0/3414717140 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400009a70 con 0x7f04080cd8e0 2023-03-03T09:16:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f0406d9d700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04081593c0 0x7f040814e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:16:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f040659c700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04080cd8e0 0x7f0408151440 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03f8016000 tx=0x7f03f8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f03f6ffd700 1 -- 172.21.15.102:0/3414717140 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8004480 con 0x7f04080cd8e0 2023-03-03T09:16:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f03f6ffd700 1 -- 172.21.15.102:0/3414717140 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f801f070 con 0x7f04080cd8e0 2023-03-03T09:16:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f03f6ffd700 1 -- 172.21.15.102:0/3414717140 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f80038e0 con 0x7f04080cd8e0 2023-03-03T09:16:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.389+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f8009550 con 0x7f04080cd8e0 2023-03-03T09:16:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.390+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04080ccde0 con 0x7f04080cd8e0 2023-03-03T09:16:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.391+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040815d160 con 0x7f04080cd8e0 2023-03-03T09:16:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.392+0000 7f03f6ffd700 1 -- 172.21.15.102:0/3414717140 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f03f802a020 con 0x7f04080cd8e0 2023-03-03T09:16:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.393+0000 7f03f6ffd700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03e8068320 0x7f03e806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.393+0000 7f03f6ffd700 1 -- 172.21.15.102:0/3414717140 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f03f8087290 con 0x7f04080cd8e0 2023-03-03T09:16:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.393+0000 7f0405d9b700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03e8068320 0x7f03e806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.394+0000 7f0405d9b700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03e8068320 0x7f03e806a7d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0408073fe0 tx=0x7f03fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.398+0000 7f03f6ffd700 1 -- 172.21.15.102:0/3414717140 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f03f80539a0 con 0x7f04080cd8e0 2023-03-03T09:16:23.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.633+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04080cd0c0 con 0x7f04080cd8e0 2023-03-03T09:16:23.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.635+0000 7f03f6ffd700 1 -- 172.21.15.102:0/3414717140 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f03f8056fc0 con 0x7f04080cd8e0 2023-03-03T09:16:23.636 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:16:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:16:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03e8068320 msgr2=0x7f03e806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03e8068320 0x7f03e806a7d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0408073fe0 tx=0x7f03fc009040 comp rx=0 tx=0).stop 2023-03-03T09:16:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04080cd8e0 msgr2=0x7f0408151440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04080cd8e0 0x7f0408151440 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03f8016000 tx=0x7f03f8004750 comp rx=0 tx=0).stop 2023-03-03T09:16:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 shutdown_connections 2023-03-03T09:16:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f03e8068320 0x7f03e806a7d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04080cd8e0 0x7f0408151440 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04081593c0 0x7f040814e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 --2- 172.21.15.102:0/3414717140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0408153290 0x7f04080b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 >> 172.21.15.102:0/3414717140 conn(0x7f0408065fb0 msgr2=0x7f0408073680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:16:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 shutdown_connections 2023-03-03T09:16:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:16:23.637+0000 7f040cbff700 1 -- 172.21.15.102:0/3414717140 wait complete. 2023-03-03T09:16:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:16:23.666 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2023-03-03T09:16:46.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:46.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:16.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:16.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:27.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:17:27.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.931+0000 7f881c7ff700 1 Processor -- start 2023-03-03T09:17:27.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.932+0000 7f881c7ff700 1 -- start start 2023-03-03T09:17:27.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.933+0000 7f881c7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f88140c9cb0 0x7f88140cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:27.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.933+0000 7f881c7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140ca190 0x7f88140cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:27.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.933+0000 7f881c7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140cf940 con 0x7f8814153260 2023-03-03T09:17:27.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.933+0000 7f881c7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88140cfe70 con 0x7f88140c9cb0 2023-03-03T09:17:27.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.933+0000 7f881c7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8814158d70 con 0x7f88140ca190 2023-03-03T09:17:27.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140ca190 0x7f88140cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:27.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f8819d9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f88140c9cb0 0x7f88140cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:27.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140ca190 0x7f88140cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60144/0 (socket says 172.21.15.102:60144) 2023-03-03T09:17:27.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 -- 172.21.15.102:0/255275798 learned_addr learned my addr 172.21.15.102:0/255275798 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:17:27.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881a59b700 1 --1- 172.21.15.102:0/255275798 >> v1:172.21.15.172:6789/0 conn(0x7f8814153260 0x7f88140ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52386/0 (socket says 172.21.15.102:52386) 2023-03-03T09:17:27.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 -- 172.21.15.102:0/255275798 >> v2:172.21.15.172:3300/0 conn(0x7f88140c9cb0 msgr2=0x7f88140cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:17:27.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 --2- 172.21.15.102:0/255275798 >> v2:172.21.15.172:3300/0 conn(0x7f88140c9cb0 0x7f88140cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:27.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 -- 172.21.15.102:0/255275798 >> v1:172.21.15.172:6789/0 conn(0x7f8814153260 legacy=0x7f88140ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:27.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 -- 172.21.15.102:0/255275798 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140cfb90 con 0x7f88140ca190 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.934+0000 7f881ad9c700 1 --2- 172.21.15.102:0/255275798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140ca190 0x7f88140cbdc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f880c009e50 tx=0x7f880c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=188b3233d2ed79dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f8819599700 1 -- 172.21.15.102:0/255275798 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c00e070 con 0x7f88140ca190 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f8819599700 1 -- 172.21.15.102:0/255275798 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880c00b910 con 0x7f88140ca190 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f8819599700 1 -- 172.21.15.102:0/255275798 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c00bcc0 con 0x7f88140ca190 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f881c7ff700 1 -- 172.21.15.102:0/255275798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140ca190 msgr2=0x7f88140cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f881c7ff700 1 --2- 172.21.15.102:0/255275798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140ca190 0x7f88140cbdc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f880c009e50 tx=0x7f880c00b330 comp rx=0 tx=0).stop 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f881c7ff700 1 -- 172.21.15.102:0/255275798 shutdown_connections 2023-03-03T09:17:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f881c7ff700 1 --2- 172.21.15.102:0/255275798 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140ca190 0x7f88140cbdc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:27.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f881c7ff700 1 --2- 172.21.15.102:0/255275798 >> v2:172.21.15.172:3300/0 conn(0x7f88140c9cb0 0x7f88140cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:27.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f881c7ff700 1 -- 172.21.15.102:0/255275798 >> 172.21.15.102:0/255275798 conn(0x7f8814066c00 msgr2=0x7f88140ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:27.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.935+0000 7f881c7ff700 1 -- 172.21.15.102:0/255275798 shutdown_connections 2023-03-03T09:17:27.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.936+0000 7f881c7ff700 1 -- 172.21.15.102:0/255275798 wait complete. 2023-03-03T09:17:27.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.937+0000 7f881c7ff700 1 Processor -- start 2023-03-03T09:17:27.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.938+0000 7f881c7ff700 1 -- start start 2023-03-03T09:17:27.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881c7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88140ca190 0x7f8814150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:27.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881c7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8814153260 0x7f88140c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:27.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881c7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140c6180 0x7f881414e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:27.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881c7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88140c6790 con 0x7f88140ca190 2023-03-03T09:17:27.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881c7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88141ee730 con 0x7f8814153260 2023-03-03T09:17:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881c7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881415a960 con 0x7f88140c6180 2023-03-03T09:17:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881a59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88140ca190 0x7f8814150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881ad9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140c6180 0x7f881414e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f8819d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8814153260 0x7f88140c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881a59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88140ca190 0x7f8814150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52808/0 (socket says 172.21.15.102:52808) 2023-03-03T09:17:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f8819d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8814153260 0x7f88140c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:41476/0 (socket says 172.21.15.102:41476) 2023-03-03T09:17:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881a59b700 1 -- 172.21.15.102:0/86610165 learned_addr learned my addr 172.21.15.102:0/86610165 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.939+0000 7f881ad9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140c6180 0x7f881414e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60146/0 (socket says 172.21.15.102:60146) 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f8819d9a700 1 -- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140c6180 msgr2=0x7f881414e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f8819d9a700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140c6180 0x7f881414e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f8819d9a700 1 -- 172.21.15.102:0/86610165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88140ca190 msgr2=0x7f8814150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f8819d9a700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88140ca190 0x7f8814150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f8819d9a700 1 -- 172.21.15.102:0/86610165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880c009b30 con 0x7f8814153260 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f881ad9c700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140c6180 0x7f881414e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:17:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f8819d9a700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8814153260 0x7f88140c7b70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f880400e930 tx=0x7f880400ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:27.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f880affd700 1 -- 172.21.15.102:0/86610165 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804013680 con 0x7f8814153260 2023-03-03T09:17:27.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f880affd700 1 -- 172.21.15.102:0/86610165 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88040137e0 con 0x7f8814153260 2023-03-03T09:17:27.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f880affd700 1 -- 172.21.15.102:0/86610165 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804013b50 con 0x7f8814153260 2023-03-03T09:17:27.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.940+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8814159260 con 0x7f8814153260 2023-03-03T09:17:27.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.941+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8814158f80 con 0x7f8814153260 2023-03-03T09:17:27.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.941+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8814159540 con 0x7f8814153260 2023-03-03T09:17:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.943+0000 7f880affd700 1 -- 172.21.15.102:0/86610165 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8804019030 con 0x7f8814153260 2023-03-03T09:17:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.944+0000 7f880affd700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f87f80682f0 0x7f87f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.944+0000 7f880affd700 1 -- 172.21.15.102:0/86610165 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8804085b60 con 0x7f8814153260 2023-03-03T09:17:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.944+0000 7f881a59b700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f87f80682f0 0x7f87f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.945+0000 7f881a59b700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f87f80682f0 0x7f87f806a7a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8810000f60 tx=0x7f8810007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:27.947+0000 7f880affd700 1 -- 172.21.15.102:0/86610165 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f88040522f0 con 0x7f8814153260 2023-03-03T09:17:28.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.181+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f881404e6e0 con 0x7f8814153260 2023-03-03T09:17:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.183+0000 7f880affd700 1 -- 172.21.15.102:0/86610165 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8804017090 con 0x7f8814153260 2023-03-03T09:17:28.184 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:17:28.184 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:17:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f87f80682f0 msgr2=0x7f87f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f87f80682f0 0x7f87f806a7a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8810000f60 tx=0x7f8810007f90 comp rx=0 tx=0).stop 2023-03-03T09:17:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8814153260 msgr2=0x7f88140c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8814153260 0x7f88140c7b70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f880400e930 tx=0x7f880400ec40 comp rx=0 tx=0).stop 2023-03-03T09:17:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 shutdown_connections 2023-03-03T09:17:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f87f80682f0 0x7f87f806a7a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f88140c6180 0x7f881414e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88140ca190 0x7f8814150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 --2- 172.21.15.102:0/86610165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8814153260 0x7f88140c7b70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 >> 172.21.15.102:0/86610165 conn(0x7f88140c9cb0 msgr2=0x7f88140b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 shutdown_connections 2023-03-03T09:17:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:28.185+0000 7f881c7ff700 1 -- 172.21.15.102:0/86610165 wait complete. 2023-03-03T09:17:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:17:28.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-03-03T09:17:33.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:17:33.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.463+0000 7f096e19d700 1 Processor -- start 2023-03-03T09:17:33.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.463+0000 7f096e19d700 1 -- start start 2023-03-03T09:17:33.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.465+0000 7f096e19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f09680ced00 0x7f09680c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.465+0000 7f096e19d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680c3d20 con 0x7f09680c41b0 2023-03-03T09:17:33.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.465+0000 7f096e19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096815cba0 con 0x7f09680b3970 2023-03-03T09:17:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.465+0000 7f096e19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09680c4930 con 0x7f09680ced00 2023-03-03T09:17:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.465+0000 7f096d19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f09680ced00 0x7f09680c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.465+0000 7f096d99c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f09680c41b0 0x7f0968163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39162/0 (socket says 172.21.15.102:39162) 2023-03-03T09:17:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.465+0000 7f096d99c700 1 -- 172.21.15.102:0/1366597161 learned_addr learned my addr 172.21.15.102:0/1366597161 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:17:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.466+0000 7f095ffff700 1 -- 172.21.15.102:0/1366597161 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414290833 0 0) 0x7f09680c3d20 con 0x7f09680c41b0 2023-03-03T09:17:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.466+0000 7f095ffff700 1 -- 172.21.15.102:0/1366597161 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09540035b0 con 0x7f09680c41b0 2023-03-03T09:17:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.466+0000 7f095ffff700 1 -- 172.21.15.102:0/1366597161 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105948008 0 0) 0x7f096815cba0 con 0x7f09680b3970 2023-03-03T09:17:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.466+0000 7f096d19b700 1 -- 172.21.15.102:0/1366597161 >> v1:172.21.15.172:6789/0 conn(0x7f09680b3970 legacy=0x7f0968158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.466+0000 7f096d19b700 1 -- 172.21.15.102:0/1366597161 >> v1:172.21.15.102:6789/0 conn(0x7f09680c41b0 legacy=0x7f0968163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.466+0000 7f096d19b700 1 -- 172.21.15.102:0/1366597161 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096815ce80 con 0x7f09680ced00 2023-03-03T09:17:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.466+0000 7f096d19b700 1 --2- 172.21.15.102:0/1366597161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680ced00 0x7f09680c5a90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0964009840 tx=0x7f0964009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f2eda7c7d61324 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f095ffff700 1 -- 172.21.15.102:0/1366597161 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964011830 con 0x7f09680ced00 2023-03-03T09:17:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f095ffff700 1 -- 172.21.15.102:0/1366597161 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0964011990 con 0x7f09680ced00 2023-03-03T09:17:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f095ffff700 1 -- 172.21.15.102:0/1366597161 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964011d20 con 0x7f09680ced00 2023-03-03T09:17:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f096e19d700 1 -- 172.21.15.102:0/1366597161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680ced00 msgr2=0x7f09680c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f096e19d700 1 --2- 172.21.15.102:0/1366597161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680ced00 0x7f09680c5a90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0964009840 tx=0x7f0964009b50 comp rx=0 tx=0).stop 2023-03-03T09:17:33.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f096e19d700 1 -- 172.21.15.102:0/1366597161 shutdown_connections 2023-03-03T09:17:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f096e19d700 1 --2- 172.21.15.102:0/1366597161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680ced00 0x7f09680c5a90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f096e19d700 1 -- 172.21.15.102:0/1366597161 >> 172.21.15.102:0/1366597161 conn(0x7f096808b1c0 msgr2=0x7f0968071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.467+0000 7f096e19d700 1 -- 172.21.15.102:0/1366597161 shutdown_connections 2023-03-03T09:17:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.468+0000 7f096e19d700 1 -- 172.21.15.102:0/1366597161 wait complete. 2023-03-03T09:17:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.469+0000 7f096e19d700 1 Processor -- start 2023-03-03T09:17:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.470+0000 7f096e19d700 1 -- start start 2023-03-03T09:17:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096e19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09680b3970 0x7f09681509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096e19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680c41b0 0x7f096815aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096e19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09680ced00 0x7f0968151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096e19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096815b420 con 0x7f09680c41b0 2023-03-03T09:17:33.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096e19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096815a560 con 0x7f09680b3970 2023-03-03T09:17:33.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096e19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0968151060 con 0x7f09680ced00 2023-03-03T09:17:33.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096c99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680c41b0 0x7f096815aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096d19b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09680b3970 0x7f09681509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096d19b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09680b3970 0x7f09681509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:41486/0 (socket says 172.21.15.102:41486) 2023-03-03T09:17:33.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096c99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680c41b0 0x7f096815aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52830/0 (socket says 172.21.15.102:52830) 2023-03-03T09:17:33.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096d19b700 1 -- 172.21.15.102:0/1271674266 learned_addr learned my addr 172.21.15.102:0/1271674266 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:17:33.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096d99c700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09680ced00 0x7f0968151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096c99a700 1 -- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09680ced00 msgr2=0x7f0968151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.471+0000 7f096c99a700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09680ced00 0x7f0968151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f096c99a700 1 -- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09680b3970 msgr2=0x7f09681509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f096c99a700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09680b3970 0x7f09681509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f096c99a700 1 -- 172.21.15.102:0/1271674266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09640094a0 con 0x7f09680c41b0 2023-03-03T09:17:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f096d19b700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09680b3970 0x7f09681509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:17:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f096c99a700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680c41b0 0x7f096815aee0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f096808c270 tx=0x7f095800c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f095dffb700 1 -- 172.21.15.102:0/1271674266 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580052d0 con 0x7f09680c41b0 2023-03-03T09:17:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f095dffb700 1 -- 172.21.15.102:0/1271674266 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0958005430 con 0x7f09680c41b0 2023-03-03T09:17:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096806c200 con 0x7f09680c41b0 2023-03-03T09:17:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f095dffb700 1 -- 172.21.15.102:0/1271674266 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580038a0 con 0x7f09680c41b0 2023-03-03T09:17:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.472+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096806bf20 con 0x7f09680c41b0 2023-03-03T09:17:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.473+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096815c890 con 0x7f09680c41b0 2023-03-03T09:17:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.475+0000 7f095dffb700 1 -- 172.21.15.102:0/1271674266 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0958019030 con 0x7f09680c41b0 2023-03-03T09:17:33.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.476+0000 7f095dffb700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f094c068320 0x7f094c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.476+0000 7f095dffb700 1 -- 172.21.15.102:0/1271674266 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0958084f20 con 0x7f09680c41b0 2023-03-03T09:17:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.476+0000 7f096d19b700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f094c068320 0x7f094c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.477+0000 7f096d19b700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f094c068320 0x7f094c06a7d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0964000c00 tx=0x7f096401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:33.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.480+0000 7f095dffb700 1 -- 172.21.15.102:0/1271674266 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0958051630 con 0x7f09680c41b0 2023-03-03T09:17:33.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.714+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f096804e6e0 con 0x7f09680c41b0 2023-03-03T09:17:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.715+0000 7f095dffb700 1 -- 172.21.15.102:0/1271674266 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0958017020 con 0x7f09680c41b0 2023-03-03T09:17:33.716 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:17:33.716 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:17:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f094c068320 msgr2=0x7f094c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f094c068320 0x7f094c06a7d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0964000c00 tx=0x7f096401c040 comp rx=0 tx=0).stop 2023-03-03T09:17:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680c41b0 msgr2=0x7f096815aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680c41b0 0x7f096815aee0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f096808c270 tx=0x7f095800c6c0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 shutdown_connections 2023-03-03T09:17:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f094c068320 0x7f094c06a7d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f09680ced00 0x7f0968151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09680b3970 0x7f09681509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 --2- 172.21.15.102:0/1271674266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09680c41b0 0x7f096815aee0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.717+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 >> 172.21.15.102:0/1271674266 conn(0x7f096808b1c0 msgr2=0x7f0968071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.718+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 shutdown_connections 2023-03-03T09:17:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:17:33.718+0000 7f096e19d700 1 -- 172.21.15.102:0/1271674266 wait complete. 2023-03-03T09:17:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:17:33.745 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T09:17:47.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:47.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:17.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:17.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:42.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:18:42.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.993+0000 7f90f75ff700 1 Processor -- start 2023-03-03T09:18:42.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.993+0000 7f90f75ff700 1 -- start start 2023-03-03T09:18:42.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.994+0000 7f90f75ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90f0153290 0x7f90f0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:42.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.994+0000 7f90f75ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90f01593c0 0x7f90f00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:42.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.994+0000 7f90f75ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:42.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.994+0000 7f90f75ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f0153660 con 0x7f90f0153290 2023-03-03T09:18:42.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.994+0000 7f90f75ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f006c360 con 0x7f90f01593c0 2023-03-03T09:18:42.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.994+0000 7f90f75ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f015cea0 con 0x7f90f00cd8e0 2023-03-03T09:18:42.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f539b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90f0153290 0x7f90f0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:42.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:42.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f4b9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90f01593c0 0x7f90f00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:42.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45152/0 (socket says 172.21.15.102:45152) 2023-03-03T09:18:42.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 -- 172.21.15.102:0/2160843542 learned_addr learned my addr 172.21.15.102:0/2160843542 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:18:42.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 -- 172.21.15.102:0/2160843542 >> v2:172.21.15.172:3300/0 conn(0x7f90f01593c0 msgr2=0x7f90f00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:18:42.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 --2- 172.21.15.102:0/2160843542 >> v2:172.21.15.172:3300/0 conn(0x7f90f01593c0 0x7f90f00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:42.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 -- 172.21.15.102:0/2160843542 >> v2:172.21.15.102:3300/0 conn(0x7f90f0153290 msgr2=0x7f90f0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:18:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 --2- 172.21.15.102:0/2160843542 >> v2:172.21.15.102:3300/0 conn(0x7f90f0153290 0x7f90f0158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 -- 172.21.15.102:0/2160843542 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f006c7f0 con 0x7f90f00cd8e0 2023-03-03T09:18:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.995+0000 7f90f5b9c700 1 --2- 172.21.15.102:0/2160843542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00ce360 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f90f0159fb0 tx=0x7f90e800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64a851553a8477c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90e7fff700 1 -- 172.21.15.102:0/2160843542 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800e070 con 0x7f90f00cd8e0 2023-03-03T09:18:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90e7fff700 1 -- 172.21.15.102:0/2160843542 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e800b910 con 0x7f90f00cd8e0 2023-03-03T09:18:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90e7fff700 1 -- 172.21.15.102:0/2160843542 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800bcc0 con 0x7f90f00cd8e0 2023-03-03T09:18:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90f75ff700 1 -- 172.21.15.102:0/2160843542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 msgr2=0x7f90f00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:43.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90f75ff700 1 --2- 172.21.15.102:0/2160843542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00ce360 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f90f0159fb0 tx=0x7f90e800b330 comp rx=0 tx=0).stop 2023-03-03T09:18:43.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90f75ff700 1 -- 172.21.15.102:0/2160843542 shutdown_connections 2023-03-03T09:18:43.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90f75ff700 1 --2- 172.21.15.102:0/2160843542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00ce360 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90f75ff700 1 --2- 172.21.15.102:0/2160843542 >> v2:172.21.15.102:3300/0 conn(0x7f90f0153290 0x7f90f0158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90f75ff700 1 --2- 172.21.15.102:0/2160843542 >> v2:172.21.15.172:3300/0 conn(0x7f90f01593c0 0x7f90f00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.996+0000 7f90f75ff700 1 -- 172.21.15.102:0/2160843542 >> 172.21.15.102:0/2160843542 conn(0x7f90f0065fb0 msgr2=0x7f90f00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:18:43.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.997+0000 7f90f75ff700 1 -- 172.21.15.102:0/2160843542 shutdown_connections 2023-03-03T09:18:43.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.997+0000 7f90f75ff700 1 -- 172.21.15.102:0/2160843542 wait complete. 2023-03-03T09:18:43.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.999+0000 7f90f75ff700 1 Processor -- start 2023-03-03T09:18:43.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:42.999+0000 7f90f75ff700 1 -- start start 2023-03-03T09:18:43.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f75ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:43.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f75ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f0153290 0x7f90f00c2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:43.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f75ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f01593c0 0x7f90f00c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:43.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f75ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01ef3b0 con 0x7f90f01593c0 2023-03-03T09:18:43.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f75ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01ee820 con 0x7f90f0153290 2023-03-03T09:18:43.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f75ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01ef7d0 con 0x7f90f00cd8e0 2023-03-03T09:18:43.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f5b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f01593c0 0x7f90f00c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:43.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f5b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f01593c0 0x7f90f00c2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54328/0 (socket says 172.21.15.102:54328) 2023-03-03T09:18:43.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f4b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f0153290 0x7f90f00c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:43.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f5b9c700 1 -- 172.21.15.102:0/975619585 learned_addr learned my addr 172.21.15.102:0/975619585 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:18:43.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f539b700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:43.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f5b9c700 1 -- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 msgr2=0x7f90f00c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:43.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.000+0000 7f90f5b9c700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90f5b9c700 1 -- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f0153290 msgr2=0x7f90f00c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:43.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90f5b9c700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f0153290 0x7f90f00c2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90f5b9c700 1 -- 172.21.15.102:0/975619585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e8009b30 con 0x7f90f01593c0 2023-03-03T09:18:43.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90f5b9c700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f01593c0 0x7f90f00c2ca0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f90e800b680 tx=0x7f90e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:43.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90e5ffb700 1 -- 172.21.15.102:0/975619585 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e80099b0 con 0x7f90f01593c0 2023-03-03T09:18:43.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90e5ffb700 1 -- 172.21.15.102:0/975619585 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e800ce80 con 0x7f90f01593c0 2023-03-03T09:18:43.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90e5ffb700 1 -- 172.21.15.102:0/975619585 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8004490 con 0x7f90f01593c0 2023-03-03T09:18:43.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f00ccde0 con 0x7f90f01593c0 2023-03-03T09:18:43.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.001+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f015d160 con 0x7f90f01593c0 2023-03-03T09:18:43.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.002+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f00cd0c0 con 0x7f90f01593c0 2023-03-03T09:18:43.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.004+0000 7f90e5ffb700 1 -- 172.21.15.102:0/975619585 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90e802b070 con 0x7f90f01593c0 2023-03-03T09:18:43.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.005+0000 7f90e5ffb700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d40682f0 0x7f90d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:43.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.005+0000 7f90e5ffb700 1 -- 172.21.15.102:0/975619585 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f90e8086d90 con 0x7f90f01593c0 2023-03-03T09:18:43.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.005+0000 7f90f539b700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d40682f0 0x7f90d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:43.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.006+0000 7f90f539b700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d40682f0 0x7f90d406a7a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009b40 tx=0x7f90ec009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:43.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.009+0000 7f90e5ffb700 1 -- 172.21.15.102:0/975619585 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90e8053530 con 0x7f90f01593c0 2023-03-03T09:18:43.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.241+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f00c31e0 con 0x7f90f01593c0 2023-03-03T09:18:43.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.241+0000 7f90e5ffb700 1 -- 172.21.15.102:0/975619585 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f90f00c31e0 con 0x7f90f01593c0 2023-03-03T09:18:43.246 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:18:43.246 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:18:43.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d40682f0 msgr2=0x7f90d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:43.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d40682f0 0x7f90d406a7a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009b40 tx=0x7f90ec009210 comp rx=0 tx=0).stop 2023-03-03T09:18:43.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f01593c0 msgr2=0x7f90f00c2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f01593c0 0x7f90f00c2ca0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f90e800b680 tx=0x7f90e8004ad0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 shutdown_connections 2023-03-03T09:18:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d40682f0 0x7f90d406a7a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f00cd8e0 0x7f90f00c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f01593c0 0x7f90f00c2ca0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 --2- 172.21.15.102:0/975619585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f0153290 0x7f90f00c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:43.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 >> 172.21.15.102:0/975619585 conn(0x7f90f0065fb0 msgr2=0x7f90f00cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:18:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 shutdown_connections 2023-03-03T09:18:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:18:43.244+0000 7f90f75ff700 1 -- 172.21.15.102:0/975619585 wait complete. 2023-03-03T09:18:43.249 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:18:43.273 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-03-03T09:18:47.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:47.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:17.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:17.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:47.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:47.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T09:20:00.000+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:20:06.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:20:06.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.527+0000 7f501e1ff700 1 Processor -- start 2023-03-03T09:20:06.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.528+0000 7f501e1ff700 1 -- start start 2023-03-03T09:20:06.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.529+0000 7f501e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f5018158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:06.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.529+0000 7f501e1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f50181593c0 0x7f50180cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:06.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.529+0000 7f501e1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5018153660 con 0x7f50180cd8e0 2023-03-03T09:20:06.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.529+0000 7f501e1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501806c360 con 0x7f50181593c0 2023-03-03T09:20:06.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.529+0000 7f501e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501815cea0 con 0x7f5018153290 2023-03-03T09:20:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5016ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f50181593c0 0x7f50180cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f50177fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f5018158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5017fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f50180cd8e0 0x7f50180ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60708/0 (socket says 172.21.15.102:60708) 2023-03-03T09:20:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5016ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f50181593c0 0x7f50180cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40650/0 (socket says 172.21.15.102:40650) 2023-03-03T09:20:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5017fff700 1 -- 172.21.15.102:0/485676875 learned_addr learned my addr 172.21.15.102:0/485676875 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:20:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5016ffd700 1 -- 172.21.15.102:0/485676875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 msgr2=0x7f5018158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:20:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5016ffd700 1 --2- 172.21.15.102:0/485676875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f5018158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5016ffd700 1 -- 172.21.15.102:0/485676875 >> v1:172.21.15.172:6789/0 conn(0x7f50180cd8e0 legacy=0x7f50180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5016ffd700 1 -- 172.21.15.102:0/485676875 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501806c7f0 con 0x7f50181593c0 2023-03-03T09:20:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.530+0000 7f5016ffd700 1 --2- 172.21.15.102:0/485676875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50181593c0 0x7f50180cde20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f500c009840 tx=0x7f500c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e525b526089c8c2b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:06.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.533+0000 7f50167fc700 1 -- 172.21.15.102:0/485676875 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c011830 con 0x7f50181593c0 2023-03-03T09:20:06.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.533+0000 7f50167fc700 1 -- 172.21.15.102:0/485676875 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c011990 con 0x7f50181593c0 2023-03-03T09:20:06.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.533+0000 7f50167fc700 1 -- 172.21.15.102:0/485676875 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c011d20 con 0x7f50181593c0 2023-03-03T09:20:06.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.533+0000 7f501e1ff700 1 -- 172.21.15.102:0/485676875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50181593c0 msgr2=0x7f50180cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:06.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.533+0000 7f501e1ff700 1 --2- 172.21.15.102:0/485676875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50181593c0 0x7f50180cde20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f500c009840 tx=0x7f500c009b50 comp rx=0 tx=0).stop 2023-03-03T09:20:06.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.534+0000 7f501e1ff700 1 -- 172.21.15.102:0/485676875 shutdown_connections 2023-03-03T09:20:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.534+0000 7f501e1ff700 1 --2- 172.21.15.102:0/485676875 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f5018158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.534+0000 7f501e1ff700 1 --2- 172.21.15.102:0/485676875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50181593c0 0x7f50180cde20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.534+0000 7f501e1ff700 1 -- 172.21.15.102:0/485676875 >> 172.21.15.102:0/485676875 conn(0x7f5018065fb0 msgr2=0x7f50180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.534+0000 7f501e1ff700 1 -- 172.21.15.102:0/485676875 shutdown_connections 2023-03-03T09:20:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.534+0000 7f501e1ff700 1 -- 172.21.15.102:0/485676875 wait complete. 2023-03-03T09:20:06.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.537+0000 7f501e1ff700 1 Processor -- start 2023-03-03T09:20:06.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.537+0000 7f501e1ff700 1 -- start start 2023-03-03T09:20:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.538+0000 7f501e1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50180cd8e0 0x7f50180b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.538+0000 7f501e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f50180b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.538+0000 7f501e1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50181593c0 0x7f50180b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.538+0000 7f501e1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181ee500 con 0x7f50181593c0 2023-03-03T09:20:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.538+0000 7f501e1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50180b4750 con 0x7f50180cd8e0 2023-03-03T09:20:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.538+0000 7f501e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50180b4a10 con 0x7f5018153290 2023-03-03T09:20:06.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f50177fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50180cd8e0 0x7f50180b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f50180b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f50177fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50180cd8e0 0x7f50180b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40658/0 (socket says 172.21.15.102:40658) 2023-03-03T09:20:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f50177fe700 1 -- 172.21.15.102:0/1849942395 learned_addr learned my addr 172.21.15.102:0/1849942395 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:20:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f5017fff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50181593c0 0x7f50180b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f5016ffd700 1 -- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50180cd8e0 msgr2=0x7f50180b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f5016ffd700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50180cd8e0 0x7f50180b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f5016ffd700 1 -- 172.21.15.102:0/1849942395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50181593c0 msgr2=0x7f50180b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:06.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f5016ffd700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50181593c0 0x7f50180b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f5016ffd700 1 -- 172.21.15.102:0/1849942395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0094a0 con 0x7f5018153290 2023-03-03T09:20:06.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.539+0000 7f50177fe700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50180cd8e0 0x7f50180b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:20:06.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.540+0000 7f5016ffd700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f50180b1c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f500c016000 tx=0x7f500c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:06.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.540+0000 7f4ffffff700 1 -- 172.21.15.102:0/1849942395 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c0036a0 con 0x7f5018153290 2023-03-03T09:20:06.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.540+0000 7f4ffffff700 1 -- 172.21.15.102:0/1849942395 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c01f070 con 0x7f5018153290 2023-03-03T09:20:06.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.540+0000 7f4ffffff700 1 -- 172.21.15.102:0/1849942395 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c003a40 con 0x7f5018153290 2023-03-03T09:20:06.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.540+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50180ccde0 con 0x7f5018153290 2023-03-03T09:20:06.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.540+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501815d160 con 0x7f5018153290 2023-03-03T09:20:06.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.541+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50180cd0c0 con 0x7f5018153290 2023-03-03T09:20:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.542+0000 7f4ffffff700 1 -- 172.21.15.102:0/1849942395 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f500c00f050 con 0x7f5018153290 2023-03-03T09:20:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.543+0000 7f4ffffff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5004068320 0x7f500406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.543+0000 7f4ffffff700 1 -- 172.21.15.102:0/1849942395 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f500c021070 con 0x7f5018153290 2023-03-03T09:20:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.544+0000 7f50177fe700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5004068320 0x7f500406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.544+0000 7f50177fe700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5004068320 0x7f500406a7d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f50080113d0 tx=0x7f5008009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:06.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.548+0000 7f4ffffff700 1 -- 172.21.15.102:0/1849942395 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f500c053cd0 con 0x7f5018153290 2023-03-03T09:20:06.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.784+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f501804e6e0 con 0x7f5018153290 2023-03-03T09:20:06.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.785+0000 7f4ffffff700 1 -- 172.21.15.102:0/1849942395 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f500c02b090 con 0x7f5018153290 2023-03-03T09:20:06.788 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:20:06.788 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:20:06.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5004068320 msgr2=0x7f500406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:06.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5004068320 0x7f500406a7d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f50080113d0 tx=0x7f5008009210 comp rx=0 tx=0).stop 2023-03-03T09:20:06.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 msgr2=0x7f50180b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:06.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f50180b1c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f500c016000 tx=0x7f500c004750 comp rx=0 tx=0).stop 2023-03-03T09:20:06.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 shutdown_connections 2023-03-03T09:20:06.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50181593c0 0x7f50180b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50180cd8e0 0x7f50180b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5004068320 0x7f500406a7d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 --2- 172.21.15.102:0/1849942395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5018153290 0x7f50180b1c70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:06.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 >> 172.21.15.102:0/1849942395 conn(0x7f5018065fb0 msgr2=0x7f501815c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:06.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 shutdown_connections 2023-03-03T09:20:06.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:06.788+0000 7f501e1ff700 1 -- 172.21.15.102:0/1849942395 wait complete. 2023-03-03T09:20:06.793 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:20:06.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 15 secs before thrashing 2023-03-03T09:20:17.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:17.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:21.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:20:22.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.068+0000 7f97129ff700 1 Processor -- start 2023-03-03T09:20:22.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.068+0000 7f97129ff700 1 -- start start 2023-03-03T09:20:22.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.069+0000 7f97129ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f970c0cd8e0 0x7f970c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:22.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.069+0000 7f97129ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c153660 con 0x7f970c153290 2023-03-03T09:20:22.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.069+0000 7f97129ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c06c360 con 0x7f970c0cd390 2023-03-03T09:20:22.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.069+0000 7f97129ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970c15cea0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9703fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f970c0cd8e0 0x7f970c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9703fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f970c0cd8e0 0x7f970c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41710/0 (socket says 172.21.15.102:41710) 2023-03-03T09:20:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9703fff700 1 -- 172.21.15.102:0/2637118394 learned_addr learned my addr 172.21.15.102:0/2637118394 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:20:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9710f9c700 1 --1- 172.21.15.102:0/2637118394 >> v1:172.21.15.172:6789/0 conn(0x7f970c0cd390 0x7f970c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50302/0 (socket says 172.21.15.102:50302) 2023-03-03T09:20:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9703fff700 1 -- 172.21.15.102:0/2637118394 >> v1:172.21.15.172:6789/0 conn(0x7f970c0cd390 legacy=0x7f970c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:22.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9703fff700 1 -- 172.21.15.102:0/2637118394 >> v1:172.21.15.102:6789/0 conn(0x7f970c153290 legacy=0x7f970c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9703fff700 1 -- 172.21.15.102:0/2637118394 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c06c7f0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.070+0000 7f9703fff700 1 --2- 172.21.15.102:0/2637118394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c0cd8e0 0x7f970c0cde20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f96f40098f0 tx=0x7f96f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9686edf28ef888b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f970b7fe700 1 -- 172.21.15.102:0/2637118394 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f400b8a0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f970b7fe700 1 -- 172.21.15.102:0/2637118394 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f400ba00 con 0x7f970c0cd8e0 2023-03-03T09:20:22.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f970b7fe700 1 -- 172.21.15.102:0/2637118394 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f400bd90 con 0x7f970c0cd8e0 2023-03-03T09:20:22.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f97129ff700 1 -- 172.21.15.102:0/2637118394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c0cd8e0 msgr2=0x7f970c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:22.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f97129ff700 1 --2- 172.21.15.102:0/2637118394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c0cd8e0 0x7f970c0cde20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f96f40098f0 tx=0x7f96f4009c00 comp rx=0 tx=0).stop 2023-03-03T09:20:22.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f97129ff700 1 -- 172.21.15.102:0/2637118394 shutdown_connections 2023-03-03T09:20:22.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f97129ff700 1 --2- 172.21.15.102:0/2637118394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c0cd8e0 0x7f970c0cde20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:22.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.071+0000 7f97129ff700 1 -- 172.21.15.102:0/2637118394 >> 172.21.15.102:0/2637118394 conn(0x7f970c065fb0 msgr2=0x7f970c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:22.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.072+0000 7f97129ff700 1 -- 172.21.15.102:0/2637118394 shutdown_connections 2023-03-03T09:20:22.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.072+0000 7f97129ff700 1 -- 172.21.15.102:0/2637118394 wait complete. 2023-03-03T09:20:22.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.074+0000 7f97129ff700 1 Processor -- start 2023-03-03T09:20:22.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.074+0000 7f97129ff700 1 -- start start 2023-03-03T09:20:22.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f97129ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f970c0cd390 0x7f970c1e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:22.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f97129ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f970c0cd8e0 0x7f970c0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:22.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f97129ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c153290 0x7f970c0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f97129ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c1efd00 con 0x7f970c153290 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f97129ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c1ef7a0 con 0x7f970c0cd390 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f97129ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c1efa20 con 0x7f970c0cd8e0 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f970bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f970c0cd390 0x7f970c1e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f9703fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f970c0cd8e0 0x7f970c0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f970bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f970c0cd390 0x7f970c1e2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54536/0 (socket says 172.21.15.102:54536) 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f970bfff700 1 -- 172.21.15.102:0/3120637419 learned_addr learned my addr 172.21.15.102:0/3120637419 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:20:22.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.075+0000 7f9710f9c700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c153290 0x7f970c0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:22.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f9703fff700 1 -- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f970c0cd390 msgr2=0x7f970c1e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f9703fff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f970c0cd390 0x7f970c1e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f9703fff700 1 -- 172.21.15.102:0/3120637419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c153290 msgr2=0x7f970c0c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f9703fff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c153290 0x7f970c0c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f9703fff700 1 -- 172.21.15.102:0/3120637419 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f4009550 con 0x7f970c0cd8e0 2023-03-03T09:20:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f970bfff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f970c0cd390 0x7f970c1e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:20:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f9703fff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f970c0cd8e0 0x7f970c0c7880 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f96f4016000 tx=0x7f96f4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f97097fa700 1 -- 172.21.15.102:0/3120637419 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4004380 con 0x7f970c0cd8e0 2023-03-03T09:20:22.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f97097fa700 1 -- 172.21.15.102:0/3120637419 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f401f070 con 0x7f970c0cd8e0 2023-03-03T09:20:22.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f97097fa700 1 -- 172.21.15.102:0/3120637419 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f40036a0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.076+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c0ccde0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.077+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970c15d160 con 0x7f970c0cd8e0 2023-03-03T09:20:22.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.077+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970c0cd0c0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.079+0000 7f97097fa700 1 -- 172.21.15.102:0/3120637419 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f96f402a020 con 0x7f970c0cd8e0 2023-03-03T09:20:22.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.080+0000 7f97097fa700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96ec0682f0 0x7f96ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:22.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.080+0000 7f97097fa700 1 -- 172.21.15.102:0/3120637419 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f96f4087330 con 0x7f970c0cd8e0 2023-03-03T09:20:22.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.080+0000 7f970bfff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96ec0682f0 0x7f96ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:22.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.081+0000 7f970bfff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96ec0682f0 0x7f96ec06a7a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f96fc001c70 tx=0x7f96fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:22.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.084+0000 7f97097fa700 1 -- 172.21.15.102:0/3120637419 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f96f40539c0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.318+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f970c15cbb0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.319+0000 7f97097fa700 1 -- 172.21.15.102:0/3120637419 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f96f4056fe0 con 0x7f970c0cd8e0 2023-03-03T09:20:22.321 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:20:22.321 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:20:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96ec0682f0 msgr2=0x7f96ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96ec0682f0 0x7f96ec06a7a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f96fc001c70 tx=0x7f96fc009040 comp rx=0 tx=0).stop 2023-03-03T09:20:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f970c0cd8e0 msgr2=0x7f970c0c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f970c0cd8e0 0x7f970c0c7880 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f96f4016000 tx=0x7f96f4004750 comp rx=0 tx=0).stop 2023-03-03T09:20:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 shutdown_connections 2023-03-03T09:20:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f970c153290 0x7f970c0c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f970c0cd390 0x7f970c1e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96ec0682f0 0x7f96ec06a7a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 --2- 172.21.15.102:0/3120637419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f970c0cd8e0 0x7f970c0c7880 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 >> 172.21.15.102:0/3120637419 conn(0x7f970c065fb0 msgr2=0x7f970c15a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 shutdown_connections 2023-03-03T09:20:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:20:22.322+0000 7f97129ff700 1 -- 172.21.15.102:0/3120637419 wait complete. 2023-03-03T09:20:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:20:22.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-03-03T09:20:48.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:48.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:04.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:21:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.605+0000 7f2a0abff700 1 Processor -- start 2023-03-03T09:21:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.605+0000 7f2a0abff700 1 -- start start 2023-03-03T09:21:04.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.606+0000 7f2a0abff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a040cd390 0x7f2a040ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:04.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.606+0000 7f2a0abff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a04153660 con 0x7f2a04153290 2023-03-03T09:21:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.606+0000 7f2a0abff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0406c360 con 0x7f2a040cd8e0 2023-03-03T09:21:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.606+0000 7f2a0abff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0415cea0 con 0x7f2a040cd390 2023-03-03T09:21:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a0899b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2a04153290 0x7f2a04158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42560/0 (socket says 172.21.15.102:42560) 2023-03-03T09:21:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a0899b700 1 -- 172.21.15.102:0/892586741 learned_addr learned my addr 172.21.15.102:0/892586741 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:21:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a0919c700 1 --2- 172.21.15.102:0/892586741 >> v2:172.21.15.172:3300/0 conn(0x7f2a040cd390 0x7f2a040ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a037fe700 1 -- 172.21.15.102:0/892586741 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008938662 0 0) 0x7f2a04153660 con 0x7f2a04153290 2023-03-03T09:21:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a0919c700 1 -- 172.21.15.102:0/892586741 >> v1:172.21.15.172:6789/0 conn(0x7f2a040cd8e0 legacy=0x7f2a040cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a0919c700 1 -- 172.21.15.102:0/892586741 >> v1:172.21.15.102:6789/0 conn(0x7f2a04153290 legacy=0x7f2a04158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a0919c700 1 -- 172.21.15.102:0/892586741 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0406c7f0 con 0x7f2a040cd390 2023-03-03T09:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.607+0000 7f2a0919c700 1 --2- 172.21.15.102:0/892586741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd390 0x7f2a040ce530 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009d90 tx=0x7f29fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca7304fde0d0cb93 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a037fe700 1 -- 172.21.15.102:0/892586741 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00e070 con 0x7f2a040cd390 2023-03-03T09:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a037fe700 1 -- 172.21.15.102:0/892586741 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc00bad0 con 0x7f2a040cd390 2023-03-03T09:21:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a037fe700 1 -- 172.21.15.102:0/892586741 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00be80 con 0x7f2a040cd390 2023-03-03T09:21:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a0abff700 1 -- 172.21.15.102:0/892586741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd390 msgr2=0x7f2a040ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a0abff700 1 --2- 172.21.15.102:0/892586741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd390 0x7f2a040ce530 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009d90 tx=0x7f29fc00b330 comp rx=0 tx=0).stop 2023-03-03T09:21:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a0abff700 1 -- 172.21.15.102:0/892586741 shutdown_connections 2023-03-03T09:21:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a0abff700 1 --2- 172.21.15.102:0/892586741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd390 0x7f2a040ce530 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a0abff700 1 -- 172.21.15.102:0/892586741 >> 172.21.15.102:0/892586741 conn(0x7f2a04065fb0 msgr2=0x7f2a040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.608+0000 7f2a0abff700 1 -- 172.21.15.102:0/892586741 shutdown_connections 2023-03-03T09:21:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.609+0000 7f2a0abff700 1 -- 172.21.15.102:0/892586741 wait complete. 2023-03-03T09:21:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.611+0000 7f2a0abff700 1 Processor -- start 2023-03-03T09:21:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.611+0000 7f2a0abff700 1 -- start start 2023-03-03T09:21:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0abff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cd390 0x7f2a040c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0abff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd8e0 0x7f2a040b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0abff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a04153290 0x7f2a040c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0abff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a041f28f0 con 0x7f2a04153290 2023-03-03T09:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0abff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040b42c0 con 0x7f2a040cd8e0 2023-03-03T09:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0abff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04153970 con 0x7f2a040cd390 2023-03-03T09:21:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0919c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a04153290 0x7f2a040c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0899b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cd390 0x7f2a040c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a03fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd8e0 0x7f2a040b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0919c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a04153290 0x7f2a040c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39002/0 (socket says 172.21.15.102:39002) 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.612+0000 7f2a0919c700 1 -- 172.21.15.102:0/1093303319 learned_addr learned my addr 172.21.15.102:0/1093303319 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a0919c700 1 -- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cd390 msgr2=0x7f2a040c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a0919c700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cd390 0x7f2a040c6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a0919c700 1 -- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd8e0 msgr2=0x7f2a040b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a0919c700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd8e0 0x7f2a040b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a0919c700 1 -- 172.21.15.102:0/1093303319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc009a70 con 0x7f2a04153290 2023-03-03T09:21:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a0919c700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a04153290 0x7f2a040c7600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2a04074540 tx=0x7f29fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a017fa700 1 -- 172.21.15.102:0/1093303319 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc012630 con 0x7f2a04153290 2023-03-03T09:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a017fa700 1 -- 172.21.15.102:0/1093303319 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc012790 con 0x7f2a04153290 2023-03-03T09:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a017fa700 1 -- 172.21.15.102:0/1093303319 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc012b40 con 0x7f2a04153290 2023-03-03T09:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.613+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a040ccde0 con 0x7f2a04153290 2023-03-03T09:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.614+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0415d160 con 0x7f2a04153290 2023-03-03T09:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.615+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a040cd0c0 con 0x7f2a04153290 2023-03-03T09:21:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.616+0000 7f2a017fa700 1 -- 172.21.15.102:0/1093303319 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f29fc012ca0 con 0x7f2a04153290 2023-03-03T09:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.617+0000 7f2a017fa700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f29e80682f0 0x7f29e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.617+0000 7f2a017fa700 1 -- 172.21.15.102:0/1093303319 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f29fc087cc0 con 0x7f2a04153290 2023-03-03T09:21:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.617+0000 7f2a0899b700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f29e80682f0 0x7f29e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.618+0000 7f2a0899b700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f29e80682f0 0x7f29e806a7a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000f60 tx=0x7f29f8007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:04.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.621+0000 7f2a017fa700 1 -- 172.21.15.102:0/1093303319 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f29fc08e080 con 0x7f2a04153290 2023-03-03T09:21:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.858+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a0404e6e0 con 0x7f2a04153290 2023-03-03T09:21:04.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.858+0000 7f2a017fa700 1 -- 172.21.15.102:0/1093303319 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f29fc054370 con 0x7f2a04153290 2023-03-03T09:21:04.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:21:04.861 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:21:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f29e80682f0 msgr2=0x7f29e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f29e80682f0 0x7f29e806a7a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000f60 tx=0x7f29f8007e70 comp rx=0 tx=0).stop 2023-03-03T09:21:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a04153290 msgr2=0x7f2a040c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:04.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a04153290 0x7f2a040c7600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2a04074540 tx=0x7f29fc004ad0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 shutdown_connections 2023-03-03T09:21:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f29e80682f0 0x7f29e806a7a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2a040cd390 0x7f2a040c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a04153290 0x7f2a040c7600 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 --2- 172.21.15.102:0/1093303319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a040cd8e0 0x7f2a040b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 >> 172.21.15.102:0/1093303319 conn(0x7f2a04065fb0 msgr2=0x7f2a04156c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:04.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 shutdown_connections 2023-03-03T09:21:04.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:21:04.861+0000 7f2a0abff700 1 -- 172.21.15.102:0/1093303319 wait complete. 2023-03-03T09:21:04.865 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:21:04.889 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-03-03T09:21:18.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:18.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:48.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:48.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:18.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:18.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:46.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:22:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.147+0000 7fae3d1ff700 1 Processor -- start 2023-03-03T09:22:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.147+0000 7fae3d1ff700 1 -- start start 2023-03-03T09:22:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.148+0000 7fae3d1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae3814fbe0 0x7fae380c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.148+0000 7fae3d1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae380c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.148+0000 7fae3d1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae380c3c80 con 0x7fae380716c0 2023-03-03T09:22:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae3d1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae380c41a0 con 0x7fae3814fbe0 2023-03-03T09:22:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae3d1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae38071cd0 con 0x7fae3814ffb0 2023-03-03T09:22:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae36d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae3814fbe0 0x7fae380c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae3759e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fae380716c0 0x7fae380706f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50340/0 (socket says 172.21.15.102:50340) 2023-03-03T09:22:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae3759e700 1 -- 172.21.15.102:0/127380029 learned_addr learned my addr 172.21.15.102:0/127380029 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:22:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae3659c700 1 --2- 172.21.15.102:0/127380029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae380c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae36d9d700 1 -- 172.21.15.102:0/127380029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 msgr2=0x7fae380c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae36d9d700 1 --2- 172.21.15.102:0/127380029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae380c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae36d9d700 1 -- 172.21.15.102:0/127380029 >> v1:172.21.15.102:6789/0 conn(0x7fae380716c0 legacy=0x7fae380706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae36d9d700 1 -- 172.21.15.102:0/127380029 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae380c3ec0 con 0x7fae3814fbe0 2023-03-03T09:22:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.149+0000 7fae35d9b700 1 -- 172.21.15.102:0/127380029 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547541950 0 0) 0x7fae30001490 con 0x7fae380716c0 2023-03-03T09:22:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.150+0000 7fae36d9d700 1 --2- 172.21.15.102:0/127380029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae380c3740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fae28009840 tx=0x7fae28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb8c2418ed216b5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.150+0000 7fae35d9b700 1 -- 172.21.15.102:0/127380029 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28011830 con 0x7fae3814fbe0 2023-03-03T09:22:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.150+0000 7fae35d9b700 1 -- 172.21.15.102:0/127380029 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae28011990 con 0x7fae3814fbe0 2023-03-03T09:22:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.150+0000 7fae35d9b700 1 -- 172.21.15.102:0/127380029 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28011d20 con 0x7fae3814fbe0 2023-03-03T09:22:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 -- 172.21.15.102:0/127380029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 msgr2=0x7fae380c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/127380029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae380c3740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fae28009840 tx=0x7fae28009b50 comp rx=0 tx=0).stop 2023-03-03T09:22:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 -- 172.21.15.102:0/127380029 shutdown_connections 2023-03-03T09:22:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/127380029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae380c3740 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/127380029 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae380c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 -- 172.21.15.102:0/127380029 >> 172.21.15.102:0/127380029 conn(0x7fae3806af90 msgr2=0x7fae3806c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 -- 172.21.15.102:0/127380029 shutdown_connections 2023-03-03T09:22:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.151+0000 7fae3d1ff700 1 -- 172.21.15.102:0/127380029 wait complete. 2023-03-03T09:22:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.153+0000 7fae3d1ff700 1 Processor -- start 2023-03-03T09:22:47.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.153+0000 7fae3d1ff700 1 -- start start 2023-03-03T09:22:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.154+0000 7fae3d1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae380716c0 0x7fae381eec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.154+0000 7fae3d1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae38154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.154+0000 7fae3d1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae38154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.154+0000 7fae3d1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3814bd60 con 0x7fae3814fbe0 2023-03-03T09:22:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.154+0000 7fae3d1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae381dd6e0 con 0x7fae380716c0 2023-03-03T09:22:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.154+0000 7fae3d1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3814a170 con 0x7fae3814ffb0 2023-03-03T09:22:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.154+0000 7fae3659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae38154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae38154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae36d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae380716c0 0x7fae381eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae38154a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48084/0 (socket says 172.21.15.102:48084) 2023-03-03T09:22:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3659c700 1 -- 172.21.15.102:0/114481882 learned_addr learned my addr 172.21.15.102:0/114481882 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:22:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae38154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38596/0 (socket says 172.21.15.102:38596) 2023-03-03T09:22:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 -- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae380716c0 msgr2=0x7fae381eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae380716c0 0x7fae381eec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 -- 172.21.15.102:0/114481882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 msgr2=0x7fae38154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae38154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 -- 172.21.15.102:0/114481882 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280094a0 con 0x7fae3814ffb0 2023-03-03T09:22:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3659c700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae38154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:22:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.155+0000 7fae3759e700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae38154f40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fae3000cbe0 tx=0x7fae3000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.156+0000 7fae1f7fe700 1 -- 172.21.15.102:0/114481882 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3000aaa0 con 0x7fae3814ffb0 2023-03-03T09:22:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.156+0000 7fae1f7fe700 1 -- 172.21.15.102:0/114481882 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3000ac20 con 0x7fae3814ffb0 2023-03-03T09:22:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.156+0000 7fae1f7fe700 1 -- 172.21.15.102:0/114481882 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae300070d0 con 0x7fae3814ffb0 2023-03-03T09:22:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.156+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae38072250 con 0x7fae3814ffb0 2023-03-03T09:22:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.156+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae38071f70 con 0x7fae3814ffb0 2023-03-03T09:22:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.157+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae38072530 con 0x7fae3814ffb0 2023-03-03T09:22:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.158+0000 7fae1f7fe700 1 -- 172.21.15.102:0/114481882 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae30010070 con 0x7fae3814ffb0 2023-03-03T09:22:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.160+0000 7fae1f7fe700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae240682f0 0x7fae2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.160+0000 7fae1f7fe700 1 -- 172.21.15.102:0/114481882 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fae300853c0 con 0x7fae3814ffb0 2023-03-03T09:22:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.160+0000 7fae36d9d700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae240682f0 0x7fae2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.160+0000 7fae36d9d700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae240682f0 0x7fae2406a7a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fae280090b0 tx=0x7fae28008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.164+0000 7fae1f7fe700 1 -- 172.21.15.102:0/114481882 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae3004b5e0 con 0x7fae3814ffb0 2023-03-03T09:22:47.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.398+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae380c3c80 con 0x7fae3814ffb0 2023-03-03T09:22:47.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.399+0000 7fae1f7fe700 1 -- 172.21.15.102:0/114481882 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fae380c3c80 con 0x7fae3814ffb0 2023-03-03T09:22:47.401 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:22:47.401 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:22:47.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae240682f0 msgr2=0x7fae2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:47.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae240682f0 0x7fae2406a7a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fae280090b0 tx=0x7fae28008030 comp rx=0 tx=0).stop 2023-03-03T09:22:47.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 msgr2=0x7fae38154f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae38154f40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fae3000cbe0 tx=0x7fae3000cef0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 shutdown_connections 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae240682f0 0x7fae2406a7a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae3814ffb0 0x7fae38154f40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae380716c0 0x7fae381eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 --2- 172.21.15.102:0/114481882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae3814fbe0 0x7fae38154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 >> 172.21.15.102:0/114481882 conn(0x7fae3806af90 msgr2=0x7fae380c4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 shutdown_connections 2023-03-03T09:22:47.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:22:47.402+0000 7fae3d1ff700 1 -- 172.21.15.102:0/114481882 wait complete. 2023-03-03T09:22:47.405 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:22:47.430 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 90 secs before thrashing 2023-03-03T09:22:48.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:48.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:19.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:19.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:49.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:49.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:17.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:24:17.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.695+0000 7f8b61fff700 1 Processor -- start 2023-03-03T09:24:17.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.696+0000 7f8b61fff700 1 -- start start 2023-03-03T09:24:17.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.697+0000 7f8b61fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.697+0000 7f8b61fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5c153660 con 0x7f8b5c0cd8e0 2023-03-03T09:24:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.697+0000 7f8b61fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5c06c360 con 0x7f8b5c153290 2023-03-03T09:24:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.697+0000 7f8b61fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c15cea0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.697+0000 7f8b5affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b5c0cd8e0 0x7f8b5c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32816/0 (socket says 172.21.15.102:32816) 2023-03-03T09:24:17.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.697+0000 7f8b5affd700 1 -- 172.21.15.102:0/3302685879 learned_addr learned my addr 172.21.15.102:0/3302685879 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:24:17.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.697+0000 7f8b5bfff700 1 --2- 172.21.15.102:0/3302685879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5a7fc700 1 -- 172.21.15.102:0/3302685879 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546252566 0 0) 0x7f8b5c06c360 con 0x7f8b5c153290 2023-03-03T09:24:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5a7fc700 1 -- 172.21.15.102:0/3302685879 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b440035b0 con 0x7f8b5c153290 2023-03-03T09:24:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5a7fc700 1 -- 172.21.15.102:0/3302685879 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623436547 0 0) 0x7f8b5c153660 con 0x7f8b5c0cd8e0 2023-03-03T09:24:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5a7fc700 1 -- 172.21.15.102:0/3302685879 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b5c06c360 con 0x7f8b5c0cd8e0 2023-03-03T09:24:17.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5bfff700 1 -- 172.21.15.102:0/3302685879 >> v1:172.21.15.172:6789/0 conn(0x7f8b5c153290 legacy=0x7f8b5c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5bfff700 1 -- 172.21.15.102:0/3302685879 >> v1:172.21.15.102:6789/0 conn(0x7f8b5c0cd8e0 legacy=0x7f8b5c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5bfff700 1 -- 172.21.15.102:0/3302685879 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c06c7f0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.698+0000 7f8b5bfff700 1 --2- 172.21.15.102:0/3302685879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0ce530 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009dc0 tx=0x7f8b5400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7035e86b0585ab46 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b5a7fc700 1 -- 172.21.15.102:0/3302685879 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400e070 con 0x7f8b5c0cd390 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b5a7fc700 1 -- 172.21.15.102:0/3302685879 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5400bb20 con 0x7f8b5c0cd390 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b5a7fc700 1 -- 172.21.15.102:0/3302685879 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54012430 con 0x7f8b5c0cd390 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b61fff700 1 -- 172.21.15.102:0/3302685879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 msgr2=0x7f8b5c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:17.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b61fff700 1 --2- 172.21.15.102:0/3302685879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0ce530 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009dc0 tx=0x7f8b5400b330 comp rx=0 tx=0).stop 2023-03-03T09:24:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b61fff700 1 -- 172.21.15.102:0/3302685879 shutdown_connections 2023-03-03T09:24:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b61fff700 1 --2- 172.21.15.102:0/3302685879 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0ce530 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b61fff700 1 -- 172.21.15.102:0/3302685879 >> 172.21.15.102:0/3302685879 conn(0x7f8b5c065fb0 msgr2=0x7f8b5c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:24:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b61fff700 1 -- 172.21.15.102:0/3302685879 shutdown_connections 2023-03-03T09:24:17.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.699+0000 7f8b61fff700 1 -- 172.21.15.102:0/3302685879 wait complete. 2023-03-03T09:24:17.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.702+0000 7f8b61fff700 1 Processor -- start 2023-03-03T09:24:17.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.702+0000 7f8b61fff700 1 -- start start 2023-03-03T09:24:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b61fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b61fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c0cd8e0 0x7f8b5c0c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b61fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b5c153290 0x7f8b5c0b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b61fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c0c71b0 con 0x7f8b5c153290 2023-03-03T09:24:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b61fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c0c62c0 con 0x7f8b5c0cd8e0 2023-03-03T09:24:17.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b61fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c0c65a0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b5b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b5affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c0cd8e0 0x7f8b5c0c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b5b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0c8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54660/0 (socket says 172.21.15.102:54660) 2023-03-03T09:24:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b5b7fe700 1 -- 172.21.15.102:0/2527909511 learned_addr learned my addr 172.21.15.102:0/2527909511 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:24:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.704+0000 7f8b5bfff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b5c153290 0x7f8b5c0b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.705+0000 7f8b5b7fe700 1 -- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c0cd8e0 msgr2=0x7f8b5c0c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.705+0000 7f8b5b7fe700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c0cd8e0 0x7f8b5c0c7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.705+0000 7f8b5b7fe700 1 -- 172.21.15.102:0/2527909511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b5c153290 msgr2=0x7f8b5c0b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:17.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.705+0000 7f8b5b7fe700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b5c153290 0x7f8b5c0b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.705+0000 7f8b5b7fe700 1 -- 172.21.15.102:0/2527909511 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b54009a70 con 0x7f8b5c0cd390 2023-03-03T09:24:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.705+0000 7f8b5affd700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c0cd8e0 0x7f8b5c0c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:24:17.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.705+0000 7f8b5b7fe700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0c8a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c00c700 tx=0x7f8b4c00ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:24:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.707+0000 7f8b43fff700 1 -- 172.21.15.102:0/2527909511 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c0052d0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.708+0000 7f8b43fff700 1 -- 172.21.15.102:0/2527909511 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c005430 con 0x7f8b5c0cd390 2023-03-03T09:24:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.708+0000 7f8b43fff700 1 -- 172.21.15.102:0/2527909511 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c0038a0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.708+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5c0ccde0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.708+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5c15d160 con 0x7f8b5c0cd390 2023-03-03T09:24:17.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.709+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5c0cd0c0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.710+0000 7f8b43fff700 1 -- 172.21.15.102:0/2527909511 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b4c019030 con 0x7f8b5c0cd390 2023-03-03T09:24:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.711+0000 7f8b43fff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b48068340 0x7f8b4806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.711+0000 7f8b43fff700 1 -- 172.21.15.102:0/2527909511 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8b4c085070 con 0x7f8b5c0cd390 2023-03-03T09:24:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.712+0000 7f8b5affd700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b48068340 0x7f8b4806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.712+0000 7f8b5affd700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b48068340 0x7f8b4806a7f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c0cb5c0 tx=0x7f8b50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:24:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.716+0000 7f8b43fff700 1 -- 172.21.15.102:0/2527909511 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b4c051830 con 0x7f8b5c0cd390 2023-03-03T09:24:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.949+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b5c0b38e0 con 0x7f8b5c0cd390 2023-03-03T09:24:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.950+0000 7f8b43fff700 1 -- 172.21.15.102:0/2527909511 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8b4c017020 con 0x7f8b5c0cd390 2023-03-03T09:24:17.951 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:24:17.952 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:24:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b48068340 msgr2=0x7f8b4806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b48068340 0x7f8b4806a7f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c0cb5c0 tx=0x7f8b50009040 comp rx=0 tx=0).stop 2023-03-03T09:24:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 msgr2=0x7f8b5c0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0c8a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c00c700 tx=0x7f8b4c00ca10 comp rx=0 tx=0).stop 2023-03-03T09:24:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 shutdown_connections 2023-03-03T09:24:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b48068340 0x7f8b4806a7f0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b5c0cd390 0x7f8b5c0c8a80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b5c153290 0x7f8b5c0b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 --2- 172.21.15.102:0/2527909511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b5c0cd8e0 0x7f8b5c0c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 >> 172.21.15.102:0/2527909511 conn(0x7f8b5c065fb0 msgr2=0x7f8b5c150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:24:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 shutdown_connections 2023-03-03T09:24:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:24:17.953+0000 7f8b61fff700 1 -- 172.21.15.102:0/2527909511 wait complete. 2023-03-03T09:24:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:24:17.981 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T09:24:19.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:19.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:49.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:49.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:15.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:25:16.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.233+0000 7f0e28fff700 1 Processor -- start 2023-03-03T09:25:16.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.233+0000 7f0e28fff700 1 -- start start 2023-03-03T09:25:16.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.234+0000 7f0e28fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0e24153290 0x7f0e24158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.234+0000 7f0e28fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0e241593c0 0x7f0e240cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.234+0000 7f0e28fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24153660 con 0x7f0e240cd8e0 2023-03-03T09:25:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.234+0000 7f0e28fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2406c360 con 0x7f0e241593c0 2023-03-03T09:25:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.234+0000 7f0e28fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2415cea0 con 0x7f0e24153290 2023-03-03T09:25:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e21d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0e241593c0 0x7f0e240cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e22d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0e240cd8e0 0x7f0e240ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58292/0 (socket says 172.21.15.102:58292) 2023-03-03T09:25:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e22d9d700 1 -- 172.21.15.102:0/3219078083 learned_addr learned my addr 172.21.15.102:0/3219078083 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:25:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e2259c700 1 --2- 172.21.15.102:0/3219078083 >> v2:172.21.15.172:3300/0 conn(0x7f0e24153290 0x7f0e24158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e21d9b700 1 -- 172.21.15.102:0/3219078083 >> v2:172.21.15.172:3300/0 conn(0x7f0e24153290 msgr2=0x7f0e24158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:25:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e21d9b700 1 --2- 172.21.15.102:0/3219078083 >> v2:172.21.15.172:3300/0 conn(0x7f0e24153290 0x7f0e24158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e21d9b700 1 -- 172.21.15.102:0/3219078083 >> v1:172.21.15.102:6789/0 conn(0x7f0e240cd8e0 legacy=0x7f0e240ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e21d9b700 1 -- 172.21.15.102:0/3219078083 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2406c7f0 con 0x7f0e241593c0 2023-03-03T09:25:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.235+0000 7f0e2159a700 1 -- 172.21.15.102:0/3219078083 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271833775 0 0) 0x7f0e24153660 con 0x7f0e240cd8e0 2023-03-03T09:25:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.236+0000 7f0e21d9b700 1 --2- 172.21.15.102:0/3219078083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 0x7f0e240cde20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0e2408cc70 tx=0x7f0e18009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53888028ff0b6322 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:25:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.236+0000 7f0e2159a700 1 -- 172.21.15.102:0/3219078083 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18011830 con 0x7f0e241593c0 2023-03-03T09:25:16.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.236+0000 7f0e2159a700 1 -- 172.21.15.102:0/3219078083 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e18011990 con 0x7f0e241593c0 2023-03-03T09:25:16.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.236+0000 7f0e2159a700 1 -- 172.21.15.102:0/3219078083 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18011d20 con 0x7f0e241593c0 2023-03-03T09:25:16.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.236+0000 7f0e28fff700 1 -- 172.21.15.102:0/3219078083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 msgr2=0x7f0e240cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:16.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.236+0000 7f0e28fff700 1 --2- 172.21.15.102:0/3219078083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 0x7f0e240cde20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0e2408cc70 tx=0x7f0e18009b20 comp rx=0 tx=0).stop 2023-03-03T09:25:16.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.237+0000 7f0e28fff700 1 -- 172.21.15.102:0/3219078083 shutdown_connections 2023-03-03T09:25:16.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.237+0000 7f0e28fff700 1 --2- 172.21.15.102:0/3219078083 >> v2:172.21.15.172:3300/0 conn(0x7f0e24153290 0x7f0e24158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.237+0000 7f0e28fff700 1 --2- 172.21.15.102:0/3219078083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 0x7f0e240cde20 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.237+0000 7f0e28fff700 1 -- 172.21.15.102:0/3219078083 >> 172.21.15.102:0/3219078083 conn(0x7f0e24065fb0 msgr2=0x7f0e240663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:25:16.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.237+0000 7f0e28fff700 1 -- 172.21.15.102:0/3219078083 shutdown_connections 2023-03-03T09:25:16.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.237+0000 7f0e28fff700 1 -- 172.21.15.102:0/3219078083 wait complete. 2023-03-03T09:25:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.239+0000 7f0e28fff700 1 Processor -- start 2023-03-03T09:25:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.239+0000 7f0e28fff700 1 -- start start 2023-03-03T09:25:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e28fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e240cd8e0 0x7f0e240c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e28fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e24153290 0x7f0e240c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e28fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 0x7f0e240c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e28fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e240c7270 con 0x7f0e241593c0 2023-03-03T09:25:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e28fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e241ef0d0 con 0x7f0e24153290 2023-03-03T09:25:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e28fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e241ef600 con 0x7f0e240cd8e0 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e21d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e24153290 0x7f0e240c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e2259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e240cd8e0 0x7f0e240c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e21d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e24153290 0x7f0e240c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59690/0 (socket says 172.21.15.102:59690) 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e21d9b700 1 -- 172.21.15.102:0/1793643318 learned_addr learned my addr 172.21.15.102:0/1793643318 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.240+0000 7f0e22d9d700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 0x7f0e240c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e21d9b700 1 -- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e240cd8e0 msgr2=0x7f0e240c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e21d9b700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e240cd8e0 0x7f0e240c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e21d9b700 1 -- 172.21.15.102:0/1793643318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 msgr2=0x7f0e240c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e21d9b700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 0x7f0e240c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e21d9b700 1 -- 172.21.15.102:0/1793643318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180094a0 con 0x7f0e24153290 2023-03-03T09:25:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e21d9b700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e24153290 0x7f0e240c76c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0e180090b0 tx=0x7f0e18004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:25:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e12ffd700 1 -- 172.21.15.102:0/1793643318 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e180036a0 con 0x7f0e24153290 2023-03-03T09:25:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e12ffd700 1 -- 172.21.15.102:0/1793643318 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1801f070 con 0x7f0e24153290 2023-03-03T09:25:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e12ffd700 1 -- 172.21.15.102:0/1793643318 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18003a20 con 0x7f0e24153290 2023-03-03T09:25:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.241+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e240ccde0 con 0x7f0e24153290 2023-03-03T09:25:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.242+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2415d160 con 0x7f0e24153290 2023-03-03T09:25:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.243+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e240cd0c0 con 0x7f0e24153290 2023-03-03T09:25:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.244+0000 7f0e12ffd700 1 -- 172.21.15.102:0/1793643318 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0e1800f050 con 0x7f0e24153290 2023-03-03T09:25:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.245+0000 7f0e12ffd700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e040682f0 0x7f0e0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.245+0000 7f0e12ffd700 1 -- 172.21.15.102:0/1793643318 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0e18021070 con 0x7f0e24153290 2023-03-03T09:25:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.246+0000 7f0e2259c700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e040682f0 0x7f0e0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.246+0000 7f0e2259c700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e040682f0 0x7f0e0406a7a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009be0 tx=0x7f0e14009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:25:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.250+0000 7f0e12ffd700 1 -- 172.21.15.102:0/1793643318 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e18053c70 con 0x7f0e24153290 2023-03-03T09:25:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.486+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e2408bee0 con 0x7f0e24153290 2023-03-03T09:25:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.487+0000 7f0e12ffd700 1 -- 172.21.15.102:0/1793643318 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0e18057290 con 0x7f0e24153290 2023-03-03T09:25:16.489 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:25:16.489 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:25:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e040682f0 msgr2=0x7f0e0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e040682f0 0x7f0e0406a7a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009be0 tx=0x7f0e14009210 comp rx=0 tx=0).stop 2023-03-03T09:25:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e24153290 msgr2=0x7f0e240c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e24153290 0x7f0e240c76c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0e180090b0 tx=0x7f0e18004750 comp rx=0 tx=0).stop 2023-03-03T09:25:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 shutdown_connections 2023-03-03T09:25:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e040682f0 0x7f0e0406a7a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e240cd8e0 0x7f0e240c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e241593c0 0x7f0e240c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 --2- 172.21.15.102:0/1793643318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e24153290 0x7f0e240c76c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 >> 172.21.15.102:0/1793643318 conn(0x7f0e24065fb0 msgr2=0x7f0e240cc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:25:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.490+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 shutdown_connections 2023-03-03T09:25:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:25:16.491+0000 7f0e28fff700 1 -- 172.21.15.102:0/1793643318 wait complete. 2023-03-03T09:25:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:25:16.518 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T09:25:19.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:19.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:49.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:49.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:18.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:26:18.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.768+0000 7f20eebff700 1 Processor -- start 2023-03-03T09:26:18.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.768+0000 7f20eebff700 1 -- start start 2023-03-03T09:26:18.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.769+0000 7f20eebff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f20e80cd8e0 0x7f20e80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:18.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.769+0000 7f20eebff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e80cd390 0x7f20e80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:18.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.769+0000 7f20eebff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e8153660 con 0x7f20e8153290 2023-03-03T09:26:18.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.769+0000 7f20eebff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e806c360 con 0x7f20e80cd8e0 2023-03-03T09:26:18.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.769+0000 7f20eebff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e815cea0 con 0x7f20e80cd390 2023-03-03T09:26:18.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.769+0000 7f20ec99b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f20e8153290 0x7f20e8158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56428/0 (socket says 172.21.15.102:56428) 2023-03-03T09:26:18.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.769+0000 7f20ec99b700 1 -- 172.21.15.102:0/1645061715 learned_addr learned my addr 172.21.15.102:0/1645061715 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:26:18.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20ed19c700 1 --2- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e80cd390 0x7f20e80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:18.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e7fff700 1 --2- 172.21.15.102:0/1645061715 >> v2:172.21.15.172:3300/0 conn(0x7f20e80cd8e0 0x7f20e80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:18.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e77fe700 1 -- 172.21.15.102:0/1645061715 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890792055 0 0) 0x7f20e8153660 con 0x7f20e8153290 2023-03-03T09:26:18.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e77fe700 1 -- 172.21.15.102:0/1645061715 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d80035b0 con 0x7f20e8153290 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e7fff700 1 -- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e80cd390 msgr2=0x7f20e80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e7fff700 1 --2- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e80cd390 0x7f20e80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e7fff700 1 -- 172.21.15.102:0/1645061715 >> v1:172.21.15.102:6789/0 conn(0x7f20e8153290 legacy=0x7f20e8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e7fff700 1 -- 172.21.15.102:0/1645061715 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e806c7f0 con 0x7f20e80cd8e0 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20ed19c700 1 --2- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e80cd390 0x7f20e80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.770+0000 7f20e7fff700 1 --2- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e80cde20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009840 tx=0x7f20d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e943df872904736e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20e77fe700 1 -- 172.21.15.102:0/1645061715 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4011830 con 0x7f20e80cd8e0 2023-03-03T09:26:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20e77fe700 1 -- 172.21.15.102:0/1645061715 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d4011990 con 0x7f20e80cd8e0 2023-03-03T09:26:18.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20e77fe700 1 -- 172.21.15.102:0/1645061715 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4011d20 con 0x7f20e80cd8e0 2023-03-03T09:26:18.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20eebff700 1 -- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 msgr2=0x7f20e80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:18.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20eebff700 1 --2- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e80cde20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009840 tx=0x7f20d4009b50 comp rx=0 tx=0).stop 2023-03-03T09:26:18.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20eebff700 1 -- 172.21.15.102:0/1645061715 shutdown_connections 2023-03-03T09:26:18.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20eebff700 1 --2- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e80cd390 0x7f20e80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:18.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20eebff700 1 --2- 172.21.15.102:0/1645061715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e80cde20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:18.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20eebff700 1 -- 172.21.15.102:0/1645061715 >> 172.21.15.102:0/1645061715 conn(0x7f20e8065fb0 msgr2=0x7f20e80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:18.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.771+0000 7f20eebff700 1 -- 172.21.15.102:0/1645061715 shutdown_connections 2023-03-03T09:26:18.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.772+0000 7f20eebff700 1 -- 172.21.15.102:0/1645061715 wait complete. 2023-03-03T09:26:18.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.773+0000 7f20eebff700 1 Processor -- start 2023-03-03T09:26:18.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.774+0000 7f20eebff700 1 -- start start 2023-03-03T09:26:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.774+0000 7f20eebff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e80cd390 0x7f20e8157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20eebff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e8159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20eebff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e8153290 0x7f20e81e5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20eebff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e81e4b70 con 0x7f20e80cd390 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20eebff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e81e4e50 con 0x7f20e80cd8e0 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20eebff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e81e33b0 con 0x7f20e8153290 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20ec99b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e80cd390 0x7f20e8157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e8159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e8159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35672/0 (socket says 172.21.15.102:35672) 2023-03-03T09:26:18.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20ed19c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e8153290 0x7f20e81e5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 -- 172.21.15.102:0/3564086266 learned_addr learned my addr 172.21.15.102:0/3564086266 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:26:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 -- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e8153290 msgr2=0x7f20e81e5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e8153290 0x7f20e81e5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 -- 172.21.15.102:0/3564086266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e80cd390 msgr2=0x7f20e8157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e80cd390 0x7f20e8157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20e7fff700 1 -- 172.21.15.102:0/3564086266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0009a70 con 0x7f20e80cd8e0 2023-03-03T09:26:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.775+0000 7f20ed19c700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e8153290 0x7f20e81e5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:26:18.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.776+0000 7f20e7fff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e8159fe0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009fd0 tx=0x7f20d4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:18.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.776+0000 7f20e57fa700 1 -- 172.21.15.102:0/3564086266 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4004550 con 0x7f20e80cd8e0 2023-03-03T09:26:18.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.776+0000 7f20e57fa700 1 -- 172.21.15.102:0/3564086266 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d401e070 con 0x7f20e80cd8e0 2023-03-03T09:26:18.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.776+0000 7f20e57fa700 1 -- 172.21.15.102:0/3564086266 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d40038e0 con 0x7f20e80cd8e0 2023-03-03T09:26:18.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.776+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d40094a0 con 0x7f20e80cd8e0 2023-03-03T09:26:18.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.776+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e80ccde0 con 0x7f20e80cd8e0 2023-03-03T09:26:18.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.777+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e815d160 con 0x7f20e80cd8e0 2023-03-03T09:26:18.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.778+0000 7f20e57fa700 1 -- 172.21.15.102:0/3564086266 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f20d400f050 con 0x7f20e80cd8e0 2023-03-03T09:26:18.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.780+0000 7f20e57fa700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20cc068320 0x7f20cc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:18.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.780+0000 7f20e57fa700 1 -- 172.21.15.102:0/3564086266 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f20d4086bf0 con 0x7f20e80cd8e0 2023-03-03T09:26:18.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.780+0000 7f20ec99b700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20cc068320 0x7f20cc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:18.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.781+0000 7f20ec99b700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20cc068320 0x7f20cc06a7d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f20dc007fd0 tx=0x7f20dc007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:18.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:18.784+0000 7f20e57fa700 1 -- 172.21.15.102:0/3564086266 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f20d4053270 con 0x7f20e80cd8e0 2023-03-03T09:26:19.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.020+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20e80cd0c0 con 0x7f20e80cd8e0 2023-03-03T09:26:19.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.021+0000 7f20e57fa700 1 -- 172.21.15.102:0/3564086266 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f20d4015020 con 0x7f20e80cd8e0 2023-03-03T09:26:19.023 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:26:19.023 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:26:19.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20cc068320 msgr2=0x7f20cc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:19.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20cc068320 0x7f20cc06a7d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f20dc007fd0 tx=0x7f20dc007dc0 comp rx=0 tx=0).stop 2023-03-03T09:26:19.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 msgr2=0x7f20e8159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:19.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e8159fe0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009fd0 tx=0x7f20d4004750 comp rx=0 tx=0).stop 2023-03-03T09:26:19.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 shutdown_connections 2023-03-03T09:26:19.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20cc068320 0x7f20cc06a7d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:19.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20e8153290 0x7f20e81e5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:19.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e80cd390 0x7f20e8157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:19.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 --2- 172.21.15.102:0/3564086266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20e80cd8e0 0x7f20e8159fe0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:19.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 >> 172.21.15.102:0/3564086266 conn(0x7f20e8065fb0 msgr2=0x7f20e80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 shutdown_connections 2023-03-03T09:26:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:19.023+0000 7f20eebff700 1 -- 172.21.15.102:0/3564086266 wait complete. 2023-03-03T09:26:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:26:19.053 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2023-03-03T09:26:19.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:19.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:46.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:26:46.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.302+0000 7f3946b9d700 1 Processor -- start 2023-03-03T09:26:46.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.302+0000 7f3946b9d700 1 -- start start 2023-03-03T09:26:46.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f3946b9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f39400ced00 0x7f39400c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:46.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f3946b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c41b0 0x7f39400cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:46.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f3946b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400c3d20 con 0x7f39400b3970 2023-03-03T09:26:46.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f3946b9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39400c4930 con 0x7f39400ced00 2023-03-03T09:26:46.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f3946b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39400cf610 con 0x7f39400c41b0 2023-03-03T09:26:46.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f3945b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f39400ced00 0x7f39400c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:46.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f394539a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f39400b3970 0x7f3940158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:55294/0 (socket says 172.21.15.102:55294) 2023-03-03T09:26:46.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f394539a700 1 -- 172.21.15.102:0/3207290424 learned_addr learned my addr 172.21.15.102:0/3207290424 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:26:46.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.304+0000 7f394639c700 1 --2- 172.21.15.102:0/3207290424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c41b0 0x7f39400cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:46.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3945b9b700 1 -- 172.21.15.102:0/3207290424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c41b0 msgr2=0x7f39400cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:46.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3945b9b700 1 --2- 172.21.15.102:0/3207290424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c41b0 0x7f39400cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3945b9b700 1 -- 172.21.15.102:0/3207290424 >> v1:172.21.15.172:6789/0 conn(0x7f39400b3970 legacy=0x7f3940158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:46.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3945b9b700 1 -- 172.21.15.102:0/3207290424 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394015cba0 con 0x7f39400ced00 2023-03-03T09:26:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3944b99700 1 -- 172.21.15.102:0/3207290424 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113001285 0 0) 0x7f39400c3d20 con 0x7f39400b3970 2023-03-03T09:26:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3945b9b700 1 --2- 172.21.15.102:0/3207290424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f39400c5a90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f393c009870 tx=0x7f393c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9557b1050eaff3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3944b99700 1 -- 172.21.15.102:0/3207290424 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c011830 con 0x7f39400ced00 2023-03-03T09:26:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3944b99700 1 -- 172.21.15.102:0/3207290424 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c011990 con 0x7f39400ced00 2023-03-03T09:26:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.305+0000 7f3944b99700 1 -- 172.21.15.102:0/3207290424 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c011d20 con 0x7f39400ced00 2023-03-03T09:26:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 -- 172.21.15.102:0/3207290424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 msgr2=0x7f39400c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 --2- 172.21.15.102:0/3207290424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f39400c5a90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f393c009870 tx=0x7f393c009b80 comp rx=0 tx=0).stop 2023-03-03T09:26:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 -- 172.21.15.102:0/3207290424 shutdown_connections 2023-03-03T09:26:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 --2- 172.21.15.102:0/3207290424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c41b0 0x7f39400cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 --2- 172.21.15.102:0/3207290424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f39400c5a90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 -- 172.21.15.102:0/3207290424 >> 172.21.15.102:0/3207290424 conn(0x7f394008b1c0 msgr2=0x7f3940071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 -- 172.21.15.102:0/3207290424 shutdown_connections 2023-03-03T09:26:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.306+0000 7f3946b9d700 1 -- 172.21.15.102:0/3207290424 wait complete. 2023-03-03T09:26:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.309+0000 7f3946b9d700 1 Processor -- start 2023-03-03T09:26:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.309+0000 7f3946b9d700 1 -- start start 2023-03-03T09:26:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.310+0000 7f3946b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400b3970 0x7f394015a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:46.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3946b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400c41b0 0x7f3940148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:46.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3946b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f3940151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:46.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3946b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394014c7e0 con 0x7f39400ced00 2023-03-03T09:26:46.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3946b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394014c500 con 0x7f39400c41b0 2023-03-03T09:26:46.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3946b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394014f520 con 0x7f39400b3970 2023-03-03T09:26:46.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f394539a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400c41b0 0x7f3940148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:46.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400b3970 0x7f394015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f394639c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f3940151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400b3970 0x7f394015a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34028/0 (socket says 172.21.15.102:34028) 2023-03-03T09:26:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 -- 172.21.15.102:0/1430079797 learned_addr learned my addr 172.21.15.102:0/1430079797 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:26:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 -- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400c41b0 msgr2=0x7f3940148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400c41b0 0x7f3940148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 -- 172.21.15.102:0/1430079797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 msgr2=0x7f3940151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f3940151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f3945b9b700 1 -- 172.21.15.102:0/1430079797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c0094a0 con 0x7f39400b3970 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.311+0000 7f394639c700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f3940151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.312+0000 7f3945b9b700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400b3970 0x7f394015a660 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f393c0090b0 tx=0x7f393c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.312+0000 7f39367fc700 1 -- 172.21.15.102:0/1430079797 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c01a8b0 con 0x7f39400b3970 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.312+0000 7f39367fc700 1 -- 172.21.15.102:0/1430079797 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c01aa10 con 0x7f39400b3970 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.312+0000 7f39367fc700 1 -- 172.21.15.102:0/1430079797 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c01ad80 con 0x7f39400b3970 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.312+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394006bf20 con 0x7f39400b3970 2023-03-03T09:26:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.312+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394015ce80 con 0x7f39400b3970 2023-03-03T09:26:46.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.313+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394006c200 con 0x7f39400b3970 2023-03-03T09:26:46.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.314+0000 7f39367fc700 1 -- 172.21.15.102:0/1430079797 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f393c00f050 con 0x7f39400b3970 2023-03-03T09:26:46.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.315+0000 7f39367fc700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39240682f0 0x7f392406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:46.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.316+0000 7f39367fc700 1 -- 172.21.15.102:0/1430079797 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f393c0870d0 con 0x7f39400b3970 2023-03-03T09:26:46.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.316+0000 7f394539a700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39240682f0 0x7f392406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:46.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.316+0000 7f394539a700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39240682f0 0x7f392406a7a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f394015c590 tx=0x7f3930007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:46.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.320+0000 7f39367fc700 1 -- 172.21.15.102:0/1430079797 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f393c0537e0 con 0x7f39400b3970 2023-03-03T09:26:46.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.555+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f394014f760 con 0x7f39400b3970 2023-03-03T09:26:46.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.556+0000 7f39367fc700 1 -- 172.21.15.102:0/1430079797 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f394014f760 con 0x7f39400b3970 2023-03-03T09:26:46.557 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:26:46.557 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:26:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.558+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39240682f0 msgr2=0x7f392406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.558+0000 7f3946b9d700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39240682f0 0x7f392406a7a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f394015c590 tx=0x7f3930007e70 comp rx=0 tx=0).stop 2023-03-03T09:26:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.558+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400b3970 msgr2=0x7f394015a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.558+0000 7f3946b9d700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400b3970 0x7f394015a660 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f393c0090b0 tx=0x7f393c004750 comp rx=0 tx=0).stop 2023-03-03T09:26:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 shutdown_connections 2023-03-03T09:26:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39240682f0 0x7f392406a7a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400b3970 0x7f394015a660 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400ced00 0x7f3940151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 --2- 172.21.15.102:0/1430079797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400c41b0 0x7f3940148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:46.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 >> 172.21.15.102:0/1430079797 conn(0x7f394008b1c0 msgr2=0x7f39401e1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:46.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 shutdown_connections 2023-03-03T09:26:46.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:26:46.559+0000 7f3946b9d700 1 -- 172.21.15.102:0/1430079797 wait complete. 2023-03-03T09:26:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:26:46.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-03-03T09:26:50.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:50.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:20.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:20.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:50.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:50.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:01.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:28:01.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.859+0000 7f14e1bff700 1 Processor -- start 2023-03-03T09:28:01.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.859+0000 7f14e1bff700 1 -- start start 2023-03-03T09:28:01.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.860+0000 7f14e1bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14dc0cd8e0 0x7f14dc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.860+0000 7f14e1bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14dc153660 con 0x7f14dc0cd390 2023-03-03T09:28:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.860+0000 7f14e1bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14dc06c360 con 0x7f14dc153290 2023-03-03T09:28:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.860+0000 7f14e1bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14dc15cea0 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.861+0000 7f14daffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14dc0cd8e0 0x7f14dc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.861+0000 7f14db7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f14dc153290 0x7f14dc158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50396/0 (socket says 172.21.15.102:50396) 2023-03-03T09:28:01.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.861+0000 7f14db7fe700 1 -- 172.21.15.102:0/3463345394 learned_addr learned my addr 172.21.15.102:0/3463345394 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:28:01.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.861+0000 7f14daffd700 1 -- 172.21.15.102:0/3463345394 >> v1:172.21.15.172:6789/0 conn(0x7f14dc153290 legacy=0x7f14dc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.861+0000 7f14daffd700 1 -- 172.21.15.102:0/3463345394 >> v1:172.21.15.102:6789/0 conn(0x7f14dc0cd390 legacy=0x7f14dc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.861+0000 7f14daffd700 1 -- 172.21.15.102:0/3463345394 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc06c7f0 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.861+0000 7f14daffd700 1 --2- 172.21.15.102:0/3463345394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc0cde20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009840 tx=0x7f14d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736a5211c4bcadd6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14da7fc700 1 -- 172.21.15.102:0/3463345394 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0011830 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14da7fc700 1 -- 172.21.15.102:0/3463345394 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d0011990 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14da7fc700 1 -- 172.21.15.102:0/3463345394 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0011d20 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14e1bff700 1 -- 172.21.15.102:0/3463345394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 msgr2=0x7f14dc0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14e1bff700 1 --2- 172.21.15.102:0/3463345394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc0cde20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009840 tx=0x7f14d0009b50 comp rx=0 tx=0).stop 2023-03-03T09:28:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14e1bff700 1 -- 172.21.15.102:0/3463345394 shutdown_connections 2023-03-03T09:28:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14e1bff700 1 --2- 172.21.15.102:0/3463345394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc0cde20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.862+0000 7f14e1bff700 1 -- 172.21.15.102:0/3463345394 >> 172.21.15.102:0/3463345394 conn(0x7f14dc065fb0 msgr2=0x7f14dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.863+0000 7f14e1bff700 1 -- 172.21.15.102:0/3463345394 shutdown_connections 2023-03-03T09:28:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.863+0000 7f14e1bff700 1 -- 172.21.15.102:0/3463345394 wait complete. 2023-03-03T09:28:01.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.864+0000 7f14e1bff700 1 Processor -- start 2023-03-03T09:28:01.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.865+0000 7f14e1bff700 1 -- start start 2023-03-03T09:28:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14e1bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14dc0cd390 0x7f14dc0c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14e1bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc1e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14e1bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14dc153290 0x7f14dc1f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14e1bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc0c9000 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14e1bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc0c8e00 con 0x7f14dc153290 2023-03-03T09:28:01.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14e1bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc153990 con 0x7f14dc0cd390 2023-03-03T09:28:01.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc1e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:01.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc1e5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34298/0 (socket says 172.21.15.102:34298) 2023-03-03T09:28:01.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 -- 172.21.15.102:0/560947910 learned_addr learned my addr 172.21.15.102:0/560947910 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:28:01.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14db7fe700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14dc0cd390 0x7f14dc0c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:01.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14dbfff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14dc153290 0x7f14dc1f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:01.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 -- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14dc0cd390 msgr2=0x7f14dc0c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:01.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14dc0cd390 0x7f14dc0c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:01.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 -- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14dc153290 msgr2=0x7f14dc1f2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:01.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14dc153290 0x7f14dc1f2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:01.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.866+0000 7f14daffd700 1 -- 172.21.15.102:0/560947910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00094a0 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.867+0000 7f14dbfff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14dc153290 0x7f14dc1f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:28:01.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.867+0000 7f14daffd700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc1e5720 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f14dc074540 tx=0x7f14d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:01.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.867+0000 7f14c3fff700 1 -- 172.21.15.102:0/560947910 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d00036a0 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.867+0000 7f14c3fff700 1 -- 172.21.15.102:0/560947910 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d001f070 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.867+0000 7f14c3fff700 1 -- 172.21.15.102:0/560947910 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0003a60 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.867+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14dc0ccde0 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.867+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14dc15d160 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.869+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14dc0cd0c0 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.870+0000 7f14c3fff700 1 -- 172.21.15.102:0/560947910 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f14d000f050 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.872+0000 7f14c3fff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14c8068320 0x7f14c806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:01.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.872+0000 7f14c3fff700 1 -- 172.21.15.102:0/560947910 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f14d0021070 con 0x7f14dc0cd8e0 2023-03-03T09:28:01.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.872+0000 7f14db7fe700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14c8068320 0x7f14c806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:01.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.873+0000 7f14db7fe700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14c8068320 0x7f14c806a7d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f14cc005910 tx=0x7f14cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:01.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:01.876+0000 7f14c3fff700 1 -- 172.21.15.102:0/560947910 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f14d0053880 con 0x7f14dc0cd8e0 2023-03-03T09:28:02.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.112+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14dc04e6e0 con 0x7f14dc0cd8e0 2023-03-03T09:28:02.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.112+0000 7f14c3fff700 1 -- 172.21.15.102:0/560947910 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f14d0015020 con 0x7f14dc0cd8e0 2023-03-03T09:28:02.115 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:28:02.115 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:28:02.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14c8068320 msgr2=0x7f14c806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:02.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14c8068320 0x7f14c806a7d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f14cc005910 tx=0x7f14cc009040 comp rx=0 tx=0).stop 2023-03-03T09:28:02.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 msgr2=0x7f14dc1e5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc1e5720 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f14dc074540 tx=0x7f14d0004750 comp rx=0 tx=0).stop 2023-03-03T09:28:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 shutdown_connections 2023-03-03T09:28:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14c8068320 0x7f14c806a7d0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14dc0cd390 0x7f14dc0c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14dc153290 0x7f14dc1f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 --2- 172.21.15.102:0/560947910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc0cd8e0 0x7f14dc1e5720 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 >> 172.21.15.102:0/560947910 conn(0x7f14dc065fb0 msgr2=0x7f14dc1580c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:02.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 shutdown_connections 2023-03-03T09:28:02.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:28:02.115+0000 7f14e1bff700 1 -- 172.21.15.102:0/560947910 wait complete. 2023-03-03T09:28:02.119 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:28:02.144 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T09:28:20.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:20.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:50.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:50.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:04.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:29:04.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.397+0000 7f208f1ff700 1 Processor -- start 2023-03-03T09:29:04.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.398+0000 7f208f1ff700 1 -- start start 2023-03-03T09:29:04.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.399+0000 7f208f1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880c9cb0 0x7f20880cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:04.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.399+0000 7f208f1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f20880ca190 0x7f20880cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:04.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.399+0000 7f208f1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20880cf940 con 0x7f2088153260 2023-03-03T09:29:04.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.399+0000 7f208f1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20880cfe70 con 0x7f20880ca190 2023-03-03T09:29:04.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.399+0000 7f208f1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2088158d70 con 0x7f20880c9cb0 2023-03-03T09:29:04.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.399+0000 7f208cf9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2088153260 0x7f20880ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44248/0 (socket says 172.21.15.102:44248) 2023-03-03T09:29:04.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f208cf9b700 1 -- 172.21.15.102:0/2522836792 learned_addr learned my addr 172.21.15.102:0/2522836792 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:29:04.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f2087fff700 1 --2- 172.21.15.102:0/2522836792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880c9cb0 0x7f20880cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f208d79c700 1 --2- 172.21.15.102:0/2522836792 >> v2:172.21.15.102:3300/0 conn(0x7f20880ca190 0x7f20880cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f20877fe700 1 -- 172.21.15.102:0/2522836792 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601708475 0 0) 0x7f20880cf940 con 0x7f2088153260 2023-03-03T09:29:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f2087fff700 1 -- 172.21.15.102:0/2522836792 >> v2:172.21.15.102:3300/0 conn(0x7f20880ca190 msgr2=0x7f20880cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:29:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f2087fff700 1 --2- 172.21.15.102:0/2522836792 >> v2:172.21.15.102:3300/0 conn(0x7f20880ca190 0x7f20880cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f2087fff700 1 -- 172.21.15.102:0/2522836792 >> v1:172.21.15.172:6789/0 conn(0x7f2088153260 legacy=0x7f20880ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f2087fff700 1 -- 172.21.15.102:0/2522836792 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20880cfb90 con 0x7f20880c9cb0 2023-03-03T09:29:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f208d79c700 1 --2- 172.21.15.102:0/2522836792 >> v2:172.21.15.102:3300/0 conn(0x7f20880ca190 0x7f20880cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.400+0000 7f2087fff700 1 --2- 172.21.15.102:0/2522836792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880c9cb0 0x7f20880cb880 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2074009840 tx=0x7f2074009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef75d88e05e50acf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f20877fe700 1 -- 172.21.15.102:0/2522836792 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074011830 con 0x7f20880c9cb0 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f20877fe700 1 -- 172.21.15.102:0/2522836792 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2074011990 con 0x7f20880c9cb0 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f20877fe700 1 -- 172.21.15.102:0/2522836792 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074011d20 con 0x7f20880c9cb0 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 -- 172.21.15.102:0/2522836792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880c9cb0 msgr2=0x7f20880cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 --2- 172.21.15.102:0/2522836792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880c9cb0 0x7f20880cb880 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2074009840 tx=0x7f2074009b50 comp rx=0 tx=0).stop 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 -- 172.21.15.102:0/2522836792 shutdown_connections 2023-03-03T09:29:04.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 --2- 172.21.15.102:0/2522836792 >> v2:172.21.15.102:3300/0 conn(0x7f20880ca190 0x7f20880cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 --2- 172.21.15.102:0/2522836792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880c9cb0 0x7f20880cb880 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 -- 172.21.15.102:0/2522836792 >> 172.21.15.102:0/2522836792 conn(0x7f2088066c00 msgr2=0x7f20880ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:29:04.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 -- 172.21.15.102:0/2522836792 shutdown_connections 2023-03-03T09:29:04.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.401+0000 7f208f1ff700 1 -- 172.21.15.102:0/2522836792 wait complete. 2023-03-03T09:29:04.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.403+0000 7f208f1ff700 1 Processor -- start 2023-03-03T09:29:04.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.403+0000 7f208f1ff700 1 -- start start 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.404+0000 7f208f1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880ca190 0x7f208814b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.404+0000 7f208f1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2088153260 0x7f20881547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.404+0000 7f208f1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20880c5920 0x7f2088154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.404+0000 7f208f1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20881ef610 con 0x7f20880c5920 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.404+0000 7f208f1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088151cf0 con 0x7f2088153260 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.404+0000 7f208f1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2088157cb0 con 0x7f20880ca190 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880ca190 0x7f208814b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880ca190 0x7f208814b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45458/0 (socket says 172.21.15.102:45458) 2023-03-03T09:29:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 -- 172.21.15.102:0/3860806358 learned_addr learned my addr 172.21.15.102:0/3860806358 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:29:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f2087fff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2088153260 0x7f20881547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208d79c700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20880c5920 0x7f2088154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 -- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2088153260 msgr2=0x7f20881547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2088153260 0x7f20881547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 -- 172.21.15.102:0/3860806358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20880c5920 msgr2=0x7f2088154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20880c5920 0x7f2088154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 -- 172.21.15.102:0/3860806358 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20740094a0 con 0x7f20880ca190 2023-03-03T09:29:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.405+0000 7f208cf9b700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880ca190 0x7f208814b290 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c750 tx=0x7f207c00ca60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:29:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.406+0000 7f20857fa700 1 -- 172.21.15.102:0/3860806358 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c004c80 con 0x7f20880ca190 2023-03-03T09:29:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.406+0000 7f20857fa700 1 -- 172.21.15.102:0/3860806358 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c004e00 con 0x7f20880ca190 2023-03-03T09:29:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.406+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2088159260 con 0x7f20880ca190 2023-03-03T09:29:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.406+0000 7f20857fa700 1 -- 172.21.15.102:0/3860806358 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c007310 con 0x7f20880ca190 2023-03-03T09:29:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.406+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2088158f80 con 0x7f20880ca190 2023-03-03T09:29:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.407+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2088159540 con 0x7f20880ca190 2023-03-03T09:29:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.408+0000 7f20857fa700 1 -- 172.21.15.102:0/3860806358 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f207c01a020 con 0x7f20880ca190 2023-03-03T09:29:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.409+0000 7f20857fa700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f206c068320 0x7f206c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.410+0000 7f20857fa700 1 -- 172.21.15.102:0/3860806358 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f207c085960 con 0x7f20880ca190 2023-03-03T09:29:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.410+0000 7f2087fff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f206c068320 0x7f206c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.410+0000 7f2087fff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f206c068320 0x7f206c06a7d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f20740090b0 tx=0x7f207401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:29:04.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.413+0000 7f20857fa700 1 -- 172.21.15.102:0/3860806358 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f207c052120 con 0x7f20880ca190 2023-03-03T09:29:04.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.646+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f208806b180 con 0x7f20880ca190 2023-03-03T09:29:04.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.647+0000 7f20857fa700 1 -- 172.21.15.102:0/3860806358 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f207c055740 con 0x7f20880ca190 2023-03-03T09:29:04.648 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:29:04.649 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:29:04.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f206c068320 msgr2=0x7f206c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:04.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f206c068320 0x7f206c06a7d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f20740090b0 tx=0x7f207401c040 comp rx=0 tx=0).stop 2023-03-03T09:29:04.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880ca190 msgr2=0x7f208814b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:04.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880ca190 0x7f208814b290 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c750 tx=0x7f207c00ca60 comp rx=0 tx=0).stop 2023-03-03T09:29:04.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 shutdown_connections 2023-03-03T09:29:04.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f206c068320 0x7f206c06a7d0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20880ca190 0x7f208814b290 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20880c5920 0x7f2088154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 --2- 172.21.15.102:0/3860806358 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2088153260 0x7f20881547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:04.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 >> 172.21.15.102:0/3860806358 conn(0x7f20880c9cb0 msgr2=0x7f20880b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:29:04.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 shutdown_connections 2023-03-03T09:29:04.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:29:04.650+0000 7f208f1ff700 1 -- 172.21.15.102:0/3860806358 wait complete. 2023-03-03T09:29:04.653 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:29:04.678 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-03-03T09:29:21.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:21.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:51.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:51.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T09:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:30:21.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:21.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:51.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:51.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:57.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:30:57.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.942+0000 7f4c4f59d700 1 Processor -- start 2023-03-03T09:30:57.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.943+0000 7f4c4f59d700 1 -- start start 2023-03-03T09:30:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4f59d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4c481564b0 0x7f4c480cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4f59d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c480cab50 0x7f4c480cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4f59d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c480c9ca0 con 0x7f4c480c9610 2023-03-03T09:30:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4f59d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c480b4920 con 0x7f4c480cab50 2023-03-03T09:30:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4f59d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c480b4c00 con 0x7f4c481564b0 2023-03-03T09:30:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4e59b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4c481564b0 0x7f4c480cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4dd9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4c480c9610 0x7f4c480ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45094/0 (socket says 172.21.15.102:45094) 2023-03-03T09:30:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4dd9a700 1 -- 172.21.15.102:0/2322408609 learned_addr learned my addr 172.21.15.102:0/2322408609 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:30:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.944+0000 7f4c4ed9c700 1 --2- 172.21.15.102:0/2322408609 >> v2:172.21.15.102:3300/0 conn(0x7f4c480cab50 0x7f4c480cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:57.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4e59b700 1 -- 172.21.15.102:0/2322408609 >> v2:172.21.15.102:3300/0 conn(0x7f4c480cab50 msgr2=0x7f4c480cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:30:57.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4e59b700 1 --2- 172.21.15.102:0/2322408609 >> v2:172.21.15.102:3300/0 conn(0x7f4c480cab50 0x7f4c480cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:57.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4d599700 1 -- 172.21.15.102:0/2322408609 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532502598 0 0) 0x7f4c480c9ca0 con 0x7f4c480c9610 2023-03-03T09:30:57.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4e59b700 1 -- 172.21.15.102:0/2322408609 >> v1:172.21.15.172:6789/0 conn(0x7f4c480c9610 legacy=0x7f4c480ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:57.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4e59b700 1 -- 172.21.15.102:0/2322408609 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c480b46f0 con 0x7f4c481564b0 2023-03-03T09:30:57.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4e59b700 1 --2- 172.21.15.102:0/2322408609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c481564b0 0x7f4c480cb660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4c4808c600 tx=0x7f4c44009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9034e1debf15dc3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:57.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4d599700 1 -- 172.21.15.102:0/2322408609 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44011830 con 0x7f4c481564b0 2023-03-03T09:30:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4d599700 1 -- 172.21.15.102:0/2322408609 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44011990 con 0x7f4c481564b0 2023-03-03T09:30:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.945+0000 7f4c4d599700 1 -- 172.21.15.102:0/2322408609 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44011d20 con 0x7f4c481564b0 2023-03-03T09:30:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2322408609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c481564b0 msgr2=0x7f4c480cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2322408609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c481564b0 0x7f4c480cb660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4c4808c600 tx=0x7f4c44009b20 comp rx=0 tx=0).stop 2023-03-03T09:30:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2322408609 shutdown_connections 2023-03-03T09:30:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2322408609 >> v2:172.21.15.102:3300/0 conn(0x7f4c480cab50 0x7f4c480cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:57.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2322408609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c481564b0 0x7f4c480cb660 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:57.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2322408609 >> 172.21.15.102:0/2322408609 conn(0x7f4c48065fb0 msgr2=0x7f4c480663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:30:57.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2322408609 shutdown_connections 2023-03-03T09:30:57.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.946+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2322408609 wait complete. 2023-03-03T09:30:57.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.948+0000 7f4c4f59d700 1 Processor -- start 2023-03-03T09:30:57.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.948+0000 7f4c4f59d700 1 -- start start 2023-03-03T09:30:57.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.949+0000 7f4c4f59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c480c9610 0x7f4c4815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.949+0000 7f4c4f59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c480cab50 0x7f4c480c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.949+0000 7f4c4f59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c481564b0 0x7f4c4815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.949+0000 7f4c4f59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4815a620 con 0x7f4c481564b0 2023-03-03T09:30:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.949+0000 7f4c4f59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c48152b40 con 0x7f4c480cab50 2023-03-03T09:30:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.949+0000 7f4c4f59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c481ee960 con 0x7f4c480c9610 2023-03-03T09:30:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c480c9610 0x7f4c4815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4dd9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c480cab50 0x7f4c480c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4ed9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c481564b0 0x7f4c4815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c480c9610 0x7f4c4815cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48476/0 (socket says 172.21.15.102:48476) 2023-03-03T09:30:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 -- 172.21.15.102:0/2014090797 learned_addr learned my addr 172.21.15.102:0/2014090797 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:30:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 -- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c480cab50 msgr2=0x7f4c480c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c480cab50 0x7f4c480c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 -- 172.21.15.102:0/2014090797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c481564b0 msgr2=0x7f4c4815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c481564b0 0x7f4c4815aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:57.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.950+0000 7f4c4e59b700 1 -- 172.21.15.102:0/2014090797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c440094a0 con 0x7f4c480c9610 2023-03-03T09:30:57.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.951+0000 7f4c4e59b700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c480c9610 0x7f4c4815cd70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c440090b0 tx=0x7f4c44004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:57.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.951+0000 7f4c3effd700 1 -- 172.21.15.102:0/2014090797 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440036a0 con 0x7f4c480c9610 2023-03-03T09:30:57.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.951+0000 7f4c3effd700 1 -- 172.21.15.102:0/2014090797 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4401e070 con 0x7f4c480c9610 2023-03-03T09:30:57.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.951+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c480b2210 con 0x7f4c480c9610 2023-03-03T09:30:57.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.951+0000 7f4c3effd700 1 -- 172.21.15.102:0/2014090797 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44003a20 con 0x7f4c480c9610 2023-03-03T09:30:57.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.951+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c480b1f30 con 0x7f4c480c9610 2023-03-03T09:30:57.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.952+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c48156880 con 0x7f4c480c9610 2023-03-03T09:30:57.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.953+0000 7f4c3effd700 1 -- 172.21.15.102:0/2014090797 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c4400f050 con 0x7f4c480c9610 2023-03-03T09:30:57.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.954+0000 7f4c3effd700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c2c0682f0 0x7f4c2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:57.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.955+0000 7f4c3effd700 1 -- 172.21.15.102:0/2014090797 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4c44086d70 con 0x7f4c480c9610 2023-03-03T09:30:57.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.955+0000 7f4c4dd9a700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c2c0682f0 0x7f4c2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:57.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.955+0000 7f4c4dd9a700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c2c0682f0 0x7f4c2c06a7a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4c480c80a0 tx=0x7f4c38007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:57.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:57.959+0000 7f4c3effd700 1 -- 172.21.15.102:0/2014090797 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c44053400 con 0x7f4c480c9610 2023-03-03T09:30:58.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.192+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c4815a890 con 0x7f4c480c9610 2023-03-03T09:30:58.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.193+0000 7f4c3effd700 1 -- 172.21.15.102:0/2014090797 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4c44015090 con 0x7f4c480c9610 2023-03-03T09:30:58.194 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:30:58.194 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:30:58.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c2c0682f0 msgr2=0x7f4c2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c2c0682f0 0x7f4c2c06a7a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4c480c80a0 tx=0x7f4c38007de0 comp rx=0 tx=0).stop 2023-03-03T09:30:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c480c9610 msgr2=0x7f4c4815cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c480c9610 0x7f4c4815cd70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c440090b0 tx=0x7f4c44004750 comp rx=0 tx=0).stop 2023-03-03T09:30:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 shutdown_connections 2023-03-03T09:30:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4c2c0682f0 0x7f4c2c06a7a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c480c9610 0x7f4c4815cd70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c481564b0 0x7f4c4815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.195+0000 7f4c4f59d700 1 --2- 172.21.15.102:0/2014090797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c480cab50 0x7f4c480c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.196+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 >> 172.21.15.102:0/2014090797 conn(0x7f4c48065fb0 msgr2=0x7f4c481547e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:30:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.196+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 shutdown_connections 2023-03-03T09:30:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:30:58.196+0000 7f4c4f59d700 1 -- 172.21.15.102:0/2014090797 wait complete. 2023-03-03T09:30:58.199 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:30:58.224 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-03-03T09:31:21.810 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:21.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:52.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:52.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:22.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:22.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:33.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:32:33.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.492+0000 7fe1a03ff700 1 Processor -- start 2023-03-03T09:32:33.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.492+0000 7fe1a03ff700 1 -- start start 2023-03-03T09:32:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.493+0000 7fe1a03ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe1980cd8e0 0x7fe1980cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.493+0000 7fe1a03ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1980cd390 0x7fe1980ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.493+0000 7fe1a03ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe198153660 con 0x7fe198153290 2023-03-03T09:32:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.493+0000 7fe1a03ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19806c360 con 0x7fe1980cd8e0 2023-03-03T09:32:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.493+0000 7fe1a03ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19815cea0 con 0x7fe1980cd390 2023-03-03T09:32:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.493+0000 7fe19d99a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe1980cd8e0 0x7fe1980cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19e19b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe198153290 0x7fe198158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45690/0 (socket says 172.21.15.102:45690) 2023-03-03T09:32:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19e19b700 1 -- 172.21.15.102:0/1914405684 learned_addr learned my addr 172.21.15.102:0/1914405684 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:32:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19e99c700 1 --2- 172.21.15.102:0/1914405684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1980cd390 0x7fe1980ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19d99a700 1 -- 172.21.15.102:0/1914405684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1980cd390 msgr2=0x7fe1980ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:32:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19d99a700 1 --2- 172.21.15.102:0/1914405684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1980cd390 0x7fe1980ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19d99a700 1 -- 172.21.15.102:0/1914405684 >> v1:172.21.15.102:6789/0 conn(0x7fe198153290 legacy=0x7fe198158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19d99a700 1 -- 172.21.15.102:0/1914405684 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19806c7f0 con 0x7fe1980cd8e0 2023-03-03T09:32:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19d199700 1 -- 172.21.15.102:0/1914405684 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149666127 0 0) 0x7fe198153660 con 0x7fe198153290 2023-03-03T09:32:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.494+0000 7fe19d99a700 1 --2- 172.21.15.102:0/1914405684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd8e0 0x7fe1980cde20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe188009840 tx=0x7fe188009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d7a5ef3842aacb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:32:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe19d199700 1 -- 172.21.15.102:0/1914405684 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188011830 con 0x7fe1980cd8e0 2023-03-03T09:32:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe19d199700 1 -- 172.21.15.102:0/1914405684 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe188011990 con 0x7fe1980cd8e0 2023-03-03T09:32:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe19d199700 1 -- 172.21.15.102:0/1914405684 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188011d20 con 0x7fe1980cd8e0 2023-03-03T09:32:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe1a03ff700 1 -- 172.21.15.102:0/1914405684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd8e0 msgr2=0x7fe1980cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/1914405684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd8e0 0x7fe1980cde20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe188009840 tx=0x7fe188009b50 comp rx=0 tx=0).stop 2023-03-03T09:32:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe1a03ff700 1 -- 172.21.15.102:0/1914405684 shutdown_connections 2023-03-03T09:32:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/1914405684 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1980cd390 0x7fe1980ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/1914405684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd8e0 0x7fe1980cde20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe1a03ff700 1 -- 172.21.15.102:0/1914405684 >> 172.21.15.102:0/1914405684 conn(0x7fe198065fb0 msgr2=0x7fe1980663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:32:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.495+0000 7fe1a03ff700 1 -- 172.21.15.102:0/1914405684 shutdown_connections 2023-03-03T09:32:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.496+0000 7fe1a03ff700 1 -- 172.21.15.102:0/1914405684 wait complete. 2023-03-03T09:32:33.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.498+0000 7fe1a03ff700 1 Processor -- start 2023-03-03T09:32:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.498+0000 7fe1a03ff700 1 -- start start 2023-03-03T09:32:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe1a03ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd390 0x7fe1981f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe1a03ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1980cd8e0 0x7fe198158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe1a03ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe198153290 0x7fe19814e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe1a03ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980c6880 con 0x7fe1980cd390 2023-03-03T09:32:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe1a03ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980c62c0 con 0x7fe1980cd8e0 2023-03-03T09:32:33.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe1a03ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980c65a0 con 0x7fe198153290 2023-03-03T09:32:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe19e19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd390 0x7fe1981f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe19e19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd390 0x7fe1981f1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48324/0 (socket says 172.21.15.102:48324) 2023-03-03T09:32:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe19e19b700 1 -- 172.21.15.102:0/819449082 learned_addr learned my addr 172.21.15.102:0/819449082 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:32:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe19d99a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1980cd8e0 0x7fe198158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.499+0000 7fe19e99c700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe198153290 0x7fe19814e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe19e19b700 1 -- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe198153290 msgr2=0x7fe19814e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe19e19b700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe198153290 0x7fe19814e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe19e19b700 1 -- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1980cd8e0 msgr2=0x7fe198158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe19e19b700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1980cd8e0 0x7fe198158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe19e19b700 1 -- 172.21.15.102:0/819449082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880094a0 con 0x7fe1980cd390 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe19d99a700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1980cd8e0 0x7fe198158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe19e19b700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd390 0x7fe1981f1c10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe1940011e0 tx=0x7fe19400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe18effd700 1 -- 172.21.15.102:0/819449082 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194004c80 con 0x7fe1980cd390 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe18effd700 1 -- 172.21.15.102:0/819449082 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe194004e00 con 0x7fe1980cd390 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe18effd700 1 -- 172.21.15.102:0/819449082 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194007310 con 0x7fe1980cd390 2023-03-03T09:32:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.500+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1980ccde0 con 0x7fe1980cd390 2023-03-03T09:32:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.501+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19815d160 con 0x7fe1980cd390 2023-03-03T09:32:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.502+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1980cd0c0 con 0x7fe1980cd390 2023-03-03T09:32:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.503+0000 7fe18effd700 1 -- 172.21.15.102:0/819449082 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe19401a020 con 0x7fe1980cd390 2023-03-03T09:32:33.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.504+0000 7fe18effd700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe17c0682f0 0x7fe17c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.505+0000 7fe18effd700 1 -- 172.21.15.102:0/819449082 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe194085880 con 0x7fe1980cd390 2023-03-03T09:32:33.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.505+0000 7fe19d99a700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe17c0682f0 0x7fe17c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:33.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.506+0000 7fe19d99a700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe17c0682f0 0x7fe17c06a7a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe1880090b0 tx=0x7fe18801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:32:33.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.508+0000 7fe18effd700 1 -- 172.21.15.102:0/819449082 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe194051f90 con 0x7fe1980cd390 2023-03-03T09:32:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.744+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe19804e6e0 con 0x7fe1980cd390 2023-03-03T09:32:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.744+0000 7fe18effd700 1 -- 172.21.15.102:0/819449082 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe1940555b0 con 0x7fe1980cd390 2023-03-03T09:32:33.746 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:32:33.746 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:32:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe17c0682f0 msgr2=0x7fe17c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe17c0682f0 0x7fe17c06a7a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe1880090b0 tx=0x7fe18801c040 comp rx=0 tx=0).stop 2023-03-03T09:32:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd390 msgr2=0x7fe1981f1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd390 0x7fe1981f1c10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe1940011e0 tx=0x7fe19400ca30 comp rx=0 tx=0).stop 2023-03-03T09:32:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 shutdown_connections 2023-03-03T09:32:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe17c0682f0 0x7fe17c06a7a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe198153290 0x7fe19814e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1980cd390 0x7fe1981f1c10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 --2- 172.21.15.102:0/819449082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1980cd8e0 0x7fe198158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 >> 172.21.15.102:0/819449082 conn(0x7fe198065fb0 msgr2=0x7fe1981575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:32:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.747+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 shutdown_connections 2023-03-03T09:32:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:32:33.748+0000 7fe1a03ff700 1 -- 172.21.15.102:0/819449082 wait complete. 2023-03-03T09:32:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:32:33.775 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T09:32:52.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:52.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:01.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:33:02.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.025+0000 7fc7003ff700 1 Processor -- start 2023-03-03T09:33:02.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.025+0000 7fc7003ff700 1 -- start start 2023-03-03T09:33:02.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.026+0000 7fc7003ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f80cd390 0x7fc6f80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:02.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.026+0000 7fc7003ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f8153660 con 0x7fc6f8153290 2023-03-03T09:33:02.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.026+0000 7fc7003ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f806c360 con 0x7fc6f80cd8e0 2023-03-03T09:33:02.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.026+0000 7fc7003ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f815cea0 con 0x7fc6f80cd390 2023-03-03T09:33:02.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fe19b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc6f8153290 0x7fc6f8158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39736/0 (socket says 172.21.15.102:39736) 2023-03-03T09:33:02.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fe19b700 1 -- 172.21.15.102:0/3787132821 learned_addr learned my addr 172.21.15.102:0/3787132821 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:33:02.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fe99c700 1 --2- 172.21.15.102:0/3787132821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f80cd390 0x7fc6f80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:02.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fd199700 1 -- 172.21.15.102:0/3787132821 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592137604 0 0) 0x7fc6f8153660 con 0x7fc6f8153290 2023-03-03T09:33:02.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fd199700 1 -- 172.21.15.102:0/3787132821 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e40035b0 con 0x7fc6f8153290 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fd199700 1 -- 172.21.15.102:0/3787132821 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476701300 0 0) 0x7fc6f806c360 con 0x7fc6f80cd8e0 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fd199700 1 -- 172.21.15.102:0/3787132821 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f8153660 con 0x7fc6f80cd8e0 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fe99c700 1 -- 172.21.15.102:0/3787132821 >> v1:172.21.15.172:6789/0 conn(0x7fc6f80cd8e0 legacy=0x7fc6f80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fe99c700 1 -- 172.21.15.102:0/3787132821 >> v1:172.21.15.102:6789/0 conn(0x7fc6f8153290 legacy=0x7fc6f8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.027+0000 7fc6fe99c700 1 -- 172.21.15.102:0/3787132821 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f806c7f0 con 0x7fc6f80cd390 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc6fe99c700 1 --2- 172.21.15.102:0/3787132821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f80cd390 0x7fc6f80ce530 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009dc0 tx=0x7fc6f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcad8637d5da42ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc6fd199700 1 -- 172.21.15.102:0/3787132821 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f000e070 con 0x7fc6f80cd390 2023-03-03T09:33:02.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc6fd199700 1 -- 172.21.15.102:0/3787132821 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f000bb20 con 0x7fc6f80cd390 2023-03-03T09:33:02.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc6fd199700 1 -- 172.21.15.102:0/3787132821 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0012430 con 0x7fc6f80cd390 2023-03-03T09:33:02.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc7003ff700 1 -- 172.21.15.102:0/3787132821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f80cd390 msgr2=0x7fc6f80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc7003ff700 1 --2- 172.21.15.102:0/3787132821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f80cd390 0x7fc6f80ce530 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009dc0 tx=0x7fc6f000b330 comp rx=0 tx=0).stop 2023-03-03T09:33:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc7003ff700 1 -- 172.21.15.102:0/3787132821 shutdown_connections 2023-03-03T09:33:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc7003ff700 1 --2- 172.21.15.102:0/3787132821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f80cd390 0x7fc6f80ce530 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.028+0000 7fc7003ff700 1 -- 172.21.15.102:0/3787132821 >> 172.21.15.102:0/3787132821 conn(0x7fc6f8065fb0 msgr2=0x7fc6f80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.029+0000 7fc7003ff700 1 -- 172.21.15.102:0/3787132821 shutdown_connections 2023-03-03T09:33:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.029+0000 7fc7003ff700 1 -- 172.21.15.102:0/3787132821 wait complete. 2023-03-03T09:33:02.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.031+0000 7fc7003ff700 1 Processor -- start 2023-03-03T09:33:02.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.031+0000 7fc7003ff700 1 -- start start 2023-03-03T09:33:02.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc7003ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f80cd390 0x7fc6f80c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc7003ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6f80cd8e0 0x7fc6f814e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc7003ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f8153290 0x7fc6f80c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc7003ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f8154360 con 0x7fc6f80cd390 2023-03-03T09:33:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc7003ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f814ea20 con 0x7fc6f80cd8e0 2023-03-03T09:33:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc7003ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f80c91c0 con 0x7fc6f8153290 2023-03-03T09:33:02.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc6fe19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f80cd390 0x7fc6f80c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:02.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc6fe19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f80cd390 0x7fc6f80c7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48864/0 (socket says 172.21.15.102:48864) 2023-03-03T09:33:02.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc6fe19b700 1 -- 172.21.15.102:0/777920631 learned_addr learned my addr 172.21.15.102:0/777920631 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:33:02.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc6fe99c700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f8153290 0x7fc6f80c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:02.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.032+0000 7fc6fd99a700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6f80cd8e0 0x7fc6f814e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:02.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6fe19b700 1 -- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f8153290 msgr2=0x7fc6f80c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:02.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6fe19b700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f8153290 0x7fc6f80c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:02.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6fe19b700 1 -- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6f80cd8e0 msgr2=0x7fc6f814e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:02.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6fe19b700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6f80cd8e0 0x7fc6f814e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:02.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6fe19b700 1 -- 172.21.15.102:0/777920631 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f0009a70 con 0x7fc6f80cd390 2023-03-03T09:33:02.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6fe19b700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f80cd390 0x7fc6f80c7da0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400c700 tx=0x7fc6f400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:02.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6eeffd700 1 -- 172.21.15.102:0/777920631 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4005d70 con 0x7fc6f80cd390 2023-03-03T09:33:02.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6eeffd700 1 -- 172.21.15.102:0/777920631 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f40070d0 con 0x7fc6f80cd390 2023-03-03T09:33:02.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f80ccde0 con 0x7fc6f80cd390 2023-03-03T09:33:02.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.033+0000 7fc6eeffd700 1 -- 172.21.15.102:0/777920631 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4007480 con 0x7fc6f80cd390 2023-03-03T09:33:02.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.034+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f815d160 con 0x7fc6f80cd390 2023-03-03T09:33:02.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.035+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f80cd0c0 con 0x7fc6f80cd390 2023-03-03T09:33:02.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.036+0000 7fc6eeffd700 1 -- 172.21.15.102:0/777920631 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc6f4019030 con 0x7fc6f80cd390 2023-03-03T09:33:02.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.038+0000 7fc6eeffd700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6dc0682f0 0x7fc6dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:02.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.038+0000 7fc6eeffd700 1 -- 172.21.15.102:0/777920631 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc6f4085130 con 0x7fc6f80cd390 2023-03-03T09:33:02.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.039+0000 7fc6fd99a700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6dc0682f0 0x7fc6dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:02.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.039+0000 7fc6fd99a700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6dc0682f0 0x7fc6dc06a7a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80c82e0 tx=0x7fc6e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:02.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.043+0000 7fc6eeffd700 1 -- 172.21.15.102:0/777920631 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6f40517b0 con 0x7fc6f80cd390 2023-03-03T09:33:02.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.276+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6f804e6e0 con 0x7fc6f80cd390 2023-03-03T09:33:02.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.277+0000 7fc6eeffd700 1 -- 172.21.15.102:0/777920631 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc6f4017070 con 0x7fc6f80cd390 2023-03-03T09:33:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:33:02.279 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:33:02.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6dc0682f0 msgr2=0x7fc6dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:02.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6dc0682f0 0x7fc6dc06a7a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80c82e0 tx=0x7fc6e8009040 comp rx=0 tx=0).stop 2023-03-03T09:33:02.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f80cd390 msgr2=0x7fc6f80c7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:02.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f80cd390 0x7fc6f80c7da0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400c700 tx=0x7fc6f400ca10 comp rx=0 tx=0).stop 2023-03-03T09:33:02.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 shutdown_connections 2023-03-03T09:33:02.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6dc0682f0 0x7fc6dc06a7a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:02.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6f8153290 0x7fc6f80c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:02.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f80cd390 0x7fc6f80c7da0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:02.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 --2- 172.21.15.102:0/777920631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6f80cd8e0 0x7fc6f814e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:02.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 >> 172.21.15.102:0/777920631 conn(0x7fc6f8065fb0 msgr2=0x7fc6f81585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:02.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 shutdown_connections 2023-03-03T09:33:02.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:33:02.280+0000 7fc7003ff700 1 -- 172.21.15.102:0/777920631 wait complete. 2023-03-03T09:33:02.283 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:33:02.308 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T09:33:22.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:22.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:52.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:52.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:22.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:22.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:29.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:34:29.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.564+0000 7f1ac29ff700 1 Processor -- start 2023-03-03T09:34:29.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.564+0000 7f1ac29ff700 1 -- start start 2023-03-03T09:34:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.565+0000 7f1ac29ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0c9cb0 0x7f1abc0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.565+0000 7f1ac29ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1abc0ca190 0x7f1abc0cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:29.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.565+0000 7f1ac29ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1abc0cf940 con 0x7f1abc153260 2023-03-03T09:34:29.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.565+0000 7f1ac29ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0cfe70 con 0x7f1abc0ca190 2023-03-03T09:34:29.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.565+0000 7f1ac29ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc158d70 con 0x7f1abc0c9cb0 2023-03-03T09:34:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1abbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1abc153260 0x7f1abc0ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35970/0 (socket says 172.21.15.102:35970) 2023-03-03T09:34:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1abbfff700 1 -- 172.21.15.102:0/1140216219 learned_addr learned my addr 172.21.15.102:0/1140216219 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:34:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1ac0f9c700 1 --2- 172.21.15.102:0/1140216219 >> v2:172.21.15.102:3300/0 conn(0x7f1abc0ca190 0x7f1abc0cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1abb7fe700 1 --2- 172.21.15.102:0/1140216219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0c9cb0 0x7f1abc0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1abaffd700 1 -- 172.21.15.102:0/1140216219 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982671314 0 0) 0x7f1abc0cf940 con 0x7f1abc153260 2023-03-03T09:34:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1ac0f9c700 1 -- 172.21.15.102:0/1140216219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0c9cb0 msgr2=0x7f1abc0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:34:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1ac0f9c700 1 --2- 172.21.15.102:0/1140216219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0c9cb0 0x7f1abc0cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1ac0f9c700 1 -- 172.21.15.102:0/1140216219 >> v1:172.21.15.102:6789/0 conn(0x7f1abc153260 legacy=0x7f1abc0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.566+0000 7f1ac0f9c700 1 -- 172.21.15.102:0/1140216219 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0cfb90 con 0x7f1abc0ca190 2023-03-03T09:34:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.567+0000 7f1ac0f9c700 1 --2- 172.21.15.102:0/1140216219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0ca190 0x7f1abc0cbdc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009d90 tx=0x7f1ab400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428b41b07edf9960 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.567+0000 7f1abaffd700 1 -- 172.21.15.102:0/1140216219 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab400e070 con 0x7f1abc0ca190 2023-03-03T09:34:29.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.567+0000 7f1abaffd700 1 -- 172.21.15.102:0/1140216219 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab400bb20 con 0x7f1abc0ca190 2023-03-03T09:34:29.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.567+0000 7f1abaffd700 1 -- 172.21.15.102:0/1140216219 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4012430 con 0x7f1abc0ca190 2023-03-03T09:34:29.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.567+0000 7f1ac29ff700 1 -- 172.21.15.102:0/1140216219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0ca190 msgr2=0x7f1abc0cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.567+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/1140216219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0ca190 0x7f1abc0cbdc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009d90 tx=0x7f1ab400b330 comp rx=0 tx=0).stop 2023-03-03T09:34:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.568+0000 7f1ac29ff700 1 -- 172.21.15.102:0/1140216219 shutdown_connections 2023-03-03T09:34:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.568+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/1140216219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0ca190 0x7f1abc0cbdc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.568+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/1140216219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0c9cb0 0x7f1abc0cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.568+0000 7f1ac29ff700 1 -- 172.21.15.102:0/1140216219 >> 172.21.15.102:0/1140216219 conn(0x7f1abc066c00 msgr2=0x7f1abc0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:34:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.568+0000 7f1ac29ff700 1 -- 172.21.15.102:0/1140216219 shutdown_connections 2023-03-03T09:34:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.568+0000 7f1ac29ff700 1 -- 172.21.15.102:0/1140216219 wait complete. 2023-03-03T09:34:29.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.570+0000 7f1ac29ff700 1 Processor -- start 2023-03-03T09:34:29.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.570+0000 7f1ac29ff700 1 -- start start 2023-03-03T09:34:29.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac29ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0ca190 0x7f1abc14da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:29.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac29ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1abc153260 0x7f1abc14e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:29.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac29ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc154730 0x7f1abc14e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:29.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac29ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc14dfc0 con 0x7f1abc154730 2023-03-03T09:34:29.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac29ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc14c650 con 0x7f1abc153260 2023-03-03T09:34:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac29ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0c5b60 con 0x7f1abc0ca190 2023-03-03T09:34:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac0f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc154730 0x7f1abc14e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1abb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1abc153260 0x7f1abc14e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1abbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0ca190 0x7f1abc14da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1abb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1abc153260 0x7f1abc14e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:41730/0 (socket says 172.21.15.102:41730) 2023-03-03T09:34:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1abb7fe700 1 -- 172.21.15.102:0/2138605169 learned_addr learned my addr 172.21.15.102:0/2138605169 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:34:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.571+0000 7f1ac0f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc154730 0x7f1abc14e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59924/0 (socket says 172.21.15.102:59924) 2023-03-03T09:34:29.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1abb7fe700 1 -- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0ca190 msgr2=0x7f1abc14da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:29.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1abb7fe700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0ca190 0x7f1abc14da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1abb7fe700 1 -- 172.21.15.102:0/2138605169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc154730 msgr2=0x7f1abc14e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:29.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1abb7fe700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc154730 0x7f1abc14e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1abb7fe700 1 -- 172.21.15.102:0/2138605169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4009a70 con 0x7f1abc153260 2023-03-03T09:34:29.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1ac0f9c700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc154730 0x7f1abc14e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:34:29.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1abb7fe700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1abc153260 0x7f1abc14e3d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e8a0 tx=0x7f1ab000ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:29.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1ab8ff9700 1 -- 172.21.15.102:0/2138605169 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0009590 con 0x7f1abc153260 2023-03-03T09:34:29.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1ab8ff9700 1 -- 172.21.15.102:0/2138605169 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab0013680 con 0x7f1abc153260 2023-03-03T09:34:29.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc159260 con 0x7f1abc153260 2023-03-03T09:34:29.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.572+0000 7f1ab8ff9700 1 -- 172.21.15.102:0/2138605169 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00139f0 con 0x7f1abc153260 2023-03-03T09:34:29.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.573+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc158f80 con 0x7f1abc153260 2023-03-03T09:34:29.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.574+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc159540 con 0x7f1abc153260 2023-03-03T09:34:29.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.575+0000 7f1ab8ff9700 1 -- 172.21.15.102:0/2138605169 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1ab0019030 con 0x7f1abc153260 2023-03-03T09:34:29.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.576+0000 7f1ab8ff9700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1aa8068320 0x7f1aa806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:29.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.577+0000 7f1ab8ff9700 1 -- 172.21.15.102:0/2138605169 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1ab0085a90 con 0x7f1abc153260 2023-03-03T09:34:29.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.577+0000 7f1abbfff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1aa8068320 0x7f1aa806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:29.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.577+0000 7f1abbfff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1aa8068320 0x7f1aa806a7d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1aac007fd0 tx=0x7f1aac007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:29.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.581+0000 7f1ab8ff9700 1 -- 172.21.15.102:0/2138605169 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ab00521a0 con 0x7f1abc153260 2023-03-03T09:34:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.814+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1abc067ce0 con 0x7f1abc153260 2023-03-03T09:34:29.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.815+0000 7f1ab8ff9700 1 -- 172.21.15.102:0/2138605169 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1ab00557c0 con 0x7f1abc153260 2023-03-03T09:34:29.817 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:34:29.817 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:34:29.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1aa8068320 msgr2=0x7f1aa806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:29.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1aa8068320 0x7f1aa806a7d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1aac007fd0 tx=0x7f1aac007de0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1abc153260 msgr2=0x7f1abc14e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:29.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1abc153260 0x7f1abc14e3d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e8a0 tx=0x7f1ab000ebb0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 shutdown_connections 2023-03-03T09:34:29.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1aa8068320 0x7f1aa806a7d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1abc0ca190 0x7f1abc14da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc154730 0x7f1abc14e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 --2- 172.21.15.102:0/2138605169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1abc153260 0x7f1abc14e3d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:29.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 >> 172.21.15.102:0/2138605169 conn(0x7f1abc0c9cb0 msgr2=0x7f1abc0cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:34:29.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 shutdown_connections 2023-03-03T09:34:29.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:34:29.818+0000 7f1ac29ff700 1 -- 172.21.15.102:0/2138605169 wait complete. 2023-03-03T09:34:29.821 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:34:29.847 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T09:34:53.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:53.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:23.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:23.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:31.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:35:32.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.099+0000 7fe5253ff700 1 Processor -- start 2023-03-03T09:35:32.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.099+0000 7fe5253ff700 1 -- start start 2023-03-03T09:35:32.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.100+0000 7fe5253ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:32.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.100+0000 7fe5253ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520153660 con 0x7fe5201593c0 2023-03-03T09:35:32.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.100+0000 7fe5253ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52006c360 con 0x7fe5200cd8e0 2023-03-03T09:35:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.100+0000 7fe5253ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200cec40 con 0x7fe520153290 2023-03-03T09:35:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51e7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe5201593c0 0x7fe5200cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36792/0 (socket says 172.21.15.102:36792) 2023-03-03T09:35:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51e7fc700 1 -- 172.21.15.102:0/3864776755 learned_addr learned my addr 172.21.15.102:0/3864776755 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:35:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51effd700 1 --2- 172.21.15.102:0/3864776755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51dffb700 1 -- 172.21.15.102:0/3864776755 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158340250 0 0) 0x7fe52006c360 con 0x7fe5200cd8e0 2023-03-03T09:35:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51dffb700 1 -- 172.21.15.102:0/3864776755 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5080035b0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51effd700 1 -- 172.21.15.102:0/3864776755 >> v1:172.21.15.172:6789/0 conn(0x7fe5200cd8e0 legacy=0x7fe5200ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51effd700 1 -- 172.21.15.102:0/3864776755 >> v1:172.21.15.102:6789/0 conn(0x7fe5201593c0 legacy=0x7fe5200cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51dffb700 1 -- 172.21.15.102:0/3864776755 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260092088 0 0) 0x7fe520153660 con 0x7fe5201593c0 2023-03-03T09:35:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.101+0000 7fe51effd700 1 -- 172.21.15.102:0/3864776755 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52015cea0 con 0x7fe520153290 2023-03-03T09:35:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe51effd700 1 --2- 172.21.15.102:0/3864776755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520158e80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe510009870 tx=0x7fe510009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bfb403a688047d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe51dffb700 1 -- 172.21.15.102:0/3864776755 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510011830 con 0x7fe520153290 2023-03-03T09:35:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe51dffb700 1 -- 172.21.15.102:0/3864776755 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe510011990 con 0x7fe520153290 2023-03-03T09:35:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe51dffb700 1 -- 172.21.15.102:0/3864776755 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510011d20 con 0x7fe520153290 2023-03-03T09:35:32.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe5253ff700 1 -- 172.21.15.102:0/3864776755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 msgr2=0x7fe520158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:32.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe5253ff700 1 --2- 172.21.15.102:0/3864776755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520158e80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe510009870 tx=0x7fe510009b80 comp rx=0 tx=0).stop 2023-03-03T09:35:32.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe5253ff700 1 -- 172.21.15.102:0/3864776755 shutdown_connections 2023-03-03T09:35:32.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe5253ff700 1 --2- 172.21.15.102:0/3864776755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520158e80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe5253ff700 1 -- 172.21.15.102:0/3864776755 >> 172.21.15.102:0/3864776755 conn(0x7fe520065fb0 msgr2=0x7fe5200663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:35:32.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.102+0000 7fe5253ff700 1 -- 172.21.15.102:0/3864776755 shutdown_connections 2023-03-03T09:35:32.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.103+0000 7fe5253ff700 1 -- 172.21.15.102:0/3864776755 wait complete. 2023-03-03T09:35:32.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.104+0000 7fe5253ff700 1 Processor -- start 2023-03-03T09:35:32.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.105+0000 7fe5253ff700 1 -- start start 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.105+0000 7fe5253ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5200cd8e0 0x7fe5201e55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe5253ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe5253ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5201593c0 0x7fe520150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe5253ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200c8d20 con 0x7fe5201593c0 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe5253ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200b3800 con 0x7fe5200cd8e0 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe5253ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200c68a0 con 0x7fe520153290 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5200cd8e0 0x7fe5201e55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:32.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5201593c0 0x7fe520150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:32.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51e7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:32.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5200cd8e0 0x7fe5201e55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36776/0 (socket says 172.21.15.102:36776) 2023-03-03T09:35:32.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 -- 172.21.15.102:0/2472244830 learned_addr learned my addr 172.21.15.102:0/2472244830 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:35:32.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 -- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 msgr2=0x7fe520150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:32.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 -- 172.21.15.102:0/2472244830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5201593c0 msgr2=0x7fe520150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:32.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5201593c0 0x7fe520150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51effd700 1 -- 172.21.15.102:0/2472244830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5100094a0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.106+0000 7fe51e7fc700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:35:32.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.107+0000 7fe51effd700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5200cd8e0 0x7fe5201e55d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe5100090b0 tx=0x7fe510004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:32.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.107+0000 7fe5077fe700 1 -- 172.21.15.102:0/2472244830 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51001a9f0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.107+0000 7fe5077fe700 1 -- 172.21.15.102:0/2472244830 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51001ab50 con 0x7fe5200cd8e0 2023-03-03T09:35:32.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.107+0000 7fe5077fe700 1 -- 172.21.15.102:0/2472244830 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5100243f0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.107+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52015d160 con 0x7fe5200cd8e0 2023-03-03T09:35:32.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.107+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52006c7f0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.109+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5200ccde0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.110+0000 7fe5077fe700 1 -- 172.21.15.102:0/2472244830 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe51000f050 con 0x7fe5200cd8e0 2023-03-03T09:35:32.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.111+0000 7fe5077fe700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe50c0682f0 0x7fe50c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:32.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.111+0000 7fe5077fe700 1 -- 172.21.15.102:0/2472244830 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe510086fd0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.111+0000 7fe51e7fc700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe50c0682f0 0x7fe50c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:32.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.112+0000 7fe51e7fc700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe50c0682f0 0x7fe50c06a7a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe52015ad40 tx=0x7fe514007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.115+0000 7fe5077fe700 1 -- 172.21.15.102:0/2472244830 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5100536e0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.347+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe52015c8d0 con 0x7fe5200cd8e0 2023-03-03T09:35:32.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.348+0000 7fe5077fe700 1 -- 172.21.15.102:0/2472244830 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe510015070 con 0x7fe5200cd8e0 2023-03-03T09:35:32.350 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:35:32.350 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:35:32.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe50c0682f0 msgr2=0x7fe50c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:32.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe50c0682f0 0x7fe50c06a7a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe52015ad40 tx=0x7fe514007de0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5200cd8e0 msgr2=0x7fe5201e55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:32.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5200cd8e0 0x7fe5201e55d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe5100090b0 tx=0x7fe510004750 comp rx=0 tx=0).stop 2023-03-03T09:35:32.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 shutdown_connections 2023-03-03T09:35:32.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5201593c0 0x7fe520150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5200cd8e0 0x7fe5201e55d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe50c0682f0 0x7fe50c06a7a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 --2- 172.21.15.102:0/2472244830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe520153290 0x7fe520150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:32.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 >> 172.21.15.102:0/2472244830 conn(0x7fe520065fb0 msgr2=0x7fe5200cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:35:32.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 shutdown_connections 2023-03-03T09:35:32.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:32.351+0000 7fe5253ff700 1 -- 172.21.15.102:0/2472244830 wait complete. 2023-03-03T09:35:32.354 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:35:32.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T09:35:41.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:35:41.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.635+0000 7f3c46dff700 1 Processor -- start 2023-03-03T09:35:41.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.635+0000 7f3c46dff700 1 -- start start 2023-03-03T09:35:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.637+0000 7f3c46dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c400cd8e0 0x7f3c400cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.637+0000 7f3c46dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c40153660 con 0x7f3c400cd390 2023-03-03T09:35:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.637+0000 7f3c46dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4006c360 con 0x7f3c40153290 2023-03-03T09:35:41.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.637+0000 7f3c46dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4015cea0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.637+0000 7f3c3ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c400cd8e0 0x7f3c400cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:41.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.637+0000 7f3c44b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3c40153290 0x7f3c40158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36254/0 (socket says 172.21.15.102:36254) 2023-03-03T09:35:41.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.637+0000 7f3c44b9b700 1 -- 172.21.15.102:0/673876978 learned_addr learned my addr 172.21.15.102:0/673876978 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:35:41.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3f7fe700 1 -- 172.21.15.102:0/673876978 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096230294 0 0) 0x7f3c4006c360 con 0x7f3c40153290 2023-03-03T09:35:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3f7fe700 1 -- 172.21.15.102:0/673876978 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c300035b0 con 0x7f3c40153290 2023-03-03T09:35:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3ffff700 1 -- 172.21.15.102:0/673876978 >> v1:172.21.15.172:6789/0 conn(0x7f3c40153290 legacy=0x7f3c40158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3f7fe700 1 -- 172.21.15.102:0/673876978 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356299185 0 0) 0x7f3c40153660 con 0x7f3c400cd390 2023-03-03T09:35:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3ffff700 1 -- 172.21.15.102:0/673876978 >> v1:172.21.15.102:6789/0 conn(0x7f3c400cd390 legacy=0x7f3c400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3ffff700 1 -- 172.21.15.102:0/673876978 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4006c7f0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3ffff700 1 --2- 172.21.15.102:0/673876978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c400cd8e0 0x7f3c400cde20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009840 tx=0x7f3c2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4057ceed0846601b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:41.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3f7fe700 1 -- 172.21.15.102:0/673876978 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c011830 con 0x7f3c400cd8e0 2023-03-03T09:35:41.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3f7fe700 1 -- 172.21.15.102:0/673876978 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c011990 con 0x7f3c400cd8e0 2023-03-03T09:35:41.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.638+0000 7f3c3f7fe700 1 -- 172.21.15.102:0/673876978 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c011d20 con 0x7f3c400cd8e0 2023-03-03T09:35:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.639+0000 7f3c46dff700 1 -- 172.21.15.102:0/673876978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c400cd8e0 msgr2=0x7f3c400cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.639+0000 7f3c46dff700 1 --2- 172.21.15.102:0/673876978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c400cd8e0 0x7f3c400cde20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009840 tx=0x7f3c2c009b50 comp rx=0 tx=0).stop 2023-03-03T09:35:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.639+0000 7f3c46dff700 1 -- 172.21.15.102:0/673876978 shutdown_connections 2023-03-03T09:35:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.639+0000 7f3c46dff700 1 --2- 172.21.15.102:0/673876978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c400cd8e0 0x7f3c400cde20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.639+0000 7f3c46dff700 1 -- 172.21.15.102:0/673876978 >> 172.21.15.102:0/673876978 conn(0x7f3c40065fb0 msgr2=0x7f3c400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:35:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.639+0000 7f3c46dff700 1 -- 172.21.15.102:0/673876978 shutdown_connections 2023-03-03T09:35:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.639+0000 7f3c46dff700 1 -- 172.21.15.102:0/673876978 wait complete. 2023-03-03T09:35:41.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.641+0000 7f3c46dff700 1 Processor -- start 2023-03-03T09:35:41.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.641+0000 7f3c46dff700 1 -- start start 2023-03-03T09:35:41.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.642+0000 7f3c46dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c400cd390 0x7f3c400b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:41.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.642+0000 7f3c46dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c400cd8e0 0x7f3c400c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.642+0000 7f3c46dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c40153290 0x7f3c400c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.642+0000 7f3c46dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c400c9210 con 0x7f3c40153290 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.642+0000 7f3c46dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c40154750 con 0x7f3c400cd8e0 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.642+0000 7f3c46dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c400c75a0 con 0x7f3c400cd390 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c4539c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c40153290 0x7f3c400c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c400cd8e0 0x7f3c400c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c400cd8e0 0x7f3c400c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42738/0 (socket says 172.21.15.102:42738) 2023-03-03T09:35:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 -- 172.21.15.102:0/2751049866 learned_addr learned my addr 172.21.15.102:0/2751049866 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:35:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c44b9b700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c400cd390 0x7f3c400b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 -- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c400cd390 msgr2=0x7f3c400b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c400cd390 0x7f3c400b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 -- 172.21.15.102:0/2751049866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c40153290 msgr2=0x7f3c400c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c40153290 0x7f3c400c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 -- 172.21.15.102:0/2751049866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c0094a0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.643+0000 7f3c3ffff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c400cd8e0 0x7f3c400c7d10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0090b0 tx=0x7f3c2c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.644+0000 7f3c3d7fa700 1 -- 172.21.15.102:0/2751049866 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c01a7e0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.644+0000 7f3c3d7fa700 1 -- 172.21.15.102:0/2751049866 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c01a940 con 0x7f3c400cd8e0 2023-03-03T09:35:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.644+0000 7f3c3d7fa700 1 -- 172.21.15.102:0/2751049866 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c01acd0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.644+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c400ccde0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.644+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4015d160 con 0x7f3c400cd8e0 2023-03-03T09:35:41.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.645+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c400cd0c0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.646+0000 7f3c3d7fa700 1 -- 172.21.15.102:0/2751049866 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c2c02b030 con 0x7f3c400cd8e0 2023-03-03T09:35:41.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.647+0000 7f3c3d7fa700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c24068320 0x7f3c2406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:41.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.648+0000 7f3c3d7fa700 1 -- 172.21.15.102:0/2751049866 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3c2c087070 con 0x7f3c400cd8e0 2023-03-03T09:35:41.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.648+0000 7f3c44b9b700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c24068320 0x7f3c2406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:41.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.648+0000 7f3c44b9b700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c24068320 0x7f3c2406a7d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3c34007fd0 tx=0x7f3c34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:41.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.652+0000 7f3c3d7fa700 1 -- 172.21.15.102:0/2751049866 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c2c0536f0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.889+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c4004e6e0 con 0x7f3c400cd8e0 2023-03-03T09:35:41.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.890+0000 7f3c3d7fa700 1 -- 172.21.15.102:0/2751049866 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3c2c029070 con 0x7f3c400cd8e0 2023-03-03T09:35:41.892 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:35:41.892 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:35:41.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c24068320 msgr2=0x7f3c2406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c24068320 0x7f3c2406a7d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3c34007fd0 tx=0x7f3c34009040 comp rx=0 tx=0).stop 2023-03-03T09:35:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c400cd8e0 msgr2=0x7f3c400c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c400cd8e0 0x7f3c400c7d10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0090b0 tx=0x7f3c2c004750 comp rx=0 tx=0).stop 2023-03-03T09:35:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 shutdown_connections 2023-03-03T09:35:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c24068320 0x7f3c2406a7d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c400cd390 0x7f3c400b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c40153290 0x7f3c400c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:41.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 --2- 172.21.15.102:0/2751049866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c400cd8e0 0x7f3c400c7d10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:41.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.892+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 >> 172.21.15.102:0/2751049866 conn(0x7f3c40065fb0 msgr2=0x7f3c40158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:35:41.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.893+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 shutdown_connections 2023-03-03T09:35:41.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:35:41.893+0000 7f3c46dff700 1 -- 172.21.15.102:0/2751049866 wait complete. 2023-03-03T09:35:41.896 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:35:41.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T09:35:53.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:53.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:23.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:23.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:54.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:54.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:18.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:37:19.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.171+0000 7f9a67fff700 1 Processor -- start 2023-03-03T09:37:19.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.171+0000 7f9a67fff700 1 -- start start 2023-03-03T09:37:19.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a67fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:19.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a67fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:19.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a67fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a680c3d20 con 0x7f9a680ced00 2023-03-03T09:37:19.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a67fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a680c4930 con 0x7f9a680b3970 2023-03-03T09:37:19.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a67fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a680cf610 con 0x7f9a680c41b0 2023-03-03T09:37:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a667fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a66ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9a680ced00 0x7f9a68158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37256/0 (socket says 172.21.15.102:37256) 2023-03-03T09:37:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a677fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.173+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3851613312 learned_addr learned my addr 172.21.15.102:0/3851613312 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:37:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a667fc700 1 -- 172.21.15.102:0/3851613312 >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 msgr2=0x7f9a680cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:37:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a667fc700 1 --2- 172.21.15.102:0/3851613312 >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a667fc700 1 -- 172.21.15.102:0/3851613312 >> v1:172.21.15.172:6789/0 conn(0x7f9a680ced00 legacy=0x7f9a68158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a667fc700 1 -- 172.21.15.102:0/3851613312 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6815cba0 con 0x7f9a680b3970 2023-03-03T09:37:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a677fe700 1 --2- 172.21.15.102:0/3851613312 >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:37:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a667fc700 1 --2- 172.21.15.102:0/3851613312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a680c5a90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9a6808ce30 tx=0x7f9a5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33264b4fe19c6ce4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a65ffb700 1 -- 172.21.15.102:0/3851613312 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c011830 con 0x7f9a680b3970 2023-03-03T09:37:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a65ffb700 1 -- 172.21.15.102:0/3851613312 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c011990 con 0x7f9a680b3970 2023-03-03T09:37:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.174+0000 7f9a65ffb700 1 -- 172.21.15.102:0/3851613312 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c011d20 con 0x7f9a680b3970 2023-03-03T09:37:19.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 -- 172.21.15.102:0/3851613312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 msgr2=0x7f9a680c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:19.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3851613312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a680c5a90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9a6808ce30 tx=0x7f9a5c009b50 comp rx=0 tx=0).stop 2023-03-03T09:37:19.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 -- 172.21.15.102:0/3851613312 shutdown_connections 2023-03-03T09:37:19.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3851613312 >> v2:172.21.15.172:3300/0 conn(0x7f9a680c41b0 0x7f9a680cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3851613312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a680c5a90 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 -- 172.21.15.102:0/3851613312 >> 172.21.15.102:0/3851613312 conn(0x7f9a6808b1c0 msgr2=0x7f9a68071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:19.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 -- 172.21.15.102:0/3851613312 shutdown_connections 2023-03-03T09:37:19.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.175+0000 7f9a67fff700 1 -- 172.21.15.102:0/3851613312 wait complete. 2023-03-03T09:37:19.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.177+0000 7f9a67fff700 1 Processor -- start 2023-03-03T09:37:19.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.177+0000 7f9a67fff700 1 -- start start 2023-03-03T09:37:19.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.178+0000 7f9a67fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a68144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.178+0000 7f9a67fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a6814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.178+0000 7f9a67fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.178+0000 7f9a67fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68145400 con 0x7f9a680b3970 2023-03-03T09:37:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.178+0000 7f9a67fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68151be0 con 0x7f9a680c41b0 2023-03-03T09:37:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.178+0000 7f9a67fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68151ec0 con 0x7f9a680ced00 2023-03-03T09:37:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a68144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a677fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a68144ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51188/0 (socket says 172.21.15.102:51188) 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3306258229 learned_addr learned my addr 172.21.15.102:0/3306258229 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a667fc700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a6814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 msgr2=0x7f9a68150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 msgr2=0x7f9a6814f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a6814f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.179+0000 7f9a66ffd700 1 -- 172.21.15.102:0/3306258229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60009a70 con 0x7f9a680b3970 2023-03-03T09:37:19.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.180+0000 7f9a66ffd700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a68144ec0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c520 tx=0x7f9a5800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:19.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.180+0000 7f9a4f7fe700 1 -- 172.21.15.102:0/3306258229 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58005d70 con 0x7f9a680b3970 2023-03-03T09:37:19.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.180+0000 7f9a4f7fe700 1 -- 172.21.15.102:0/3306258229 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a580070d0 con 0x7f9a680b3970 2023-03-03T09:37:19.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.180+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c0094a0 con 0x7f9a680b3970 2023-03-03T09:37:19.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.180+0000 7f9a4f7fe700 1 -- 172.21.15.102:0/3306258229 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58007440 con 0x7f9a680b3970 2023-03-03T09:37:19.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.180+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6806bf20 con 0x7f9a680b3970 2023-03-03T09:37:19.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.181+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6815ce80 con 0x7f9a680b3970 2023-03-03T09:37:19.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.182+0000 7f9a4f7fe700 1 -- 172.21.15.102:0/3306258229 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a58019030 con 0x7f9a680b3970 2023-03-03T09:37:19.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.184+0000 7f9a4f7fe700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:19.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.184+0000 7f9a4f7fe700 1 -- 172.21.15.102:0/3306258229 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9a58084f30 con 0x7f9a680b3970 2023-03-03T09:37:19.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.185+0000 7f9a667fc700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:19.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.185+0000 7f9a667fc700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c000c00 tx=0x7f9a5c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:19.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.189+0000 7f9a4f7fe700 1 -- 172.21.15.102:0/3306258229 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a5804b5e0 con 0x7f9a680b3970 2023-03-03T09:37:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.425+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6806c200 con 0x7f9a680b3970 2023-03-03T09:37:19.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.426+0000 7f9a4f7fe700 1 -- 172.21.15.102:0/3306258229 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9a58017090 con 0x7f9a680b3970 2023-03-03T09:37:19.428 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:37:19.428 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:37:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.428+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 msgr2=0x7f9a5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.428+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c000c00 tx=0x7f9a5c01c040 comp rx=0 tx=0).stop 2023-03-03T09:37:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 msgr2=0x7f9a68144ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a68144ec0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c520 tx=0x7f9a5800c830 comp rx=0 tx=0).stop 2023-03-03T09:37:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 shutdown_connections 2023-03-03T09:37:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a680ced00 0x7f9a68150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a680b3970 0x7f9a68144ec0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 --2- 172.21.15.102:0/3306258229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a680c41b0 0x7f9a6814f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 >> 172.21.15.102:0/3306258229 conn(0x7f9a6808b1c0 msgr2=0x7f9a68071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 shutdown_connections 2023-03-03T09:37:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:37:19.429+0000 7f9a67fff700 1 -- 172.21.15.102:0/3306258229 wait complete. 2023-03-03T09:37:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:37:19.458 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T09:37:24.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:24.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:54.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:54.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:19.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:38:19.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.712+0000 7f6c3cfff700 1 Processor -- start 2023-03-03T09:38:19.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.712+0000 7f6c3cfff700 1 -- start start 2023-03-03T09:38:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.713+0000 7f6c3cfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6c380cd8e0 0x7f6c380cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.713+0000 7f6c3cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c380ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.713+0000 7f6c3cfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c38153660 con 0x7f6c38153290 2023-03-03T09:38:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.713+0000 7f6c3cfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3806c360 con 0x7f6c380cd8e0 2023-03-03T09:38:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.713+0000 7f6c3cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3815cea0 con 0x7f6c380cd390 2023-03-03T09:38:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c35d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6c380cd8e0 0x7f6c380cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c3659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6c38153290 0x7f6c38158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49396/0 (socket says 172.21.15.102:49396) 2023-03-03T09:38:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c3659c700 1 -- 172.21.15.102:0/3709031880 learned_addr learned my addr 172.21.15.102:0/3709031880 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:38:19.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c36d9d700 1 --2- 172.21.15.102:0/3709031880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c380ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c35d9b700 1 -- 172.21.15.102:0/3709031880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 msgr2=0x7f6c380ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:38:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c35d9b700 1 --2- 172.21.15.102:0/3709031880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c380ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c35d9b700 1 -- 172.21.15.102:0/3709031880 >> v1:172.21.15.102:6789/0 conn(0x7f6c38153290 legacy=0x7f6c38158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c35d9b700 1 -- 172.21.15.102:0/3709031880 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3806c7f0 con 0x7f6c380cd8e0 2023-03-03T09:38:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c36d9d700 1 --2- 172.21.15.102:0/3709031880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c380ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:38:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.714+0000 7f6c3559a700 1 -- 172.21.15.102:0/3709031880 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810379331 0 0) 0x7f6c38153660 con 0x7f6c38153290 2023-03-03T09:38:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.715+0000 7f6c35d9b700 1 --2- 172.21.15.102:0/3709031880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c380cde20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009840 tx=0x7f6c2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e68cdcdaba355c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.715+0000 7f6c3559a700 1 -- 172.21.15.102:0/3709031880 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c011830 con 0x7f6c380cd8e0 2023-03-03T09:38:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.715+0000 7f6c3559a700 1 -- 172.21.15.102:0/3709031880 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c011990 con 0x7f6c380cd8e0 2023-03-03T09:38:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.715+0000 7f6c3559a700 1 -- 172.21.15.102:0/3709031880 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c011d20 con 0x7f6c380cd8e0 2023-03-03T09:38:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.715+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3709031880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 msgr2=0x7f6c380cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.716+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3709031880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c380cde20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009840 tx=0x7f6c2c009b50 comp rx=0 tx=0).stop 2023-03-03T09:38:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.716+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3709031880 shutdown_connections 2023-03-03T09:38:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.716+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3709031880 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c380ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.716+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3709031880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c380cde20 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.716+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3709031880 >> 172.21.15.102:0/3709031880 conn(0x7f6c38065fb0 msgr2=0x7f6c380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.716+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3709031880 shutdown_connections 2023-03-03T09:38:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.716+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3709031880 wait complete. 2023-03-03T09:38:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.718+0000 7f6c3cfff700 1 Processor -- start 2023-03-03T09:38:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.718+0000 7f6c3cfff700 1 -- start start 2023-03-03T09:38:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c3cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c381f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c3cfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c381f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c3cfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38153290 0x7f6c381f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c3cfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c381f0310 con 0x7f6c380cd8e0 2023-03-03T09:38:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c3cfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3815ca40 con 0x7f6c38153290 2023-03-03T09:38:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c3cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380c6270 con 0x7f6c380cd390 2023-03-03T09:38:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c35d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c381f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c3659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c381f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c35d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c381f0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55926/0 (socket says 172.21.15.102:55926) 2023-03-03T09:38:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c35d9b700 1 -- 172.21.15.102:0/3643482027 learned_addr learned my addr 172.21.15.102:0/3643482027 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:38:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.719+0000 7f6c36d9d700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38153290 0x7f6c381f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.720+0000 7f6c35d9b700 1 -- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 msgr2=0x7f6c381f0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.720+0000 7f6c35d9b700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c381f0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.720+0000 7f6c35d9b700 1 -- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38153290 msgr2=0x7f6c381f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.720+0000 7f6c35d9b700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38153290 0x7f6c381f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.720+0000 7f6c35d9b700 1 -- 172.21.15.102:0/3643482027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c0094a0 con 0x7f6c380cd8e0 2023-03-03T09:38:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.720+0000 7f6c35d9b700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c381f0fc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c014040 tx=0x7f6c2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.720+0000 7f6c26ffd700 1 -- 172.21.15.102:0/3643482027 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c0036a0 con 0x7f6c380cd8e0 2023-03-03T09:38:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.721+0000 7f6c26ffd700 1 -- 172.21.15.102:0/3643482027 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c01e070 con 0x7f6c380cd8e0 2023-03-03T09:38:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.721+0000 7f6c26ffd700 1 -- 172.21.15.102:0/3643482027 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c003a20 con 0x7f6c380cd8e0 2023-03-03T09:38:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.721+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c380ccde0 con 0x7f6c380cd8e0 2023-03-03T09:38:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.721+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3815d160 con 0x7f6c380cd8e0 2023-03-03T09:38:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.722+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c380cd0c0 con 0x7f6c380cd8e0 2023-03-03T09:38:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.723+0000 7f6c26ffd700 1 -- 172.21.15.102:0/3643482027 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c2c00f050 con 0x7f6c380cd8e0 2023-03-03T09:38:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.724+0000 7f6c26ffd700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.724+0000 7f6c26ffd700 1 -- 172.21.15.102:0/3643482027 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6c2c086cc0 con 0x7f6c380cd8e0 2023-03-03T09:38:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.724+0000 7f6c3659c700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.725+0000 7f6c3659c700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6c28007fd0 tx=0x7f6c28007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:19.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.729+0000 7f6c26ffd700 1 -- 172.21.15.102:0/3643482027 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c2c08d080 con 0x7f6c380cd8e0 2023-03-03T09:38:19.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.963+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c3804e6e0 con 0x7f6c380cd8e0 2023-03-03T09:38:19.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.963+0000 7f6c26ffd700 1 -- 172.21.15.102:0/3643482027 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6c2c015090 con 0x7f6c380cd8e0 2023-03-03T09:38:19.965 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:38:19.966 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:38:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 msgr2=0x7f6c1806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6c28007fd0 tx=0x7f6c28007de0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 msgr2=0x7f6c381f0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c381f0fc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c014040 tx=0x7f6c2c004750 comp rx=0 tx=0).stop 2023-03-03T09:38:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 shutdown_connections 2023-03-03T09:38:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380cd390 0x7f6c381f0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38153290 0x7f6c381f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 --2- 172.21.15.102:0/3643482027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c380cd8e0 0x7f6c381f0fc0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 >> 172.21.15.102:0/3643482027 conn(0x7f6c38065fb0 msgr2=0x7f6c380cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 shutdown_connections 2023-03-03T09:38:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:19.966+0000 7f6c3cfff700 1 -- 172.21.15.102:0/3643482027 wait complete. 2023-03-03T09:38:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:38:19.995 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 22 secs before thrashing 2023-03-03T09:38:24.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:24.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:41.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:38:42.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.246+0000 7fab499ff700 1 Processor -- start 2023-03-03T09:38:42.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.247+0000 7fab499ff700 1 -- start start 2023-03-03T09:38:42.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab499ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fab44153290 0x7fab44158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab499ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab441593c0 0x7fab440cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab499ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab44153660 con 0x7fab440cd8e0 2023-03-03T09:38:42.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab499ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4406c360 con 0x7fab44153290 2023-03-03T09:38:42.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab499ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4415cea0 con 0x7fab441593c0 2023-03-03T09:38:42.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab42ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fab44153290 0x7fab44158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab42ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fab44153290 0x7fab44158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58792/0 (socket says 172.21.15.102:58792) 2023-03-03T09:38:42.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.248+0000 7fab42ffd700 1 -- 172.21.15.102:0/1265439139 learned_addr learned my addr 172.21.15.102:0/1265439139 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:38:42.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.249+0000 7fab427fc700 1 --2- 172.21.15.102:0/1265439139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab441593c0 0x7fab440cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.249+0000 7fab42ffd700 1 -- 172.21.15.102:0/1265439139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab441593c0 msgr2=0x7fab440cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:38:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.249+0000 7fab42ffd700 1 --2- 172.21.15.102:0/1265439139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab441593c0 0x7fab440cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.249+0000 7fab42ffd700 1 -- 172.21.15.102:0/1265439139 >> v1:172.21.15.102:6789/0 conn(0x7fab440cd8e0 legacy=0x7fab440ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.249+0000 7fab42ffd700 1 -- 172.21.15.102:0/1265439139 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4406c7f0 con 0x7fab44153290 2023-03-03T09:38:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.249+0000 7fab42ffd700 1 --2- 172.21.15.102:0/1265439139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab44153290 0x7fab44158e80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fab340098f0 tx=0x7fab34009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29bfd65b3c5985c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.249+0000 7fab41ffb700 1 -- 172.21.15.102:0/1265439139 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34011830 con 0x7fab44153290 2023-03-03T09:38:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab41ffb700 1 -- 172.21.15.102:0/1265439139 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34011990 con 0x7fab44153290 2023-03-03T09:38:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab41ffb700 1 -- 172.21.15.102:0/1265439139 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34011d20 con 0x7fab44153290 2023-03-03T09:38:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 -- 172.21.15.102:0/1265439139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab44153290 msgr2=0x7fab44158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 --2- 172.21.15.102:0/1265439139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab44153290 0x7fab44158e80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fab340098f0 tx=0x7fab34009c00 comp rx=0 tx=0).stop 2023-03-03T09:38:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 -- 172.21.15.102:0/1265439139 shutdown_connections 2023-03-03T09:38:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 --2- 172.21.15.102:0/1265439139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab44153290 0x7fab44158e80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 --2- 172.21.15.102:0/1265439139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab441593c0 0x7fab440cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 -- 172.21.15.102:0/1265439139 >> 172.21.15.102:0/1265439139 conn(0x7fab44065fb0 msgr2=0x7fab440663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 -- 172.21.15.102:0/1265439139 shutdown_connections 2023-03-03T09:38:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.250+0000 7fab499ff700 1 -- 172.21.15.102:0/1265439139 wait complete. 2023-03-03T09:38:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.253+0000 7fab499ff700 1 Processor -- start 2023-03-03T09:38:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.254+0000 7fab499ff700 1 -- start start 2023-03-03T09:38:42.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab499ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab440cd8e0 0x7fab44151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab499ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab44153290 0x7fab441eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab499ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab441593c0 0x7fab441ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab499ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab441f3090 con 0x7fab440cd8e0 2023-03-03T09:38:42.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab499ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab441ef630 con 0x7fab441593c0 2023-03-03T09:38:42.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab499ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab441f2b30 con 0x7fab44153290 2023-03-03T09:38:42.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab42ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab440cd8e0 0x7fab44151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab427fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab44153290 0x7fab441eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab42ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab440cd8e0 0x7fab44151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58804/0 (socket says 172.21.15.102:58804) 2023-03-03T09:38:42.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab42ffd700 1 -- 172.21.15.102:0/1736602391 learned_addr learned my addr 172.21.15.102:0/1736602391 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:38:42.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.255+0000 7fab437fe700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab441593c0 0x7fab441ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab42ffd700 1 -- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab44153290 msgr2=0x7fab441eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab42ffd700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab44153290 0x7fab441eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab42ffd700 1 -- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab441593c0 msgr2=0x7fab441ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab42ffd700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab441593c0 0x7fab441ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab42ffd700 1 -- 172.21.15.102:0/1736602391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34009550 con 0x7fab440cd8e0 2023-03-03T09:38:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab42ffd700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab440cd8e0 0x7fab44151130 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fab34000c00 tx=0x7fab34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:42.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab2bfff700 1 -- 172.21.15.102:0/1736602391 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34004480 con 0x7fab440cd8e0 2023-03-03T09:38:42.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab2bfff700 1 -- 172.21.15.102:0/1736602391 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3401f070 con 0x7fab440cd8e0 2023-03-03T09:38:42.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab2bfff700 1 -- 172.21.15.102:0/1736602391 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab340038c0 con 0x7fab440cd8e0 2023-03-03T09:38:42.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.256+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab440ccde0 con 0x7fab440cd8e0 2023-03-03T09:38:42.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.257+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4415d160 con 0x7fab440cd8e0 2023-03-03T09:38:42.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.258+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab440cd0c0 con 0x7fab440cd8e0 2023-03-03T09:38:42.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.259+0000 7fab2bfff700 1 -- 172.21.15.102:0/1736602391 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab3402a020 con 0x7fab440cd8e0 2023-03-03T09:38:42.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.260+0000 7fab2bfff700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab300682c0 0x7fab3006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.260+0000 7fab2bfff700 1 -- 172.21.15.102:0/1736602391 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fab34087330 con 0x7fab440cd8e0 2023-03-03T09:38:42.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.261+0000 7fab427fc700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab300682c0 0x7fab3006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.261+0000 7fab427fc700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab300682c0 0x7fab3006a770 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fab440b4d70 tx=0x7fab38009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:42.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.265+0000 7fab2bfff700 1 -- 172.21.15.102:0/1736602391 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab34053a40 con 0x7fab440cd8e0 2023-03-03T09:38:42.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.502+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab4406d430 con 0x7fab440cd8e0 2023-03-03T09:38:42.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.503+0000 7fab2bfff700 1 -- 172.21.15.102:0/1736602391 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fab34057060 con 0x7fab440cd8e0 2023-03-03T09:38:42.505 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:38:42.505 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:38:42.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.505+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab300682c0 msgr2=0x7fab3006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.505+0000 7fab499ff700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab300682c0 0x7fab3006a770 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fab440b4d70 tx=0x7fab38009210 comp rx=0 tx=0).stop 2023-03-03T09:38:42.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab440cd8e0 msgr2=0x7fab44151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab440cd8e0 0x7fab44151130 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fab34000c00 tx=0x7fab34004750 comp rx=0 tx=0).stop 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 shutdown_connections 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab441593c0 0x7fab441ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab440cd8e0 0x7fab44151130 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab300682c0 0x7fab3006a770 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 --2- 172.21.15.102:0/1736602391 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab44153290 0x7fab441eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 >> 172.21.15.102:0/1736602391 conn(0x7fab44065fb0 msgr2=0x7fab440cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 shutdown_connections 2023-03-03T09:38:42.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:38:42.506+0000 7fab499ff700 1 -- 172.21.15.102:0/1736602391 wait complete. 2023-03-03T09:38:42.509 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:38:42.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T09:38:54.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:54.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:22.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:39:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.804+0000 7f99415ff700 1 Processor -- start 2023-03-03T09:39:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.804+0000 7f99415ff700 1 -- start start 2023-03-03T09:39:22.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.805+0000 7f99415ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:22.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f99415ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f993c0cd390 0x7f993c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:22.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f99415ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c153660 con 0x7f993c153290 2023-03-03T09:39:22.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f99415ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f993c06c360 con 0x7f993c0cd390 2023-03-03T09:39:22.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f99415ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993c15cea0 con 0x7f993c0cd8e0 2023-03-03T09:39:22.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f993a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:22.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f993b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f993c0cd390 0x7f993c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:22.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f993a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55088/0 (socket says 172.21.15.102:55088) 2023-03-03T09:39:22.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f993a7fc700 1 -- 172.21.15.102:0/2837120096 learned_addr learned my addr 172.21.15.102:0/2837120096 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:39:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.806+0000 7f993affd700 1 --1- 172.21.15.102:0/2837120096 >> v1:172.21.15.172:6789/0 conn(0x7f993c153290 0x7f993c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37654/0 (socket says 172.21.15.102:37654) 2023-03-03T09:39:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f993a7fc700 1 -- 172.21.15.102:0/2837120096 >> v2:172.21.15.172:3300/0 conn(0x7f993c0cd390 msgr2=0x7f993c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:39:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f993a7fc700 1 --2- 172.21.15.102:0/2837120096 >> v2:172.21.15.172:3300/0 conn(0x7f993c0cd390 0x7f993c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f993a7fc700 1 -- 172.21.15.102:0/2837120096 >> v1:172.21.15.172:6789/0 conn(0x7f993c153290 legacy=0x7f993c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f993a7fc700 1 -- 172.21.15.102:0/2837120096 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c06c7f0 con 0x7f993c0cd8e0 2023-03-03T09:39:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f993a7fc700 1 --2- 172.21.15.102:0/2837120096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c0cde20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f99300098f0 tx=0x7f9930009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73ff909b4f1eaa5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f9939ffb700 1 -- 172.21.15.102:0/2837120096 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930011830 con 0x7f993c0cd8e0 2023-03-03T09:39:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f9939ffb700 1 -- 172.21.15.102:0/2837120096 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930011990 con 0x7f993c0cd8e0 2023-03-03T09:39:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.807+0000 7f9939ffb700 1 -- 172.21.15.102:0/2837120096 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930011d20 con 0x7f993c0cd8e0 2023-03-03T09:39:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 -- 172.21.15.102:0/2837120096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 msgr2=0x7f993c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 --2- 172.21.15.102:0/2837120096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c0cde20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f99300098f0 tx=0x7f9930009c00 comp rx=0 tx=0).stop 2023-03-03T09:39:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 -- 172.21.15.102:0/2837120096 shutdown_connections 2023-03-03T09:39:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 --2- 172.21.15.102:0/2837120096 >> v2:172.21.15.172:3300/0 conn(0x7f993c0cd390 0x7f993c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 --2- 172.21.15.102:0/2837120096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c0cde20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 -- 172.21.15.102:0/2837120096 >> 172.21.15.102:0/2837120096 conn(0x7f993c065fb0 msgr2=0x7f993c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 -- 172.21.15.102:0/2837120096 shutdown_connections 2023-03-03T09:39:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.808+0000 7f99415ff700 1 -- 172.21.15.102:0/2837120096 wait complete. 2023-03-03T09:39:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.810+0000 7f99415ff700 1 Processor -- start 2023-03-03T09:39:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.810+0000 7f99415ff700 1 -- start start 2023-03-03T09:39:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.811+0000 7f99415ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f993c0cd390 0x7f993c14fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.811+0000 7f99415ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.811+0000 7f99415ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f993c153290 0x7f993c14dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.811+0000 7f99415ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993c14f890 con 0x7f993c153290 2023-03-03T09:39:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.811+0000 7f99415ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993c150260 con 0x7f993c0cd390 2023-03-03T09:39:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.811+0000 7f99415ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993c1ef840 con 0x7f993c0cd8e0 2023-03-03T09:39:22.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:22.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f993c0cd390 0x7f993c14fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55104/0 (socket says 172.21.15.102:55104) 2023-03-03T09:39:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993a7fc700 1 -- 172.21.15.102:0/3781632311 learned_addr learned my addr 172.21.15.102:0/3781632311 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:39:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993b7fe700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f993c153290 0x7f993c14dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993affd700 1 -- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 msgr2=0x7f993c158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993affd700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993affd700 1 -- 172.21.15.102:0/3781632311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f993c153290 msgr2=0x7f993c14dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993affd700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f993c153290 0x7f993c14dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993affd700 1 -- 172.21.15.102:0/3781632311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930009550 con 0x7f993c0cd390 2023-03-03T09:39:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.812+0000 7f993affd700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f993c0cd390 0x7f993c14fd20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c450 tx=0x7f992c00c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.813+0000 7f99237fe700 1 -- 172.21.15.102:0/3781632311 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c005cb0 con 0x7f993c0cd390 2023-03-03T09:39:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.813+0000 7f99237fe700 1 -- 172.21.15.102:0/3781632311 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992c005e10 con 0x7f993c0cd390 2023-03-03T09:39:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.813+0000 7f99237fe700 1 -- 172.21.15.102:0/3781632311 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c0072d0 con 0x7f993c0cd390 2023-03-03T09:39:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.813+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993c0ccde0 con 0x7f993c0cd390 2023-03-03T09:39:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.813+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f993c15d160 con 0x7f993c0cd390 2023-03-03T09:39:22.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.814+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993c0cd0c0 con 0x7f993c0cd390 2023-03-03T09:39:22.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.816+0000 7f99237fe700 1 -- 172.21.15.102:0/3781632311 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f992c019030 con 0x7f993c0cd390 2023-03-03T09:39:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.817+0000 7f99237fe700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f99280682f0 0x7f992806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.817+0000 7f99237fe700 1 -- 172.21.15.102:0/3781632311 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f992c084e30 con 0x7f993c0cd390 2023-03-03T09:39:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.817+0000 7f993a7fc700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f99280682f0 0x7f992806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.818+0000 7f993a7fc700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f99280682f0 0x7f992806a7a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f993c0c9360 tx=0x7f9930008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:22.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:22.821+0000 7f99237fe700 1 -- 172.21.15.102:0/3781632311 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f992c04b5e0 con 0x7f993c0cd390 2023-03-03T09:39:23.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.057+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f993c14fb70 con 0x7f993c0cd390 2023-03-03T09:39:23.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.058+0000 7f99237fe700 1 -- 172.21.15.102:0/3781632311 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f992c017020 con 0x7f993c0cd390 2023-03-03T09:39:23.059 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:39:23.059 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:39:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f99280682f0 msgr2=0x7f992806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f99280682f0 0x7f992806a7a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f993c0c9360 tx=0x7f9930008030 comp rx=0 tx=0).stop 2023-03-03T09:39:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f993c0cd390 msgr2=0x7f993c14fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f993c0cd390 0x7f993c14fd20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c450 tx=0x7f992c00c760 comp rx=0 tx=0).stop 2023-03-03T09:39:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 shutdown_connections 2023-03-03T09:39:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f993c153290 0x7f993c14dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f993c0cd390 0x7f993c14fd20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f99280682f0 0x7f992806a7a0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.060+0000 7f99415ff700 1 --2- 172.21.15.102:0/3781632311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f993c0cd8e0 0x7f993c158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.061+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 >> 172.21.15.102:0/3781632311 conn(0x7f993c065fb0 msgr2=0x7f993c1572d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.061+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 shutdown_connections 2023-03-03T09:39:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:23.061+0000 7f99415ff700 1 -- 172.21.15.102:0/3781632311 wait complete. 2023-03-03T09:39:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:39:23.089 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 20 secs before thrashing 2023-03-03T09:39:25.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:25.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:43.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:39:43.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.338+0000 7f80d4dff700 1 Processor -- start 2023-03-03T09:39:43.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.338+0000 7f80d4dff700 1 -- start start 2023-03-03T09:39:43.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80d4dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f80d0153290 0x7f80d0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:43.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80d4dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:43.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80d4dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d0153660 con 0x7f80d01593c0 2023-03-03T09:39:43.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80d4dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d006c360 con 0x7f80d0153290 2023-03-03T09:39:43.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80d4dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d015cea0 con 0x7f80d00cd8e0 2023-03-03T09:39:43.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80ce59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f80d0153290 0x7f80d0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80ce59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f80d0153290 0x7f80d0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46436/0 (socket says 172.21.15.102:46436) 2023-03-03T09:39:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80ce59c700 1 -- 172.21.15.102:0/633377270 learned_addr learned my addr 172.21.15.102:0/633377270 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:39:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.340+0000 7f80ced9d700 1 --2- 172.21.15.102:0/633377270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80ce59c700 1 -- 172.21.15.102:0/633377270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 msgr2=0x7f80d00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:39:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80ce59c700 1 --2- 172.21.15.102:0/633377270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80ce59c700 1 -- 172.21.15.102:0/633377270 >> v1:172.21.15.172:6789/0 conn(0x7f80d01593c0 legacy=0x7f80d00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80ce59c700 1 -- 172.21.15.102:0/633377270 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d006c7f0 con 0x7f80d0153290 2023-03-03T09:39:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80cd59a700 1 -- 172.21.15.102:0/633377270 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132609875 0 0) 0x7f80d0153660 con 0x7f80d01593c0 2023-03-03T09:39:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80ce59c700 1 --2- 172.21.15.102:0/633377270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d0158e80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f80c00098f0 tx=0x7f80c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b992bbffbc4bdb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80cd59a700 1 -- 172.21.15.102:0/633377270 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0011830 con 0x7f80d0153290 2023-03-03T09:39:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80cd59a700 1 -- 172.21.15.102:0/633377270 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c0011990 con 0x7f80d0153290 2023-03-03T09:39:43.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.341+0000 7f80cd59a700 1 -- 172.21.15.102:0/633377270 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0011d20 con 0x7f80d0153290 2023-03-03T09:39:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 -- 172.21.15.102:0/633377270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 msgr2=0x7f80d0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 --2- 172.21.15.102:0/633377270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d0158e80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f80c00098f0 tx=0x7f80c0009c00 comp rx=0 tx=0).stop 2023-03-03T09:39:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 -- 172.21.15.102:0/633377270 shutdown_connections 2023-03-03T09:39:43.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 --2- 172.21.15.102:0/633377270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 --2- 172.21.15.102:0/633377270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d0158e80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 -- 172.21.15.102:0/633377270 >> 172.21.15.102:0/633377270 conn(0x7f80d0065fb0 msgr2=0x7f80d00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 -- 172.21.15.102:0/633377270 shutdown_connections 2023-03-03T09:39:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.342+0000 7f80d4dff700 1 -- 172.21.15.102:0/633377270 wait complete. 2023-03-03T09:39:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.344+0000 7f80d4dff700 1 Processor -- start 2023-03-03T09:39:43.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.344+0000 7f80d4dff700 1 -- start start 2023-03-03T09:39:43.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80d4dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:43.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80d4dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d00b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:43.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80d4dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80d01593c0 0x7f80d00caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80d4dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d0157350 con 0x7f80d0153290 2023-03-03T09:39:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80d4dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d00c6cc0 con 0x7f80d01593c0 2023-03-03T09:39:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80d4dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d01e44c0 con 0x7f80d00cd8e0 2023-03-03T09:39:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80cdd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d00b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.345+0000 7f80ce59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80cdd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d00b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46452/0 (socket says 172.21.15.102:46452) 2023-03-03T09:39:43.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80cdd9b700 1 -- 172.21.15.102:0/3398728894 learned_addr learned my addr 172.21.15.102:0/3398728894 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:39:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80ced9d700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80d01593c0 0x7f80d00caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80ce59c700 1 -- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80d01593c0 msgr2=0x7f80d00caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80ce59c700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80d01593c0 0x7f80d00caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80ce59c700 1 -- 172.21.15.102:0/3398728894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 msgr2=0x7f80d00b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80ce59c700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d00b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80ce59c700 1 -- 172.21.15.102:0/3398728894 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0009550 con 0x7f80d00cd8e0 2023-03-03T09:39:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80cdd9b700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d00b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:39:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.346+0000 7f80ce59c700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00b1ee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f80d0159fb0 tx=0x7f80c0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.347+0000 7f80beffd700 1 -- 172.21.15.102:0/3398728894 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0004510 con 0x7f80d00cd8e0 2023-03-03T09:39:43.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.347+0000 7f80beffd700 1 -- 172.21.15.102:0/3398728894 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c001e070 con 0x7f80d00cd8e0 2023-03-03T09:39:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.347+0000 7f80beffd700 1 -- 172.21.15.102:0/3398728894 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c00038e0 con 0x7f80d00cd8e0 2023-03-03T09:39:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.347+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d00ccde0 con 0x7f80d00cd8e0 2023-03-03T09:39:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.347+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d015d160 con 0x7f80d00cd8e0 2023-03-03T09:39:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.348+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d00cd0c0 con 0x7f80d00cd8e0 2023-03-03T09:39:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.349+0000 7f80beffd700 1 -- 172.21.15.102:0/3398728894 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80c000f050 con 0x7f80d00cd8e0 2023-03-03T09:39:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.350+0000 7f80beffd700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b0068340 0x7f80b006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.350+0000 7f80beffd700 1 -- 172.21.15.102:0/3398728894 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f80c0086b60 con 0x7f80d00cd8e0 2023-03-03T09:39:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.351+0000 7f80cdd9b700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b0068340 0x7f80b006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.352+0000 7f80cdd9b700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b0068340 0x7f80b006a7f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f80d006b290 tx=0x7f80c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.355+0000 7f80beffd700 1 -- 172.21.15.102:0/3398728894 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80c00532f0 con 0x7f80d00cd8e0 2023-03-03T09:39:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.587+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80d004e6e0 con 0x7f80d00cd8e0 2023-03-03T09:39:43.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.588+0000 7f80beffd700 1 -- 172.21.15.102:0/3398728894 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f80c0020090 con 0x7f80d00cd8e0 2023-03-03T09:39:43.590 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:39:43.590 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:39:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b0068340 msgr2=0x7f80b006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b0068340 0x7f80b006a7f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f80d006b290 tx=0x7f80c4009040 comp rx=0 tx=0).stop 2023-03-03T09:39:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 msgr2=0x7f80d00b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00b1ee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f80d0159fb0 tx=0x7f80c0004750 comp rx=0 tx=0).stop 2023-03-03T09:39:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 shutdown_connections 2023-03-03T09:39:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b0068340 0x7f80b006a7f0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80d00cd8e0 0x7f80d00b1ee0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80d01593c0 0x7f80d00caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 --2- 172.21.15.102:0/3398728894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80d0153290 0x7f80d00b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 >> 172.21.15.102:0/3398728894 conn(0x7f80d0065fb0 msgr2=0x7f80d015b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 shutdown_connections 2023-03-03T09:39:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:39:43.591+0000 7f80d4dff700 1 -- 172.21.15.102:0/3398728894 wait complete. 2023-03-03T09:39:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:39:43.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2023-03-03T09:39:55.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:55.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T09:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:40:25.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:25.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:55.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:55.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:11.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:41:11.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.888+0000 7f1107fff700 1 Processor -- start 2023-03-03T09:41:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.888+0000 7f1107fff700 1 -- start start 2023-03-03T09:41:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f1107fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11080b3970 0x7f11080c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f1107fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f11080cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f1107fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11080c3d20 con 0x7f11080ced00 2023-03-03T09:41:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f1107fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11080c4930 con 0x7f11080b3970 2023-03-03T09:41:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f1107fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11080cf610 con 0x7f11080c41b0 2023-03-03T09:41:11.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f10fffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11080b3970 0x7f11080c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:11.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f10fffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11080b3970 0x7f11080c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45106/0 (socket says 172.21.15.102:45106) 2023-03-03T09:41:11.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f10fffff700 1 -- 172.21.15.102:0/3062506210 learned_addr learned my addr 172.21.15.102:0/3062506210 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:41:11.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.890+0000 7f11077fe700 1 --2- 172.21.15.102:0/3062506210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f11080cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:11.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f10fffff700 1 -- 172.21.15.102:0/3062506210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 msgr2=0x7f11080cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:11.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f10fffff700 1 --2- 172.21.15.102:0/3062506210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f11080cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f10fffff700 1 -- 172.21.15.102:0/3062506210 >> v1:172.21.15.172:6789/0 conn(0x7f11080ced00 legacy=0x7f1108158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f11067fc700 1 -- 172.21.15.102:0/3062506210 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116939539 0 0) 0x7f11080c3d20 con 0x7f11080ced00 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f10fffff700 1 -- 172.21.15.102:0/3062506210 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110815cba0 con 0x7f11080b3970 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f11077fe700 1 --2- 172.21.15.102:0/3062506210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f11080cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f10fffff700 1 --2- 172.21.15.102:0/3062506210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080b3970 0x7f11080c5a90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009920 tx=0x7f10f0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6bd232c1bd4ad0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f11067fc700 1 -- 172.21.15.102:0/3062506210 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011830 con 0x7f11080b3970 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f11067fc700 1 -- 172.21.15.102:0/3062506210 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f0011990 con 0x7f11080b3970 2023-03-03T09:41:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.891+0000 7f11067fc700 1 -- 172.21.15.102:0/3062506210 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011d20 con 0x7f11080b3970 2023-03-03T09:41:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 -- 172.21.15.102:0/3062506210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080b3970 msgr2=0x7f11080c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 --2- 172.21.15.102:0/3062506210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080b3970 0x7f11080c5a90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009920 tx=0x7f10f0009c30 comp rx=0 tx=0).stop 2023-03-03T09:41:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 -- 172.21.15.102:0/3062506210 shutdown_connections 2023-03-03T09:41:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 --2- 172.21.15.102:0/3062506210 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f11080cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 --2- 172.21.15.102:0/3062506210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080b3970 0x7f11080c5a90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 -- 172.21.15.102:0/3062506210 >> 172.21.15.102:0/3062506210 conn(0x7f110808b1c0 msgr2=0x7f1108071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 -- 172.21.15.102:0/3062506210 shutdown_connections 2023-03-03T09:41:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.892+0000 7f1107fff700 1 -- 172.21.15.102:0/3062506210 wait complete. 2023-03-03T09:41:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.894+0000 7f1107fff700 1 Processor -- start 2023-03-03T09:41:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.894+0000 7f1107fff700 1 -- start start 2023-03-03T09:41:11.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.895+0000 7f1107fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11080b3970 0x7f1108148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:11.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.895+0000 7f1107fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f1108148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:11.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.895+0000 7f1107fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080ced00 0x7f11081f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:11.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.895+0000 7f1107fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1108151490 con 0x7f11080ced00 2023-03-03T09:41:11.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.895+0000 7f1107fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081f1890 con 0x7f11080b3970 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.895+0000 7f1107fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11081511b0 con 0x7f11080c41b0 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f11077fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080ced00 0x7f11081f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f10fffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f1108148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f1106ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11080b3970 0x7f1108148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f11077fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080ced00 0x7f11081f1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45110/0 (socket says 172.21.15.102:45110) 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f11077fe700 1 -- 172.21.15.102:0/3095930400 learned_addr learned my addr 172.21.15.102:0/3095930400 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f10fffff700 1 -- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11080b3970 msgr2=0x7f1108148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f10fffff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11080b3970 0x7f1108148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f10fffff700 1 -- 172.21.15.102:0/3095930400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080ced00 msgr2=0x7f11081f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f10fffff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080ced00 0x7f11081f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f10fffff700 1 -- 172.21.15.102:0/3095930400 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100009a70 con 0x7f11080c41b0 2023-03-03T09:41:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.896+0000 7f10fffff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f1108148570 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f10f0016000 tx=0x7f10f0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.897+0000 7f10ff7fe700 1 -- 172.21.15.102:0/3095930400 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0004510 con 0x7f11080c41b0 2023-03-03T09:41:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.897+0000 7f10ff7fe700 1 -- 172.21.15.102:0/3095930400 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f001e070 con 0x7f11080c41b0 2023-03-03T09:41:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.897+0000 7f10ff7fe700 1 -- 172.21.15.102:0/3095930400 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00038e0 con 0x7f11080c41b0 2023-03-03T09:41:11.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.897+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f0009550 con 0x7f11080c41b0 2023-03-03T09:41:11.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.897+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110806bf20 con 0x7f11080c41b0 2023-03-03T09:41:11.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.898+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110815ce80 con 0x7f11080c41b0 2023-03-03T09:41:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.899+0000 7f10ff7fe700 1 -- 172.21.15.102:0/3095930400 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f10f000f050 con 0x7f11080c41b0 2023-03-03T09:41:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.900+0000 7f10ff7fe700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e8068320 0x7f10e806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.900+0000 7f10ff7fe700 1 -- 172.21.15.102:0/3095930400 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f10f0086ca0 con 0x7f11080c41b0 2023-03-03T09:41:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.901+0000 7f1106ffd700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e8068320 0x7f10e806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.901+0000 7f1106ffd700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e8068320 0x7f10e806a7d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f10f8007fd0 tx=0x7f10f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:11.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:11.905+0000 7f10ff7fe700 1 -- 172.21.15.102:0/3095930400 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10f00533b0 con 0x7f11080c41b0 2023-03-03T09:41:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.138+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f110806c200 con 0x7f11080c41b0 2023-03-03T09:41:12.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.139+0000 7f10ff7fe700 1 -- 172.21.15.102:0/3095930400 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f10f00569d0 con 0x7f11080c41b0 2023-03-03T09:41:12.141 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:41:12.141 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:41:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e8068320 msgr2=0x7f10e806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e8068320 0x7f10e806a7d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f10f8007fd0 tx=0x7f10f8009040 comp rx=0 tx=0).stop 2023-03-03T09:41:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 msgr2=0x7f1108148570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f1108148570 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f10f0016000 tx=0x7f10f0004750 comp rx=0 tx=0).stop 2023-03-03T09:41:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 shutdown_connections 2023-03-03T09:41:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11080ced00 0x7f11081f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11080b3970 0x7f1108148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10e8068320 0x7f10e806a7d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 --2- 172.21.15.102:0/3095930400 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11080c41b0 0x7f1108148570 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 >> 172.21.15.102:0/3095930400 conn(0x7f110808b1c0 msgr2=0x7f11080cdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 shutdown_connections 2023-03-03T09:41:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:12.142+0000 7f1107fff700 1 -- 172.21.15.102:0/3095930400 wait complete. 2023-03-03T09:41:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:41:12.171 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-03-03T09:41:25.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:25.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:56.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:56.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:59.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:41:59.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.421+0000 7f3edd5ff700 1 Processor -- start 2023-03-03T09:41:59.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.422+0000 7f3edd5ff700 1 -- start start 2023-03-03T09:41:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3edd5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3edd5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ed80cd8e0 0x7f3ed80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3edd5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed8153660 con 0x7f3ed81593c0 2023-03-03T09:41:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3edd5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed806c360 con 0x7f3ed8153290 2023-03-03T09:41:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3edd5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed815cea0 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3ed77fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ed80cd8e0 0x7f3ed80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:59.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3ed67fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3ed81593c0 0x7f3ed80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36972/0 (socket says 172.21.15.102:36972) 2023-03-03T09:41:59.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3ed67fc700 1 -- 172.21.15.102:0/2785823627 learned_addr learned my addr 172.21.15.102:0/2785823627 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:41:59.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.423+0000 7f3ed6ffd700 1 --2- 172.21.15.102:0/2785823627 >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:59.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed77fe700 1 -- 172.21.15.102:0/2785823627 >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 msgr2=0x7f3ed8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:59.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed77fe700 1 --2- 172.21.15.102:0/2785823627 >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed77fe700 1 -- 172.21.15.102:0/2785823627 >> v1:172.21.15.172:6789/0 conn(0x7f3ed81593c0 legacy=0x7f3ed80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:59.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed77fe700 1 -- 172.21.15.102:0/2785823627 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed806c7f0 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed77fe700 1 --2- 172.21.15.102:0/2785823627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed80ce530 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80c9390 tx=0x7f3ed000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12b77f32cf223298 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed5ffb700 1 -- 172.21.15.102:0/2785823627 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000e070 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed5ffb700 1 -- 172.21.15.102:0/2785823627 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed000bb20 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.424+0000 7f3ed5ffb700 1 -- 172.21.15.102:0/2785823627 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0012430 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 -- 172.21.15.102:0/2785823627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed80cd8e0 msgr2=0x7f3ed80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:59.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/2785823627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed80ce530 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80c9390 tx=0x7f3ed000b330 comp rx=0 tx=0).stop 2023-03-03T09:41:59.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 -- 172.21.15.102:0/2785823627 shutdown_connections 2023-03-03T09:41:59.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/2785823627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed80ce530 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/2785823627 >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 -- 172.21.15.102:0/2785823627 >> 172.21.15.102:0/2785823627 conn(0x7f3ed8065fb0 msgr2=0x7f3ed80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:59.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 -- 172.21.15.102:0/2785823627 shutdown_connections 2023-03-03T09:41:59.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.425+0000 7f3edd5ff700 1 -- 172.21.15.102:0/2785823627 wait complete. 2023-03-03T09:41:59.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.427+0000 7f3edd5ff700 1 Processor -- start 2023-03-03T09:41:59.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.427+0000 7f3edd5ff700 1 -- start start 2023-03-03T09:41:59.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.428+0000 7f3edd5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed8156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:59.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.428+0000 7f3edd5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed8153290 0x7f3ed80c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:59.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.428+0000 7f3edd5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed81593c0 0x7f3ed80c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:59.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.428+0000 7f3edd5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80c8360 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.428+0000 7f3edd5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80c8b50 con 0x7f3ed81593c0 2023-03-03T09:41:59.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.428+0000 7f3edd5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80c8e30 con 0x7f3ed8153290 2023-03-03T09:41:59.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.428+0000 7f3ed6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed8156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:59.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed8156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47820/0 (socket says 172.21.15.102:47820) 2023-03-03T09:41:59.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 -- 172.21.15.102:0/1434075644 learned_addr learned my addr 172.21.15.102:0/1434075644 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:41:59.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed77fe700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed81593c0 0x7f3ed80c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:59.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed67fc700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed8153290 0x7f3ed80c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:59.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 -- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed8153290 msgr2=0x7f3ed80c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:59.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed8153290 0x7f3ed80c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 -- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed81593c0 msgr2=0x7f3ed80c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed81593c0 0x7f3ed80c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 -- 172.21.15.102:0/1434075644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0009a70 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed67fc700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed8153290 0x7f3ed80c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:41:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.429+0000 7f3ed6ffd700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed8156ac0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800e9b0 tx=0x7f3ec800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.430+0000 7f3ebf7fe700 1 -- 172.21.15.102:0/1434075644 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8009710 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.430+0000 7f3ebf7fe700 1 -- 172.21.15.102:0/1434075644 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec800c850 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.430+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed80ccde0 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.430+0000 7f3ebf7fe700 1 -- 172.21.15.102:0/1434075644 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec800cbc0 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.430+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed815d160 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.431+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed80cd0c0 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.432+0000 7f3ebf7fe700 1 -- 172.21.15.102:0/1434075644 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3ec8023070 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.433+0000 7f3ebf7fe700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3ec4068320 0x7f3ec406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:59.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.433+0000 7f3ebf7fe700 1 -- 172.21.15.102:0/1434075644 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3ec8086100 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.434+0000 7f3ed67fc700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3ec4068320 0x7f3ec406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:59.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.434+0000 7f3ed67fc700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3ec4068320 0x7f3ec406a7d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc007fd0 tx=0x7f3ecc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:59.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.438+0000 7f3ebf7fe700 1 -- 172.21.15.102:0/1434075644 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3ec8052780 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.671+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ed81ee7c0 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.671+0000 7f3ebf7fe700 1 -- 172.21.15.102:0/1434075644 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3ec8055da0 con 0x7f3ed80cd8e0 2023-03-03T09:41:59.674 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:41:59.674 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:41:59.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3ec4068320 msgr2=0x7f3ec406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:59.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3ec4068320 0x7f3ec406a7d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc007fd0 tx=0x7f3ecc009040 comp rx=0 tx=0).stop 2023-03-03T09:41:59.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed80cd8e0 msgr2=0x7f3ed8156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:59.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed8156ac0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800e9b0 tx=0x7f3ec800ecc0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 shutdown_connections 2023-03-03T09:41:59.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed81593c0 0x7f3ed80c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed80cd8e0 0x7f3ed8156ac0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3ec4068320 0x7f3ec406a7d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 --2- 172.21.15.102:0/1434075644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed8153290 0x7f3ed80c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:59.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 >> 172.21.15.102:0/1434075644 conn(0x7f3ed8065fb0 msgr2=0x7f3ed80cc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:59.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 shutdown_connections 2023-03-03T09:41:59.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:41:59.674+0000 7f3edd5ff700 1 -- 172.21.15.102:0/1434075644 wait complete. 2023-03-03T09:41:59.678 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:41:59.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 65 secs before thrashing 2023-03-03T09:42:26.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:26.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:56.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:56.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:04.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:43:04.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.960+0000 7f3edcbff700 1 Processor -- start 2023-03-03T09:43:04.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.960+0000 7f3edcbff700 1 -- start start 2023-03-03T09:43:04.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3edcbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:04.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3edcbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ed80cd8e0 0x7f3ed80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:04.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3edcbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed8153660 con 0x7f3ed81593c0 2023-03-03T09:43:04.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3edcbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed806c360 con 0x7f3ed8153290 2023-03-03T09:43:04.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3edcbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed815cea0 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3ed659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:04.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3ed659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49248/0 (socket says 172.21.15.102:49248) 2023-03-03T09:43:04.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3ed659c700 1 -- 172.21.15.102:0/2885584096 learned_addr learned my addr 172.21.15.102:0/2885584096 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:43:04.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.962+0000 7f3ed6d9d700 1 --2- 172.21.15.102:0/2885584096 >> v2:172.21.15.172:3300/0 conn(0x7f3ed80cd8e0 0x7f3ed80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:04.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed659c700 1 -- 172.21.15.102:0/2885584096 >> v2:172.21.15.172:3300/0 conn(0x7f3ed80cd8e0 msgr2=0x7f3ed80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:04.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed659c700 1 --2- 172.21.15.102:0/2885584096 >> v2:172.21.15.172:3300/0 conn(0x7f3ed80cd8e0 0x7f3ed80ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:04.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed659c700 1 -- 172.21.15.102:0/2885584096 >> v1:172.21.15.102:6789/0 conn(0x7f3ed81593c0 legacy=0x7f3ed80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:04.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed659c700 1 -- 172.21.15.102:0/2885584096 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed806c7f0 con 0x7f3ed8153290 2023-03-03T09:43:04.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed559a700 1 -- 172.21.15.102:0/2885584096 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119972348 0 0) 0x7f3ed8153660 con 0x7f3ed81593c0 2023-03-03T09:43:04.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed659c700 1 --2- 172.21.15.102:0/2885584096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed8153290 0x7f3ed8158e80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80c9390 tx=0x7f3ec8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d886aa2e80066b88 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:04.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed559a700 1 -- 172.21.15.102:0/2885584096 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8011830 con 0x7f3ed8153290 2023-03-03T09:43:04.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed559a700 1 -- 172.21.15.102:0/2885584096 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8011990 con 0x7f3ed8153290 2023-03-03T09:43:04.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.963+0000 7f3ed559a700 1 -- 172.21.15.102:0/2885584096 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8011d20 con 0x7f3ed8153290 2023-03-03T09:43:04.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 -- 172.21.15.102:0/2885584096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed8153290 msgr2=0x7f3ed8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:04.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 --2- 172.21.15.102:0/2885584096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed8153290 0x7f3ed8158e80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80c9390 tx=0x7f3ec8009bd0 comp rx=0 tx=0).stop 2023-03-03T09:43:04.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 -- 172.21.15.102:0/2885584096 shutdown_connections 2023-03-03T09:43:04.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 --2- 172.21.15.102:0/2885584096 >> v2:172.21.15.172:3300/0 conn(0x7f3ed80cd8e0 0x7f3ed80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:04.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 --2- 172.21.15.102:0/2885584096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed8153290 0x7f3ed8158e80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:04.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 -- 172.21.15.102:0/2885584096 >> 172.21.15.102:0/2885584096 conn(0x7f3ed8065fb0 msgr2=0x7f3ed80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:04.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 -- 172.21.15.102:0/2885584096 shutdown_connections 2023-03-03T09:43:04.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.964+0000 7f3edcbff700 1 -- 172.21.15.102:0/2885584096 wait complete. 2023-03-03T09:43:04.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.967+0000 7f3edcbff700 1 Processor -- start 2023-03-03T09:43:04.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.967+0000 7f3edcbff700 1 -- start start 2023-03-03T09:43:04.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3edcbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed80cd8e0 0x7f3ed80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:04.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3edcbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed8153290 0x7f3ed80c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:04.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3edcbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed81593c0 0x7f3ed814dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:04.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3edcbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81eef20 con 0x7f3ed81593c0 2023-03-03T09:43:04.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3edcbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81ee650 con 0x7f3ed8153290 2023-03-03T09:43:04.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3edcbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed8156130 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3ed6d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed81593c0 0x7f3ed814dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3ed659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed80cd8e0 0x7f3ed80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3ed659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed80cd8e0 0x7f3ed80b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44180/0 (socket says 172.21.15.102:44180) 2023-03-03T09:43:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3ed659c700 1 -- 172.21.15.102:0/670289185 learned_addr learned my addr 172.21.15.102:0/670289185 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:43:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.969+0000 7f3ed5d9b700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed8153290 0x7f3ed80c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ed659c700 1 -- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed8153290 msgr2=0x7f3ed80c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ed659c700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed8153290 0x7f3ed80c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ed659c700 1 -- 172.21.15.102:0/670289185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed81593c0 msgr2=0x7f3ed814dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ed659c700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed81593c0 0x7f3ed814dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ed659c700 1 -- 172.21.15.102:0/670289185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8009550 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ed659c700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed80cd8e0 0x7f3ed80b2cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009c10 tx=0x7f3ec8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ec6ffd700 1 -- 172.21.15.102:0/670289185 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8004510 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ec6ffd700 1 -- 172.21.15.102:0/670289185 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec80036a0 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3ec6ffd700 1 -- 172.21.15.102:0/670289185 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8003a10 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed80ccde0 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.970+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed815d160 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.971+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed80cd0c0 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.972+0000 7f3ec6ffd700 1 -- 172.21.15.102:0/670289185 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3ec802d070 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.974+0000 7f3ec6ffd700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3eb8068320 0x7f3eb806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.974+0000 7f3ec6ffd700 1 -- 172.21.15.102:0/670289185 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3ec8021030 con 0x7f3ed80cd8e0 2023-03-03T09:43:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.974+0000 7f3ed5d9b700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3eb8068320 0x7f3eb806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.975+0000 7f3ed5d9b700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3eb8068320 0x7f3eb806a7d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8157320 tx=0x7f3ecc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:04.978+0000 7f3ec6ffd700 1 -- 172.21.15.102:0/670289185 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3ec8053b10 con 0x7f3ed80cd8e0 2023-03-03T09:43:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.213+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ed8157370 con 0x7f3ed80cd8e0 2023-03-03T09:43:05.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.214+0000 7f3ec6ffd700 1 -- 172.21.15.102:0/670289185 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3ec8015090 con 0x7f3ed80cd8e0 2023-03-03T09:43:05.215 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:43:05.216 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:43:05.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3eb8068320 msgr2=0x7f3eb806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:05.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3eb8068320 0x7f3eb806a7d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8157320 tx=0x7f3ecc009040 comp rx=0 tx=0).stop 2023-03-03T09:43:05.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed80cd8e0 msgr2=0x7f3ed80b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:05.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed80cd8e0 0x7f3ed80b2cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009c10 tx=0x7f3ec8004750 comp rx=0 tx=0).stop 2023-03-03T09:43:05.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 shutdown_connections 2023-03-03T09:43:05.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3eb8068320 0x7f3eb806a7d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:05.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ed80cd8e0 0x7f3ed80b2cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:05.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed81593c0 0x7f3ed814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:05.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 --2- 172.21.15.102:0/670289185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ed8153290 0x7f3ed80c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:05.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 >> 172.21.15.102:0/670289185 conn(0x7f3ed8065fb0 msgr2=0x7f3ed8150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:05.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 shutdown_connections 2023-03-03T09:43:05.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:05.217+0000 7f3edcbff700 1 -- 172.21.15.102:0/670289185 wait complete. 2023-03-03T09:43:05.220 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:43:05.246 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2023-03-03T09:43:26.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:26.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:54.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:43:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.510+0000 7f4d1d19d700 1 Processor -- start 2023-03-03T09:43:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.510+0000 7f4d1d19d700 1 -- start start 2023-03-03T09:43:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d1d19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180ced00 0x7f4d180c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d1d19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d180b3970 0x7f4d180c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d1d19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4d180cf0d0 0x7f4d18158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d1d19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d180c4930 con 0x7f4d180b3970 2023-03-03T09:43:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d1d19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d180c3d20 con 0x7f4d180cf0d0 2023-03-03T09:43:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d1d19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1815cba0 con 0x7f4d180ced00 2023-03-03T09:43:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d177fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d180b3970 0x7f4d180c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d177fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d180b3970 0x7f4d180c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38090/0 (socket says 172.21.15.102:38090) 2023-03-03T09:43:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d177fe700 1 -- 172.21.15.102:0/2977594301 learned_addr learned my addr 172.21.15.102:0/2977594301 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d17fff700 1 --2- 172.21.15.102:0/2977594301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180ced00 0x7f4d180c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.512+0000 7f4d1c99c700 1 --2- 172.21.15.102:0/2977594301 >> v2:172.21.15.172:3300/0 conn(0x7f4d180cf0d0 0x7f4d18158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d177fe700 1 -- 172.21.15.102:0/2977594301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180ced00 msgr2=0x7f4d180c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d177fe700 1 --2- 172.21.15.102:0/2977594301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180ced00 0x7f4d180c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d177fe700 1 -- 172.21.15.102:0/2977594301 >> v2:172.21.15.172:3300/0 conn(0x7f4d180cf0d0 msgr2=0x7f4d18158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d177fe700 1 --2- 172.21.15.102:0/2977594301 >> v2:172.21.15.172:3300/0 conn(0x7f4d180cf0d0 0x7f4d18158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d177fe700 1 -- 172.21.15.102:0/2977594301 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1815ce80 con 0x7f4d180b3970 2023-03-03T09:43:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d177fe700 1 --2- 172.21.15.102:0/2977594301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180b3970 0x7f4d180c41b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4d180cb6d0 tx=0x7f4d0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af15046a5d588711 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d16ffd700 1 -- 172.21.15.102:0/2977594301 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c011830 con 0x7f4d180b3970 2023-03-03T09:43:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d16ffd700 1 -- 172.21.15.102:0/2977594301 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0c011990 con 0x7f4d180b3970 2023-03-03T09:43:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.513+0000 7f4d16ffd700 1 -- 172.21.15.102:0/2977594301 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c011d20 con 0x7f4d180b3970 2023-03-03T09:43:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 -- 172.21.15.102:0/2977594301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180b3970 msgr2=0x7f4d180c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/2977594301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180b3970 0x7f4d180c41b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4d180cb6d0 tx=0x7f4d0c009bd0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 -- 172.21.15.102:0/2977594301 shutdown_connections 2023-03-03T09:43:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/2977594301 >> v2:172.21.15.172:3300/0 conn(0x7f4d180cf0d0 0x7f4d18158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/2977594301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180ced00 0x7f4d180c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/2977594301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180b3970 0x7f4d180c41b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 -- 172.21.15.102:0/2977594301 >> 172.21.15.102:0/2977594301 conn(0x7f4d1808b1c0 msgr2=0x7f4d18071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 -- 172.21.15.102:0/2977594301 shutdown_connections 2023-03-03T09:43:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.514+0000 7f4d1d19d700 1 -- 172.21.15.102:0/2977594301 wait complete. 2023-03-03T09:43:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.516+0000 7f4d1d19d700 1 Processor -- start 2023-03-03T09:43:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.516+0000 7f4d1d19d700 1 -- start start 2023-03-03T09:43:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.517+0000 7f4d1d19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180b3970 0x7f4d180c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.517+0000 7f4d1d19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d180ced00 0x7f4d181e5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.517+0000 7f4d1d19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180cf0d0 0x7f4d181f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.517+0000 7f4d1d19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d181f2780 con 0x7f4d180cf0d0 2023-03-03T09:43:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.517+0000 7f4d1d19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d180c65b0 con 0x7f4d180ced00 2023-03-03T09:43:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.517+0000 7f4d1d19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1814c7c0 con 0x7f4d180b3970 2023-03-03T09:43:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d180ced00 0x7f4d181e5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d17fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180b3970 0x7f4d180c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d180ced00 0x7f4d181e5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40796/0 (socket says 172.21.15.102:40796) 2023-03-03T09:43:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 -- 172.21.15.102:0/3166167130 learned_addr learned my addr 172.21.15.102:0/3166167130 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:43:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d1c99c700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180cf0d0 0x7f4d181f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 -- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180b3970 msgr2=0x7f4d180c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180b3970 0x7f4d180c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 -- 172.21.15.102:0/3166167130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180cf0d0 msgr2=0x7f4d181f1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180cf0d0 0x7f4d181f1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 -- 172.21.15.102:0/3166167130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c009550 con 0x7f4d180ced00 2023-03-03T09:43:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.518+0000 7f4d177fe700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d180ced00 0x7f4d181e5710 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c009c10 tx=0x7f4d0c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.519+0000 7f4d14ff9700 1 -- 172.21.15.102:0/3166167130 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c004450 con 0x7f4d180ced00 2023-03-03T09:43:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.519+0000 7f4d14ff9700 1 -- 172.21.15.102:0/3166167130 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0c01e070 con 0x7f4d180ced00 2023-03-03T09:43:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.519+0000 7f4d14ff9700 1 -- 172.21.15.102:0/3166167130 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c0038e0 con 0x7f4d180ced00 2023-03-03T09:43:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.519+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1806c200 con 0x7f4d180ced00 2023-03-03T09:43:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.519+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1806bf20 con 0x7f4d180ced00 2023-03-03T09:43:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.520+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d18159040 con 0x7f4d180ced00 2023-03-03T09:43:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.521+0000 7f4d14ff9700 1 -- 172.21.15.102:0/3166167130 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4d0c00f050 con 0x7f4d180ced00 2023-03-03T09:43:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.522+0000 7f4d14ff9700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d04068320 0x7f4d0406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.523+0000 7f4d14ff9700 1 -- 172.21.15.102:0/3166167130 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4d0c086ae0 con 0x7f4d180ced00 2023-03-03T09:43:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.523+0000 7f4d17fff700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d04068320 0x7f4d0406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.523+0000 7f4d17fff700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d04068320 0x7f4d0406a7d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009b60 tx=0x7f4d08009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.527+0000 7f4d14ff9700 1 -- 172.21.15.102:0/3166167130 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4d0c053170 con 0x7f4d180ced00 2023-03-03T09:43:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.760+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d1804e6e0 con 0x7f4d180ced00 2023-03-03T09:43:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.761+0000 7f4d14ff9700 1 -- 172.21.15.102:0/3166167130 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4d0c020090 con 0x7f4d180ced00 2023-03-03T09:43:54.762 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:43:54.762 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:43:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d04068320 msgr2=0x7f4d0406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d04068320 0x7f4d0406a7d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009b60 tx=0x7f4d08009210 comp rx=0 tx=0).stop 2023-03-03T09:43:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d180ced00 msgr2=0x7f4d181e5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d180ced00 0x7f4d181e5710 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c009c10 tx=0x7f4d0c004750 comp rx=0 tx=0).stop 2023-03-03T09:43:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 shutdown_connections 2023-03-03T09:43:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d04068320 0x7f4d0406a7d0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d180b3970 0x7f4d180c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d180cf0d0 0x7f4d181f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 --2- 172.21.15.102:0/3166167130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d180ced00 0x7f4d181e5710 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 >> 172.21.15.102:0/3166167130 conn(0x7f4d1808b1c0 msgr2=0x7f4d18071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 shutdown_connections 2023-03-03T09:43:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:43:54.763+0000 7f4d1d19d700 1 -- 172.21.15.102:0/3166167130 wait complete. 2023-03-03T09:43:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:43:54.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 63 secs before thrashing 2023-03-03T09:43:56.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:56.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:27.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:27.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:57.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:57.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:57.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:44:58.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.039+0000 7f83a299d700 1 Processor -- start 2023-03-03T09:44:58.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.039+0000 7f83a299d700 1 -- start start 2023-03-03T09:44:58.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a299d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:58.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a299d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f839c0c41b0 0x7f839c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:58.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a299d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c0c3d20 con 0x7f839c0ced00 2023-03-03T09:44:58.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a299d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c0c4930 con 0x7f839c0c41b0 2023-03-03T09:44:58.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a299d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0cf610 con 0x7f839c0b3970 2023-03-03T09:44:58.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a219c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f839c0c41b0 0x7f839c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a199b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f839c0ced00 0x7f839c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47134/0 (socket says 172.21.15.102:47134) 2023-03-03T09:44:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a199b700 1 -- 172.21.15.102:0/1043137416 learned_addr learned my addr 172.21.15.102:0/1043137416 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:44:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.041+0000 7f83a119a700 1 --2- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a219c700 1 -- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 msgr2=0x7f839c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:44:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a219c700 1 --2- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a219c700 1 -- 172.21.15.102:0/1043137416 >> v1:172.21.15.172:6789/0 conn(0x7f839c0ced00 legacy=0x7f839c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a219c700 1 -- 172.21.15.102:0/1043137416 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c15cba0 con 0x7f839c0c41b0 2023-03-03T09:44:58.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a0999700 1 -- 172.21.15.102:0/1043137416 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710763300 0 0) 0x7f839c0c3d20 con 0x7f839c0ced00 2023-03-03T09:44:58.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a219c700 1 --2- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0c41b0 0x7f839c0cf0d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8394009dc0 tx=0x7f839400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ed43248df133b8f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:58.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a0999700 1 -- 172.21.15.102:0/1043137416 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839400e070 con 0x7f839c0c41b0 2023-03-03T09:44:58.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a0999700 1 -- 172.21.15.102:0/1043137416 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839400bae0 con 0x7f839c0c41b0 2023-03-03T09:44:58.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.042+0000 7f83a0999700 1 -- 172.21.15.102:0/1043137416 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839400be90 con 0x7f839c0c41b0 2023-03-03T09:44:58.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 -- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0c41b0 msgr2=0x7f839c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:58.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 --2- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0c41b0 0x7f839c0cf0d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8394009dc0 tx=0x7f839400b330 comp rx=0 tx=0).stop 2023-03-03T09:44:58.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 -- 172.21.15.102:0/1043137416 shutdown_connections 2023-03-03T09:44:58.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 --2- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0c41b0 0x7f839c0cf0d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 --2- 172.21.15.102:0/1043137416 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 -- 172.21.15.102:0/1043137416 >> 172.21.15.102:0/1043137416 conn(0x7f839c08b1c0 msgr2=0x7f839c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:44:58.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 -- 172.21.15.102:0/1043137416 shutdown_connections 2023-03-03T09:44:58.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.043+0000 7f83a299d700 1 -- 172.21.15.102:0/1043137416 wait complete. 2023-03-03T09:44:58.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.045+0000 7f83a299d700 1 Processor -- start 2023-03-03T09:44:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.045+0000 7f83a299d700 1 -- start start 2023-03-03T09:44:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.046+0000 7f83a299d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a299d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f839c0c41b0 0x7f839c15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a299d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0ced00 0x7f839c1eef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a299d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c1eeac0 con 0x7f839c0c41b0 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a299d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0ce7a0 con 0x7f839c0ced00 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a299d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c1ee490 con 0x7f839c0b3970 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36522/0 (socket says 172.21.15.102:36522) 2023-03-03T09:44:58.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 -- 172.21.15.102:0/3242542497 learned_addr learned my addr 172.21.15.102:0/3242542497 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:44:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a119a700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f839c0c41b0 0x7f839c15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a219c700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0ced00 0x7f839c1eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 -- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0ced00 msgr2=0x7f839c1eef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0ced00 0x7f839c1eef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 -- 172.21.15.102:0/3242542497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f839c0c41b0 msgr2=0x7f839c15bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f839c0c41b0 0x7f839c15bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.047+0000 7f83a199b700 1 -- 172.21.15.102:0/3242542497 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8394009a70 con 0x7f839c0b3970 2023-03-03T09:44:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.048+0000 7f83a199b700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c153410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f83980011e0 tx=0x7f839800ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.048+0000 7f83927fc700 1 -- 172.21.15.102:0/3242542497 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398004c80 con 0x7f839c0b3970 2023-03-03T09:44:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.048+0000 7f83927fc700 1 -- 172.21.15.102:0/3242542497 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398004e00 con 0x7f839c0b3970 2023-03-03T09:44:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.048+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c06bf20 con 0x7f839c0b3970 2023-03-03T09:44:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.048+0000 7f83927fc700 1 -- 172.21.15.102:0/3242542497 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398007310 con 0x7f839c0b3970 2023-03-03T09:44:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.048+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c15ce80 con 0x7f839c0b3970 2023-03-03T09:44:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.049+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c06c200 con 0x7f839c0b3970 2023-03-03T09:44:58.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.051+0000 7f83927fc700 1 -- 172.21.15.102:0/3242542497 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f839801a020 con 0x7f839c0b3970 2023-03-03T09:44:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.052+0000 7f83927fc700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f83800682f0 0x7f838006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.052+0000 7f83927fc700 1 -- 172.21.15.102:0/3242542497 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8398085950 con 0x7f839c0b3970 2023-03-03T09:44:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.052+0000 7f83a119a700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f83800682f0 0x7f838006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.053+0000 7f83a119a700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f83800682f0 0x7f838006a7a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f839c0b1800 tx=0x7f838c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:58.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.056+0000 7f83927fc700 1 -- 172.21.15.102:0/3242542497 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8398051fd0 con 0x7f839c0b3970 2023-03-03T09:44:58.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.291+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f839c04e6e0 con 0x7f839c0b3970 2023-03-03T09:44:58.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.292+0000 7f83927fc700 1 -- 172.21.15.102:0/3242542497 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f83980555f0 con 0x7f839c0b3970 2023-03-03T09:44:58.294 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:44:58.294 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:44:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f83800682f0 msgr2=0x7f838006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f83800682f0 0x7f838006a7a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f839c0b1800 tx=0x7f838c009210 comp rx=0 tx=0).stop 2023-03-03T09:44:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 msgr2=0x7f839c153410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c153410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f83980011e0 tx=0x7f839800ca30 comp rx=0 tx=0).stop 2023-03-03T09:44:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 shutdown_connections 2023-03-03T09:44:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f83800682f0 0x7f838006a7a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f839c0b3970 0x7f839c153410 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f839c0ced00 0x7f839c1eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 --2- 172.21.15.102:0/3242542497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f839c0c41b0 0x7f839c15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 >> 172.21.15.102:0/3242542497 conn(0x7f839c08b1c0 msgr2=0x7f839c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:44:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 shutdown_connections 2023-03-03T09:44:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:44:58.295+0000 7f83a299d700 1 -- 172.21.15.102:0/3242542497 wait complete. 2023-03-03T09:44:58.298 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:44:58.323 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T09:45:27.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:27.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:57.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:57.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:27.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:27.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:49.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:46:49.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.572+0000 7fbcf9d9d700 1 Processor -- start 2023-03-03T09:46:49.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.572+0000 7fbcf9d9d700 1 -- start start 2023-03-03T09:46:49.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf9d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbcf40c5a90 0x7fbcf40c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:49.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf9d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf40c3d20 con 0x7fbcf40b3970 2023-03-03T09:46:49.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf9d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf415cba0 con 0x7fbcf40ced00 2023-03-03T09:46:49.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf9d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf40c4930 con 0x7fbcf40c5a90 2023-03-03T09:46:49.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf8d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbcf40ced00 0x7fbcf4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35120/0 (socket says 172.21.15.102:35120) 2023-03-03T09:46:49.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf8d9b700 1 -- 172.21.15.102:0/1145107082 learned_addr learned my addr 172.21.15.102:0/1145107082 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:46:49.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf959c700 1 --2- 172.21.15.102:0/1145107082 >> v2:172.21.15.172:3300/0 conn(0x7fbcf40c5a90 0x7fbcf40c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:49.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.574+0000 7fbcf37fe700 1 -- 172.21.15.102:0/1145107082 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147795191 0 0) 0x7fbcf415cba0 con 0x7fbcf40ced00 2023-03-03T09:46:49.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf37fe700 1 -- 172.21.15.102:0/1145107082 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce40035b0 con 0x7fbcf40ced00 2023-03-03T09:46:49.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf37fe700 1 -- 172.21.15.102:0/1145107082 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183402064 0 0) 0x7fbcf40c3d20 con 0x7fbcf40b3970 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf37fe700 1 -- 172.21.15.102:0/1145107082 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf415cba0 con 0x7fbcf40b3970 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf959c700 1 -- 172.21.15.102:0/1145107082 >> v1:172.21.15.172:6789/0 conn(0x7fbcf40ced00 legacy=0x7fbcf4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf959c700 1 -- 172.21.15.102:0/1145107082 >> v1:172.21.15.102:6789/0 conn(0x7fbcf40b3970 legacy=0x7fbcf4161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf959c700 1 -- 172.21.15.102:0/1145107082 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf415ce80 con 0x7fbcf40c5a90 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf959c700 1 --2- 172.21.15.102:0/1145107082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 0x7fbcf40c41b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009d90 tx=0x7fbcec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3c8730b8ad32b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf37fe700 1 -- 172.21.15.102:0/1145107082 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec00e070 con 0x7fbcf40c5a90 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf37fe700 1 -- 172.21.15.102:0/1145107082 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec00bb20 con 0x7fbcf40c5a90 2023-03-03T09:46:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.575+0000 7fbcf37fe700 1 -- 172.21.15.102:0/1145107082 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec012430 con 0x7fbcf40c5a90 2023-03-03T09:46:49.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.576+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/1145107082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 msgr2=0x7fbcf40c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:49.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.576+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/1145107082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 0x7fbcf40c41b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009d90 tx=0x7fbcec00b330 comp rx=0 tx=0).stop 2023-03-03T09:46:49.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.576+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/1145107082 shutdown_connections 2023-03-03T09:46:49.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.576+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/1145107082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 0x7fbcf40c41b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:49.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.576+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/1145107082 >> 172.21.15.102:0/1145107082 conn(0x7fbcf408b1c0 msgr2=0x7fbcf4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:49.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.576+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/1145107082 shutdown_connections 2023-03-03T09:46:49.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.576+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/1145107082 wait complete. 2023-03-03T09:46:49.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.578+0000 7fbcf9d9d700 1 Processor -- start 2023-03-03T09:46:49.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.578+0000 7fbcf9d9d700 1 -- start start 2023-03-03T09:46:49.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.579+0000 7fbcf9d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcf40b3970 0x7fbcf415a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:49.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.579+0000 7fbcf9d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 0x7fbcf414cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:49.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.579+0000 7fbcf9d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcf40ced00 0x7fbcf414d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:49.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.579+0000 7fbcf9d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf41e4840 con 0x7fbcf40b3970 2023-03-03T09:46:49.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.579+0000 7fbcf9d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf415b200 con 0x7fbcf40c5a90 2023-03-03T09:46:49.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.579+0000 7fbcf9d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf41e35c0 con 0x7fbcf40ced00 2023-03-03T09:46:49.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf8d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcf40b3970 0x7fbcf415a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:49.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf959c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcf40ced00 0x7fbcf414d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:49.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf959c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcf40ced00 0x7fbcf414d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44972/0 (socket says 172.21.15.102:44972) 2023-03-03T09:46:49.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 0x7fbcf414cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:49.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf959c700 1 -- 172.21.15.102:0/2381113563 learned_addr learned my addr 172.21.15.102:0/2381113563 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:46:49.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf8d9b700 1 -- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcf40ced00 msgr2=0x7fbcf414d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:49.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf8d9b700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcf40ced00 0x7fbcf414d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:49.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf8d9b700 1 -- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 msgr2=0x7fbcf414cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:49.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf8d9b700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 0x7fbcf414cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf8d9b700 1 -- 172.21.15.102:0/2381113563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec009a70 con 0x7fbcf40b3970 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf959c700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcf40ced00 0x7fbcf414d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.580+0000 7fbcf8d9b700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcf40b3970 0x7fbcf415a260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c500 tx=0x7fbce800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.581+0000 7fbcf17fa700 1 -- 172.21.15.102:0/2381113563 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8005d70 con 0x7fbcf40b3970 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.581+0000 7fbcf17fa700 1 -- 172.21.15.102:0/2381113563 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce80070d0 con 0x7fbcf40b3970 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.581+0000 7fbcf17fa700 1 -- 172.21.15.102:0/2381113563 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8007480 con 0x7fbcf40b3970 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.581+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf406c200 con 0x7fbcf40b3970 2023-03-03T09:46:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.581+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf406bf20 con 0x7fbcf40b3970 2023-03-03T09:46:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.582+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf415c890 con 0x7fbcf40b3970 2023-03-03T09:46:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.583+0000 7fbcf17fa700 1 -- 172.21.15.102:0/2381113563 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbce8019030 con 0x7fbcf40b3970 2023-03-03T09:46:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.584+0000 7fbcf17fa700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcd80682f0 0x7fbcd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:49.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.584+0000 7fbcf17fa700 1 -- 172.21.15.102:0/2381113563 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbce8085070 con 0x7fbcf40b3970 2023-03-03T09:46:49.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.585+0000 7fbcf3fff700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcd80682f0 0x7fbcd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:49.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.585+0000 7fbcf3fff700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcd80682f0 0x7fbcd806a7a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40cb4f0 tx=0x7fbce0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:49.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.588+0000 7fbcf17fa700 1 -- 172.21.15.102:0/2381113563 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbce8051840 con 0x7fbcf40b3970 2023-03-03T09:46:49.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.821+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcf404e6e0 con 0x7fbcf40b3970 2023-03-03T09:46:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.822+0000 7fbcf17fa700 1 -- 172.21.15.102:0/2381113563 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbce8054e60 con 0x7fbcf40b3970 2023-03-03T09:46:49.824 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:46:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:46:49.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcd80682f0 msgr2=0x7fbcd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcd80682f0 0x7fbcd806a7a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40cb4f0 tx=0x7fbce0009040 comp rx=0 tx=0).stop 2023-03-03T09:46:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcf40b3970 msgr2=0x7fbcf415a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcf40b3970 0x7fbcf415a260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c500 tx=0x7fbce800c810 comp rx=0 tx=0).stop 2023-03-03T09:46:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 shutdown_connections 2023-03-03T09:46:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbcd80682f0 0x7fbcd806a7a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbcf40ced00 0x7fbcf414d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcf40b3970 0x7fbcf415a260 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 --2- 172.21.15.102:0/2381113563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf40c5a90 0x7fbcf414cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 >> 172.21.15.102:0/2381113563 conn(0x7fbcf408b1c0 msgr2=0x7fbcf4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.824+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 shutdown_connections 2023-03-03T09:46:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:46:49.825+0000 7fbcf9d9d700 1 -- 172.21.15.102:0/2381113563 wait complete. 2023-03-03T09:46:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:46:49.853 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 65 secs before thrashing 2023-03-03T09:46:57.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:57.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:28.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:28.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:54.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:47:55.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.105+0000 7fa99f59e700 1 Processor -- start 2023-03-03T09:47:55.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.105+0000 7fa99f59e700 1 -- start start 2023-03-03T09:47:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99f59e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9a00ced00 0x7fa9a00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99f59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9a00b3970 0x7fa9a00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99f59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a00c3d20 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99f59e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a00c4930 con 0x7fa9a00ced00 2023-03-03T09:47:55.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99f59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a00cf570 con 0x7fa9a00b3970 2023-03-03T09:47:55.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99dd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9a00b3970 0x7fa9a00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9a00ced00 0x7fa9a00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9a00ced00 0x7fa9a00c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37010/0 (socket says 172.21.15.102:37010) 2023-03-03T09:47:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99e59c700 1 -- 172.21.15.102:0/3753140322 learned_addr learned my addr 172.21.15.102:0/3753140322 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:47:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99dd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9a00b3970 0x7fa9a00c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53746/0 (socket says 172.21.15.102:53746) 2023-03-03T09:47:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.107+0000 7fa99ed9d700 1 --1- 172.21.15.102:0/3753140322 >> v1:172.21.15.102:6789/0 conn(0x7fa9a00cf0d0 0x7fa9a0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50952/0 (socket says 172.21.15.102:50952) 2023-03-03T09:47:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99e59c700 1 -- 172.21.15.102:0/3753140322 >> v2:172.21.15.172:3300/0 conn(0x7fa9a00b3970 msgr2=0x7fa9a00c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:47:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99e59c700 1 --2- 172.21.15.102:0/3753140322 >> v2:172.21.15.172:3300/0 conn(0x7fa9a00b3970 0x7fa9a00c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99e59c700 1 -- 172.21.15.102:0/3753140322 >> v1:172.21.15.102:6789/0 conn(0x7fa9a00cf0d0 legacy=0x7fa9a0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99e59c700 1 -- 172.21.15.102:0/3753140322 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a015cba0 con 0x7fa9a00ced00 2023-03-03T09:47:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99e59c700 1 --2- 172.21.15.102:0/3753140322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00ced00 0x7fa9a00c5a90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00c4ba0 tx=0x7fa990009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3c9eacfababe59 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:47:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99d59a700 1 -- 172.21.15.102:0/3753140322 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990011830 con 0x7fa9a00ced00 2023-03-03T09:47:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99d59a700 1 -- 172.21.15.102:0/3753140322 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa990011990 con 0x7fa9a00ced00 2023-03-03T09:47:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.108+0000 7fa99d59a700 1 -- 172.21.15.102:0/3753140322 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990011d20 con 0x7fa9a00ced00 2023-03-03T09:47:55.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 -- 172.21.15.102:0/3753140322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00ced00 msgr2=0x7fa9a00c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:55.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 --2- 172.21.15.102:0/3753140322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00ced00 0x7fa9a00c5a90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00c4ba0 tx=0x7fa990009bd0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 -- 172.21.15.102:0/3753140322 shutdown_connections 2023-03-03T09:47:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 --2- 172.21.15.102:0/3753140322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00ced00 0x7fa9a00c5a90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 --2- 172.21.15.102:0/3753140322 >> v2:172.21.15.172:3300/0 conn(0x7fa9a00b3970 0x7fa9a00c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 -- 172.21.15.102:0/3753140322 >> 172.21.15.102:0/3753140322 conn(0x7fa9a008b1c0 msgr2=0x7fa9a0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:47:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 -- 172.21.15.102:0/3753140322 shutdown_connections 2023-03-03T09:47:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.109+0000 7fa99f59e700 1 -- 172.21.15.102:0/3753140322 wait complete. 2023-03-03T09:47:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.111+0000 7fa99f59e700 1 Processor -- start 2023-03-03T09:47:55.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.111+0000 7fa99f59e700 1 -- start start 2023-03-03T09:47:55.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.112+0000 7fa99f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00b3970 0x7fa9a014dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.112+0000 7fa99f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a00ced00 0x7fa9a015b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.112+0000 7fa99f59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00cf0d0 0x7fa9a00c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.112+0000 7fa99f59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01512c0 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.112+0000 7fa99f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a0071c50 con 0x7fa9a00ced00 2023-03-03T09:47:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.112+0000 7fa99f59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a0151d90 con 0x7fa9a00b3970 2023-03-03T09:47:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00cf0d0 0x7fa9a00c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00b3970 0x7fa9a014dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00b3970 0x7fa9a014dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:40344/0 (socket says 172.21.15.102:40344) 2023-03-03T09:47:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99e59c700 1 -- 172.21.15.102:0/2631868920 learned_addr learned my addr 172.21.15.102:0/2631868920 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:47:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99dd9b700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a00ced00 0x7fa9a015b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99ed9d700 1 -- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00b3970 msgr2=0x7fa9a014dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99ed9d700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00b3970 0x7fa9a014dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99ed9d700 1 -- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a00ced00 msgr2=0x7fa9a015b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99ed9d700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a00ced00 0x7fa9a015b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99ed9d700 1 -- 172.21.15.102:0/2631868920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa990009550 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99e59c700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00b3970 0x7fa9a014dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:47:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.113+0000 7fa99dd9b700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a00ced00 0x7fa9a015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:47:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.114+0000 7fa99ed9d700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00cf0d0 0x7fa9a00c82d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa99800cb10 tx=0x7fa99800ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:47:55.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.114+0000 7fa98effd700 1 -- 172.21.15.102:0/2631868920 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99800a980 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.114+0000 7fa98effd700 1 -- 172.21.15.102:0/2631868920 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99800aae0 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.114+0000 7fa98effd700 1 -- 172.21.15.102:0/2631868920 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99800ae90 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.114+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a006bf20 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.114+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a015ce80 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.115+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a006c200 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.117+0000 7fa98effd700 1 -- 172.21.15.102:0/2631868920 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa998019030 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.118+0000 7fa98effd700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9800682f0 0x7fa98006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:55.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.119+0000 7fa98effd700 1 -- 172.21.15.102:0/2631868920 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa998085880 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.119+0000 7fa99e59c700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9800682f0 0x7fa98006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:55.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.120+0000 7fa99e59c700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9800682f0 0x7fa98006a7a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa990009c10 tx=0x7fa990008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:47:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.123+0000 7fa98effd700 1 -- 172.21.15.102:0/2631868920 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa998051f90 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.358+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9a00c9ef0 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.358+0000 7fa98effd700 1 -- 172.21.15.102:0/2631868920 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa998001bc0 con 0x7fa9a00cf0d0 2023-03-03T09:47:55.360 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:47:55.360 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:47:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9800682f0 msgr2=0x7fa98006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9800682f0 0x7fa98006a7a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa990009c10 tx=0x7fa990008030 comp rx=0 tx=0).stop 2023-03-03T09:47:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00cf0d0 msgr2=0x7fa9a00c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00cf0d0 0x7fa9a00c82d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa99800cb10 tx=0x7fa99800ce20 comp rx=0 tx=0).stop 2023-03-03T09:47:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 shutdown_connections 2023-03-03T09:47:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9800682f0 0x7fa98006a7a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a00b3970 0x7fa9a014dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a00cf0d0 0x7fa9a00c82d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 --2- 172.21.15.102:0/2631868920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a00ced00 0x7fa9a015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:55.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 >> 172.21.15.102:0/2631868920 conn(0x7fa9a008b1c0 msgr2=0x7fa9a00cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:47:55.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 shutdown_connections 2023-03-03T09:47:55.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:47:55.361+0000 7fa99f59e700 1 -- 172.21.15.102:0/2631868920 wait complete. 2023-03-03T09:47:55.366 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:47:55.390 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-03-03T09:47:58.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:58.192 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:18.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:48:18.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.637+0000 7f460979d700 1 Processor -- start 2023-03-03T09:48:18.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.637+0000 7f460979d700 1 -- start start 2023-03-03T09:48:18.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.639+0000 7f460979d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f46040b3970 0x7f46040c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:18.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.639+0000 7f460979d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040c3d20 con 0x7f46040c41b0 2023-03-03T09:48:18.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.639+0000 7f460979d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460415cba0 con 0x7f46040ced00 2023-03-03T09:48:18.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.639+0000 7f460979d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46040c4930 con 0x7f46040b3970 2023-03-03T09:48:18.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.639+0000 7f45fbfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f46040b3970 0x7f46040c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:18.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.639+0000 7f4603fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f46040ced00 0x7f4604158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47256/0 (socket says 172.21.15.102:47256) 2023-03-03T09:48:18.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.639+0000 7f4603fff700 1 -- 172.21.15.102:0/2308151497 learned_addr learned my addr 172.21.15.102:0/2308151497 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:48:18.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f45fbfff700 1 -- 172.21.15.102:0/2308151497 >> v1:172.21.15.172:6789/0 conn(0x7f46040ced00 legacy=0x7f4604158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:18.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f45fbfff700 1 -- 172.21.15.102:0/2308151497 >> v1:172.21.15.102:6789/0 conn(0x7f46040c41b0 legacy=0x7f4604163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f45fbfff700 1 -- 172.21.15.102:0/2308151497 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460415ce80 con 0x7f46040b3970 2023-03-03T09:48:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f46037fe700 1 -- 172.21.15.102:0/2308151497 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104957409 0 0) 0x7f460415cba0 con 0x7f46040ced00 2023-03-03T09:48:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f45fbfff700 1 --2- 172.21.15.102:0/2308151497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040b3970 0x7f46040c5a90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009840 tx=0x7f45ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=559cc286f6dcae70 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f46037fe700 1 -- 172.21.15.102:0/2308151497 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec004230 con 0x7f46040b3970 2023-03-03T09:48:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f46037fe700 1 -- 172.21.15.102:0/2308151497 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec004390 con 0x7f46040b3970 2023-03-03T09:48:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.640+0000 7f46037fe700 1 -- 172.21.15.102:0/2308151497 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec0036a0 con 0x7f46040b3970 2023-03-03T09:48:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.641+0000 7f460979d700 1 -- 172.21.15.102:0/2308151497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040b3970 msgr2=0x7f46040c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.641+0000 7f460979d700 1 --2- 172.21.15.102:0/2308151497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040b3970 0x7f46040c5a90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009840 tx=0x7f45ec009b50 comp rx=0 tx=0).stop 2023-03-03T09:48:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.641+0000 7f460979d700 1 -- 172.21.15.102:0/2308151497 shutdown_connections 2023-03-03T09:48:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.641+0000 7f460979d700 1 --2- 172.21.15.102:0/2308151497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040b3970 0x7f46040c5a90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.641+0000 7f460979d700 1 -- 172.21.15.102:0/2308151497 >> 172.21.15.102:0/2308151497 conn(0x7f460408b1c0 msgr2=0x7f4604071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.641+0000 7f460979d700 1 -- 172.21.15.102:0/2308151497 shutdown_connections 2023-03-03T09:48:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.641+0000 7f460979d700 1 -- 172.21.15.102:0/2308151497 wait complete. 2023-03-03T09:48:18.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.643+0000 7f460979d700 1 Processor -- start 2023-03-03T09:48:18.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.643+0000 7f460979d700 1 -- start start 2023-03-03T09:48:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f460979d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46040b3970 0x7f46040c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f460979d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46040c41b0 0x7f46040c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f460979d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040ced00 0x7f46041f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f460979d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46041f22c0 con 0x7f46040c41b0 2023-03-03T09:48:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f460979d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46040c73a0 con 0x7f46040ced00 2023-03-03T09:48:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f460979d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46040c8a90 con 0x7f46040b3970 2023-03-03T09:48:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f45fbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46040c41b0 0x7f46040c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f4603fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46040b3970 0x7f46040c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.644+0000 7f4608f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040ced00 0x7f46041f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46040c41b0 0x7f46040c7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57980/0 (socket says 172.21.15.102:57980) 2023-03-03T09:48:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 -- 172.21.15.102:0/593555926 learned_addr learned my addr 172.21.15.102:0/593555926 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:48:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 -- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46040b3970 msgr2=0x7f46040c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46040b3970 0x7f46040c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 -- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040ced00 msgr2=0x7f46041f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040ced00 0x7f46041f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 -- 172.21.15.102:0/593555926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec0094a0 con 0x7f46040c41b0 2023-03-03T09:48:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.645+0000 7f45fbfff700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46040c41b0 0x7f46040c7f10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f460408ba70 tx=0x7f45ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.646+0000 7f46017fa700 1 -- 172.21.15.102:0/593555926 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec003cd0 con 0x7f46040c41b0 2023-03-03T09:48:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.646+0000 7f46017fa700 1 -- 172.21.15.102:0/593555926 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec01f070 con 0x7f46040c41b0 2023-03-03T09:48:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.646+0000 7f46017fa700 1 -- 172.21.15.102:0/593555926 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec011870 con 0x7f46040c41b0 2023-03-03T09:48:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.646+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460406c200 con 0x7f46040c41b0 2023-03-03T09:48:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.646+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460406bf20 con 0x7f46040c41b0 2023-03-03T09:48:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.647+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460415c890 con 0x7f46040c41b0 2023-03-03T09:48:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.648+0000 7f46017fa700 1 -- 172.21.15.102:0/593555926 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f45ec02a020 con 0x7f46040c41b0 2023-03-03T09:48:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.649+0000 7f46017fa700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45e40682f0 0x7f45e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.649+0000 7f46017fa700 1 -- 172.21.15.102:0/593555926 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f45ec087320 con 0x7f46040c41b0 2023-03-03T09:48:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.650+0000 7f4603fff700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45e40682f0 0x7f45e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.650+0000 7f4603fff700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45e40682f0 0x7f45e406a7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f45f4007b10 tx=0x7f45f4007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:18.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.654+0000 7f46017fa700 1 -- 172.21.15.102:0/593555926 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f45ec053a30 con 0x7f46040c41b0 2023-03-03T09:48:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.890+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f460415c890 con 0x7f46040c41b0 2023-03-03T09:48:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.891+0000 7f46017fa700 1 -- 172.21.15.102:0/593555926 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f45ec057050 con 0x7f46040c41b0 2023-03-03T09:48:18.893 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:48:18.894 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:48:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.893+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45e40682f0 msgr2=0x7f45e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.893+0000 7f460979d700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45e40682f0 0x7f45e406a7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f45f4007b10 tx=0x7f45f4007a60 comp rx=0 tx=0).stop 2023-03-03T09:48:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.893+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46040c41b0 msgr2=0x7f46040c7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.893+0000 7f460979d700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46040c41b0 0x7f46040c7f10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f460408ba70 tx=0x7f45ec004750 comp rx=0 tx=0).stop 2023-03-03T09:48:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 shutdown_connections 2023-03-03T09:48:18.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f45e40682f0 0x7f45e406a7a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f46040b3970 0x7f46040c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46040ced00 0x7f46041f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 --2- 172.21.15.102:0/593555926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46040c41b0 0x7f46040c7f10 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 >> 172.21.15.102:0/593555926 conn(0x7f460408b1c0 msgr2=0x7f46040669c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 shutdown_connections 2023-03-03T09:48:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:18.894+0000 7f460979d700 1 -- 172.21.15.102:0/593555926 wait complete. 2023-03-03T09:48:18.898 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:48:18.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T09:48:28.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:28.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:46.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:48:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.177+0000 7fe1d0f9d700 1 Processor -- start 2023-03-03T09:48:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.178+0000 7fe1d0f9d700 1 -- start start 2023-03-03T09:48:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.179+0000 7fe1d0f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1cc0ced00 0x7fe1cc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.179+0000 7fe1d0f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0c41b0 0x7fe1cc0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.179+0000 7fe1d0f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc0c3d20 con 0x7fe1cc0b3970 2023-03-03T09:48:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.179+0000 7fe1d0f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1cc0c4930 con 0x7fe1cc0ced00 2023-03-03T09:48:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.179+0000 7fe1d0f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0cf610 con 0x7fe1cc0c41b0 2023-03-03T09:48:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1cbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0c41b0 0x7fe1cc0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1cb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1cc0ced00 0x7fe1cc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1caffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe1cc0b3970 0x7fe1cc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55186/0 (socket says 172.21.15.102:55186) 2023-03-03T09:48:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1caffd700 1 -- 172.21.15.102:0/1273252364 learned_addr learned my addr 172.21.15.102:0/1273252364 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:48:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1cbfff700 1 -- 172.21.15.102:0/1273252364 >> v2:172.21.15.172:3300/0 conn(0x7fe1cc0ced00 msgr2=0x7fe1cc0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:48:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1cbfff700 1 --2- 172.21.15.102:0/1273252364 >> v2:172.21.15.172:3300/0 conn(0x7fe1cc0ced00 0x7fe1cc0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1cbfff700 1 -- 172.21.15.102:0/1273252364 >> v1:172.21.15.102:6789/0 conn(0x7fe1cc0b3970 legacy=0x7fe1cc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1cbfff700 1 -- 172.21.15.102:0/1273252364 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc15cba0 con 0x7fe1cc0c41b0 2023-03-03T09:48:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.180+0000 7fe1cbfff700 1 --2- 172.21.15.102:0/1273252364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0c41b0 0x7fe1cc0cf0d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009d90 tx=0x7fe1c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=636d0cc78b117666 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1ca7fc700 1 -- 172.21.15.102:0/1273252364 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c400e070 con 0x7fe1cc0c41b0 2023-03-03T09:48:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1ca7fc700 1 -- 172.21.15.102:0/1273252364 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c400bb20 con 0x7fe1cc0c41b0 2023-03-03T09:48:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1ca7fc700 1 -- 172.21.15.102:0/1273252364 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4012430 con 0x7fe1cc0c41b0 2023-03-03T09:48:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/1273252364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0c41b0 msgr2=0x7fe1cc0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/1273252364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0c41b0 0x7fe1cc0cf0d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009d90 tx=0x7fe1c400b330 comp rx=0 tx=0).stop 2023-03-03T09:48:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/1273252364 shutdown_connections 2023-03-03T09:48:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/1273252364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0c41b0 0x7fe1cc0cf0d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/1273252364 >> v2:172.21.15.172:3300/0 conn(0x7fe1cc0ced00 0x7fe1cc0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/1273252364 >> 172.21.15.102:0/1273252364 conn(0x7fe1cc08b1c0 msgr2=0x7fe1cc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/1273252364 shutdown_connections 2023-03-03T09:48:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.181+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/1273252364 wait complete. 2023-03-03T09:48:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.183+0000 7fe1d0f9d700 1 Processor -- start 2023-03-03T09:48:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.184+0000 7fe1d0f9d700 1 -- start start 2023-03-03T09:48:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1d0f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0b3970 0x7fe1cc1eef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1d0f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1cc0c41b0 0x7fe1cc1514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1d0f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1cc0ced00 0x7fe1cc151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1d0f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc15b380 con 0x7fe1cc0ced00 2023-03-03T09:48:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1d0f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc15bee0 con 0x7fe1cc0c41b0 2023-03-03T09:48:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1d0f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc15b640 con 0x7fe1cc0b3970 2023-03-03T09:48:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1cb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0b3970 0x7fe1cc1eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1caffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1cc0c41b0 0x7fe1cc1514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1cb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0b3970 0x7fe1cc1eef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39462/0 (socket says 172.21.15.102:39462) 2023-03-03T09:48:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1cb7fe700 1 -- 172.21.15.102:0/4076980509 learned_addr learned my addr 172.21.15.102:0/4076980509 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:48:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.185+0000 7fe1cbfff700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1cc0ced00 0x7fe1cc151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1cb7fe700 1 -- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1cc0c41b0 msgr2=0x7fe1cc1514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1cb7fe700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1cc0c41b0 0x7fe1cc1514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1cb7fe700 1 -- 172.21.15.102:0/4076980509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1cc0ced00 msgr2=0x7fe1cc151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1cb7fe700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1cc0ced00 0x7fe1cc151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1cb7fe700 1 -- 172.21.15.102:0/4076980509 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c4009a70 con 0x7fe1cc0b3970 2023-03-03T09:48:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1cb7fe700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0b3970 0x7fe1cc1eef40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00e9b0 tx=0x7fe1bc00ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1b3fff700 1 -- 172.21.15.102:0/4076980509 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc009710 con 0x7fe1cc0b3970 2023-03-03T09:48:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1b3fff700 1 -- 172.21.15.102:0/4076980509 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc00ee50 con 0x7fe1cc0b3970 2023-03-03T09:48:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1cc06bf20 con 0x7fe1cc0b3970 2023-03-03T09:48:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1b3fff700 1 -- 172.21.15.102:0/4076980509 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc00ca50 con 0x7fe1cc0b3970 2023-03-03T09:48:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.186+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1cc15ce80 con 0x7fe1cc0b3970 2023-03-03T09:48:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.187+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1cc06c200 con 0x7fe1cc0b3970 2023-03-03T09:48:47.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.189+0000 7fe1b3fff700 1 -- 172.21.15.102:0/4076980509 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe1bc019030 con 0x7fe1cc0b3970 2023-03-03T09:48:47.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.190+0000 7fe1b3fff700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1b80682f0 0x7fe1b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:47.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.190+0000 7fe1b3fff700 1 -- 172.21.15.102:0/4076980509 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe1bc085ce0 con 0x7fe1cc0b3970 2023-03-03T09:48:47.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.190+0000 7fe1caffd700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1b80682f0 0x7fe1b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:47.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.191+0000 7fe1caffd700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1b80682f0 0x7fe1b806a7a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc0cd180 tx=0x7fe1c0007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:47.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.193+0000 7fe1b3fff700 1 -- 172.21.15.102:0/4076980509 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1bc055e10 con 0x7fe1cc0b3970 2023-03-03T09:48:47.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.428+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1cc04e6e0 con 0x7fe1cc0b3970 2023-03-03T09:48:47.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.429+0000 7fe1b3fff700 1 -- 172.21.15.102:0/4076980509 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe1bc021090 con 0x7fe1cc0b3970 2023-03-03T09:48:47.430 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:48:47.431 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:48:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1b80682f0 msgr2=0x7fe1b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1b80682f0 0x7fe1b806a7a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc0cd180 tx=0x7fe1c0007f10 comp rx=0 tx=0).stop 2023-03-03T09:48:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0b3970 msgr2=0x7fe1cc1eef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0b3970 0x7fe1cc1eef40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00e9b0 tx=0x7fe1bc00ecc0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 shutdown_connections 2023-03-03T09:48:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1b80682f0 0x7fe1b806a7a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1cc0b3970 0x7fe1cc1eef40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1cc0ced00 0x7fe1cc151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 --2- 172.21.15.102:0/4076980509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1cc0c41b0 0x7fe1cc1514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 >> 172.21.15.102:0/4076980509 conn(0x7fe1cc08b1c0 msgr2=0x7fe1cc1e1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:47.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 shutdown_connections 2023-03-03T09:48:47.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:48:47.432+0000 7fe1d0f9d700 1 -- 172.21.15.102:0/4076980509 wait complete. 2023-03-03T09:48:47.435 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:48:47.461 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T09:48:58.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:58.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:28.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:28.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:58.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:59.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:00.002 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T09:50:00.000+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:50:14.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:50:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.735+0000 7f09fffff700 1 Processor -- start 2023-03-03T09:50:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.735+0000 7f09fffff700 1 -- start start 2023-03-03T09:50:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.737+0000 7f09fffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000b3970 0x7f0a000c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.737+0000 7f09fffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a000c41b0 0x7f0a000cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.737+0000 7f09fffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000c3d20 con 0x7f0a000ced00 2023-03-03T09:50:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.737+0000 7f09fffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a000c4930 con 0x7f0a000c41b0 2023-03-03T09:50:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.737+0000 7f09fffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000cf610 con 0x7f0a000b3970 2023-03-03T09:50:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.737+0000 7f09fe7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000b3970 0x7f0a000c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.737+0000 7f09ff7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a000c41b0 0x7f0a000cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09ff7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a000c41b0 0x7f0a000cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60050/0 (socket says 172.21.15.102:60050) 2023-03-03T09:50:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09ff7fe700 1 -- 172.21.15.102:0/3379636408 learned_addr learned my addr 172.21.15.102:0/3379636408 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:50:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09feffd700 1 --1- 172.21.15.102:0/3379636408 >> v1:172.21.15.172:6789/0 conn(0x7f0a000ced00 0x7f0a00158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41464/0 (socket says 172.21.15.102:41464) 2023-03-03T09:50:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09ff7fe700 1 -- 172.21.15.102:0/3379636408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000b3970 msgr2=0x7f0a000c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09ff7fe700 1 --2- 172.21.15.102:0/3379636408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000b3970 0x7f0a000c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09ff7fe700 1 -- 172.21.15.102:0/3379636408 >> v1:172.21.15.172:6789/0 conn(0x7f0a000ced00 legacy=0x7f0a00158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09ff7fe700 1 -- 172.21.15.102:0/3379636408 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0015cba0 con 0x7f0a000c41b0 2023-03-03T09:50:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09fdffb700 1 -- 172.21.15.102:0/3379636408 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263505578 0 0) 0x7f0a000c3d20 con 0x7f0a000ced00 2023-03-03T09:50:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.738+0000 7f09ff7fe700 1 --2- 172.21.15.102:0/3379636408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c41b0 0x7f0a000cf0d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009e80 tx=0x7f09f800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e4327b11cb9ae81 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fdffb700 1 -- 172.21.15.102:0/3379636408 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800e070 con 0x7f0a000c41b0 2023-03-03T09:50:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fdffb700 1 -- 172.21.15.102:0/3379636408 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f800b8d0 con 0x7f0a000c41b0 2023-03-03T09:50:14.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fdffb700 1 -- 172.21.15.102:0/3379636408 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800bc80 con 0x7f0a000c41b0 2023-03-03T09:50:14.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 -- 172.21.15.102:0/3379636408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c41b0 msgr2=0x7f0a000cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:14.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 --2- 172.21.15.102:0/3379636408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c41b0 0x7f0a000cf0d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009e80 tx=0x7f09f800b330 comp rx=0 tx=0).stop 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 -- 172.21.15.102:0/3379636408 shutdown_connections 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 --2- 172.21.15.102:0/3379636408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000c41b0 0x7f0a000cf0d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 --2- 172.21.15.102:0/3379636408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000b3970 0x7f0a000c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 -- 172.21.15.102:0/3379636408 >> 172.21.15.102:0/3379636408 conn(0x7f0a0008b1c0 msgr2=0x7f0a00071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 -- 172.21.15.102:0/3379636408 shutdown_connections 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.739+0000 7f09fffff700 1 -- 172.21.15.102:0/3379636408 wait complete. 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.741+0000 7f09fffff700 1 Processor -- start 2023-03-03T09:50:14.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.741+0000 7f09fffff700 1 -- start start 2023-03-03T09:50:14.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.742+0000 7f09fffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000b3970 0x7f0a0014e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:14.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.742+0000 7f09fffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000c41b0 0x7f0a00153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:14.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.742+0000 7f09fffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a000ced00 0x7f0a00154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:14.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.742+0000 7f09fffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00155b40 con 0x7f0a000ced00 2023-03-03T09:50:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.742+0000 7f09fffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a001538b0 con 0x7f0a000b3970 2023-03-03T09:50:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.742+0000 7f09fffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0015bbd0 con 0x7f0a000c41b0 2023-03-03T09:50:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000b3970 0x7f0a0014e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09fe7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000c41b0 0x7f0a00153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000b3970 0x7f0a0014e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60060/0 (socket says 172.21.15.102:60060) 2023-03-03T09:50:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 -- 172.21.15.102:0/1888109754 learned_addr learned my addr 172.21.15.102:0/1888109754 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:50:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09ff7fe700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a000ced00 0x7f0a00154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 -- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000c41b0 msgr2=0x7f0a00153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000c41b0 0x7f0a00153d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 -- 172.21.15.102:0/1888109754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a000ced00 msgr2=0x7f0a00154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a000ced00 0x7f0a00154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09feffd700 1 -- 172.21.15.102:0/1888109754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8009b30 con 0x7f0a000b3970 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.743+0000 7f09fe7fc700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000c41b0 0x7f0a00153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.744+0000 7f09feffd700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000b3970 0x7f0a0014e470 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c750 tx=0x7f09f000ca60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.744+0000 7f09e7fff700 1 -- 172.21.15.102:0/1888109754 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0004c80 con 0x7f0a000b3970 2023-03-03T09:50:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.744+0000 7f09e7fff700 1 -- 172.21.15.102:0/1888109754 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f0004e00 con 0x7f0a000b3970 2023-03-03T09:50:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.744+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0006bf20 con 0x7f0a000b3970 2023-03-03T09:50:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.744+0000 7f09e7fff700 1 -- 172.21.15.102:0/1888109754 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0007310 con 0x7f0a000b3970 2023-03-03T09:50:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.744+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0015ce80 con 0x7f0a000b3970 2023-03-03T09:50:14.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.745+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0006c200 con 0x7f0a000b3970 2023-03-03T09:50:14.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.747+0000 7f09e7fff700 1 -- 172.21.15.102:0/1888109754 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f09f001a020 con 0x7f0a000b3970 2023-03-03T09:50:14.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.748+0000 7f09e7fff700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09ec0682f0 0x7f09ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:14.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.748+0000 7f09e7fff700 1 -- 172.21.15.102:0/1888109754 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f09f0085920 con 0x7f0a000b3970 2023-03-03T09:50:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.748+0000 7f09fe7fc700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09ec0682f0 0x7f09ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.749+0000 7f09fe7fc700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09ec0682f0 0x7f09ec06a7a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0a0014cd00 tx=0x7f09f4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:14.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.752+0000 7f09e7fff700 1 -- 172.21.15.102:0/1888109754 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f09f0051fd0 con 0x7f0a000b3970 2023-03-03T09:50:14.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.986+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a00153b90 con 0x7f0a000b3970 2023-03-03T09:50:14.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.987+0000 7f09e7fff700 1 -- 172.21.15.102:0/1888109754 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f09f00555f0 con 0x7f0a000b3970 2023-03-03T09:50:14.989 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:50:14.989 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:50:14.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09ec0682f0 msgr2=0x7f09ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:14.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09ec0682f0 0x7f09ec06a7a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0a0014cd00 tx=0x7f09f4009210 comp rx=0 tx=0).stop 2023-03-03T09:50:14.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000b3970 msgr2=0x7f0a0014e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:14.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000b3970 0x7f0a0014e470 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c750 tx=0x7f09f000ca60 comp rx=0 tx=0).stop 2023-03-03T09:50:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 shutdown_connections 2023-03-03T09:50:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a000ced00 0x7f0a00154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a000b3970 0x7f0a0014e470 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f09ec0682f0 0x7f09ec06a7a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 --2- 172.21.15.102:0/1888109754 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0a000c41b0 0x7f0a00153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 >> 172.21.15.102:0/1888109754 conn(0x7f0a0008b1c0 msgr2=0x7f0a000b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:50:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 shutdown_connections 2023-03-03T09:50:14.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:50:14.990+0000 7f09fffff700 1 -- 172.21.15.102:0/1888109754 wait complete. 2023-03-03T09:50:14.993 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:50:15.018 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T09:50:29.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:29.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:59.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:59.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:29.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:29.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:34.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:51:34.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.267+0000 7f37eddff700 1 Processor -- start 2023-03-03T09:51:34.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.267+0000 7f37eddff700 1 -- start start 2023-03-03T09:51:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37eddff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37eddff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37e80cd8e0 0x7f37e80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37eddff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e8153660 con 0x7f37e81593c0 2023-03-03T09:51:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37eddff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e806c360 con 0x7f37e80cd8e0 2023-03-03T09:51:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37eddff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e815cea0 con 0x7f37e8153290 2023-03-03T09:51:34.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37e6ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f37e81593c0 0x7f37e80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56202/0 (socket says 172.21.15.102:56202) 2023-03-03T09:51:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37e6ffd700 1 -- 172.21.15.102:0/1554203189 learned_addr learned my addr 172.21.15.102:0/1554203189 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:51:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37e7fff700 1 --2- 172.21.15.102:0/1554203189 >> v2:172.21.15.172:3300/0 conn(0x7f37e80cd8e0 0x7f37e80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.269+0000 7f37e77fe700 1 --2- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e7fff700 1 -- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 msgr2=0x7f37e8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e7fff700 1 --2- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e7fff700 1 -- 172.21.15.102:0/1554203189 >> v1:172.21.15.102:6789/0 conn(0x7f37e81593c0 legacy=0x7f37e80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e7fff700 1 -- 172.21.15.102:0/1554203189 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e806c7f0 con 0x7f37e80cd8e0 2023-03-03T09:51:34.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e77fe700 1 --2- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:51:34.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e67fc700 1 -- 172.21.15.102:0/1554203189 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037106863 0 0) 0x7f37e8153660 con 0x7f37e81593c0 2023-03-03T09:51:34.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e7fff700 1 --2- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e80cd8e0 0x7f37e80ce530 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009840 tx=0x7f37e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fffbaa79c704f0a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:34.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e67fc700 1 -- 172.21.15.102:0/1554203189 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0011830 con 0x7f37e80cd8e0 2023-03-03T09:51:34.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e67fc700 1 -- 172.21.15.102:0/1554203189 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e0011990 con 0x7f37e80cd8e0 2023-03-03T09:51:34.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.270+0000 7f37e67fc700 1 -- 172.21.15.102:0/1554203189 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0011d20 con 0x7f37e80cd8e0 2023-03-03T09:51:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 -- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e80cd8e0 msgr2=0x7f37e80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 --2- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e80cd8e0 0x7f37e80ce530 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009840 tx=0x7f37e0009b50 comp rx=0 tx=0).stop 2023-03-03T09:51:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 -- 172.21.15.102:0/1554203189 shutdown_connections 2023-03-03T09:51:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 --2- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e80cd8e0 0x7f37e80ce530 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 --2- 172.21.15.102:0/1554203189 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 -- 172.21.15.102:0/1554203189 >> 172.21.15.102:0/1554203189 conn(0x7f37e8065fb0 msgr2=0x7f37e80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 -- 172.21.15.102:0/1554203189 shutdown_connections 2023-03-03T09:51:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.271+0000 7f37eddff700 1 -- 172.21.15.102:0/1554203189 wait complete. 2023-03-03T09:51:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.273+0000 7f37eddff700 1 Processor -- start 2023-03-03T09:51:34.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.273+0000 7f37eddff700 1 -- start start 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.274+0000 7f37eddff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37e80cd8e0 0x7f37e80c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.274+0000 7f37eddff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.274+0000 7f37eddff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e81593c0 0x7f37e814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.274+0000 7f37eddff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80c8f10 con 0x7f37e80cd8e0 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.274+0000 7f37eddff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e814c9f0 con 0x7f37e81593c0 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.274+0000 7f37eddff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80c7d60 con 0x7f37e8153290 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37e80cd8e0 0x7f37e80c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37e80cd8e0 0x7f37e80c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42034/0 (socket says 172.21.15.102:42034) 2023-03-03T09:51:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 -- 172.21.15.102:0/3264763350 learned_addr learned my addr 172.21.15.102:0/3264763350 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:51:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e6ffd700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e7fff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e81593c0 0x7f37e814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 -- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 msgr2=0x7f37e814ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e814ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 -- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e81593c0 msgr2=0x7f37e814d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e81593c0 0x7f37e814d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.275+0000 7f37e77fe700 1 -- 172.21.15.102:0/3264763350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d8009a70 con 0x7f37e80cd8e0 2023-03-03T09:51:34.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.276+0000 7f37e77fe700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37e80cd8e0 0x7f37e80c63b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f37d800efc0 tx=0x7f37d800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:34.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.276+0000 7f37cffff700 1 -- 172.21.15.102:0/3264763350 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d800cdd0 con 0x7f37e80cd8e0 2023-03-03T09:51:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.276+0000 7f37cffff700 1 -- 172.21.15.102:0/3264763350 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d800f040 con 0x7f37e80cd8e0 2023-03-03T09:51:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.276+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e00094a0 con 0x7f37e80cd8e0 2023-03-03T09:51:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.276+0000 7f37cffff700 1 -- 172.21.15.102:0/3264763350 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d80146a0 con 0x7f37e80cd8e0 2023-03-03T09:51:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.276+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e80ccde0 con 0x7f37e80cd8e0 2023-03-03T09:51:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.277+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e815d160 con 0x7f37e80cd8e0 2023-03-03T09:51:34.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.278+0000 7f37cffff700 1 -- 172.21.15.102:0/3264763350 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37d8014800 con 0x7f37e80cd8e0 2023-03-03T09:51:34.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.279+0000 7f37cffff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37d40682f0 0x7f37d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:34.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.280+0000 7f37cffff700 1 -- 172.21.15.102:0/3264763350 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f37d8086c60 con 0x7f37e80cd8e0 2023-03-03T09:51:34.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.280+0000 7f37e6ffd700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37d40682f0 0x7f37d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:34.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.281+0000 7f37e6ffd700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37d40682f0 0x7f37d406a7a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f37e815aad0 tx=0x7f37dc007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.284+0000 7f37cffff700 1 -- 172.21.15.102:0/3264763350 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37d80532e0 con 0x7f37e80cd8e0 2023-03-03T09:51:34.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.518+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37e80cd0c0 con 0x7f37e80cd8e0 2023-03-03T09:51:34.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.518+0000 7f37cffff700 1 -- 172.21.15.102:0/3264763350 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f37d8056900 con 0x7f37e80cd8e0 2023-03-03T09:51:34.521 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:51:34.521 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:51:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37d40682f0 msgr2=0x7f37d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37d40682f0 0x7f37d406a7a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f37e815aad0 tx=0x7f37dc007de0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37e80cd8e0 msgr2=0x7f37e80c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37e80cd8e0 0x7f37e80c63b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f37d800efc0 tx=0x7f37d800c3a0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 shutdown_connections 2023-03-03T09:51:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e81593c0 0x7f37e814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37e80cd8e0 0x7f37e80c63b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37d40682f0 0x7f37d406a7a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 --2- 172.21.15.102:0/3264763350 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37e8153290 0x7f37e814ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 >> 172.21.15.102:0/3264763350 conn(0x7f37e8065fb0 msgr2=0x7f37e81524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 shutdown_connections 2023-03-03T09:51:34.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:34.521+0000 7f37eddff700 1 -- 172.21.15.102:0/3264763350 wait complete. 2023-03-03T09:51:34.525 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:51:34.550 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 24 secs before thrashing 2023-03-03T09:51:58.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:51:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.800+0000 7fa5691ff700 1 Processor -- start 2023-03-03T09:51:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.800+0000 7fa5691ff700 1 -- start start 2023-03-03T09:51:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa5691ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa5640cd8e0 0x7fa5640cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa5691ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa5640cd390 0x7fa5640ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa5691ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa564153660 con 0x7fa564153290 2023-03-03T09:51:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa5691ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56406c360 con 0x7fa5640cd390 2023-03-03T09:51:58.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa5691ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56415cea0 con 0x7fa5640cd8e0 2023-03-03T09:51:58.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa562d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa564153290 0x7fa564158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44628/0 (socket says 172.21.15.102:44628) 2023-03-03T09:51:58.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa562d9d700 1 -- 172.21.15.102:0/1001428296 learned_addr learned my addr 172.21.15.102:0/1001428296 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:51:58.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.801+0000 7fa56259c700 1 --2- 172.21.15.102:0/1001428296 >> v2:172.21.15.172:3300/0 conn(0x7fa5640cd8e0 0x7fa5640cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:58.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.802+0000 7fa56359e700 1 --2- 172.21.15.102:0/1001428296 >> v2:172.21.15.102:3300/0 conn(0x7fa5640cd390 0x7fa5640ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:58.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.802+0000 7fa561d9b700 1 -- 172.21.15.102:0/1001428296 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942147156 0 0) 0x7fa564153660 con 0x7fa564153290 2023-03-03T09:51:58.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.802+0000 7fa56259c700 1 -- 172.21.15.102:0/1001428296 >> v2:172.21.15.102:3300/0 conn(0x7fa5640cd390 msgr2=0x7fa5640ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.802+0000 7fa56259c700 1 --2- 172.21.15.102:0/1001428296 >> v2:172.21.15.102:3300/0 conn(0x7fa5640cd390 0x7fa5640ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.802+0000 7fa56259c700 1 -- 172.21.15.102:0/1001428296 >> v1:172.21.15.102:6789/0 conn(0x7fa564153290 legacy=0x7fa564158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.802+0000 7fa56259c700 1 -- 172.21.15.102:0/1001428296 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56406c7f0 con 0x7fa5640cd8e0 2023-03-03T09:51:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.802+0000 7fa56259c700 1 --2- 172.21.15.102:0/1001428296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5640cde20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa564159fb0 tx=0x7fa558009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b73b8e0a70ef1da server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa561d9b700 1 -- 172.21.15.102:0/1001428296 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558011830 con 0x7fa5640cd8e0 2023-03-03T09:51:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa561d9b700 1 -- 172.21.15.102:0/1001428296 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa558011990 con 0x7fa5640cd8e0 2023-03-03T09:51:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa561d9b700 1 -- 172.21.15.102:0/1001428296 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558011d20 con 0x7fa5640cd8e0 2023-03-03T09:51:58.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 -- 172.21.15.102:0/1001428296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 msgr2=0x7fa5640cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:58.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1001428296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5640cde20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa564159fb0 tx=0x7fa558009b20 comp rx=0 tx=0).stop 2023-03-03T09:51:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 -- 172.21.15.102:0/1001428296 shutdown_connections 2023-03-03T09:51:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1001428296 >> v2:172.21.15.102:3300/0 conn(0x7fa5640cd390 0x7fa5640ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1001428296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5640cde20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 -- 172.21.15.102:0/1001428296 >> 172.21.15.102:0/1001428296 conn(0x7fa564065fb0 msgr2=0x7fa5640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 -- 172.21.15.102:0/1001428296 shutdown_connections 2023-03-03T09:51:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.803+0000 7fa5691ff700 1 -- 172.21.15.102:0/1001428296 wait complete. 2023-03-03T09:51:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.805+0000 7fa5691ff700 1 Processor -- start 2023-03-03T09:51:58.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.805+0000 7fa5691ff700 1 -- start start 2023-03-03T09:51:58.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.806+0000 7fa5691ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5640cd390 0x7fa56414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:58.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.806+0000 7fa5691ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5641ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:58.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.806+0000 7fa5691ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa564153290 0x7fa5640c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:58.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.806+0000 7fa5691ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56414ea90 con 0x7fa564153290 2023-03-03T09:51:58.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.806+0000 7fa5691ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa564151890 con 0x7fa5640cd8e0 2023-03-03T09:51:58.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.806+0000 7fa5691ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5641efe20 con 0x7fa5640cd390 2023-03-03T09:51:58.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa562d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5640cd390 0x7fa56414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:58.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5641ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:58.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5641ef8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39306/0 (socket says 172.21.15.102:39306) 2023-03-03T09:51:58.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 -- 172.21.15.102:0/1595778920 learned_addr learned my addr 172.21.15.102:0/1595778920 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56359e700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa564153290 0x7fa5640c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 -- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5640cd390 msgr2=0x7fa56414e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5640cd390 0x7fa56414e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 -- 172.21.15.102:0/1595778920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa564153290 msgr2=0x7fa5640c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa564153290 0x7fa5640c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56259c700 1 -- 172.21.15.102:0/1595778920 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580094a0 con 0x7fa5640cd8e0 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.807+0000 7fa56359e700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa564153290 0x7fa5640c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:51:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.808+0000 7fa56259c700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5641ef8e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa5580090b0 tx=0x7fa558004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:58.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.808+0000 7fa54b7fe700 1 -- 172.21.15.102:0/1595778920 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5580036a0 con 0x7fa5640cd8e0 2023-03-03T09:51:58.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.808+0000 7fa54b7fe700 1 -- 172.21.15.102:0/1595778920 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55801f070 con 0x7fa5640cd8e0 2023-03-03T09:51:58.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.808+0000 7fa54b7fe700 1 -- 172.21.15.102:0/1595778920 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558003a00 con 0x7fa5640cd8e0 2023-03-03T09:51:58.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.808+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5640ccde0 con 0x7fa5640cd8e0 2023-03-03T09:51:58.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.808+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56415d160 con 0x7fa5640cd8e0 2023-03-03T09:51:58.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.809+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5640cd0c0 con 0x7fa5640cd8e0 2023-03-03T09:51:58.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.811+0000 7fa54b7fe700 1 -- 172.21.15.102:0/1595778920 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa55800f050 con 0x7fa5640cd8e0 2023-03-03T09:51:58.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.812+0000 7fa54b7fe700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5500682f0 0x7fa55006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:58.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.812+0000 7fa54b7fe700 1 -- 172.21.15.102:0/1595778920 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa558021070 con 0x7fa5640cd8e0 2023-03-03T09:51:58.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.812+0000 7fa562d9d700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5500682f0 0x7fa55006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:58.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.813+0000 7fa562d9d700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5500682f0 0x7fa55006a7a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa554000f60 tx=0x7fa554007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:58.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:58.815+0000 7fa54b7fe700 1 -- 172.21.15.102:0/1595778920 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa558053e20 con 0x7fa5640cd8e0 2023-03-03T09:51:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.047+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa56404e6e0 con 0x7fa5640cd8e0 2023-03-03T09:51:59.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.048+0000 7fa54b7fe700 1 -- 172.21.15.102:0/1595778920 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa558057440 con 0x7fa5640cd8e0 2023-03-03T09:51:59.050 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:51:59.050 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5500682f0 msgr2=0x7fa55006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5500682f0 0x7fa55006a7a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa554000f60 tx=0x7fa554007e70 comp rx=0 tx=0).stop 2023-03-03T09:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 msgr2=0x7fa5641ef8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5641ef8e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa5580090b0 tx=0x7fa558004750 comp rx=0 tx=0).stop 2023-03-03T09:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 shutdown_connections 2023-03-03T09:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5500682f0 0x7fa55006a7a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5640cd390 0x7fa56414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa564153290 0x7fa5640c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 --2- 172.21.15.102:0/1595778920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5640cd8e0 0x7fa5641ef8e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:59.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 >> 172.21.15.102:0/1595778920 conn(0x7fa564065fb0 msgr2=0x7fa5640cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:59.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 shutdown_connections 2023-03-03T09:51:59.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:51:59.051+0000 7fa5691ff700 1 -- 172.21.15.102:0/1595778920 wait complete. 2023-03-03T09:51:59.054 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:51:59.080 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2023-03-03T09:51:59.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:59.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:29.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:29.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:00.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:00.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:30.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:30.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:44.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:53:44.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.335+0000 7f1f57dff700 1 Processor -- start 2023-03-03T09:53:44.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.336+0000 7f1f57dff700 1 -- start start 2023-03-03T09:53:44.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f57dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f50153290 0x7f1f50158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:44.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f57dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f501593c0 0x7f1f500cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:44.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f57dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f500cd8e0 0x7f1f500ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:44.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f57dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f50153660 con 0x7f1f501593c0 2023-03-03T09:53:44.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f57dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5006c360 con 0x7f1f50153290 2023-03-03T09:53:44.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f57dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5015cea0 con 0x7f1f500cd8e0 2023-03-03T09:53:44.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f5539a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f501593c0 0x7f1f500cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:44.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f55b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f50153290 0x7f1f50158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:44.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f5539a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f501593c0 0x7f1f500cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36106/0 (socket says 172.21.15.102:36106) 2023-03-03T09:53:44.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f5539a700 1 -- 172.21.15.102:0/1366536083 learned_addr learned my addr 172.21.15.102:0/1366536083 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:53:44.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.337+0000 7f1f5639c700 1 --2- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f500cd8e0 0x7f1f500ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:44.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f55b9b700 1 -- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f500cd8e0 msgr2=0x7f1f500ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:53:44.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f55b9b700 1 --2- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f500cd8e0 0x7f1f500ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f55b9b700 1 -- 172.21.15.102:0/1366536083 >> v2:172.21.15.102:3300/0 conn(0x7f1f501593c0 msgr2=0x7f1f500cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:53:44.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f55b9b700 1 --2- 172.21.15.102:0/1366536083 >> v2:172.21.15.102:3300/0 conn(0x7f1f501593c0 0x7f1f500cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f55b9b700 1 -- 172.21.15.102:0/1366536083 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5006c7f0 con 0x7f1f50153290 2023-03-03T09:53:44.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f5539a700 1 --2- 172.21.15.102:0/1366536083 >> v2:172.21.15.102:3300/0 conn(0x7f1f501593c0 0x7f1f500cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:53:44.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f5639c700 1 --2- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f500cd8e0 0x7f1f500ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:53:44.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f55b9b700 1 --2- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f50153290 0x7f1f50158e80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1f5008cc70 tx=0x7f1f4c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6c00aa9f8facb60 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:44.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.338+0000 7f1f54b99700 1 -- 172.21.15.102:0/1366536083 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c011830 con 0x7f1f50153290 2023-03-03T09:53:44.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f54b99700 1 -- 172.21.15.102:0/1366536083 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c011990 con 0x7f1f50153290 2023-03-03T09:53:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f54b99700 1 -- 172.21.15.102:0/1366536083 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c011d20 con 0x7f1f50153290 2023-03-03T09:53:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 -- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f50153290 msgr2=0x7f1f50158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 --2- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f50153290 0x7f1f50158e80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1f5008cc70 tx=0x7f1f4c009b20 comp rx=0 tx=0).stop 2023-03-03T09:53:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 -- 172.21.15.102:0/1366536083 shutdown_connections 2023-03-03T09:53:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 --2- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f500cd8e0 0x7f1f500ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 --2- 172.21.15.102:0/1366536083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f50153290 0x7f1f50158e80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 --2- 172.21.15.102:0/1366536083 >> v2:172.21.15.102:3300/0 conn(0x7f1f501593c0 0x7f1f500cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 -- 172.21.15.102:0/1366536083 >> 172.21.15.102:0/1366536083 conn(0x7f1f50065fb0 msgr2=0x7f1f500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:53:44.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 -- 172.21.15.102:0/1366536083 shutdown_connections 2023-03-03T09:53:44.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.339+0000 7f1f57dff700 1 -- 172.21.15.102:0/1366536083 wait complete. 2023-03-03T09:53:44.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.341+0000 7f1f57dff700 1 Processor -- start 2023-03-03T09:53:44.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.342+0000 7f1f57dff700 1 -- start start 2023-03-03T09:53:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f57dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500cd8e0 0x7f1f501ef3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f57dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f50153290 0x7f1f500c2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f57dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f501593c0 0x7f1f50158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f57dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f500c3840 con 0x7f1f501593c0 2023-03-03T09:53:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f57dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f501eef40 con 0x7f1f500cd8e0 2023-03-03T09:53:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f57dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5014c990 con 0x7f1f50153290 2023-03-03T09:53:44.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f5639c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f501593c0 0x7f1f50158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:44.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f55b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500cd8e0 0x7f1f501ef3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:44.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f55b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500cd8e0 0x7f1f501ef3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47452/0 (socket says 172.21.15.102:47452) 2023-03-03T09:53:44.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f5639c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f501593c0 0x7f1f50158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36116/0 (socket says 172.21.15.102:36116) 2023-03-03T09:53:44.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f55b9b700 1 -- 172.21.15.102:0/3287257053 learned_addr learned my addr 172.21.15.102:0/3287257053 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:53:44.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f5539a700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f50153290 0x7f1f500c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:44.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f55b9b700 1 -- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f50153290 msgr2=0x7f1f500c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:44.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.343+0000 7f1f55b9b700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f50153290 0x7f1f500c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f55b9b700 1 -- 172.21.15.102:0/3287257053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f501593c0 msgr2=0x7f1f50158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:44.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f55b9b700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f501593c0 0x7f1f50158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f55b9b700 1 -- 172.21.15.102:0/3287257053 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f48009a70 con 0x7f1f500cd8e0 2023-03-03T09:53:44.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f55b9b700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500cd8e0 0x7f1f501ef3d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c000c00 tx=0x7f1f4c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f467fc700 1 -- 172.21.15.102:0/3287257053 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c004450 con 0x7f1f500cd8e0 2023-03-03T09:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f467fc700 1 -- 172.21.15.102:0/3287257053 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c01e070 con 0x7f1f500cd8e0 2023-03-03T09:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f467fc700 1 -- 172.21.15.102:0/3287257053 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0038e0 con 0x7f1f500cd8e0 2023-03-03T09:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f40009550 con 0x7f1f500cd8e0 2023-03-03T09:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.344+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c0094a0 con 0x7f1f500cd8e0 2023-03-03T09:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.345+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f500ccde0 con 0x7f1f500cd8e0 2023-03-03T09:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.347+0000 7f1f467fc700 1 -- 172.21.15.102:0/3287257053 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f4c00f050 con 0x7f1f500cd8e0 2023-03-03T09:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.348+0000 7f1f467fc700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f340682f0 0x7f1f3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.348+0000 7f1f467fc700 1 -- 172.21.15.102:0/3287257053 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1f4c086a30 con 0x7f1f500cd8e0 2023-03-03T09:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.348+0000 7f1f5539a700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f340682f0 0x7f1f3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:44.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.349+0000 7f1f5539a700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f340682f0 0x7f1f3406a7a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1f5014e260 tx=0x7f1f400092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:44.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.352+0000 7f1f467fc700 1 -- 172.21.15.102:0/3287257053 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f4c053140 con 0x7f1f500cd8e0 2023-03-03T09:53:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.588+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f500cd0c0 con 0x7f1f500cd8e0 2023-03-03T09:53:44.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.589+0000 7f1f467fc700 1 -- 172.21.15.102:0/3287257053 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1f4c020090 con 0x7f1f500cd8e0 2023-03-03T09:53:44.591 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:53:44.592 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:53:44.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f340682f0 msgr2=0x7f1f3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:44.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f340682f0 0x7f1f3406a7a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1f5014e260 tx=0x7f1f400092c0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500cd8e0 msgr2=0x7f1f501ef3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:44.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500cd8e0 0x7f1f501ef3d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c000c00 tx=0x7f1f4c004750 comp rx=0 tx=0).stop 2023-03-03T09:53:44.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 shutdown_connections 2023-03-03T09:53:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f501593c0 0x7f1f50158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500cd8e0 0x7f1f501ef3d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f340682f0 0x7f1f3406a7a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 --2- 172.21.15.102:0/3287257053 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f50153290 0x7f1f500c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 >> 172.21.15.102:0/3287257053 conn(0x7f1f50065fb0 msgr2=0x7f1f5015be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:53:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 shutdown_connections 2023-03-03T09:53:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:53:44.592+0000 7f1f57dff700 1 -- 172.21.15.102:0/3287257053 wait complete. 2023-03-03T09:53:44.596 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:53:44.621 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T09:54:00.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:00.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:30.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:30.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:42.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:54:42.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.870+0000 7ff7747ff700 1 Processor -- start 2023-03-03T09:54:42.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.871+0000 7ff7747ff700 1 -- start start 2023-03-03T09:54:42.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff7747ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff76c0cd8e0 0x7ff76c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:42.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff7747ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff76c0cd390 0x7ff76c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:42.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff7747ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c153660 con 0x7ff76c153290 2023-03-03T09:54:42.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff7747ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76c06c360 con 0x7ff76c0cd390 2023-03-03T09:54:42.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff7747ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76c15cea0 con 0x7ff76c0cd8e0 2023-03-03T09:54:42.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff772d9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff76c0cd390 0x7ff76c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:42.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff771d9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff76c0cd8e0 0x7ff76c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:42.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff77259b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff76c153290 0x7ff76c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44854/0 (socket says 172.21.15.102:44854) 2023-03-03T09:54:42.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.872+0000 7ff77259b700 1 -- 172.21.15.102:0/2655386625 learned_addr learned my addr 172.21.15.102:0/2655386625 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:54:42.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.873+0000 7ff771d9a700 1 -- 172.21.15.102:0/2655386625 >> v2:172.21.15.102:3300/0 conn(0x7ff76c0cd390 msgr2=0x7ff76c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:54:42.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.873+0000 7ff771d9a700 1 --2- 172.21.15.102:0/2655386625 >> v2:172.21.15.102:3300/0 conn(0x7ff76c0cd390 0x7ff76c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:42.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.873+0000 7ff771d9a700 1 -- 172.21.15.102:0/2655386625 >> v1:172.21.15.172:6789/0 conn(0x7ff76c153290 legacy=0x7ff76c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:42.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.873+0000 7ff771d9a700 1 -- 172.21.15.102:0/2655386625 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c06c7f0 con 0x7ff76c0cd8e0 2023-03-03T09:54:42.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.873+0000 7ff771599700 1 -- 172.21.15.102:0/2655386625 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359878617 0 0) 0x7ff76c153660 con 0x7ff76c153290 2023-03-03T09:54:42.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.873+0000 7ff771d9a700 1 --2- 172.21.15.102:0/2655386625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c0cde20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff76c159fb0 tx=0x7ff75c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f9b67bfc904f83a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:54:42.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.873+0000 7ff771599700 1 -- 172.21.15.102:0/2655386625 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c011830 con 0x7ff76c0cd8e0 2023-03-03T09:54:42.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff771599700 1 -- 172.21.15.102:0/2655386625 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c011990 con 0x7ff76c0cd8e0 2023-03-03T09:54:42.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff771599700 1 -- 172.21.15.102:0/2655386625 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c011d20 con 0x7ff76c0cd8e0 2023-03-03T09:54:42.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 -- 172.21.15.102:0/2655386625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c0cd8e0 msgr2=0x7ff76c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:42.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 --2- 172.21.15.102:0/2655386625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c0cde20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff76c159fb0 tx=0x7ff75c009b20 comp rx=0 tx=0).stop 2023-03-03T09:54:42.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 -- 172.21.15.102:0/2655386625 shutdown_connections 2023-03-03T09:54:42.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 --2- 172.21.15.102:0/2655386625 >> v2:172.21.15.102:3300/0 conn(0x7ff76c0cd390 0x7ff76c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:42.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 --2- 172.21.15.102:0/2655386625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c0cde20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:42.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 -- 172.21.15.102:0/2655386625 >> 172.21.15.102:0/2655386625 conn(0x7ff76c065fb0 msgr2=0x7ff76c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:54:42.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 -- 172.21.15.102:0/2655386625 shutdown_connections 2023-03-03T09:54:42.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.874+0000 7ff7747ff700 1 -- 172.21.15.102:0/2655386625 wait complete. 2023-03-03T09:54:42.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.877+0000 7ff7747ff700 1 Processor -- start 2023-03-03T09:54:42.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.877+0000 7ff7747ff700 1 -- start start 2023-03-03T09:54:42.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff7747ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd390 0x7ff76c14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:42.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff7747ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c1ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:42.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff7747ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:42.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff7747ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c14ea90 con 0x7ff76c0cd8e0 2023-03-03T09:54:42.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff7747ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c151890 con 0x7ff76c153290 2023-03-03T09:54:42.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff7747ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c1efe20 con 0x7ff76c0cd390 2023-03-03T09:54:42.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff771d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c1ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:42.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff77259b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd390 0x7ff76c14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:42.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff772d9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:42.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff77259b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd390 0x7ff76c14e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57288/0 (socket says 172.21.15.102:57288) 2023-03-03T09:54:42.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff77259b700 1 -- 172.21.15.102:0/1722013327 learned_addr learned my addr 172.21.15.102:0/1722013327 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:54:42.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.879+0000 7ff771d9a700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c1ef8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44772/0 (socket says 172.21.15.102:44772) 2023-03-03T09:54:42.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff77259b700 1 -- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 msgr2=0x7ff76c0c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff77259b700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff77259b700 1 -- 172.21.15.102:0/1722013327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c0cd8e0 msgr2=0x7ff76c1ef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff77259b700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c1ef8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff77259b700 1 -- 172.21.15.102:0/1722013327 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c0094a0 con 0x7ff76c0cd390 2023-03-03T09:54:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff772d9c700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:54:42.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff771d9a700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c1ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:54:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff77259b700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd390 0x7ff76c14e550 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c550 tx=0x7ff76800c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:54:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff762ffd700 1 -- 172.21.15.102:0/1722013327 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768004c80 con 0x7ff76c0cd390 2023-03-03T09:54:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff762ffd700 1 -- 172.21.15.102:0/1722013327 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff768004e00 con 0x7ff76c0cd390 2023-03-03T09:54:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff762ffd700 1 -- 172.21.15.102:0/1722013327 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768007310 con 0x7ff76c0cd390 2023-03-03T09:54:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c0ccde0 con 0x7ff76c0cd390 2023-03-03T09:54:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.880+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c15d160 con 0x7ff76c0cd390 2023-03-03T09:54:42.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.881+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c0cd0c0 con 0x7ff76c0cd390 2023-03-03T09:54:42.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.883+0000 7ff762ffd700 1 -- 172.21.15.102:0/1722013327 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff76801a020 con 0x7ff76c0cd390 2023-03-03T09:54:42.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.885+0000 7ff762ffd700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7500682f0 0x7ff75006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:42.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.885+0000 7ff762ffd700 1 -- 172.21.15.102:0/1722013327 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff768085870 con 0x7ff76c0cd390 2023-03-03T09:54:42.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.885+0000 7ff771d9a700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7500682f0 0x7ff75006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:42.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.886+0000 7ff771d9a700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7500682f0 0x7ff75006a7a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0090b0 tx=0x7ff75c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:54:42.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:42.889+0000 7ff762ffd700 1 -- 172.21.15.102:0/1722013327 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff768051f80 con 0x7ff76c0cd390 2023-03-03T09:54:43.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.123+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff76c04e6e0 con 0x7ff76c0cd390 2023-03-03T09:54:43.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.124+0000 7ff762ffd700 1 -- 172.21.15.102:0/1722013327 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff7680555a0 con 0x7ff76c0cd390 2023-03-03T09:54:43.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:54:43.126 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:54:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7500682f0 msgr2=0x7ff75006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7500682f0 0x7ff75006a7a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0090b0 tx=0x7ff75c008030 comp rx=0 tx=0).stop 2023-03-03T09:54:43.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd390 msgr2=0x7ff76c14e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd390 0x7ff76c14e550 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c550 tx=0x7ff76800c860 comp rx=0 tx=0).stop 2023-03-03T09:54:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 shutdown_connections 2023-03-03T09:54:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7500682f0 0x7ff75006a7a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd390 0x7ff76c14e550 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 --2- 172.21.15.102:0/1722013327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c0cd8e0 0x7ff76c1ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:43.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 >> 172.21.15.102:0/1722013327 conn(0x7ff76c065fb0 msgr2=0x7ff76c0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:54:43.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 shutdown_connections 2023-03-03T09:54:43.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:54:43.127+0000 7ff7747ff700 1 -- 172.21.15.102:0/1722013327 wait complete. 2023-03-03T09:54:43.131 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:54:43.156 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T09:55:00.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:00.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:26.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:55:26.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.418+0000 7f08cd19d700 1 Processor -- start 2023-03-03T09:55:26.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.418+0000 7f08cd19d700 1 -- start start 2023-03-03T09:55:26.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.420+0000 7f08cd19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f08c80ced00 0x7f08c80c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:26.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.420+0000 7f08cd19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f08c80b3970 0x7f08c80c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:26.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.420+0000 7f08cd19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c80c3d20 con 0x7f08c80cf0d0 2023-03-03T09:55:26.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.420+0000 7f08cd19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c80c4930 con 0x7f08c80ced00 2023-03-03T09:55:26.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.420+0000 7f08cd19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c80cf570 con 0x7f08c80b3970 2023-03-03T09:55:26.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f08c80ced00 0x7f08c80c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:26.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f08c80ced00 0x7f08c80c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38506/0 (socket says 172.21.15.102:38506) 2023-03-03T09:55:26.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c7fff700 1 -- 172.21.15.102:0/4106139796 learned_addr learned my addr 172.21.15.102:0/4106139796 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:55:26.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08cc99c700 1 --1- 172.21.15.102:0/4106139796 >> v1:172.21.15.172:6789/0 conn(0x7f08c80cf0d0 0x7f08c8158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35274/0 (socket says 172.21.15.102:35274) 2023-03-03T09:55:26.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08bffff700 1 --2- 172.21.15.102:0/4106139796 >> v2:172.21.15.172:3300/0 conn(0x7f08c80b3970 0x7f08c80c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:26.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c77fe700 1 -- 172.21.15.102:0/4106139796 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417289363 0 0) 0x7f08c80c3d20 con 0x7f08c80cf0d0 2023-03-03T09:55:26.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c7fff700 1 -- 172.21.15.102:0/4106139796 >> v2:172.21.15.172:3300/0 conn(0x7f08c80b3970 msgr2=0x7f08c80c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:55:26.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c7fff700 1 --2- 172.21.15.102:0/4106139796 >> v2:172.21.15.172:3300/0 conn(0x7f08c80b3970 0x7f08c80c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c7fff700 1 -- 172.21.15.102:0/4106139796 >> v1:172.21.15.172:6789/0 conn(0x7f08c80cf0d0 legacy=0x7f08c8158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:26.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.421+0000 7f08c7fff700 1 -- 172.21.15.102:0/4106139796 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c815cba0 con 0x7f08c80ced00 2023-03-03T09:55:26.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08c7fff700 1 --2- 172.21.15.102:0/4106139796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80ced00 0x7f08c80c5a90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f08c80c4ba0 tx=0x7f08b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0daa436213e6feb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:26.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08c77fe700 1 -- 172.21.15.102:0/4106139796 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8011830 con 0x7f08c80ced00 2023-03-03T09:55:26.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08c77fe700 1 -- 172.21.15.102:0/4106139796 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b8011990 con 0x7f08c80ced00 2023-03-03T09:55:26.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08c77fe700 1 -- 172.21.15.102:0/4106139796 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8011d20 con 0x7f08c80ced00 2023-03-03T09:55:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08cd19d700 1 -- 172.21.15.102:0/4106139796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80ced00 msgr2=0x7f08c80c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08cd19d700 1 --2- 172.21.15.102:0/4106139796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80ced00 0x7f08c80c5a90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f08c80c4ba0 tx=0x7f08b8009bd0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08cd19d700 1 -- 172.21.15.102:0/4106139796 shutdown_connections 2023-03-03T09:55:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08cd19d700 1 --2- 172.21.15.102:0/4106139796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80ced00 0x7f08c80c5a90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08cd19d700 1 --2- 172.21.15.102:0/4106139796 >> v2:172.21.15.172:3300/0 conn(0x7f08c80b3970 0x7f08c80c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.422+0000 7f08cd19d700 1 -- 172.21.15.102:0/4106139796 >> 172.21.15.102:0/4106139796 conn(0x7f08c808b1c0 msgr2=0x7f08c8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:55:26.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.423+0000 7f08cd19d700 1 -- 172.21.15.102:0/4106139796 shutdown_connections 2023-03-03T09:55:26.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.423+0000 7f08cd19d700 1 -- 172.21.15.102:0/4106139796 wait complete. 2023-03-03T09:55:26.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.425+0000 7f08cd19d700 1 Processor -- start 2023-03-03T09:55:26.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.425+0000 7f08cd19d700 1 -- start start 2023-03-03T09:55:26.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08cd19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80b3970 0x7f08c81f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08cd19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c80ced00 0x7f08c815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08cd19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c80cf0d0 0x7f08c8150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08cd19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c81518e0 con 0x7f08c80b3970 2023-03-03T09:55:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08cd19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c8151490 con 0x7f08c80cf0d0 2023-03-03T09:55:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08cd19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c815c3d0 con 0x7f08c80ced00 2023-03-03T09:55:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08c7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80b3970 0x7f08c81f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08c7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80b3970 0x7f08c81f18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38518/0 (socket says 172.21.15.102:38518) 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08c7fff700 1 -- 172.21.15.102:0/662192859 learned_addr learned my addr 172.21.15.102:0/662192859 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08bffff700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c80ced00 0x7f08c815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.426+0000 7f08cc99c700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c80cf0d0 0x7f08c8150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c7fff700 1 -- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c80ced00 msgr2=0x7f08c815a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c7fff700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c80ced00 0x7f08c815a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c7fff700 1 -- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c80cf0d0 msgr2=0x7f08c8150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c7fff700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c80cf0d0 0x7f08c8150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c7fff700 1 -- 172.21.15.102:0/662192859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b8009550 con 0x7f08c80b3970 2023-03-03T09:55:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08cc99c700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c80cf0d0 0x7f08c8150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:55:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c7fff700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80b3970 0x7f08c81f18e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009c10 tx=0x7f08b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c57fa700 1 -- 172.21.15.102:0/662192859 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8004590 con 0x7f08c80b3970 2023-03-03T09:55:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c57fa700 1 -- 172.21.15.102:0/662192859 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b801e070 con 0x7f08c80b3970 2023-03-03T09:55:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08c57fa700 1 -- 172.21.15.102:0/662192859 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b80038a0 con 0x7f08c80b3970 2023-03-03T09:55:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.427+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c806bf20 con 0x7f08c80b3970 2023-03-03T09:55:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.428+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c815ce80 con 0x7f08c80b3970 2023-03-03T09:55:26.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.429+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c806c200 con 0x7f08c80b3970 2023-03-03T09:55:26.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.430+0000 7f08c57fa700 1 -- 172.21.15.102:0/662192859 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f08b800f050 con 0x7f08c80b3970 2023-03-03T09:55:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.431+0000 7f08c57fa700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08a80682f0 0x7f08a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.431+0000 7f08c57fa700 1 -- 172.21.15.102:0/662192859 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f08b8086da0 con 0x7f08c80b3970 2023-03-03T09:55:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.431+0000 7f08bffff700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08a80682f0 0x7f08a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.432+0000 7f08bffff700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08a80682f0 0x7f08a806a7a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f08c81516c0 tx=0x7f08b0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:26.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.435+0000 7f08c57fa700 1 -- 172.21.15.102:0/662192859 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08b8053560 con 0x7f08c80b3970 2023-03-03T09:55:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.670+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08c804e6e0 con 0x7f08c80b3970 2023-03-03T09:55:26.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.671+0000 7f08c57fa700 1 -- 172.21.15.102:0/662192859 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f08b8020070 con 0x7f08c80b3970 2023-03-03T09:55:26.673 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:55:26.674 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:55:26.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08a80682f0 msgr2=0x7f08a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:26.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08a80682f0 0x7f08a806a7a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f08c81516c0 tx=0x7f08b0009210 comp rx=0 tx=0).stop 2023-03-03T09:55:26.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80b3970 msgr2=0x7f08c81f18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:26.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80b3970 0x7f08c81f18e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009c10 tx=0x7f08b8004750 comp rx=0 tx=0).stop 2023-03-03T09:55:26.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 shutdown_connections 2023-03-03T09:55:26.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08c80cf0d0 0x7f08c8150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08c80b3970 0x7f08c81f18e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08a80682f0 0x7f08a806a7a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 --2- 172.21.15.102:0/662192859 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f08c80ced00 0x7f08c815a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:26.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 >> 172.21.15.102:0/662192859 conn(0x7f08c808b1c0 msgr2=0x7f08c80ca0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:55:26.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 shutdown_connections 2023-03-03T09:55:26.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:55:26.674+0000 7f08cd19d700 1 -- 172.21.15.102:0/662192859 wait complete. 2023-03-03T09:55:26.678 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:55:26.702 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 51 secs before thrashing 2023-03-03T09:55:31.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:31.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:01.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:01.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:17.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:56:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.951+0000 7f5e0b59e700 1 Processor -- start 2023-03-03T09:56:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.951+0000 7f5e0b59e700 1 -- start start 2023-03-03T09:56:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.953+0000 7f5e0b59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5e0c0ced00 0x7f5e0c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.953+0000 7f5e0b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.953+0000 7f5e0b59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0c3d20 con 0x7f5e0c0b3970 2023-03-03T09:56:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.953+0000 7f5e0b59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c0c4930 con 0x7f5e0c0ced00 2023-03-03T09:56:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.953+0000 7f5e0b59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0cf610 con 0x7f5e0c0c41b0 2023-03-03T09:56:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.953+0000 7f5e0ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e0a59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5e0c0ced00 0x7f5e0c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e03fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5e0c0b3970 0x7f5e0c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51614/0 (socket says 172.21.15.102:51614) 2023-03-03T09:56:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e0ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c0cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44660/0 (socket says 172.21.15.102:44660) 2023-03-03T09:56:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e03fff700 1 -- 172.21.15.102:0/1008585477 learned_addr learned my addr 172.21.15.102:0/1008585477 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:56:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e0ad9d700 1 -- 172.21.15.102:0/1008585477 >> v2:172.21.15.172:3300/0 conn(0x7f5e0c0ced00 msgr2=0x7f5e0c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:56:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e0ad9d700 1 --2- 172.21.15.102:0/1008585477 >> v2:172.21.15.172:3300/0 conn(0x7f5e0c0ced00 0x7f5e0c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e0ad9d700 1 -- 172.21.15.102:0/1008585477 >> v1:172.21.15.172:6789/0 conn(0x7f5e0c0b3970 legacy=0x7f5e0c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e0ad9d700 1 -- 172.21.15.102:0/1008585477 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c15cba0 con 0x7f5e0c0c41b0 2023-03-03T09:56:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e09d9b700 1 -- 172.21.15.102:0/1008585477 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724734734 0 0) 0x7f5e0c0c3d20 con 0x7f5e0c0b3970 2023-03-03T09:56:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.954+0000 7f5e0ad9d700 1 --2- 172.21.15.102:0/1008585477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c0cf0d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=395bd74893b71297 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e09d9b700 1 -- 172.21.15.102:0/1008585477 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0400e070 con 0x7f5e0c0c41b0 2023-03-03T09:56:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e09d9b700 1 -- 172.21.15.102:0/1008585477 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0400bb20 con 0x7f5e0c0c41b0 2023-03-03T09:56:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e09d9b700 1 -- 172.21.15.102:0/1008585477 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04012430 con 0x7f5e0c0c41b0 2023-03-03T09:56:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 -- 172.21.15.102:0/1008585477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 msgr2=0x7f5e0c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/1008585477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c0cf0d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0400b330 comp rx=0 tx=0).stop 2023-03-03T09:56:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 -- 172.21.15.102:0/1008585477 shutdown_connections 2023-03-03T09:56:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/1008585477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c0cf0d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/1008585477 >> v2:172.21.15.172:3300/0 conn(0x7f5e0c0ced00 0x7f5e0c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 -- 172.21.15.102:0/1008585477 >> 172.21.15.102:0/1008585477 conn(0x7f5e0c08b1c0 msgr2=0x7f5e0c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 -- 172.21.15.102:0/1008585477 shutdown_connections 2023-03-03T09:56:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.955+0000 7f5e0b59e700 1 -- 172.21.15.102:0/1008585477 wait complete. 2023-03-03T09:56:17.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.957+0000 7f5e0b59e700 1 Processor -- start 2023-03-03T09:56:17.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.958+0000 7f5e0b59e700 1 -- start start 2023-03-03T09:56:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.958+0000 7f5e0b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c0b3970 0x7f5e0c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.958+0000 7f5e0b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.958+0000 7f5e0b59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e0c0ced00 0x7f5e0c151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0b59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c15b1d0 con 0x7f5e0c0ced00 2023-03-03T09:56:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0b59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1f1930 con 0x7f5e0c0b3970 2023-03-03T09:56:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0b59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1f1bd0 con 0x7f5e0c0c41b0 2023-03-03T09:56:17.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e0c0ced00 0x7f5e0c151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e0c0ced00 0x7f5e0c151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38464/0 (socket says 172.21.15.102:38464) 2023-03-03T09:56:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 -- 172.21.15.102:0/3766177483 learned_addr learned my addr 172.21.15.102:0/3766177483 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:56:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e03fff700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0a59c700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c0b3970 0x7f5e0c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 -- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 msgr2=0x7f5e0c159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 -- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c0b3970 msgr2=0x7f5e0c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c0b3970 0x7f5e0c15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0ad9d700 1 -- 172.21.15.102:0/3766177483 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e04009a70 con 0x7f5e0c0ced00 2023-03-03T09:56:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.959+0000 7f5e0a59c700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c0b3970 0x7f5e0c15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:56:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.960+0000 7f5e0ad9d700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e0c0ced00 0x7f5e0c151860 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400b370 tx=0x7f5e04004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.960+0000 7f5e02ffd700 1 -- 172.21.15.102:0/3766177483 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04012430 con 0x7f5e0c0ced00 2023-03-03T09:56:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.960+0000 7f5e02ffd700 1 -- 172.21.15.102:0/3766177483 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e04020070 con 0x7f5e0c0ced00 2023-03-03T09:56:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.960+0000 7f5e02ffd700 1 -- 172.21.15.102:0/3766177483 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04012ea0 con 0x7f5e0c0ced00 2023-03-03T09:56:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.960+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c06bf20 con 0x7f5e0c0ced00 2023-03-03T09:56:17.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.960+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c15ce80 con 0x7f5e0c0ced00 2023-03-03T09:56:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.962+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c06c200 con 0x7f5e0c0ced00 2023-03-03T09:56:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.963+0000 7f5e02ffd700 1 -- 172.21.15.102:0/3766177483 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5e0402e070 con 0x7f5e0c0ced00 2023-03-03T09:56:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.964+0000 7f5e02ffd700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dec0682f0 0x7f5dec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.964+0000 7f5e02ffd700 1 -- 172.21.15.102:0/3766177483 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5e04022070 con 0x7f5e0c0ced00 2023-03-03T09:56:17.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.965+0000 7f5e0a59c700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dec0682f0 0x7f5dec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:17.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.965+0000 7f5e0a59c700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dec0682f0 0x7f5dec06a7a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009fd0 tx=0x7f5dfc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:17.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:17.969+0000 7f5e02ffd700 1 -- 172.21.15.102:0/3766177483 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e040540b0 con 0x7f5e0c0ced00 2023-03-03T09:56:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.206+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e0c04e6e0 con 0x7f5e0c0ced00 2023-03-03T09:56:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.206+0000 7f5e02ffd700 1 -- 172.21.15.102:0/3766177483 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5e0402c090 con 0x7f5e0c0ced00 2023-03-03T09:56:18.209 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:56:18.209 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:56:18.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dec0682f0 msgr2=0x7f5dec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:18.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dec0682f0 0x7f5dec06a7a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009fd0 tx=0x7f5dfc009210 comp rx=0 tx=0).stop 2023-03-03T09:56:18.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e0c0ced00 msgr2=0x7f5e0c151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:18.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e0c0ced00 0x7f5e0c151860 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400b370 tx=0x7f5e04004b20 comp rx=0 tx=0).stop 2023-03-03T09:56:18.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 shutdown_connections 2023-03-03T09:56:18.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e0c0ced00 0x7f5e0c151860 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:18.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e0c0b3970 0x7f5e0c15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:18.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dec0682f0 0x7f5dec06a7a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:18.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 --2- 172.21.15.102:0/3766177483 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e0c0c41b0 0x7f5e0c159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:18.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 >> 172.21.15.102:0/3766177483 conn(0x7f5e0c08b1c0 msgr2=0x7f5e0c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:18.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 shutdown_connections 2023-03-03T09:56:18.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:18.209+0000 7f5e0b59e700 1 -- 172.21.15.102:0/3766177483 wait complete. 2023-03-03T09:56:18.213 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:56:18.238 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-03-03T09:56:21.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:56:21.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.484+0000 7f54e2f9d700 1 Processor -- start 2023-03-03T09:56:21.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.485+0000 7f54e2f9d700 1 -- start start 2023-03-03T09:56:21.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e2f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc1564b0 0x7f54dc0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:21.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e2f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f54dc0c9610 0x7f54dc0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:21.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e2f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54dc0ca440 0x7f54dc0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:21.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e2f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0c9ca0 con 0x7f54dc0c9610 2023-03-03T09:56:21.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e2f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0b46f0 con 0x7f54dc0ca440 2023-03-03T09:56:21.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e2f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0cadc0 con 0x7f54dc1564b0 2023-03-03T09:56:21.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e179a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f54dc0c9610 0x7f54dc0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:21.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e279c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54dc0ca440 0x7f54dc0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:21.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e179a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f54dc0c9610 0x7f54dc0cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38468/0 (socket says 172.21.15.102:38468) 2023-03-03T09:56:21.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e179a700 1 -- 172.21.15.102:0/458759048 learned_addr learned my addr 172.21.15.102:0/458759048 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:56:21.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.486+0000 7f54e1f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc1564b0 0x7f54dc0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e279c700 1 -- 172.21.15.102:0/458759048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc1564b0 msgr2=0x7f54dc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:56:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e279c700 1 --2- 172.21.15.102:0/458759048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc1564b0 0x7f54dc0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e279c700 1 -- 172.21.15.102:0/458759048 >> v2:172.21.15.102:3300/0 conn(0x7f54dc0c9610 msgr2=0x7f54dc0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:56:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e279c700 1 --2- 172.21.15.102:0/458759048 >> v2:172.21.15.102:3300/0 conn(0x7f54dc0c9610 0x7f54dc0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e279c700 1 -- 172.21.15.102:0/458759048 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0b4920 con 0x7f54dc0ca440 2023-03-03T09:56:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e179a700 1 --2- 172.21.15.102:0/458759048 >> v2:172.21.15.102:3300/0 conn(0x7f54dc0c9610 0x7f54dc0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:56:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e279c700 1 --2- 172.21.15.102:0/458759048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0ca440 0x7f54dc0ca880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0b1c10 tx=0x7f54d400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=716d00ae10dcf8e3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e0f99700 1 -- 172.21.15.102:0/458759048 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d400e070 con 0x7f54dc0ca440 2023-03-03T09:56:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e0f99700 1 -- 172.21.15.102:0/458759048 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d400bb20 con 0x7f54dc0ca440 2023-03-03T09:56:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.487+0000 7f54e0f99700 1 -- 172.21.15.102:0/458759048 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4012430 con 0x7f54dc0ca440 2023-03-03T09:56:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 -- 172.21.15.102:0/458759048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0ca440 msgr2=0x7f54dc0ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/458759048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0ca440 0x7f54dc0ca880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0b1c10 tx=0x7f54d400b330 comp rx=0 tx=0).stop 2023-03-03T09:56:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 -- 172.21.15.102:0/458759048 shutdown_connections 2023-03-03T09:56:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/458759048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0ca440 0x7f54dc0ca880 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/458759048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc1564b0 0x7f54dc0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/458759048 >> v2:172.21.15.102:3300/0 conn(0x7f54dc0c9610 0x7f54dc0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 -- 172.21.15.102:0/458759048 >> 172.21.15.102:0/458759048 conn(0x7f54dc065fb0 msgr2=0x7f54dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 -- 172.21.15.102:0/458759048 shutdown_connections 2023-03-03T09:56:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.488+0000 7f54e2f9d700 1 -- 172.21.15.102:0/458759048 wait complete. 2023-03-03T09:56:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.490+0000 7f54e2f9d700 1 Processor -- start 2023-03-03T09:56:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.490+0000 7f54e2f9d700 1 -- start start 2023-03-03T09:56:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.491+0000 7f54e2f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54dc0c9610 0x7f54dc0c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.491+0000 7f54e2f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc0ca440 0x7f54dc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:21.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.491+0000 7f54e2f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc1564b0 0x7f54dc1f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:21.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.491+0000 7f54e2f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c6e80 con 0x7f54dc0c9610 2023-03-03T09:56:21.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.491+0000 7f54e2f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c7940 con 0x7f54dc1564b0 2023-03-03T09:56:21.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.491+0000 7f54e2f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c7c10 con 0x7f54dc0ca440 2023-03-03T09:56:21.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e1f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54dc0c9610 0x7f54dc0c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:21.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc1564b0 0x7f54dc1f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:21.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc1564b0 0x7f54dc1f00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53670/0 (socket says 172.21.15.102:53670) 2023-03-03T09:56:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 -- 172.21.15.102:0/3964022034 learned_addr learned my addr 172.21.15.102:0/3964022034 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:56:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e179a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc0ca440 0x7f54dc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 -- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc0ca440 msgr2=0x7f54dc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc0ca440 0x7f54dc1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 -- 172.21.15.102:0/3964022034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54dc0c9610 msgr2=0x7f54dc0c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54dc0c9610 0x7f54dc0c6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 -- 172.21.15.102:0/3964022034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc009550 con 0x7f54dc1564b0 2023-03-03T09:56:21.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.492+0000 7f54e279c700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc1564b0 0x7f54dc1f00e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f54d4014040 tx=0x7f54d4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:21.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.493+0000 7f54d27fc700 1 -- 172.21.15.102:0/3964022034 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d40048b0 con 0x7f54dc1564b0 2023-03-03T09:56:21.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.493+0000 7f54d27fc700 1 -- 172.21.15.102:0/3964022034 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4020070 con 0x7f54dc1564b0 2023-03-03T09:56:21.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.493+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d4009a70 con 0x7f54dc1564b0 2023-03-03T09:56:21.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.493+0000 7f54d27fc700 1 -- 172.21.15.102:0/3964022034 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d40276c0 con 0x7f54dc1564b0 2023-03-03T09:56:21.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.493+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc0b1f30 con 0x7f54dc1564b0 2023-03-03T09:56:21.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.494+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc0b4c00 con 0x7f54dc1564b0 2023-03-03T09:56:21.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.495+0000 7f54d27fc700 1 -- 172.21.15.102:0/3964022034 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f54d4012430 con 0x7f54dc1564b0 2023-03-03T09:56:21.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.496+0000 7f54d27fc700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54c00682f0 0x7f54c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:21.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.497+0000 7f54d27fc700 1 -- 172.21.15.102:0/3964022034 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f54d4022070 con 0x7f54dc1564b0 2023-03-03T09:56:21.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.497+0000 7f54e1f9b700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54c00682f0 0x7f54c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:21.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.497+0000 7f54e1f9b700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54c00682f0 0x7f54c006a7a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009b40 tx=0x7f54d8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:21.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.501+0000 7f54d27fc700 1 -- 172.21.15.102:0/3964022034 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f54d4054690 con 0x7f54dc1564b0 2023-03-03T09:56:21.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.736+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54dc0b2210 con 0x7f54dc1564b0 2023-03-03T09:56:21.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.737+0000 7f54d27fc700 1 -- 172.21.15.102:0/3964022034 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f54d4057cb0 con 0x7f54dc1564b0 2023-03-03T09:56:21.739 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:56:21.739 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:56:21.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54c00682f0 msgr2=0x7f54c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54c00682f0 0x7f54c006a7a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009b40 tx=0x7f54d8009210 comp rx=0 tx=0).stop 2023-03-03T09:56:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc1564b0 msgr2=0x7f54dc1f00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc1564b0 0x7f54dc1f00e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f54d4014040 tx=0x7f54d4004ad0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 shutdown_connections 2023-03-03T09:56:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc1564b0 0x7f54dc1f00e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54dc0c9610 0x7f54dc0c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f54c00682f0 0x7f54c006a7a0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 --2- 172.21.15.102:0/3964022034 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54dc0ca440 0x7f54dc1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:21.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 >> 172.21.15.102:0/3964022034 conn(0x7f54dc065fb0 msgr2=0x7f54dc154a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:21.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.739+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 shutdown_connections 2023-03-03T09:56:21.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:21.740+0000 7f54e2f9d700 1 -- 172.21.15.102:0/3964022034 wait complete. 2023-03-03T09:56:21.744 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:56:21.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T09:56:30.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:56:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.019+0000 7f868ffff700 1 Processor -- start 2023-03-03T09:56:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.019+0000 7f868ffff700 1 -- start start 2023-03-03T09:56:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.020+0000 7f868ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f86880cd8e0 0x7f86880cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.020+0000 7f868ffff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8688153660 con 0x7f8688153290 2023-03-03T09:56:31.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.020+0000 7f868ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868806c360 con 0x7f86880cd390 2023-03-03T09:56:31.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.020+0000 7f868ffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868815cea0 con 0x7f86880cd8e0 2023-03-03T09:56:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868d59a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f86880cd8e0 0x7f86880cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8688153290 0x7f8688158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47578/0 (socket says 172.21.15.102:47578) 2023-03-03T09:56:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868dd9b700 1 -- 172.21.15.102:0/2463548144 learned_addr learned my addr 172.21.15.102:0/2463548144 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:56:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868cd99700 1 -- 172.21.15.102:0/2463548144 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114412558 0 0) 0x7f868806c360 con 0x7f86880cd390 2023-03-03T09:56:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868cd99700 1 -- 172.21.15.102:0/2463548144 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86740035b0 con 0x7f86880cd390 2023-03-03T09:56:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868cd99700 1 -- 172.21.15.102:0/2463548144 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886437249 0 0) 0x7f8688153660 con 0x7f8688153290 2023-03-03T09:56:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868d59a700 1 -- 172.21.15.102:0/2463548144 >> v1:172.21.15.172:6789/0 conn(0x7f86880cd390 legacy=0x7f86880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868d59a700 1 -- 172.21.15.102:0/2463548144 >> v1:172.21.15.102:6789/0 conn(0x7f8688153290 legacy=0x7f8688158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.021+0000 7f868d59a700 1 -- 172.21.15.102:0/2463548144 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868806c7f0 con 0x7f86880cd8e0 2023-03-03T09:56:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.022+0000 7f868d59a700 1 --2- 172.21.15.102:0/2463548144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd8e0 0x7f86880cde20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8678009840 tx=0x7f8678009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82f80a98492129d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.022+0000 7f868cd99700 1 -- 172.21.15.102:0/2463548144 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678011830 con 0x7f86880cd8e0 2023-03-03T09:56:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.022+0000 7f868cd99700 1 -- 172.21.15.102:0/2463548144 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8678011990 con 0x7f86880cd8e0 2023-03-03T09:56:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.022+0000 7f868cd99700 1 -- 172.21.15.102:0/2463548144 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678011d20 con 0x7f86880cd8e0 2023-03-03T09:56:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.022+0000 7f868ffff700 1 -- 172.21.15.102:0/2463548144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd8e0 msgr2=0x7f86880cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.022+0000 7f868ffff700 1 --2- 172.21.15.102:0/2463548144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd8e0 0x7f86880cde20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8678009840 tx=0x7f8678009b50 comp rx=0 tx=0).stop 2023-03-03T09:56:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.023+0000 7f868ffff700 1 -- 172.21.15.102:0/2463548144 shutdown_connections 2023-03-03T09:56:31.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.023+0000 7f868ffff700 1 --2- 172.21.15.102:0/2463548144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd8e0 0x7f86880cde20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:31.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.023+0000 7f868ffff700 1 -- 172.21.15.102:0/2463548144 >> 172.21.15.102:0/2463548144 conn(0x7f8688065fb0 msgr2=0x7f86880663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:31.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.023+0000 7f868ffff700 1 -- 172.21.15.102:0/2463548144 shutdown_connections 2023-03-03T09:56:31.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.023+0000 7f868ffff700 1 -- 172.21.15.102:0/2463548144 wait complete. 2023-03-03T09:56:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.025+0000 7f868ffff700 1 Processor -- start 2023-03-03T09:56:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.025+0000 7f868ffff700 1 -- start start 2023-03-03T09:56:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd390 0x7f86881e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86880cd8e0 0x7f86881500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8688153290 0x7f8688150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688158a80 con 0x7f86880cd8e0 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868814f520 con 0x7f86880cd390 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86880c57c0 con 0x7f8688153290 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868d59a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86880cd8e0 0x7f86881500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868d59a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86880cd8e0 0x7f86881500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51202/0 (socket says 172.21.15.102:51202) 2023-03-03T09:56:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868d59a700 1 -- 172.21.15.102:0/970695954 learned_addr learned my addr 172.21.15.102:0/970695954 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:56:31.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868dd9b700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd390 0x7f86881e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:31.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.026+0000 7f868e59c700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8688153290 0x7f8688150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868d59a700 1 -- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8688153290 msgr2=0x7f8688150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868d59a700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8688153290 0x7f8688150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868d59a700 1 -- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd390 msgr2=0x7f86881e54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868d59a700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd390 0x7f86881e54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868d59a700 1 -- 172.21.15.102:0/970695954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86780094a0 con 0x7f86880cd8e0 2023-03-03T09:56:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868e59c700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8688153290 0x7f8688150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:56:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868d59a700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86880cd8e0 0x7f86881500c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f86780090b0 tx=0x7f8678004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f867e7fc700 1 -- 172.21.15.102:0/970695954 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867801a860 con 0x7f86880cd8e0 2023-03-03T09:56:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f867e7fc700 1 -- 172.21.15.102:0/970695954 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867801a9c0 con 0x7f86880cd8e0 2023-03-03T09:56:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f867e7fc700 1 -- 172.21.15.102:0/970695954 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867801ad30 con 0x7f86880cd8e0 2023-03-03T09:56:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.027+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86880ccde0 con 0x7f86880cd8e0 2023-03-03T09:56:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.028+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868815d160 con 0x7f86880cd8e0 2023-03-03T09:56:31.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.028+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86880cd0c0 con 0x7f86880cd8e0 2023-03-03T09:56:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.030+0000 7f867e7fc700 1 -- 172.21.15.102:0/970695954 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f867800f050 con 0x7f86880cd8e0 2023-03-03T09:56:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.031+0000 7f867e7fc700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f866c0682f0 0x7f866c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.031+0000 7f867e7fc700 1 -- 172.21.15.102:0/970695954 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8678086c40 con 0x7f86880cd8e0 2023-03-03T09:56:31.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.031+0000 7f868dd9b700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f866c0682f0 0x7f866c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:31.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.032+0000 7f868dd9b700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f866c0682f0 0x7f866c06a7a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8684000f60 tx=0x7f8684007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:31.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.036+0000 7f867e7fc700 1 -- 172.21.15.102:0/970695954 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8678053400 con 0x7f86880cd8e0 2023-03-03T09:56:31.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.267+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86881ef2e0 con 0x7f86880cd8e0 2023-03-03T09:56:31.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.268+0000 7f867e7fc700 1 -- 172.21.15.102:0/970695954 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8678029090 con 0x7f86880cd8e0 2023-03-03T09:56:31.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:56:31.270 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:56:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.270+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f866c0682f0 msgr2=0x7f866c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.270+0000 7f868ffff700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f866c0682f0 0x7f866c06a7a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8684000f60 tx=0x7f8684007e70 comp rx=0 tx=0).stop 2023-03-03T09:56:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.270+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86880cd8e0 msgr2=0x7f86881500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.270+0000 7f868ffff700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86880cd8e0 0x7f86881500c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f86780090b0 tx=0x7f8678004750 comp rx=0 tx=0).stop 2023-03-03T09:56:31.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 shutdown_connections 2023-03-03T09:56:31.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f866c0682f0 0x7f866c06a7a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:31.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8688153290 0x7f8688150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:31.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86880cd390 0x7f86881e54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:31.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 --2- 172.21.15.102:0/970695954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86880cd8e0 0x7f86881500c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:31.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 >> 172.21.15.102:0/970695954 conn(0x7f8688065fb0 msgr2=0x7f8688156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:31.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 shutdown_connections 2023-03-03T09:56:31.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:56:31.271+0000 7f868ffff700 1 -- 172.21.15.102:0/970695954 wait complete. 2023-03-03T09:56:31.274 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:56:31.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-03T09:56:31.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:31.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:01.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:01.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:31.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:31.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:42.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:57:42.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.548+0000 7f517cfff700 1 Processor -- start 2023-03-03T09:57:42.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.548+0000 7f517cfff700 1 -- start start 2023-03-03T09:57:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517cfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5178153290 0x7f5178158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f51780ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517cfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5178153660 con 0x7f51781593c0 2023-03-03T09:57:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517cfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517806c360 con 0x7f5178153290 2023-03-03T09:57:42.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517815cea0 con 0x7f51780cd8e0 2023-03-03T09:57:42.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5178153290 0x7f5178158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:42.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5178153290 0x7f5178158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47820/0 (socket says 172.21.15.102:47820) 2023-03-03T09:57:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f517659c700 1 -- 172.21.15.102:0/3660193591 learned_addr learned my addr 172.21.15.102:0/3660193591 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:57:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.550+0000 7f5176d9d700 1 --2- 172.21.15.102:0/3660193591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f51780ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517659c700 1 -- 172.21.15.102:0/3660193591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 msgr2=0x7f51780ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:57:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517659c700 1 --2- 172.21.15.102:0/3660193591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f51780ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517659c700 1 -- 172.21.15.102:0/3660193591 >> v1:172.21.15.102:6789/0 conn(0x7f51781593c0 legacy=0x7f51780cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517659c700 1 -- 172.21.15.102:0/3660193591 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517806c7f0 con 0x7f5178153290 2023-03-03T09:57:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517659c700 1 --2- 172.21.15.102:0/3660193591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 0x7f5178158e80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f51680098f0 tx=0x7f5168009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba60e8937725753 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:57:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517559a700 1 -- 172.21.15.102:0/3660193591 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168011830 con 0x7f5178153290 2023-03-03T09:57:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517559a700 1 -- 172.21.15.102:0/3660193591 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5168011990 con 0x7f5178153290 2023-03-03T09:57:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.551+0000 7f517559a700 1 -- 172.21.15.102:0/3660193591 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168011d20 con 0x7f5178153290 2023-03-03T09:57:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 -- 172.21.15.102:0/3660193591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 msgr2=0x7f5178158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 --2- 172.21.15.102:0/3660193591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 0x7f5178158e80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f51680098f0 tx=0x7f5168009c00 comp rx=0 tx=0).stop 2023-03-03T09:57:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 -- 172.21.15.102:0/3660193591 shutdown_connections 2023-03-03T09:57:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 --2- 172.21.15.102:0/3660193591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f51780ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 --2- 172.21.15.102:0/3660193591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 0x7f5178158e80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 -- 172.21.15.102:0/3660193591 >> 172.21.15.102:0/3660193591 conn(0x7f5178065fb0 msgr2=0x7f51780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:57:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 -- 172.21.15.102:0/3660193591 shutdown_connections 2023-03-03T09:57:42.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.552+0000 7f517cfff700 1 -- 172.21.15.102:0/3660193591 wait complete. 2023-03-03T09:57:42.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.554+0000 7f517cfff700 1 Processor -- start 2023-03-03T09:57:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.554+0000 7f517cfff700 1 -- start start 2023-03-03T09:57:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.555+0000 7f517cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f5178156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.555+0000 7f517cfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 0x7f517815b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.555+0000 7f517cfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51781593c0 0x7f5178150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.555+0000 7f517cfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51780b44e0 con 0x7f5178153290 2023-03-03T09:57:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.555+0000 7f517cfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51781ee5d0 con 0x7f51781593c0 2023-03-03T09:57:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.555+0000 7f517cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51781ee8b0 con 0x7f51780cd8e0 2023-03-03T09:57:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f5178156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f5178156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36834/0 (socket says 172.21.15.102:36834) 2023-03-03T09:57:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f5175d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 0x7f517815b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 -- 172.21.15.102:0/766511454 learned_addr learned my addr 172.21.15.102:0/766511454 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:57:42.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f5176d9d700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51781593c0 0x7f5178150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:42.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 -- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51781593c0 msgr2=0x7f5178150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51781593c0 0x7f5178150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 -- 172.21.15.102:0/766511454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 msgr2=0x7f517815b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 0x7f517815b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 -- 172.21.15.102:0/766511454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5168009550 con 0x7f51780cd8e0 2023-03-03T09:57:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f5176d9d700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51781593c0 0x7f5178150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:57:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.556+0000 7f517659c700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f5178156290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5168009c40 tx=0x7f5168004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:57:42.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.557+0000 7f5166ffd700 1 -- 172.21.15.102:0/766511454 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168004510 con 0x7f51780cd8e0 2023-03-03T09:57:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.557+0000 7f5166ffd700 1 -- 172.21.15.102:0/766511454 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516801e070 con 0x7f51780cd8e0 2023-03-03T09:57:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.557+0000 7f5166ffd700 1 -- 172.21.15.102:0/766511454 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51680038e0 con 0x7f51780cd8e0 2023-03-03T09:57:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.557+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51780ccde0 con 0x7f51780cd8e0 2023-03-03T09:57:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.557+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517815d160 con 0x7f51780cd8e0 2023-03-03T09:57:42.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.558+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51780cd0c0 con 0x7f51780cd8e0 2023-03-03T09:57:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.559+0000 7f5166ffd700 1 -- 172.21.15.102:0/766511454 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f516800f050 con 0x7f51780cd8e0 2023-03-03T09:57:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.561+0000 7f5166ffd700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51580682f0 0x7f515806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.561+0000 7f5166ffd700 1 -- 172.21.15.102:0/766511454 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5168087510 con 0x7f51780cd8e0 2023-03-03T09:57:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.561+0000 7f5175d9b700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51580682f0 0x7f515806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:42.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.562+0000 7f5175d9b700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51580682f0 0x7f515806a7a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f517815b890 tx=0x7f516c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:57:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.564+0000 7f5166ffd700 1 -- 172.21.15.102:0/766511454 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5168053c20 con 0x7f51780cd8e0 2023-03-03T09:57:42.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.796+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f517804e6e0 con 0x7f51780cd8e0 2023-03-03T09:57:42.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.798+0000 7f5166ffd700 1 -- 172.21.15.102:0/766511454 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5168057240 con 0x7f51780cd8e0 2023-03-03T09:57:42.800 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:57:42.800 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:57:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51580682f0 msgr2=0x7f515806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51580682f0 0x7f515806a7a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f517815b890 tx=0x7f516c009040 comp rx=0 tx=0).stop 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 msgr2=0x7f5178156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f5178156290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5168009c40 tx=0x7f5168004750 comp rx=0 tx=0).stop 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 shutdown_connections 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51580682f0 0x7f515806a7a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51780cd8e0 0x7f5178156290 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51781593c0 0x7f5178150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 --2- 172.21.15.102:0/766511454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5178153290 0x7f517815b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 >> 172.21.15.102:0/766511454 conn(0x7f5178065fb0 msgr2=0x7f5178156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:57:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 shutdown_connections 2023-03-03T09:57:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:57:42.800+0000 7f517cfff700 1 -- 172.21.15.102:0/766511454 wait complete. 2023-03-03T09:57:42.804 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:57:42.829 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 78 secs before thrashing 2023-03-03T09:58:02.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:02.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:32.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:32.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:00.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:59:01.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.080+0000 7fc9c67ff700 1 Processor -- start 2023-03-03T09:59:01.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.080+0000 7fc9c67ff700 1 -- start start 2023-03-03T09:59:01.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9c67ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9c00cd8e0 0x7fc9c00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:01.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9c67ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c0153660 con 0x7fc9c0153290 2023-03-03T09:59:01.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9c67ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c006c360 con 0x7fc9c00cd390 2023-03-03T09:59:01.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9c67ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c015cea0 con 0x7fc9c00cd8e0 2023-03-03T09:59:01.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9bffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc9c0153290 0x7fc9c0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44076/0 (socket says 172.21.15.102:44076) 2023-03-03T09:59:01.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9bffff700 1 -- 172.21.15.102:0/3835550470 learned_addr learned my addr 172.21.15.102:0/3835550470 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T09:59:01.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9b7fff700 1 --2- 172.21.15.102:0/3835550470 >> v2:172.21.15.102:3300/0 conn(0x7fc9c00cd8e0 0x7fc9c00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:01.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.082+0000 7fc9bf7fe700 1 -- 172.21.15.102:0/3835550470 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865751818 0 0) 0x7fc9c0153660 con 0x7fc9c0153290 2023-03-03T09:59:01.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9bf7fe700 1 -- 172.21.15.102:0/3835550470 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ac0035b0 con 0x7fc9c0153290 2023-03-03T09:59:01.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9bf7fe700 1 -- 172.21.15.102:0/3835550470 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240702010 0 0) 0x7fc9c006c360 con 0x7fc9c00cd390 2023-03-03T09:59:01.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9b7fff700 1 -- 172.21.15.102:0/3835550470 >> v1:172.21.15.172:6789/0 conn(0x7fc9c00cd390 legacy=0x7fc9c00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:01.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9b7fff700 1 -- 172.21.15.102:0/3835550470 >> v1:172.21.15.102:6789/0 conn(0x7fc9c0153290 legacy=0x7fc9c0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:01.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9b7fff700 1 -- 172.21.15.102:0/3835550470 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c006c7f0 con 0x7fc9c00cd8e0 2023-03-03T09:59:01.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9b7fff700 1 --2- 172.21.15.102:0/3835550470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c00cde20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009840 tx=0x7fc9a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5abd79cf30471ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:01.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9bf7fe700 1 -- 172.21.15.102:0/3835550470 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8011830 con 0x7fc9c00cd8e0 2023-03-03T09:59:01.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9bf7fe700 1 -- 172.21.15.102:0/3835550470 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a8011990 con 0x7fc9c00cd8e0 2023-03-03T09:59:01.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.083+0000 7fc9bf7fe700 1 -- 172.21.15.102:0/3835550470 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8011d20 con 0x7fc9c00cd8e0 2023-03-03T09:59:01.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.084+0000 7fc9c67ff700 1 -- 172.21.15.102:0/3835550470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd8e0 msgr2=0x7fc9c00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:01.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.084+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/3835550470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c00cde20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009840 tx=0x7fc9a8009b50 comp rx=0 tx=0).stop 2023-03-03T09:59:01.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.084+0000 7fc9c67ff700 1 -- 172.21.15.102:0/3835550470 shutdown_connections 2023-03-03T09:59:01.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.084+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/3835550470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c00cde20 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:01.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.084+0000 7fc9c67ff700 1 -- 172.21.15.102:0/3835550470 >> 172.21.15.102:0/3835550470 conn(0x7fc9c0065fb0 msgr2=0x7fc9c00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:01.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.084+0000 7fc9c67ff700 1 -- 172.21.15.102:0/3835550470 shutdown_connections 2023-03-03T09:59:01.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.084+0000 7fc9c67ff700 1 -- 172.21.15.102:0/3835550470 wait complete. 2023-03-03T09:59:01.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.086+0000 7fc9c67ff700 1 Processor -- start 2023-03-03T09:59:01.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.086+0000 7fc9c67ff700 1 -- start start 2023-03-03T09:59:01.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9c67ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd390 0x7fc9c0157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:01.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9c67ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c01ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:01.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9c67ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c0153290 0x7fc9c01eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:01.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9c67ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c00c6df0 con 0x7fc9c00cd390 2023-03-03T09:59:01.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9c67ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c00c8790 con 0x7fc9c00cd8e0 2023-03-03T09:59:01.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9c67ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c00c8a70 con 0x7fc9c0153290 2023-03-03T09:59:01.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9bffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd390 0x7fc9c0157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:01.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.087+0000 7fc9bffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd390 0x7fc9c0157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59586/0 (socket says 172.21.15.102:59586) 2023-03-03T09:59:01.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9bffff700 1 -- 172.21.15.102:0/886578239 learned_addr learned my addr 172.21.15.102:0/886578239 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T09:59:01.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9b7fff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c01ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:01.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9c4d9c700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c0153290 0x7fc9c01eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:01.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9bffff700 1 -- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c0153290 msgr2=0x7fc9c01eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:01.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9bffff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c0153290 0x7fc9c01eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:01.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9bffff700 1 -- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c00cd8e0 msgr2=0x7fc9c01ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:01.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9bffff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c01ee8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:01.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9bffff700 1 -- 172.21.15.102:0/886578239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80094a0 con 0x7fc9c00cd390 2023-03-03T09:59:01.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9b7fff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c01ee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:59:01.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.088+0000 7fc9bffff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd390 0x7fc9c0157a00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c420 tx=0x7fc9b000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:01.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.089+0000 7fc9bd7fa700 1 -- 172.21.15.102:0/886578239 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0005d70 con 0x7fc9c00cd390 2023-03-03T09:59:01.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.089+0000 7fc9bd7fa700 1 -- 172.21.15.102:0/886578239 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b000ce30 con 0x7fc9c00cd390 2023-03-03T09:59:01.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.089+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c00ccde0 con 0x7fc9c00cd390 2023-03-03T09:59:01.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.089+0000 7fc9bd7fa700 1 -- 172.21.15.102:0/886578239 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b00072d0 con 0x7fc9c00cd390 2023-03-03T09:59:01.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.089+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c015d160 con 0x7fc9c00cd390 2023-03-03T09:59:01.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.090+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c00cd0c0 con 0x7fc9c00cd390 2023-03-03T09:59:01.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.091+0000 7fc9bd7fa700 1 -- 172.21.15.102:0/886578239 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc9b0019030 con 0x7fc9c00cd390 2023-03-03T09:59:01.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.093+0000 7fc9bd7fa700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a0068320 0x7fc9a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:01.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.093+0000 7fc9bd7fa700 1 -- 172.21.15.102:0/886578239 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc9b0084f00 con 0x7fc9c00cd390 2023-03-03T09:59:01.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.094+0000 7fc9b7fff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a0068320 0x7fc9a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:01.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.094+0000 7fc9b7fff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a0068320 0x7fc9a006a7d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009fd0 tx=0x7fc9a801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:01.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.098+0000 7fc9bd7fa700 1 -- 172.21.15.102:0/886578239 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc9b004b5e0 con 0x7fc9c00cd390 2023-03-03T09:59:01.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.338+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9c004e6e0 con 0x7fc9c00cd390 2023-03-03T09:59:01.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.339+0000 7fc9bd7fa700 1 -- 172.21.15.102:0/886578239 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc9b0017070 con 0x7fc9c00cd390 2023-03-03T09:59:01.340 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T09:59:01.340 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T09:59:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a0068320 msgr2=0x7fc9a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a0068320 0x7fc9a006a7d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009fd0 tx=0x7fc9a801c040 comp rx=0 tx=0).stop 2023-03-03T09:59:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd390 msgr2=0x7fc9c0157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:01.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd390 0x7fc9c0157a00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c420 tx=0x7fc9b000c730 comp rx=0 tx=0).stop 2023-03-03T09:59:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 shutdown_connections 2023-03-03T09:59:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9a0068320 0x7fc9a006a7d0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9c0153290 0x7fc9c01eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9c00cd390 0x7fc9c0157a00 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 --2- 172.21.15.102:0/886578239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9c00cd8e0 0x7fc9c01ee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 >> 172.21.15.102:0/886578239 conn(0x7fc9c0065fb0 msgr2=0x7fc9c0158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 shutdown_connections 2023-03-03T09:59:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T09:59:01.342+0000 7fc9c67ff700 1 -- 172.21.15.102:0/886578239 wait complete. 2023-03-03T09:59:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T09:59:01.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T09:59:02.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:02.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:32.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:32.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T10:00:00.000+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:00:02.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:02.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:33.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:33.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:55.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:00:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.636+0000 7fcbd75ff700 1 Processor -- start 2023-03-03T10:00:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.636+0000 7fcbd75ff700 1 -- start start 2023-03-03T10:00:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd75ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcbd0153290 0x7fcbd0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd75ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcbd01593c0 0x7fcbd00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd75ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd0153660 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd75ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd006c360 con 0x7fcbd01593c0 2023-03-03T10:00:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd75ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd015cea0 con 0x7fcbd0153290 2023-03-03T10:00:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd539b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcbd0153290 0x7fcbd0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd539b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcbd0153290 0x7fcbd0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57482/0 (socket says 172.21.15.102:57482) 2023-03-03T10:00:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd5b9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcbd00cd8e0 0x7fcbd00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:53744/0 (socket says 172.21.15.102:53744) 2023-03-03T10:00:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd539b700 1 -- 172.21.15.102:0/4018529340 learned_addr learned my addr 172.21.15.102:0/4018529340 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:00:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.638+0000 7fcbd4b9a700 1 --2- 172.21.15.102:0/4018529340 >> v2:172.21.15.102:3300/0 conn(0x7fcbd01593c0 0x7fcbd00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbd539b700 1 -- 172.21.15.102:0/4018529340 >> v2:172.21.15.102:3300/0 conn(0x7fcbd01593c0 msgr2=0x7fcbd00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:00:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbd539b700 1 --2- 172.21.15.102:0/4018529340 >> v2:172.21.15.102:3300/0 conn(0x7fcbd01593c0 0x7fcbd00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbd539b700 1 -- 172.21.15.102:0/4018529340 >> v1:172.21.15.172:6789/0 conn(0x7fcbd00cd8e0 legacy=0x7fcbd00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbd539b700 1 -- 172.21.15.102:0/4018529340 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd006c7f0 con 0x7fcbd0153290 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbc7fff700 1 -- 172.21.15.102:0/4018529340 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335769594 0 0) 0x7fcbd0153660 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbd539b700 1 --2- 172.21.15.102:0/4018529340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 0x7fcbd0158e80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcbd008cc70 tx=0x7fcbcc009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cad938e74aec0855 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbc7fff700 1 -- 172.21.15.102:0/4018529340 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc011830 con 0x7fcbd0153290 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbc7fff700 1 -- 172.21.15.102:0/4018529340 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc011990 con 0x7fcbd0153290 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.639+0000 7fcbc7fff700 1 -- 172.21.15.102:0/4018529340 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc011d20 con 0x7fcbd0153290 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 -- 172.21.15.102:0/4018529340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 msgr2=0x7fcbd0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/4018529340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 0x7fcbd0158e80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcbd008cc70 tx=0x7fcbcc009bd0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 -- 172.21.15.102:0/4018529340 shutdown_connections 2023-03-03T10:00:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/4018529340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 0x7fcbd0158e80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/4018529340 >> v2:172.21.15.102:3300/0 conn(0x7fcbd01593c0 0x7fcbd00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 -- 172.21.15.102:0/4018529340 >> 172.21.15.102:0/4018529340 conn(0x7fcbd0065fb0 msgr2=0x7fcbd00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 -- 172.21.15.102:0/4018529340 shutdown_connections 2023-03-03T10:00:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.640+0000 7fcbd75ff700 1 -- 172.21.15.102:0/4018529340 wait complete. 2023-03-03T10:00:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.642+0000 7fcbd75ff700 1 Processor -- start 2023-03-03T10:00:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.642+0000 7fcbd75ff700 1 -- start start 2023-03-03T10:00:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.643+0000 7fcbd75ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbd00cd8e0 0x7fcbd01e4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.643+0000 7fcbd75ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 0x7fcbd0151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.643+0000 7fcbd75ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbd01593c0 0x7fcbd01ef830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.643+0000 7fcbd75ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd01f2240 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.643+0000 7fcbd75ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd01f2730 con 0x7fcbd0153290 2023-03-03T10:00:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.643+0000 7fcbd75ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd01e4a40 con 0x7fcbd01593c0 2023-03-03T10:00:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbd00cd8e0 0x7fcbd01e4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbd00cd8e0 0x7fcbd01e4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41548/0 (socket says 172.21.15.102:41548) 2023-03-03T10:00:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 -- 172.21.15.102:0/3022093369 learned_addr learned my addr 172.21.15.102:0/3022093369 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:00:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd4b9a700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 0x7fcbd0151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd5b9c700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbd01593c0 0x7fcbd01ef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 -- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbd01593c0 msgr2=0x7fcbd01ef830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbd01593c0 0x7fcbd01ef830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 -- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 msgr2=0x7fcbd0151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 0x7fcbd0151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.644+0000 7fcbd539b700 1 -- 172.21.15.102:0/3022093369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc009550 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.645+0000 7fcbd539b700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbd00cd8e0 0x7fcbd01e4060 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009c10 tx=0x7fcbcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.645+0000 7fcbc5ffb700 1 -- 172.21.15.102:0/3022093369 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc004510 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.645+0000 7fcbc5ffb700 1 -- 172.21.15.102:0/3022093369 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0036a0 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.645+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd00ccde0 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.645+0000 7fcbc5ffb700 1 -- 172.21.15.102:0/3022093369 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc003a10 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.645+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd015d160 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.646+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd00cd0c0 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.648+0000 7fcbc5ffb700 1 -- 172.21.15.102:0/3022093369 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcbcc02d070 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.649+0000 7fcbc5ffb700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcbb40682f0 0x7fcbb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.649+0000 7fcbc5ffb700 1 -- 172.21.15.102:0/3022093369 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcbcc021030 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.649+0000 7fcbd4b9a700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcbb40682f0 0x7fcbb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.650+0000 7fcbd4b9a700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcbb40682f0 0x7fcbb406a7a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcbd014e960 tx=0x7fcbc0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.653+0000 7fcbc5ffb700 1 -- 172.21.15.102:0/3022093369 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcbcc053ac0 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.887+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbd01e3c10 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.888+0000 7fcbc5ffb700 1 -- 172.21.15.102:0/3022093369 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcbcc015090 con 0x7fcbd00cd8e0 2023-03-03T10:00:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:00:55.891 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcbb40682f0 msgr2=0x7fcbb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcbb40682f0 0x7fcbb406a7a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcbd014e960 tx=0x7fcbc0009210 comp rx=0 tx=0).stop 2023-03-03T10:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbd00cd8e0 msgr2=0x7fcbd01e4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbd00cd8e0 0x7fcbd01e4060 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009c10 tx=0x7fcbcc004750 comp rx=0 tx=0).stop 2023-03-03T10:00:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 shutdown_connections 2023-03-03T10:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcbb40682f0 0x7fcbb406a7a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcbd01593c0 0x7fcbd01ef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbd00cd8e0 0x7fcbd01e4060 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 --2- 172.21.15.102:0/3022093369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd0153290 0x7fcbd0151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 >> 172.21.15.102:0/3022093369 conn(0x7fcbd0065fb0 msgr2=0x7fcbd00cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 shutdown_connections 2023-03-03T10:00:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:00:55.891+0000 7fcbd75ff700 1 -- 172.21.15.102:0/3022093369 wait complete. 2023-03-03T10:00:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:00:55.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T10:01:03.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:03.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:33.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:33.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:38.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:01:39.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.176+0000 7f67e17ff700 1 Processor -- start 2023-03-03T10:01:39.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.177+0000 7f67e17ff700 1 -- start start 2023-03-03T10:01:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67e17ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f67dc0c9cb0 0x7f67dc0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:01:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67e17ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0ca190 0x7f67dc0cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:01:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67e17ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc0cf940 con 0x7f67dc153260 2023-03-03T10:01:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67e17ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67dc0cfe70 con 0x7f67dc0c9cb0 2023-03-03T10:01:39.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67e17ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc158d70 con 0x7f67dc0ca190 2023-03-03T10:01:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67d3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f67dc0c9cb0 0x7f67dc0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:01:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67daffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f67dc153260 0x7f67dc0ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51874/0 (socket says 172.21.15.102:51874) 2023-03-03T10:01:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67daffd700 1 -- 172.21.15.102:0/4141033150 learned_addr learned my addr 172.21.15.102:0/4141033150 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:01:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.178+0000 7f67db7fe700 1 --2- 172.21.15.102:0/4141033150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0ca190 0x7f67dc0cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:01:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67d3fff700 1 -- 172.21.15.102:0/4141033150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0ca190 msgr2=0x7f67dc0cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:01:39.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67d3fff700 1 --2- 172.21.15.102:0/4141033150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0ca190 0x7f67dc0cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67d3fff700 1 -- 172.21.15.102:0/4141033150 >> v1:172.21.15.172:6789/0 conn(0x7f67dc153260 legacy=0x7f67dc0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:01:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67d3fff700 1 -- 172.21.15.102:0/4141033150 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc0cfb90 con 0x7f67dc0c9cb0 2023-03-03T10:01:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67da7fc700 1 -- 172.21.15.102:0/4141033150 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179384348 0 0) 0x7f67dc0cf940 con 0x7f67dc153260 2023-03-03T10:01:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67d3fff700 1 --2- 172.21.15.102:0/4141033150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc0c9cb0 0x7f67dc0cb880 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009840 tx=0x7f67c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d14604e752cf57 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:01:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67da7fc700 1 -- 172.21.15.102:0/4141033150 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4011830 con 0x7f67dc0c9cb0 2023-03-03T10:01:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67da7fc700 1 -- 172.21.15.102:0/4141033150 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c4011990 con 0x7f67dc0c9cb0 2023-03-03T10:01:39.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.179+0000 7f67da7fc700 1 -- 172.21.15.102:0/4141033150 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4011d20 con 0x7f67dc0c9cb0 2023-03-03T10:01:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 -- 172.21.15.102:0/4141033150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc0c9cb0 msgr2=0x7f67dc0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:01:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 --2- 172.21.15.102:0/4141033150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc0c9cb0 0x7f67dc0cb880 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009840 tx=0x7f67c4009b50 comp rx=0 tx=0).stop 2023-03-03T10:01:39.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 -- 172.21.15.102:0/4141033150 shutdown_connections 2023-03-03T10:01:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 --2- 172.21.15.102:0/4141033150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc0ca190 0x7f67dc0cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 --2- 172.21.15.102:0/4141033150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc0c9cb0 0x7f67dc0cb880 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 -- 172.21.15.102:0/4141033150 >> 172.21.15.102:0/4141033150 conn(0x7f67dc066c00 msgr2=0x7f67dc0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:01:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 -- 172.21.15.102:0/4141033150 shutdown_connections 2023-03-03T10:01:39.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.180+0000 7f67e17ff700 1 -- 172.21.15.102:0/4141033150 wait complete. 2023-03-03T10:01:39.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.183+0000 7f67e17ff700 1 Processor -- start 2023-03-03T10:01:39.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.184+0000 7f67e17ff700 1 -- start start 2023-03-03T10:01:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67e17ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0ca190 0x7f67dc15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:01:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67e17ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc153260 0x7f67dc15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:01:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67e17ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc148fc0 0x7f67dc1ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:01:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67e17ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc15b100 con 0x7f67dc153260 2023-03-03T10:01:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67e17ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc15b3e0 con 0x7f67dc0ca190 2023-03-03T10:01:39.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67e17ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc159f70 con 0x7f67dc148fc0 2023-03-03T10:01:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67d3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc153260 0x7f67dc15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:01:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67d3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc153260 0x7f67dc15b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60086/0 (socket says 172.21.15.102:60086) 2023-03-03T10:01:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67d3fff700 1 -- 172.21.15.102:0/2656063842 learned_addr learned my addr 172.21.15.102:0/2656063842 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:01:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67daffd700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0ca190 0x7f67dc15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:01:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.185+0000 7f67db7fe700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc148fc0 0x7f67dc1ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:01:39.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d3fff700 1 -- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc148fc0 msgr2=0x7f67dc1ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d3fff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc148fc0 0x7f67dc1ee970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d3fff700 1 -- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0ca190 msgr2=0x7f67dc15a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d3fff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0ca190 0x7f67dc15a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d3fff700 1 -- 172.21.15.102:0/2656063842 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c40094a0 con 0x7f67dc153260 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d3fff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc153260 0x7f67dc15b8b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d37fe700 1 -- 172.21.15.102:0/2656063842 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c40036a0 con 0x7f67dc153260 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d37fe700 1 -- 172.21.15.102:0/2656063842 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c401f070 con 0x7f67dc153260 2023-03-03T10:01:39.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67d37fe700 1 -- 172.21.15.102:0/2656063842 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4003a20 con 0x7f67dc153260 2023-03-03T10:01:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.186+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc159260 con 0x7f67dc153260 2023-03-03T10:01:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.187+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc158f80 con 0x7f67dc153260 2023-03-03T10:01:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.188+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc159540 con 0x7f67dc153260 2023-03-03T10:01:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.189+0000 7f67d37fe700 1 -- 172.21.15.102:0/2656063842 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f67c400f050 con 0x7f67dc153260 2023-03-03T10:01:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.190+0000 7f67d37fe700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67bc0682f0 0x7f67bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:01:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.190+0000 7f67d37fe700 1 -- 172.21.15.102:0/2656063842 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f67c4021070 con 0x7f67dc153260 2023-03-03T10:01:39.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.190+0000 7f67daffd700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67bc0682f0 0x7f67bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:01:39.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.191+0000 7f67daffd700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67bc0682f0 0x7f67bc06a7a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f67cc000f60 tx=0x7f67cc007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:01:39.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.195+0000 7f67d37fe700 1 -- 172.21.15.102:0/2656063842 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f67c4053cb0 con 0x7f67dc153260 2023-03-03T10:01:39.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.431+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67dc15a490 con 0x7f67dc153260 2023-03-03T10:01:39.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.431+0000 7f67d37fe700 1 -- 172.21.15.102:0/2656063842 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f67dc15a490 con 0x7f67dc153260 2023-03-03T10:01:39.434 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:01:39.434 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:01:39.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67bc0682f0 msgr2=0x7f67bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:01:39.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67bc0682f0 0x7f67bc06a7a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f67cc000f60 tx=0x7f67cc007e70 comp rx=0 tx=0).stop 2023-03-03T10:01:39.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc153260 msgr2=0x7f67dc15b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:01:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc153260 0x7f67dc15b8b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c4004750 comp rx=0 tx=0).stop 2023-03-03T10:01:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 shutdown_connections 2023-03-03T10:01:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67bc0682f0 0x7f67bc06a7a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67dc148fc0 0x7f67dc1ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67dc0ca190 0x7f67dc15a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 --2- 172.21.15.102:0/2656063842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67dc153260 0x7f67dc15b8b0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:01:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 >> 172.21.15.102:0/2656063842 conn(0x7f67dc0c9cb0 msgr2=0x7f67dc152680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:01:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 shutdown_connections 2023-03-03T10:01:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:01:39.434+0000 7f67e17ff700 1 -- 172.21.15.102:0/2656063842 wait complete. 2023-03-03T10:01:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:01:39.464 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-03-03T10:02:03.941 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:03.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:34.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:34.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:04.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:04.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:32.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:03:32.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.731+0000 7f3743fff700 1 Processor -- start 2023-03-03T10:03:32.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.731+0000 7f3743fff700 1 -- start start 2023-03-03T10:03:32.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3743fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440ced00 0x7f37440c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:32.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3743fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37440b3970 0x7f37440c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:32.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3743fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37440cf0d0 0x7f3744158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:32.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3743fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440c4930 con 0x7f37440b3970 2023-03-03T10:03:32.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3743fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440c3d20 con 0x7f37440cf0d0 2023-03-03T10:03:32.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3743fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374415cba0 con 0x7f37440ced00 2023-03-03T10:03:32.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3742ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440ced00 0x7f37440c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:32.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f37427fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37440b3970 0x7f37440c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:32.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3742ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440ced00 0x7f37440c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44416/0 (socket says 172.21.15.102:44416) 2023-03-03T10:03:32.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f3742ffd700 1 -- 172.21.15.102:0/4110197984 learned_addr learned my addr 172.21.15.102:0/4110197984 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:03:32.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.733+0000 7f37437fe700 1 --2- 172.21.15.102:0/4110197984 >> v2:172.21.15.172:3300/0 conn(0x7f37440cf0d0 0x7f3744158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:32.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.734+0000 7f3742ffd700 1 -- 172.21.15.102:0/4110197984 >> v2:172.21.15.172:3300/0 conn(0x7f37440cf0d0 msgr2=0x7f3744158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:03:32.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.734+0000 7f3742ffd700 1 --2- 172.21.15.102:0/4110197984 >> v2:172.21.15.172:3300/0 conn(0x7f37440cf0d0 0x7f3744158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.734+0000 7f3742ffd700 1 -- 172.21.15.102:0/4110197984 >> v2:172.21.15.102:3300/0 conn(0x7f37440b3970 msgr2=0x7f37440c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:03:32.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.734+0000 7f3742ffd700 1 --2- 172.21.15.102:0/4110197984 >> v2:172.21.15.102:3300/0 conn(0x7f37440b3970 0x7f37440c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.734+0000 7f3742ffd700 1 -- 172.21.15.102:0/4110197984 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374415ce80 con 0x7f37440ced00 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.734+0000 7f37427fc700 1 --2- 172.21.15.102:0/4110197984 >> v2:172.21.15.102:3300/0 conn(0x7f37440b3970 0x7f37440c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.734+0000 7f3742ffd700 1 --2- 172.21.15.102:0/4110197984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440ced00 0x7f37440c5a90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37440cb6d0 tx=0x7f3734009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=716a30aa1f9acbb8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3741ffb700 1 -- 172.21.15.102:0/4110197984 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734011830 con 0x7f37440ced00 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3741ffb700 1 -- 172.21.15.102:0/4110197984 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3734011990 con 0x7f37440ced00 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3741ffb700 1 -- 172.21.15.102:0/4110197984 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734011d20 con 0x7f37440ced00 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 -- 172.21.15.102:0/4110197984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440ced00 msgr2=0x7f37440c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 --2- 172.21.15.102:0/4110197984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440ced00 0x7f37440c5a90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37440cb6d0 tx=0x7f3734009bd0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 -- 172.21.15.102:0/4110197984 shutdown_connections 2023-03-03T10:03:32.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 --2- 172.21.15.102:0/4110197984 >> v2:172.21.15.172:3300/0 conn(0x7f37440cf0d0 0x7f3744158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 --2- 172.21.15.102:0/4110197984 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440ced00 0x7f37440c5a90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 --2- 172.21.15.102:0/4110197984 >> v2:172.21.15.102:3300/0 conn(0x7f37440b3970 0x7f37440c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 -- 172.21.15.102:0/4110197984 >> 172.21.15.102:0/4110197984 conn(0x7f374408b1c0 msgr2=0x7f3744071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:03:32.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 -- 172.21.15.102:0/4110197984 shutdown_connections 2023-03-03T10:03:32.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.735+0000 7f3743fff700 1 -- 172.21.15.102:0/4110197984 wait complete. 2023-03-03T10:03:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.737+0000 7f3743fff700 1 Processor -- start 2023-03-03T10:03:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.738+0000 7f3743fff700 1 -- start start 2023-03-03T10:03:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f3743fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37440b3970 0x7f374414cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f3743fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37440ced00 0x7f37441ef090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f3743fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440cf0d0 0x7f374414ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f3743fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374414d2e0 con 0x7f37440ced00 2023-03-03T10:03:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f3743fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37441ef5d0 con 0x7f37440b3970 2023-03-03T10:03:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f3743fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374414e7f0 con 0x7f37440cf0d0 2023-03-03T10:03:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37440ced00 0x7f37441ef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37440ced00 0x7f37441ef090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37038/0 (socket says 172.21.15.102:37038) 2023-03-03T10:03:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 -- 172.21.15.102:0/1495633822 learned_addr learned my addr 172.21.15.102:0/1495633822 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:03:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f3742ffd700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37440b3970 0x7f374414cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37437fe700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440cf0d0 0x7f374414ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 -- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440cf0d0 msgr2=0x7f374414ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440cf0d0 0x7f374414ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 -- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37440b3970 msgr2=0x7f374414cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37440b3970 0x7f374414cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37427fc700 1 -- 172.21.15.102:0/1495633822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380094a0 con 0x7f37440ced00 2023-03-03T10:03:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.739+0000 7f37437fe700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440cf0d0 0x7f374414ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:03:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.740+0000 7f37427fc700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37440ced00 0x7f37441ef090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f373800e980 tx=0x7f373800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:03:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.740+0000 7f372b7fe700 1 -- 172.21.15.102:0/1495633822 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37380096e0 con 0x7f37440ced00 2023-03-03T10:03:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.740+0000 7f372b7fe700 1 -- 172.21.15.102:0/1495633822 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373800eed0 con 0x7f37440ced00 2023-03-03T10:03:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.740+0000 7f372b7fe700 1 -- 172.21.15.102:0/1495633822 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373800ca70 con 0x7f37440ced00 2023-03-03T10:03:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.740+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3734009550 con 0x7f37440ced00 2023-03-03T10:03:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.740+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374406c200 con 0x7f37440ced00 2023-03-03T10:03:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.741+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374406bf20 con 0x7f37440ced00 2023-03-03T10:03:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.742+0000 7f372b7fe700 1 -- 172.21.15.102:0/1495633822 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3738019030 con 0x7f37440ced00 2023-03-03T10:03:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.744+0000 7f372b7fe700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37300682f0 0x7f373006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:32.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.745+0000 7f372b7fe700 1 -- 172.21.15.102:0/1495633822 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3738085ce0 con 0x7f37440ced00 2023-03-03T10:03:32.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.745+0000 7f3742ffd700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37300682f0 0x7f373006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:32.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.745+0000 7f3742ffd700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37300682f0 0x7f373006a7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3734009c10 tx=0x7f3734008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:03:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.749+0000 7f372b7fe700 1 -- 172.21.15.102:0/1495633822 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3738052360 con 0x7f37440ced00 2023-03-03T10:03:32.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.985+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3744159040 con 0x7f37440ced00 2023-03-03T10:03:32.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.986+0000 7f372b7fe700 1 -- 172.21.15.102:0/1495633822 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3738017090 con 0x7f37440ced00 2023-03-03T10:03:32.987 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:03:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:03:32.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37300682f0 msgr2=0x7f373006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37300682f0 0x7f373006a7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3734009c10 tx=0x7f3734008030 comp rx=0 tx=0).stop 2023-03-03T10:03:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37440ced00 msgr2=0x7f37441ef090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37440ced00 0x7f37441ef090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f373800e980 tx=0x7f373800ec90 comp rx=0 tx=0).stop 2023-03-03T10:03:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 shutdown_connections 2023-03-03T10:03:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37300682f0 0x7f373006a7a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37440cf0d0 0x7f374414ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37440b3970 0x7f374414cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 --2- 172.21.15.102:0/1495633822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37440ced00 0x7f37441ef090 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.988+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 >> 172.21.15.102:0/1495633822 conn(0x7f374408b1c0 msgr2=0x7f3744071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:03:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.989+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 shutdown_connections 2023-03-03T10:03:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:03:32.989+0000 7f3743fff700 1 -- 172.21.15.102:0/1495633822 wait complete. 2023-03-03T10:03:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:03:33.016 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 118 secs before thrashing 2023-03-03T10:03:34.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:34.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:04.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:04.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:35.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:35.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:05.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:05.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:31.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:05:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.283+0000 7f4bcf7ff700 1 Processor -- start 2023-03-03T10:05:31.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.284+0000 7f4bcf7ff700 1 -- start start 2023-03-03T10:05:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bcf7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4bc80cd8e0 0x7f4bc80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bcf7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd390 0x7f4bc80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bcf7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc8153660 con 0x7f4bc8153290 2023-03-03T10:05:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bcf7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc806c360 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bcf7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc815cea0 con 0x7f4bc80cd390 2023-03-03T10:05:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bccd9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4bc80cd8e0 0x7f4bc80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:31.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bccd9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4bc80cd8e0 0x7f4bc80cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45872/0 (socket says 172.21.15.102:45872) 2023-03-03T10:05:31.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bccd9a700 1 -- 172.21.15.102:0/926873748 learned_addr learned my addr 172.21.15.102:0/926873748 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:05:31.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.285+0000 7f4bcdd9c700 1 --2- 172.21.15.102:0/926873748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd390 0x7f4bc80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:31.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bccd9a700 1 -- 172.21.15.102:0/926873748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd390 msgr2=0x7f4bc80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:05:31.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bccd9a700 1 --2- 172.21.15.102:0/926873748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd390 0x7f4bc80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bccd9a700 1 -- 172.21.15.102:0/926873748 >> v1:172.21.15.172:6789/0 conn(0x7f4bc8153290 legacy=0x7f4bc8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:31.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bccd9a700 1 -- 172.21.15.102:0/926873748 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc806c7f0 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bccd9a700 1 --2- 172.21.15.102:0/926873748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80cd8e0 0x7f4bc80cde20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80098f0 tx=0x7f4bb8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7650daaa1345f4bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:31.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bbffff700 1 -- 172.21.15.102:0/926873748 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011830 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bbffff700 1 -- 172.21.15.102:0/926873748 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8011990 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.286+0000 7f4bbffff700 1 -- 172.21.15.102:0/926873748 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011d20 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/926873748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80cd8e0 msgr2=0x7f4bc80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/926873748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80cd8e0 0x7f4bc80cde20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80098f0 tx=0x7f4bb8009c00 comp rx=0 tx=0).stop 2023-03-03T10:05:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/926873748 shutdown_connections 2023-03-03T10:05:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/926873748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd390 0x7f4bc80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/926873748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc80cd8e0 0x7f4bc80cde20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/926873748 >> 172.21.15.102:0/926873748 conn(0x7f4bc8065fb0 msgr2=0x7f4bc80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:05:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/926873748 shutdown_connections 2023-03-03T10:05:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.287+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/926873748 wait complete. 2023-03-03T10:05:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.289+0000 7f4bcf7ff700 1 Processor -- start 2023-03-03T10:05:31.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.289+0000 7f4bcf7ff700 1 -- start start 2023-03-03T10:05:31.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.290+0000 7f4bcf7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80cd390 0x7f4bc80b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:31.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.290+0000 7f4bcf7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd8e0 0x7f4bc80c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:31.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.290+0000 7f4bcf7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc8153290 0x7f4bc81f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:31.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.290+0000 7f4bcf7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80b4410 con 0x7f4bc8153290 2023-03-03T10:05:31.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.290+0000 7f4bcf7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80b2ac0 con 0x7f4bc80cd390 2023-03-03T10:05:31.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.290+0000 7f4bcf7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80b36c0 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bcd59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80cd390 0x7f4bc80b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:31.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bccd9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd8e0 0x7f4bc80c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:31.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bcdd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc8153290 0x7f4bc81f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bcd59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80cd390 0x7f4bc80b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48126/0 (socket says 172.21.15.102:48126) 2023-03-03T10:05:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bcd59b700 1 -- 172.21.15.102:0/3082937874 learned_addr learned my addr 172.21.15.102:0/3082937874 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:05:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bccd9a700 1 -- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80cd390 msgr2=0x7f4bc80b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bccd9a700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80cd390 0x7f4bc80b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bccd9a700 1 -- 172.21.15.102:0/3082937874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc8153290 msgr2=0x7f4bc81f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bccd9a700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc8153290 0x7f4bc81f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bccd9a700 1 -- 172.21.15.102:0/3082937874 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009550 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bcdd9c700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc8153290 0x7f4bc81f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:05:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.291+0000 7f4bccd9a700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd8e0 0x7f4bc80c6140 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4bc808cc70 tx=0x7f4bb8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.292+0000 7f4bbdffb700 1 -- 172.21.15.102:0/3082937874 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8004510 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.292+0000 7f4bbdffb700 1 -- 172.21.15.102:0/3082937874 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb801e070 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.292+0000 7f4bbdffb700 1 -- 172.21.15.102:0/3082937874 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80038c0 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.292+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc80ccde0 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.292+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc815d160 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.293+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc80cd0c0 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.294+0000 7f4bbdffb700 1 -- 172.21.15.102:0/3082937874 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4bb800f050 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.295+0000 7f4bbdffb700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bac0682f0 0x7f4bac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.296+0000 7f4bbdffb700 1 -- 172.21.15.102:0/3082937874 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4bb8086b80 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.296+0000 7f4bcd59b700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bac0682f0 0x7f4bac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.296+0000 7f4bcd59b700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bac0682f0 0x7f4bac06a7a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4001920 tx=0x7f4bc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:31.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.300+0000 7f4bbdffb700 1 -- 172.21.15.102:0/3082937874 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4bb8053290 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.533+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc804e6e0 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.534+0000 7f4bbdffb700 1 -- 172.21.15.102:0/3082937874 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4bb8020090 con 0x7f4bc80cd8e0 2023-03-03T10:05:31.536 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:05:31.536 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:05:31.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bac0682f0 msgr2=0x7f4bac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:31.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bac0682f0 0x7f4bac06a7a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4001920 tx=0x7f4bc4009040 comp rx=0 tx=0).stop 2023-03-03T10:05:31.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd8e0 msgr2=0x7f4bc80c6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:31.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd8e0 0x7f4bc80c6140 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4bc808cc70 tx=0x7f4bb8004750 comp rx=0 tx=0).stop 2023-03-03T10:05:31.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 shutdown_connections 2023-03-03T10:05:31.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bc8153290 0x7f4bc81f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bc80cd390 0x7f4bc80b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bac0682f0 0x7f4bac06a7a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 --2- 172.21.15.102:0/3082937874 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4bc80cd8e0 0x7f4bc80c6140 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 >> 172.21.15.102:0/3082937874 conn(0x7f4bc8065fb0 msgr2=0x7f4bc80cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:05:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 shutdown_connections 2023-03-03T10:05:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:31.537+0000 7f4bcf7ff700 1 -- 172.21.15.102:0/3082937874 wait complete. 2023-03-03T10:05:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:05:31.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 26 secs before thrashing 2023-03-03T10:05:35.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:35.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:57.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:05:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.812+0000 7f25123ff700 1 Processor -- start 2023-03-03T10:05:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.813+0000 7f25123ff700 1 -- start start 2023-03-03T10:05:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.814+0000 7f25123ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f250c0ca130 0x7f250c15c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.814+0000 7f25123ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f250c0c8eb0 0x7f250c15c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.814+0000 7f25123ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0c9590 con 0x7f250c152e50 2023-03-03T10:05:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.814+0000 7f25123ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250c159880 con 0x7f250c0ca130 2023-03-03T10:05:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.814+0000 7f25123ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250c0c7ae0 con 0x7f250c0c8eb0 2023-03-03T10:05:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.814+0000 7f250b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f250c0ca130 0x7f250c15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f250c0ca130 0x7f250c15c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57800/0 (socket says 172.21.15.102:57800) 2023-03-03T10:05:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250b7fe700 1 -- 172.21.15.102:0/3391044044 learned_addr learned my addr 172.21.15.102:0/3391044044 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:05:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f251099c700 1 --2- 172.21.15.102:0/3391044044 >> v2:172.21.15.172:3300/0 conn(0x7f250c0c8eb0 0x7f250c15c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250b7fe700 1 -- 172.21.15.102:0/3391044044 >> v2:172.21.15.172:3300/0 conn(0x7f250c0c8eb0 msgr2=0x7f250c15c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:05:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250b7fe700 1 --2- 172.21.15.102:0/3391044044 >> v2:172.21.15.172:3300/0 conn(0x7f250c0c8eb0 0x7f250c15c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250b7fe700 1 -- 172.21.15.102:0/3391044044 >> v1:172.21.15.172:6789/0 conn(0x7f250c152e50 legacy=0x7f250c0c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250b7fe700 1 -- 172.21.15.102:0/3391044044 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c0c7dc0 con 0x7f250c0ca130 2023-03-03T10:05:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250affd700 1 -- 172.21.15.102:0/3391044044 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812044939 0 0) 0x7f250c0c9590 con 0x7f250c152e50 2023-03-03T10:05:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.815+0000 7f250b7fe700 1 --2- 172.21.15.102:0/3391044044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c0ca130 0x7f250c15c250 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f250c0c3fe0 tx=0x7f2500009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55216fd9b7e840f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f250affd700 1 -- 172.21.15.102:0/3391044044 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500011830 con 0x7f250c0ca130 2023-03-03T10:05:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f250affd700 1 -- 172.21.15.102:0/3391044044 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2500011990 con 0x7f250c0ca130 2023-03-03T10:05:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f250affd700 1 -- 172.21.15.102:0/3391044044 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500011d20 con 0x7f250c0ca130 2023-03-03T10:05:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 -- 172.21.15.102:0/3391044044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c0ca130 msgr2=0x7f250c15c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 --2- 172.21.15.102:0/3391044044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c0ca130 0x7f250c15c250 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f250c0c3fe0 tx=0x7f2500009bd0 comp rx=0 tx=0).stop 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 -- 172.21.15.102:0/3391044044 shutdown_connections 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 --2- 172.21.15.102:0/3391044044 >> v2:172.21.15.172:3300/0 conn(0x7f250c0c8eb0 0x7f250c15c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 --2- 172.21.15.102:0/3391044044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c0ca130 0x7f250c15c250 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 -- 172.21.15.102:0/3391044044 >> 172.21.15.102:0/3391044044 conn(0x7f250c0704c0 msgr2=0x7f250c0708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 -- 172.21.15.102:0/3391044044 shutdown_connections 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.816+0000 7f25123ff700 1 -- 172.21.15.102:0/3391044044 wait complete. 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.818+0000 7f25123ff700 1 Processor -- start 2023-03-03T10:05:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.819+0000 7f25123ff700 1 -- start start 2023-03-03T10:05:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.819+0000 7f25123ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f250c0c8eb0 0x7f250c1eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.819+0000 7f25123ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f250c0ca130 0x7f250c06c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f25123ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c152e50 0x7f250c14a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f25123ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c1ee980 con 0x7f250c152e50 2023-03-03T10:05:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f25123ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c1f28b0 con 0x7f250c0c8eb0 2023-03-03T10:05:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f25123ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250c1f2350 con 0x7f250c0ca130 2023-03-03T10:05:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c152e50 0x7f250c14a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c152e50 0x7f250c14a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57808/0 (socket says 172.21.15.102:57808) 2023-03-03T10:05:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 -- 172.21.15.102:0/2528345171 learned_addr learned my addr 172.21.15.102:0/2528345171 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:05:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f250b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f250c0ca130 0x7f250c06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f250bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f250c0c8eb0 0x7f250c1eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 -- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f250c0ca130 msgr2=0x7f250c06c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f250c0ca130 0x7f250c06c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 -- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f250c0c8eb0 msgr2=0x7f250c1eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f250c0c8eb0 0x7f250c1eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.820+0000 7f251099c700 1 -- 172.21.15.102:0/2528345171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500009550 con 0x7f250c152e50 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.821+0000 7f251099c700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c152e50 0x7f250c14a7b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2504009dd0 tx=0x7f250400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.821+0000 7f2508ff9700 1 -- 172.21.15.102:0/2528345171 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504009c40 con 0x7f250c152e50 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.821+0000 7f2508ff9700 1 -- 172.21.15.102:0/2528345171 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250400f040 con 0x7f250c152e50 2023-03-03T10:05:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.821+0000 7f2508ff9700 1 -- 172.21.15.102:0/2528345171 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250400ceb0 con 0x7f250c152e50 2023-03-03T10:05:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.821+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c159b60 con 0x7f250c152e50 2023-03-03T10:05:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.821+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c0c80a0 con 0x7f250c152e50 2023-03-03T10:05:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.822+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c0c7690 con 0x7f250c152e50 2023-03-03T10:05:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.824+0000 7f2508ff9700 1 -- 172.21.15.102:0/2528345171 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2504014480 con 0x7f250c152e50 2023-03-03T10:05:57.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.825+0000 7f2508ff9700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24f80682f0 0x7f24f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:57.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.825+0000 7f2508ff9700 1 -- 172.21.15.102:0/2528345171 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f25040866a0 con 0x7f250c152e50 2023-03-03T10:05:57.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.825+0000 7f250bfff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24f80682f0 0x7f24f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.826+0000 7f250bfff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24f80682f0 0x7f24f806a7a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f24fc000f60 tx=0x7f24fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:57.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:57.829+0000 7f2508ff9700 1 -- 172.21.15.102:0/2528345171 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2504052d50 con 0x7f250c152e50 2023-03-03T10:05:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.064+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f250c1eeac0 con 0x7f250c152e50 2023-03-03T10:05:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.064+0000 7f2508ff9700 1 -- 172.21.15.102:0/2528345171 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f250c1eeac0 con 0x7f250c152e50 2023-03-03T10:05:58.065 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:05:58.066 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:05:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24f80682f0 msgr2=0x7f24f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24f80682f0 0x7f24f806a7a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f24fc000f60 tx=0x7f24fc009040 comp rx=0 tx=0).stop 2023-03-03T10:05:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c152e50 msgr2=0x7f250c14a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c152e50 0x7f250c14a7b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2504009dd0 tx=0x7f250400c3a0 comp rx=0 tx=0).stop 2023-03-03T10:05:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 shutdown_connections 2023-03-03T10:05:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250c152e50 0x7f250c14a7b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f250c0c8eb0 0x7f250c1eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24f80682f0 0x7f24f806a7a0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 --2- 172.21.15.102:0/2528345171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f250c0ca130 0x7f250c06c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 >> 172.21.15.102:0/2528345171 conn(0x7f250c0704c0 msgr2=0x7f250c14cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:05:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 shutdown_connections 2023-03-03T10:05:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:05:58.067+0000 7f25123ff700 1 -- 172.21.15.102:0/2528345171 wait complete. 2023-03-03T10:05:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:05:58.095 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T10:06:05.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:05.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:26.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:06:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.345+0000 7f0f8a39d700 1 Processor -- start 2023-03-03T10:06:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.345+0000 7f0f8a39d700 1 -- start start 2023-03-03T10:06:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f8a39d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0f840b3970 0x7f0f840c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f8a39d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f840c41b0 0x7f0f840cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f8a39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f840c3d20 con 0x7f0f840ced00 2023-03-03T10:06:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f8a39d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f840c4930 con 0x7f0f840c41b0 2023-03-03T10:06:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f8a39d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f840cf610 con 0x7f0f840b3970 2023-03-03T10:06:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f89b9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f840c41b0 0x7f0f840cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f8939b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0f840ced00 0x7f0f84158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36756/0 (socket says 172.21.15.102:36756) 2023-03-03T10:06:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f8939b700 1 -- 172.21.15.102:0/2605382989 learned_addr learned my addr 172.21.15.102:0/2605382989 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:06:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.347+0000 7f0f88b9a700 1 --2- 172.21.15.102:0/2605382989 >> v2:172.21.15.172:3300/0 conn(0x7f0f840b3970 0x7f0f840c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:26.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f89b9c700 1 -- 172.21.15.102:0/2605382989 >> v2:172.21.15.172:3300/0 conn(0x7f0f840b3970 msgr2=0x7f0f840c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:06:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f89b9c700 1 --2- 172.21.15.102:0/2605382989 >> v2:172.21.15.172:3300/0 conn(0x7f0f840b3970 0x7f0f840c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f89b9c700 1 -- 172.21.15.102:0/2605382989 >> v1:172.21.15.102:6789/0 conn(0x7f0f840ced00 legacy=0x7f0f84158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f89b9c700 1 -- 172.21.15.102:0/2605382989 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8415cba0 con 0x7f0f840c41b0 2023-03-03T10:06:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f7bfff700 1 -- 172.21.15.102:0/2605382989 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741091535 0 0) 0x7f0f840c3d20 con 0x7f0f840ced00 2023-03-03T10:06:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f89b9c700 1 --2- 172.21.15.102:0/2605382989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840c41b0 0x7f0f840cf0d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0f840c4ba0 tx=0x7f0f7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2079bb23ae1cad8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f7bfff700 1 -- 172.21.15.102:0/2605382989 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00e070 con 0x7f0f840c41b0 2023-03-03T10:06:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f7bfff700 1 -- 172.21.15.102:0/2605382989 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c00bae0 con 0x7f0f840c41b0 2023-03-03T10:06:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.348+0000 7f0f7bfff700 1 -- 172.21.15.102:0/2605382989 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00be90 con 0x7f0f840c41b0 2023-03-03T10:06:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2605382989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840c41b0 msgr2=0x7f0f840cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2605382989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840c41b0 0x7f0f840cf0d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0f840c4ba0 tx=0x7f0f7c00b330 comp rx=0 tx=0).stop 2023-03-03T10:06:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2605382989 shutdown_connections 2023-03-03T10:06:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2605382989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840c41b0 0x7f0f840cf0d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2605382989 >> v2:172.21.15.172:3300/0 conn(0x7f0f840b3970 0x7f0f840c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2605382989 >> 172.21.15.102:0/2605382989 conn(0x7f0f8408b1c0 msgr2=0x7f0f84071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2605382989 shutdown_connections 2023-03-03T10:06:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.349+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2605382989 wait complete. 2023-03-03T10:06:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.352+0000 7f0f8a39d700 1 Processor -- start 2023-03-03T10:06:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.352+0000 7f0f8a39d700 1 -- start start 2023-03-03T10:06:26.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8a39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840b3970 0x7f0f841516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8a39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f840c41b0 0x7f0f8414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8a39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f840ced00 0x7f0f8415ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8a39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f84150c70 con 0x7f0f840b3970 2023-03-03T10:06:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8a39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8415bd70 con 0x7f0f840ced00 2023-03-03T10:06:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8a39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8414e520 con 0x7f0f840c41b0 2023-03-03T10:06:26.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8939b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840b3970 0x7f0f841516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:26.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8939b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840b3970 0x7f0f841516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36716/0 (socket says 172.21.15.102:36716) 2023-03-03T10:06:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8939b700 1 -- 172.21.15.102:0/2719421259 learned_addr learned my addr 172.21.15.102:0/2719421259 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:06:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f89b9c700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f840ced00 0x7f0f8415ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f88b9a700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f840c41b0 0x7f0f8414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8939b700 1 -- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f840c41b0 msgr2=0x7f0f8414d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8939b700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f840c41b0 0x7f0f8414d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8939b700 1 -- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f840ced00 msgr2=0x7f0f8415ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.354+0000 7f0f8939b700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f840ced00 0x7f0f8415ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.355+0000 7f0f8939b700 1 -- 172.21.15.102:0/2719421259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c009a70 con 0x7f0f840b3970 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.355+0000 7f0f8939b700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840b3970 0x7f0f841516f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0f800011e0 tx=0x7f0f8000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.355+0000 7f0f79ffb700 1 -- 172.21.15.102:0/2719421259 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80004c80 con 0x7f0f840b3970 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.355+0000 7f0f79ffb700 1 -- 172.21.15.102:0/2719421259 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f80004e00 con 0x7f0f840b3970 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.355+0000 7f0f79ffb700 1 -- 172.21.15.102:0/2719421259 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80007310 con 0x7f0f840b3970 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.355+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8406bf20 con 0x7f0f840b3970 2023-03-03T10:06:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.355+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8415ce80 con 0x7f0f840b3970 2023-03-03T10:06:26.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.357+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8406c200 con 0x7f0f840b3970 2023-03-03T10:06:26.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.358+0000 7f0f79ffb700 1 -- 172.21.15.102:0/2719421259 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f8001a020 con 0x7f0f840b3970 2023-03-03T10:06:26.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.359+0000 7f0f79ffb700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f680682f0 0x7f0f6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:26.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.359+0000 7f0f79ffb700 1 -- 172.21.15.102:0/2719421259 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0f800858e0 con 0x7f0f840b3970 2023-03-03T10:06:26.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.359+0000 7f0f88b9a700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f680682f0 0x7f0f6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.360+0000 7f0f88b9a700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f680682f0 0x7f0f6806a7a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f0f840b2860 tx=0x7f0f74009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:26.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.363+0000 7f0f79ffb700 1 -- 172.21.15.102:0/2719421259 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f80051f90 con 0x7f0f840b3970 2023-03-03T10:06:26.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.596+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f8404e6e0 con 0x7f0f840b3970 2023-03-03T10:06:26.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.596+0000 7f0f79ffb700 1 -- 172.21.15.102:0/2719421259 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0f800555b0 con 0x7f0f840b3970 2023-03-03T10:06:26.599 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:06:26.599 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:06:26.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f680682f0 msgr2=0x7f0f6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:26.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f680682f0 0x7f0f6806a7a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f0f840b2860 tx=0x7f0f74009210 comp rx=0 tx=0).stop 2023-03-03T10:06:26.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840b3970 msgr2=0x7f0f841516f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:26.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840b3970 0x7f0f841516f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0f800011e0 tx=0x7f0f8000ca30 comp rx=0 tx=0).stop 2023-03-03T10:06:26.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 shutdown_connections 2023-03-03T10:06:26.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f840ced00 0x7f0f8415ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f840b3970 0x7f0f841516f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f680682f0 0x7f0f6806a7a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 --2- 172.21.15.102:0/2719421259 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f840c41b0 0x7f0f8414d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:26.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 >> 172.21.15.102:0/2719421259 conn(0x7f0f8408b1c0 msgr2=0x7f0f84071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:26.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.599+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 shutdown_connections 2023-03-03T10:06:26.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:26.600+0000 7f0f8a39d700 1 -- 172.21.15.102:0/2719421259 wait complete. 2023-03-03T10:06:26.603 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:06:26.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 18 secs before thrashing 2023-03-03T10:06:35.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:35.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:44.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:06:44.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.884+0000 7fa88bfff700 1 Processor -- start 2023-03-03T10:06:44.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.884+0000 7fa88bfff700 1 -- start start 2023-03-03T10:06:44.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa88c0ced00 0x7fa88c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:44.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa88c0b3970 0x7fa88c0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:44.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88bfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa88c0c3d20 con 0x7fa88c0cf0d0 2023-03-03T10:06:44.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88bfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa88c0c4930 con 0x7fa88c0ced00 2023-03-03T10:06:44.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88bfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa88c0cf570 con 0x7fa88c0b3970 2023-03-03T10:06:44.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa88c0ced00 0x7fa88c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:44.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa88c0ced00 0x7fa88c0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34296/0 (socket says 172.21.15.102:34296) 2023-03-03T10:06:44.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88affd700 1 -- 172.21.15.102:0/2931196570 learned_addr learned my addr 172.21.15.102:0/2931196570 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:06:44.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.886+0000 7fa88a7fc700 1 --2- 172.21.15.102:0/2931196570 >> v2:172.21.15.172:3300/0 conn(0x7fa88c0b3970 0x7fa88c0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:44.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa88a7fc700 1 -- 172.21.15.102:0/2931196570 >> v2:172.21.15.102:3300/0 conn(0x7fa88c0ced00 msgr2=0x7fa88c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:06:44.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa88a7fc700 1 --2- 172.21.15.102:0/2931196570 >> v2:172.21.15.102:3300/0 conn(0x7fa88c0ced00 0x7fa88c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:44.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa88a7fc700 1 -- 172.21.15.102:0/2931196570 >> v1:172.21.15.172:6789/0 conn(0x7fa88c0cf0d0 legacy=0x7fa88c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:44.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa88a7fc700 1 -- 172.21.15.102:0/2931196570 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c15cba0 con 0x7fa88c0b3970 2023-03-03T10:06:44.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa889ffb700 1 -- 172.21.15.102:0/2931196570 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366683396 0 0) 0x7fa88c0c3d20 con 0x7fa88c0cf0d0 2023-03-03T10:06:44.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa88a7fc700 1 --2- 172.21.15.102:0/2931196570 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0b3970 0x7fa88c0c41b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa88c0c4ba0 tx=0x7fa880009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ecf3d261469f4c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:44.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa889ffb700 1 -- 172.21.15.102:0/2931196570 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880011830 con 0x7fa88c0b3970 2023-03-03T10:06:44.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa889ffb700 1 -- 172.21.15.102:0/2931196570 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa880011990 con 0x7fa88c0b3970 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.887+0000 7fa889ffb700 1 -- 172.21.15.102:0/2931196570 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880011d20 con 0x7fa88c0b3970 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 -- 172.21.15.102:0/2931196570 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0b3970 msgr2=0x7fa88c0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 --2- 172.21.15.102:0/2931196570 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0b3970 0x7fa88c0c41b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa88c0c4ba0 tx=0x7fa880009b20 comp rx=0 tx=0).stop 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 -- 172.21.15.102:0/2931196570 shutdown_connections 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 --2- 172.21.15.102:0/2931196570 >> v2:172.21.15.102:3300/0 conn(0x7fa88c0ced00 0x7fa88c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 --2- 172.21.15.102:0/2931196570 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0b3970 0x7fa88c0c41b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 -- 172.21.15.102:0/2931196570 >> 172.21.15.102:0/2931196570 conn(0x7fa88c08b1c0 msgr2=0x7fa88c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:44.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 -- 172.21.15.102:0/2931196570 shutdown_connections 2023-03-03T10:06:44.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.888+0000 7fa88bfff700 1 -- 172.21.15.102:0/2931196570 wait complete. 2023-03-03T10:06:44.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.891+0000 7fa88bfff700 1 Processor -- start 2023-03-03T10:06:44.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.891+0000 7fa88bfff700 1 -- start start 2023-03-03T10:06:44.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.892+0000 7fa88bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa88c0b3970 0x7fa88c150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:44.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.892+0000 7fa88bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa88c0ced00 0x7fa88c159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:44.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.892+0000 7fa88bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0cf0d0 0x7fa88c15a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:44.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.892+0000 7fa88bfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c14ed60 con 0x7fa88c0ced00 2023-03-03T10:06:44.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.892+0000 7fa88bfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c15aa70 con 0x7fa88c0cf0d0 2023-03-03T10:06:44.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.892+0000 7fa88bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c14d0e0 con 0x7fa88c0b3970 2023-03-03T10:06:44.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa88c0b3970 0x7fa88c150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:44.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa88c0b3970 0x7fa88c150d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34058/0 (socket says 172.21.15.102:34058) 2023-03-03T10:06:44.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88affd700 1 -- 172.21.15.102:0/3060186328 learned_addr learned my addr 172.21.15.102:0/3060186328 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:06:44.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88a7fc700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa88c0ced00 0x7fa88c159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:44.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88b7fe700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0cf0d0 0x7fa88c15a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:44.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88a7fc700 1 -- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa88c0b3970 msgr2=0x7fa88c150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:44.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88a7fc700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa88c0b3970 0x7fa88c150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:44.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88a7fc700 1 -- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0cf0d0 msgr2=0x7fa88c15a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88a7fc700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0cf0d0 0x7fa88c15a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88a7fc700 1 -- 172.21.15.102:0/3060186328 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8800094a0 con 0x7fa88c0ced00 2023-03-03T10:06:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88affd700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa88c0b3970 0x7fa88c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:06:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.893+0000 7fa88a7fc700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa88c0ced00 0x7fa88c159ff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa880016000 tx=0x7fa880004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.894+0000 7fa873fff700 1 -- 172.21.15.102:0/3060186328 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8800036a0 con 0x7fa88c0ced00 2023-03-03T10:06:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.894+0000 7fa873fff700 1 -- 172.21.15.102:0/3060186328 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88001e070 con 0x7fa88c0ced00 2023-03-03T10:06:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.894+0000 7fa873fff700 1 -- 172.21.15.102:0/3060186328 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880003a60 con 0x7fa88c0ced00 2023-03-03T10:06:44.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.894+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88c06bf20 con 0x7fa88c0ced00 2023-03-03T10:06:44.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.894+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88c15ce80 con 0x7fa88c0ced00 2023-03-03T10:06:44.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.895+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88c06c200 con 0x7fa88c0ced00 2023-03-03T10:06:44.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.896+0000 7fa873fff700 1 -- 172.21.15.102:0/3060186328 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa88000f050 con 0x7fa88c0ced00 2023-03-03T10:06:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.898+0000 7fa873fff700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa8780682f0 0x7fa87806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.898+0000 7fa873fff700 1 -- 172.21.15.102:0/3060186328 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa880086be0 con 0x7fa88c0ced00 2023-03-03T10:06:44.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.899+0000 7fa88affd700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa8780682f0 0x7fa87806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:44.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.899+0000 7fa88affd700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa8780682f0 0x7fa87806a7a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa87c006fd0 tx=0x7fa87c0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:44.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:44.902+0000 7fa873fff700 1 -- 172.21.15.102:0/3060186328 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa8800532f0 con 0x7fa88c0ced00 2023-03-03T10:06:45.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.138+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa88c04e6e0 con 0x7fa88c0ced00 2023-03-03T10:06:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.139+0000 7fa873fff700 1 -- 172.21.15.102:0/3060186328 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa880020090 con 0x7fa88c0ced00 2023-03-03T10:06:45.140 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:06:45.140 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:06:45.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa8780682f0 msgr2=0x7fa87806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa8780682f0 0x7fa87806a7a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa87c006fd0 tx=0x7fa87c0092c0 comp rx=0 tx=0).stop 2023-03-03T10:06:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa88c0ced00 msgr2=0x7fa88c159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa88c0ced00 0x7fa88c159ff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa880016000 tx=0x7fa880004750 comp rx=0 tx=0).stop 2023-03-03T10:06:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 shutdown_connections 2023-03-03T10:06:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa8780682f0 0x7fa87806a7a0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:45.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa88c0b3970 0x7fa88c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa88c0cf0d0 0x7fa88c15a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 --2- 172.21.15.102:0/3060186328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa88c0ced00 0x7fa88c159ff0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.141+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 >> 172.21.15.102:0/3060186328 conn(0x7fa88c08b1c0 msgr2=0x7fa88c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.142+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 shutdown_connections 2023-03-03T10:06:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:06:45.142+0000 7fa88bfff700 1 -- 172.21.15.102:0/3060186328 wait complete. 2023-03-03T10:06:45.144 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:06:45.170 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T10:07:06.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:06.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:36.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:36.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:06.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:06.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:09.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:08:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.418+0000 7f1df9fff700 1 Processor -- start 2023-03-03T10:08:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.418+0000 7f1df9fff700 1 -- start start 2023-03-03T10:08:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df9fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1df4153260 0x7f1df40ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:08:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df9fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df40c9cb0 0x7f1df40cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:08:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df9fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40cf940 con 0x7f1df40cac20 2023-03-03T10:08:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df9fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df40cfe70 con 0x7f1df4153260 2023-03-03T10:08:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df9fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4158d70 con 0x7f1df40c9cb0 2023-03-03T10:08:09.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df37fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1df4153260 0x7f1df40ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:08:09.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1df40cac20 0x7f1df415c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59272/0 (socket says 172.21.15.102:59272) 2023-03-03T10:08:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df3fff700 1 -- 172.21.15.102:0/2256016183 learned_addr learned my addr 172.21.15.102:0/2256016183 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:08:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df40c9cb0 0x7f1df40cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:08:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.420+0000 7f1df37fe700 1 --2- 172.21.15.102:0/2256016183 >> v2:172.21.15.102:3300/0 conn(0x7f1df4153260 0x7f1df40ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33586/0 (socket says 172.21.15.102:33586) 2023-03-03T10:08:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df37fe700 1 -- 172.21.15.102:0/2256016183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df40c9cb0 msgr2=0x7f1df40cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:08:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df37fe700 1 --2- 172.21.15.102:0/2256016183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df40c9cb0 0x7f1df40cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df37fe700 1 -- 172.21.15.102:0/2256016183 >> v1:172.21.15.102:6789/0 conn(0x7f1df40cac20 legacy=0x7f1df415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:08:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df37fe700 1 -- 172.21.15.102:0/2256016183 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df40cfb90 con 0x7f1df4153260 2023-03-03T10:08:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df37fe700 1 --2- 172.21.15.102:0/2256016183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df40ca6e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009840 tx=0x7f1de4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2f3aedd0cf315c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:08:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df27fc700 1 -- 172.21.15.102:0/2256016183 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4011830 con 0x7f1df4153260 2023-03-03T10:08:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df27fc700 1 -- 172.21.15.102:0/2256016183 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de4011990 con 0x7f1df4153260 2023-03-03T10:08:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.421+0000 7f1df27fc700 1 -- 172.21.15.102:0/2256016183 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4011d20 con 0x7f1df4153260 2023-03-03T10:08:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 -- 172.21.15.102:0/2256016183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 msgr2=0x7f1df40ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:08:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 --2- 172.21.15.102:0/2256016183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df40ca6e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009840 tx=0x7f1de4009b50 comp rx=0 tx=0).stop 2023-03-03T10:08:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 -- 172.21.15.102:0/2256016183 shutdown_connections 2023-03-03T10:08:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 --2- 172.21.15.102:0/2256016183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df40ca6e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 --2- 172.21.15.102:0/2256016183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df40c9cb0 0x7f1df40cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 -- 172.21.15.102:0/2256016183 >> 172.21.15.102:0/2256016183 conn(0x7f1df4066c00 msgr2=0x7f1df40ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:08:09.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 -- 172.21.15.102:0/2256016183 shutdown_connections 2023-03-03T10:08:09.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.422+0000 7f1df9fff700 1 -- 172.21.15.102:0/2256016183 wait complete. 2023-03-03T10:08:09.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.425+0000 7f1df9fff700 1 Processor -- start 2023-03-03T10:08:09.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.425+0000 7f1df9fff700 1 -- start start 2023-03-03T10:08:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.426+0000 7f1df9fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1df40cac20 0x7f1df414fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:08:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.426+0000 7f1df9fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df41f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:08:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df9fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df41519e0 0x7f1df414d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df9fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4154a70 con 0x7f1df4153260 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df9fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4154d40 con 0x7f1df40cac20 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df9fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df40c6f80 con 0x7f1df41519e0 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1df40cac20 0x7f1df414fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df41f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df41f1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33598/0 (socket says 172.21.15.102:33598) 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 -- 172.21.15.102:0/1215707841 learned_addr learned my addr 172.21.15.102:0/1215707841 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:08:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df3fff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df41519e0 0x7f1df414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:08:09.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 -- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df41519e0 msgr2=0x7f1df414d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:08:09.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df41519e0 0x7f1df414d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 -- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1df40cac20 msgr2=0x7f1df414fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:08:09.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1df40cac20 0x7f1df414fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.427+0000 7f1df2ffd700 1 -- 172.21.15.102:0/1215707841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de40094a0 con 0x7f1df4153260 2023-03-03T10:08:09.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.428+0000 7f1df2ffd700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df41f1b50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1de800e870 tx=0x7f1de800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:08:09.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.428+0000 7f1ddbfff700 1 -- 172.21.15.102:0/1215707841 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8009590 con 0x7f1df4153260 2023-03-03T10:08:09.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.428+0000 7f1ddbfff700 1 -- 172.21.15.102:0/1215707841 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de8013680 con 0x7f1df4153260 2023-03-03T10:08:09.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.428+0000 7f1ddbfff700 1 -- 172.21.15.102:0/1215707841 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80139f0 con 0x7f1df4153260 2023-03-03T10:08:09.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.428+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df4159260 con 0x7f1df4153260 2023-03-03T10:08:09.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.428+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df4158f80 con 0x7f1df4153260 2023-03-03T10:08:09.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.429+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df4159540 con 0x7f1df4153260 2023-03-03T10:08:09.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.431+0000 7f1ddbfff700 1 -- 172.21.15.102:0/1215707841 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1de8019030 con 0x7f1df4153260 2023-03-03T10:08:09.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.432+0000 7f1ddbfff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de00682f0 0x7f1de006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:08:09.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.432+0000 7f1ddbfff700 1 -- 172.21.15.102:0/1215707841 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1de8085a70 con 0x7f1df4153260 2023-03-03T10:08:09.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.432+0000 7f1df37fe700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de00682f0 0x7f1de006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:08:09.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.433+0000 7f1df37fe700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de00682f0 0x7f1de006a7a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009fd0 tx=0x7f1de4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:08:09.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.436+0000 7f1ddbfff700 1 -- 172.21.15.102:0/1215707841 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1de8052180 con 0x7f1df4153260 2023-03-03T10:08:09.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.670+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1df4154ef0 con 0x7f1df4153260 2023-03-03T10:08:09.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.671+0000 7f1ddbfff700 1 -- 172.21.15.102:0/1215707841 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1df4154ef0 con 0x7f1df4153260 2023-03-03T10:08:09.673 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:08:09.673 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:08:09.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de00682f0 msgr2=0x7f1de006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:08:09.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de00682f0 0x7f1de006a7a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009fd0 tx=0x7f1de4008030 comp rx=0 tx=0).stop 2023-03-03T10:08:09.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 msgr2=0x7f1df41f1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:08:09.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df41f1b50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1de800e870 tx=0x7f1de800eb80 comp rx=0 tx=0).stop 2023-03-03T10:08:09.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 shutdown_connections 2023-03-03T10:08:09.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de00682f0 0x7f1de006a7a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1df41519e0 0x7f1df414d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1df40cac20 0x7f1df414fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 --2- 172.21.15.102:0/1215707841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df4153260 0x7f1df41f1b50 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:08:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 >> 172.21.15.102:0/1215707841 conn(0x7f1df40c9cb0 msgr2=0x7f1df4158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:08:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 shutdown_connections 2023-03-03T10:08:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:08:09.674+0000 7f1df9fff700 1 -- 172.21.15.102:0/1215707841 wait complete. 2023-03-03T10:08:09.678 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:08:09.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-03-03T10:08:36.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:36.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:06.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:06.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:36.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:36.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:38.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:09:38.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.951+0000 7f0e645ff700 1 Processor -- start 2023-03-03T10:09:38.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.951+0000 7f0e645ff700 1 -- start start 2023-03-03T10:09:38.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.952+0000 7f0e645ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0e5c153290 0x7f0e5c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:38.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.952+0000 7f0e645ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0e5c1593c0 0x7f0e5c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:38.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.952+0000 7f0e645ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c0cd8e0 0x7f0e5c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:38.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.952+0000 7f0e645ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c153660 con 0x7f0e5c1593c0 2023-03-03T10:09:38.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.952+0000 7f0e645ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c06c360 con 0x7f0e5c153290 2023-03-03T10:09:38.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.952+0000 7f0e645ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c15cea0 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e61b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0e5c1593c0 0x7f0e5c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:38.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e61b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0e5c1593c0 0x7f0e5c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59240/0 (socket says 172.21.15.102:59240) 2023-03-03T10:09:38.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e62b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c0cd8e0 0x7f0e5c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:38.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e61b9a700 1 -- 172.21.15.102:0/3601493857 learned_addr learned my addr 172.21.15.102:0/3601493857 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:09:38.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e6239b700 1 --2- 172.21.15.102:0/3601493857 >> v2:172.21.15.172:3300/0 conn(0x7f0e5c153290 0x7f0e5c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:38.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e62b9c700 1 -- 172.21.15.102:0/3601493857 >> v2:172.21.15.172:3300/0 conn(0x7f0e5c153290 msgr2=0x7f0e5c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:09:38.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e62b9c700 1 --2- 172.21.15.102:0/3601493857 >> v2:172.21.15.172:3300/0 conn(0x7f0e5c153290 0x7f0e5c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:38.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e62b9c700 1 -- 172.21.15.102:0/3601493857 >> v2:172.21.15.102:3300/0 conn(0x7f0e5c1593c0 msgr2=0x7f0e5c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:09:38.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e62b9c700 1 --2- 172.21.15.102:0/3601493857 >> v2:172.21.15.102:3300/0 conn(0x7f0e5c1593c0 0x7f0e5c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:38.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e62b9c700 1 -- 172.21.15.102:0/3601493857 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c06c7f0 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.953+0000 7f0e61b9a700 1 --2- 172.21.15.102:0/3601493857 >> v2:172.21.15.102:3300/0 conn(0x7f0e5c1593c0 0x7f0e5c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:09:38.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.954+0000 7f0e62b9c700 1 --2- 172.21.15.102:0/3601493857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c0cd8e0 0x7f0e5c0ce360 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c08cc70 tx=0x7f0e54009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38928a43571d3f57 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:38.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.954+0000 7f0e61399700 1 -- 172.21.15.102:0/3601493857 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54011830 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.954+0000 7f0e61399700 1 -- 172.21.15.102:0/3601493857 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54011990 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.954+0000 7f0e61399700 1 -- 172.21.15.102:0/3601493857 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54011d20 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.954+0000 7f0e645ff700 1 -- 172.21.15.102:0/3601493857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c0cd8e0 msgr2=0x7f0e5c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:38.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.954+0000 7f0e645ff700 1 --2- 172.21.15.102:0/3601493857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c0cd8e0 0x7f0e5c0ce360 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c08cc70 tx=0x7f0e54009b20 comp rx=0 tx=0).stop 2023-03-03T10:09:38.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.955+0000 7f0e645ff700 1 -- 172.21.15.102:0/3601493857 shutdown_connections 2023-03-03T10:09:38.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.955+0000 7f0e645ff700 1 --2- 172.21.15.102:0/3601493857 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c0cd8e0 0x7f0e5c0ce360 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:38.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.955+0000 7f0e645ff700 1 --2- 172.21.15.102:0/3601493857 >> v2:172.21.15.172:3300/0 conn(0x7f0e5c153290 0x7f0e5c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:38.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.955+0000 7f0e645ff700 1 --2- 172.21.15.102:0/3601493857 >> v2:172.21.15.102:3300/0 conn(0x7f0e5c1593c0 0x7f0e5c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:38.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.955+0000 7f0e645ff700 1 -- 172.21.15.102:0/3601493857 >> 172.21.15.102:0/3601493857 conn(0x7f0e5c065fb0 msgr2=0x7f0e5c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:38.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.955+0000 7f0e645ff700 1 -- 172.21.15.102:0/3601493857 shutdown_connections 2023-03-03T10:09:38.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.955+0000 7f0e645ff700 1 -- 172.21.15.102:0/3601493857 wait complete. 2023-03-03T10:09:38.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.957+0000 7f0e645ff700 1 Processor -- start 2023-03-03T10:09:38.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.957+0000 7f0e645ff700 1 -- start start 2023-03-03T10:09:38.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e645ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e5c0cd8e0 0x7f0e5c1e5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:38.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e645ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c153290 0x7f0e5c1efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:38.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e645ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e5c1593c0 0x7f0e5c1f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:38.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e645ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c14ea70 con 0x7f0e5c1593c0 2023-03-03T10:09:38.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e645ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b4b50 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e645ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1e4220 con 0x7f0e5c153290 2023-03-03T10:09:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e6239b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e5c0cd8e0 0x7f0e5c1e5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e6239b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e5c0cd8e0 0x7f0e5c1e5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59102/0 (socket says 172.21.15.102:59102) 2023-03-03T10:09:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e6239b700 1 -- 172.21.15.102:0/1863668110 learned_addr learned my addr 172.21.15.102:0/1863668110 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:09:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e61b9a700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c153290 0x7f0e5c1efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.958+0000 7f0e62b9c700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e5c1593c0 0x7f0e5c1f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e6239b700 1 -- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c153290 msgr2=0x7f0e5c1efbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e6239b700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c153290 0x7f0e5c1efbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e6239b700 1 -- 172.21.15.102:0/1863668110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e5c1593c0 msgr2=0x7f0e5c1f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e6239b700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e5c1593c0 0x7f0e5c1f0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e6239b700 1 -- 172.21.15.102:0/1863668110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c009550 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e6239b700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e5c0cd8e0 0x7f0e5c1e5be0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009d10 tx=0x7f0e5800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e52ffd700 1 -- 172.21.15.102:0/1863668110 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5800cd10 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e52ffd700 1 -- 172.21.15.102:0/1863668110 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5800f040 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e540094a0 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e52ffd700 1 -- 172.21.15.102:0/1863668110 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e580146a0 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.959+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c0ccde0 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.960+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c15d160 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.962+0000 7f0e52ffd700 1 -- 172.21.15.102:0/1863668110 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0e58014800 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.963+0000 7f0e52ffd700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e400682f0 0x7f0e4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:38.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.963+0000 7f0e52ffd700 1 -- 172.21.15.102:0/1863668110 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0e58086940 con 0x7f0e5c0cd8e0 2023-03-03T10:09:38.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.964+0000 7f0e61b9a700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e400682f0 0x7f0e4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:38.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.964+0000 7f0e61b9a700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e400682f0 0x7f0e4006a7a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c156560 tx=0x7f0e4c0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:38.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:38.966+0000 7f0e52ffd700 1 -- 172.21.15.102:0/1863668110 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e58053040 con 0x7f0e5c0cd8e0 2023-03-03T10:09:39.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.201+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e5c0cd0c0 con 0x7f0e5c0cd8e0 2023-03-03T10:09:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.203+0000 7f0e52ffd700 1 -- 172.21.15.102:0/1863668110 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0e58009e60 con 0x7f0e5c0cd8e0 2023-03-03T10:09:39.204 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:09:39.204 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:09:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e400682f0 msgr2=0x7f0e4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e400682f0 0x7f0e4006a7a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c156560 tx=0x7f0e4c0092c0 comp rx=0 tx=0).stop 2023-03-03T10:09:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e5c0cd8e0 msgr2=0x7f0e5c1e5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e5c0cd8e0 0x7f0e5c1e5be0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009d10 tx=0x7f0e5800c3a0 comp rx=0 tx=0).stop 2023-03-03T10:09:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 shutdown_connections 2023-03-03T10:09:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e5c1593c0 0x7f0e5c1f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e5c0cd8e0 0x7f0e5c1e5be0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0e400682f0 0x7f0e4006a7a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 --2- 172.21.15.102:0/1863668110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0e5c153290 0x7f0e5c1efbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 >> 172.21.15.102:0/1863668110 conn(0x7f0e5c065fb0 msgr2=0x7f0e5c150ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 shutdown_connections 2023-03-03T10:09:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:09:39.205+0000 7f0e645ff700 1 -- 172.21.15.102:0/1863668110 wait complete. 2023-03-03T10:09:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:09:39.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T10:10:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T10:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:10:07.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:07.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:11.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:10:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.483+0000 7f22531ff700 1 Processor -- start 2023-03-03T10:10:11.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.483+0000 7f22531ff700 1 -- start start 2023-03-03T10:10:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f22531ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c153290 0x7f224c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f22531ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f224c0cd8e0 0x7f224c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f22531ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c153660 con 0x7f224c1593c0 2023-03-03T10:10:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f22531ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224c06c360 con 0x7f224c0cd8e0 2023-03-03T10:10:11.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f22531ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c15cea0 con 0x7f224c153290 2023-03-03T10:10:11.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f224bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f224c1593c0 0x7f224c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45976/0 (socket says 172.21.15.102:45976) 2023-03-03T10:10:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f224bfff700 1 -- 172.21.15.102:0/1013355159 learned_addr learned my addr 172.21.15.102:0/1013355159 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:10:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f2250f9b700 1 --2- 172.21.15.102:0/1013355159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c153290 0x7f224c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f225179c700 1 --2- 172.21.15.102:0/1013355159 >> v2:172.21.15.172:3300/0 conn(0x7f224c0cd8e0 0x7f224c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.485+0000 7f224b7fe700 1 -- 172.21.15.102:0/1013355159 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80606600 0 0) 0x7f224c153660 con 0x7f224c1593c0 2023-03-03T10:10:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f224b7fe700 1 -- 172.21.15.102:0/1013355159 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c0035b0 con 0x7f224c1593c0 2023-03-03T10:10:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f2250f9b700 1 -- 172.21.15.102:0/1013355159 >> v2:172.21.15.172:3300/0 conn(0x7f224c0cd8e0 msgr2=0x7f224c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:10:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f2250f9b700 1 --2- 172.21.15.102:0/1013355159 >> v2:172.21.15.172:3300/0 conn(0x7f224c0cd8e0 0x7f224c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f2250f9b700 1 -- 172.21.15.102:0/1013355159 >> v1:172.21.15.102:6789/0 conn(0x7f224c1593c0 legacy=0x7f224c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f2250f9b700 1 -- 172.21.15.102:0/1013355159 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c06c7f0 con 0x7f224c153290 2023-03-03T10:10:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f225179c700 1 --2- 172.21.15.102:0/1013355159 >> v2:172.21.15.172:3300/0 conn(0x7f224c0cd8e0 0x7f224c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:10:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f2250f9b700 1 --2- 172.21.15.102:0/1013355159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c153290 0x7f224c158e80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2240009840 tx=0x7f2240009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f8a023a3d08f66a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f224b7fe700 1 -- 172.21.15.102:0/1013355159 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240011830 con 0x7f224c153290 2023-03-03T10:10:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f224b7fe700 1 -- 172.21.15.102:0/1013355159 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2240011990 con 0x7f224c153290 2023-03-03T10:10:11.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.486+0000 7f224b7fe700 1 -- 172.21.15.102:0/1013355159 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240011d20 con 0x7f224c153290 2023-03-03T10:10:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 -- 172.21.15.102:0/1013355159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c153290 msgr2=0x7f224c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 --2- 172.21.15.102:0/1013355159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c153290 0x7f224c158e80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2240009840 tx=0x7f2240009b50 comp rx=0 tx=0).stop 2023-03-03T10:10:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 -- 172.21.15.102:0/1013355159 shutdown_connections 2023-03-03T10:10:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 --2- 172.21.15.102:0/1013355159 >> v2:172.21.15.172:3300/0 conn(0x7f224c0cd8e0 0x7f224c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 --2- 172.21.15.102:0/1013355159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c153290 0x7f224c158e80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 -- 172.21.15.102:0/1013355159 >> 172.21.15.102:0/1013355159 conn(0x7f224c065fb0 msgr2=0x7f224c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 -- 172.21.15.102:0/1013355159 shutdown_connections 2023-03-03T10:10:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.487+0000 7f22531ff700 1 -- 172.21.15.102:0/1013355159 wait complete. 2023-03-03T10:10:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.489+0000 7f22531ff700 1 Processor -- start 2023-03-03T10:10:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.489+0000 7f22531ff700 1 -- start start 2023-03-03T10:10:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.490+0000 7f22531ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c0cd8e0 0x7f224c14e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.490+0000 7f22531ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f224c153290 0x7f224c1ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.490+0000 7f22531ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224c1593c0 0x7f224c1eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.490+0000 7f22531ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0c6b50 con 0x7f224c153290 2023-03-03T10:10:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.490+0000 7f22531ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c14cf60 con 0x7f224c1593c0 2023-03-03T10:10:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.490+0000 7f22531ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0c6580 con 0x7f224c0cd8e0 2023-03-03T10:10:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.490+0000 7f2250f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c0cd8e0 0x7f224c14e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f224bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f224c153290 0x7f224c1ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c0cd8e0 0x7f224c14e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:49564/0 (socket says 172.21.15.102:49564) 2023-03-03T10:10:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 -- 172.21.15.102:0/1794694212 learned_addr learned my addr 172.21.15.102:0/1794694212 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:10:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f225179c700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224c1593c0 0x7f224c1eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 -- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224c1593c0 msgr2=0x7f224c1eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:11.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224c1593c0 0x7f224c1eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 -- 172.21.15.102:0/1794694212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f224c153290 msgr2=0x7f224c1ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f224c153290 0x7f224c1ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 -- 172.21.15.102:0/1794694212 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2244009a70 con 0x7f224c0cd8e0 2023-03-03T10:10:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.491+0000 7f2250f9b700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c0cd8e0 0x7f224c14e9b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2240000c00 tx=0x7f2240004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.492+0000 7f22497fa700 1 -- 172.21.15.102:0/1794694212 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240004550 con 0x7f224c0cd8e0 2023-03-03T10:10:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.492+0000 7f22497fa700 1 -- 172.21.15.102:0/1794694212 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224001f070 con 0x7f224c0cd8e0 2023-03-03T10:10:11.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.492+0000 7f22497fa700 1 -- 172.21.15.102:0/1794694212 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22400038c0 con 0x7f224c0cd8e0 2023-03-03T10:10:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.492+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22400094a0 con 0x7f224c0cd8e0 2023-03-03T10:10:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.492+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c0ccde0 con 0x7f224c0cd8e0 2023-03-03T10:10:11.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.493+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c15d160 con 0x7f224c0cd8e0 2023-03-03T10:10:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.494+0000 7f22497fa700 1 -- 172.21.15.102:0/1794694212 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f224000f050 con 0x7f224c0cd8e0 2023-03-03T10:10:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.495+0000 7f22497fa700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22300682f0 0x7f223006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.495+0000 7f22497fa700 1 -- 172.21.15.102:0/1794694212 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2240021070 con 0x7f224c0cd8e0 2023-03-03T10:10:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.496+0000 7f224bfff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22300682f0 0x7f223006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:11.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.496+0000 7f224bfff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22300682f0 0x7f223006a7a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f224c073c90 tx=0x7f2238007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.500+0000 7f22497fa700 1 -- 172.21.15.102:0/1794694212 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2240053d10 con 0x7f224c0cd8e0 2023-03-03T10:10:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.733+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f224c0cd0c0 con 0x7f224c0cd8e0 2023-03-03T10:10:11.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.735+0000 7f22497fa700 1 -- 172.21.15.102:0/1794694212 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2240057330 con 0x7f224c0cd8e0 2023-03-03T10:10:11.735 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:10:11.736 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:10:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22300682f0 msgr2=0x7f223006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22300682f0 0x7f223006a7a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f224c073c90 tx=0x7f2238007dc0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c0cd8e0 msgr2=0x7f224c14e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c0cd8e0 0x7f224c14e9b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2240000c00 tx=0x7f2240004750 comp rx=0 tx=0).stop 2023-03-03T10:10:11.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 shutdown_connections 2023-03-03T10:10:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f22300682f0 0x7f223006a7a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f224c0cd8e0 0x7f224c14e9b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f224c1593c0 0x7f224c1eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 --2- 172.21.15.102:0/1794694212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f224c153290 0x7f224c1ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 >> 172.21.15.102:0/1794694212 conn(0x7f224c065fb0 msgr2=0x7f224c0cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 shutdown_connections 2023-03-03T10:10:11.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:11.737+0000 7f22531ff700 1 -- 172.21.15.102:0/1794694212 wait complete. 2023-03-03T10:10:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:10:11.765 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-03-03T10:10:37.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:37.348 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:44.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:10:45.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.016+0000 7f959b1ff700 1 Processor -- start 2023-03-03T10:10:45.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.016+0000 7f959b1ff700 1 -- start start 2023-03-03T10:10:45.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.017+0000 7f959b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9594153290 0x7f9594158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.017+0000 7f959b1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f95941593c0 0x7f95940cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.017+0000 7f959b1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95940cd8e0 0x7f95940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.017+0000 7f959b1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9594153660 con 0x7f95940cd8e0 2023-03-03T10:10:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.017+0000 7f959b1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f959406c360 con 0x7f95941593c0 2023-03-03T10:10:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.017+0000 7f959b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959415cea0 con 0x7f9594153290 2023-03-03T10:10:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95940cd8e0 0x7f95940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f9593fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f95941593c0 0x7f95940cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f9598f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9594153290 0x7f9594158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95940cd8e0 0x7f95940ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58026/0 (socket says 172.21.15.102:58026) 2023-03-03T10:10:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 -- 172.21.15.102:0/4234766877 learned_addr learned my addr 172.21.15.102:0/4234766877 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:10:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 -- 172.21.15.102:0/4234766877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9594153290 msgr2=0x7f9594158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:10:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 --2- 172.21.15.102:0/4234766877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9594153290 0x7f9594158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 -- 172.21.15.102:0/4234766877 >> v2:172.21.15.172:3300/0 conn(0x7f95941593c0 msgr2=0x7f95940cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:10:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 --2- 172.21.15.102:0/4234766877 >> v2:172.21.15.172:3300/0 conn(0x7f95941593c0 0x7f95940cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 -- 172.21.15.102:0/4234766877 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959406c7f0 con 0x7f95940cd8e0 2023-03-03T10:10:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.018+0000 7f959979c700 1 --2- 172.21.15.102:0/4234766877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95940cd8e0 0x7f95940ce360 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f95940b1970 tx=0x7f958c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1712aeed3b520d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f95937fe700 1 -- 172.21.15.102:0/4234766877 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c00e070 con 0x7f95940cd8e0 2023-03-03T10:10:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f95937fe700 1 -- 172.21.15.102:0/4234766877 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958c00b910 con 0x7f95940cd8e0 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f95937fe700 1 -- 172.21.15.102:0/4234766877 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c00bcc0 con 0x7f95940cd8e0 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f959b1ff700 1 -- 172.21.15.102:0/4234766877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95940cd8e0 msgr2=0x7f95940ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f959b1ff700 1 --2- 172.21.15.102:0/4234766877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95940cd8e0 0x7f95940ce360 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f95940b1970 tx=0x7f958c00b330 comp rx=0 tx=0).stop 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f959b1ff700 1 -- 172.21.15.102:0/4234766877 shutdown_connections 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f959b1ff700 1 --2- 172.21.15.102:0/4234766877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95940cd8e0 0x7f95940ce360 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f959b1ff700 1 --2- 172.21.15.102:0/4234766877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9594153290 0x7f9594158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f959b1ff700 1 --2- 172.21.15.102:0/4234766877 >> v2:172.21.15.172:3300/0 conn(0x7f95941593c0 0x7f95940cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.019+0000 7f959b1ff700 1 -- 172.21.15.102:0/4234766877 >> 172.21.15.102:0/4234766877 conn(0x7f9594065fb0 msgr2=0x7f95940663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:45.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.020+0000 7f959b1ff700 1 -- 172.21.15.102:0/4234766877 shutdown_connections 2023-03-03T10:10:45.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.020+0000 7f959b1ff700 1 -- 172.21.15.102:0/4234766877 wait complete. 2023-03-03T10:10:45.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.022+0000 7f959b1ff700 1 Processor -- start 2023-03-03T10:10:45.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.022+0000 7f959b1ff700 1 -- start start 2023-03-03T10:10:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f959b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95940cd8e0 0x7f95941548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f959b1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9594153290 0x7f95940b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f959b1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95941593c0 0x7f95940cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f959b1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95940b4c20 con 0x7f9594153290 2023-03-03T10:10:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f959b1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95940b2ed0 con 0x7f95941593c0 2023-03-03T10:10:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f959b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9594154df0 con 0x7f95940cd8e0 2023-03-03T10:10:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9594153290 0x7f95940b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9594153290 0x7f95940b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58040/0 (socket says 172.21.15.102:58040) 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 -- 172.21.15.102:0/1012746995 learned_addr learned my addr 172.21.15.102:0/1012746995 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f959979c700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95941593c0 0x7f95940cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9598f9b700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95940cd8e0 0x7f95941548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 -- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95940cd8e0 msgr2=0x7f95941548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95940cd8e0 0x7f95941548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 -- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95941593c0 msgr2=0x7f95940cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95941593c0 0x7f95940cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.023+0000 7f9593fff700 1 -- 172.21.15.102:0/1012746995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c009b30 con 0x7f9594153290 2023-03-03T10:10:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.024+0000 7f9593fff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9594153290 0x7f95940b3360 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f958000e9b0 tx=0x7f958000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.024+0000 7f95917fa700 1 -- 172.21.15.102:0/1012746995 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580009710 con 0x7f9594153290 2023-03-03T10:10:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.024+0000 7f95917fa700 1 -- 172.21.15.102:0/1012746995 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000ee50 con 0x7f9594153290 2023-03-03T10:10:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.024+0000 7f95917fa700 1 -- 172.21.15.102:0/1012746995 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000ca50 con 0x7f9594153290 2023-03-03T10:10:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.024+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95940ccde0 con 0x7f9594153290 2023-03-03T10:10:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.024+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959415d160 con 0x7f9594153290 2023-03-03T10:10:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.025+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95940cd0c0 con 0x7f9594153290 2023-03-03T10:10:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.027+0000 7f95917fa700 1 -- 172.21.15.102:0/1012746995 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9580019030 con 0x7f9594153290 2023-03-03T10:10:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.028+0000 7f95917fa700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9578068320 0x7f957806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.028+0000 7f95917fa700 1 -- 172.21.15.102:0/1012746995 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9580085bc0 con 0x7f9594153290 2023-03-03T10:10:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.028+0000 7f9598f9b700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9578068320 0x7f957806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.029+0000 7f9598f9b700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9578068320 0x7f957806a7d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9588009b40 tx=0x7f9588009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:45.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.032+0000 7f95917fa700 1 -- 172.21.15.102:0/1012746995 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9580052270 con 0x7f9594153290 2023-03-03T10:10:45.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.267+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95940b38a0 con 0x7f9594153290 2023-03-03T10:10:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.267+0000 7f95917fa700 1 -- 172.21.15.102:0/1012746995 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9580055890 con 0x7f9594153290 2023-03-03T10:10:45.269 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:10:45.269 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:10:45.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9578068320 msgr2=0x7f957806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9578068320 0x7f957806a7d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9588009b40 tx=0x7f9588009210 comp rx=0 tx=0).stop 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9594153290 msgr2=0x7f95940b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9594153290 0x7f95940b3360 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f958000e9b0 tx=0x7f958000ecc0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 shutdown_connections 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9578068320 0x7f957806a7d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95940cd8e0 0x7f95941548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95941593c0 0x7f95940cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 --2- 172.21.15.102:0/1012746995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9594153290 0x7f95940b3360 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 >> 172.21.15.102:0/1012746995 conn(0x7f9594065fb0 msgr2=0x7f95940caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:45.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 shutdown_connections 2023-03-03T10:10:45.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:10:45.270+0000 7f959b1ff700 1 -- 172.21.15.102:0/1012746995 wait complete. 2023-03-03T10:10:45.273 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:10:45.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2023-03-03T10:11:07.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:07.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:37.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:37.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:07.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:07.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:30.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:12:30.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.547+0000 7f8b86bff700 1 Processor -- start 2023-03-03T10:12:30.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.547+0000 7f8b86bff700 1 -- start start 2023-03-03T10:12:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.548+0000 7f8b86bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b800cd8e0 0x7f8b800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.548+0000 7f8b86bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b80153660 con 0x7f8b80153290 2023-03-03T10:12:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.548+0000 7f8b86bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8006c360 con 0x7f8b800cd390 2023-03-03T10:12:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.549+0000 7f8b86bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8015cea0 con 0x7f8b800cd8e0 2023-03-03T10:12:30.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.549+0000 7f8b77fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b800cd8e0 0x7f8b800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:30.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.549+0000 7f8b8499b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b80153290 0x7f8b80158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51040/0 (socket says 172.21.15.102:51040) 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.549+0000 7f8b8499b700 1 -- 172.21.15.102:0/1340069594 learned_addr learned my addr 172.21.15.102:0/1340069594 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.549+0000 7f8b77fff700 1 -- 172.21.15.102:0/1340069594 >> v1:172.21.15.172:6789/0 conn(0x7f8b800cd390 legacy=0x7f8b800ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.549+0000 7f8b77fff700 1 -- 172.21.15.102:0/1340069594 >> v1:172.21.15.102:6789/0 conn(0x7f8b80153290 legacy=0x7f8b80158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.549+0000 7f8b77fff700 1 -- 172.21.15.102:0/1340069594 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8006c7f0 con 0x7f8b800cd8e0 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.550+0000 7f8b77fff700 1 --2- 172.21.15.102:0/1340069594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b800cd8e0 0x7f8b800cde20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009840 tx=0x7f8b70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19bf1313b7ff1bd8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.550+0000 7f8b777fe700 1 -- 172.21.15.102:0/1340069594 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70011830 con 0x7f8b800cd8e0 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.550+0000 7f8b777fe700 1 -- 172.21.15.102:0/1340069594 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b70011990 con 0x7f8b800cd8e0 2023-03-03T10:12:30.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.550+0000 7f8b777fe700 1 -- 172.21.15.102:0/1340069594 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70011d20 con 0x7f8b800cd8e0 2023-03-03T10:12:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.550+0000 7f8b86bff700 1 -- 172.21.15.102:0/1340069594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b800cd8e0 msgr2=0x7f8b800cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.550+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1340069594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b800cd8e0 0x7f8b800cde20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009840 tx=0x7f8b70009b50 comp rx=0 tx=0).stop 2023-03-03T10:12:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.551+0000 7f8b86bff700 1 -- 172.21.15.102:0/1340069594 shutdown_connections 2023-03-03T10:12:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.551+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1340069594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b800cd8e0 0x7f8b800cde20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.551+0000 7f8b86bff700 1 -- 172.21.15.102:0/1340069594 >> 172.21.15.102:0/1340069594 conn(0x7f8b80065fb0 msgr2=0x7f8b800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:12:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.551+0000 7f8b86bff700 1 -- 172.21.15.102:0/1340069594 shutdown_connections 2023-03-03T10:12:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.551+0000 7f8b86bff700 1 -- 172.21.15.102:0/1340069594 wait complete. 2023-03-03T10:12:30.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.553+0000 7f8b86bff700 1 Processor -- start 2023-03-03T10:12:30.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.554+0000 7f8b86bff700 1 -- start start 2023-03-03T10:12:30.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.555+0000 7f8b86bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b800cd390 0x7f8b800c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.555+0000 7f8b86bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b800cd8e0 0x7f8b801f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.555+0000 7f8b86bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b80153290 0x7f8b800b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.555+0000 7f8b86bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b800c59b0 con 0x7f8b80153290 2023-03-03T10:12:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.555+0000 7f8b86bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b801ef650 con 0x7f8b800cd8e0 2023-03-03T10:12:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.555+0000 7f8b86bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b801f2e60 con 0x7f8b800cd390 2023-03-03T10:12:30.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.555+0000 7f8b8519c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b80153290 0x7f8b800b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b800cd390 0x7f8b800c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b77fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b800cd8e0 0x7f8b801f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b800cd390 0x7f8b800c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:52910/0 (socket says 172.21.15.102:52910) 2023-03-03T10:12:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 -- 172.21.15.102:0/1154679344 learned_addr learned my addr 172.21.15.102:0/1154679344 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:12:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 -- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b800cd8e0 msgr2=0x7f8b801f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b800cd8e0 0x7f8b801f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 -- 172.21.15.102:0/1154679344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b80153290 msgr2=0x7f8b800b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b80153290 0x7f8b800b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 -- 172.21.15.102:0/1154679344 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b700094a0 con 0x7f8b800cd390 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8519c700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b80153290 0x7f8b800b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.556+0000 7f8b8499b700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b800cd390 0x7f8b800c5470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c4b0 tx=0x7f8b7c00c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.557+0000 7f8b757fa700 1 -- 172.21.15.102:0/1154679344 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c005d70 con 0x7f8b800cd390 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.557+0000 7f8b757fa700 1 -- 172.21.15.102:0/1154679344 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c00ce80 con 0x7f8b800cd390 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.557+0000 7f8b757fa700 1 -- 172.21.15.102:0/1154679344 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c0072d0 con 0x7f8b800cd390 2023-03-03T10:12:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.557+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b800ccde0 con 0x7f8b800cd390 2023-03-03T10:12:30.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.557+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8015d160 con 0x7f8b800cd390 2023-03-03T10:12:30.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.558+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b800cd0c0 con 0x7f8b800cd390 2023-03-03T10:12:30.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.559+0000 7f8b757fa700 1 -- 172.21.15.102:0/1154679344 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b7c019030 con 0x7f8b800cd390 2023-03-03T10:12:30.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.560+0000 7f8b757fa700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b6c068320 0x7f8b6c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:30.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.561+0000 7f8b757fa700 1 -- 172.21.15.102:0/1154679344 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8b7c084f00 con 0x7f8b800cd390 2023-03-03T10:12:30.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.561+0000 7f8b77fff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b6c068320 0x7f8b6c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:30.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.562+0000 7f8b77fff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b6c068320 0x7f8b6c06a7d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8b700090b0 tx=0x7f8b70008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:12:30.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.565+0000 7f8b757fa700 1 -- 172.21.15.102:0/1154679344 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b7c04b5e0 con 0x7f8b800cd390 2023-03-03T10:12:30.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.797+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b8004e6e0 con 0x7f8b800cd390 2023-03-03T10:12:30.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.798+0000 7f8b757fa700 1 -- 172.21.15.102:0/1154679344 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8b7c017090 con 0x7f8b800cd390 2023-03-03T10:12:30.800 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:12:30.801 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:12:30.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.800+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b6c068320 msgr2=0x7f8b6c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:30.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.800+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b6c068320 0x7f8b6c06a7d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8b700090b0 tx=0x7f8b70008030 comp rx=0 tx=0).stop 2023-03-03T10:12:30.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.800+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b800cd390 msgr2=0x7f8b800c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:30.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.800+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b800cd390 0x7f8b800c5470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c4b0 tx=0x7f8b7c00c7c0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 shutdown_connections 2023-03-03T10:12:30.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b6c068320 0x7f8b6c06a7d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b800cd390 0x7f8b800c5470 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b80153290 0x7f8b800b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 --2- 172.21.15.102:0/1154679344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b800cd8e0 0x7f8b801f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:30.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 >> 172.21.15.102:0/1154679344 conn(0x7f8b80065fb0 msgr2=0x7f8b800b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:12:30.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 shutdown_connections 2023-03-03T10:12:30.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:12:30.801+0000 7f8b86bff700 1 -- 172.21.15.102:0/1154679344 wait complete. 2023-03-03T10:12:30.805 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:12:30.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-03-03T10:12:38.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:38.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:08.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:08.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:38.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:38.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:08.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:08.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:17.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:14:18.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.082+0000 7f67989ff700 1 Processor -- start 2023-03-03T10:14:18.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.082+0000 7f67989ff700 1 -- start start 2023-03-03T10:14:18.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.083+0000 7f67989ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6794153290 0x7f6794158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:18.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.083+0000 7f67989ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f67941593c0 0x7f67940cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:18.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.083+0000 7f67989ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6794153660 con 0x7f67940cd8e0 2023-03-03T10:14:18.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f67989ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679406c360 con 0x7f67941593c0 2023-03-03T10:14:18.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f67989ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679415cea0 con 0x7f6794153290 2023-03-03T10:14:18.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f6791d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f67941593c0 0x7f67940cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:18.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f679259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6794153290 0x7f6794158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:18.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f6792d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f67940cd8e0 0x7f67940ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35982/0 (socket says 172.21.15.102:35982) 2023-03-03T10:14:18.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f6792d9d700 1 -- 172.21.15.102:0/2252090255 learned_addr learned my addr 172.21.15.102:0/2252090255 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:14:18.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f679259c700 1 -- 172.21.15.102:0/2252090255 >> v2:172.21.15.102:3300/0 conn(0x7f67941593c0 msgr2=0x7f67940cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:14:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f679259c700 1 --2- 172.21.15.102:0/2252090255 >> v2:172.21.15.102:3300/0 conn(0x7f67941593c0 0x7f67940cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f679259c700 1 -- 172.21.15.102:0/2252090255 >> v1:172.21.15.102:6789/0 conn(0x7f67940cd8e0 legacy=0x7f67940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.084+0000 7f679259c700 1 -- 172.21.15.102:0/2252090255 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679406c7f0 con 0x7f6794153290 2023-03-03T10:14:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.085+0000 7f679259c700 1 --2- 172.21.15.102:0/2252090255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 0x7f6794158e80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f679408cc70 tx=0x7f6784009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f39ba0e3101db2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.085+0000 7f679159a700 1 -- 172.21.15.102:0/2252090255 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784011830 con 0x7f6794153290 2023-03-03T10:14:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.085+0000 7f679159a700 1 -- 172.21.15.102:0/2252090255 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784011990 con 0x7f6794153290 2023-03-03T10:14:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.085+0000 7f679159a700 1 -- 172.21.15.102:0/2252090255 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784011d20 con 0x7f6794153290 2023-03-03T10:14:18.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.085+0000 7f67989ff700 1 -- 172.21.15.102:0/2252090255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 msgr2=0x7f6794158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:18.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.085+0000 7f67989ff700 1 --2- 172.21.15.102:0/2252090255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 0x7f6794158e80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f679408cc70 tx=0x7f6784009b20 comp rx=0 tx=0).stop 2023-03-03T10:14:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.086+0000 7f67989ff700 1 -- 172.21.15.102:0/2252090255 shutdown_connections 2023-03-03T10:14:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.086+0000 7f67989ff700 1 --2- 172.21.15.102:0/2252090255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 0x7f6794158e80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.086+0000 7f67989ff700 1 --2- 172.21.15.102:0/2252090255 >> v2:172.21.15.102:3300/0 conn(0x7f67941593c0 0x7f67940cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.086+0000 7f67989ff700 1 -- 172.21.15.102:0/2252090255 >> 172.21.15.102:0/2252090255 conn(0x7f6794065fb0 msgr2=0x7f67940663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:14:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.086+0000 7f67989ff700 1 -- 172.21.15.102:0/2252090255 shutdown_connections 2023-03-03T10:14:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.086+0000 7f67989ff700 1 -- 172.21.15.102:0/2252090255 wait complete. 2023-03-03T10:14:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.088+0000 7f67989ff700 1 Processor -- start 2023-03-03T10:14:18.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.088+0000 7f67989ff700 1 -- start start 2023-03-03T10:14:18.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f67989ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67940cd8e0 0x7f67940b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:18.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f67989ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 0x7f67940c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:18.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f67989ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67941593c0 0x7f6794150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:18.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f67989ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67941f21e0 con 0x7f67940cd8e0 2023-03-03T10:14:18.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f67989ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67941eea40 con 0x7f6794153290 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f67989ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6794151250 con 0x7f67941593c0 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f679259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67940cd8e0 0x7f67940b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f679259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67940cd8e0 0x7f67940b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41952/0 (socket says 172.21.15.102:41952) 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f679259c700 1 -- 172.21.15.102:0/3207660355 learned_addr learned my addr 172.21.15.102:0/3207660355 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f6791d9b700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 0x7f67940c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.089+0000 7f6792d9d700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67941593c0 0x7f6794150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f679259c700 1 -- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67941593c0 msgr2=0x7f6794150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f679259c700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67941593c0 0x7f6794150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f679259c700 1 -- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 msgr2=0x7f67940c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:18.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f679259c700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 0x7f67940c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f679259c700 1 -- 172.21.15.102:0/3207660355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67840094a0 con 0x7f67940cd8e0 2023-03-03T10:14:18.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f6792d9d700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67941593c0 0x7f6794150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:14:18.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f679259c700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67940cd8e0 0x7f67940b2a30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6784009b60 tx=0x7f6784004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:18.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f6782ffd700 1 -- 172.21.15.102:0/3207660355 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67840036a0 con 0x7f67940cd8e0 2023-03-03T10:14:18.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f6782ffd700 1 -- 172.21.15.102:0/3207660355 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678401f070 con 0x7f67940cd8e0 2023-03-03T10:14:18.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f6782ffd700 1 -- 172.21.15.102:0/3207660355 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784003a40 con 0x7f67940cd8e0 2023-03-03T10:14:18.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.090+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67940ccde0 con 0x7f67940cd8e0 2023-03-03T10:14:18.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.091+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679415d160 con 0x7f67940cd8e0 2023-03-03T10:14:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.092+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67940cd0c0 con 0x7f67940cd8e0 2023-03-03T10:14:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.093+0000 7f6782ffd700 1 -- 172.21.15.102:0/3207660355 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f678400f050 con 0x7f67940cd8e0 2023-03-03T10:14:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.094+0000 7f6782ffd700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6774068320 0x7f677406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.094+0000 7f6782ffd700 1 -- 172.21.15.102:0/3207660355 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6784021070 con 0x7f67940cd8e0 2023-03-03T10:14:18.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.095+0000 7f6791d9b700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6774068320 0x7f677406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:18.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.095+0000 7f6791d9b700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6774068320 0x7f677406a7d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f67940b1740 tx=0x7f6788009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:18.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.099+0000 7f6782ffd700 1 -- 172.21.15.102:0/3207660355 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6784053d70 con 0x7f67940cd8e0 2023-03-03T10:14:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.331+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6794151810 con 0x7f67940cd8e0 2023-03-03T10:14:18.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.332+0000 7f6782ffd700 1 -- 172.21.15.102:0/3207660355 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6784057390 con 0x7f67940cd8e0 2023-03-03T10:14:18.336 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:14:18.336 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:14:18.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6774068320 msgr2=0x7f677406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:18.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6774068320 0x7f677406a7d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f67940b1740 tx=0x7f6788009210 comp rx=0 tx=0).stop 2023-03-03T10:14:18.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67940cd8e0 msgr2=0x7f67940b2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:18.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67940cd8e0 0x7f67940b2a30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6784009b60 tx=0x7f6784004750 comp rx=0 tx=0).stop 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 shutdown_connections 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6774068320 0x7f677406a7d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67941593c0 0x7f6794150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67940cd8e0 0x7f67940b2a30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 --2- 172.21.15.102:0/3207660355 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6794153290 0x7f67940c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 >> 172.21.15.102:0/3207660355 conn(0x7f6794065fb0 msgr2=0x7f67940cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 shutdown_connections 2023-03-03T10:14:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:14:18.335+0000 7f67989ff700 1 -- 172.21.15.102:0/3207660355 wait complete. 2023-03-03T10:14:18.340 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:14:18.365 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 119 secs before thrashing 2023-03-03T10:14:38.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:38.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:09.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:09.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:39.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:39.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:09.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:09.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:17.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:16:17.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.617+0000 7fd082bff700 1 Processor -- start 2023-03-03T10:16:17.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.618+0000 7fd082bff700 1 -- start start 2023-03-03T10:16:17.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd082bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd07c0cd8e0 0x7fd07c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd082bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd390 0x7fd07c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd082bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c153660 con 0x7fd07c153290 2023-03-03T10:16:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd082bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c06c360 con 0x7fd07c0cd8e0 2023-03-03T10:16:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd082bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c15cea0 con 0x7fd07c0cd390 2023-03-03T10:16:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd08099b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd07c153290 0x7fd07c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34744/0 (socket says 172.21.15.102:34744) 2023-03-03T10:16:17.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd08099b700 1 -- 172.21.15.102:0/1578895422 learned_addr learned my addr 172.21.15.102:0/1578895422 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:16:17.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd07bfff700 1 --2- 172.21.15.102:0/1578895422 >> v2:172.21.15.102:3300/0 conn(0x7fd07c0cd8e0 0x7fd07c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:17.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.620+0000 7fd08119c700 1 --2- 172.21.15.102:0/1578895422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd390 0x7fd07c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:17.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.621+0000 7fd07bfff700 1 -- 172.21.15.102:0/1578895422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd390 msgr2=0x7fd07c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:16:17.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.621+0000 7fd07bfff700 1 --2- 172.21.15.102:0/1578895422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd390 0x7fd07c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.621+0000 7fd07b7fe700 1 -- 172.21.15.102:0/1578895422 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093212946 0 0) 0x7fd07c153660 con 0x7fd07c153290 2023-03-03T10:16:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.621+0000 7fd07bfff700 1 -- 172.21.15.102:0/1578895422 >> v1:172.21.15.102:6789/0 conn(0x7fd07c153290 legacy=0x7fd07c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.621+0000 7fd07bfff700 1 -- 172.21.15.102:0/1578895422 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c06c7f0 con 0x7fd07c0cd8e0 2023-03-03T10:16:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.621+0000 7fd07bfff700 1 --2- 172.21.15.102:0/1578895422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd8e0 0x7fd07c0cde20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd068009840 tx=0x7fd068009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb16232ee88eb17 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:16:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd07b7fe700 1 -- 172.21.15.102:0/1578895422 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068011830 con 0x7fd07c0cd8e0 2023-03-03T10:16:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd07b7fe700 1 -- 172.21.15.102:0/1578895422 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd068011990 con 0x7fd07c0cd8e0 2023-03-03T10:16:17.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd07b7fe700 1 -- 172.21.15.102:0/1578895422 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068011d20 con 0x7fd07c0cd8e0 2023-03-03T10:16:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 -- 172.21.15.102:0/1578895422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd8e0 msgr2=0x7fd07c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 --2- 172.21.15.102:0/1578895422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd8e0 0x7fd07c0cde20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd068009840 tx=0x7fd068009b50 comp rx=0 tx=0).stop 2023-03-03T10:16:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 -- 172.21.15.102:0/1578895422 shutdown_connections 2023-03-03T10:16:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 --2- 172.21.15.102:0/1578895422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd390 0x7fd07c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 --2- 172.21.15.102:0/1578895422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd8e0 0x7fd07c0cde20 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 -- 172.21.15.102:0/1578895422 >> 172.21.15.102:0/1578895422 conn(0x7fd07c065fb0 msgr2=0x7fd07c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:16:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 -- 172.21.15.102:0/1578895422 shutdown_connections 2023-03-03T10:16:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.622+0000 7fd082bff700 1 -- 172.21.15.102:0/1578895422 wait complete. 2023-03-03T10:16:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.625+0000 7fd082bff700 1 Processor -- start 2023-03-03T10:16:17.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.625+0000 7fd082bff700 1 -- start start 2023-03-03T10:16:17.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd082bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd390 0x7fd07c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd082bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd8e0 0x7fd07c15ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd082bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd07c153290 0x7fd07c0c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd082bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1e3ed0 con 0x7fd07c0cd390 2023-03-03T10:16:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd082bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1ef270 con 0x7fd07c153290 2023-03-03T10:16:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd082bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1ef550 con 0x7fd07c0cd8e0 2023-03-03T10:16:17.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd08099b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd390 0x7fd07c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd08099b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd390 0x7fd07c0b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60650/0 (socket says 172.21.15.102:60650) 2023-03-03T10:16:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd08099b700 1 -- 172.21.15.102:0/1162566716 learned_addr learned my addr 172.21.15.102:0/1162566716 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:16:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd07bfff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd8e0 0x7fd07c15ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.627+0000 7fd08119c700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd07c153290 0x7fd07c0c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd08099b700 1 -- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd8e0 msgr2=0x7fd07c15ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd08099b700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd8e0 0x7fd07c15ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd08099b700 1 -- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd07c153290 msgr2=0x7fd07c0c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd08099b700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd07c153290 0x7fd07c0c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd08099b700 1 -- 172.21.15.102:0/1162566716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0680094a0 con 0x7fd07c0cd390 2023-03-03T10:16:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd08099b700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd390 0x7fd07c0b1e80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd0700011e0 tx=0x7fd07000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:16:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd0797fa700 1 -- 172.21.15.102:0/1162566716 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070004c80 con 0x7fd07c0cd390 2023-03-03T10:16:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd0797fa700 1 -- 172.21.15.102:0/1162566716 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070004e00 con 0x7fd07c0cd390 2023-03-03T10:16:17.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd0797fa700 1 -- 172.21.15.102:0/1162566716 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070007310 con 0x7fd07c0cd390 2023-03-03T10:16:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.628+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c0ccde0 con 0x7fd07c0cd390 2023-03-03T10:16:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.629+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c15d160 con 0x7fd07c0cd390 2023-03-03T10:16:17.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.629+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c0cd0c0 con 0x7fd07c0cd390 2023-03-03T10:16:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.631+0000 7fd0797fa700 1 -- 172.21.15.102:0/1162566716 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd07001a020 con 0x7fd07c0cd390 2023-03-03T10:16:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.632+0000 7fd0797fa700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd060068320 0x7fd06006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.632+0000 7fd0797fa700 1 -- 172.21.15.102:0/1162566716 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd070085930 con 0x7fd07c0cd390 2023-03-03T10:16:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.632+0000 7fd07bfff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd060068320 0x7fd06006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:17.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.633+0000 7fd07bfff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd060068320 0x7fd06006a7d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd0680090b0 tx=0x7fd06801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:16:17.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.636+0000 7fd0797fa700 1 -- 172.21.15.102:0/1162566716 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0700520f0 con 0x7fd07c0cd390 2023-03-03T10:16:17.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.871+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd07c04e6e0 con 0x7fd07c0cd390 2023-03-03T10:16:17.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.872+0000 7fd0797fa700 1 -- 172.21.15.102:0/1162566716 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd070055710 con 0x7fd07c0cd390 2023-03-03T10:16:17.874 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:16:17.874 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:16:17.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd060068320 msgr2=0x7fd06006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:17.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd060068320 0x7fd06006a7d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd0680090b0 tx=0x7fd06801c040 comp rx=0 tx=0).stop 2023-03-03T10:16:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd390 msgr2=0x7fd07c0b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd390 0x7fd07c0b1e80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd0700011e0 tx=0x7fd07000ca30 comp rx=0 tx=0).stop 2023-03-03T10:16:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 shutdown_connections 2023-03-03T10:16:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd07c153290 0x7fd07c0c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd07c0cd390 0x7fd07c0b1e80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd060068320 0x7fd06006a7d0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 --2- 172.21.15.102:0/1162566716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd07c0cd8e0 0x7fd07c15ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:17.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 >> 172.21.15.102:0/1162566716 conn(0x7fd07c065fb0 msgr2=0x7fd07c158940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:16:17.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.874+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 shutdown_connections 2023-03-03T10:16:17.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:16:17.875+0000 7fd082bff700 1 -- 172.21.15.102:0/1162566716 wait complete. 2023-03-03T10:16:17.878 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:16:17.903 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 76 secs before thrashing 2023-03-03T10:16:39.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:39.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:09.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:09.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:33.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:17:34.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.157+0000 7f9214d9d700 1 Processor -- start 2023-03-03T10:17:34.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.157+0000 7f9214d9d700 1 -- start start 2023-03-03T10:17:34.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f9214d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f92100ced00 0x7f92100c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:34.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f9214d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92100b3970 0x7f92100c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:34.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f9214d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100cf0d0 0x7f9210158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:34.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f9214d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92100c4930 con 0x7f92100b3970 2023-03-03T10:17:34.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f9214d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92100c3d20 con 0x7f92100ced00 2023-03-03T10:17:34.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f9214d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92100cf570 con 0x7f92100cf0d0 2023-03-03T10:17:34.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f920effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92100b3970 0x7f92100c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f920effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92100b3970 0x7f92100c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48648/0 (socket says 172.21.15.102:48648) 2023-03-03T10:17:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f920effd700 1 -- 172.21.15.102:0/1034648629 learned_addr learned my addr 172.21.15.102:0/1034648629 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:17:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f920f7fe700 1 --2- 172.21.15.102:0/1034648629 >> v2:172.21.15.172:3300/0 conn(0x7f92100ced00 0x7f92100c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.159+0000 7f920ffff700 1 --2- 172.21.15.102:0/1034648629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100cf0d0 0x7f9210158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920effd700 1 -- 172.21.15.102:0/1034648629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100cf0d0 msgr2=0x7f9210158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920effd700 1 --2- 172.21.15.102:0/1034648629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100cf0d0 0x7f9210158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920effd700 1 -- 172.21.15.102:0/1034648629 >> v2:172.21.15.172:3300/0 conn(0x7f92100ced00 msgr2=0x7f92100c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920effd700 1 --2- 172.21.15.102:0/1034648629 >> v2:172.21.15.172:3300/0 conn(0x7f92100ced00 0x7f92100c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920effd700 1 -- 172.21.15.102:0/1034648629 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921015cba0 con 0x7f92100b3970 2023-03-03T10:17:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920f7fe700 1 --2- 172.21.15.102:0/1034648629 >> v2:172.21.15.172:3300/0 conn(0x7f92100ced00 0x7f92100c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:17:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920effd700 1 --2- 172.21.15.102:0/1034648629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100b3970 0x7f92100c41b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f92100b1b40 tx=0x7f9204009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de039426020859 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920e7fc700 1 -- 172.21.15.102:0/1034648629 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9204011830 con 0x7f92100b3970 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920e7fc700 1 -- 172.21.15.102:0/1034648629 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9204011990 con 0x7f92100b3970 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.160+0000 7f920e7fc700 1 -- 172.21.15.102:0/1034648629 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9204011d20 con 0x7f92100b3970 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 -- 172.21.15.102:0/1034648629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100b3970 msgr2=0x7f92100c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 --2- 172.21.15.102:0/1034648629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100b3970 0x7f92100c41b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f92100b1b40 tx=0x7f9204009bd0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 -- 172.21.15.102:0/1034648629 shutdown_connections 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 --2- 172.21.15.102:0/1034648629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100cf0d0 0x7f9210158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 --2- 172.21.15.102:0/1034648629 >> v2:172.21.15.172:3300/0 conn(0x7f92100ced00 0x7f92100c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 --2- 172.21.15.102:0/1034648629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100b3970 0x7f92100c41b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 -- 172.21.15.102:0/1034648629 >> 172.21.15.102:0/1034648629 conn(0x7f921008b1c0 msgr2=0x7f9210071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 -- 172.21.15.102:0/1034648629 shutdown_connections 2023-03-03T10:17:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.161+0000 7f9214d9d700 1 -- 172.21.15.102:0/1034648629 wait complete. 2023-03-03T10:17:34.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.163+0000 7f9214d9d700 1 Processor -- start 2023-03-03T10:17:34.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.163+0000 7f9214d9d700 1 -- start start 2023-03-03T10:17:34.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.164+0000 7f9214d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100b3970 0x7f92100c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:34.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.164+0000 7f9214d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100ced00 0x7f92101f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:34.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.164+0000 7f9214d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92100cf0d0 0x7f921014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:34.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f9214d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921006b8e0 con 0x7f92100ced00 2023-03-03T10:17:34.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f9214d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100c6a80 con 0x7f92100cf0d0 2023-03-03T10:17:34.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f9214d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92101f1aa0 con 0x7f92100b3970 2023-03-03T10:17:34.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100b3970 0x7f92100c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:34.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100b3970 0x7f92100c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36716/0 (socket says 172.21.15.102:36716) 2023-03-03T10:17:34.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 -- 172.21.15.102:0/2401189899 learned_addr learned my addr 172.21.15.102:0/2401189899 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:17:34.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920effd700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100ced00 0x7f92101f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:34.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920ffff700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92100cf0d0 0x7f921014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:34.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 -- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92100cf0d0 msgr2=0x7f921014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92100cf0d0 0x7f921014dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 -- 172.21.15.102:0/2401189899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100ced00 msgr2=0x7f92101f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100ced00 0x7f92101f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.165+0000 7f920f7fe700 1 -- 172.21.15.102:0/2401189899 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92000094a0 con 0x7f92100b3970 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.166+0000 7f920f7fe700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100b3970 0x7f92100c8650 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f920000ebe0 tx=0x7f920000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.166+0000 7f91f7fff700 1 -- 172.21.15.102:0/2401189899 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200009780 con 0x7f92100b3970 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.166+0000 7f91f7fff700 1 -- 172.21.15.102:0/2401189899 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920000c850 con 0x7f92100b3970 2023-03-03T10:17:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.166+0000 7f91f7fff700 1 -- 172.21.15.102:0/2401189899 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920000cbe0 con 0x7f92100b3970 2023-03-03T10:17:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.166+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9204009550 con 0x7f92100b3970 2023-03-03T10:17:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.166+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921006bf20 con 0x7f92100b3970 2023-03-03T10:17:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.167+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921015ce80 con 0x7f92100b3970 2023-03-03T10:17:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.169+0000 7f91f7fff700 1 -- 172.21.15.102:0/2401189899 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9200023070 con 0x7f92100b3970 2023-03-03T10:17:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.170+0000 7f91f7fff700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f91fc068320 0x7f91fc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.170+0000 7f91f7fff700 1 -- 172.21.15.102:0/2401189899 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9200085fb0 con 0x7f92100b3970 2023-03-03T10:17:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.170+0000 7f920effd700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f91fc068320 0x7f91fc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:34.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.171+0000 7f920effd700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f91fc068320 0x7f91fc06a7d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f92100b1b40 tx=0x7f9204008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:34.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.174+0000 7f91f7fff700 1 -- 172.21.15.102:0/2401189899 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92000526c0 con 0x7f92100b3970 2023-03-03T10:17:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.407+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f921006c200 con 0x7f92100b3970 2023-03-03T10:17:34.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.408+0000 7f91f7fff700 1 -- 172.21.15.102:0/2401189899 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9200055ce0 con 0x7f92100b3970 2023-03-03T10:17:34.411 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:17:34.411 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:17:34.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.410+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f91fc068320 msgr2=0x7f91fc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.410+0000 7f9214d9d700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f91fc068320 0x7f91fc06a7d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f92100b1b40 tx=0x7f9204008030 comp rx=0 tx=0).stop 2023-03-03T10:17:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.410+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100b3970 msgr2=0x7f92100c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.410+0000 7f9214d9d700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100b3970 0x7f92100c8650 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f920000ebe0 tx=0x7f920000eef0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 shutdown_connections 2023-03-03T10:17:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f91fc068320 0x7f91fc06a7d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f92100b3970 0x7f92100c8650 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92100cf0d0 0x7f921014dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 --2- 172.21.15.102:0/2401189899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92100ced00 0x7f92101f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:34.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 >> 172.21.15.102:0/2401189899 conn(0x7f921008b1c0 msgr2=0x7f9210065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:34.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 shutdown_connections 2023-03-03T10:17:34.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:17:34.411+0000 7f9214d9d700 1 -- 172.21.15.102:0/2401189899 wait complete. 2023-03-03T10:17:34.415 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:17:34.441 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T10:17:39.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:40.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:10.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:10.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:40.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:40.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:58.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:18:58.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.698+0000 7f5c65b9d700 1 Processor -- start 2023-03-03T10:18:58.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.698+0000 7f5c65b9d700 1 -- start start 2023-03-03T10:18:58.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c65b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5c600ced00 0x7f5c600c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:58.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c65b9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c600b3970 0x7f5c600c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:58.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c65b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c600c3d20 con 0x7f5c600cf0d0 2023-03-03T10:18:58.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c65b9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c600c4930 con 0x7f5c600b3970 2023-03-03T10:18:58.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c65b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c600cf570 con 0x7f5c600ced00 2023-03-03T10:18:58.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c5ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c600b3970 0x7f5c600c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:58.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c5ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c600b3970 0x7f5c600c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59952/0 (socket says 172.21.15.102:59952) 2023-03-03T10:18:58.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c64b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5c600ced00 0x7f5c600c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:58.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c5ffff700 1 -- 172.21.15.102:0/3129946041 learned_addr learned my addr 172.21.15.102:0/3129946041 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:18:58.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.700+0000 7f5c6539c700 1 --1- 172.21.15.102:0/3129946041 >> v1:172.21.15.102:6789/0 conn(0x7f5c600cf0d0 0x7f5c60158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59578/0 (socket says 172.21.15.102:59578) 2023-03-03T10:18:58.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5ffff700 1 -- 172.21.15.102:0/3129946041 >> v2:172.21.15.172:3300/0 conn(0x7f5c600ced00 msgr2=0x7f5c600c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:18:58.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5ffff700 1 --2- 172.21.15.102:0/3129946041 >> v2:172.21.15.172:3300/0 conn(0x7f5c600ced00 0x7f5c600c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5ffff700 1 -- 172.21.15.102:0/3129946041 >> v1:172.21.15.102:6789/0 conn(0x7f5c600cf0d0 legacy=0x7f5c60158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:58.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5ffff700 1 -- 172.21.15.102:0/3129946041 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6015cba0 con 0x7f5c600b3970 2023-03-03T10:18:58.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5ffff700 1 --2- 172.21.15.102:0/3129946041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600b3970 0x7f5c600c41b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5c600b1b40 tx=0x7f5c4c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b49f5bf3c236c18 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:18:58.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5f7fe700 1 -- 172.21.15.102:0/3129946041 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c011830 con 0x7f5c600b3970 2023-03-03T10:18:58.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5f7fe700 1 -- 172.21.15.102:0/3129946041 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c011990 con 0x7f5c600b3970 2023-03-03T10:18:58.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.701+0000 7f5c5f7fe700 1 -- 172.21.15.102:0/3129946041 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c011d20 con 0x7f5c600b3970 2023-03-03T10:18:58.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 -- 172.21.15.102:0/3129946041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600b3970 msgr2=0x7f5c600c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:58.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/3129946041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600b3970 0x7f5c600c41b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5c600b1b40 tx=0x7f5c4c009bd0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 -- 172.21.15.102:0/3129946041 shutdown_connections 2023-03-03T10:18:58.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/3129946041 >> v2:172.21.15.172:3300/0 conn(0x7f5c600ced00 0x7f5c600c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/3129946041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600b3970 0x7f5c600c41b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 -- 172.21.15.102:0/3129946041 >> 172.21.15.102:0/3129946041 conn(0x7f5c6008b1c0 msgr2=0x7f5c60071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:18:58.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 -- 172.21.15.102:0/3129946041 shutdown_connections 2023-03-03T10:18:58.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.702+0000 7f5c65b9d700 1 -- 172.21.15.102:0/3129946041 wait complete. 2023-03-03T10:18:58.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.705+0000 7f5c65b9d700 1 Processor -- start 2023-03-03T10:18:58.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.705+0000 7f5c65b9d700 1 -- start start 2023-03-03T10:18:58.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c65b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c600b3970 0x7f5c601e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:58.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c65b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600ced00 0x7f5c600b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:58.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c65b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c600cf0d0 0x7f5c600c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:58.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c65b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601f2cd0 con 0x7f5c600ced00 2023-03-03T10:18:58.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c65b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c600c82b0 con 0x7f5c600b3970 2023-03-03T10:18:58.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c65b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601f20e0 con 0x7f5c600cf0d0 2023-03-03T10:18:58.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c5ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600ced00 0x7f5c600b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:58.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c600b3970 0x7f5c601e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:58.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c600b3970 0x7f5c601e5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47186/0 (socket says 172.21.15.102:47186) 2023-03-03T10:18:58.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 -- 172.21.15.102:0/2325116615 learned_addr learned my addr 172.21.15.102:0/2325116615 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:18:58.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c6539c700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c600cf0d0 0x7f5c600c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:58.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 -- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c600cf0d0 msgr2=0x7f5c600c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:58.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c600cf0d0 0x7f5c600c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 -- 172.21.15.102:0/2325116615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600ced00 msgr2=0x7f5c600b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:58.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600ced00 0x7f5c600b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c64b9b700 1 -- 172.21.15.102:0/2325116615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009550 con 0x7f5c600b3970 2023-03-03T10:18:58.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.707+0000 7f5c6539c700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c600cf0d0 0x7f5c600c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:18:58.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.708+0000 7f5c64b9b700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c600b3970 0x7f5c601e5c70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e870 tx=0x7f5c5400eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:18:58.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.708+0000 7f5c5d7fa700 1 -- 172.21.15.102:0/2325116615 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c54009590 con 0x7f5c600b3970 2023-03-03T10:18:58.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.708+0000 7f5c5d7fa700 1 -- 172.21.15.102:0/2325116615 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54013680 con 0x7f5c600b3970 2023-03-03T10:18:58.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.708+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6006bf20 con 0x7f5c600b3970 2023-03-03T10:18:58.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.708+0000 7f5c5d7fa700 1 -- 172.21.15.102:0/2325116615 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c540139f0 con 0x7f5c600b3970 2023-03-03T10:18:58.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.708+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6015ce80 con 0x7f5c600b3970 2023-03-03T10:18:58.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.709+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6006c200 con 0x7f5c600b3970 2023-03-03T10:18:58.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.711+0000 7f5c5d7fa700 1 -- 172.21.15.102:0/2325116615 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c54023070 con 0x7f5c600b3970 2023-03-03T10:18:58.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.712+0000 7f5c5d7fa700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c440682f0 0x7f5c4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:58.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.712+0000 7f5c5d7fa700 1 -- 172.21.15.102:0/2325116615 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5c54085f20 con 0x7f5c600b3970 2023-03-03T10:18:58.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.712+0000 7f5c5ffff700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c440682f0 0x7f5c4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:58.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.713+0000 7f5c5ffff700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c440682f0 0x7f5c4406a7a0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5c600b1b40 tx=0x7f5c4c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:18:58.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.716+0000 7f5c5d7fa700 1 -- 172.21.15.102:0/2325116615 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c540525d0 con 0x7f5c600b3970 2023-03-03T10:18:58.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.950+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c600cf940 con 0x7f5c600b3970 2023-03-03T10:18:58.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.951+0000 7f5c5d7fa700 1 -- 172.21.15.102:0/2325116615 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5c54055bf0 con 0x7f5c600b3970 2023-03-03T10:18:58.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:18:58.955 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:18:58.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.954+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c440682f0 msgr2=0x7f5c4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:58.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.954+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c440682f0 0x7f5c4406a7a0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5c600b1b40 tx=0x7f5c4c008030 comp rx=0 tx=0).stop 2023-03-03T10:18:58.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.954+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c600b3970 msgr2=0x7f5c601e5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:58.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.954+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c600b3970 0x7f5c601e5c70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e870 tx=0x7f5c5400eb80 comp rx=0 tx=0).stop 2023-03-03T10:18:58.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 shutdown_connections 2023-03-03T10:18:58.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5c440682f0 0x7f5c4406a7a0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5c600cf0d0 0x7f5c600c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c600b3970 0x7f5c601e5c70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 --2- 172.21.15.102:0/2325116615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c600ced00 0x7f5c600b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:58.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 >> 172.21.15.102:0/2325116615 conn(0x7f5c6008b1c0 msgr2=0x7f5c600c4100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:18:58.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 shutdown_connections 2023-03-03T10:18:58.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:18:58.955+0000 7f5c65b9d700 1 -- 172.21.15.102:0/2325116615 wait complete. 2023-03-03T10:18:58.960 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:18:58.985 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-03-03T10:19:10.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:10.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:40.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:40.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T10:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:20:10.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:10.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:15.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:20:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.250+0000 7faebcfff700 1 Processor -- start 2023-03-03T10:20:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.250+0000 7faebcfff700 1 -- start start 2023-03-03T10:20:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.251+0000 7faebcfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faeb8153290 0x7faeb8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.251+0000 7faebcfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faeb80cd8e0 0x7faeb80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.251+0000 7faebcfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb8153660 con 0x7faeb81593c0 2023-03-03T10:20:16.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.251+0000 7faebcfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb806c360 con 0x7faeb8153290 2023-03-03T10:20:16.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.251+0000 7faebcfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb815cea0 con 0x7faeb80cd8e0 2023-03-03T10:20:16.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faeb8153290 0x7faeb8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:16.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faeb8153290 0x7faeb8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54814/0 (socket says 172.21.15.102:54814) 2023-03-03T10:20:16.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb659c700 1 -- 172.21.15.102:0/704383588 learned_addr learned my addr 172.21.15.102:0/704383588 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:20:16.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb6d9d700 1 --2- 172.21.15.102:0/704383588 >> v2:172.21.15.172:3300/0 conn(0x7faeb80cd8e0 0x7faeb80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:16.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb659c700 1 -- 172.21.15.102:0/704383588 >> v2:172.21.15.172:3300/0 conn(0x7faeb80cd8e0 msgr2=0x7faeb80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:20:16.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb659c700 1 --2- 172.21.15.102:0/704383588 >> v2:172.21.15.172:3300/0 conn(0x7faeb80cd8e0 0x7faeb80ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb659c700 1 -- 172.21.15.102:0/704383588 >> v1:172.21.15.172:6789/0 conn(0x7faeb81593c0 legacy=0x7faeb80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:16.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb659c700 1 -- 172.21.15.102:0/704383588 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb806c7f0 con 0x7faeb8153290 2023-03-03T10:20:16.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.252+0000 7faeb559a700 1 -- 172.21.15.102:0/704383588 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413061294 0 0) 0x7faeb8153660 con 0x7faeb81593c0 2023-03-03T10:20:16.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.253+0000 7faeb659c700 1 --2- 172.21.15.102:0/704383588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb8153290 0x7faeb8158e80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faeb80c9390 tx=0x7faea8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec733f6d52ed8cf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:16.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.253+0000 7faeb559a700 1 -- 172.21.15.102:0/704383588 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8011830 con 0x7faeb8153290 2023-03-03T10:20:16.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.253+0000 7faeb559a700 1 -- 172.21.15.102:0/704383588 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea8011990 con 0x7faeb8153290 2023-03-03T10:20:16.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.253+0000 7faeb559a700 1 -- 172.21.15.102:0/704383588 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8011d20 con 0x7faeb8153290 2023-03-03T10:20:16.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.253+0000 7faebcfff700 1 -- 172.21.15.102:0/704383588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb8153290 msgr2=0x7faeb8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:16.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.253+0000 7faebcfff700 1 --2- 172.21.15.102:0/704383588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb8153290 0x7faeb8158e80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faeb80c9390 tx=0x7faea8009bd0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.254+0000 7faebcfff700 1 -- 172.21.15.102:0/704383588 shutdown_connections 2023-03-03T10:20:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.254+0000 7faebcfff700 1 --2- 172.21.15.102:0/704383588 >> v2:172.21.15.172:3300/0 conn(0x7faeb80cd8e0 0x7faeb80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.254+0000 7faebcfff700 1 --2- 172.21.15.102:0/704383588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb8153290 0x7faeb8158e80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.254+0000 7faebcfff700 1 -- 172.21.15.102:0/704383588 >> 172.21.15.102:0/704383588 conn(0x7faeb8065fb0 msgr2=0x7faeb80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:20:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.254+0000 7faebcfff700 1 -- 172.21.15.102:0/704383588 shutdown_connections 2023-03-03T10:20:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.254+0000 7faebcfff700 1 -- 172.21.15.102:0/704383588 wait complete. 2023-03-03T10:20:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.256+0000 7faebcfff700 1 Processor -- start 2023-03-03T10:20:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.256+0000 7faebcfff700 1 -- start start 2023-03-03T10:20:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faebcfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb80cd8e0 0x7faeb80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faebcfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8153290 0x7faeb80c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faebcfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb81593c0 0x7faeb814dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faebcfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb81eef20 con 0x7faeb81593c0 2023-03-03T10:20:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faebcfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb81ee650 con 0x7faeb80cd8e0 2023-03-03T10:20:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faebcfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8156130 con 0x7faeb8153290 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faeb659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb80cd8e0 0x7faeb80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faeb659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb80cd8e0 0x7faeb80b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47662/0 (socket says 172.21.15.102:47662) 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faeb659c700 1 -- 172.21.15.102:0/341848561 learned_addr learned my addr 172.21.15.102:0/341848561 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faeb6d9d700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb81593c0 0x7faeb814dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.257+0000 7faeb5d9b700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8153290 0x7faeb80c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faeb659c700 1 -- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8153290 msgr2=0x7faeb80c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faeb659c700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8153290 0x7faeb80c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faeb659c700 1 -- 172.21.15.102:0/341848561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb81593c0 msgr2=0x7faeb814dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:16.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faeb659c700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb81593c0 0x7faeb814dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faeb659c700 1 -- 172.21.15.102:0/341848561 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea8009550 con 0x7faeb80cd8e0 2023-03-03T10:20:16.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faeb659c700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb80cd8e0 0x7faeb80b2cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7faea8009c10 tx=0x7faea8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faea6ffd700 1 -- 172.21.15.102:0/341848561 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8004510 con 0x7faeb80cd8e0 2023-03-03T10:20:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faea6ffd700 1 -- 172.21.15.102:0/341848561 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea80036a0 con 0x7faeb80cd8e0 2023-03-03T10:20:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faea6ffd700 1 -- 172.21.15.102:0/341848561 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8003a10 con 0x7faeb80cd8e0 2023-03-03T10:20:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.258+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb80ccde0 con 0x7faeb80cd8e0 2023-03-03T10:20:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.259+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb815d160 con 0x7faeb80cd8e0 2023-03-03T10:20:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.259+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb80cd0c0 con 0x7faeb80cd8e0 2023-03-03T10:20:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.261+0000 7faea6ffd700 1 -- 172.21.15.102:0/341848561 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faea802d070 con 0x7faeb80cd8e0 2023-03-03T10:20:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.262+0000 7faea6ffd700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae98068320 0x7fae9806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.262+0000 7faea6ffd700 1 -- 172.21.15.102:0/341848561 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7faea8021030 con 0x7faeb80cd8e0 2023-03-03T10:20:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.262+0000 7faeb5d9b700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae98068320 0x7fae9806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:16.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.263+0000 7faeb5d9b700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae98068320 0x7fae9806a7d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faeb8157320 tx=0x7faeac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:16.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.267+0000 7faea6ffd700 1 -- 172.21.15.102:0/341848561 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faea8053b10 con 0x7faeb80cd8e0 2023-03-03T10:20:16.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.502+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeb8157370 con 0x7faeb80cd8e0 2023-03-03T10:20:16.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.503+0000 7faea6ffd700 1 -- 172.21.15.102:0/341848561 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7faea8015090 con 0x7faeb80cd8e0 2023-03-03T10:20:16.505 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:20:16.505 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:20:16.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae98068320 msgr2=0x7fae9806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:16.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae98068320 0x7fae9806a7d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faeb8157320 tx=0x7faeac009040 comp rx=0 tx=0).stop 2023-03-03T10:20:16.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb80cd8e0 msgr2=0x7faeb80b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:16.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb80cd8e0 0x7faeb80b2cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7faea8009c10 tx=0x7faea8004750 comp rx=0 tx=0).stop 2023-03-03T10:20:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 shutdown_connections 2023-03-03T10:20:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb81593c0 0x7faeb814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faeb80cd8e0 0x7faeb80b2cc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae98068320 0x7fae9806a7d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 --2- 172.21.15.102:0/341848561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faeb8153290 0x7faeb80c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 >> 172.21.15.102:0/341848561 conn(0x7faeb8065fb0 msgr2=0x7faeb8150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:20:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 shutdown_connections 2023-03-03T10:20:16.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:16.506+0000 7faebcfff700 1 -- 172.21.15.102:0/341848561 wait complete. 2023-03-03T10:20:16.510 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:20:16.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2023-03-03T10:20:29.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:20:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.787+0000 7f03ab59e700 1 Processor -- start 2023-03-03T10:20:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.787+0000 7f03ab59e700 1 -- start start 2023-03-03T10:20:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03ab59e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03ac1564b0 0x7f03ac0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03ab59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0c9610 0x7f03ac0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03ab59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f03ac0ca440 0x7f03ac0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03ab59e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ac0c9ca0 con 0x7f03ac1564b0 2023-03-03T10:20:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03ab59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ac0b46f0 con 0x7f03ac0ca440 2023-03-03T10:20:29.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03ab59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0cadc0 con 0x7f03ac0c9610 2023-03-03T10:20:29.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03aa59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03ac1564b0 0x7f03ac0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03aa59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03ac1564b0 0x7f03ac0cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49342/0 (socket says 172.21.15.102:49342) 2023-03-03T10:20:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03aa59c700 1 -- 172.21.15.102:0/2673974923 learned_addr learned my addr 172.21.15.102:0/2673974923 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:20:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03a3fff700 1 --2- 172.21.15.102:0/2673974923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0c9610 0x7f03ac0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:29.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.789+0000 7f03aad9d700 1 --2- 172.21.15.102:0/2673974923 >> v2:172.21.15.172:3300/0 conn(0x7f03ac0ca440 0x7f03ac0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.790+0000 7f03aa59c700 1 -- 172.21.15.102:0/2673974923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0c9610 msgr2=0x7f03ac0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:20:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.790+0000 7f03aa59c700 1 --2- 172.21.15.102:0/2673974923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0c9610 0x7f03ac0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.790+0000 7f03aa59c700 1 -- 172.21.15.102:0/2673974923 >> v2:172.21.15.172:3300/0 conn(0x7f03ac0ca440 msgr2=0x7f03ac0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:20:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.790+0000 7f03aa59c700 1 --2- 172.21.15.102:0/2673974923 >> v2:172.21.15.172:3300/0 conn(0x7f03ac0ca440 0x7f03ac0ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.790+0000 7f03aa59c700 1 -- 172.21.15.102:0/2673974923 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac0b4920 con 0x7f03ac1564b0 2023-03-03T10:20:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.790+0000 7f03aa59c700 1 --2- 172.21.15.102:0/2673974923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac0cb660 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b1c10 tx=0x7f039c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d1e618176c1927 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.790+0000 7f03a9d9b700 1 -- 172.21.15.102:0/2673974923 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c011830 con 0x7f03ac1564b0 2023-03-03T10:20:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03a9d9b700 1 -- 172.21.15.102:0/2673974923 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c011990 con 0x7f03ac1564b0 2023-03-03T10:20:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03a9d9b700 1 -- 172.21.15.102:0/2673974923 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c011d20 con 0x7f03ac1564b0 2023-03-03T10:20:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 -- 172.21.15.102:0/2673974923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 msgr2=0x7f03ac0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 --2- 172.21.15.102:0/2673974923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac0cb660 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b1c10 tx=0x7f039c009bd0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 -- 172.21.15.102:0/2673974923 shutdown_connections 2023-03-03T10:20:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 --2- 172.21.15.102:0/2673974923 >> v2:172.21.15.172:3300/0 conn(0x7f03ac0ca440 0x7f03ac0ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 --2- 172.21.15.102:0/2673974923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac0cb660 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 --2- 172.21.15.102:0/2673974923 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0c9610 0x7f03ac0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 -- 172.21.15.102:0/2673974923 >> 172.21.15.102:0/2673974923 conn(0x7f03ac065fb0 msgr2=0x7f03ac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:20:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 -- 172.21.15.102:0/2673974923 shutdown_connections 2023-03-03T10:20:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.791+0000 7f03ab59e700 1 -- 172.21.15.102:0/2673974923 wait complete. 2023-03-03T10:20:29.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.793+0000 7f03ab59e700 1 Processor -- start 2023-03-03T10:20:29.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.794+0000 7f03ab59e700 1 -- start start 2023-03-03T10:20:29.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.794+0000 7f03ab59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03ac0c9610 0x7f03ac158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.794+0000 7f03ab59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0ca440 0x7f03ac0ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03ab59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03ab59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac1f1ab0 con 0x7f03ac1564b0 2023-03-03T10:20:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03ab59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0cd3e0 con 0x7f03ac0c9610 2023-03-03T10:20:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03ab59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0cd6c0 con 0x7f03ac0ca440 2023-03-03T10:20:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03ac0c9610 0x7f03ac158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03ac0c9610 0x7f03ac158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34784/0 (socket says 172.21.15.102:34784) 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 -- 172.21.15.102:0/4226329854 learned_addr learned my addr 172.21.15.102:0/4226329854 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03a3fff700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0ca440 0x7f03ac0ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 -- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0ca440 msgr2=0x7f03ac0ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0ca440 0x7f03ac0ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 -- 172.21.15.102:0/4226329854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 msgr2=0x7f03ac152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aa59c700 1 -- 172.21.15.102:0/4226329854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c009550 con 0x7f03ac0c9610 2023-03-03T10:20:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.795+0000 7f03aad9d700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:20:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.796+0000 7f03aa59c700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03ac0c9610 0x7f03ac158aa0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f039c009c10 tx=0x7f039c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.796+0000 7f03a2ffd700 1 -- 172.21.15.102:0/4226329854 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c004450 con 0x7f03ac0c9610 2023-03-03T10:20:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.796+0000 7f03a2ffd700 1 -- 172.21.15.102:0/4226329854 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c01e070 con 0x7f03ac0c9610 2023-03-03T10:20:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.796+0000 7f03a2ffd700 1 -- 172.21.15.102:0/4226329854 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c0038e0 con 0x7f03ac0c9610 2023-03-03T10:20:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.796+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ac0b1f30 con 0x7f03ac0c9610 2023-03-03T10:20:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.796+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ac0b4c00 con 0x7f03ac0c9610 2023-03-03T10:20:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.797+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ac0b2210 con 0x7f03ac0c9610 2023-03-03T10:20:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.798+0000 7f03a2ffd700 1 -- 172.21.15.102:0/4226329854 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f039c00f050 con 0x7f03ac0c9610 2023-03-03T10:20:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.800+0000 7f03a2ffd700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f038c0682f0 0x7f038c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.800+0000 7f03a2ffd700 1 -- 172.21.15.102:0/4226329854 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f039c086a90 con 0x7f03ac0c9610 2023-03-03T10:20:29.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.800+0000 7f03a3fff700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f038c0682f0 0x7f038c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.801+0000 7f03a3fff700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f038c0682f0 0x7f038c06a7a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f03ac066c30 tx=0x7f0394009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:29.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:29.804+0000 7f03a2ffd700 1 -- 172.21.15.102:0/4226329854 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f039c0531a0 con 0x7f03ac0c9610 2023-03-03T10:20:30.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.038+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03ac0677f0 con 0x7f03ac0c9610 2023-03-03T10:20:30.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.039+0000 7f03a2ffd700 1 -- 172.21.15.102:0/4226329854 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f039c020090 con 0x7f03ac0c9610 2023-03-03T10:20:30.041 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:20:30.041 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:20:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f038c0682f0 msgr2=0x7f038c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f038c0682f0 0x7f038c06a7a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f03ac066c30 tx=0x7f0394009210 comp rx=0 tx=0).stop 2023-03-03T10:20:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03ac0c9610 msgr2=0x7f03ac158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03ac0c9610 0x7f03ac158aa0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f039c009c10 tx=0x7f039c004750 comp rx=0 tx=0).stop 2023-03-03T10:20:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 shutdown_connections 2023-03-03T10:20:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac1564b0 0x7f03ac152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:30.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03ac0c9610 0x7f03ac158aa0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:30.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f038c0682f0 0x7f038c06a7a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:30.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.041+0000 7f03ab59e700 1 --2- 172.21.15.102:0/4226329854 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f03ac0ca440 0x7f03ac0ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:30.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.042+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 >> 172.21.15.102:0/4226329854 conn(0x7f03ac065fb0 msgr2=0x7f03ac0c7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:20:30.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.042+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 shutdown_connections 2023-03-03T10:20:30.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:20:30.042+0000 7f03ab59e700 1 -- 172.21.15.102:0/4226329854 wait complete. 2023-03-03T10:20:30.046 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:20:30.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T10:20:41.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:41.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:11.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:11.298 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:14.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:21:14.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.321+0000 7fd2863ff700 1 Processor -- start 2023-03-03T10:21:14.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.321+0000 7fd2863ff700 1 -- start start 2023-03-03T10:21:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.322+0000 7fd2863ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd280153290 0x7fd280158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.322+0000 7fd2863ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2800cd8e0 0x7fd2800ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.322+0000 7fd2863ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd280153660 con 0x7fd2801593c0 2023-03-03T10:21:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.322+0000 7fd2863ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28006c360 con 0x7fd280153290 2023-03-03T10:21:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.322+0000 7fd2863ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28015cea0 con 0x7fd2800cd8e0 2023-03-03T10:21:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd2801593c0 0x7fd2800cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44830/0 (socket says 172.21.15.102:44830) 2023-03-03T10:21:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27f7fe700 1 -- 172.21.15.102:0/1140095209 learned_addr learned my addr 172.21.15.102:0/1140095209 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:21:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27ffff700 1 --2- 172.21.15.102:0/1140095209 >> v2:172.21.15.102:3300/0 conn(0x7fd280153290 0x7fd280158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd28499c700 1 --2- 172.21.15.102:0/1140095209 >> v2:172.21.15.172:3300/0 conn(0x7fd2800cd8e0 0x7fd2800ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27effd700 1 -- 172.21.15.102:0/1140095209 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814808789 0 0) 0x7fd280153660 con 0x7fd2801593c0 2023-03-03T10:21:14.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27effd700 1 -- 172.21.15.102:0/1140095209 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2680035b0 con 0x7fd2801593c0 2023-03-03T10:21:14.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27ffff700 1 -- 172.21.15.102:0/1140095209 >> v2:172.21.15.172:3300/0 conn(0x7fd2800cd8e0 msgr2=0x7fd2800ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:14.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27ffff700 1 --2- 172.21.15.102:0/1140095209 >> v2:172.21.15.172:3300/0 conn(0x7fd2800cd8e0 0x7fd2800ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27ffff700 1 -- 172.21.15.102:0/1140095209 >> v1:172.21.15.172:6789/0 conn(0x7fd2801593c0 legacy=0x7fd2800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:14.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.323+0000 7fd27ffff700 1 -- 172.21.15.102:0/1140095209 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28006c7f0 con 0x7fd280153290 2023-03-03T10:21:14.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.324+0000 7fd27ffff700 1 --2- 172.21.15.102:0/1140095209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd280158e80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd2800c9390 tx=0x7fd270009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be27fabb6cdf3cf1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:14.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.324+0000 7fd27effd700 1 -- 172.21.15.102:0/1140095209 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270011830 con 0x7fd280153290 2023-03-03T10:21:14.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.324+0000 7fd27effd700 1 -- 172.21.15.102:0/1140095209 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd270011990 con 0x7fd280153290 2023-03-03T10:21:14.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.324+0000 7fd27effd700 1 -- 172.21.15.102:0/1140095209 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270011d00 con 0x7fd280153290 2023-03-03T10:21:14.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.324+0000 7fd2863ff700 1 -- 172.21.15.102:0/1140095209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 msgr2=0x7fd280158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:14.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.324+0000 7fd2863ff700 1 --2- 172.21.15.102:0/1140095209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd280158e80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd2800c9390 tx=0x7fd270009b20 comp rx=0 tx=0).stop 2023-03-03T10:21:14.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.325+0000 7fd2863ff700 1 -- 172.21.15.102:0/1140095209 shutdown_connections 2023-03-03T10:21:14.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.325+0000 7fd2863ff700 1 --2- 172.21.15.102:0/1140095209 >> v2:172.21.15.172:3300/0 conn(0x7fd2800cd8e0 0x7fd2800ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.325+0000 7fd2863ff700 1 --2- 172.21.15.102:0/1140095209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd280158e80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.325+0000 7fd2863ff700 1 -- 172.21.15.102:0/1140095209 >> 172.21.15.102:0/1140095209 conn(0x7fd280065fb0 msgr2=0x7fd2800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:14.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.325+0000 7fd2863ff700 1 -- 172.21.15.102:0/1140095209 shutdown_connections 2023-03-03T10:21:14.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.325+0000 7fd2863ff700 1 -- 172.21.15.102:0/1140095209 wait complete. 2023-03-03T10:21:14.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.327+0000 7fd2863ff700 1 Processor -- start 2023-03-03T10:21:14.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.327+0000 7fd2863ff700 1 -- start start 2023-03-03T10:21:14.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd2863ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2800cd8e0 0x7fd2800c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:14.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd2863ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd28014e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:14.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd2863ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2801593c0 0x7fd2801e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:14.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd2863ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800b2be0 con 0x7fd280153290 2023-03-03T10:21:14.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd2863ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800b1d80 con 0x7fd2801593c0 2023-03-03T10:21:14.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd2863ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800b2060 con 0x7fd2800cd8e0 2023-03-03T10:21:14.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd27f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd28014e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:14.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd28499c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2801593c0 0x7fd2801e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:14.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd27ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2800cd8e0 0x7fd2800c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:14.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd28499c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2801593c0 0x7fd2801e5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35328/0 (socket says 172.21.15.102:35328) 2023-03-03T10:21:14.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.328+0000 7fd28499c700 1 -- 172.21.15.102:0/2764837910 learned_addr learned my addr 172.21.15.102:0/2764837910 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:21:14.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd28499c700 1 -- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2800cd8e0 msgr2=0x7fd2800c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:14.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd28499c700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2800cd8e0 0x7fd2800c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd28499c700 1 -- 172.21.15.102:0/2764837910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 msgr2=0x7fd28014e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:14.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd28499c700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd28014e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd28499c700 1 -- 172.21.15.102:0/2764837910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2700094a0 con 0x7fd2801593c0 2023-03-03T10:21:14.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd27f7fe700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd28014e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:21:14.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd27ffff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2800cd8e0 0x7fd2800c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:21:14.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd28499c700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2801593c0 0x7fd2801e5740 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd27800efc0 tx=0x7fd27800c0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:14.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd27cff9700 1 -- 172.21.15.102:0/2764837910 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27800cad0 con 0x7fd2801593c0 2023-03-03T10:21:14.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd27cff9700 1 -- 172.21.15.102:0/2764837910 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27800f040 con 0x7fd2801593c0 2023-03-03T10:21:14.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd27cff9700 1 -- 172.21.15.102:0/2764837910 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27800ce70 con 0x7fd2801593c0 2023-03-03T10:21:14.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2800ccde0 con 0x7fd2801593c0 2023-03-03T10:21:14.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.329+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28015d160 con 0x7fd2801593c0 2023-03-03T10:21:14.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.330+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2800cd0c0 con 0x7fd2801593c0 2023-03-03T10:21:14.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.332+0000 7fd27cff9700 1 -- 172.21.15.102:0/2764837910 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd278014480 con 0x7fd2801593c0 2023-03-03T10:21:14.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.333+0000 7fd27cff9700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd26c0682f0 0x7fd26c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:14.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.333+0000 7fd27cff9700 1 -- 172.21.15.102:0/2764837910 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd278086670 con 0x7fd2801593c0 2023-03-03T10:21:14.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.333+0000 7fd27ffff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd26c0682f0 0x7fd26c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:14.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.334+0000 7fd27ffff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd26c0682f0 0x7fd26c06a7a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd270009b60 tx=0x7fd270008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:14.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.337+0000 7fd27cff9700 1 -- 172.21.15.102:0/2764837910 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd278052d20 con 0x7fd2801593c0 2023-03-03T10:21:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.573+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd28015bed0 con 0x7fd2801593c0 2023-03-03T10:21:14.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.574+0000 7fd27cff9700 1 -- 172.21.15.102:0/2764837910 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd278056340 con 0x7fd2801593c0 2023-03-03T10:21:14.576 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:21:14.576 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:21:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd26c0682f0 msgr2=0x7fd26c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd26c0682f0 0x7fd26c06a7a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd270009b60 tx=0x7fd270008030 comp rx=0 tx=0).stop 2023-03-03T10:21:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2801593c0 msgr2=0x7fd2801e5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2801593c0 0x7fd2801e5740 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd27800efc0 tx=0x7fd27800c0c0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 shutdown_connections 2023-03-03T10:21:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd26c0682f0 0x7fd26c06a7a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd2800cd8e0 0x7fd2800c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2801593c0 0x7fd2801e5740 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 --2- 172.21.15.102:0/2764837910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd280153290 0x7fd28014e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 >> 172.21.15.102:0/2764837910 conn(0x7fd280065fb0 msgr2=0x7fd28015ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 shutdown_connections 2023-03-03T10:21:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:14.577+0000 7fd2863ff700 1 -- 172.21.15.102:0/2764837910 wait complete. 2023-03-03T10:21:14.581 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:21:14.606 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T10:21:41.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:41.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:46.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:21:46.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.866+0000 7f1f2d3ff700 1 Processor -- start 2023-03-03T10:21:46.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.866+0000 7f1f2d3ff700 1 -- start start 2023-03-03T10:21:46.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.867+0000 7f1f2d3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f28153290 0x7f1f28158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:46.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.867+0000 7f1f2d3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f281593c0 0x7f1f280cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.867+0000 7f1f2d3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f280cd8e0 0x7f1f280ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.867+0000 7f1f2d3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f28153660 con 0x7f1f28153290 2023-03-03T10:21:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.867+0000 7f1f2d3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2806c360 con 0x7f1f280cd8e0 2023-03-03T10:21:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.867+0000 7f1f2d3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2815cea0 con 0x7f1f281593c0 2023-03-03T10:21:46.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f26ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f28153290 0x7f1f28158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f277fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f280cd8e0 0x7f1f280ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f267fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f281593c0 0x7f1f280cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f26ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f28153290 0x7f1f28158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54252/0 (socket says 172.21.15.102:54252) 2023-03-03T10:21:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f277fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f280cd8e0 0x7f1f280ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57852/0 (socket says 172.21.15.102:57852) 2023-03-03T10:21:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f26ffd700 1 -- 172.21.15.102:0/3377834231 learned_addr learned my addr 172.21.15.102:0/3377834231 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:21:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f277fe700 1 -- 172.21.15.102:0/3377834231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f281593c0 msgr2=0x7f1f280cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f277fe700 1 --2- 172.21.15.102:0/3377834231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f281593c0 0x7f1f280cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f277fe700 1 -- 172.21.15.102:0/3377834231 >> v2:172.21.15.102:3300/0 conn(0x7f1f28153290 msgr2=0x7f1f28158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f277fe700 1 --2- 172.21.15.102:0/3377834231 >> v2:172.21.15.102:3300/0 conn(0x7f1f28153290 0x7f1f28158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f277fe700 1 -- 172.21.15.102:0/3377834231 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2806c7f0 con 0x7f1f280cd8e0 2023-03-03T10:21:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.868+0000 7f1f26ffd700 1 --2- 172.21.15.102:0/3377834231 >> v2:172.21.15.102:3300/0 conn(0x7f1f28153290 0x7f1f28158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:21:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.869+0000 7f1f277fe700 1 --2- 172.21.15.102:0/3377834231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f280cd8e0 0x7f1f280ce360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1f2806d0e0 tx=0x7f1f2000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a7f2fda9e42fe44 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.869+0000 7f1f25ffb700 1 -- 172.21.15.102:0/3377834231 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2000e070 con 0x7f1f280cd8e0 2023-03-03T10:21:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.869+0000 7f1f25ffb700 1 -- 172.21.15.102:0/3377834231 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2000bb20 con 0x7f1f280cd8e0 2023-03-03T10:21:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.869+0000 7f1f25ffb700 1 -- 172.21.15.102:0/3377834231 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20012430 con 0x7f1f280cd8e0 2023-03-03T10:21:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.869+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/3377834231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f280cd8e0 msgr2=0x7f1f280ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.869+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/3377834231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f280cd8e0 0x7f1f280ce360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1f2806d0e0 tx=0x7f1f2000b330 comp rx=0 tx=0).stop 2023-03-03T10:21:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.870+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/3377834231 shutdown_connections 2023-03-03T10:21:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.870+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/3377834231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f280cd8e0 0x7f1f280ce360 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.870+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/3377834231 >> v2:172.21.15.102:3300/0 conn(0x7f1f28153290 0x7f1f28158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.870+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/3377834231 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f281593c0 0x7f1f280cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.870+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/3377834231 >> 172.21.15.102:0/3377834231 conn(0x7f1f28065fb0 msgr2=0x7f1f280663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.870+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/3377834231 shutdown_connections 2023-03-03T10:21:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.870+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/3377834231 wait complete. 2023-03-03T10:21:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.872+0000 7f1f2d3ff700 1 Processor -- start 2023-03-03T10:21:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.872+0000 7f1f2d3ff700 1 -- start start 2023-03-03T10:21:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f2d3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f280cd8e0 0x7f1f280b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f2d3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f28153290 0x7f1f281577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f2d3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f281593c0 0x7f1f28157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f2d3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280c6140 con 0x7f1f281593c0 2023-03-03T10:21:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f2d3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281eef50 con 0x7f1f28153290 2023-03-03T10:21:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f2d3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280c6370 con 0x7f1f280cd8e0 2023-03-03T10:21:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f26ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f280cd8e0 0x7f1f280b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f277fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f281593c0 0x7f1f28157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f26ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f280cd8e0 0x7f1f280b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59206/0 (socket says 172.21.15.102:59206) 2023-03-03T10:21:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f267fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f28153290 0x7f1f281577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:46.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.873+0000 7f1f26ffd700 1 -- 172.21.15.102:0/2463013514 learned_addr learned my addr 172.21.15.102:0/2463013514 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f267fc700 1 -- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f280cd8e0 msgr2=0x7f1f280b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f267fc700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f280cd8e0 0x7f1f280b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f267fc700 1 -- 172.21.15.102:0/2463013514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f281593c0 msgr2=0x7f1f28157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f267fc700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f281593c0 0x7f1f28157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f267fc700 1 -- 172.21.15.102:0/2463013514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f18009550 con 0x7f1f28153290 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f277fe700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f281593c0 0x7f1f28157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f26ffd700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f280cd8e0 0x7f1f280b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:21:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f267fc700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f28153290 0x7f1f281577a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00e900 tx=0x7f1f1c00ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:46.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f0f7fe700 1 -- 172.21.15.102:0/2463013514 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c013680 con 0x7f1f28153290 2023-03-03T10:21:46.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f0f7fe700 1 -- 172.21.15.102:0/2463013514 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c0137e0 con 0x7f1f28153290 2023-03-03T10:21:46.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f20009a70 con 0x7f1f28153290 2023-03-03T10:21:46.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f0f7fe700 1 -- 172.21.15.102:0/2463013514 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c013b50 con 0x7f1f28153290 2023-03-03T10:21:46.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.874+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f280ccde0 con 0x7f1f28153290 2023-03-03T10:21:46.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.876+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2815d160 con 0x7f1f28153290 2023-03-03T10:21:46.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.877+0000 7f1f0f7fe700 1 -- 172.21.15.102:0/2463013514 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f1c019030 con 0x7f1f28153290 2023-03-03T10:21:46.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.878+0000 7f1f0f7fe700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f140682f0 0x7f1f1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:46.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.878+0000 7f1f0f7fe700 1 -- 172.21.15.102:0/2463013514 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1f1c085b70 con 0x7f1f28153290 2023-03-03T10:21:46.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.879+0000 7f1f26ffd700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f140682f0 0x7f1f1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:46.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.879+0000 7f1f26ffd700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f140682f0 0x7f1f1406a7a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1f180114b0 tx=0x7f1f180092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:46.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:46.882+0000 7f1f0f7fe700 1 -- 172.21.15.102:0/2463013514 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f1c052280 con 0x7f1f28153290 2023-03-03T10:21:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.117+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f280cd0c0 con 0x7f1f28153290 2023-03-03T10:21:47.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.118+0000 7f1f0f7fe700 1 -- 172.21.15.102:0/2463013514 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1f1c017090 con 0x7f1f28153290 2023-03-03T10:21:47.120 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:21:47.121 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:21:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f140682f0 msgr2=0x7f1f1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f140682f0 0x7f1f1406a7a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1f180114b0 tx=0x7f1f180092c0 comp rx=0 tx=0).stop 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f28153290 msgr2=0x7f1f281577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f28153290 0x7f1f281577a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00e900 tx=0x7f1f1c00ec10 comp rx=0 tx=0).stop 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 shutdown_connections 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f140682f0 0x7f1f1406a7a0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f280cd8e0 0x7f1f280b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f281593c0 0x7f1f28157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 --2- 172.21.15.102:0/2463013514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f28153290 0x7f1f281577a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 >> 172.21.15.102:0/2463013514 conn(0x7f1f28065fb0 msgr2=0x7f1f2815ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 shutdown_connections 2023-03-03T10:21:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:21:47.121+0000 7f1f2d3ff700 1 -- 172.21.15.102:0/2463013514 wait complete. 2023-03-03T10:21:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:21:47.150 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-03-03T10:22:11.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:11.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:41.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:41.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:12.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:12.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:12.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:23:12.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.407+0000 7ffb86b9d700 1 Processor -- start 2023-03-03T10:23:12.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.407+0000 7ffb86b9d700 1 -- start start 2023-03-03T10:23:12.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb86b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb800ced00 0x7ffb800c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:12.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb86b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800b3970 0x7ffb800c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:12.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb86b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb800c3d20 con 0x7ffb800cf0d0 2023-03-03T10:23:12.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb86b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb800c4930 con 0x7ffb800ced00 2023-03-03T10:23:12.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb86b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8015cba0 con 0x7ffb800b3970 2023-03-03T10:23:12.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb85b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb800ced00 0x7ffb800c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:12.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb8539a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800b3970 0x7ffb800c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:12.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb8639c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ffb800cf0d0 0x7ffb80158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44200/0 (socket says 172.21.15.102:44200) 2023-03-03T10:23:12.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb85b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb800ced00 0x7ffb800c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43146/0 (socket says 172.21.15.102:43146) 2023-03-03T10:23:12.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.409+0000 7ffb8639c700 1 -- 172.21.15.102:0/2787730234 learned_addr learned my addr 172.21.15.102:0/2787730234 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:23:12.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb85b9b700 1 -- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800b3970 msgr2=0x7ffb800c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:23:12.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb85b9b700 1 --2- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800b3970 0x7ffb800c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb85b9b700 1 -- 172.21.15.102:0/2787730234 >> v1:172.21.15.172:6789/0 conn(0x7ffb800cf0d0 legacy=0x7ffb80158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:12.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb85b9b700 1 -- 172.21.15.102:0/2787730234 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8015ce80 con 0x7ffb800ced00 2023-03-03T10:23:12.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb8539a700 1 --2- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800b3970 0x7ffb800c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:23:12.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb84b99700 1 -- 172.21.15.102:0/2787730234 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106016697 0 0) 0x7ffb800c3d20 con 0x7ffb800cf0d0 2023-03-03T10:23:12.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb85b9b700 1 --2- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb800c5a90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009840 tx=0x7ffb7c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded76ff6a8953778 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:12.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb84b99700 1 -- 172.21.15.102:0/2787730234 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c011830 con 0x7ffb800ced00 2023-03-03T10:23:12.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb84b99700 1 -- 172.21.15.102:0/2787730234 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c011990 con 0x7ffb800ced00 2023-03-03T10:23:12.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.410+0000 7ffb84b99700 1 -- 172.21.15.102:0/2787730234 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c011d20 con 0x7ffb800ced00 2023-03-03T10:23:12.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 -- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 msgr2=0x7ffb800c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:12.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb800c5a90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009840 tx=0x7ffb7c009b50 comp rx=0 tx=0).stop 2023-03-03T10:23:12.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 -- 172.21.15.102:0/2787730234 shutdown_connections 2023-03-03T10:23:12.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb800c5a90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/2787730234 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800b3970 0x7ffb800c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 -- 172.21.15.102:0/2787730234 >> 172.21.15.102:0/2787730234 conn(0x7ffb8008b1c0 msgr2=0x7ffb80071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:23:12.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 -- 172.21.15.102:0/2787730234 shutdown_connections 2023-03-03T10:23:12.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.411+0000 7ffb86b9d700 1 -- 172.21.15.102:0/2787730234 wait complete. 2023-03-03T10:23:12.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.413+0000 7ffb86b9d700 1 Processor -- start 2023-03-03T10:23:12.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.413+0000 7ffb86b9d700 1 -- start start 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.414+0000 7ffb86b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb800b3970 0x7ffb80150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.414+0000 7ffb86b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb801f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.414+0000 7ffb86b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800cf0d0 0x7ffb801547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.414+0000 7ffb86b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80153520 con 0x7ffb800b3970 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.414+0000 7ffb86b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801f1870 con 0x7ffb800ced00 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.414+0000 7ffb86b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801537c0 con 0x7ffb800cf0d0 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb800b3970 0x7ffb80150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb8639c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800cf0d0 0x7ffb801547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb8539a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb801f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb800b3970 0x7ffb80150970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54394/0 (socket says 172.21.15.102:54394) 2023-03-03T10:23:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 -- 172.21.15.102:0/736043213 learned_addr learned my addr 172.21.15.102:0/736043213 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:23:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 -- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800cf0d0 msgr2=0x7ffb801547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800cf0d0 0x7ffb801547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 -- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 msgr2=0x7ffb801f2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb801f2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb85b9b700 1 -- 172.21.15.102:0/736043213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700094a0 con 0x7ffb800b3970 2023-03-03T10:23:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb8539a700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb801f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:23:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.415+0000 7ffb8639c700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800cf0d0 0x7ffb801547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:23:12.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.416+0000 7ffb85b9b700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb800b3970 0x7ffb80150970 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009b90 tx=0x7ffb7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.416+0000 7ffb767fc700 1 -- 172.21.15.102:0/736043213 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c0036a0 con 0x7ffb800b3970 2023-03-03T10:23:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.416+0000 7ffb767fc700 1 -- 172.21.15.102:0/736043213 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c01f070 con 0x7ffb800b3970 2023-03-03T10:23:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.416+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7c0094a0 con 0x7ffb800b3970 2023-03-03T10:23:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.416+0000 7ffb767fc700 1 -- 172.21.15.102:0/736043213 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c003a40 con 0x7ffb800b3970 2023-03-03T10:23:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.416+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8006c200 con 0x7ffb800b3970 2023-03-03T10:23:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.417+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8006bf20 con 0x7ffb800b3970 2023-03-03T10:23:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.418+0000 7ffb767fc700 1 -- 172.21.15.102:0/736043213 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb7c00f050 con 0x7ffb800b3970 2023-03-03T10:23:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.419+0000 7ffb767fc700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb64068320 0x7ffb6406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:12.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.419+0000 7ffb767fc700 1 -- 172.21.15.102:0/736043213 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ffb7c021070 con 0x7ffb800b3970 2023-03-03T10:23:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.420+0000 7ffb8539a700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb64068320 0x7ffb6406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.421+0000 7ffb8539a700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb64068320 0x7ffb6406a7d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ffb800c5c40 tx=0x7ffb70009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:12.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.424+0000 7ffb767fc700 1 -- 172.21.15.102:0/736043213 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb7c053c90 con 0x7ffb800b3970 2023-03-03T10:23:12.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.668+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb800cf540 con 0x7ffb800b3970 2023-03-03T10:23:12.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.669+0000 7ffb767fc700 1 -- 172.21.15.102:0/736043213 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ffb7c0193b0 con 0x7ffb800b3970 2023-03-03T10:23:12.671 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:23:12.671 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:23:12.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb64068320 msgr2=0x7ffb6406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:12.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb64068320 0x7ffb6406a7d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ffb800c5c40 tx=0x7ffb70009210 comp rx=0 tx=0).stop 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb800b3970 msgr2=0x7ffb80150970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb800b3970 0x7ffb80150970 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009b90 tx=0x7ffb7c004750 comp rx=0 tx=0).stop 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 shutdown_connections 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb64068320 0x7ffb6406a7d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb800cf0d0 0x7ffb801547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb800b3970 0x7ffb80150970 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 --2- 172.21.15.102:0/736043213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb800ced00 0x7ffb801f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:12.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 >> 172.21.15.102:0/736043213 conn(0x7ffb8008b1c0 msgr2=0x7ffb800b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:23:12.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 shutdown_connections 2023-03-03T10:23:12.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:23:12.672+0000 7ffb86b9d700 1 -- 172.21.15.102:0/736043213 wait complete. 2023-03-03T10:23:12.676 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:23:12.701 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T10:23:42.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:42.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:08.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:24:08.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.950+0000 7f72ba3ff700 1 Processor -- start 2023-03-03T10:24:08.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.951+0000 7f72ba3ff700 1 -- start start 2023-03-03T10:24:08.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.952+0000 7f72ba3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:08.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.952+0000 7f72ba3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:08.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.952+0000 7f72ba3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f72b40cd8e0 0x7f72b40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:08.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.952+0000 7f72ba3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b4153660 con 0x7f72b4153290 2023-03-03T10:24:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.952+0000 7f72ba3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b406c360 con 0x7f72b40cd8e0 2023-03-03T10:24:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.952+0000 7f72ba3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b415cea0 con 0x7f72b41593c0 2023-03-03T10:24:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.952+0000 7f72b3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:08.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56086/0 (socket says 172.21.15.102:56086) 2023-03-03T10:24:08.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 -- 172.21.15.102:0/3101790830 learned_addr learned my addr 172.21.15.102:0/3101790830 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:24:08.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72abfff700 1 --2- 172.21.15.102:0/3101790830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:08.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b899c700 1 --2- 172.21.15.102:0/3101790830 >> v2:172.21.15.172:3300/0 conn(0x7f72b40cd8e0 0x7f72b40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:08.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 -- 172.21.15.102:0/3101790830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b41593c0 msgr2=0x7f72b40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:24:08.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 --2- 172.21.15.102:0/3101790830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 -- 172.21.15.102:0/3101790830 >> v2:172.21.15.172:3300/0 conn(0x7f72b40cd8e0 msgr2=0x7f72b40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:24:08.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 --2- 172.21.15.102:0/3101790830 >> v2:172.21.15.172:3300/0 conn(0x7f72b40cd8e0 0x7f72b40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 -- 172.21.15.102:0/3101790830 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b406c7f0 con 0x7f72b4153290 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.953+0000 7f72b3fff700 1 --2- 172.21.15.102:0/3101790830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 0x7f72b4158e80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f72b406d0e0 tx=0x7f72a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2154fc3c20335b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72b37fe700 1 -- 172.21.15.102:0/3101790830 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4011830 con 0x7f72b4153290 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72b37fe700 1 -- 172.21.15.102:0/3101790830 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a4011990 con 0x7f72b4153290 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72b37fe700 1 -- 172.21.15.102:0/3101790830 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4011d20 con 0x7f72b4153290 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72ba3ff700 1 -- 172.21.15.102:0/3101790830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 msgr2=0x7f72b4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/3101790830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 0x7f72b4158e80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f72b406d0e0 tx=0x7f72a4009bd0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72ba3ff700 1 -- 172.21.15.102:0/3101790830 shutdown_connections 2023-03-03T10:24:08.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/3101790830 >> v2:172.21.15.172:3300/0 conn(0x7f72b40cd8e0 0x7f72b40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/3101790830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/3101790830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.954+0000 7f72ba3ff700 1 -- 172.21.15.102:0/3101790830 >> 172.21.15.102:0/3101790830 conn(0x7f72b4065fb0 msgr2=0x7f72b40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:08.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.955+0000 7f72ba3ff700 1 -- 172.21.15.102:0/3101790830 shutdown_connections 2023-03-03T10:24:08.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.955+0000 7f72ba3ff700 1 -- 172.21.15.102:0/3101790830 wait complete. 2023-03-03T10:24:08.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.956+0000 7f72ba3ff700 1 Processor -- start 2023-03-03T10:24:08.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.957+0000 7f72ba3ff700 1 -- start start 2023-03-03T10:24:08.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72ba3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b4158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:08.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72ba3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:08.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72ba3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b4157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:08.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72ba3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41e31d0 con 0x7f72b41593c0 2023-03-03T10:24:08.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72ba3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4154da0 con 0x7f72b40cd8e0 2023-03-03T10:24:08.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72ba3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4153890 con 0x7f72b4153290 2023-03-03T10:24:08.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b899c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b4157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b4158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46978/0 (socket says 172.21.15.102:46978) 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b3fff700 1 -- 172.21.15.102:0/942545524 learned_addr learned my addr 172.21.15.102:0/942545524 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72abfff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b899c700 1 -- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 msgr2=0x7f72b41564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b899c700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b899c700 1 -- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 msgr2=0x7f72b4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:08.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b899c700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b4158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:08.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b899c700 1 -- 172.21.15.102:0/942545524 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4009550 con 0x7f72b41593c0 2023-03-03T10:24:08.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.958+0000 7f72b3fff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:24:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.959+0000 7f72b899c700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b4157a00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00eba0 tx=0x7f72ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.959+0000 7f72b17fa700 1 -- 172.21.15.102:0/942545524 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac009a80 con 0x7f72b41593c0 2023-03-03T10:24:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.959+0000 7f72b17fa700 1 -- 172.21.15.102:0/942545524 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac00f040 con 0x7f72b41593c0 2023-03-03T10:24:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.959+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b40ccde0 con 0x7f72b41593c0 2023-03-03T10:24:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.959+0000 7f72b17fa700 1 -- 172.21.15.102:0/942545524 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac00cc40 con 0x7f72b41593c0 2023-03-03T10:24:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.959+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b415d160 con 0x7f72b41593c0 2023-03-03T10:24:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.960+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b40cd0c0 con 0x7f72b41593c0 2023-03-03T10:24:08.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.961+0000 7f72b17fa700 1 -- 172.21.15.102:0/942545524 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72ac014480 con 0x7f72b41593c0 2023-03-03T10:24:08.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.963+0000 7f72b17fa700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72940682f0 0x7f729406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:08.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.963+0000 7f72b17fa700 1 -- 172.21.15.102:0/942545524 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f72ac0868b0 con 0x7f72b41593c0 2023-03-03T10:24:08.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.963+0000 7f72b3fff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72940682f0 0x7f729406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:08.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.964+0000 7f72b3fff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72940682f0 0x7f729406a7a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009fd0 tx=0x7f72a4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:08.967+0000 7f72b17fa700 1 -- 172.21.15.102:0/942545524 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72ac052f60 con 0x7f72b41593c0 2023-03-03T10:24:09.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.201+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72b404e6e0 con 0x7f72b41593c0 2023-03-03T10:24:09.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.201+0000 7f72b17fa700 1 -- 172.21.15.102:0/942545524 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f72ac056580 con 0x7f72b41593c0 2023-03-03T10:24:09.204 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:24:09.204 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:24:09.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72940682f0 msgr2=0x7f729406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:09.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72940682f0 0x7f729406a7a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009fd0 tx=0x7f72a4008030 comp rx=0 tx=0).stop 2023-03-03T10:24:09.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 msgr2=0x7f72b4157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b4157a00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00eba0 tx=0x7f72ac00eeb0 comp rx=0 tx=0).stop 2023-03-03T10:24:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 shutdown_connections 2023-03-03T10:24:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b4157a00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72940682f0 0x7f729406a7a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 --2- 172.21.15.102:0/942545524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 >> 172.21.15.102:0/942545524 conn(0x7f72b4065fb0 msgr2=0x7f72b40cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:09.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 shutdown_connections 2023-03-03T10:24:09.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:09.204+0000 7f72ba3ff700 1 -- 172.21.15.102:0/942545524 wait complete. 2023-03-03T10:24:09.208 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:24:09.233 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 25 secs before thrashing 2023-03-03T10:24:12.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:12.602 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:34.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:24:34.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.479+0000 7fdc7c5ff700 1 Processor -- start 2023-03-03T10:24:34.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.480+0000 7fdc7c5ff700 1 -- start start 2023-03-03T10:24:34.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc7c5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdc740cd8e0 0x7fdc740cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:34.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc7c5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc74153660 con 0x7fdc74153290 2023-03-03T10:24:34.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc7c5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7406c360 con 0x7fdc740cd390 2023-03-03T10:24:34.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc7c5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7415cea0 con 0x7fdc740cd8e0 2023-03-03T10:24:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc79b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdc740cd8e0 0x7fdc740cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc7ab9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdc740cd390 0x7fdc740ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56660/0 (socket says 172.21.15.102:56660) 2023-03-03T10:24:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc7ab9c700 1 -- 172.21.15.102:0/1465162877 learned_addr learned my addr 172.21.15.102:0/1465162877 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:24:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.481+0000 7fdc79b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdc740cd8e0 0x7fdc740cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56470/0 (socket says 172.21.15.102:56470) 2023-03-03T10:24:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.482+0000 7fdc79399700 1 -- 172.21.15.102:0/1465162877 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720682347 0 0) 0x7fdc7406c360 con 0x7fdc740cd390 2023-03-03T10:24:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.482+0000 7fdc79399700 1 -- 172.21.15.102:0/1465162877 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc600035b0 con 0x7fdc740cd390 2023-03-03T10:24:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.482+0000 7fdc79b9a700 1 -- 172.21.15.102:0/1465162877 >> v1:172.21.15.172:6789/0 conn(0x7fdc740cd390 legacy=0x7fdc740ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.482+0000 7fdc79399700 1 -- 172.21.15.102:0/1465162877 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944788077 0 0) 0x7fdc74153660 con 0x7fdc74153290 2023-03-03T10:24:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.482+0000 7fdc79b9a700 1 -- 172.21.15.102:0/1465162877 >> v1:172.21.15.102:6789/0 conn(0x7fdc74153290 legacy=0x7fdc74158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.482+0000 7fdc79b9a700 1 -- 172.21.15.102:0/1465162877 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7406c7f0 con 0x7fdc740cd8e0 2023-03-03T10:24:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.482+0000 7fdc79b9a700 1 --2- 172.21.15.102:0/1465162877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc740cd8e0 0x7fdc740cde20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009840 tx=0x7fdc64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88a6e546e7c9088 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:34.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc79399700 1 -- 172.21.15.102:0/1465162877 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64011830 con 0x7fdc740cd8e0 2023-03-03T10:24:34.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc79399700 1 -- 172.21.15.102:0/1465162877 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc64011990 con 0x7fdc740cd8e0 2023-03-03T10:24:34.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc79399700 1 -- 172.21.15.102:0/1465162877 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64011d20 con 0x7fdc740cd8e0 2023-03-03T10:24:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1465162877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc740cd8e0 msgr2=0x7fdc740cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1465162877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc740cd8e0 0x7fdc740cde20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009840 tx=0x7fdc64009b50 comp rx=0 tx=0).stop 2023-03-03T10:24:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1465162877 shutdown_connections 2023-03-03T10:24:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1465162877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc740cd8e0 0x7fdc740cde20 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1465162877 >> 172.21.15.102:0/1465162877 conn(0x7fdc74065fb0 msgr2=0x7fdc740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1465162877 shutdown_connections 2023-03-03T10:24:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.483+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1465162877 wait complete. 2023-03-03T10:24:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.485+0000 7fdc7c5ff700 1 Processor -- start 2023-03-03T10:24:34.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.486+0000 7fdc7c5ff700 1 -- start start 2023-03-03T10:24:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7c5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc740cd390 0x7fdc741e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7c5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc740cd8e0 0x7fdc741500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7c5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc74153290 0x7fdc74150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7c5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc74158a80 con 0x7fdc74153290 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7c5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7414f520 con 0x7fdc740cd390 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7c5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc740c57c0 con 0x7fdc740cd8e0 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7ab9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc74153290 0x7fdc74150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7a39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc740cd390 0x7fdc741e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc79b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc740cd8e0 0x7fdc741500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7a39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc740cd390 0x7fdc741e54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36512/0 (socket says 172.21.15.102:36512) 2023-03-03T10:24:34.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.487+0000 7fdc7a39b700 1 -- 172.21.15.102:0/1977655608 learned_addr learned my addr 172.21.15.102:0/1977655608 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:24:34.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7a39b700 1 -- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc740cd8e0 msgr2=0x7fdc741500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:34.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7a39b700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc740cd8e0 0x7fdc741500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:34.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7a39b700 1 -- 172.21.15.102:0/1977655608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc74153290 msgr2=0x7fdc74150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:34.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7a39b700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc74153290 0x7fdc74150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:34.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7a39b700 1 -- 172.21.15.102:0/1977655608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc640094a0 con 0x7fdc740cd390 2023-03-03T10:24:34.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7a39b700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc740cd390 0x7fdc741e54b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000c550 tx=0x7fdc7000c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:34.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc6affd700 1 -- 172.21.15.102:0/1977655608 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc700054e0 con 0x7fdc740cd390 2023-03-03T10:24:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc6affd700 1 -- 172.21.15.102:0/1977655608 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc70003680 con 0x7fdc740cd390 2023-03-03T10:24:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc6affd700 1 -- 172.21.15.102:0/1977655608 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70003a30 con 0x7fdc740cd390 2023-03-03T10:24:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc740ccde0 con 0x7fdc740cd390 2023-03-03T10:24:34.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.488+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7415d160 con 0x7fdc740cd390 2023-03-03T10:24:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.490+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc740cd0c0 con 0x7fdc740cd390 2023-03-03T10:24:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.491+0000 7fdc6affd700 1 -- 172.21.15.102:0/1977655608 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc70019030 con 0x7fdc740cd390 2023-03-03T10:24:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.492+0000 7fdc6affd700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc580682f0 0x7fdc5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.492+0000 7fdc6affd700 1 -- 172.21.15.102:0/1977655608 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdc70084ff0 con 0x7fdc740cd390 2023-03-03T10:24:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.492+0000 7fdc79b9a700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc580682f0 0x7fdc5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:34.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.493+0000 7fdc79b9a700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc580682f0 0x7fdc5806a7a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009fd0 tx=0x7fdc6401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:34.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.497+0000 7fdc6affd700 1 -- 172.21.15.102:0/1977655608 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc70051720 con 0x7fdc740cd390 2023-03-03T10:24:34.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.733+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc741ef2e0 con 0x7fdc740cd390 2023-03-03T10:24:34.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.734+0000 7fdc6affd700 1 -- 172.21.15.102:0/1977655608 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdc70017070 con 0x7fdc740cd390 2023-03-03T10:24:34.737 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:24:34.737 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:24:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc580682f0 msgr2=0x7fdc5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc580682f0 0x7fdc5806a7a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009fd0 tx=0x7fdc6401c040 comp rx=0 tx=0).stop 2023-03-03T10:24:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc740cd390 msgr2=0x7fdc741e54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc740cd390 0x7fdc741e54b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000c550 tx=0x7fdc7000c860 comp rx=0 tx=0).stop 2023-03-03T10:24:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 shutdown_connections 2023-03-03T10:24:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc74153290 0x7fdc74150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc740cd390 0x7fdc741e54b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc580682f0 0x7fdc5806a7a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:34.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 --2- 172.21.15.102:0/1977655608 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc740cd8e0 0x7fdc741500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:34.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 >> 172.21.15.102:0/1977655608 conn(0x7fdc74065fb0 msgr2=0x7fdc74156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:34.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 shutdown_connections 2023-03-03T10:24:34.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:24:34.736+0000 7fdc7c5ff700 1 -- 172.21.15.102:0/1977655608 wait complete. 2023-03-03T10:24:34.741 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:24:34.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-03-03T10:24:42.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:42.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:12.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:12.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:43.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:43.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:03.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:26:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.015+0000 7fececbff700 1 Processor -- start 2023-03-03T10:26:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.015+0000 7fececbff700 1 -- start start 2023-03-03T10:26:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.016+0000 7fececbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fece8153290 0x7fece8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.016+0000 7fececbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cd8e0 0x7fece80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:04.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.016+0000 7fececbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece8153660 con 0x7fece81593c0 2023-03-03T10:26:04.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.016+0000 7fececbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece806c360 con 0x7fece8153290 2023-03-03T10:26:04.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.016+0000 7fececbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece815cea0 con 0x7fece80cd8e0 2023-03-03T10:26:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fece8153290 0x7fece8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece5d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fece81593c0 0x7fece80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59204/0 (socket says 172.21.15.102:59204) 2023-03-03T10:26:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece6d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cd8e0 0x7fece80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece5d9b700 1 -- 172.21.15.102:0/4288477772 learned_addr learned my addr 172.21.15.102:0/4288477772 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:26:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece559a700 1 -- 172.21.15.102:0/4288477772 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522347093 0 0) 0x7fece8153660 con 0x7fece81593c0 2023-03-03T10:26:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece6d9d700 1 -- 172.21.15.102:0/4288477772 >> v2:172.21.15.102:3300/0 conn(0x7fece8153290 msgr2=0x7fece8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:26:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece6d9d700 1 --2- 172.21.15.102:0/4288477772 >> v2:172.21.15.102:3300/0 conn(0x7fece8153290 0x7fece8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece6d9d700 1 -- 172.21.15.102:0/4288477772 >> v1:172.21.15.172:6789/0 conn(0x7fece81593c0 legacy=0x7fece80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece6d9d700 1 -- 172.21.15.102:0/4288477772 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece806c7f0 con 0x7fece80cd8e0 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.017+0000 7fece6d9d700 1 --2- 172.21.15.102:0/4288477772 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cd8e0 0x7fece80ce530 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fece0009d90 tx=0x7fece000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb9b0cb1325feea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fece559a700 1 -- 172.21.15.102:0/4288477772 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece000e070 con 0x7fece80cd8e0 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fece559a700 1 -- 172.21.15.102:0/4288477772 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece000bb20 con 0x7fece80cd8e0 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fece559a700 1 -- 172.21.15.102:0/4288477772 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0012430 con 0x7fece80cd8e0 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fececbff700 1 -- 172.21.15.102:0/4288477772 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cd8e0 msgr2=0x7fece80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fececbff700 1 --2- 172.21.15.102:0/4288477772 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cd8e0 0x7fece80ce530 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fece0009d90 tx=0x7fece000b330 comp rx=0 tx=0).stop 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fececbff700 1 -- 172.21.15.102:0/4288477772 shutdown_connections 2023-03-03T10:26:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fececbff700 1 --2- 172.21.15.102:0/4288477772 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece80cd8e0 0x7fece80ce530 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fececbff700 1 --2- 172.21.15.102:0/4288477772 >> v2:172.21.15.102:3300/0 conn(0x7fece8153290 0x7fece8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.018+0000 7fececbff700 1 -- 172.21.15.102:0/4288477772 >> 172.21.15.102:0/4288477772 conn(0x7fece8065fb0 msgr2=0x7fece80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:26:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.019+0000 7fececbff700 1 -- 172.21.15.102:0/4288477772 shutdown_connections 2023-03-03T10:26:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.019+0000 7fececbff700 1 -- 172.21.15.102:0/4288477772 wait complete. 2023-03-03T10:26:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.021+0000 7fececbff700 1 Processor -- start 2023-03-03T10:26:04.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.022+0000 7fececbff700 1 -- start start 2023-03-03T10:26:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fececbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece80cd8e0 0x7fece80c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fececbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece8153290 0x7fece81f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fececbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fece81593c0 0x7fece80c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fececbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80c83d0 con 0x7fece81593c0 2023-03-03T10:26:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fececbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80cbe90 con 0x7fece80cd8e0 2023-03-03T10:26:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fececbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece81e60d0 con 0x7fece8153290 2023-03-03T10:26:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fece6d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fece81593c0 0x7fece80c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fece659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece80cd8e0 0x7fece80c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fece5d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece8153290 0x7fece81f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fece6d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fece81593c0 0x7fece80c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51808/0 (socket says 172.21.15.102:51808) 2023-03-03T10:26:04.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.023+0000 7fece6d9d700 1 -- 172.21.15.102:0/3325621294 learned_addr learned my addr 172.21.15.102:0/3325621294 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:26:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fece6d9d700 1 -- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece8153290 msgr2=0x7fece81f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fece6d9d700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece8153290 0x7fece81f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fece6d9d700 1 -- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece80cd8e0 msgr2=0x7fece80c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fece6d9d700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece80cd8e0 0x7fece80c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fece6d9d700 1 -- 172.21.15.102:0/3325621294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0009a70 con 0x7fece81593c0 2023-03-03T10:26:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fece6d9d700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fece81593c0 0x7fece80c6ae0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fece815cfe0 tx=0x7fece0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:26:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fecd6ffd700 1 -- 172.21.15.102:0/3325621294 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0012430 con 0x7fece81593c0 2023-03-03T10:26:04.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fecd6ffd700 1 -- 172.21.15.102:0/3325621294 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece0020070 con 0x7fece81593c0 2023-03-03T10:26:04.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fecd6ffd700 1 -- 172.21.15.102:0/3325621294 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0004280 con 0x7fece81593c0 2023-03-03T10:26:04.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.024+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece80ccde0 con 0x7fece81593c0 2023-03-03T10:26:04.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.025+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece815d160 con 0x7fece81593c0 2023-03-03T10:26:04.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.026+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece80cd0c0 con 0x7fece81593c0 2023-03-03T10:26:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.027+0000 7fecd6ffd700 1 -- 172.21.15.102:0/3325621294 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fece002e070 con 0x7fece81593c0 2023-03-03T10:26:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.028+0000 7fecd6ffd700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecc80682f0 0x7fecc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.028+0000 7fecd6ffd700 1 -- 172.21.15.102:0/3325621294 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fece0022070 con 0x7fece81593c0 2023-03-03T10:26:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.028+0000 7fece659c700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecc80682f0 0x7fecc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.029+0000 7fece659c700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecc80682f0 0x7fecc806a7a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009b40 tx=0x7fecd8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:26:04.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.032+0000 7fecd6ffd700 1 -- 172.21.15.102:0/3325621294 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fece00540f0 con 0x7fece81593c0 2023-03-03T10:26:04.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.265+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fece804e6e0 con 0x7fece81593c0 2023-03-03T10:26:04.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.266+0000 7fecd6ffd700 1 -- 172.21.15.102:0/3325621294 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fece0057710 con 0x7fece81593c0 2023-03-03T10:26:04.269 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:26:04.269 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:26:04.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecc80682f0 msgr2=0x7fecc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:04.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecc80682f0 0x7fecc806a7a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009b40 tx=0x7fecd8009210 comp rx=0 tx=0).stop 2023-03-03T10:26:04.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fece81593c0 msgr2=0x7fece80c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:04.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fece81593c0 0x7fece80c6ae0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fece815cfe0 tx=0x7fece0004ad0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 shutdown_connections 2023-03-03T10:26:04.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fece81593c0 0x7fece80c6ae0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fece80cd8e0 0x7fece80c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecc80682f0 0x7fecc806a7a0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 --2- 172.21.15.102:0/3325621294 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fece8153290 0x7fece81f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:04.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 >> 172.21.15.102:0/3325621294 conn(0x7fece8065fb0 msgr2=0x7fece8156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:26:04.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.268+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 shutdown_connections 2023-03-03T10:26:04.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:26:04.269+0000 7fececbff700 1 -- 172.21.15.102:0/3325621294 wait complete. 2023-03-03T10:26:04.273 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:26:04.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T10:26:13.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:13.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:43.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:43.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:13.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:13.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:28.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:27:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.550+0000 7f69a91ff700 1 Processor -- start 2023-03-03T10:27:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.550+0000 7f69a91ff700 1 -- start start 2023-03-03T10:27:28.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.551+0000 7f69a91ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:28.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.551+0000 7f69a91ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f69a40cd390 0x7f69a40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:28.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.551+0000 7f69a91ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a4153660 con 0x7f69a4153290 2023-03-03T10:27:28.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.551+0000 7f69a91ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a406c360 con 0x7f69a40cd8e0 2023-03-03T10:27:28.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.551+0000 7f69a91ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a415cea0 con 0x7f69a40cd390 2023-03-03T10:27:28.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a259c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:28.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a2d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f69a4153290 0x7f69a4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49028/0 (socket says 172.21.15.102:49028) 2023-03-03T10:27:28.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a2d9d700 1 -- 172.21.15.102:0/2987830522 learned_addr learned my addr 172.21.15.102:0/2987830522 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:27:28.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a359e700 1 --2- 172.21.15.102:0/2987830522 >> v2:172.21.15.172:3300/0 conn(0x7f69a40cd390 0x7f69a40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a1d9b700 1 -- 172.21.15.102:0/2987830522 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52431021 0 0) 0x7f69a4153660 con 0x7f69a4153290 2023-03-03T10:27:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a1d9b700 1 -- 172.21.15.102:0/2987830522 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f698c0035b0 con 0x7f69a4153290 2023-03-03T10:27:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a259c700 1 -- 172.21.15.102:0/2987830522 >> v2:172.21.15.172:3300/0 conn(0x7f69a40cd390 msgr2=0x7f69a40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:27:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a259c700 1 --2- 172.21.15.102:0/2987830522 >> v2:172.21.15.172:3300/0 conn(0x7f69a40cd390 0x7f69a40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a259c700 1 -- 172.21.15.102:0/2987830522 >> v1:172.21.15.172:6789/0 conn(0x7f69a4153290 legacy=0x7f69a4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.552+0000 7f69a259c700 1 -- 172.21.15.102:0/2987830522 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a406c7f0 con 0x7f69a40cd8e0 2023-03-03T10:27:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.553+0000 7f69a259c700 1 --2- 172.21.15.102:0/2987830522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 0x7f69a40cde20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f69a40c9390 tx=0x7f6998009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77bd07505797dee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:27:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.553+0000 7f69a1d9b700 1 -- 172.21.15.102:0/2987830522 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998011830 con 0x7f69a40cd8e0 2023-03-03T10:27:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.553+0000 7f69a1d9b700 1 -- 172.21.15.102:0/2987830522 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6998011990 con 0x7f69a40cd8e0 2023-03-03T10:27:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.553+0000 7f69a1d9b700 1 -- 172.21.15.102:0/2987830522 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998011d00 con 0x7f69a40cd8e0 2023-03-03T10:27:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.553+0000 7f69a91ff700 1 -- 172.21.15.102:0/2987830522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 msgr2=0x7f69a40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.553+0000 7f69a91ff700 1 --2- 172.21.15.102:0/2987830522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 0x7f69a40cde20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f69a40c9390 tx=0x7f6998009b20 comp rx=0 tx=0).stop 2023-03-03T10:27:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.554+0000 7f69a91ff700 1 -- 172.21.15.102:0/2987830522 shutdown_connections 2023-03-03T10:27:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.554+0000 7f69a91ff700 1 --2- 172.21.15.102:0/2987830522 >> v2:172.21.15.172:3300/0 conn(0x7f69a40cd390 0x7f69a40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.554+0000 7f69a91ff700 1 --2- 172.21.15.102:0/2987830522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 0x7f69a40cde20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.554+0000 7f69a91ff700 1 -- 172.21.15.102:0/2987830522 >> 172.21.15.102:0/2987830522 conn(0x7f69a4065fb0 msgr2=0x7f69a40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:27:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.554+0000 7f69a91ff700 1 -- 172.21.15.102:0/2987830522 shutdown_connections 2023-03-03T10:27:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.554+0000 7f69a91ff700 1 -- 172.21.15.102:0/2987830522 wait complete. 2023-03-03T10:27:28.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.556+0000 7f69a91ff700 1 Processor -- start 2023-03-03T10:27:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.556+0000 7f69a91ff700 1 -- start start 2023-03-03T10:27:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a91ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a91ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 0x7f69a41e4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a91ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4153290 0x7f69a41e4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a91ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a41e2b60 con 0x7f69a40cd8e0 2023-03-03T10:27:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a91ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a40b27a0 con 0x7f69a4153290 2023-03-03T10:27:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a91ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a4158380 con 0x7f69a40cd390 2023-03-03T10:27:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 0x7f69a41e4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4153290 0x7f69a41e4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4153290 0x7f69a41e4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55668/0 (socket says 172.21.15.102:55668) 2023-03-03T10:27:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.557+0000 7f69a359e700 1 -- 172.21.15.102:0/3712470248 learned_addr learned my addr 172.21.15.102:0/3712470248 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:27:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f69a359e700 1 -- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 msgr2=0x7f69a40b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f69a359e700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f69a359e700 1 -- 172.21.15.102:0/3712470248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 msgr2=0x7f69a41e4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f69a359e700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 0x7f69a41e4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f69a359e700 1 -- 172.21.15.102:0/3712470248 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980094a0 con 0x7f69a4153290 2023-03-03T10:27:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f69a359e700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4153290 0x7f69a41e4550 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f699c00efc0 tx=0x7f699c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:27:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f698b7fe700 1 -- 172.21.15.102:0/3712470248 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c009ea0 con 0x7f69a4153290 2023-03-03T10:27:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f698b7fe700 1 -- 172.21.15.102:0/3712470248 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c00f040 con 0x7f69a4153290 2023-03-03T10:27:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f698b7fe700 1 -- 172.21.15.102:0/3712470248 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c00ced0 con 0x7f69a4153290 2023-03-03T10:27:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.558+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a40ccde0 con 0x7f69a4153290 2023-03-03T10:27:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.559+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a415d160 con 0x7f69a4153290 2023-03-03T10:27:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.560+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a40cd0c0 con 0x7f69a4153290 2023-03-03T10:27:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.561+0000 7f698b7fe700 1 -- 172.21.15.102:0/3712470248 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f699c014480 con 0x7f69a4153290 2023-03-03T10:27:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.562+0000 7f698b7fe700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f69900682f0 0x7f699006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.563+0000 7f698b7fe700 1 -- 172.21.15.102:0/3712470248 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f699c086ac0 con 0x7f69a4153290 2023-03-03T10:27:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.563+0000 7f69a2d9d700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f69900682f0 0x7f699006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:28.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.563+0000 7f69a2d9d700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f69900682f0 0x7f699006a7a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6994007fd0 tx=0x7f6994007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:27:28.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.567+0000 7f698b7fe700 1 -- 172.21.15.102:0/3712470248 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f699c053170 con 0x7f69a4153290 2023-03-03T10:27:28.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.800+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a41e2ca0 con 0x7f69a4153290 2023-03-03T10:27:28.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.801+0000 7f698b7fe700 1 -- 172.21.15.102:0/3712470248 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f69a41e2ca0 con 0x7f69a4153290 2023-03-03T10:27:28.803 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:27:28.803 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:27:28.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f69900682f0 msgr2=0x7f699006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:28.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f69900682f0 0x7f699006a7a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6994007fd0 tx=0x7f6994007dc0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4153290 msgr2=0x7f69a41e4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:28.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4153290 0x7f69a41e4550 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f699c00efc0 tx=0x7f699c00c3a0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 shutdown_connections 2023-03-03T10:27:28.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f69900682f0 0x7f699006a7a0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69a40cd390 0x7f69a40b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69a4153290 0x7f69a41e4550 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 --2- 172.21.15.102:0/3712470248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69a40cd8e0 0x7f69a41e4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:28.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 >> 172.21.15.102:0/3712470248 conn(0x7f69a4065fb0 msgr2=0x7f69a41570a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:27:28.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 shutdown_connections 2023-03-03T10:27:28.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:27:28.804+0000 7f69a91ff700 1 -- 172.21.15.102:0/3712470248 wait complete. 2023-03-03T10:27:28.808 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:27:28.832 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T10:27:43.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:43.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:03.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:28:04.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.082+0000 7f6837fff700 1 Processor -- start 2023-03-03T10:28:04.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.082+0000 7f6837fff700 1 -- start start 2023-03-03T10:28:04.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.083+0000 7f6837fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f68381564b0 0x7f68380cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:04.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.083+0000 7f6837fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380c9610 0x7f68380cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:04.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6837fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68380c9ca0 con 0x7f68380ca440 2023-03-03T10:28:04.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6837fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68380b4920 con 0x7f68381564b0 2023-03-03T10:28:04.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6837fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380b4c00 con 0x7f68380c9610 2023-03-03T10:28:04.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6836ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f68381564b0 0x7f68380cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:04.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f68377fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f68380ca440 0x7f68380ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48500/0 (socket says 172.21.15.102:48500) 2023-03-03T10:28:04.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f68377fe700 1 -- 172.21.15.102:0/4051925499 learned_addr learned my addr 172.21.15.102:0/4051925499 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:28:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f68367fc700 1 --2- 172.21.15.102:0/4051925499 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380c9610 0x7f68380cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6835ffb700 1 -- 172.21.15.102:0/4051925499 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351383388 0 0) 0x7f68380c9ca0 con 0x7f68380ca440 2023-03-03T10:28:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6835ffb700 1 -- 172.21.15.102:0/4051925499 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68200035b0 con 0x7f68380ca440 2023-03-03T10:28:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6836ffd700 1 -- 172.21.15.102:0/4051925499 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380c9610 msgr2=0x7f68380cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:28:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6836ffd700 1 --2- 172.21.15.102:0/4051925499 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380c9610 0x7f68380cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6836ffd700 1 -- 172.21.15.102:0/4051925499 >> v1:172.21.15.102:6789/0 conn(0x7f68380ca440 legacy=0x7f68380ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.084+0000 7f6836ffd700 1 -- 172.21.15.102:0/4051925499 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380b46f0 con 0x7f68381564b0 2023-03-03T10:28:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.085+0000 7f6836ffd700 1 --2- 172.21.15.102:0/4051925499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68381564b0 0x7f68380cb660 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6828009840 tx=0x7f6828009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b23c3832565394 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.085+0000 7f6835ffb700 1 -- 172.21.15.102:0/4051925499 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828011830 con 0x7f68381564b0 2023-03-03T10:28:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.085+0000 7f6835ffb700 1 -- 172.21.15.102:0/4051925499 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6828011990 con 0x7f68381564b0 2023-03-03T10:28:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.085+0000 7f6835ffb700 1 -- 172.21.15.102:0/4051925499 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828011d20 con 0x7f68381564b0 2023-03-03T10:28:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 -- 172.21.15.102:0/4051925499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68381564b0 msgr2=0x7f68380cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 --2- 172.21.15.102:0/4051925499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68381564b0 0x7f68380cb660 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6828009840 tx=0x7f6828009b50 comp rx=0 tx=0).stop 2023-03-03T10:28:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 -- 172.21.15.102:0/4051925499 shutdown_connections 2023-03-03T10:28:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 --2- 172.21.15.102:0/4051925499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68381564b0 0x7f68380cb660 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 --2- 172.21.15.102:0/4051925499 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380c9610 0x7f68380cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 -- 172.21.15.102:0/4051925499 >> 172.21.15.102:0/4051925499 conn(0x7f6838065fb0 msgr2=0x7f68380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 -- 172.21.15.102:0/4051925499 shutdown_connections 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.086+0000 7f6837fff700 1 -- 172.21.15.102:0/4051925499 wait complete. 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.088+0000 7f6837fff700 1 Processor -- start 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.088+0000 7f6837fff700 1 -- start start 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6837fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68380c9610 0x7f68381f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6837fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380ca440 0x7f68380c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6837fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68381564b0 0x7f683806b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6837fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68381eda70 con 0x7f68380c9610 2023-03-03T10:28:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6837fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380ce9a0 con 0x7f68381564b0 2023-03-03T10:28:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6837fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68381f1f30 con 0x7f68380ca440 2023-03-03T10:28:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6836ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68380c9610 0x7f68381f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6836ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68380c9610 0x7f68381f19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50422/0 (socket says 172.21.15.102:50422) 2023-03-03T10:28:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.089+0000 7f6836ffd700 1 -- 172.21.15.102:0/1965698768 learned_addr learned my addr 172.21.15.102:0/1965698768 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:28:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f68377fe700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68381564b0 0x7f683806b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f68367fc700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380ca440 0x7f68380c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:04.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f6836ffd700 1 -- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380ca440 msgr2=0x7f68380c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:04.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f6836ffd700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380ca440 0x7f68380c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f6836ffd700 1 -- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68381564b0 msgr2=0x7f683806b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:04.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f6836ffd700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68381564b0 0x7f683806b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f6836ffd700 1 -- 172.21.15.102:0/1965698768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68280094a0 con 0x7f68380c9610 2023-03-03T10:28:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.090+0000 7f6836ffd700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68380c9610 0x7f68381f19f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f68280090b0 tx=0x7f6828004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.091+0000 7f681f7fe700 1 -- 172.21.15.102:0/1965698768 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828004590 con 0x7f68380c9610 2023-03-03T10:28:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.091+0000 7f681f7fe700 1 -- 172.21.15.102:0/1965698768 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682801e070 con 0x7f68380c9610 2023-03-03T10:28:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.091+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68380b2210 con 0x7f68380c9610 2023-03-03T10:28:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.091+0000 7f681f7fe700 1 -- 172.21.15.102:0/1965698768 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280038e0 con 0x7f68380c9610 2023-03-03T10:28:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.091+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68380b1f30 con 0x7f68380c9610 2023-03-03T10:28:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.092+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6838156880 con 0x7f68380c9610 2023-03-03T10:28:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.093+0000 7f681f7fe700 1 -- 172.21.15.102:0/1965698768 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f682800f050 con 0x7f68380c9610 2023-03-03T10:28:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.094+0000 7f681f7fe700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6824068320 0x7f682406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.094+0000 7f681f7fe700 1 -- 172.21.15.102:0/1965698768 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6828086bb0 con 0x7f68380c9610 2023-03-03T10:28:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.095+0000 7f68367fc700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6824068320 0x7f682406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.096+0000 7f68367fc700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6824068320 0x7f682406a7d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f68380c94c0 tx=0x7f682c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.099+0000 7f681f7fe700 1 -- 172.21.15.102:0/1965698768 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68280532c0 con 0x7f68380c9610 2023-03-03T10:28:04.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.331+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68380677f0 con 0x7f68380c9610 2023-03-03T10:28:04.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.332+0000 7f681f7fe700 1 -- 172.21.15.102:0/1965698768 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6828015090 con 0x7f68380c9610 2023-03-03T10:28:04.334 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:28:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:28:04.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.334+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6824068320 msgr2=0x7f682406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:04.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.334+0000 7f6837fff700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6824068320 0x7f682406a7d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f68380c94c0 tx=0x7f682c009210 comp rx=0 tx=0).stop 2023-03-03T10:28:04.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.334+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68380c9610 msgr2=0x7f68381f19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:04.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.334+0000 7f6837fff700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68380c9610 0x7f68381f19f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f68280090b0 tx=0x7f6828004750 comp rx=0 tx=0).stop 2023-03-03T10:28:04.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 shutdown_connections 2023-03-03T10:28:04.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68381564b0 0x7f683806b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68380c9610 0x7f68381f19f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6824068320 0x7f682406a7d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 --2- 172.21.15.102:0/1965698768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f68380ca440 0x7f68380c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:04.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 >> 172.21.15.102:0/1965698768 conn(0x7f6838065fb0 msgr2=0x7f6838155520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:04.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 shutdown_connections 2023-03-03T10:28:04.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:04.335+0000 7f6837fff700 1 -- 172.21.15.102:0/1965698768 wait complete. 2023-03-03T10:28:04.339 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:28:04.364 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 18 secs before thrashing 2023-03-03T10:28:14.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:14.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:22.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:28:22.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.614+0000 7fa9acbff700 1 Processor -- start 2023-03-03T10:28:22.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.614+0000 7fa9acbff700 1 -- start start 2023-03-03T10:28:22.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.615+0000 7fa9acbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:22.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.615+0000 7fa9acbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a8153660 con 0x7fa9a81593c0 2023-03-03T10:28:22.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.615+0000 7fa9acbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a806c360 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.615+0000 7fa9acbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a80cec40 con 0x7fa9a8153290 2023-03-03T10:28:22.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa99ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa9a81593c0 0x7fa9a80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35690/0 (socket says 172.21.15.102:35690) 2023-03-03T10:28:22.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa99ffff700 1 -- 172.21.15.102:0/702358473 learned_addr learned my addr 172.21.15.102:0/702358473 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:28:22.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a659c700 1 --2- 172.21.15.102:0/702358473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:22.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a5d9b700 1 -- 172.21.15.102:0/702358473 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977397395 0 0) 0x7fa9a806c360 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a5d9b700 1 -- 172.21.15.102:0/702358473 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9940035b0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a5d9b700 1 -- 172.21.15.102:0/702358473 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209672779 0 0) 0x7fa9a8153660 con 0x7fa9a81593c0 2023-03-03T10:28:22.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a5d9b700 1 -- 172.21.15.102:0/702358473 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a806c360 con 0x7fa9a81593c0 2023-03-03T10:28:22.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a659c700 1 -- 172.21.15.102:0/702358473 >> v1:172.21.15.172:6789/0 conn(0x7fa9a80cd8e0 legacy=0x7fa9a80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:22.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a659c700 1 -- 172.21.15.102:0/702358473 >> v1:172.21.15.102:6789/0 conn(0x7fa9a81593c0 legacy=0x7fa9a80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:22.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.616+0000 7fa9a659c700 1 -- 172.21.15.102:0/702358473 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a815cea0 con 0x7fa9a8153290 2023-03-03T10:28:22.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.617+0000 7fa9a659c700 1 --2- 172.21.15.102:0/702358473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a8158e80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa998009870 tx=0x7fa998009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b26572b57b1c8c2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:22.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.617+0000 7fa9a5d9b700 1 -- 172.21.15.102:0/702358473 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998011830 con 0x7fa9a8153290 2023-03-03T10:28:22.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.617+0000 7fa9a5d9b700 1 -- 172.21.15.102:0/702358473 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998011990 con 0x7fa9a8153290 2023-03-03T10:28:22.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.617+0000 7fa9a5d9b700 1 -- 172.21.15.102:0/702358473 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998011d20 con 0x7fa9a8153290 2023-03-03T10:28:22.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.617+0000 7fa9acbff700 1 -- 172.21.15.102:0/702358473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 msgr2=0x7fa9a8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:22.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.617+0000 7fa9acbff700 1 --2- 172.21.15.102:0/702358473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a8158e80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa998009870 tx=0x7fa998009b80 comp rx=0 tx=0).stop 2023-03-03T10:28:22.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.618+0000 7fa9acbff700 1 -- 172.21.15.102:0/702358473 shutdown_connections 2023-03-03T10:28:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.618+0000 7fa9acbff700 1 --2- 172.21.15.102:0/702358473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.618+0000 7fa9acbff700 1 -- 172.21.15.102:0/702358473 >> 172.21.15.102:0/702358473 conn(0x7fa9a8065fb0 msgr2=0x7fa9a80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.618+0000 7fa9acbff700 1 -- 172.21.15.102:0/702358473 shutdown_connections 2023-03-03T10:28:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.618+0000 7fa9acbff700 1 -- 172.21.15.102:0/702358473 wait complete. 2023-03-03T10:28:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.620+0000 7fa9acbff700 1 Processor -- start 2023-03-03T10:28:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.620+0000 7fa9acbff700 1 -- start start 2023-03-03T10:28:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9acbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a80c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9acbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a8153290 0x7fa9a815ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9acbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9acbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a81f02c0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9acbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a80c69e0 con 0x7fa9a8153290 2023-03-03T10:28:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9acbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a80b2f90 con 0x7fa9a81593c0 2023-03-03T10:28:22.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9a659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a80c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:22.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9a659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a80c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53352/0 (socket says 172.21.15.102:53352) 2023-03-03T10:28:22.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9a659c700 1 -- 172.21.15.102:0/3289883165 learned_addr learned my addr 172.21.15.102:0/3289883165 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:28:22.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa99ffff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a8153290 0x7fa9a815ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:22.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.621+0000 7fa9a6d9d700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:22.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa9a659c700 1 -- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 msgr2=0x7fa9a814e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:22.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa9a659c700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a814e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa9a659c700 1 -- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a8153290 msgr2=0x7fa9a815ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa9a659c700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a8153290 0x7fa9a815ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa9a659c700 1 -- 172.21.15.102:0/3289883165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980094a0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa9a659c700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a80c5730 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa998014040 tx=0x7fa998004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa99effd700 1 -- 172.21.15.102:0/3289883165 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99801a6e0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa99effd700 1 -- 172.21.15.102:0/3289883165 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99801a840 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa99effd700 1 -- 172.21.15.102:0/3289883165 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99801abb0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.622+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a815d160 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.623+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a806c7f0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.623+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a80ccde0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.625+0000 7fa99effd700 1 -- 172.21.15.102:0/3289883165 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa99800f050 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.626+0000 7fa99effd700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9880682f0 0x7fa98806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:22.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.626+0000 7fa99effd700 1 -- 172.21.15.102:0/3289883165 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa998086a60 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.626+0000 7fa99ffff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9880682f0 0x7fa98806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:22.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.627+0000 7fa99ffff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9880682f0 0x7fa98806a7a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa9a81528c0 tx=0x7fa990007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:22.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.630+0000 7fa99effd700 1 -- 172.21.15.102:0/3289883165 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa998053220 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.863+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9a804e6e0 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.863+0000 7fa99effd700 1 -- 172.21.15.102:0/3289883165 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa998029090 con 0x7fa9a80cd8e0 2023-03-03T10:28:22.867 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:28:22.867 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:28:22.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9880682f0 msgr2=0x7fa98806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:22.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9880682f0 0x7fa98806a7a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa9a81528c0 tx=0x7fa990007dc0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a80cd8e0 msgr2=0x7fa9a80c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:22.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a80c5730 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa998014040 tx=0x7fa998004750 comp rx=0 tx=0).stop 2023-03-03T10:28:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 shutdown_connections 2023-03-03T10:28:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9880682f0 0x7fa98806a7a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a814e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a80c5730 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 --2- 172.21.15.102:0/3289883165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a8153290 0x7fa9a815ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 >> 172.21.15.102:0/3289883165 conn(0x7fa9a8065fb0 msgr2=0x7fa9a80cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:22.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 shutdown_connections 2023-03-03T10:28:22.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:28:22.866+0000 7fa9acbff700 1 -- 172.21.15.102:0/3289883165 wait complete. 2023-03-03T10:28:22.871 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:28:22.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2023-03-03T10:28:44.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:44.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:14.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:14.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:42.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:29:43.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.141+0000 7f37d03ff700 1 Processor -- start 2023-03-03T10:29:43.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.141+0000 7f37d03ff700 1 -- start start 2023-03-03T10:29:43.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.142+0000 7f37d03ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37c80cd390 0x7f37c80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37d03ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c8153660 con 0x7f37c80cd8e0 2023-03-03T10:29:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37d03ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c806c360 con 0x7f37c8153290 2023-03-03T10:29:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37d03ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c815cea0 con 0x7f37c80cd390 2023-03-03T10:29:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37cd99a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f37c80cd8e0 0x7f37c80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39386/0 (socket says 172.21.15.102:39386) 2023-03-03T10:29:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37cd99a700 1 -- 172.21.15.102:0/1766389557 learned_addr learned my addr 172.21.15.102:0/1766389557 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:29:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37ce99c700 1 --2- 172.21.15.102:0/1766389557 >> v2:172.21.15.172:3300/0 conn(0x7f37c80cd390 0x7f37c80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37cd199700 1 -- 172.21.15.102:0/1766389557 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720659283 0 0) 0x7f37c8153660 con 0x7f37c80cd8e0 2023-03-03T10:29:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37cd199700 1 -- 172.21.15.102:0/1766389557 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b40035b0 con 0x7f37c80cd8e0 2023-03-03T10:29:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37cd199700 1 -- 172.21.15.102:0/1766389557 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807110636 0 0) 0x7f37c806c360 con 0x7f37c8153290 2023-03-03T10:29:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.143+0000 7f37cd199700 1 -- 172.21.15.102:0/1766389557 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c8153660 con 0x7f37c8153290 2023-03-03T10:29:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.144+0000 7f37ce99c700 1 -- 172.21.15.102:0/1766389557 >> v1:172.21.15.172:6789/0 conn(0x7f37c8153290 legacy=0x7f37c8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.144+0000 7f37ce99c700 1 -- 172.21.15.102:0/1766389557 >> v1:172.21.15.102:6789/0 conn(0x7f37c80cd8e0 legacy=0x7f37c80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.144+0000 7f37ce99c700 1 -- 172.21.15.102:0/1766389557 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c806c7f0 con 0x7f37c80cd390 2023-03-03T10:29:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.144+0000 7f37ce99c700 1 --2- 172.21.15.102:0/1766389557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd390 0x7f37c80ce530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009d90 tx=0x7f37c000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fce826e9e708d332 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.144+0000 7f37cd199700 1 -- 172.21.15.102:0/1766389557 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c000e070 con 0x7f37c80cd390 2023-03-03T10:29:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.144+0000 7f37cd199700 1 -- 172.21.15.102:0/1766389557 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c000bb20 con 0x7f37c80cd390 2023-03-03T10:29:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.144+0000 7f37cd199700 1 -- 172.21.15.102:0/1766389557 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0012430 con 0x7f37c80cd390 2023-03-03T10:29:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.145+0000 7f37d03ff700 1 -- 172.21.15.102:0/1766389557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd390 msgr2=0x7f37c80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.145+0000 7f37d03ff700 1 --2- 172.21.15.102:0/1766389557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd390 0x7f37c80ce530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009d90 tx=0x7f37c000b330 comp rx=0 tx=0).stop 2023-03-03T10:29:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.145+0000 7f37d03ff700 1 -- 172.21.15.102:0/1766389557 shutdown_connections 2023-03-03T10:29:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.145+0000 7f37d03ff700 1 --2- 172.21.15.102:0/1766389557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd390 0x7f37c80ce530 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.145+0000 7f37d03ff700 1 -- 172.21.15.102:0/1766389557 >> 172.21.15.102:0/1766389557 conn(0x7f37c8065fb0 msgr2=0x7f37c80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:29:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.145+0000 7f37d03ff700 1 -- 172.21.15.102:0/1766389557 shutdown_connections 2023-03-03T10:29:43.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.145+0000 7f37d03ff700 1 -- 172.21.15.102:0/1766389557 wait complete. 2023-03-03T10:29:43.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.147+0000 7f37d03ff700 1 Processor -- start 2023-03-03T10:29:43.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.147+0000 7f37d03ff700 1 -- start start 2023-03-03T10:29:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37d03ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37c80cd390 0x7f37c80c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37d03ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd8e0 0x7f37c81502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37d03ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37c8153290 0x7f37c814e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37d03ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8150820 con 0x7f37c80cd390 2023-03-03T10:29:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37d03ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8151a10 con 0x7f37c80cd8e0 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37d03ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80b2000 con 0x7f37c8153290 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37cd99a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd8e0 0x7f37c81502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37ce19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37c80cd390 0x7f37c80c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37ce99c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37c8153290 0x7f37c814e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37cd99a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd8e0 0x7f37c81502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52544/0 (socket says 172.21.15.102:52544) 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37ce99c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37c8153290 0x7f37c814e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42820/0 (socket says 172.21.15.102:42820) 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.148+0000 7f37cd99a700 1 -- 172.21.15.102:0/3651263236 learned_addr learned my addr 172.21.15.102:0/3651263236 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:29:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37cd99a700 1 -- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37c8153290 msgr2=0x7f37c814e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37cd99a700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37c8153290 0x7f37c814e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37cd99a700 1 -- 172.21.15.102:0/3651263236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37c80cd390 msgr2=0x7f37c80c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37cd99a700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37c80cd390 0x7f37c80c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37cd99a700 1 -- 172.21.15.102:0/3651263236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c0009a70 con 0x7f37c80cd8e0 2023-03-03T10:29:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37ce19b700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37c80cd390 0x7f37c80c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:29:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37cd99a700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd8e0 0x7f37c81502e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f37b800c730 tx=0x7f37b800ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37beffd700 1 -- 172.21.15.102:0/3651263236 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8005d70 con 0x7f37c80cd8e0 2023-03-03T10:29:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37beffd700 1 -- 172.21.15.102:0/3651263236 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b80070d0 con 0x7f37c80cd8e0 2023-03-03T10:29:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c80ccde0 con 0x7f37c80cd8e0 2023-03-03T10:29:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.149+0000 7f37beffd700 1 -- 172.21.15.102:0/3651263236 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8007440 con 0x7f37c80cd8e0 2023-03-03T10:29:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.150+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c815d160 con 0x7f37c80cd8e0 2023-03-03T10:29:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.151+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c80cd0c0 con 0x7f37c80cd8e0 2023-03-03T10:29:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.152+0000 7f37beffd700 1 -- 172.21.15.102:0/3651263236 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37b8019030 con 0x7f37c80cd8e0 2023-03-03T10:29:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.153+0000 7f37beffd700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.153+0000 7f37beffd700 1 -- 172.21.15.102:0/3651263236 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f37b80853d0 con 0x7f37c80cd8e0 2023-03-03T10:29:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.154+0000 7f37ce19b700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.154+0000 7f37ce19b700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f37c40011e0 tx=0x7f37c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:43.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.156+0000 7f37beffd700 1 -- 172.21.15.102:0/3651263236 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37b8051c10 con 0x7f37c80cd8e0 2023-03-03T10:29:43.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.389+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37c804e6e0 con 0x7f37c80cd8e0 2023-03-03T10:29:43.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.390+0000 7f37beffd700 1 -- 172.21.15.102:0/3651263236 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f37b8055230 con 0x7f37c80cd8e0 2023-03-03T10:29:43.392 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:29:43.392 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:29:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 msgr2=0x7f37ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f37c40011e0 tx=0x7f37c4009040 comp rx=0 tx=0).stop 2023-03-03T10:29:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd8e0 msgr2=0x7f37c81502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd8e0 0x7f37c81502e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f37b800c730 tx=0x7f37b800ca40 comp rx=0 tx=0).stop 2023-03-03T10:29:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 shutdown_connections 2023-03-03T10:29:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37c8153290 0x7f37c814e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37c80cd390 0x7f37c80c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 --2- 172.21.15.102:0/3651263236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37c80cd8e0 0x7f37c81502e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 >> 172.21.15.102:0/3651263236 conn(0x7f37c8065fb0 msgr2=0x7f37c815aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:29:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 shutdown_connections 2023-03-03T10:29:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:29:43.393+0000 7f37d03ff700 1 -- 172.21.15.102:0/3651263236 wait complete. 2023-03-03T10:29:43.397 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:29:43.422 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-03-03T10:29:44.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:44.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T10:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:30:14.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:14.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:44.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:44.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:58.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:30:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.671+0000 7fdf5359e700 1 Processor -- start 2023-03-03T10:30:58.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.671+0000 7fdf5359e700 1 -- start start 2023-03-03T10:30:58.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf5359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdf540b3970 0x7fdf540c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:58.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf5359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdf540c41b0 0x7fdf540cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:58.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf5359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540c3d20 con 0x7fdf540ced00 2023-03-03T10:30:58.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf5359e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf540c4930 con 0x7fdf540c41b0 2023-03-03T10:30:58.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf5359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf540cf610 con 0x7fdf540b3970 2023-03-03T10:30:58.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf5259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdf540ced00 0x7fdf54158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53364/0 (socket says 172.21.15.102:53364) 2023-03-03T10:30:58.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf5259c700 1 -- 172.21.15.102:0/1444946150 learned_addr learned my addr 172.21.15.102:0/1444946150 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:30:58.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf52d9d700 1 --2- 172.21.15.102:0/1444946150 >> v2:172.21.15.102:3300/0 conn(0x7fdf540c41b0 0x7fdf540cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:58.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.673+0000 7fdf51d9b700 1 --2- 172.21.15.102:0/1444946150 >> v2:172.21.15.172:3300/0 conn(0x7fdf540b3970 0x7fdf540c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:58.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf5159a700 1 -- 172.21.15.102:0/1444946150 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576415180 0 0) 0x7fdf540c3d20 con 0x7fdf540ced00 2023-03-03T10:30:58.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf5159a700 1 -- 172.21.15.102:0/1444946150 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c0035b0 con 0x7fdf540ced00 2023-03-03T10:30:58.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf51d9b700 1 -- 172.21.15.102:0/1444946150 >> v2:172.21.15.102:3300/0 conn(0x7fdf540c41b0 msgr2=0x7fdf540cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:30:58.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf51d9b700 1 --2- 172.21.15.102:0/1444946150 >> v2:172.21.15.102:3300/0 conn(0x7fdf540c41b0 0x7fdf540cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf51d9b700 1 -- 172.21.15.102:0/1444946150 >> v1:172.21.15.102:6789/0 conn(0x7fdf540ced00 legacy=0x7fdf54158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf51d9b700 1 -- 172.21.15.102:0/1444946150 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5415cba0 con 0x7fdf540b3970 2023-03-03T10:30:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf52d9d700 1 --2- 172.21.15.102:0/1444946150 >> v2:172.21.15.102:3300/0 conn(0x7fdf540c41b0 0x7fdf540cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:30:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf51d9b700 1 --2- 172.21.15.102:0/1444946150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540b3970 0x7fdf540c5a90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdf540c4ba0 tx=0x7fdf48009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff979b72f662a08d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.674+0000 7fdf5159a700 1 -- 172.21.15.102:0/1444946150 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48011830 con 0x7fdf540b3970 2023-03-03T10:30:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5159a700 1 -- 172.21.15.102:0/1444946150 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf48011990 con 0x7fdf540b3970 2023-03-03T10:30:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5159a700 1 -- 172.21.15.102:0/1444946150 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48011d20 con 0x7fdf540b3970 2023-03-03T10:30:58.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 -- 172.21.15.102:0/1444946150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540b3970 msgr2=0x7fdf540c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:58.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 --2- 172.21.15.102:0/1444946150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540b3970 0x7fdf540c5a90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdf540c4ba0 tx=0x7fdf48009b20 comp rx=0 tx=0).stop 2023-03-03T10:30:58.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 -- 172.21.15.102:0/1444946150 shutdown_connections 2023-03-03T10:30:58.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 --2- 172.21.15.102:0/1444946150 >> v2:172.21.15.102:3300/0 conn(0x7fdf540c41b0 0x7fdf540cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 --2- 172.21.15.102:0/1444946150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540b3970 0x7fdf540c5a90 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 -- 172.21.15.102:0/1444946150 >> 172.21.15.102:0/1444946150 conn(0x7fdf5408b1c0 msgr2=0x7fdf54071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:30:58.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 -- 172.21.15.102:0/1444946150 shutdown_connections 2023-03-03T10:30:58.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.675+0000 7fdf5359e700 1 -- 172.21.15.102:0/1444946150 wait complete. 2023-03-03T10:30:58.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.678+0000 7fdf5359e700 1 Processor -- start 2023-03-03T10:30:58.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.678+0000 7fdf5359e700 1 -- start start 2023-03-03T10:30:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf540b3970 0x7fdf541e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540c41b0 0x7fdf54159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf540ced00 0x7fdf540cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5415abf0 con 0x7fdf540b3970 2023-03-03T10:30:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf541e47b0 con 0x7fdf540c41b0 2023-03-03T10:30:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf541f19e0 con 0x7fdf540ced00 2023-03-03T10:30:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf540b3970 0x7fdf541e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf540b3970 0x7fdf541e4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42282/0 (socket says 172.21.15.102:42282) 2023-03-03T10:30:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf5259c700 1 -- 172.21.15.102:0/864047830 learned_addr learned my addr 172.21.15.102:0/864047830 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:30:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf51d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540c41b0 0x7fdf54159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.680+0000 7fdf52d9d700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf540ced00 0x7fdf540cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf5259c700 1 -- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf540ced00 msgr2=0x7fdf540cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf5259c700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf540ced00 0x7fdf540cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf5259c700 1 -- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540c41b0 msgr2=0x7fdf54159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf5259c700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540c41b0 0x7fdf54159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf5259c700 1 -- 172.21.15.102:0/864047830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c009a70 con 0x7fdf540b3970 2023-03-03T10:30:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf5259c700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf540b3970 0x7fdf541e4bb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400c700 tx=0x7fdf4400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf42ffd700 1 -- 172.21.15.102:0/864047830 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44005d70 con 0x7fdf540b3970 2023-03-03T10:30:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf42ffd700 1 -- 172.21.15.102:0/864047830 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf440070d0 con 0x7fdf540b3970 2023-03-03T10:30:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf42ffd700 1 -- 172.21.15.102:0/864047830 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44007460 con 0x7fdf540b3970 2023-03-03T10:30:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.681+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf480094a0 con 0x7fdf540b3970 2023-03-03T10:30:58.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.682+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5406bf20 con 0x7fdf540b3970 2023-03-03T10:30:58.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.683+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5415ce80 con 0x7fdf540b3970 2023-03-03T10:30:58.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.684+0000 7fdf42ffd700 1 -- 172.21.15.102:0/864047830 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf44019030 con 0x7fdf540b3970 2023-03-03T10:30:58.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.685+0000 7fdf42ffd700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf34068320 0x7fdf3406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:58.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.685+0000 7fdf42ffd700 1 -- 172.21.15.102:0/864047830 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdf44085160 con 0x7fdf540b3970 2023-03-03T10:30:58.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.686+0000 7fdf51d9b700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf34068320 0x7fdf3406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:58.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.686+0000 7fdf51d9b700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf34068320 0x7fdf3406a7d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdf48014040 tx=0x7fdf4801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:58.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.689+0000 7fdf42ffd700 1 -- 172.21.15.102:0/864047830 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf440517f0 con 0x7fdf540b3970 2023-03-03T10:30:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.924+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf5406c1e0 con 0x7fdf540b3970 2023-03-03T10:30:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.925+0000 7fdf42ffd700 1 -- 172.21.15.102:0/864047830 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdf44054e10 con 0x7fdf540b3970 2023-03-03T10:30:58.928 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:30:58.928 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:30:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf34068320 msgr2=0x7fdf3406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf34068320 0x7fdf3406a7d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdf48014040 tx=0x7fdf4801c040 comp rx=0 tx=0).stop 2023-03-03T10:30:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf540b3970 msgr2=0x7fdf541e4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf540b3970 0x7fdf541e4bb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400c700 tx=0x7fdf4400ca10 comp rx=0 tx=0).stop 2023-03-03T10:30:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 shutdown_connections 2023-03-03T10:30:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdf34068320 0x7fdf3406a7d0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdf540ced00 0x7fdf540cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf540b3970 0x7fdf541e4bb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 --2- 172.21.15.102:0/864047830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf540c41b0 0x7fdf54159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 >> 172.21.15.102:0/864047830 conn(0x7fdf5408b1c0 msgr2=0x7fdf54071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:30:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 shutdown_connections 2023-03-03T10:30:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:30:58.928+0000 7fdf5359e700 1 -- 172.21.15.102:0/864047830 wait complete. 2023-03-03T10:30:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:30:58.958 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 119 secs before thrashing 2023-03-03T10:31:15.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:15.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:45.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:45.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:15.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:15.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:45.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:45.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:57.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:32:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.224+0000 7f50db9ff700 1 Processor -- start 2023-03-03T10:32:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.225+0000 7f50db9ff700 1 -- start start 2023-03-03T10:32:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.226+0000 7f50db9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd8e0 0x7f50d40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.226+0000 7f50db9ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d4153660 con 0x7f50d40cd390 2023-03-03T10:32:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.226+0000 7f50db9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d406c360 con 0x7f50d4153290 2023-03-03T10:32:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.226+0000 7f50db9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d415cea0 con 0x7f50d40cd8e0 2023-03-03T10:32:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.226+0000 7f50d979b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f50d4153290 0x7f50d4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36038/0 (socket says 172.21.15.102:36038) 2023-03-03T10:32:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.226+0000 7f50d979b700 1 -- 172.21.15.102:0/3782830891 learned_addr learned my addr 172.21.15.102:0/3782830891 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:32:58.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.226+0000 7f50d8f9a700 1 --2- 172.21.15.102:0/3782830891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd8e0 0x7f50d40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:58.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50cbfff700 1 -- 172.21.15.102:0/3782830891 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820358046 0 0) 0x7f50d406c360 con 0x7f50d4153290 2023-03-03T10:32:58.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50cbfff700 1 -- 172.21.15.102:0/3782830891 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c00035b0 con 0x7f50d4153290 2023-03-03T10:32:58.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50cbfff700 1 -- 172.21.15.102:0/3782830891 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248518815 0 0) 0x7f50d4153660 con 0x7f50d40cd390 2023-03-03T10:32:58.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50d8f9a700 1 -- 172.21.15.102:0/3782830891 >> v1:172.21.15.172:6789/0 conn(0x7f50d4153290 legacy=0x7f50d4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:58.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50d8f9a700 1 -- 172.21.15.102:0/3782830891 >> v1:172.21.15.102:6789/0 conn(0x7f50d40cd390 legacy=0x7f50d40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50d8f9a700 1 -- 172.21.15.102:0/3782830891 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d406c7f0 con 0x7f50d40cd8e0 2023-03-03T10:32:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50d8f9a700 1 --2- 172.21.15.102:0/3782830891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd8e0 0x7f50d40cde20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b61aa58d466e0b63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.227+0000 7f50cbfff700 1 -- 172.21.15.102:0/3782830891 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4011830 con 0x7f50d40cd8e0 2023-03-03T10:32:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50cbfff700 1 -- 172.21.15.102:0/3782830891 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c4011990 con 0x7f50d40cd8e0 2023-03-03T10:32:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50cbfff700 1 -- 172.21.15.102:0/3782830891 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4011d20 con 0x7f50d40cd8e0 2023-03-03T10:32:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50db9ff700 1 -- 172.21.15.102:0/3782830891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd8e0 msgr2=0x7f50d40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50db9ff700 1 --2- 172.21.15.102:0/3782830891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd8e0 0x7f50d40cde20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c4009b80 comp rx=0 tx=0).stop 2023-03-03T10:32:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50db9ff700 1 -- 172.21.15.102:0/3782830891 shutdown_connections 2023-03-03T10:32:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50db9ff700 1 --2- 172.21.15.102:0/3782830891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd8e0 0x7f50d40cde20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50db9ff700 1 -- 172.21.15.102:0/3782830891 >> 172.21.15.102:0/3782830891 conn(0x7f50d4065fb0 msgr2=0x7f50d40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50db9ff700 1 -- 172.21.15.102:0/3782830891 shutdown_connections 2023-03-03T10:32:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.228+0000 7f50db9ff700 1 -- 172.21.15.102:0/3782830891 wait complete. 2023-03-03T10:32:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.230+0000 7f50db9ff700 1 Processor -- start 2023-03-03T10:32:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.230+0000 7f50db9ff700 1 -- start start 2023-03-03T10:32:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.231+0000 7f50db9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd390 0x7f50d40c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.231+0000 7f50db9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50d40cd8e0 0x7f50d40cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.231+0000 7f50db9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50d4153290 0x7f50d40c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.231+0000 7f50db9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d41ee5a0 con 0x7f50d40cd8e0 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.231+0000 7f50db9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d41ef3c0 con 0x7f50d4153290 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.231+0000 7f50db9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d41ee840 con 0x7f50d40cd390 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd390 0x7f50d40c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd390 0x7f50d40c84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42062/0 (socket says 172.21.15.102:42062) 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 -- 172.21.15.102:0/2439790001 learned_addr learned my addr 172.21.15.102:0/2439790001 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d9f9c700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50d4153290 0x7f50d40c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d8f9a700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50d40cd8e0 0x7f50d40cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 -- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50d4153290 msgr2=0x7f50d40c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50d4153290 0x7f50d40c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 -- 172.21.15.102:0/2439790001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50d40cd8e0 msgr2=0x7f50d40cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50d40cd8e0 0x7f50d40cbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 -- 172.21.15.102:0/2439790001 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40094a0 con 0x7f50d40cd390 2023-03-03T10:32:58.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.232+0000 7f50d979b700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd390 0x7f50d40c84b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f50d000c730 tx=0x7f50d000ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:58.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.233+0000 7f50c9ffb700 1 -- 172.21.15.102:0/2439790001 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0005d70 con 0x7f50d40cd390 2023-03-03T10:32:58.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.233+0000 7f50c9ffb700 1 -- 172.21.15.102:0/2439790001 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d00070d0 con 0x7f50d40cd390 2023-03-03T10:32:58.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.233+0000 7f50c9ffb700 1 -- 172.21.15.102:0/2439790001 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0007460 con 0x7f50d40cd390 2023-03-03T10:32:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.233+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d40ccde0 con 0x7f50d40cd390 2023-03-03T10:32:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.233+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d415d160 con 0x7f50d40cd390 2023-03-03T10:32:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.234+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d40cd0c0 con 0x7f50d40cd390 2023-03-03T10:32:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.235+0000 7f50c9ffb700 1 -- 172.21.15.102:0/2439790001 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f50d0019030 con 0x7f50d40cd390 2023-03-03T10:32:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.236+0000 7f50c9ffb700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50b8068320 0x7f50b806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.236+0000 7f50c9ffb700 1 -- 172.21.15.102:0/2439790001 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f50d0085090 con 0x7f50d40cd390 2023-03-03T10:32:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.237+0000 7f50d8f9a700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50b8068320 0x7f50b806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.237+0000 7f50d8f9a700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50b8068320 0x7f50b806a7d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f50c40097c0 tx=0x7f50c401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:58.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.241+0000 7f50c9ffb700 1 -- 172.21.15.102:0/2439790001 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50d00517a0 con 0x7f50d40cd390 2023-03-03T10:32:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.476+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50d40c6140 con 0x7f50d40cd390 2023-03-03T10:32:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.477+0000 7f50c9ffb700 1 -- 172.21.15.102:0/2439790001 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f50d0017020 con 0x7f50d40cd390 2023-03-03T10:32:58.480 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:32:58.480 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:32:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50b8068320 msgr2=0x7f50b806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50b8068320 0x7f50b806a7d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f50c40097c0 tx=0x7f50c401c040 comp rx=0 tx=0).stop 2023-03-03T10:32:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd390 msgr2=0x7f50d40c84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd390 0x7f50d40c84b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f50d000c730 tx=0x7f50d000ca40 comp rx=0 tx=0).stop 2023-03-03T10:32:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 shutdown_connections 2023-03-03T10:32:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50b8068320 0x7f50b806a7d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50d40cd390 0x7f50d40c84b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50d4153290 0x7f50d40c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 --2- 172.21.15.102:0/2439790001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50d40cd8e0 0x7f50d40cbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:58.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 >> 172.21.15.102:0/2439790001 conn(0x7f50d4065fb0 msgr2=0x7f50d4150c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:58.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 shutdown_connections 2023-03-03T10:32:58.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:32:58.480+0000 7f50db9ff700 1 -- 172.21.15.102:0/2439790001 wait complete. 2023-03-03T10:32:58.484 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:32:58.509 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T10:33:07.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:33:07.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.757+0000 7f2c0bfff700 1 Processor -- start 2023-03-03T10:33:07.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.757+0000 7f2c0bfff700 1 -- start start 2023-03-03T10:33:07.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c0c0ced00 0x7f2c0c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:33:07.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0bfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c0c3d20 con 0x7f2c0c0b3970 2023-03-03T10:33:07.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0bfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c15cba0 con 0x7f2c0c0c41b0 2023-03-03T10:33:07.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0bfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c0c4930 con 0x7f2c0c0ced00 2023-03-03T10:33:07.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0a7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2c0c0b3970 0x7f2c0c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37816/0 (socket says 172.21.15.102:37816) 2023-03-03T10:33:07.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0affd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c0c0ced00 0x7f2c0c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:33:07.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0a7fc700 1 -- 172.21.15.102:0/1427477738 learned_addr learned my addr 172.21.15.102:0/1427477738 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:33:07.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.759+0000 7f2c0b7fe700 1 --1- 172.21.15.102:0/1427477738 >> v1:172.21.15.172:6789/0 conn(0x7f2c0c0c41b0 0x7f2c0c163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33602/0 (socket says 172.21.15.102:33602) 2023-03-03T10:33:07.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c09ffb700 1 -- 172.21.15.102:0/1427477738 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545681931 0 0) 0x7f2c0c15cba0 con 0x7f2c0c0c41b0 2023-03-03T10:33:07.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c09ffb700 1 -- 172.21.15.102:0/1427477738 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf40035b0 con 0x7f2c0c0c41b0 2023-03-03T10:33:07.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c0affd700 1 -- 172.21.15.102:0/1427477738 >> v1:172.21.15.172:6789/0 conn(0x7f2c0c0c41b0 legacy=0x7f2c0c163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:33:07.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c0affd700 1 -- 172.21.15.102:0/1427477738 >> v1:172.21.15.102:6789/0 conn(0x7f2c0c0b3970 legacy=0x7f2c0c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:33:07.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c0affd700 1 -- 172.21.15.102:0/1427477738 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c15ce80 con 0x7f2c0c0ced00 2023-03-03T10:33:07.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c0affd700 1 --2- 172.21.15.102:0/1427477738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c0c5a90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009840 tx=0x7f2bfc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f922a35ebdc44d5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:33:07.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c09ffb700 1 -- 172.21.15.102:0/1427477738 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc011830 con 0x7f2c0c0ced00 2023-03-03T10:33:07.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c09ffb700 1 -- 172.21.15.102:0/1427477738 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bfc011990 con 0x7f2c0c0ced00 2023-03-03T10:33:07.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.760+0000 7f2c09ffb700 1 -- 172.21.15.102:0/1427477738 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc011d20 con 0x7f2c0c0ced00 2023-03-03T10:33:07.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.761+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1427477738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 msgr2=0x7f2c0c0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:33:07.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.761+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1427477738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c0c5a90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009840 tx=0x7f2bfc009b50 comp rx=0 tx=0).stop 2023-03-03T10:33:07.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.761+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1427477738 shutdown_connections 2023-03-03T10:33:07.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.761+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1427477738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c0c5a90 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:33:07.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.761+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1427477738 >> 172.21.15.102:0/1427477738 conn(0x7f2c0c08b1c0 msgr2=0x7f2c0c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:33:07.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.761+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1427477738 shutdown_connections 2023-03-03T10:33:07.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.761+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1427477738 wait complete. 2023-03-03T10:33:07.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.764+0000 7f2c0bfff700 1 Processor -- start 2023-03-03T10:33:07.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.764+0000 7f2c0bfff700 1 -- start start 2023-03-03T10:33:07.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.765+0000 7f2c0bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c0c0b3970 0x7f2c0c1f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:33:07.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0c0c41b0 0x7f2c0c151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:33:07.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c1507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:33:07.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0bfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c14edb0 con 0x7f2c0c0c41b0 2023-03-03T10:33:07.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0bfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c14d000 con 0x7f2c0c0ced00 2023-03-03T10:33:07.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c14e3c0 con 0x7f2c0c0b3970 2023-03-03T10:33:07.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0a7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0c0c41b0 0x7f2c0c151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:33:07.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c0c0b3970 0x7f2c0c1f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:33:07.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c1507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:33:07.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0a7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0c0c41b0 0x7f2c0c151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41082/0 (socket says 172.21.15.102:41082) 2023-03-03T10:33:07.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0a7fc700 1 -- 172.21.15.102:0/1222460902 learned_addr learned my addr 172.21.15.102:0/1222460902 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:33:07.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c0c0b3970 0x7f2c0c1f16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60256/0 (socket says 172.21.15.102:60256) 2023-03-03T10:33:07.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0affd700 1 -- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 msgr2=0x7f2c0c1507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:33:07.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.766+0000 7f2c0affd700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c1507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:33:07.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2c0affd700 1 -- 172.21.15.102:0/1222460902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0c0c41b0 msgr2=0x7f2c0c151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:33:07.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2c0affd700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0c0c41b0 0x7f2c0c151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:33:07.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2c0affd700 1 -- 172.21.15.102:0/1222460902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0094a0 con 0x7f2c0c0b3970 2023-03-03T10:33:07.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2c0b7fe700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c1507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:33:07.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2c0affd700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c0c0b3970 0x7f2c0c1f16d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0c4ba0 tx=0x7f2bfc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:33:07.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2bf3fff700 1 -- 172.21.15.102:0/1222460902 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc0036a0 con 0x7f2c0c0b3970 2023-03-03T10:33:07.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2bf3fff700 1 -- 172.21.15.102:0/1222460902 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bfc01f070 con 0x7f2c0c0b3970 2023-03-03T10:33:07.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2bf3fff700 1 -- 172.21.15.102:0/1222460902 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc003a20 con 0x7f2c0c0b3970 2023-03-03T10:33:07.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.767+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c06c200 con 0x7f2c0c0b3970 2023-03-03T10:33:07.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.768+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c06bf20 con 0x7f2c0c0b3970 2023-03-03T10:33:07.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.769+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c15c890 con 0x7f2c0c0b3970 2023-03-03T10:33:07.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.770+0000 7f2bf3fff700 1 -- 172.21.15.102:0/1222460902 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2bfc00f050 con 0x7f2c0c0b3970 2023-03-03T10:33:07.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.771+0000 7f2bf3fff700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf80682f0 0x7f2bf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:33:07.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.771+0000 7f2bf3fff700 1 -- 172.21.15.102:0/1222460902 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2bfc021070 con 0x7f2c0c0b3970 2023-03-03T10:33:07.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.771+0000 7f2c0a7fc700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf80682f0 0x7f2bf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:33:07.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.772+0000 7f2c0a7fc700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf80682f0 0x7f2bf806a7a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cebd0 tx=0x7f2c00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:33:07.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:07.774+0000 7f2bf3fff700 1 -- 172.21.15.102:0/1222460902 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2bfc0543e0 con 0x7f2c0c0b3970 2023-03-03T10:33:08.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.009+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c0c04e6e0 con 0x7f2c0c0b3970 2023-03-03T10:33:08.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.010+0000 7f2bf3fff700 1 -- 172.21.15.102:0/1222460902 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2bfc02b090 con 0x7f2c0c0b3970 2023-03-03T10:33:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:33:08.014 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:33:08.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf80682f0 msgr2=0x7f2bf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:33:08.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf80682f0 0x7f2bf806a7a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cebd0 tx=0x7f2c00009040 comp rx=0 tx=0).stop 2023-03-03T10:33:08.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c0c0b3970 msgr2=0x7f2c0c1f16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:33:08.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c0c0b3970 0x7f2c0c1f16d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0c4ba0 tx=0x7f2bfc004750 comp rx=0 tx=0).stop 2023-03-03T10:33:08.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 shutdown_connections 2023-03-03T10:33:08.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf80682f0 0x7f2bf806a7a0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:33:08.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c0c0b3970 0x7f2c0c1f16d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:33:08.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c0c0ced00 0x7f2c0c1507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:33:08.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 --2- 172.21.15.102:0/1222460902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0c0c41b0 0x7f2c0c151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:33:08.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 >> 172.21.15.102:0/1222460902 conn(0x7f2c0c08b1c0 msgr2=0x7f2c0c0cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:33:08.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 shutdown_connections 2023-03-03T10:33:08.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:33:08.014+0000 7f2c0bfff700 1 -- 172.21.15.102:0/1222460902 wait complete. 2023-03-03T10:33:08.019 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:33:08.044 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2023-03-03T10:33:15.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:15.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:46.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:46.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:16.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:16.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:46.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:46.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:00.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:35:00.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.298+0000 7f8aa2dff700 1 Processor -- start 2023-03-03T10:35:00.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.298+0000 7f8aa2dff700 1 -- start start 2023-03-03T10:35:00.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.299+0000 7f8aa2dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a9c0cd8e0 0x7f8a9c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:00.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.299+0000 7f8aa2dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:00.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8aa2dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c153660 con 0x7f8a9c153290 2023-03-03T10:35:00.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8aa2dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9c06c360 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8aa2dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c15cea0 con 0x7f8a9c0cd390 2023-03-03T10:35:00.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8aa0b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8a9c153290 0x7f8a9c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59488/0 (socket says 172.21.15.102:59488) 2023-03-03T10:35:00.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8aa0b9b700 1 -- 172.21.15.102:0/3959706545 learned_addr learned my addr 172.21.15.102:0/3959706545 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:35:00.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8a93fff700 1 --2- 172.21.15.102:0/3959706545 >> v2:172.21.15.102:3300/0 conn(0x7f8a9c0cd8e0 0x7f8a9c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:00.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8aa139c700 1 --2- 172.21.15.102:0/3959706545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8a937fe700 1 -- 172.21.15.102:0/3959706545 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205711319 0 0) 0x7f8a9c153660 con 0x7f8a9c153290 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8a937fe700 1 -- 172.21.15.102:0/3959706545 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a840035b0 con 0x7f8a9c153290 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8a93fff700 1 -- 172.21.15.102:0/3959706545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 msgr2=0x7f8a9c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8a93fff700 1 --2- 172.21.15.102:0/3959706545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8a93fff700 1 -- 172.21.15.102:0/3959706545 >> v1:172.21.15.172:6789/0 conn(0x7f8a9c153290 legacy=0x7f8a9c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8a93fff700 1 -- 172.21.15.102:0/3959706545 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c06c7f0 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.300+0000 7f8aa139c700 1 --2- 172.21.15.102:0/3959706545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:35:00.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.301+0000 7f8a93fff700 1 --2- 172.21.15.102:0/3959706545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c0cde20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009840 tx=0x7f8a8c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f21232c7495588a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.301+0000 7f8a937fe700 1 -- 172.21.15.102:0/3959706545 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c011830 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.301+0000 7f8a937fe700 1 -- 172.21.15.102:0/3959706545 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c011990 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.301+0000 7f8a937fe700 1 -- 172.21.15.102:0/3959706545 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c011d20 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.301+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3959706545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 msgr2=0x7f8a9c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:00.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.301+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3959706545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c0cde20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009840 tx=0x7f8a8c009b50 comp rx=0 tx=0).stop 2023-03-03T10:35:00.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.302+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3959706545 shutdown_connections 2023-03-03T10:35:00.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.302+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3959706545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.302+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3959706545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c0cde20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.302+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3959706545 >> 172.21.15.102:0/3959706545 conn(0x7f8a9c065fb0 msgr2=0x7f8a9c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:00.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.302+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3959706545 shutdown_connections 2023-03-03T10:35:00.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.302+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3959706545 wait complete. 2023-03-03T10:35:00.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.304+0000 7f8aa2dff700 1 Processor -- start 2023-03-03T10:35:00.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.304+0000 7f8aa2dff700 1 -- start start 2023-03-03T10:35:00.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa2dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:00.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa2dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:00.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa2dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a9c153290 0x7f8a9c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:00.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa2dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0b36d0 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa2dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1550b0 con 0x7f8a9c153290 2023-03-03T10:35:00.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa2dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0b3ee0 con 0x7f8a9c0cd390 2023-03-03T10:35:00.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:00.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44576/0 (socket says 172.21.15.102:44576) 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8a93fff700 1 -- 172.21.15.102:0/3527932701 learned_addr learned my addr 172.21.15.102:0/3527932701 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa0b9b700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.305+0000 7f8aa139c700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a9c153290 0x7f8a9c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a93fff700 1 -- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 msgr2=0x7f8a9c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a93fff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a93fff700 1 -- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a9c153290 msgr2=0x7f8a9c155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a93fff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a9c153290 0x7f8a9c155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a93fff700 1 -- 172.21.15.102:0/3527932701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94009a70 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8aa0b9b700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:35:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a93fff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c158300 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0090b0 tx=0x7f8a8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a917fa700 1 -- 172.21.15.102:0/3527932701 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c0045d0 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a917fa700 1 -- 172.21.15.102:0/3527932701 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c01e070 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8a917fa700 1 -- 172.21.15.102:0/3527932701 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c0038e0 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c0094a0 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.306+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c0ccde0 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.308+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c15d160 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.309+0000 7f8a917fa700 1 -- 172.21.15.102:0/3527932701 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a8c00f050 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.311+0000 7f8a917fa700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a880682f0 0x7f8a8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:00.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.311+0000 7f8a917fa700 1 -- 172.21.15.102:0/3527932701 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8a8c086c50 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.311+0000 7f8aa0b9b700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a880682f0 0x7f8a8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:00.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.312+0000 7f8aa0b9b700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a880682f0 0x7f8a8806a7a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8a980011e0 tx=0x7f8a98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:00.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.316+0000 7f8a917fa700 1 -- 172.21.15.102:0/3527932701 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a8c053360 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.554+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a9c0cd0c0 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.554+0000 7f8a917fa700 1 -- 172.21.15.102:0/3527932701 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8a8c020090 con 0x7f8a9c0cd8e0 2023-03-03T10:35:00.556 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:35:00.556 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:35:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a880682f0 msgr2=0x7f8a8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a880682f0 0x7f8a8806a7a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8a980011e0 tx=0x7f8a98009040 comp rx=0 tx=0).stop 2023-03-03T10:35:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 msgr2=0x7f8a9c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c158300 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0090b0 tx=0x7f8a8c004750 comp rx=0 tx=0).stop 2023-03-03T10:35:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 shutdown_connections 2023-03-03T10:35:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a880682f0 0x7f8a8806a7a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a9c0cd390 0x7f8a9c0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a9c153290 0x7f8a9c155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 --2- 172.21.15.102:0/3527932701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a9c0cd8e0 0x7f8a9c158300 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 >> 172.21.15.102:0/3527932701 conn(0x7f8a9c065fb0 msgr2=0x7f8a9c073810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 shutdown_connections 2023-03-03T10:35:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:00.557+0000 7f8aa2dff700 1 -- 172.21.15.102:0/3527932701 wait complete. 2023-03-03T10:35:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:35:00.586 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-03-03T10:35:16.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:16.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:33.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:35:33.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.833+0000 7f3be319d700 1 Processor -- start 2023-03-03T10:35:33.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.833+0000 7f3be319d700 1 -- start start 2023-03-03T10:35:33.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be319d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0ced00 0x7f3bdc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:33.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be319d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0b3970 0x7f3bdc0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:33.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be319d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bdc0c3d20 con 0x7f3bdc0cf0d0 2023-03-03T10:35:33.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be319d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bdc0c4930 con 0x7f3bdc0ced00 2023-03-03T10:35:33.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be319d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc15cba0 con 0x7f3bdc0b3970 2023-03-03T10:35:33.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be199a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0b3970 0x7f3bdc0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:33.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be219b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0ced00 0x7f3bdc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:33.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be299c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3bdc0cf0d0 0x7f3bdc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56916/0 (socket says 172.21.15.102:56916) 2023-03-03T10:35:33.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.835+0000 7f3be299c700 1 -- 172.21.15.102:0/2434071841 learned_addr learned my addr 172.21.15.102:0/2434071841 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:35:33.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be199a700 1 -- 172.21.15.102:0/2434071841 >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0ced00 msgr2=0x7f3bdc0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:35:33.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be199a700 1 --2- 172.21.15.102:0/2434071841 >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0ced00 0x7f3bdc0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:33.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be199a700 1 -- 172.21.15.102:0/2434071841 >> v1:172.21.15.172:6789/0 conn(0x7f3bdc0cf0d0 legacy=0x7f3bdc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:33.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be199a700 1 -- 172.21.15.102:0/2434071841 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc15ce80 con 0x7f3bdc0b3970 2023-03-03T10:35:33.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be219b700 1 --2- 172.21.15.102:0/2434071841 >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0ced00 0x7f3bdc0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:35:33.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be1199700 1 -- 172.21.15.102:0/2434071841 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791162556 0 0) 0x7f3bdc0c3d20 con 0x7f3bdc0cf0d0 2023-03-03T10:35:33.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be199a700 1 --2- 172.21.15.102:0/2434071841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0b3970 0x7f3bdc0c41b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009840 tx=0x7f3bcc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a3cbda1eaa22df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:33.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be1199700 1 -- 172.21.15.102:0/2434071841 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc011830 con 0x7f3bdc0b3970 2023-03-03T10:35:33.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be1199700 1 -- 172.21.15.102:0/2434071841 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc011990 con 0x7f3bdc0b3970 2023-03-03T10:35:33.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.836+0000 7f3be1199700 1 -- 172.21.15.102:0/2434071841 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc011d20 con 0x7f3bdc0b3970 2023-03-03T10:35:33.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 -- 172.21.15.102:0/2434071841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0b3970 msgr2=0x7f3bdc0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:33.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 --2- 172.21.15.102:0/2434071841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0b3970 0x7f3bdc0c41b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009840 tx=0x7f3bcc009b50 comp rx=0 tx=0).stop 2023-03-03T10:35:33.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 -- 172.21.15.102:0/2434071841 shutdown_connections 2023-03-03T10:35:33.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 --2- 172.21.15.102:0/2434071841 >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0ced00 0x7f3bdc0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:33.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 --2- 172.21.15.102:0/2434071841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0b3970 0x7f3bdc0c41b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 -- 172.21.15.102:0/2434071841 >> 172.21.15.102:0/2434071841 conn(0x7f3bdc08b1c0 msgr2=0x7f3bdc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 -- 172.21.15.102:0/2434071841 shutdown_connections 2023-03-03T10:35:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.837+0000 7f3be319d700 1 -- 172.21.15.102:0/2434071841 wait complete. 2023-03-03T10:35:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.839+0000 7f3be319d700 1 Processor -- start 2023-03-03T10:35:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.839+0000 7f3be319d700 1 -- start start 2023-03-03T10:35:33.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.840+0000 7f3be319d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bdc0b3970 0x7f3bdc159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.840+0000 7f3be319d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0ced00 0x7f3bdc1e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.840+0000 7f3be319d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc0cf0d0 0x7f3bdc14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.840+0000 7f3be319d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc15a200 con 0x7f3bdc0cf0d0 2023-03-03T10:35:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.840+0000 7f3be319d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc0cd400 con 0x7f3bdc0b3970 2023-03-03T10:35:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.840+0000 7f3be319d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc15a420 con 0x7f3bdc0ced00 2023-03-03T10:35:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.840+0000 7f3be219b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bdc0b3970 0x7f3bdc159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bdc0b3970 0x7f3bdc159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52194/0 (socket says 172.21.15.102:52194) 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be199a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0ced00 0x7f3bdc1e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 -- 172.21.15.102:0/3230338241 learned_addr learned my addr 172.21.15.102:0/3230338241 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be299c700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc0cf0d0 0x7f3bdc14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 -- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0ced00 msgr2=0x7f3bdc1e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0ced00 0x7f3bdc1e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 -- 172.21.15.102:0/3230338241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc0cf0d0 msgr2=0x7f3bdc14e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc0cf0d0 0x7f3bdc14e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 -- 172.21.15.102:0/3230338241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd80094a0 con 0x7f3bdc0b3970 2023-03-03T10:35:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3be219b700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bdc0b3970 0x7f3bdc159cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800ebe0 tx=0x7f3bd800eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.841+0000 7f3bd2ffd700 1 -- 172.21.15.102:0/3230338241 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd800c850 con 0x7f3bdc0b3970 2023-03-03T10:35:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.842+0000 7f3bd2ffd700 1 -- 172.21.15.102:0/3230338241 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd800c9b0 con 0x7f3bdc0b3970 2023-03-03T10:35:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.842+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bcc0094a0 con 0x7f3bdc0b3970 2023-03-03T10:35:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.842+0000 7f3bd2ffd700 1 -- 172.21.15.102:0/3230338241 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd800cd40 con 0x7f3bdc0b3970 2023-03-03T10:35:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.842+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bdc06c200 con 0x7f3bdc0b3970 2023-03-03T10:35:33.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.843+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bdc06bf20 con 0x7f3bdc0b3970 2023-03-03T10:35:33.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.844+0000 7f3bd2ffd700 1 -- 172.21.15.102:0/3230338241 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3bd8023070 con 0x7f3bdc0b3970 2023-03-03T10:35:33.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.845+0000 7f3bd2ffd700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bc00682f0 0x7f3bc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:33.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.845+0000 7f3bd2ffd700 1 -- 172.21.15.102:0/3230338241 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3bd8086110 con 0x7f3bdc0b3970 2023-03-03T10:35:33.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.846+0000 7f3be199a700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bc00682f0 0x7f3bc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:33.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.846+0000 7f3be199a700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bc00682f0 0x7f3bc006a7a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009840 tx=0x7f3bcc01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:33.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:33.849+0000 7f3bd2ffd700 1 -- 172.21.15.102:0/3230338241 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3bd8052820 con 0x7f3bdc0b3970 2023-03-03T10:35:34.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.083+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bdc0cf540 con 0x7f3bdc0b3970 2023-03-03T10:35:34.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.084+0000 7f3bd2ffd700 1 -- 172.21.15.102:0/3230338241 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3bd800cea0 con 0x7f3bdc0b3970 2023-03-03T10:35:34.087 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:35:34.088 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:35:34.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bc00682f0 msgr2=0x7f3bc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:34.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bc00682f0 0x7f3bc006a7a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009840 tx=0x7f3bcc01c040 comp rx=0 tx=0).stop 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bdc0b3970 msgr2=0x7f3bdc159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bdc0b3970 0x7f3bdc159cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800ebe0 tx=0x7f3bd800eef0 comp rx=0 tx=0).stop 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 shutdown_connections 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc0cf0d0 0x7f3bdc14e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bdc0b3970 0x7f3bdc159cc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bc00682f0 0x7f3bc006a7a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 --2- 172.21.15.102:0/3230338241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3bdc0ced00 0x7f3bdc1e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:34.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 >> 172.21.15.102:0/3230338241 conn(0x7f3bdc08b1c0 msgr2=0x7f3bdc0669c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:34.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 shutdown_connections 2023-03-03T10:35:34.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:34.087+0000 7f3be319d700 1 -- 172.21.15.102:0/3230338241 wait complete. 2023-03-03T10:35:34.092 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:35:34.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-03-03T10:35:37.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:35:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.368+0000 7f177dbff700 1 Processor -- start 2023-03-03T10:35:37.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.368+0000 7f177dbff700 1 -- start start 2023-03-03T10:35:37.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.369+0000 7f177dbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17781531c0 0x7f177815c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:37.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.369+0000 7f177dbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815c890 0x7f17780c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:37.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f177dbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17780723d0 con 0x7f17780c86f0 2023-03-03T10:35:37.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f177dbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1778153590 con 0x7f17781531c0 2023-03-03T10:35:37.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f177dbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1778072860 con 0x7f177815c890 2023-03-03T10:35:37.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f17777fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17781531c0 0x7f177815c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f17777fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17781531c0 0x7f177815c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35500/0 (socket says 172.21.15.102:35500) 2023-03-03T10:35:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f176ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815c890 0x7f17780c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f17777fe700 1 -- 172.21.15.102:0/1309229843 learned_addr learned my addr 172.21.15.102:0/1309229843 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:35:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f176ffff700 1 -- 172.21.15.102:0/1309229843 >> v2:172.21.15.102:3300/0 conn(0x7f17781531c0 msgr2=0x7f177815c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:35:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f176ffff700 1 --2- 172.21.15.102:0/1309229843 >> v2:172.21.15.102:3300/0 conn(0x7f17781531c0 0x7f177815c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f176ffff700 1 -- 172.21.15.102:0/1309229843 >> v1:172.21.15.172:6789/0 conn(0x7f17780c86f0 legacy=0x7f17780c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:37.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f176ffff700 1 -- 172.21.15.102:0/1309229843 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780c8c50 con 0x7f177815c890 2023-03-03T10:35:37.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f1776ffd700 1 -- 172.21.15.102:0/1309229843 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493423337 0 0) 0x7f17780723d0 con 0x7f17780c86f0 2023-03-03T10:35:37.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.370+0000 7f17777fe700 1 --2- 172.21.15.102:0/1309229843 >> v2:172.21.15.102:3300/0 conn(0x7f17781531c0 0x7f177815c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:35:37.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.371+0000 7f176ffff700 1 --2- 172.21.15.102:0/1309229843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815c890 0x7f17780c81b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1760009840 tx=0x7f1760009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e98c58ded40f4d83 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:37.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.371+0000 7f1776ffd700 1 -- 172.21.15.102:0/1309229843 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1760011830 con 0x7f177815c890 2023-03-03T10:35:37.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.371+0000 7f1776ffd700 1 -- 172.21.15.102:0/1309229843 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1760011990 con 0x7f177815c890 2023-03-03T10:35:37.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.371+0000 7f1776ffd700 1 -- 172.21.15.102:0/1309229843 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1760011d20 con 0x7f177815c890 2023-03-03T10:35:37.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.371+0000 7f177dbff700 1 -- 172.21.15.102:0/1309229843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815c890 msgr2=0x7f17780c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:37.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.371+0000 7f177dbff700 1 --2- 172.21.15.102:0/1309229843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815c890 0x7f17780c81b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1760009840 tx=0x7f1760009b50 comp rx=0 tx=0).stop 2023-03-03T10:35:37.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.372+0000 7f177dbff700 1 -- 172.21.15.102:0/1309229843 shutdown_connections 2023-03-03T10:35:37.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.372+0000 7f177dbff700 1 --2- 172.21.15.102:0/1309229843 >> v2:172.21.15.102:3300/0 conn(0x7f17781531c0 0x7f177815c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.372+0000 7f177dbff700 1 --2- 172.21.15.102:0/1309229843 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f177815c890 0x7f17780c81b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.372+0000 7f177dbff700 1 -- 172.21.15.102:0/1309229843 >> 172.21.15.102:0/1309229843 conn(0x7f177806b410 msgr2=0x7f177806b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:37.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.372+0000 7f177dbff700 1 -- 172.21.15.102:0/1309229843 shutdown_connections 2023-03-03T10:35:37.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.372+0000 7f177dbff700 1 -- 172.21.15.102:0/1309229843 wait complete. 2023-03-03T10:35:37.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.374+0000 7f177dbff700 1 Processor -- start 2023-03-03T10:35:37.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.374+0000 7f177dbff700 1 -- start start 2023-03-03T10:35:37.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f177dbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17780c86f0 0x7f17781477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:37.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f177dbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17781531c0 0x7f17781556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:37.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f177dbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f177815c890 0x7f1778155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:37.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f177dbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778156660 con 0x7f17781531c0 2023-03-03T10:35:37.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f177dbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778156380 con 0x7f177815c890 2023-03-03T10:35:37.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f177dbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780c1ca0 con 0x7f17780c86f0 2023-03-03T10:35:37.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f17777fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17780c86f0 0x7f17781477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:37.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f17777fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17780c86f0 0x7f17781477a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:47082/0 (socket says 172.21.15.102:47082) 2023-03-03T10:35:37.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f1777fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f177815c890 0x7f1778155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:37.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.375+0000 7f17777fe700 1 -- 172.21.15.102:0/1651185709 learned_addr learned my addr 172.21.15.102:0/1651185709 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:35:37.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f176ffff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17781531c0 0x7f17781556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:37.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f17777fe700 1 -- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f177815c890 msgr2=0x7f1778155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:37.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f17777fe700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f177815c890 0x7f1778155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f17777fe700 1 -- 172.21.15.102:0/1651185709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17781531c0 msgr2=0x7f17781556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:37.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f17777fe700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17781531c0 0x7f17781556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f17777fe700 1 -- 172.21.15.102:0/1651185709 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1768009550 con 0x7f17780c86f0 2023-03-03T10:35:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f17777fe700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17780c86f0 0x7f17781477a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f176800ec10 tx=0x7f176800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f1774ff9700 1 -- 172.21.15.102:0/1651185709 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176800c850 con 0x7f17780c86f0 2023-03-03T10:35:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.376+0000 7f1774ff9700 1 -- 172.21.15.102:0/1651185709 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176800c9b0 con 0x7f17780c86f0 2023-03-03T10:35:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.377+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17600094a0 con 0x7f17780c86f0 2023-03-03T10:35:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.377+0000 7f1774ff9700 1 -- 172.21.15.102:0/1651185709 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176800cd40 con 0x7f17780c86f0 2023-03-03T10:35:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.377+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1778071aa0 con 0x7f17780c86f0 2023-03-03T10:35:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.378+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17780c8f30 con 0x7f17780c86f0 2023-03-03T10:35:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.379+0000 7f1774ff9700 1 -- 172.21.15.102:0/1651185709 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1768019030 con 0x7f17780c86f0 2023-03-03T10:35:37.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.380+0000 7f1774ff9700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17580682f0 0x7f175806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:37.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.380+0000 7f1774ff9700 1 -- 172.21.15.102:0/1651185709 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1768085de0 con 0x7f17780c86f0 2023-03-03T10:35:37.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.381+0000 7f176ffff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17580682f0 0x7f175806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:37.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.381+0000 7f176ffff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17580682f0 0x7f175806a7a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1760009840 tx=0x7f1760008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:37.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.385+0000 7f1774ff9700 1 -- 172.21.15.102:0/1651185709 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17680524f0 con 0x7f17780c86f0 2023-03-03T10:35:37.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.618+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17780c5f50 con 0x7f17780c86f0 2023-03-03T10:35:37.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.618+0000 7f1774ff9700 1 -- 172.21.15.102:0/1651185709 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1768021090 con 0x7f17780c86f0 2023-03-03T10:35:37.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:35:37.622 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:35:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17580682f0 msgr2=0x7f175806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17580682f0 0x7f175806a7a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1760009840 tx=0x7f1760008030 comp rx=0 tx=0).stop 2023-03-03T10:35:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17780c86f0 msgr2=0x7f17781477a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:37.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17780c86f0 0x7f17781477a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f176800ec10 tx=0x7f176800ef20 comp rx=0 tx=0).stop 2023-03-03T10:35:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 shutdown_connections 2023-03-03T10:35:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17580682f0 0x7f175806a7a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17780c86f0 0x7f17781477a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f177815c890 0x7f1778155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 --2- 172.21.15.102:0/1651185709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17781531c0 0x7f17781556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:37.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 >> 172.21.15.102:0/1651185709 conn(0x7f177806b410 msgr2=0x7f17780cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:37.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 shutdown_connections 2023-03-03T10:35:37.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:35:37.621+0000 7f177dbff700 1 -- 172.21.15.102:0/1651185709 wait complete. 2023-03-03T10:35:37.626 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:35:37.650 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T10:35:46.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:46.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:17.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:17.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:17.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:36:17.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.897+0000 7fc7429ff700 1 Processor -- start 2023-03-03T10:36:17.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.898+0000 7fc7429ff700 1 -- start start 2023-03-03T10:36:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc7429ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc73c153290 0x7fc73c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc7429ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc73c0cd8e0 0x7fc73c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc7429ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c153660 con 0x7fc73c1593c0 2023-03-03T10:36:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc7429ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c06c360 con 0x7fc73c153290 2023-03-03T10:36:17.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc7429ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c15cea0 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc73bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc73c153290 0x7fc73c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:17.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc73bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc73c153290 0x7fc73c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46430/0 (socket says 172.21.15.102:46430) 2023-03-03T10:36:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc73bfff700 1 -- 172.21.15.102:0/206230372 learned_addr learned my addr 172.21.15.102:0/206230372 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:36:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.899+0000 7fc740f9c700 1 --2- 172.21.15.102:0/206230372 >> v2:172.21.15.172:3300/0 conn(0x7fc73c0cd8e0 0x7fc73c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.900+0000 7fc73bfff700 1 -- 172.21.15.102:0/206230372 >> v2:172.21.15.172:3300/0 conn(0x7fc73c0cd8e0 msgr2=0x7fc73c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.900+0000 7fc73bfff700 1 --2- 172.21.15.102:0/206230372 >> v2:172.21.15.172:3300/0 conn(0x7fc73c0cd8e0 0x7fc73c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.900+0000 7fc73bfff700 1 -- 172.21.15.102:0/206230372 >> v1:172.21.15.102:6789/0 conn(0x7fc73c1593c0 legacy=0x7fc73c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.900+0000 7fc73bfff700 1 -- 172.21.15.102:0/206230372 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c06c7f0 con 0x7fc73c153290 2023-03-03T10:36:17.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.900+0000 7fc740f9c700 1 --2- 172.21.15.102:0/206230372 >> v2:172.21.15.172:3300/0 conn(0x7fc73c0cd8e0 0x7fc73c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:36:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.900+0000 7fc73affd700 1 -- 172.21.15.102:0/206230372 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544473727 0 0) 0x7fc73c153660 con 0x7fc73c1593c0 2023-03-03T10:36:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.900+0000 7fc73bfff700 1 --2- 172.21.15.102:0/206230372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c158e80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0c9390 tx=0x7fc72c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7cae83f358b01d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc73affd700 1 -- 172.21.15.102:0/206230372 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00b870 con 0x7fc73c153290 2023-03-03T10:36:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc73affd700 1 -- 172.21.15.102:0/206230372 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c00b9d0 con 0x7fc73c153290 2023-03-03T10:36:17.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc73affd700 1 -- 172.21.15.102:0/206230372 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00bd60 con 0x7fc73c153290 2023-03-03T10:36:17.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 -- 172.21.15.102:0/206230372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 msgr2=0x7fc73c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:17.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 --2- 172.21.15.102:0/206230372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c158e80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0c9390 tx=0x7fc72c009bd0 comp rx=0 tx=0).stop 2023-03-03T10:36:17.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 -- 172.21.15.102:0/206230372 shutdown_connections 2023-03-03T10:36:17.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 --2- 172.21.15.102:0/206230372 >> v2:172.21.15.172:3300/0 conn(0x7fc73c0cd8e0 0x7fc73c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 --2- 172.21.15.102:0/206230372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c158e80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 -- 172.21.15.102:0/206230372 >> 172.21.15.102:0/206230372 conn(0x7fc73c065fb0 msgr2=0x7fc73c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 -- 172.21.15.102:0/206230372 shutdown_connections 2023-03-03T10:36:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.901+0000 7fc7429ff700 1 -- 172.21.15.102:0/206230372 wait complete. 2023-03-03T10:36:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.903+0000 7fc7429ff700 1 Processor -- start 2023-03-03T10:36:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.903+0000 7fc7429ff700 1 -- start start 2023-03-03T10:36:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.904+0000 7fc7429ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc73c0cd8e0 0x7fc73c157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.904+0000 7fc7429ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c1f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc7429ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc73c1593c0 0x7fc73c0c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc7429ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0b3f10 con 0x7fc73c153290 2023-03-03T10:36:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc7429ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c1ef790 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc7429ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0c72e0 con 0x7fc73c1593c0 2023-03-03T10:36:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc73c0cd8e0 0x7fc73c157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c1f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc73c0cd8e0 0x7fc73c157f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36068/0 (socket says 172.21.15.102:36068) 2023-03-03T10:36:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 -- 172.21.15.102:0/2464945359 learned_addr learned my addr 172.21.15.102:0/2464945359 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:36:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc740f9c700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc73c1593c0 0x7fc73c0c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 -- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc73c1593c0 msgr2=0x7fc73c0c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:17.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc73c1593c0 0x7fc73c0c4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 -- 172.21.15.102:0/2464945359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 msgr2=0x7fc73c1f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c1f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73bfff700 1 -- 172.21.15.102:0/2464945359 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c009550 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.905+0000 7fc73b7fe700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c1f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.906+0000 7fc73bfff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc73c0cd8e0 0x7fc73c157f80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc73c074540 tx=0x7fc72c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.906+0000 7fc738ff9700 1 -- 172.21.15.102:0/2464945359 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c004250 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.906+0000 7fc738ff9700 1 -- 172.21.15.102:0/2464945359 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c01f070 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.906+0000 7fc738ff9700 1 -- 172.21.15.102:0/2464945359 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c0045b0 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.906+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c0ccde0 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.906+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c15d160 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.907+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c0cd0c0 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.908+0000 7fc738ff9700 1 -- 172.21.15.102:0/2464945359 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc72c00f050 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.909+0000 7fc738ff9700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7280682f0 0x7fc72806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.910+0000 7fc738ff9700 1 -- 172.21.15.102:0/2464945359 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc72c087100 con 0x7fc73c0cd8e0 2023-03-03T10:36:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.910+0000 7fc73b7fe700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7280682f0 0x7fc72806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.910+0000 7fc73b7fe700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7280682f0 0x7fc72806a7a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0c9670 tx=0x7fc730009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:17.914+0000 7fc738ff9700 1 -- 172.21.15.102:0/2464945359 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc72c053810 con 0x7fc73c0cd8e0 2023-03-03T10:36:18.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.145+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc73c04e6e0 con 0x7fc73c0cd8e0 2023-03-03T10:36:18.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.146+0000 7fc738ff9700 1 -- 172.21.15.102:0/2464945359 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc72c056e30 con 0x7fc73c0cd8e0 2023-03-03T10:36:18.149 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:36:18.149 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:36:18.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7280682f0 msgr2=0x7fc72806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:18.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7280682f0 0x7fc72806a7a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0c9670 tx=0x7fc730009040 comp rx=0 tx=0).stop 2023-03-03T10:36:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc73c0cd8e0 msgr2=0x7fc73c157f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc73c0cd8e0 0x7fc73c157f80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc73c074540 tx=0x7fc72c004750 comp rx=0 tx=0).stop 2023-03-03T10:36:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 shutdown_connections 2023-03-03T10:36:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7280682f0 0x7fc72806a7a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc73c1593c0 0x7fc73c0c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc73c0cd8e0 0x7fc73c157f80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:18.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 --2- 172.21.15.102:0/2464945359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc73c153290 0x7fc73c1f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.148+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 >> 172.21.15.102:0/2464945359 conn(0x7fc73c065fb0 msgr2=0x7fc73c150e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.149+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 shutdown_connections 2023-03-03T10:36:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:18.149+0000 7fc7429ff700 1 -- 172.21.15.102:0/2464945359 wait complete. 2023-03-03T10:36:18.153 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:36:18.179 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T10:36:47.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:47.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:50.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:36:50.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.427+0000 7f8a7359d700 1 Processor -- start 2023-03-03T10:36:50.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.427+0000 7f8a7359d700 1 -- start start 2023-03-03T10:36:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a7359d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8a6c0ced00 0x7f8a6c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:50.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a7359d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a6c0c41b0 0x7f8a6c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:50.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a7359d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6c0c3d20 con 0x7f8a6c0b3970 2023-03-03T10:36:50.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a7359d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6c0c4930 con 0x7f8a6c0c41b0 2023-03-03T10:36:50.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a7359d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6c0cf610 con 0x7f8a6c0ced00 2023-03-03T10:36:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a72d9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a6c0c41b0 0x7f8a6c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a72d9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a6c0c41b0 0x7f8a6c0cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48222/0 (socket says 172.21.15.102:48222) 2023-03-03T10:36:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a72d9c700 1 -- 172.21.15.102:0/4056140772 learned_addr learned my addr 172.21.15.102:0/4056140772 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:36:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.429+0000 7f8a7259b700 1 --2- 172.21.15.102:0/4056140772 >> v2:172.21.15.172:3300/0 conn(0x7f8a6c0ced00 0x7f8a6c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a72d9c700 1 -- 172.21.15.102:0/4056140772 >> v2:172.21.15.172:3300/0 conn(0x7f8a6c0ced00 msgr2=0x7f8a6c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a72d9c700 1 --2- 172.21.15.102:0/4056140772 >> v2:172.21.15.172:3300/0 conn(0x7f8a6c0ced00 0x7f8a6c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a72d9c700 1 -- 172.21.15.102:0/4056140772 >> v1:172.21.15.172:6789/0 conn(0x7f8a6c0b3970 legacy=0x7f8a6c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a72d9c700 1 -- 172.21.15.102:0/4056140772 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c15cba0 con 0x7f8a6c0c41b0 2023-03-03T10:36:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a72d9c700 1 --2- 172.21.15.102:0/4056140772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0c41b0 0x7f8a6c0cf0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0b1b40 tx=0x7f8a6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c03e5f797cec014 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:50.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a71599700 1 -- 172.21.15.102:0/4056140772 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400e070 con 0x7f8a6c0c41b0 2023-03-03T10:36:50.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a71599700 1 -- 172.21.15.102:0/4056140772 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6400b8d0 con 0x7f8a6c0c41b0 2023-03-03T10:36:50.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.430+0000 7f8a71599700 1 -- 172.21.15.102:0/4056140772 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400bc80 con 0x7f8a6c0c41b0 2023-03-03T10:36:50.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 -- 172.21.15.102:0/4056140772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0c41b0 msgr2=0x7f8a6c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:50.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 --2- 172.21.15.102:0/4056140772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0c41b0 0x7f8a6c0cf0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0b1b40 tx=0x7f8a6400b330 comp rx=0 tx=0).stop 2023-03-03T10:36:50.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 -- 172.21.15.102:0/4056140772 shutdown_connections 2023-03-03T10:36:50.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 --2- 172.21.15.102:0/4056140772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0c41b0 0x7f8a6c0cf0d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 --2- 172.21.15.102:0/4056140772 >> v2:172.21.15.172:3300/0 conn(0x7f8a6c0ced00 0x7f8a6c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 -- 172.21.15.102:0/4056140772 >> 172.21.15.102:0/4056140772 conn(0x7f8a6c08b1c0 msgr2=0x7f8a6c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:50.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 -- 172.21.15.102:0/4056140772 shutdown_connections 2023-03-03T10:36:50.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.431+0000 7f8a7359d700 1 -- 172.21.15.102:0/4056140772 wait complete. 2023-03-03T10:36:50.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.433+0000 7f8a7359d700 1 Processor -- start 2023-03-03T10:36:50.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.433+0000 7f8a7359d700 1 -- start start 2023-03-03T10:36:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a7359d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a6c0b3970 0x7f8a6c15bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a7359d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a6c0c41b0 0x7f8a6c14af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a7359d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0ced00 0x7f8a6c0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a7359d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c14cab0 con 0x7f8a6c0ced00 2023-03-03T10:36:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a7359d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c14b460 con 0x7f8a6c0b3970 2023-03-03T10:36:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a7359d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c14efa0 con 0x7f8a6c0c41b0 2023-03-03T10:36:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a7259b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a6c0b3970 0x7f8a6c15bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.434+0000 7f8a72d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0ced00 0x7f8a6c0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:50.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a71d9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a6c0c41b0 0x7f8a6c14af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:50.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a71d9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a6c0c41b0 0x7f8a6c14af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51758/0 (socket says 172.21.15.102:51758) 2023-03-03T10:36:50.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a71d9a700 1 -- 172.21.15.102:0/1874859578 learned_addr learned my addr 172.21.15.102:0/1874859578 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:36:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a72d9c700 1 -- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a6c0c41b0 msgr2=0x7f8a6c14af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a72d9c700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a6c0c41b0 0x7f8a6c14af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a72d9c700 1 -- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a6c0b3970 msgr2=0x7f8a6c15bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a72d9c700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a6c0b3970 0x7f8a6c15bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a72d9c700 1 -- 172.21.15.102:0/1874859578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a64009b30 con 0x7f8a6c0ced00 2023-03-03T10:36:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a71d9a700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a6c0c41b0 0x7f8a6c14af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:36:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.435+0000 7f8a7259b700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a6c0b3970 0x7f8a6c15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:36:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.436+0000 7f8a72d9c700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0ced00 0x7f8a6c0cd300 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8a64015000 tx=0x7f8a64004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:50.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.436+0000 7f8a62ffd700 1 -- 172.21.15.102:0/1874859578 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a640047c0 con 0x7f8a6c0ced00 2023-03-03T10:36:50.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.436+0000 7f8a62ffd700 1 -- 172.21.15.102:0/1874859578 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a640092e0 con 0x7f8a6c0ced00 2023-03-03T10:36:50.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.436+0000 7f8a62ffd700 1 -- 172.21.15.102:0/1874859578 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6401c680 con 0x7f8a6c0ced00 2023-03-03T10:36:50.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.436+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c06bf20 con 0x7f8a6c0ced00 2023-03-03T10:36:50.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.436+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c15ce80 con 0x7f8a6c0ced00 2023-03-03T10:36:50.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.437+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c06c200 con 0x7f8a6c0ced00 2023-03-03T10:36:50.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.438+0000 7f8a62ffd700 1 -- 172.21.15.102:0/1874859578 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a64022030 con 0x7f8a6c0ced00 2023-03-03T10:36:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.439+0000 7f8a62ffd700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a50068320 0x7f8a5006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.440+0000 7f8a62ffd700 1 -- 172.21.15.102:0/1874859578 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8a640874d0 con 0x7f8a6c0ced00 2023-03-03T10:36:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.440+0000 7f8a7259b700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a50068320 0x7f8a5006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.441+0000 7f8a7259b700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a50068320 0x7f8a5006a7d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009fd0 tx=0x7f8a68009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.444+0000 7f8a62ffd700 1 -- 172.21.15.102:0/1874859578 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a64053be0 con 0x7f8a6c0ced00 2023-03-03T10:36:50.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.679+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a6c04e6e0 con 0x7f8a6c0ced00 2023-03-03T10:36:50.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.679+0000 7f8a62ffd700 1 -- 172.21.15.102:0/1874859578 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8a64057200 con 0x7f8a6c0ced00 2023-03-03T10:36:50.681 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:36:50.682 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:36:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a50068320 msgr2=0x7f8a5006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a50068320 0x7f8a5006a7d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009fd0 tx=0x7f8a68009210 comp rx=0 tx=0).stop 2023-03-03T10:36:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0ced00 msgr2=0x7f8a6c0cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0ced00 0x7f8a6c0cd300 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8a64015000 tx=0x7f8a64004b20 comp rx=0 tx=0).stop 2023-03-03T10:36:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 shutdown_connections 2023-03-03T10:36:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0ced00 0x7f8a6c0cd300 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a6c0b3970 0x7f8a6c15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8a50068320 0x7f8a5006a7d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 --2- 172.21.15.102:0/1874859578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8a6c0c41b0 0x7f8a6c14af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.682+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 >> 172.21.15.102:0/1874859578 conn(0x7f8a6c08b1c0 msgr2=0x7f8a6c065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.683+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 shutdown_connections 2023-03-03T10:36:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:36:50.683+0000 7f8a7359d700 1 -- 172.21.15.102:0/1874859578 wait complete. 2023-03-03T10:36:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:36:50.711 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2023-03-03T10:37:17.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:17.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:47.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:47.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:17.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:17.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:47.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:38:47.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:47.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:47.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.979+0000 7f16743ff700 1 Processor -- start 2023-03-03T10:38:47.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.980+0000 7f16743ff700 1 -- start start 2023-03-03T10:38:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.981+0000 7f16743ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f166c153260 0x7f166c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.981+0000 7f16743ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c0cf940 con 0x7f166c0cac20 2023-03-03T10:38:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.982+0000 7f16743ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c0cfb90 con 0x7f166c0c9cb0 2023-03-03T10:38:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.982+0000 7f16743ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c0cfe70 con 0x7f166c153260 2023-03-03T10:38:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.982+0000 7f167199a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f166c0c9cb0 0x7f166c0cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:43854/0 (socket says 172.21.15.102:43854) 2023-03-03T10:38:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.982+0000 7f167199a700 1 -- 172.21.15.102:0/795544835 learned_addr learned my addr 172.21.15.102:0/795544835 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:38:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.982+0000 7f167219b700 1 --2- 172.21.15.102:0/795544835 >> v2:172.21.15.102:3300/0 conn(0x7f166c153260 0x7f166c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:47.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.982+0000 7f1671199700 1 -- 172.21.15.102:0/795544835 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334687637 0 0) 0x7f166c0cfb90 con 0x7f166c0c9cb0 2023-03-03T10:38:47.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.982+0000 7f1671199700 1 -- 172.21.15.102:0/795544835 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16580035b0 con 0x7f166c0c9cb0 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f1671199700 1 -- 172.21.15.102:0/795544835 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204568408 0 0) 0x7f166c0cf940 con 0x7f166c0cac20 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f167219b700 1 -- 172.21.15.102:0/795544835 >> v1:172.21.15.172:6789/0 conn(0x7f166c0c9cb0 legacy=0x7f166c0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f167219b700 1 -- 172.21.15.102:0/795544835 >> v1:172.21.15.102:6789/0 conn(0x7f166c0cac20 legacy=0x7f166c163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f167219b700 1 -- 172.21.15.102:0/795544835 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c158f80 con 0x7f166c153260 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f167219b700 1 --2- 172.21.15.102:0/795544835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c153260 0x7f166c0ca6e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1668009840 tx=0x7f1668009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b530a874470392 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f1671199700 1 -- 172.21.15.102:0/795544835 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668011830 con 0x7f166c153260 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f1671199700 1 -- 172.21.15.102:0/795544835 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1668011990 con 0x7f166c153260 2023-03-03T10:38:47.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.983+0000 7f1671199700 1 -- 172.21.15.102:0/795544835 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668011d20 con 0x7f166c153260 2023-03-03T10:38:47.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.984+0000 7f16743ff700 1 -- 172.21.15.102:0/795544835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c153260 msgr2=0x7f166c0ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:47.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.984+0000 7f16743ff700 1 --2- 172.21.15.102:0/795544835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c153260 0x7f166c0ca6e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1668009840 tx=0x7f1668009b50 comp rx=0 tx=0).stop 2023-03-03T10:38:47.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.984+0000 7f16743ff700 1 -- 172.21.15.102:0/795544835 shutdown_connections 2023-03-03T10:38:47.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.984+0000 7f16743ff700 1 --2- 172.21.15.102:0/795544835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c153260 0x7f166c0ca6e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:47.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.984+0000 7f16743ff700 1 -- 172.21.15.102:0/795544835 >> 172.21.15.102:0/795544835 conn(0x7f166c066c00 msgr2=0x7f166c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:38:47.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.984+0000 7f16743ff700 1 -- 172.21.15.102:0/795544835 shutdown_connections 2023-03-03T10:38:47.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.984+0000 7f16743ff700 1 -- 172.21.15.102:0/795544835 wait complete. 2023-03-03T10:38:47.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.986+0000 7f16743ff700 1 Processor -- start 2023-03-03T10:38:47.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.987+0000 7f16743ff700 1 -- start start 2023-03-03T10:38:47.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f16743ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f166c0cac20 0x7f166c1f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:47.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f16743ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f166c153260 0x7f166c0b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:47.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f16743ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c1f1ce0 0x7f166c0b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:47.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f16743ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b23f0 con 0x7f166c1f1ce0 2023-03-03T10:38:47.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f16743ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0aed30 con 0x7f166c153260 2023-03-03T10:38:47.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f16743ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c14b560 con 0x7f166c0cac20 2023-03-03T10:38:47.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167219b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f166c0cac20 0x7f166c1f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:47.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167199a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f166c153260 0x7f166c0b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:47.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167219b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f166c0cac20 0x7f166c1f17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:43184/0 (socket says 172.21.15.102:43184) 2023-03-03T10:38:47.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167219b700 1 -- 172.21.15.102:0/1321055118 learned_addr learned my addr 172.21.15.102:0/1321055118 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:38:47.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167299c700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c1f1ce0 0x7f166c0b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:47.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167199a700 1 -- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f166c0cac20 msgr2=0x7f166c1f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:47.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167199a700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f166c0cac20 0x7f166c1f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:47.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167199a700 1 -- 172.21.15.102:0/1321055118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c1f1ce0 msgr2=0x7f166c0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:47.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167199a700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c1f1ce0 0x7f166c0b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:47.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167199a700 1 -- 172.21.15.102:0/1321055118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16680094a0 con 0x7f166c153260 2023-03-03T10:38:47.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.988+0000 7f167219b700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f166c0cac20 0x7f166c1f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:38:47.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.989+0000 7f167199a700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f166c153260 0x7f166c0b2880 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c490 tx=0x7f165c00c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:47.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.989+0000 7f1662ffd700 1 -- 172.21.15.102:0/1321055118 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c005d70 con 0x7f166c153260 2023-03-03T10:38:47.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.989+0000 7f1662ffd700 1 -- 172.21.15.102:0/1321055118 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c0070d0 con 0x7f166c153260 2023-03-03T10:38:47.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.989+0000 7f1662ffd700 1 -- 172.21.15.102:0/1321055118 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c007440 con 0x7f166c153260 2023-03-03T10:38:47.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.989+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c159540 con 0x7f166c153260 2023-03-03T10:38:47.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.989+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c159260 con 0x7f166c153260 2023-03-03T10:38:47.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.991+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c159820 con 0x7f166c153260 2023-03-03T10:38:47.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.992+0000 7f1662ffd700 1 -- 172.21.15.102:0/1321055118 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f165c010070 con 0x7f166c153260 2023-03-03T10:38:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.993+0000 7f1662ffd700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.993+0000 7f1662ffd700 1 -- 172.21.15.102:0/1321055118 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f165c084bc0 con 0x7f166c153260 2023-03-03T10:38:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.994+0000 7f167219b700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.994+0000 7f167219b700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1668000c00 tx=0x7f166801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:48.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:47.997+0000 7f1662ffd700 1 -- 172.21.15.102:0/1321055118 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f165c04a5e0 con 0x7f166c153260 2023-03-03T10:38:48.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.235+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f166c0b2560 con 0x7f166c153260 2023-03-03T10:38:48.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.237+0000 7f1662ffd700 1 -- 172.21.15.102:0/1321055118 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f165c051380 con 0x7f166c153260 2023-03-03T10:38:48.239 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:38:48.239 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:38:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 msgr2=0x7f165006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1668000c00 tx=0x7f166801c040 comp rx=0 tx=0).stop 2023-03-03T10:38:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f166c153260 msgr2=0x7f166c0b2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f166c153260 0x7f166c0b2880 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c490 tx=0x7f165c00c7a0 comp rx=0 tx=0).stop 2023-03-03T10:38:48.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 shutdown_connections 2023-03-03T10:38:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f166c0cac20 0x7f166c1f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f166c1f1ce0 0x7f166c0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 --2- 172.21.15.102:0/1321055118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f166c153260 0x7f166c0b2880 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 >> 172.21.15.102:0/1321055118 conn(0x7f166c0c9cb0 msgr2=0x7f166c0cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:38:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 shutdown_connections 2023-03-03T10:38:48.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:48.239+0000 7f16743ff700 1 -- 172.21.15.102:0/1321055118 wait complete. 2023-03-03T10:38:48.244 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:38:48.268 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2023-03-03T10:38:54.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:38:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.516+0000 7ff703dff700 1 Processor -- start 2023-03-03T10:38:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.516+0000 7ff703dff700 1 -- start start 2023-03-03T10:38:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff703dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff6fc153290 0x7ff6fc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff703dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff6fc0cd8e0 0x7ff6fc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff703dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc153660 con 0x7ff6fc1593c0 2023-03-03T10:38:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff703dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6fc06c360 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff703dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6fc15cea0 con 0x7ff6fc153290 2023-03-03T10:38:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff70239c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff6fc0cd8e0 0x7ff6fc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff70239c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff6fc0cd8e0 0x7ff6fc0ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58966/0 (socket says 172.21.15.102:58966) 2023-03-03T10:38:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff70139a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff6fc1593c0 0x7ff6fc0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:43858/0 (socket says 172.21.15.102:43858) 2023-03-03T10:38:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff70239c700 1 -- 172.21.15.102:0/2087629488 learned_addr learned my addr 172.21.15.102:0/2087629488 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:38:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.518+0000 7ff701b9b700 1 --2- 172.21.15.102:0/2087629488 >> v2:172.21.15.172:3300/0 conn(0x7ff6fc153290 0x7ff6fc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff70239c700 1 -- 172.21.15.102:0/2087629488 >> v2:172.21.15.172:3300/0 conn(0x7ff6fc153290 msgr2=0x7ff6fc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:38:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff70239c700 1 --2- 172.21.15.102:0/2087629488 >> v2:172.21.15.172:3300/0 conn(0x7ff6fc153290 0x7ff6fc158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff70239c700 1 -- 172.21.15.102:0/2087629488 >> v1:172.21.15.172:6789/0 conn(0x7ff6fc1593c0 legacy=0x7ff6fc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff70239c700 1 -- 172.21.15.102:0/2087629488 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc06c7f0 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff70239c700 1 --2- 172.21.15.102:0/2087629488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0ce530 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc08cc70 tx=0x7ff6f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdd8ae59620f978 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff700b99700 1 -- 172.21.15.102:0/2087629488 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f400e070 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff700b99700 1 -- 172.21.15.102:0/2087629488 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f400b910 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.519+0000 7ff700b99700 1 -- 172.21.15.102:0/2087629488 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f400bcc0 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 -- 172.21.15.102:0/2087629488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc0cd8e0 msgr2=0x7ff6fc0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 --2- 172.21.15.102:0/2087629488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0ce530 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc08cc70 tx=0x7ff6f400b330 comp rx=0 tx=0).stop 2023-03-03T10:38:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 -- 172.21.15.102:0/2087629488 shutdown_connections 2023-03-03T10:38:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 --2- 172.21.15.102:0/2087629488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0ce530 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 --2- 172.21.15.102:0/2087629488 >> v2:172.21.15.172:3300/0 conn(0x7ff6fc153290 0x7ff6fc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 -- 172.21.15.102:0/2087629488 >> 172.21.15.102:0/2087629488 conn(0x7ff6fc065fb0 msgr2=0x7ff6fc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:38:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 -- 172.21.15.102:0/2087629488 shutdown_connections 2023-03-03T10:38:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.520+0000 7ff703dff700 1 -- 172.21.15.102:0/2087629488 wait complete. 2023-03-03T10:38:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.522+0000 7ff703dff700 1 Processor -- start 2023-03-03T10:38:54.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.522+0000 7ff703dff700 1 -- start start 2023-03-03T10:38:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff703dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff703dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6fc153290 0x7ff6fc1501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff703dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc1593c0 0x7ff6fc1f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff703dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc1567f0 con 0x7ff6fc1593c0 2023-03-03T10:38:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff703dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0cf920 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff703dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0cf3e0 con 0x7ff6fc153290 2023-03-03T10:38:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff701b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff70139a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6fc153290 0x7ff6fc1501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff70239c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc1593c0 0x7ff6fc1f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff701b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37106/0 (socket says 172.21.15.102:37106) 2023-03-03T10:38:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.523+0000 7ff701b9b700 1 -- 172.21.15.102:0/1368540408 learned_addr learned my addr 172.21.15.102:0/1368540408 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:38:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff701b9b700 1 -- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6fc153290 msgr2=0x7ff6fc1501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff701b9b700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6fc153290 0x7ff6fc1501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff701b9b700 1 -- 172.21.15.102:0/1368540408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc1593c0 msgr2=0x7ff6fc1f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff701b9b700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc1593c0 0x7ff6fc1f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff701b9b700 1 -- 172.21.15.102:0/1368540408 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4009b30 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff701b9b700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0c87b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800e9b0 tx=0x7ff6f800ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff6f27fc700 1 -- 172.21.15.102:0/1368540408 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f800ee50 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff6f27fc700 1 -- 172.21.15.102:0/1368540408 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f800c850 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6fc0ccde0 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.524+0000 7ff6f27fc700 1 -- 172.21.15.102:0/1368540408 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f800cbc0 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.525+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6fc15d160 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.526+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6fc0cd0c0 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.527+0000 7ff6f27fc700 1 -- 172.21.15.102:0/1368540408 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff6f8019030 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.528+0000 7ff6f27fc700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff6e0068320 0x7ff6e006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:54.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.528+0000 7ff6f27fc700 1 -- 172.21.15.102:0/1368540408 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff6f8085c40 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.528+0000 7ff70139a700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff6e0068320 0x7ff6e006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.529+0000 7ff70139a700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff6e0068320 0x7ff6e006a7d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc15a1c0 tx=0x7ff6ec007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:54.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.532+0000 7ff6f27fc700 1 -- 172.21.15.102:0/1368540408 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff6f8052350 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.768+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6fc04e6e0 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.769+0000 7ff6f27fc700 1 -- 172.21.15.102:0/1368540408 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff6f8017090 con 0x7ff6fc0cd8e0 2023-03-03T10:38:54.771 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:38:54.771 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:38:54.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff6e0068320 msgr2=0x7ff6e006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff6e0068320 0x7ff6e006a7d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc15a1c0 tx=0x7ff6ec007ed0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6fc0cd8e0 msgr2=0x7ff6fc0c87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0c87b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800e9b0 tx=0x7ff6f800ecc0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 shutdown_connections 2023-03-03T10:38:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc1593c0 0x7ff6fc1f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6fc0cd8e0 0x7ff6fc0c87b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff6e0068320 0x7ff6e006a7d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 --2- 172.21.15.102:0/1368540408 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6fc153290 0x7ff6fc1501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 >> 172.21.15.102:0/1368540408 conn(0x7ff6fc065fb0 msgr2=0x7ff6fc15ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:38:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.771+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 shutdown_connections 2023-03-03T10:38:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:38:54.772+0000 7ff703dff700 1 -- 172.21.15.102:0/1368540408 wait complete. 2023-03-03T10:38:54.775 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:38:54.800 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2023-03-03T10:39:18.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:18.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:48.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:48.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T10:40:00.000+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:40:14.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:40:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.069+0000 7f2e695ff700 1 Processor -- start 2023-03-03T10:40:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.070+0000 7f2e695ff700 1 -- start start 2023-03-03T10:40:15.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e695ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e64153290 0x7f2e64158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:15.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e695ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2e640cd8e0 0x7f2e640ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e695ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e64153660 con 0x7f2e641593c0 2023-03-03T10:40:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e695ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6406c360 con 0x7f2e64153290 2023-03-03T10:40:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e695ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6415cea0 con 0x7f2e640cd8e0 2023-03-03T10:40:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e62ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e64153290 0x7f2e64158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e62ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e64153290 0x7f2e64158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46340/0 (socket says 172.21.15.102:46340) 2023-03-03T10:40:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e62ffd700 1 -- 172.21.15.102:0/2811335099 learned_addr learned my addr 172.21.15.102:0/2811335099 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:40:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e637fe700 1 --2- 172.21.15.102:0/2811335099 >> v2:172.21.15.172:3300/0 conn(0x7f2e640cd8e0 0x7f2e640ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:15.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.071+0000 7f2e627fc700 1 --1- 172.21.15.102:0/2811335099 >> v1:172.21.15.102:6789/0 conn(0x7f2e641593c0 0x7f2e640cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51484/0 (socket says 172.21.15.102:51484) 2023-03-03T10:40:15.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.072+0000 7f2e62ffd700 1 -- 172.21.15.102:0/2811335099 >> v2:172.21.15.172:3300/0 conn(0x7f2e640cd8e0 msgr2=0x7f2e640ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:15.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.072+0000 7f2e62ffd700 1 --2- 172.21.15.102:0/2811335099 >> v2:172.21.15.172:3300/0 conn(0x7f2e640cd8e0 0x7f2e640ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.072+0000 7f2e62ffd700 1 -- 172.21.15.102:0/2811335099 >> v1:172.21.15.102:6789/0 conn(0x7f2e641593c0 legacy=0x7f2e640cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:15.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.072+0000 7f2e62ffd700 1 -- 172.21.15.102:0/2811335099 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6406c7f0 con 0x7f2e64153290 2023-03-03T10:40:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.072+0000 7f2e62ffd700 1 --2- 172.21.15.102:0/2811335099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e64153290 0x7f2e64158e80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2e640c9390 tx=0x7f2e54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83328a66c9ac45b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.072+0000 7f2e61ffb700 1 -- 172.21.15.102:0/2811335099 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54011830 con 0x7f2e64153290 2023-03-03T10:40:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e61ffb700 1 -- 172.21.15.102:0/2811335099 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54011990 con 0x7f2e64153290 2023-03-03T10:40:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e61ffb700 1 -- 172.21.15.102:0/2811335099 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54011d20 con 0x7f2e64153290 2023-03-03T10:40:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 -- 172.21.15.102:0/2811335099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e64153290 msgr2=0x7f2e64158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 --2- 172.21.15.102:0/2811335099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e64153290 0x7f2e64158e80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2e640c9390 tx=0x7f2e54009bd0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 -- 172.21.15.102:0/2811335099 shutdown_connections 2023-03-03T10:40:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 --2- 172.21.15.102:0/2811335099 >> v2:172.21.15.172:3300/0 conn(0x7f2e640cd8e0 0x7f2e640ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 --2- 172.21.15.102:0/2811335099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e64153290 0x7f2e64158e80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 -- 172.21.15.102:0/2811335099 >> 172.21.15.102:0/2811335099 conn(0x7f2e64065fb0 msgr2=0x7f2e640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 -- 172.21.15.102:0/2811335099 shutdown_connections 2023-03-03T10:40:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.073+0000 7f2e695ff700 1 -- 172.21.15.102:0/2811335099 wait complete. 2023-03-03T10:40:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.075+0000 7f2e695ff700 1 Processor -- start 2023-03-03T10:40:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.076+0000 7f2e695ff700 1 -- start start 2023-03-03T10:40:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.076+0000 7f2e695ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640cd8e0 0x7f2e64156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e695ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e64153290 0x7f2e640c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e695ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641593c0 0x7f2e640c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e695ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6415b900 con 0x7f2e641593c0 2023-03-03T10:40:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e695ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64156a80 con 0x7f2e640cd8e0 2023-03-03T10:40:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e695ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64153990 con 0x7f2e64153290 2023-03-03T10:40:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e637fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641593c0 0x7f2e640c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e637fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641593c0 0x7f2e640c8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46346/0 (socket says 172.21.15.102:46346) 2023-03-03T10:40:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e627fc700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e64153290 0x7f2e640c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e62ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640cd8e0 0x7f2e64156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e637fe700 1 -- 172.21.15.102:0/1646883717 learned_addr learned my addr 172.21.15.102:0/1646883717 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:40:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e62ffd700 1 -- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e64153290 msgr2=0x7f2e640c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e62ffd700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e64153290 0x7f2e640c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e62ffd700 1 -- 172.21.15.102:0/1646883717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641593c0 msgr2=0x7f2e640c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e62ffd700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641593c0 0x7f2e640c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e62ffd700 1 -- 172.21.15.102:0/1646883717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e54009550 con 0x7f2e640cd8e0 2023-03-03T10:40:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.077+0000 7f2e637fe700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641593c0 0x7f2e640c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:40:15.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.078+0000 7f2e62ffd700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640cd8e0 0x7f2e64156110 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2e64159fb0 tx=0x7f2e54004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.078+0000 7f2e4b7fe700 1 -- 172.21.15.102:0/1646883717 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54004380 con 0x7f2e640cd8e0 2023-03-03T10:40:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.078+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e640ccde0 con 0x7f2e640cd8e0 2023-03-03T10:40:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.078+0000 7f2e4b7fe700 1 -- 172.21.15.102:0/1646883717 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5401f070 con 0x7f2e640cd8e0 2023-03-03T10:40:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.078+0000 7f2e4b7fe700 1 -- 172.21.15.102:0/1646883717 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e540036a0 con 0x7f2e640cd8e0 2023-03-03T10:40:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.078+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6415d160 con 0x7f2e640cd8e0 2023-03-03T10:40:15.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.079+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e640cd0c0 con 0x7f2e640cd8e0 2023-03-03T10:40:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.081+0000 7f2e4b7fe700 1 -- 172.21.15.102:0/1646883717 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e5402a020 con 0x7f2e640cd8e0 2023-03-03T10:40:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.082+0000 7f2e4b7fe700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e500682f0 0x7f2e5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.082+0000 7f2e4b7fe700 1 -- 172.21.15.102:0/1646883717 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2e54087b90 con 0x7f2e640cd8e0 2023-03-03T10:40:15.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.082+0000 7f2e627fc700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e500682f0 0x7f2e5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:15.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.083+0000 7f2e627fc700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e500682f0 0x7f2e5006a7a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2e64159c70 tx=0x7f2e58007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:15.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.085+0000 7f2e4b7fe700 1 -- 172.21.15.102:0/1646883717 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e540542a0 con 0x7f2e640cd8e0 2023-03-03T10:40:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.321+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e6404e6e0 con 0x7f2e640cd8e0 2023-03-03T10:40:15.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.322+0000 7f2e4b7fe700 1 -- 172.21.15.102:0/1646883717 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2e54015090 con 0x7f2e640cd8e0 2023-03-03T10:40:15.325 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:40:15.326 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:40:15.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e500682f0 msgr2=0x7f2e5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:15.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e500682f0 0x7f2e5006a7a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2e64159c70 tx=0x7f2e58007ed0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640cd8e0 msgr2=0x7f2e64156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:15.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640cd8e0 0x7f2e64156110 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2e64159fb0 tx=0x7f2e54004750 comp rx=0 tx=0).stop 2023-03-03T10:40:15.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 shutdown_connections 2023-03-03T10:40:15.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e641593c0 0x7f2e640c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e640cd8e0 0x7f2e64156110 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e500682f0 0x7f2e5006a7a0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 --2- 172.21.15.102:0/1646883717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e64153290 0x7f2e640c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:15.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 >> 172.21.15.102:0/1646883717 conn(0x7f2e64065fb0 msgr2=0x7f2e64156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:15.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 shutdown_connections 2023-03-03T10:40:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:15.325+0000 7f2e695ff700 1 -- 172.21.15.102:0/1646883717 wait complete. 2023-03-03T10:40:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:40:15.356 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-03-03T10:40:18.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:18.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:38.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:40:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.604+0000 7f36f7fff700 1 Processor -- start 2023-03-03T10:40:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.604+0000 7f36f7fff700 1 -- start start 2023-03-03T10:40:38.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f36f80b3970 0x7f36f80c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f7fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80c3d20 con 0x7f36f80ced00 2023-03-03T10:40:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f7fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f80c4930 con 0x7f36f80b3970 2023-03-03T10:40:38.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80cf610 con 0x7f36f80c41b0 2023-03-03T10:40:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f6ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f36f80ced00 0x7f36f8158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59596/0 (socket says 172.21.15.102:59596) 2023-03-03T10:40:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f77fe700 1 --2- 172.21.15.102:0/155269351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:38.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f6ffd700 1 -- 172.21.15.102:0/155269351 learned_addr learned my addr 172.21.15.102:0/155269351 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:40:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.606+0000 7f36f67fc700 1 --2- 172.21.15.102:0/155269351 >> v2:172.21.15.102:3300/0 conn(0x7f36f80b3970 0x7f36f80c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f5ffb700 1 -- 172.21.15.102:0/155269351 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147847521 0 0) 0x7f36f80c3d20 con 0x7f36f80ced00 2023-03-03T10:40:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f5ffb700 1 -- 172.21.15.102:0/155269351 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e00035b0 con 0x7f36f80ced00 2023-03-03T10:40:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f77fe700 1 -- 172.21.15.102:0/155269351 >> v2:172.21.15.102:3300/0 conn(0x7f36f80b3970 msgr2=0x7f36f80c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f77fe700 1 --2- 172.21.15.102:0/155269351 >> v2:172.21.15.102:3300/0 conn(0x7f36f80b3970 0x7f36f80c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f77fe700 1 -- 172.21.15.102:0/155269351 >> v1:172.21.15.172:6789/0 conn(0x7f36f80ced00 legacy=0x7f36f8158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:38.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f77fe700 1 -- 172.21.15.102:0/155269351 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f815cba0 con 0x7f36f80c41b0 2023-03-03T10:40:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f77fe700 1 --2- 172.21.15.102:0/155269351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f80cf0d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009d90 tx=0x7f36f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdb56362c84d8577 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f5ffb700 1 -- 172.21.15.102:0/155269351 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f000e070 con 0x7f36f80c41b0 2023-03-03T10:40:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f5ffb700 1 -- 172.21.15.102:0/155269351 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f000bb20 con 0x7f36f80c41b0 2023-03-03T10:40:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.607+0000 7f36f5ffb700 1 -- 172.21.15.102:0/155269351 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0012430 con 0x7f36f80c41b0 2023-03-03T10:40:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 -- 172.21.15.102:0/155269351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 msgr2=0x7f36f80cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 --2- 172.21.15.102:0/155269351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f80cf0d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009d90 tx=0x7f36f000b330 comp rx=0 tx=0).stop 2023-03-03T10:40:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 -- 172.21.15.102:0/155269351 shutdown_connections 2023-03-03T10:40:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 --2- 172.21.15.102:0/155269351 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f80cf0d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 --2- 172.21.15.102:0/155269351 >> v2:172.21.15.102:3300/0 conn(0x7f36f80b3970 0x7f36f80c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 -- 172.21.15.102:0/155269351 >> 172.21.15.102:0/155269351 conn(0x7f36f808b1c0 msgr2=0x7f36f8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 -- 172.21.15.102:0/155269351 shutdown_connections 2023-03-03T10:40:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.608+0000 7f36f7fff700 1 -- 172.21.15.102:0/155269351 wait complete. 2023-03-03T10:40:38.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.610+0000 7f36f7fff700 1 Processor -- start 2023-03-03T10:40:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.610+0000 7f36f7fff700 1 -- start start 2023-03-03T10:40:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.611+0000 7f36f7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f80b3970 0x7f36f81e28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.611+0000 7f36f7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f81e2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36f80ced00 0x7f36f81f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f7fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f814c210 con 0x7f36f80b3970 2023-03-03T10:40:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81e1fa0 con 0x7f36f80ced00 2023-03-03T10:40:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81e4d80 con 0x7f36f80c41b0 2023-03-03T10:40:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f80b3970 0x7f36f81e28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f80b3970 0x7f36f81e28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60832/0 (socket says 172.21.15.102:60832) 2023-03-03T10:40:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 -- 172.21.15.102:0/1600733548 learned_addr learned my addr 172.21.15.102:0/1600733548 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:40:38.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f67fc700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f81e2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f77fe700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36f80ced00 0x7f36f81f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 -- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 msgr2=0x7f36f81e2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f81e2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 -- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36f80ced00 msgr2=0x7f36f81f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36f80ced00 0x7f36f81f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.612+0000 7f36f6ffd700 1 -- 172.21.15.102:0/1600733548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f0009a70 con 0x7f36f80b3970 2023-03-03T10:40:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.613+0000 7f36f6ffd700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f80b3970 0x7f36f81e28a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f36e800c700 tx=0x7f36e800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.613+0000 7f36df7fe700 1 -- 172.21.15.102:0/1600733548 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8005d70 con 0x7f36f80b3970 2023-03-03T10:40:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.613+0000 7f36df7fe700 1 -- 172.21.15.102:0/1600733548 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e80070d0 con 0x7f36f80b3970 2023-03-03T10:40:38.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.613+0000 7f36df7fe700 1 -- 172.21.15.102:0/1600733548 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8007460 con 0x7f36f80b3970 2023-03-03T10:40:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.613+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f806bf20 con 0x7f36f80b3970 2023-03-03T10:40:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.613+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f815ce80 con 0x7f36f80b3970 2023-03-03T10:40:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.614+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f806c200 con 0x7f36f80b3970 2023-03-03T10:40:38.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.616+0000 7f36df7fe700 1 -- 172.21.15.102:0/1600733548 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f36e8019030 con 0x7f36f80b3970 2023-03-03T10:40:38.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.618+0000 7f36df7fe700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f36e40682f0 0x7f36e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:38.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.618+0000 7f36df7fe700 1 -- 172.21.15.102:0/1600733548 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f36e8085120 con 0x7f36f80b3970 2023-03-03T10:40:38.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.618+0000 7f36f67fc700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f36e40682f0 0x7f36e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:38.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.619+0000 7f36f67fc700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f36e40682f0 0x7f36e406a7a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f36f80b18a0 tx=0x7f36ec009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:38.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.622+0000 7f36df7fe700 1 -- 172.21.15.102:0/1600733548 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f36e80517a0 con 0x7f36f80b3970 2023-03-03T10:40:38.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.855+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36f804e6e0 con 0x7f36f80b3970 2023-03-03T10:40:38.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.856+0000 7f36df7fe700 1 -- 172.21.15.102:0/1600733548 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f36e8017070 con 0x7f36f80b3970 2023-03-03T10:40:38.858 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:40:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:40:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f36e40682f0 msgr2=0x7f36e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f36e40682f0 0x7f36e406a7a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f36f80b18a0 tx=0x7f36ec009210 comp rx=0 tx=0).stop 2023-03-03T10:40:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f80b3970 msgr2=0x7f36f81e28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f80b3970 0x7f36f81e28a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f36e800c700 tx=0x7f36e800ca10 comp rx=0 tx=0).stop 2023-03-03T10:40:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 shutdown_connections 2023-03-03T10:40:38.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36f80ced00 0x7f36f81f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f80b3970 0x7f36f81e28a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f36e40682f0 0x7f36e406a7a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 --2- 172.21.15.102:0/1600733548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36f80c41b0 0x7f36f81e2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 >> 172.21.15.102:0/1600733548 conn(0x7f36f808b1c0 msgr2=0x7f36f80ca960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 shutdown_connections 2023-03-03T10:40:38.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:40:38.859+0000 7f36f7fff700 1 -- 172.21.15.102:0/1600733548 wait complete. 2023-03-03T10:40:38.863 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:40:38.888 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T10:40:48.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:48.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:18.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:18.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:38.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:41:39.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.134+0000 7f52591ff700 1 Processor -- start 2023-03-03T10:41:39.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.135+0000 7f52591ff700 1 -- start start 2023-03-03T10:41:39.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f52591ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52540cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:39.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f52591ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f52540cd390 0x7f52540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f52591ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5254153660 con 0x7f5254153290 2023-03-03T10:41:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f52591ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525406c360 con 0x7f52540cd390 2023-03-03T10:41:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f52591ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525415cea0 con 0x7f52540cd8e0 2023-03-03T10:41:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f525259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52540cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f5252d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5254153290 0x7f5254158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45476/0 (socket says 172.21.15.102:45476) 2023-03-03T10:41:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.136+0000 7f5252d9d700 1 -- 172.21.15.102:0/1228885289 learned_addr learned my addr 172.21.15.102:0/1228885289 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:41:39.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.137+0000 7f525359e700 1 --2- 172.21.15.102:0/1228885289 >> v2:172.21.15.172:3300/0 conn(0x7f52540cd390 0x7f52540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.137+0000 7f525259c700 1 -- 172.21.15.102:0/1228885289 >> v2:172.21.15.172:3300/0 conn(0x7f52540cd390 msgr2=0x7f52540ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:41:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.137+0000 7f525259c700 1 --2- 172.21.15.102:0/1228885289 >> v2:172.21.15.172:3300/0 conn(0x7f52540cd390 0x7f52540ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.137+0000 7f525259c700 1 -- 172.21.15.102:0/1228885289 >> v1:172.21.15.172:6789/0 conn(0x7f5254153290 legacy=0x7f5254158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.137+0000 7f525259c700 1 -- 172.21.15.102:0/1228885289 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525406c7f0 con 0x7f52540cd8e0 2023-03-03T10:41:39.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.137+0000 7f525259c700 1 --2- 172.21.15.102:0/1228885289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52540cde20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5248009840 tx=0x7f5248009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46c6b9492643394c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:41:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.137+0000 7f5251d9b700 1 -- 172.21.15.102:0/1228885289 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5248011830 con 0x7f52540cd8e0 2023-03-03T10:41:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f5251d9b700 1 -- 172.21.15.102:0/1228885289 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5248011990 con 0x7f52540cd8e0 2023-03-03T10:41:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f5251d9b700 1 -- 172.21.15.102:0/1228885289 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5248011d20 con 0x7f52540cd8e0 2023-03-03T10:41:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 -- 172.21.15.102:0/1228885289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 msgr2=0x7f52540cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 --2- 172.21.15.102:0/1228885289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52540cde20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5248009840 tx=0x7f5248009b50 comp rx=0 tx=0).stop 2023-03-03T10:41:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 -- 172.21.15.102:0/1228885289 shutdown_connections 2023-03-03T10:41:39.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 --2- 172.21.15.102:0/1228885289 >> v2:172.21.15.172:3300/0 conn(0x7f52540cd390 0x7f52540ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 --2- 172.21.15.102:0/1228885289 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52540cde20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 -- 172.21.15.102:0/1228885289 >> 172.21.15.102:0/1228885289 conn(0x7f5254065fb0 msgr2=0x7f52540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:41:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 -- 172.21.15.102:0/1228885289 shutdown_connections 2023-03-03T10:41:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.138+0000 7f52591ff700 1 -- 172.21.15.102:0/1228885289 wait complete. 2023-03-03T10:41:39.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.141+0000 7f52591ff700 1 Processor -- start 2023-03-03T10:41:39.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.141+0000 7f52591ff700 1 -- start start 2023-03-03T10:41:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f52591ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52540cd390 0x7f52540b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f52591ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52541e58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f52591ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5254153290 0x7f52541eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f52591ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52540c8a70 con 0x7f5254153290 2023-03-03T10:41:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f52591ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52540c80c0 con 0x7f52540cd390 2023-03-03T10:41:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f52591ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52541f2c60 con 0x7f52540cd8e0 2023-03-03T10:41:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f5252d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52540cd390 0x7f52540b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f5252d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52540cd390 0x7f52540b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38600/0 (socket says 172.21.15.102:38600) 2023-03-03T10:41:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f525259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52541e58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f5252d9d700 1 -- 172.21.15.102:0/2581167096 learned_addr learned my addr 172.21.15.102:0/2581167096 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:41:39.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.143+0000 7f525359e700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5254153290 0x7f52541eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:39.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f5252d9d700 1 -- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 msgr2=0x7f52541e58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:39.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f5252d9d700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52541e58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f5252d9d700 1 -- 172.21.15.102:0/2581167096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5254153290 msgr2=0x7f52541eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:39.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f5252d9d700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5254153290 0x7f52541eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f5252d9d700 1 -- 172.21.15.102:0/2581167096 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52480094a0 con 0x7f52540cd390 2023-03-03T10:41:39.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f5252d9d700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52540cd390 0x7f52540b4710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f524400c520 tx=0x7f524400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:41:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f523b7fe700 1 -- 172.21.15.102:0/2581167096 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244005d70 con 0x7f52540cd390 2023-03-03T10:41:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f523b7fe700 1 -- 172.21.15.102:0/2581167096 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52440070d0 con 0x7f52540cd390 2023-03-03T10:41:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f523b7fe700 1 -- 172.21.15.102:0/2581167096 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244007440 con 0x7f52540cd390 2023-03-03T10:41:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.144+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52540ccde0 con 0x7f52540cd390 2023-03-03T10:41:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.145+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525415d160 con 0x7f52540cd390 2023-03-03T10:41:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.146+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52540cd0c0 con 0x7f52540cd390 2023-03-03T10:41:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.147+0000 7f523b7fe700 1 -- 172.21.15.102:0/2581167096 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5244019030 con 0x7f52540cd390 2023-03-03T10:41:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.148+0000 7f523b7fe700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5240068320 0x7f524006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:39.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.149+0000 7f523b7fe700 1 -- 172.21.15.102:0/2581167096 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5244084f50 con 0x7f52540cd390 2023-03-03T10:41:39.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.149+0000 7f525259c700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5240068320 0x7f524006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:39.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.149+0000 7f525259c700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5240068320 0x7f524006a7d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5248000c00 tx=0x7f5248008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:41:39.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.153+0000 7f523b7fe700 1 -- 172.21.15.102:0/2581167096 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f524404b5e0 con 0x7f52540cd390 2023-03-03T10:41:39.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.388+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5254073e40 con 0x7f52540cd390 2023-03-03T10:41:39.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.389+0000 7f523b7fe700 1 -- 172.21.15.102:0/2581167096 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5244017090 con 0x7f52540cd390 2023-03-03T10:41:39.392 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:41:39.392 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:41:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.391+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5240068320 msgr2=0x7f524006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.391+0000 7f52591ff700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5240068320 0x7f524006a7d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5248000c00 tx=0x7f5248008030 comp rx=0 tx=0).stop 2023-03-03T10:41:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.391+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52540cd390 msgr2=0x7f52540b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.391+0000 7f52591ff700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52540cd390 0x7f52540b4710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f524400c520 tx=0x7f524400c830 comp rx=0 tx=0).stop 2023-03-03T10:41:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 shutdown_connections 2023-03-03T10:41:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5254153290 0x7f52541eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52540cd390 0x7f52540b4710 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5240068320 0x7f524006a7d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 --2- 172.21.15.102:0/2581167096 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52540cd8e0 0x7f52541e58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:39.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 >> 172.21.15.102:0/2581167096 conn(0x7f5254065fb0 msgr2=0x7f5254157800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:41:39.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 shutdown_connections 2023-03-03T10:41:39.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:41:39.392+0000 7f52591ff700 1 -- 172.21.15.102:0/2581167096 wait complete. 2023-03-03T10:41:39.397 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:41:39.421 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 68 secs before thrashing 2023-03-03T10:41:48.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:48.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:19.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:19.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:47.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:42:47.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.667+0000 7f16191ff700 1 Processor -- start 2023-03-03T10:42:47.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.667+0000 7f16191ff700 1 -- start start 2023-03-03T10:42:47.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.668+0000 7f16191ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1614153290 0x7f1614158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:47.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.668+0000 7f16191ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1614153660 con 0x7f16141593c0 2023-03-03T10:42:47.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f16191ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161406c360 con 0x7f16140cd8e0 2023-03-03T10:42:47.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f16191ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16140cec40 con 0x7f1614153290 2023-03-03T10:42:47.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f160bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f16141593c0 0x7f16140cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40842/0 (socket says 172.21.15.102:40842) 2023-03-03T10:42:47.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f160bfff700 1 -- 172.21.15.102:0/3426239440 learned_addr learned my addr 172.21.15.102:0/3426239440 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:42:47.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f1612d9d700 1 --2- 172.21.15.102:0/3426239440 >> v2:172.21.15.172:3300/0 conn(0x7f1614153290 0x7f1614158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:47.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f161259c700 1 -- 172.21.15.102:0/3426239440 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075013453 0 0) 0x7f161406c360 con 0x7f16140cd8e0 2023-03-03T10:42:47.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f161259c700 1 -- 172.21.15.102:0/3426239440 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16000035b0 con 0x7f16140cd8e0 2023-03-03T10:42:47.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.669+0000 7f161259c700 1 -- 172.21.15.102:0/3426239440 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473795718 0 0) 0x7f1614153660 con 0x7f16141593c0 2023-03-03T10:42:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f161259c700 1 -- 172.21.15.102:0/3426239440 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f161406c360 con 0x7f16141593c0 2023-03-03T10:42:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f1612d9d700 1 -- 172.21.15.102:0/3426239440 >> v1:172.21.15.172:6789/0 conn(0x7f16140cd8e0 legacy=0x7f16140ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f1612d9d700 1 -- 172.21.15.102:0/3426239440 >> v1:172.21.15.102:6789/0 conn(0x7f16141593c0 legacy=0x7f16140cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f1612d9d700 1 -- 172.21.15.102:0/3426239440 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161415cea0 con 0x7f1614153290 2023-03-03T10:42:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f1612d9d700 1 --2- 172.21.15.102:0/3426239440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1614153290 0x7f1614158e80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1604009840 tx=0x7f1604009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16cc4861201a3d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:42:47.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f161259c700 1 -- 172.21.15.102:0/3426239440 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604011830 con 0x7f1614153290 2023-03-03T10:42:47.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f161259c700 1 -- 172.21.15.102:0/3426239440 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1604011990 con 0x7f1614153290 2023-03-03T10:42:47.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.670+0000 7f161259c700 1 -- 172.21.15.102:0/3426239440 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604011d20 con 0x7f1614153290 2023-03-03T10:42:47.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.671+0000 7f16191ff700 1 -- 172.21.15.102:0/3426239440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1614153290 msgr2=0x7f1614158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:47.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.671+0000 7f16191ff700 1 --2- 172.21.15.102:0/3426239440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1614153290 0x7f1614158e80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1604009840 tx=0x7f1604009b50 comp rx=0 tx=0).stop 2023-03-03T10:42:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.671+0000 7f16191ff700 1 -- 172.21.15.102:0/3426239440 shutdown_connections 2023-03-03T10:42:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.671+0000 7f16191ff700 1 --2- 172.21.15.102:0/3426239440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1614153290 0x7f1614158e80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.671+0000 7f16191ff700 1 -- 172.21.15.102:0/3426239440 >> 172.21.15.102:0/3426239440 conn(0x7f1614065fb0 msgr2=0x7f16140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:42:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.671+0000 7f16191ff700 1 -- 172.21.15.102:0/3426239440 shutdown_connections 2023-03-03T10:42:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.671+0000 7f16191ff700 1 -- 172.21.15.102:0/3426239440 wait complete. 2023-03-03T10:42:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.673+0000 7f16191ff700 1 Processor -- start 2023-03-03T10:42:47.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.673+0000 7f16191ff700 1 -- start start 2023-03-03T10:42:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f16191ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16140cd8e0 0x7f161415c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f16191ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1614153290 0x7f16141549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f16191ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16141593c0 0x7f1614152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f16191ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16141f2170 con 0x7f16141593c0 2023-03-03T10:42:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f16191ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16141f1880 con 0x7f16140cd8e0 2023-03-03T10:42:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f16191ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16141eee50 con 0x7f1614153290 2023-03-03T10:42:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f1612d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16140cd8e0 0x7f161415c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f161359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16141593c0 0x7f1614152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f1612d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16140cd8e0 0x7f161415c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46930/0 (socket says 172.21.15.102:46930) 2023-03-03T10:42:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.674+0000 7f1612d9d700 1 -- 172.21.15.102:0/2212657141 learned_addr learned my addr 172.21.15.102:0/2212657141 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:42:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.675+0000 7f160bfff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1614153290 0x7f16141549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.675+0000 7f1612d9d700 1 -- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1614153290 msgr2=0x7f16141549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.675+0000 7f1612d9d700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1614153290 0x7f16141549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.675+0000 7f1612d9d700 1 -- 172.21.15.102:0/2212657141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16141593c0 msgr2=0x7f1614152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.675+0000 7f1612d9d700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16141593c0 0x7f1614152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.675+0000 7f1612d9d700 1 -- 172.21.15.102:0/2212657141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16040094a0 con 0x7f16140cd8e0 2023-03-03T10:42:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.675+0000 7f1612d9d700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16140cd8e0 0x7f161415c1e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1604014040 tx=0x7f1604004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:42:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.676+0000 7f160b7fe700 1 -- 172.21.15.102:0/2212657141 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160401a6e0 con 0x7f16140cd8e0 2023-03-03T10:42:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.676+0000 7f160b7fe700 1 -- 172.21.15.102:0/2212657141 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160401a840 con 0x7f16140cd8e0 2023-03-03T10:42:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.676+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161415d160 con 0x7f16140cd8e0 2023-03-03T10:42:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.676+0000 7f160b7fe700 1 -- 172.21.15.102:0/2212657141 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160401abb0 con 0x7f16140cd8e0 2023-03-03T10:42:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.676+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161406c7f0 con 0x7f16140cd8e0 2023-03-03T10:42:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.677+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16140ccde0 con 0x7f16140cd8e0 2023-03-03T10:42:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.678+0000 7f160b7fe700 1 -- 172.21.15.102:0/2212657141 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f160400f050 con 0x7f16140cd8e0 2023-03-03T10:42:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.679+0000 7f160b7fe700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f15f4068320 0x7f15f406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.679+0000 7f160b7fe700 1 -- 172.21.15.102:0/2212657141 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1604086a60 con 0x7f16140cd8e0 2023-03-03T10:42:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.679+0000 7f160bfff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f15f4068320 0x7f15f406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:47.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.680+0000 7f160bfff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f15f4068320 0x7f15f406a7d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1614158050 tx=0x7f15fc007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:42:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.684+0000 7f160b7fe700 1 -- 172.21.15.102:0/2212657141 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1604053220 con 0x7f16140cd8e0 2023-03-03T10:42:47.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.918+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16140c9490 con 0x7f16140cd8e0 2023-03-03T10:42:47.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.919+0000 7f160b7fe700 1 -- 172.21.15.102:0/2212657141 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1604029090 con 0x7f16140cd8e0 2023-03-03T10:42:47.921 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:42:47.921 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:42:47.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f15f4068320 msgr2=0x7f15f406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:47.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f15f4068320 0x7f15f406a7d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1614158050 tx=0x7f15fc007dc0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16140cd8e0 msgr2=0x7f161415c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:47.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16140cd8e0 0x7f161415c1e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1604014040 tx=0x7f1604004750 comp rx=0 tx=0).stop 2023-03-03T10:42:47.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 shutdown_connections 2023-03-03T10:42:47.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16141593c0 0x7f1614152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16140cd8e0 0x7f161415c1e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f15f4068320 0x7f15f406a7d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 --2- 172.21.15.102:0/2212657141 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1614153290 0x7f16141549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:47.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 >> 172.21.15.102:0/2212657141 conn(0x7f1614065fb0 msgr2=0x7f16140cb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:42:47.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 shutdown_connections 2023-03-03T10:42:47.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:42:47.922+0000 7f16191ff700 1 -- 172.21.15.102:0/2212657141 wait complete. 2023-03-03T10:42:47.926 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:42:47.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T10:42:49.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:49.602 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:19.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:19.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:19.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:43:20.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.197+0000 7faaf2d9d700 1 Processor -- start 2023-03-03T10:43:20.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.197+0000 7faaf2d9d700 1 -- start start 2023-03-03T10:43:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf2d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faaec0ced00 0x7faaec0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf2d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faaec0b3970 0x7faaec0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf2d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec0c3d20 con 0x7faaec0cf0d0 2023-03-03T10:43:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf2d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec0c4930 con 0x7faaec0b3970 2023-03-03T10:43:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf2d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec0cf570 con 0x7faaec0ced00 2023-03-03T10:43:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf159a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faaec0b3970 0x7faaec0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7faaec0cf0d0 0x7faaec158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56950/0 (socket says 172.21.15.102:56950) 2023-03-03T10:43:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf259c700 1 -- 172.21.15.102:0/3263430261 learned_addr learned my addr 172.21.15.102:0/3263430261 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.199+0000 7faaf1d9b700 1 --2- 172.21.15.102:0/3263430261 >> v2:172.21.15.172:3300/0 conn(0x7faaec0ced00 0x7faaec0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf159a700 1 -- 172.21.15.102:0/3263430261 >> v2:172.21.15.172:3300/0 conn(0x7faaec0ced00 msgr2=0x7faaec0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf159a700 1 --2- 172.21.15.102:0/3263430261 >> v2:172.21.15.172:3300/0 conn(0x7faaec0ced00 0x7faaec0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf159a700 1 -- 172.21.15.102:0/3263430261 >> v1:172.21.15.102:6789/0 conn(0x7faaec0cf0d0 legacy=0x7faaec158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf159a700 1 -- 172.21.15.102:0/3263430261 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec15cba0 con 0x7faaec0b3970 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf159a700 1 --2- 172.21.15.102:0/3263430261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0b3970 0x7faaec0c41b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faaec0b1b40 tx=0x7faadc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5033d501b4fefeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf0d99700 1 -- 172.21.15.102:0/3263430261 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc011830 con 0x7faaec0b3970 2023-03-03T10:43:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf0d99700 1 -- 172.21.15.102:0/3263430261 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc011990 con 0x7faaec0b3970 2023-03-03T10:43:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.200+0000 7faaf0d99700 1 -- 172.21.15.102:0/3263430261 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc011d20 con 0x7faaec0b3970 2023-03-03T10:43:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 -- 172.21.15.102:0/3263430261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0b3970 msgr2=0x7faaec0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/3263430261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0b3970 0x7faaec0c41b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faaec0b1b40 tx=0x7faadc009b20 comp rx=0 tx=0).stop 2023-03-03T10:43:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 -- 172.21.15.102:0/3263430261 shutdown_connections 2023-03-03T10:43:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/3263430261 >> v2:172.21.15.172:3300/0 conn(0x7faaec0ced00 0x7faaec0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/3263430261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0b3970 0x7faaec0c41b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 -- 172.21.15.102:0/3263430261 >> 172.21.15.102:0/3263430261 conn(0x7faaec08b1c0 msgr2=0x7faaec071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 -- 172.21.15.102:0/3263430261 shutdown_connections 2023-03-03T10:43:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.201+0000 7faaf2d9d700 1 -- 172.21.15.102:0/3263430261 wait complete. 2023-03-03T10:43:20.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.203+0000 7faaf2d9d700 1 Processor -- start 2023-03-03T10:43:20.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.203+0000 7faaf2d9d700 1 -- start start 2023-03-03T10:43:20.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.204+0000 7faaf2d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaec0b3970 0x7faaec1e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:20.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.204+0000 7faaf2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0ced00 0x7faaec0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.204+0000 7faaf2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faaec0cf0d0 0x7faaec0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.204+0000 7faaf2d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec1f2cd0 con 0x7faaec0ced00 2023-03-03T10:43:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.204+0000 7faaf2d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0c82b0 con 0x7faaec0b3970 2023-03-03T10:43:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.204+0000 7faaf2d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec1f20e0 con 0x7faaec0cf0d0 2023-03-03T10:43:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaec0b3970 0x7faaec1e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf159a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0ced00 0x7faaec0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaec0b3970 0x7faaec1e5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57300/0 (socket says 172.21.15.102:57300) 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 -- 172.21.15.102:0/1196385503 learned_addr learned my addr 172.21.15.102:0/1196385503 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf259c700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faaec0cf0d0 0x7faaec0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 -- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faaec0cf0d0 msgr2=0x7faaec0c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faaec0cf0d0 0x7faaec0c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 -- 172.21.15.102:0/1196385503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0ced00 msgr2=0x7faaec0b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0ced00 0x7faaec0b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf1d9b700 1 -- 172.21.15.102:0/1196385503 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc0094a0 con 0x7faaec0b3970 2023-03-03T10:43:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.205+0000 7faaf159a700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0ced00 0x7faaec0b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:43:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.206+0000 7faaf1d9b700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaec0b3970 0x7faaec1e5c70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faae800e870 tx=0x7faae800eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.206+0000 7faae27fc700 1 -- 172.21.15.102:0/1196385503 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8009590 con 0x7faaec0b3970 2023-03-03T10:43:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.206+0000 7faae27fc700 1 -- 172.21.15.102:0/1196385503 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae8013680 con 0x7faaec0b3970 2023-03-03T10:43:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.206+0000 7faae27fc700 1 -- 172.21.15.102:0/1196385503 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae80139f0 con 0x7faaec0b3970 2023-03-03T10:43:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.206+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec06bf20 con 0x7faaec0b3970 2023-03-03T10:43:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.206+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec15ce80 con 0x7faaec0b3970 2023-03-03T10:43:20.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.207+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec06c200 con 0x7faaec0b3970 2023-03-03T10:43:20.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.209+0000 7faae27fc700 1 -- 172.21.15.102:0/1196385503 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faae8023070 con 0x7faaec0b3970 2023-03-03T10:43:20.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.210+0000 7faae27fc700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faad00682f0 0x7faad006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:20.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.210+0000 7faae27fc700 1 -- 172.21.15.102:0/1196385503 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7faae8085e80 con 0x7faaec0b3970 2023-03-03T10:43:20.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.210+0000 7faaf159a700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faad00682f0 0x7faad006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:20.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.211+0000 7faaf159a700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faad00682f0 0x7faad006a7a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7faaec0b1b40 tx=0x7faadc01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.214+0000 7faae27fc700 1 -- 172.21.15.102:0/1196385503 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faae8052530 con 0x7faaec0b3970 2023-03-03T10:43:20.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.449+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaec1f2f40 con 0x7faaec0b3970 2023-03-03T10:43:20.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.450+0000 7faae27fc700 1 -- 172.21.15.102:0/1196385503 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7faae8055b50 con 0x7faaec0b3970 2023-03-03T10:43:20.453 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:43:20.453 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:43:20.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faad00682f0 msgr2=0x7faad006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:20.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faad00682f0 0x7faad006a7a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7faaec0b1b40 tx=0x7faadc01c040 comp rx=0 tx=0).stop 2023-03-03T10:43:20.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaec0b3970 msgr2=0x7faaec1e5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:20.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaec0b3970 0x7faaec1e5c70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faae800e870 tx=0x7faae800eb80 comp rx=0 tx=0).stop 2023-03-03T10:43:20.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 shutdown_connections 2023-03-03T10:43:20.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faad00682f0 0x7faad006a7a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faaec0cf0d0 0x7faaec0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaec0b3970 0x7faaec1e5c70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 --2- 172.21.15.102:0/1196385503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaec0ced00 0x7faaec0b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:20.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 >> 172.21.15.102:0/1196385503 conn(0x7faaec08b1c0 msgr2=0x7faaec0c4100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:20.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 shutdown_connections 2023-03-03T10:43:20.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:20.453+0000 7faaf2d9d700 1 -- 172.21.15.102:0/1196385503 wait complete. 2023-03-03T10:43:20.458 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:43:20.483 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-03-03T10:43:30.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:43:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.732+0000 7f1b7559d700 1 Processor -- start 2023-03-03T10:43:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.732+0000 7f1b7559d700 1 -- start start 2023-03-03T10:43:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b7559d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701564b0 0x7f1b700cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:30.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b7559d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b700c9610 0x7f1b700cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:30.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b7559d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b700c9ca0 con 0x7f1b700ca440 2023-03-03T10:43:30.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b7559d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b700b4920 con 0x7f1b700c9610 2023-03-03T10:43:30.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b7559d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700b4c00 con 0x7f1b701564b0 2023-03-03T10:43:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b6f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b700c9610 0x7f1b700cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b74d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1b700ca440 0x7f1b700ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37172/0 (socket says 172.21.15.102:37172) 2023-03-03T10:43:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b74d9c700 1 -- 172.21.15.102:0/2295066840 learned_addr learned my addr 172.21.15.102:0/2295066840 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:43:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.734+0000 7f1b6ffff700 1 --2- 172.21.15.102:0/2295066840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701564b0 0x7f1b700cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6f7fe700 1 -- 172.21.15.102:0/2295066840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701564b0 msgr2=0x7f1b700cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:43:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6f7fe700 1 --2- 172.21.15.102:0/2295066840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701564b0 0x7f1b700cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6f7fe700 1 -- 172.21.15.102:0/2295066840 >> v1:172.21.15.102:6789/0 conn(0x7f1b700ca440 legacy=0x7f1b700ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6f7fe700 1 -- 172.21.15.102:0/2295066840 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700b46f0 con 0x7f1b700c9610 2023-03-03T10:43:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6effd700 1 -- 172.21.15.102:0/2295066840 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422275506 0 0) 0x7f1b700c9ca0 con 0x7f1b700ca440 2023-03-03T10:43:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6f7fe700 1 --2- 172.21.15.102:0/2295066840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b700cbba0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009840 tx=0x7f1b64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b1beb7a3f9b288 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6effd700 1 -- 172.21.15.102:0/2295066840 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64011830 con 0x7f1b700c9610 2023-03-03T10:43:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6effd700 1 -- 172.21.15.102:0/2295066840 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64011990 con 0x7f1b700c9610 2023-03-03T10:43:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.735+0000 7f1b6effd700 1 -- 172.21.15.102:0/2295066840 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64011d20 con 0x7f1b700c9610 2023-03-03T10:43:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 -- 172.21.15.102:0/2295066840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 msgr2=0x7f1b700cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2295066840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b700cbba0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009840 tx=0x7f1b64009b50 comp rx=0 tx=0).stop 2023-03-03T10:43:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 -- 172.21.15.102:0/2295066840 shutdown_connections 2023-03-03T10:43:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2295066840 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b701564b0 0x7f1b700cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2295066840 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b700cbba0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 -- 172.21.15.102:0/2295066840 >> 172.21.15.102:0/2295066840 conn(0x7f1b70065fb0 msgr2=0x7f1b700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 -- 172.21.15.102:0/2295066840 shutdown_connections 2023-03-03T10:43:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.736+0000 7f1b7559d700 1 -- 172.21.15.102:0/2295066840 wait complete. 2023-03-03T10:43:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.738+0000 7f1b7559d700 1 Processor -- start 2023-03-03T10:43:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.738+0000 7f1b7559d700 1 -- start start 2023-03-03T10:43:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.739+0000 7f1b7559d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b70157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.739+0000 7f1b7559d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b700ca440 0x7f1b701f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.739+0000 7f1b7559d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b701564b0 0x7f1b7014df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.739+0000 7f1b7559d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701596f0 con 0x7f1b700c9610 2023-03-03T10:43:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.739+0000 7f1b7559d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c5b90 con 0x7f1b701564b0 2023-03-03T10:43:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b7559d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701f2850 con 0x7f1b700ca440 2023-03-03T10:43:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b70157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b70157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56964/0 (socket says 172.21.15.102:56964) 2023-03-03T10:43:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 -- 172.21.15.102:0/2496744761 learned_addr learned my addr 172.21.15.102:0/2496744761 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:43:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6f7fe700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b700ca440 0x7f1b701f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b74d9c700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b701564b0 0x7f1b7014df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 -- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b700ca440 msgr2=0x7f1b701f1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b700ca440 0x7f1b701f1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 -- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b701564b0 msgr2=0x7f1b7014df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b701564b0 0x7f1b7014df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.740+0000 7f1b6ffff700 1 -- 172.21.15.102:0/2496744761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640094a0 con 0x7f1b700c9610 2023-03-03T10:43:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.741+0000 7f1b6ffff700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b70157270 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000e870 tx=0x7f1b6000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.741+0000 7f1b6cff9700 1 -- 172.21.15.102:0/2496744761 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60009590 con 0x7f1b700c9610 2023-03-03T10:43:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.741+0000 7f1b6cff9700 1 -- 172.21.15.102:0/2496744761 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b60013680 con 0x7f1b700c9610 2023-03-03T10:43:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.741+0000 7f1b6cff9700 1 -- 172.21.15.102:0/2496744761 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b600139f0 con 0x7f1b700c9610 2023-03-03T10:43:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.741+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b700b2210 con 0x7f1b700c9610 2023-03-03T10:43:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.741+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b700b1f30 con 0x7f1b700c9610 2023-03-03T10:43:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.742+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b70156880 con 0x7f1b700c9610 2023-03-03T10:43:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.743+0000 7f1b6cff9700 1 -- 172.21.15.102:0/2496744761 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b60019030 con 0x7f1b700c9610 2023-03-03T10:43:30.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.745+0000 7f1b6cff9700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b5c068320 0x7f1b5c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:30.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.745+0000 7f1b6cff9700 1 -- 172.21.15.102:0/2496744761 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1b60085a70 con 0x7f1b700c9610 2023-03-03T10:43:30.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.746+0000 7f1b6f7fe700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b5c068320 0x7f1b5c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:30.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.746+0000 7f1b6f7fe700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b5c068320 0x7f1b5c06a7d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1b640090b0 tx=0x7f1b64008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:30.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.750+0000 7f1b6cff9700 1 -- 172.21.15.102:0/2496744761 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b60052180 con 0x7f1b700c9610 2023-03-03T10:43:30.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.985+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b700677f0 con 0x7f1b700c9610 2023-03-03T10:43:30.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.985+0000 7f1b6cff9700 1 -- 172.21.15.102:0/2496744761 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1b600557a0 con 0x7f1b700c9610 2023-03-03T10:43:30.987 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:43:30.988 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:43:30.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b5c068320 msgr2=0x7f1b5c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:30.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b5c068320 0x7f1b5c06a7d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1b640090b0 tx=0x7f1b64008030 comp rx=0 tx=0).stop 2023-03-03T10:43:30.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 msgr2=0x7f1b70157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:30.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b70157270 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000e870 tx=0x7f1b6000eb80 comp rx=0 tx=0).stop 2023-03-03T10:43:30.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 shutdown_connections 2023-03-03T10:43:30.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b701564b0 0x7f1b7014df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b700c9610 0x7f1b70157270 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b5c068320 0x7f1b5c06a7d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 --2- 172.21.15.102:0/2496744761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b700ca440 0x7f1b701f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:30.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 >> 172.21.15.102:0/2496744761 conn(0x7f1b70065fb0 msgr2=0x7f1b700ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:30.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 shutdown_connections 2023-03-03T10:43:30.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:43:30.988+0000 7f1b7559d700 1 -- 172.21.15.102:0/2496744761 wait complete. 2023-03-03T10:43:30.992 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:43:31.017 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-03-03T10:43:49.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:49.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:20.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:20.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:50.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:50.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:20.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:20.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:24.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:45:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.266+0000 7f0b6e5ff700 1 Processor -- start 2023-03-03T10:45:24.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.266+0000 7f0b6e5ff700 1 -- start start 2023-03-03T10:45:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.267+0000 7f0b6e5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0b68153260 0x7f0b680ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.267+0000 7f0b6e5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0b680cac20 0x7f0b68163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:24.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.267+0000 7f0b6e5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b680cf940 con 0x7f0b680c9cb0 2023-03-03T10:45:24.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.267+0000 7f0b6e5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b680cfe70 con 0x7f0b68153260 2023-03-03T10:45:24.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.267+0000 7f0b6e5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b68158d70 con 0x7f0b680cac20 2023-03-03T10:45:24.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b67fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0b68153260 0x7f0b680ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b67fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0b68153260 0x7f0b680ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46212/0 (socket says 172.21.15.102:46212) 2023-03-03T10:45:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b67fff700 1 -- 172.21.15.102:0/4068170007 learned_addr learned my addr 172.21.15.102:0/4068170007 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:45:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b6cb9c700 1 --2- 172.21.15.102:0/4068170007 >> v2:172.21.15.172:3300/0 conn(0x7f0b680cac20 0x7f0b68163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b67fff700 1 -- 172.21.15.102:0/4068170007 >> v2:172.21.15.172:3300/0 conn(0x7f0b680cac20 msgr2=0x7f0b68163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:45:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b67fff700 1 --2- 172.21.15.102:0/4068170007 >> v2:172.21.15.172:3300/0 conn(0x7f0b680cac20 0x7f0b68163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b67fff700 1 -- 172.21.15.102:0/4068170007 >> v1:172.21.15.172:6789/0 conn(0x7f0b680c9cb0 legacy=0x7f0b680cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.268+0000 7f0b67fff700 1 -- 172.21.15.102:0/4068170007 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680cfb90 con 0x7f0b68153260 2023-03-03T10:45:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.269+0000 7f0b67fff700 1 --2- 172.21.15.102:0/4068170007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b68153260 0x7f0b680ca6e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0b6808cdd0 tx=0x7f0b58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bb8721c85e3e75 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.269+0000 7f0b66ffd700 1 -- 172.21.15.102:0/4068170007 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58011830 con 0x7f0b68153260 2023-03-03T10:45:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.269+0000 7f0b66ffd700 1 -- 172.21.15.102:0/4068170007 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b58011990 con 0x7f0b68153260 2023-03-03T10:45:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.269+0000 7f0b66ffd700 1 -- 172.21.15.102:0/4068170007 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58011d20 con 0x7f0b68153260 2023-03-03T10:45:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.269+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/4068170007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b68153260 msgr2=0x7f0b680ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.269+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/4068170007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b68153260 0x7f0b680ca6e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0b6808cdd0 tx=0x7f0b58009bd0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.270+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/4068170007 shutdown_connections 2023-03-03T10:45:24.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.270+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/4068170007 >> v2:172.21.15.172:3300/0 conn(0x7f0b680cac20 0x7f0b68163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.270+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/4068170007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b68153260 0x7f0b680ca6e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.270+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/4068170007 >> 172.21.15.102:0/4068170007 conn(0x7f0b68066c00 msgr2=0x7f0b680ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:24.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.270+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/4068170007 shutdown_connections 2023-03-03T10:45:24.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.270+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/4068170007 wait complete. 2023-03-03T10:45:24.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.272+0000 7f0b6e5ff700 1 Processor -- start 2023-03-03T10:45:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.272+0000 7f0b6e5ff700 1 -- start start 2023-03-03T10:45:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b6e5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b680cac20 0x7f0b681f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b6e5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b68153260 0x7f0b68151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b6e5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b681ee990 0x7f0b680cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b6e5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68150c90 con 0x7f0b680cac20 2023-03-03T10:45:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b6e5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b681f2410 con 0x7f0b681ee990 2023-03-03T10:45:24.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b6e5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b681f2de0 con 0x7f0b68153260 2023-03-03T10:45:24.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b67fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b680cac20 0x7f0b681f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:24.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b67fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b680cac20 0x7f0b681f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46218/0 (socket says 172.21.15.102:46218) 2023-03-03T10:45:24.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b67fff700 1 -- 172.21.15.102:0/1848949698 learned_addr learned my addr 172.21.15.102:0/1848949698 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:45:24.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b6cb9c700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b681ee990 0x7f0b680cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:24.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b677fe700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b68153260 0x7f0b68151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:24.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b67fff700 1 -- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b68153260 msgr2=0x7f0b68151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:24.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.273+0000 7f0b67fff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b68153260 0x7f0b68151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b67fff700 1 -- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b681ee990 msgr2=0x7f0b680cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:24.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b67fff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b681ee990 0x7f0b680cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b67fff700 1 -- 172.21.15.102:0/1848949698 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58009550 con 0x7f0b680cac20 2023-03-03T10:45:24.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b677fe700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b68153260 0x7f0b68151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:45:24.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b67fff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b680cac20 0x7f0b681f28a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0b58000c00 tx=0x7f0b58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:24.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b64ff9700 1 -- 172.21.15.102:0/1848949698 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58004480 con 0x7f0b680cac20 2023-03-03T10:45:24.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b64ff9700 1 -- 172.21.15.102:0/1848949698 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5801f070 con 0x7f0b680cac20 2023-03-03T10:45:24.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b64ff9700 1 -- 172.21.15.102:0/1848949698 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b580038c0 con 0x7f0b680cac20 2023-03-03T10:45:24.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b68159260 con 0x7f0b680cac20 2023-03-03T10:45:24.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.274+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b68158f80 con 0x7f0b680cac20 2023-03-03T10:45:24.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.276+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b68159540 con 0x7f0b680cac20 2023-03-03T10:45:24.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.277+0000 7f0b64ff9700 1 -- 172.21.15.102:0/1848949698 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b5802a020 con 0x7f0b680cac20 2023-03-03T10:45:24.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.279+0000 7f0b64ff9700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:24.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.279+0000 7f0b64ff9700 1 -- 172.21.15.102:0/1848949698 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0b580873a0 con 0x7f0b680cac20 2023-03-03T10:45:24.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.279+0000 7f0b677fe700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.280+0000 7f0b677fe700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0b680b17f0 tx=0x7f0b5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.283+0000 7f0b64ff9700 1 -- 172.21.15.102:0/1848949698 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b58053ab0 con 0x7f0b680cac20 2023-03-03T10:45:24.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.516+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b6804e6e0 con 0x7f0b680cac20 2023-03-03T10:45:24.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.517+0000 7f0b64ff9700 1 -- 172.21.15.102:0/1848949698 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0b580570d0 con 0x7f0b680cac20 2023-03-03T10:45:24.524 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:45:24.524 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:45:24.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 msgr2=0x7f0b5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0b680b17f0 tx=0x7f0b5c009040 comp rx=0 tx=0).stop 2023-03-03T10:45:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b680cac20 msgr2=0x7f0b681f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b680cac20 0x7f0b681f28a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0b58000c00 tx=0x7f0b58004750 comp rx=0 tx=0).stop 2023-03-03T10:45:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 shutdown_connections 2023-03-03T10:45:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b681ee990 0x7f0b680cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b680cac20 0x7f0b681f28a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b540682f0 0x7f0b5406a7a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 --2- 172.21.15.102:0/1848949698 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b68153260 0x7f0b68151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:24.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 >> 172.21.15.102:0/1848949698 conn(0x7f0b680c9cb0 msgr2=0x7f0b68158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:24.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 shutdown_connections 2023-03-03T10:45:24.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:24.520+0000 7f0b6e5ff700 1 -- 172.21.15.102:0/1848949698 wait complete. 2023-03-03T10:45:24.527 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:45:24.549 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-03-03T10:45:50.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:50.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:57.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:45:57.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.796+0000 7f84617ff700 1 Processor -- start 2023-03-03T10:45:57.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.796+0000 7f84617ff700 1 -- start start 2023-03-03T10:45:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.797+0000 7f84617ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c153260 0x7f845c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.797+0000 7f84617ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f845c0cac20 0x7f845c163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.797+0000 7f84617ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c0cf940 con 0x7f845c0c9cb0 2023-03-03T10:45:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.797+0000 7f84617ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c0cfe70 con 0x7f845c0cac20 2023-03-03T10:45:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.797+0000 7f84617ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c158d70 con 0x7f845c153260 2023-03-03T10:45:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c153260 0x7f845c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f845c0cac20 0x7f845c163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f8453fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f845c0c9cb0 0x7f845c0cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56654/0 (socket says 172.21.15.102:56654) 2023-03-03T10:45:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f8453fff700 1 -- 172.21.15.102:0/23096473 learned_addr learned my addr 172.21.15.102:0/23096473 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:45:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845b7fe700 1 -- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c153260 msgr2=0x7f845c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:45:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845b7fe700 1 --2- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c153260 0x7f845c0ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845b7fe700 1 -- 172.21.15.102:0/23096473 >> v1:172.21.15.172:6789/0 conn(0x7f845c0c9cb0 legacy=0x7f845c0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845b7fe700 1 -- 172.21.15.102:0/23096473 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c0cfb90 con 0x7f845c0cac20 2023-03-03T10:45:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845affd700 1 --2- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c153260 0x7f845c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:45:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.798+0000 7f845a7fc700 1 -- 172.21.15.102:0/23096473 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611607709 0 0) 0x7f845c0cf940 con 0x7f845c0c9cb0 2023-03-03T10:45:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f845b7fe700 1 --2- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c0cac20 0x7f845c163610 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8454009d90 tx=0x7f845400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fb4c6d82c02a642 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f845a7fc700 1 -- 172.21.15.102:0/23096473 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845400e070 con 0x7f845c0cac20 2023-03-03T10:45:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f845a7fc700 1 -- 172.21.15.102:0/23096473 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845400bb20 con 0x7f845c0cac20 2023-03-03T10:45:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f845a7fc700 1 -- 172.21.15.102:0/23096473 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454012430 con 0x7f845c0cac20 2023-03-03T10:45:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f84617ff700 1 -- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c0cac20 msgr2=0x7f845c163610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f84617ff700 1 --2- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c0cac20 0x7f845c163610 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8454009d90 tx=0x7f845400b330 comp rx=0 tx=0).stop 2023-03-03T10:45:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f84617ff700 1 -- 172.21.15.102:0/23096473 shutdown_connections 2023-03-03T10:45:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f84617ff700 1 --2- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c0cac20 0x7f845c163610 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f84617ff700 1 --2- 172.21.15.102:0/23096473 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c153260 0x7f845c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.799+0000 7f84617ff700 1 -- 172.21.15.102:0/23096473 >> 172.21.15.102:0/23096473 conn(0x7f845c066c00 msgr2=0x7f845c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.800+0000 7f84617ff700 1 -- 172.21.15.102:0/23096473 shutdown_connections 2023-03-03T10:45:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.800+0000 7f84617ff700 1 -- 172.21.15.102:0/23096473 wait complete. 2023-03-03T10:45:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.802+0000 7f84617ff700 1 Processor -- start 2023-03-03T10:45:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.802+0000 7f84617ff700 1 -- start start 2023-03-03T10:45:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f84617ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c0cac20 0x7f845c1f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f84617ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c153260 0x7f845c14fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f84617ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c1f1d00 0x7f845c150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f84617ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c14f350 con 0x7f845c0cac20 2023-03-03T10:45:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f84617ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1f2230 con 0x7f845c153260 2023-03-03T10:45:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f84617ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c153880 con 0x7f845c1f1d00 2023-03-03T10:45:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f845affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c0cac20 0x7f845c1f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f845affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c0cac20 0x7f845c1f17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53762/0 (socket says 172.21.15.102:53762) 2023-03-03T10:45:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.803+0000 7f845affd700 1 -- 172.21.15.102:0/883739444 learned_addr learned my addr 172.21.15.102:0/883739444 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:45:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f8453fff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c153260 0x7f845c14fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f845b7fe700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c1f1d00 0x7f845c150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f845affd700 1 -- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c1f1d00 msgr2=0x7f845c150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f845affd700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c1f1d00 0x7f845c150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f845affd700 1 -- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c153260 msgr2=0x7f845c14fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f845affd700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c153260 0x7f845c14fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f845affd700 1 -- 172.21.15.102:0/883739444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0094a0 con 0x7f845c0cac20 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.804+0000 7f845affd700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c0cac20 0x7f845c1f17c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f844c00ebe0 tx=0x7f844c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.805+0000 7f84537fe700 1 -- 172.21.15.102:0/883739444 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c00c850 con 0x7f845c0cac20 2023-03-03T10:45:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.805+0000 7f84537fe700 1 -- 172.21.15.102:0/883739444 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c00c9b0 con 0x7f845c0cac20 2023-03-03T10:45:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.805+0000 7f84537fe700 1 -- 172.21.15.102:0/883739444 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c00cd40 con 0x7f845c0cac20 2023-03-03T10:45:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.805+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8454009a70 con 0x7f845c0cac20 2023-03-03T10:45:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.805+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c159260 con 0x7f845c0cac20 2023-03-03T10:45:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.806+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c158f80 con 0x7f845c0cac20 2023-03-03T10:45:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.807+0000 7f84537fe700 1 -- 172.21.15.102:0/883739444 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f844c019030 con 0x7f845c0cac20 2023-03-03T10:45:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.808+0000 7f84537fe700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f843c068320 0x7f843c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.808+0000 7f84537fe700 1 -- 172.21.15.102:0/883739444 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f844c085da0 con 0x7f845c0cac20 2023-03-03T10:45:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.809+0000 7f8453fff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f843c068320 0x7f843c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.809+0000 7f8453fff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f843c068320 0x7f843c06a7d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f845c0c5910 tx=0x7f8444007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:57.813+0000 7f84537fe700 1 -- 172.21.15.102:0/883739444 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f844c052420 con 0x7f845c0cac20 2023-03-03T10:45:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.053+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f845c159540 con 0x7f845c0cac20 2023-03-03T10:45:58.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.054+0000 7f84537fe700 1 -- 172.21.15.102:0/883739444 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f844c017090 con 0x7f845c0cac20 2023-03-03T10:45:58.056 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:45:58.056 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:45:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f843c068320 msgr2=0x7f843c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f843c068320 0x7f843c06a7d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f845c0c5910 tx=0x7f8444007e70 comp rx=0 tx=0).stop 2023-03-03T10:45:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c0cac20 msgr2=0x7f845c1f17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c0cac20 0x7f845c1f17c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f844c00ebe0 tx=0x7f844c00eef0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 shutdown_connections 2023-03-03T10:45:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f843c068320 0x7f843c06a7d0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f845c1f1d00 0x7f845c150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c0cac20 0x7f845c1f17c0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 --2- 172.21.15.102:0/883739444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f845c153260 0x7f845c14fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 >> 172.21.15.102:0/883739444 conn(0x7f845c0c9cb0 msgr2=0x7f845c157f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.056+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 shutdown_connections 2023-03-03T10:45:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:45:58.057+0000 7f84617ff700 1 -- 172.21.15.102:0/883739444 wait complete. 2023-03-03T10:45:58.060 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:45:58.085 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 45 secs before thrashing 2023-03-03T10:46:20.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:20.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:43.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:46:43.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.351+0000 7fa9b07ff700 1 Processor -- start 2023-03-03T10:46:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.352+0000 7fa9b07ff700 1 -- start start 2023-03-03T10:46:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.353+0000 7fa9b07ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.353+0000 7fa9b07ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.353+0000 7fa9b07ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a8153660 con 0x7fa9a80cd8e0 2023-03-03T10:46:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.353+0000 7fa9b07ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a806c360 con 0x7fa9a8153290 2023-03-03T10:46:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.353+0000 7fa9b07ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a815cea0 con 0x7fa9a81593c0 2023-03-03T10:46:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9ae59b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9add9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9aed9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa9a80cd8e0 0x7fa9a80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48462/0 (socket says 172.21.15.102:48462) 2023-03-03T10:46:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9aed9c700 1 -- 172.21.15.102:0/3120587089 learned_addr learned my addr 172.21.15.102:0/3120587089 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:46:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9add9a700 1 -- 172.21.15.102:0/3120587089 >> v2:172.21.15.172:3300/0 conn(0x7fa9a8153290 msgr2=0x7fa9a8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:46:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9add9a700 1 --2- 172.21.15.102:0/3120587089 >> v2:172.21.15.172:3300/0 conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9add9a700 1 -- 172.21.15.102:0/3120587089 >> v1:172.21.15.102:6789/0 conn(0x7fa9a80cd8e0 legacy=0x7fa9a80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9ad599700 1 -- 172.21.15.102:0/3120587089 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163898196 0 0) 0x7fa9a8153660 con 0x7fa9a80cd8e0 2023-03-03T10:46:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9add9a700 1 -- 172.21.15.102:0/3120587089 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a806c7f0 con 0x7fa9a81593c0 2023-03-03T10:46:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9ae59b700 1 --2- 172.21.15.102:0/3120587089 >> v2:172.21.15.172:3300/0 conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:46:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.354+0000 7fa9add9a700 1 --2- 172.21.15.102:0/3120587089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a80cde20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa998009840 tx=0x7fa998009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aa55d4e128d670 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:46:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.355+0000 7fa9ad599700 1 -- 172.21.15.102:0/3120587089 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998011830 con 0x7fa9a81593c0 2023-03-03T10:46:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.355+0000 7fa9ad599700 1 -- 172.21.15.102:0/3120587089 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998011990 con 0x7fa9a81593c0 2023-03-03T10:46:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.355+0000 7fa9ad599700 1 -- 172.21.15.102:0/3120587089 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998011d20 con 0x7fa9a81593c0 2023-03-03T10:46:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.355+0000 7fa9b07ff700 1 -- 172.21.15.102:0/3120587089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 msgr2=0x7fa9a80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.355+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/3120587089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a80cde20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa998009840 tx=0x7fa998009b50 comp rx=0 tx=0).stop 2023-03-03T10:46:43.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.355+0000 7fa9b07ff700 1 -- 172.21.15.102:0/3120587089 shutdown_connections 2023-03-03T10:46:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.355+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/3120587089 >> v2:172.21.15.172:3300/0 conn(0x7fa9a8153290 0x7fa9a8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.356+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/3120587089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a81593c0 0x7fa9a80cde20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.356+0000 7fa9b07ff700 1 -- 172.21.15.102:0/3120587089 >> 172.21.15.102:0/3120587089 conn(0x7fa9a8065fb0 msgr2=0x7fa9a80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:46:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.356+0000 7fa9b07ff700 1 -- 172.21.15.102:0/3120587089 shutdown_connections 2023-03-03T10:46:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.356+0000 7fa9b07ff700 1 -- 172.21.15.102:0/3120587089 wait complete. 2023-03-03T10:46:43.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.358+0000 7fa9b07ff700 1 Processor -- start 2023-03-03T10:46:43.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.358+0000 7fa9b07ff700 1 -- start start 2023-03-03T10:46:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9b07ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a814d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9b07ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a81e5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9b07ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a81593c0 0x7fa9a81e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9b07ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a814f570 con 0x7fa9a81593c0 2023-03-03T10:46:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9b07ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8150610 con 0x7fa9a80cd8e0 2023-03-03T10:46:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9b07ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a814eb50 con 0x7fa9a8153290 2023-03-03T10:46:43.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9aed9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a81593c0 0x7fa9a81e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9add9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a81e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9ae59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a814d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9add9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a81e5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60942/0 (socket says 172.21.15.102:60942) 2023-03-03T10:46:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.359+0000 7fa9add9a700 1 -- 172.21.15.102:0/1046739365 learned_addr learned my addr 172.21.15.102:0/1046739365 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:46:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9add9a700 1 -- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a80cd8e0 msgr2=0x7fa9a814d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9add9a700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a814d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9add9a700 1 -- 172.21.15.102:0/1046739365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a81593c0 msgr2=0x7fa9a81e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9add9a700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a81593c0 0x7fa9a81e5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9add9a700 1 -- 172.21.15.102:0/1046739365 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a40094a0 con 0x7fa9a8153290 2023-03-03T10:46:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9ae59b700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a814d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:46:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9aed9c700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a81593c0 0x7fa9a81e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:46:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9add9a700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a81e5870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa998009b90 tx=0x7fa998004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:46:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa99effd700 1 -- 172.21.15.102:0/1046739365 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9980036a0 con 0x7fa9a8153290 2023-03-03T10:46:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9980094a0 con 0x7fa9a8153290 2023-03-03T10:46:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.360+0000 7fa99effd700 1 -- 172.21.15.102:0/1046739365 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99801f070 con 0x7fa9a8153290 2023-03-03T10:46:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.361+0000 7fa99effd700 1 -- 172.21.15.102:0/1046739365 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998003a40 con 0x7fa9a8153290 2023-03-03T10:46:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.361+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a80ccde0 con 0x7fa9a8153290 2023-03-03T10:46:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.362+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a815d160 con 0x7fa9a8153290 2023-03-03T10:46:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.363+0000 7fa99effd700 1 -- 172.21.15.102:0/1046739365 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa99800f050 con 0x7fa9a8153290 2023-03-03T10:46:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.364+0000 7fa99effd700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa98c0682f0 0x7fa98c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.364+0000 7fa99effd700 1 -- 172.21.15.102:0/1046739365 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa998021070 con 0x7fa9a8153290 2023-03-03T10:46:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.365+0000 7fa9ae59b700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa98c0682f0 0x7fa98c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.365+0000 7fa9ae59b700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa98c0682f0 0x7fa98c06a7a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4011380 tx=0x7fa9a4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:46:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.369+0000 7fa99effd700 1 -- 172.21.15.102:0/1046739365 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa998053db0 con 0x7fa9a8153290 2023-03-03T10:46:43.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.606+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9a80cd0c0 con 0x7fa9a8153290 2023-03-03T10:46:43.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.607+0000 7fa99effd700 1 -- 172.21.15.102:0/1046739365 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa9980573d0 con 0x7fa9a8153290 2023-03-03T10:46:43.609 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:46:43.610 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:46:43.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa98c0682f0 msgr2=0x7fa98c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:43.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa98c0682f0 0x7fa98c06a7a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4011380 tx=0x7fa9a4009210 comp rx=0 tx=0).stop 2023-03-03T10:46:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 msgr2=0x7fa9a81e5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a81e5870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa998009b90 tx=0x7fa998004750 comp rx=0 tx=0).stop 2023-03-03T10:46:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 shutdown_connections 2023-03-03T10:46:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9a81593c0 0x7fa9a81e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9a80cd8e0 0x7fa9a814d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa98c0682f0 0x7fa98c06a7a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 --2- 172.21.15.102:0/1046739365 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9a8153290 0x7fa9a81e5870 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:43.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 >> 172.21.15.102:0/1046739365 conn(0x7fa9a8065fb0 msgr2=0x7fa9a80cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:46:43.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 shutdown_connections 2023-03-03T10:46:43.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:46:43.610+0000 7fa9b07ff700 1 -- 172.21.15.102:0/1046739365 wait complete. 2023-03-03T10:46:43.614 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:46:43.639 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T10:46:51.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:51.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:21.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:21.209 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:51.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:51.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:20.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:48:20.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.888+0000 7f64c539d700 1 Processor -- start 2023-03-03T10:48:20.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.888+0000 7f64c539d700 1 -- start start 2023-03-03T10:48:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64c539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64c539d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64c00b3970 0x7f64c00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64c539d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00c3d20 con 0x7f64c00cf0d0 2023-03-03T10:48:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64c539d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c00c4930 con 0x7f64c00b3970 2023-03-03T10:48:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64c539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c015cba0 con 0x7f64c00ced00 2023-03-03T10:48:20.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64b7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64c00b3970 0x7f64c00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:20.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64c4b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f64c00cf0d0 0x7f64c0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49502/0 (socket says 172.21.15.102:49502) 2023-03-03T10:48:20.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64c4b9c700 1 -- 172.21.15.102:0/3242576770 learned_addr learned my addr 172.21.15.102:0/3242576770 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:48:20.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.890+0000 7f64bffff700 1 --2- 172.21.15.102:0/3242576770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:20.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64b7fff700 1 -- 172.21.15.102:0/3242576770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 msgr2=0x7f64c00c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:48:20.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64b7fff700 1 --2- 172.21.15.102:0/3242576770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c00c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:20.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64b7fff700 1 -- 172.21.15.102:0/3242576770 >> v1:172.21.15.102:6789/0 conn(0x7f64c00cf0d0 legacy=0x7f64c0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:20.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64b7fff700 1 -- 172.21.15.102:0/3242576770 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c015ce80 con 0x7f64c00b3970 2023-03-03T10:48:20.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64bf7fe700 1 -- 172.21.15.102:0/3242576770 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576508742 0 0) 0x7f64c00c3d20 con 0x7f64c00cf0d0 2023-03-03T10:48:20.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64b7fff700 1 --2- 172.21.15.102:0/3242576770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00b3970 0x7f64c00c41b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009840 tx=0x7f64a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d487fc6f06e634 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:20.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64bf7fe700 1 -- 172.21.15.102:0/3242576770 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8011830 con 0x7f64c00b3970 2023-03-03T10:48:20.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64bf7fe700 1 -- 172.21.15.102:0/3242576770 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a8011990 con 0x7f64c00b3970 2023-03-03T10:48:20.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.891+0000 7f64bf7fe700 1 -- 172.21.15.102:0/3242576770 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8011d20 con 0x7f64c00b3970 2023-03-03T10:48:20.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 -- 172.21.15.102:0/3242576770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00b3970 msgr2=0x7f64c00c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:20.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 --2- 172.21.15.102:0/3242576770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00b3970 0x7f64c00c41b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009840 tx=0x7f64a8009b50 comp rx=0 tx=0).stop 2023-03-03T10:48:20.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 -- 172.21.15.102:0/3242576770 shutdown_connections 2023-03-03T10:48:20.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 --2- 172.21.15.102:0/3242576770 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:20.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 --2- 172.21.15.102:0/3242576770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00b3970 0x7f64c00c41b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:20.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 -- 172.21.15.102:0/3242576770 >> 172.21.15.102:0/3242576770 conn(0x7f64c008b1c0 msgr2=0x7f64c0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:20.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 -- 172.21.15.102:0/3242576770 shutdown_connections 2023-03-03T10:48:20.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.892+0000 7f64c539d700 1 -- 172.21.15.102:0/3242576770 wait complete. 2023-03-03T10:48:20.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.894+0000 7f64c539d700 1 Processor -- start 2023-03-03T10:48:20.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.894+0000 7f64c539d700 1 -- start start 2023-03-03T10:48:20.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.895+0000 7f64c539d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64c00b3970 0x7f64c0148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.895+0000 7f64c539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c01ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.895+0000 7f64c539d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00cf0d0 0x7f64c01efbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.895+0000 7f64c539d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c01e4440 con 0x7f64c00cf0d0 2023-03-03T10:48:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.895+0000 7f64c539d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c01e5eb0 con 0x7f64c00b3970 2023-03-03T10:48:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.895+0000 7f64c539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c01e3240 con 0x7f64c00ced00 2023-03-03T10:48:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c01ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:20.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64bffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64c00b3970 0x7f64c0148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:20.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64c4b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00cf0d0 0x7f64c01efbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:20.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c01ef6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:43262/0 (socket says 172.21.15.102:43262) 2023-03-03T10:48:20.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64bffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64c00b3970 0x7f64c0148df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50028/0 (socket says 172.21.15.102:50028) 2023-03-03T10:48:20.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 -- 172.21.15.102:0/3278712562 learned_addr learned my addr 172.21.15.102:0/3278712562 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:48:20.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 -- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64c00b3970 msgr2=0x7f64c0148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:20.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64c00b3970 0x7f64c0148df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:20.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 -- 172.21.15.102:0/3278712562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00cf0d0 msgr2=0x7f64c01efbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:20.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00cf0d0 0x7f64c01efbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:20.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64b7fff700 1 -- 172.21.15.102:0/3278712562 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80094a0 con 0x7f64c00ced00 2023-03-03T10:48:20.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.896+0000 7f64bffff700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64c00b3970 0x7f64c0148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:48:20.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.897+0000 7f64b7fff700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c01ef6b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f64a8014040 tx=0x7f64a8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:20.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.897+0000 7f64bd7fa700 1 -- 172.21.15.102:0/3278712562 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a80036a0 con 0x7f64c00ced00 2023-03-03T10:48:20.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.897+0000 7f64bd7fa700 1 -- 172.21.15.102:0/3278712562 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a801e070 con 0x7f64c00ced00 2023-03-03T10:48:20.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.897+0000 7f64bd7fa700 1 -- 172.21.15.102:0/3278712562 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8003a40 con 0x7f64c00ced00 2023-03-03T10:48:20.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.897+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c006c200 con 0x7f64c00ced00 2023-03-03T10:48:20.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.897+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c006bf20 con 0x7f64c00ced00 2023-03-03T10:48:20.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.898+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c015c890 con 0x7f64c00ced00 2023-03-03T10:48:20.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.900+0000 7f64bd7fa700 1 -- 172.21.15.102:0/3278712562 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f64a800f050 con 0x7f64c00ced00 2023-03-03T10:48:20.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.901+0000 7f64bd7fa700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64a0068320 0x7f64a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:20.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.901+0000 7f64bd7fa700 1 -- 172.21.15.102:0/3278712562 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f64a8086c60 con 0x7f64c00ced00 2023-03-03T10:48:20.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.901+0000 7f64bffff700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64a0068320 0x7f64a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:20.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.902+0000 7f64bffff700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64a0068320 0x7f64a006a7d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009fd0 tx=0x7f64b0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:20.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:20.905+0000 7f64bd7fa700 1 -- 172.21.15.102:0/3278712562 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f64a80532f0 con 0x7f64c00ced00 2023-03-03T10:48:21.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.141+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64c004e6e0 con 0x7f64c00ced00 2023-03-03T10:48:21.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.142+0000 7f64bd7fa700 1 -- 172.21.15.102:0/3278712562 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f64a8020090 con 0x7f64c00ced00 2023-03-03T10:48:21.145 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:48:21.145 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:48:21.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64a0068320 msgr2=0x7f64a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:21.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64a0068320 0x7f64a006a7d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009fd0 tx=0x7f64b0009210 comp rx=0 tx=0).stop 2023-03-03T10:48:21.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 msgr2=0x7f64c01ef6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:21.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c01ef6b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f64a8014040 tx=0x7f64a8004750 comp rx=0 tx=0).stop 2023-03-03T10:48:21.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 shutdown_connections 2023-03-03T10:48:21.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64c00cf0d0 0x7f64c01efbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:21.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64c00b3970 0x7f64c0148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:21.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64a0068320 0x7f64a006a7d0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:21.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.145+0000 7f64c539d700 1 --2- 172.21.15.102:0/3278712562 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64c00ced00 0x7f64c01ef6b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:21.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.146+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 >> 172.21.15.102:0/3278712562 conn(0x7f64c008b1c0 msgr2=0x7f64c00ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:21.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.146+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 shutdown_connections 2023-03-03T10:48:21.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:48:21.146+0000 7f64c539d700 1 -- 172.21.15.102:0/3278712562 wait complete. 2023-03-03T10:48:21.149 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:48:21.175 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T10:48:21.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:21.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:51.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:51.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:21.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:21.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:52.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:52.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T10:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:50:16.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:50:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.432+0000 7f42c639d700 1 Processor -- start 2023-03-03T10:50:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.432+0000 7f42c639d700 1 -- start start 2023-03-03T10:50:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.434+0000 7f42c639d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42c00ced00 0x7f42c00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.434+0000 7f42c639d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c00c3d20 con 0x7f42c00b3970 2023-03-03T10:50:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.434+0000 7f42c639d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c015cba0 con 0x7f42c00c41b0 2023-03-03T10:50:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.434+0000 7f42c639d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c00c4930 con 0x7f42c00ced00 2023-03-03T10:50:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.434+0000 7f42c539b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42c00ced00 0x7f42c00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.434+0000 7f42c539b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42c00ced00 0x7f42c00c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49952/0 (socket says 172.21.15.102:49952) 2023-03-03T10:50:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.434+0000 7f42c539b700 1 -- 172.21.15.102:0/2003357649 learned_addr learned my addr 172.21.15.102:0/2003357649 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:50:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.435+0000 7f42c539b700 1 -- 172.21.15.102:0/2003357649 >> v1:172.21.15.172:6789/0 conn(0x7f42c00c41b0 legacy=0x7f42c0163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.435+0000 7f42c539b700 1 -- 172.21.15.102:0/2003357649 >> v1:172.21.15.102:6789/0 conn(0x7f42c00b3970 legacy=0x7f42c0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.435+0000 7f42c539b700 1 -- 172.21.15.102:0/2003357649 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c015ce80 con 0x7f42c00ced00 2023-03-03T10:50:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.435+0000 7f42c539b700 1 --2- 172.21.15.102:0/2003357649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00ced00 0x7f42c00c5a90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098f0 tx=0x7f42bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15c61286a6e4044 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.435+0000 7f42b7fff700 1 -- 172.21.15.102:0/2003357649 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00b8a0 con 0x7f42c00ced00 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.435+0000 7f42b7fff700 1 -- 172.21.15.102:0/2003357649 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc00ba00 con 0x7f42c00ced00 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.435+0000 7f42b7fff700 1 -- 172.21.15.102:0/2003357649 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00bd90 con 0x7f42c00ced00 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.436+0000 7f42c639d700 1 -- 172.21.15.102:0/2003357649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00ced00 msgr2=0x7f42c00c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.436+0000 7f42c639d700 1 --2- 172.21.15.102:0/2003357649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00ced00 0x7f42c00c5a90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098f0 tx=0x7f42bc009c00 comp rx=0 tx=0).stop 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.436+0000 7f42c639d700 1 -- 172.21.15.102:0/2003357649 shutdown_connections 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.436+0000 7f42c639d700 1 --2- 172.21.15.102:0/2003357649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00ced00 0x7f42c00c5a90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.436+0000 7f42c639d700 1 -- 172.21.15.102:0/2003357649 >> 172.21.15.102:0/2003357649 conn(0x7f42c008b1c0 msgr2=0x7f42c0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:50:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.436+0000 7f42c639d700 1 -- 172.21.15.102:0/2003357649 shutdown_connections 2023-03-03T10:50:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.436+0000 7f42c639d700 1 -- 172.21.15.102:0/2003357649 wait complete. 2023-03-03T10:50:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.438+0000 7f42c639d700 1 Processor -- start 2023-03-03T10:50:16.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.438+0000 7f42c639d700 1 -- start start 2023-03-03T10:50:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.439+0000 7f42c639d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c00b3970 0x7f42c01ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.439+0000 7f42c639d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00c41b0 0x7f42c0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.439+0000 7f42c639d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42c00ced00 0x7f42c015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.439+0000 7f42c639d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0152790 con 0x7f42c00c41b0 2023-03-03T10:50:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.439+0000 7f42c639d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c015a9b0 con 0x7f42c00b3970 2023-03-03T10:50:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.439+0000 7f42c639d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c01521d0 con 0x7f42c00ced00 2023-03-03T10:50:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.439+0000 7f42c4b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00c41b0 0x7f42c0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00c41b0 0x7f42c0159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49962/0 (socket says 172.21.15.102:49962) 2023-03-03T10:50:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c539b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c00b3970 0x7f42c01ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 -- 172.21.15.102:0/3339964211 learned_addr learned my addr 172.21.15.102:0/3339964211 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:50:16.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c5b9c700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42c00ced00 0x7f42c015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 -- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42c00ced00 msgr2=0x7f42c015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42c00ced00 0x7f42c015a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 -- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c00b3970 msgr2=0x7f42c01ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c00b3970 0x7f42c01ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 -- 172.21.15.102:0/3339964211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc009550 con 0x7f42c00c41b0 2023-03-03T10:50:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c5b9c700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42c00ced00 0x7f42c015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:50:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c4b9a700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00c41b0 0x7f42c0159f30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f42b000c450 tx=0x7f42b000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42b5ffb700 1 -- 172.21.15.102:0/3339964211 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b00052d0 con 0x7f42c00c41b0 2023-03-03T10:50:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42b5ffb700 1 -- 172.21.15.102:0/3339964211 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b0005430 con 0x7f42c00c41b0 2023-03-03T10:50:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42b5ffb700 1 -- 172.21.15.102:0/3339964211 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b00038c0 con 0x7f42c00c41b0 2023-03-03T10:50:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.440+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c006c200 con 0x7f42c00c41b0 2023-03-03T10:50:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.441+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c006bf20 con 0x7f42c00c41b0 2023-03-03T10:50:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.442+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c015c890 con 0x7f42c00c41b0 2023-03-03T10:50:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.443+0000 7f42b5ffb700 1 -- 172.21.15.102:0/3339964211 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f42b0019030 con 0x7f42c00c41b0 2023-03-03T10:50:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.444+0000 7f42b5ffb700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a4068320 0x7f42a406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.444+0000 7f42b5ffb700 1 -- 172.21.15.102:0/3339964211 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f42b0084fe0 con 0x7f42c00c41b0 2023-03-03T10:50:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.445+0000 7f42c539b700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a4068320 0x7f42a406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.446+0000 7f42c539b700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a4068320 0x7f42a406a7d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009c40 tx=0x7f42bc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:16.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.449+0000 7f42b5ffb700 1 -- 172.21.15.102:0/3339964211 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42b0051730 con 0x7f42c00c41b0 2023-03-03T10:50:16.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.684+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42c004e6e0 con 0x7f42c00c41b0 2023-03-03T10:50:16.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.685+0000 7f42b5ffb700 1 -- 172.21.15.102:0/3339964211 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f42b0017090 con 0x7f42c00c41b0 2023-03-03T10:50:16.688 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:50:16.688 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:50:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a4068320 msgr2=0x7f42a406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a4068320 0x7f42a406a7d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009c40 tx=0x7f42bc008030 comp rx=0 tx=0).stop 2023-03-03T10:50:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00c41b0 msgr2=0x7f42c0159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00c41b0 0x7f42c0159f30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f42b000c450 tx=0x7f42b000c760 comp rx=0 tx=0).stop 2023-03-03T10:50:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 shutdown_connections 2023-03-03T10:50:16.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42a4068320 0x7f42a406a7d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42c00ced00 0x7f42c015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c00b3970 0x7f42c01ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 --2- 172.21.15.102:0/3339964211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42c00c41b0 0x7f42c0159f30 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 >> 172.21.15.102:0/3339964211 conn(0x7f42c008b1c0 msgr2=0x7f42c0066930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:50:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 shutdown_connections 2023-03-03T10:50:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:50:16.688+0000 7f42c639d700 1 -- 172.21.15.102:0/3339964211 wait complete. 2023-03-03T10:50:16.693 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:50:16.718 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T10:50:22.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:22.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:52.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:52.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:22.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:22.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:52.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:52.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:07.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:52:07.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.967+0000 7f24b5f9d700 1 Processor -- start 2023-03-03T10:52:07.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.967+0000 7f24b5f9d700 1 -- start start 2023-03-03T10:52:07.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.969+0000 7f24b5f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f24b00ced00 0x7f24b00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:07.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.969+0000 7f24b5f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:07.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.969+0000 7f24b5f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24b00cf0d0 0x7f24b0158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:07.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.969+0000 7f24b5f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b00c4930 con 0x7f24b00cf0d0 2023-03-03T10:52:07.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.969+0000 7f24b5f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b00c3d20 con 0x7f24b00ced00 2023-03-03T10:52:07.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.969+0000 7f24b5f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b015cba0 con 0x7f24b00b3970 2023-03-03T10:52:07.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:07.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24b4f9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f24b00ced00 0x7f24b00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:07.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b00c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:33282/0 (socket says 172.21.15.102:33282) 2023-03-03T10:52:07.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 -- 172.21.15.102:0/71133571 learned_addr learned my addr 172.21.15.102:0/71133571 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:52:07.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24b579c700 1 --2- 172.21.15.102:0/71133571 >> v2:172.21.15.102:3300/0 conn(0x7f24b00cf0d0 0x7f24b0158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:07.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 -- 172.21.15.102:0/71133571 >> v2:172.21.15.172:3300/0 conn(0x7f24b00ced00 msgr2=0x7f24b00c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:52:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 --2- 172.21.15.102:0/71133571 >> v2:172.21.15.172:3300/0 conn(0x7f24b00ced00 0x7f24b00c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 -- 172.21.15.102:0/71133571 >> v2:172.21.15.102:3300/0 conn(0x7f24b00cf0d0 msgr2=0x7f24b0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:52:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 --2- 172.21.15.102:0/71133571 >> v2:172.21.15.102:3300/0 conn(0x7f24b00cf0d0 0x7f24b0158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 -- 172.21.15.102:0/71133571 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b015ce80 con 0x7f24b00b3970 2023-03-03T10:52:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24b4f9b700 1 --2- 172.21.15.102:0/71133571 >> v2:172.21.15.172:3300/0 conn(0x7f24b00ced00 0x7f24b00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:52:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24b579c700 1 --2- 172.21.15.102:0/71133571 >> v2:172.21.15.102:3300/0 conn(0x7f24b00cf0d0 0x7f24b0158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:52:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.970+0000 7f24affff700 1 --2- 172.21.15.102:0/71133571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b00c41b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f24b00b1b40 tx=0x7f249c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74315053525db211 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.972+0000 7f24af7fe700 1 -- 172.21.15.102:0/71133571 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c011830 con 0x7f24b00b3970 2023-03-03T10:52:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.972+0000 7f24af7fe700 1 -- 172.21.15.102:0/71133571 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c011990 con 0x7f24b00b3970 2023-03-03T10:52:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.972+0000 7f24af7fe700 1 -- 172.21.15.102:0/71133571 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c011d20 con 0x7f24b00b3970 2023-03-03T10:52:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 -- 172.21.15.102:0/71133571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 msgr2=0x7f24b00c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/71133571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b00c41b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f24b00b1b40 tx=0x7f249c009bd0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 -- 172.21.15.102:0/71133571 shutdown_connections 2023-03-03T10:52:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/71133571 >> v2:172.21.15.102:3300/0 conn(0x7f24b00cf0d0 0x7f24b0158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/71133571 >> v2:172.21.15.172:3300/0 conn(0x7f24b00ced00 0x7f24b00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/71133571 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b00c41b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 -- 172.21.15.102:0/71133571 >> 172.21.15.102:0/71133571 conn(0x7f24b008b1c0 msgr2=0x7f24b0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:07.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 -- 172.21.15.102:0/71133571 shutdown_connections 2023-03-03T10:52:07.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.973+0000 7f24b5f9d700 1 -- 172.21.15.102:0/71133571 wait complete. 2023-03-03T10:52:07.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.975+0000 7f24b5f9d700 1 Processor -- start 2023-03-03T10:52:07.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.975+0000 7f24b5f9d700 1 -- start start 2023-03-03T10:52:07.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.976+0000 7f24b5f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:07.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.976+0000 7f24b5f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b00ced00 0x7f24b01ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:07.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.976+0000 7f24b5f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24b00cf0d0 0x7f24b01ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:07.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b5f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b01f14c0 con 0x7f24b00ced00 2023-03-03T10:52:07.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b5f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b01f1710 con 0x7f24b00cf0d0 2023-03-03T10:52:07.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b5f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b01f19f0 con 0x7f24b00b3970 2023-03-03T10:52:07.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24affff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b00ced00 0x7f24b01ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:07.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24b00cf0d0 0x7f24b01ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:07.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b4f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:07.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24b00cf0d0 0x7f24b01ee700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40044/0 (socket says 172.21.15.102:40044) 2023-03-03T10:52:07.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 -- 172.21.15.102:0/597612596 learned_addr learned my addr 172.21.15.102:0/597612596 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:52:07.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 -- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 msgr2=0x7f24b015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:07.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 -- 172.21.15.102:0/597612596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b00ced00 msgr2=0x7f24b01ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:07.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b00ced00 0x7f24b01ee1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:07.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24b579c700 1 -- 172.21.15.102:0/597612596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a40094a0 con 0x7f24b00cf0d0 2023-03-03T10:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.977+0000 7f24affff700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b00ced00 0x7f24b01ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.978+0000 7f24b579c700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24b00cf0d0 0x7f24b01ee700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f24a800efc0 tx=0x7f24a800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.978+0000 7f24ad7fa700 1 -- 172.21.15.102:0/597612596 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a800ca90 con 0x7f24b00cf0d0 2023-03-03T10:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.978+0000 7f24ad7fa700 1 -- 172.21.15.102:0/597612596 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a800f040 con 0x7f24b00cf0d0 2023-03-03T10:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.978+0000 7f24ad7fa700 1 -- 172.21.15.102:0/597612596 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a800ce10 con 0x7f24b00cf0d0 2023-03-03T10:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.978+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a8009a70 con 0x7f24b00cf0d0 2023-03-03T10:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.978+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c009550 con 0x7f24b00cf0d0 2023-03-03T10:52:07.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.979+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b006c200 con 0x7f24b00cf0d0 2023-03-03T10:52:07.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.981+0000 7f24ad7fa700 1 -- 172.21.15.102:0/597612596 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24a8014480 con 0x7f24b00cf0d0 2023-03-03T10:52:07.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.982+0000 7f24ad7fa700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24940682f0 0x7f249406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:07.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.982+0000 7f24ad7fa700 1 -- 172.21.15.102:0/597612596 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f24a80865f0 con 0x7f24b00cf0d0 2023-03-03T10:52:07.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.982+0000 7f24b4f9b700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24940682f0 0x7f249406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:07.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.983+0000 7f24b4f9b700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24940682f0 0x7f249406a7a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009470 tx=0x7f24a4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:07.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:07.986+0000 7f24ad7fa700 1 -- 172.21.15.102:0/597612596 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24a8052ca0 con 0x7f24b00cf0d0 2023-03-03T10:52:08.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.224+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24b0159040 con 0x7f24b00cf0d0 2023-03-03T10:52:08.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.224+0000 7f24ad7fa700 1 -- 172.21.15.102:0/597612596 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f24a8009d70 con 0x7f24b00cf0d0 2023-03-03T10:52:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:52:08.228 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:52:08.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24940682f0 msgr2=0x7f249406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:08.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24940682f0 0x7f249406a7a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009470 tx=0x7f24a4009210 comp rx=0 tx=0).stop 2023-03-03T10:52:08.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24b00cf0d0 msgr2=0x7f24b01ee700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:08.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24b00cf0d0 0x7f24b01ee700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f24a800efc0 tx=0x7f24a800c3a0 comp rx=0 tx=0).stop 2023-03-03T10:52:08.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 shutdown_connections 2023-03-03T10:52:08.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f24940682f0 0x7f249406a7a0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:08.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24b00b3970 0x7f24b015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:08.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24b00cf0d0 0x7f24b01ee700 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:08.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 --2- 172.21.15.102:0/597612596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b00ced00 0x7f24b01ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:08.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 >> 172.21.15.102:0/597612596 conn(0x7f24b008b1c0 msgr2=0x7f24b0065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:08.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 shutdown_connections 2023-03-03T10:52:08.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:52:08.227+0000 7f24b5f9d700 1 -- 172.21.15.102:0/597612596 wait complete. 2023-03-03T10:52:08.232 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:52:08.257 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T10:52:23.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:23.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:53.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:53.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:23.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:23.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:53.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:53.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:59.259 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:53:59.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.507+0000 7fb7235ff700 1 Processor -- start 2023-03-03T10:53:59.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.507+0000 7fb7235ff700 1 -- start start 2023-03-03T10:53:59.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb7235ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb71c0c5f30 0x7fb71c06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:53:59.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb7235ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c2020 0x7fb71c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:53:59.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb7235ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c0c5390 con 0x7fb71c150c80 2023-03-03T10:53:59.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb7235ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71c157ad0 con 0x7fb71c0c5f30 2023-03-03T10:53:59.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb7235ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c0c4a20 con 0x7fb71c0c2020 2023-03-03T10:53:59.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb72139b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb71c150c80 0x7fb71c0c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56174/0 (socket says 172.21.15.102:56174) 2023-03-03T10:53:59.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb72139b700 1 -- 172.21.15.102:0/1078410792 learned_addr learned my addr 172.21.15.102:0/1078410792 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:53:59.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb721b9c700 1 --2- 172.21.15.102:0/1078410792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c2020 0x7fb71c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:53:59.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb720b9a700 1 --2- 172.21.15.102:0/1078410792 >> v2:172.21.15.172:3300/0 conn(0x7fb71c0c5f30 0x7fb71c06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:53:59.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.509+0000 7fb713fff700 1 -- 172.21.15.102:0/1078410792 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364803041 0 0) 0x7fb71c0c5390 con 0x7fb71c150c80 2023-03-03T10:53:59.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb713fff700 1 -- 172.21.15.102:0/1078410792 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7080035b0 con 0x7fb71c150c80 2023-03-03T10:53:59.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb721b9c700 1 -- 172.21.15.102:0/1078410792 >> v2:172.21.15.172:3300/0 conn(0x7fb71c0c5f30 msgr2=0x7fb71c06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:53:59.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb721b9c700 1 --2- 172.21.15.102:0/1078410792 >> v2:172.21.15.172:3300/0 conn(0x7fb71c0c5f30 0x7fb71c06d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb721b9c700 1 -- 172.21.15.102:0/1078410792 >> v1:172.21.15.102:6789/0 conn(0x7fb71c150c80 legacy=0x7fb71c0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:53:59.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb721b9c700 1 -- 172.21.15.102:0/1078410792 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c06f010 con 0x7fb71c0c2020 2023-03-03T10:53:59.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb721b9c700 1 --2- 172.21.15.102:0/1078410792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c2020 0x7fb71c0c2720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb714009d90 tx=0x7fb71400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e096cf8617cd473e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:53:59.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb713fff700 1 -- 172.21.15.102:0/1078410792 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71400e070 con 0x7fb71c0c2020 2023-03-03T10:53:59.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb713fff700 1 -- 172.21.15.102:0/1078410792 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71400bb20 con 0x7fb71c0c2020 2023-03-03T10:53:59.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.510+0000 7fb713fff700 1 -- 172.21.15.102:0/1078410792 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb714012430 con 0x7fb71c0c2020 2023-03-03T10:53:59.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 -- 172.21.15.102:0/1078410792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c2020 msgr2=0x7fb71c0c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:53:59.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 --2- 172.21.15.102:0/1078410792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c2020 0x7fb71c0c2720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb714009d90 tx=0x7fb71400b330 comp rx=0 tx=0).stop 2023-03-03T10:53:59.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 -- 172.21.15.102:0/1078410792 shutdown_connections 2023-03-03T10:53:59.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 --2- 172.21.15.102:0/1078410792 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c2020 0x7fb71c0c2720 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 --2- 172.21.15.102:0/1078410792 >> v2:172.21.15.172:3300/0 conn(0x7fb71c0c5f30 0x7fb71c06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 -- 172.21.15.102:0/1078410792 >> 172.21.15.102:0/1078410792 conn(0x7fb71c06af90 msgr2=0x7fb71c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:53:59.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 -- 172.21.15.102:0/1078410792 shutdown_connections 2023-03-03T10:53:59.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.511+0000 7fb7235ff700 1 -- 172.21.15.102:0/1078410792 wait complete. 2023-03-03T10:53:59.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.513+0000 7fb7235ff700 1 Processor -- start 2023-03-03T10:53:59.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.513+0000 7fb7235ff700 1 -- start start 2023-03-03T10:53:59.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.514+0000 7fb7235ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb71c0c2020 0x7fb71c1ec3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:53:59.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.514+0000 7fb7235ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c5f30 0x7fb71c152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:53:59.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.514+0000 7fb7235ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb71c150c80 0x7fb71c0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:53:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.514+0000 7fb7235ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c1516c0 con 0x7fb71c0c2020 2023-03-03T10:53:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.514+0000 7fb7235ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c0c9c60 con 0x7fb71c150c80 2023-03-03T10:53:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.514+0000 7fb7235ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c0c8d00 con 0x7fb71c0c5f30 2023-03-03T10:53:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.514+0000 7fb72139b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb71c0c2020 0x7fb71c1ec3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:53:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb721b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb71c150c80 0x7fb71c0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:53:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb72139b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb71c0c2020 0x7fb71c1ec3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50386/0 (socket says 172.21.15.102:50386) 2023-03-03T10:53:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb72139b700 1 -- 172.21.15.102:0/2988669621 learned_addr learned my addr 172.21.15.102:0/2988669621 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:53:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb720b9a700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c5f30 0x7fb71c152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:53:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb721b9c700 1 -- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c5f30 msgr2=0x7fb71c152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:53:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb721b9c700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c5f30 0x7fb71c152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb721b9c700 1 -- 172.21.15.102:0/2988669621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb71c0c2020 msgr2=0x7fb71c1ec3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:53:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb721b9c700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb71c0c2020 0x7fb71c1ec3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb721b9c700 1 -- 172.21.15.102:0/2988669621 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb714009a70 con 0x7fb71c150c80 2023-03-03T10:53:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb72139b700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb71c0c2020 0x7fb71c1ec3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:53:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb720b9a700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c5f30 0x7fb71c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:53:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.515+0000 7fb721b9c700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb71c150c80 0x7fb71c0ca080 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb71c072a90 tx=0x7fb714004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:53:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.516+0000 7fb711ffb700 1 -- 172.21.15.102:0/2988669621 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb714012430 con 0x7fb71c150c80 2023-03-03T10:53:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.516+0000 7fb711ffb700 1 -- 172.21.15.102:0/2988669621 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb714020070 con 0x7fb71c150c80 2023-03-03T10:53:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.516+0000 7fb711ffb700 1 -- 172.21.15.102:0/2988669621 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7140098f0 con 0x7fb71c150c80 2023-03-03T10:53:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.516+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c1577f0 con 0x7fb71c150c80 2023-03-03T10:53:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.516+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c157510 con 0x7fb71c150c80 2023-03-03T10:53:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.517+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c0c4cc0 con 0x7fb71c150c80 2023-03-03T10:53:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.518+0000 7fb711ffb700 1 -- 172.21.15.102:0/2988669621 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb71402e070 con 0x7fb71c150c80 2023-03-03T10:53:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.519+0000 7fb711ffb700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb7000682f0 0x7fb70006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:53:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.519+0000 7fb711ffb700 1 -- 172.21.15.102:0/2988669621 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb714022070 con 0x7fb71c150c80 2023-03-03T10:53:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.520+0000 7fb72139b700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb7000682f0 0x7fb70006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:53:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.520+0000 7fb72139b700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb7000682f0 0x7fb70006a7a0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb7180011e0 tx=0x7fb718009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:53:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.524+0000 7fb711ffb700 1 -- 172.21.15.102:0/2988669621 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb714054000 con 0x7fb71c150c80 2023-03-03T10:53:59.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.759+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb71c04e6e0 con 0x7fb71c150c80 2023-03-03T10:53:59.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.761+0000 7fb711ffb700 1 -- 172.21.15.102:0/2988669621 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb714057620 con 0x7fb71c150c80 2023-03-03T10:53:59.763 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:53:59.763 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:53:59.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb7000682f0 msgr2=0x7fb70006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:53:59.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb7000682f0 0x7fb70006a7a0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb7180011e0 tx=0x7fb718009040 comp rx=0 tx=0).stop 2023-03-03T10:53:59.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb71c150c80 msgr2=0x7fb71c0ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:53:59.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb71c150c80 0x7fb71c0ca080 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb71c072a90 tx=0x7fb714004ad0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 shutdown_connections 2023-03-03T10:53:59.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb71c150c80 0x7fb71c0ca080 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb71c0c2020 0x7fb71c1ec3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb7000682f0 0x7fb70006a7a0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 --2- 172.21.15.102:0/2988669621 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb71c0c5f30 0x7fb71c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:53:59.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 >> 172.21.15.102:0/2988669621 conn(0x7fb71c06af90 msgr2=0x7fb71c14e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:53:59.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.763+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 shutdown_connections 2023-03-03T10:53:59.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:53:59.764+0000 7fb7235ff700 1 -- 172.21.15.102:0/2988669621 wait complete. 2023-03-03T10:53:59.767 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:53:59.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 103 secs before thrashing 2023-03-03T10:54:23.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:23.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:54.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:54.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:24.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:24.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:42.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:55:43.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.052+0000 7f49639ff700 1 Processor -- start 2023-03-03T10:55:43.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.053+0000 7f49639ff700 1 -- start start 2023-03-03T10:55:43.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f49639ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f495c0cd8e0 0x7f495c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:55:43.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f49639ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495c153660 con 0x7f495c0cd390 2023-03-03T10:55:43.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f49639ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495c06c360 con 0x7f495c153290 2023-03-03T10:55:43.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f49639ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c15cea0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f4960f9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f495c0cd8e0 0x7f495c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:55:43.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f4960f9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f495c0cd8e0 0x7f495c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42284/0 (socket says 172.21.15.102:42284) 2023-03-03T10:55:43.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f4960f9a700 1 -- 172.21.15.102:0/3747855074 learned_addr learned my addr 172.21.15.102:0/3747855074 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:55:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.054+0000 7f496179b700 1 --1- 172.21.15.102:0/3747855074 >> v1:172.21.15.172:6789/0 conn(0x7f495c153290 0x7f495c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40812/0 (socket says 172.21.15.102:40812) 2023-03-03T10:55:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4960f9a700 1 -- 172.21.15.102:0/3747855074 >> v1:172.21.15.172:6789/0 conn(0x7f495c153290 legacy=0x7f495c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:55:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4960f9a700 1 -- 172.21.15.102:0/3747855074 >> v1:172.21.15.102:6789/0 conn(0x7f495c0cd390 legacy=0x7f495c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:55:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4960f9a700 1 -- 172.21.15.102:0/3747855074 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c06c7f0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4953fff700 1 -- 172.21.15.102:0/3747855074 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931635979 0 0) 0x7f495c06c360 con 0x7f495c153290 2023-03-03T10:55:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4960f9a700 1 --2- 172.21.15.102:0/3747855074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd8e0 0x7f495c0cde20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f494c0098f0 tx=0x7f494c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb412043147ce5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:55:43.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4953fff700 1 -- 172.21.15.102:0/3747855074 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c00b8a0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4953fff700 1 -- 172.21.15.102:0/3747855074 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c00ba00 con 0x7f495c0cd8e0 2023-03-03T10:55:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.055+0000 7f4953fff700 1 -- 172.21.15.102:0/3747855074 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c00bd90 con 0x7f495c0cd8e0 2023-03-03T10:55:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.056+0000 7f49639ff700 1 -- 172.21.15.102:0/3747855074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd8e0 msgr2=0x7f495c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:55:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.056+0000 7f49639ff700 1 --2- 172.21.15.102:0/3747855074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd8e0 0x7f495c0cde20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f494c0098f0 tx=0x7f494c009c00 comp rx=0 tx=0).stop 2023-03-03T10:55:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.056+0000 7f49639ff700 1 -- 172.21.15.102:0/3747855074 shutdown_connections 2023-03-03T10:55:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.056+0000 7f49639ff700 1 --2- 172.21.15.102:0/3747855074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd8e0 0x7f495c0cde20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:55:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.056+0000 7f49639ff700 1 -- 172.21.15.102:0/3747855074 >> 172.21.15.102:0/3747855074 conn(0x7f495c065fb0 msgr2=0x7f495c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:55:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.056+0000 7f49639ff700 1 -- 172.21.15.102:0/3747855074 shutdown_connections 2023-03-03T10:55:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.056+0000 7f49639ff700 1 -- 172.21.15.102:0/3747855074 wait complete. 2023-03-03T10:55:43.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.059+0000 7f49639ff700 1 Processor -- start 2023-03-03T10:55:43.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.059+0000 7f49639ff700 1 -- start start 2023-03-03T10:55:43.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f49639ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd390 0x7f495c0b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:55:43.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f49639ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f495c0cd8e0 0x7f495c1e3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:55:43.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f49639ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f495c153290 0x7f495c1e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:55:43.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f49639ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0b2140 con 0x7f495c0cd390 2023-03-03T10:55:43.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f49639ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c1e4fc0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f49639ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c1e4a90 con 0x7f495c153290 2023-03-03T10:55:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f4960f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f495c0cd8e0 0x7f495c1e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:55:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f496179b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd390 0x7f495c0b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:55:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f4960f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f495c0cd8e0 0x7f495c1e3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55020/0 (socket says 172.21.15.102:55020) 2023-03-03T10:55:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f4960f9a700 1 -- 172.21.15.102:0/3098197537 learned_addr learned my addr 172.21.15.102:0/3098197537 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:55:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.061+0000 7f4961f9c700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f495c153290 0x7f495c1e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:55:43.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4960f9a700 1 -- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f495c153290 msgr2=0x7f495c1e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:55:43.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4960f9a700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f495c153290 0x7f495c1e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:55:43.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4960f9a700 1 -- 172.21.15.102:0/3098197537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd390 msgr2=0x7f495c0b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:55:43.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4960f9a700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd390 0x7f495c0b3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:55:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4960f9a700 1 -- 172.21.15.102:0/3098197537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c009550 con 0x7f495c0cd8e0 2023-03-03T10:55:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4960f9a700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f495c0cd8e0 0x7f495c1e3680 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f495c074540 tx=0x7f494c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:55:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4951ffb700 1 -- 172.21.15.102:0/3098197537 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c01a900 con 0x7f495c0cd8e0 2023-03-03T10:55:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4951ffb700 1 -- 172.21.15.102:0/3098197537 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c01f070 con 0x7f495c0cd8e0 2023-03-03T10:55:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495c0ccde0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f4951ffb700 1 -- 172.21.15.102:0/3098197537 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c01ac60 con 0x7f495c0cd8e0 2023-03-03T10:55:43.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.062+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c15d160 con 0x7f495c0cd8e0 2023-03-03T10:55:43.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.063+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c0cd0c0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.065+0000 7f4951ffb700 1 -- 172.21.15.102:0/3098197537 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f494c00f050 con 0x7f495c0cd8e0 2023-03-03T10:55:43.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.066+0000 7f4951ffb700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49400682f0 0x7f494006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:55:43.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.066+0000 7f4951ffb700 1 -- 172.21.15.102:0/3098197537 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f494c021070 con 0x7f495c0cd8e0 2023-03-03T10:55:43.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.066+0000 7f496179b700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49400682f0 0x7f494006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:55:43.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.067+0000 7f496179b700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49400682f0 0x7f494006a7a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4958000f60 tx=0x7f4958007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:55:43.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.070+0000 7f4951ffb700 1 -- 172.21.15.102:0/3098197537 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f494c0536f0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.304+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f495c04e6e0 con 0x7f495c0cd8e0 2023-03-03T10:55:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.305+0000 7f4951ffb700 1 -- 172.21.15.102:0/3098197537 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f494c015020 con 0x7f495c0cd8e0 2023-03-03T10:55:43.307 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:55:43.307 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:55:43.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49400682f0 msgr2=0x7f494006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:55:43.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49400682f0 0x7f494006a7a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4958000f60 tx=0x7f4958007e70 comp rx=0 tx=0).stop 2023-03-03T10:55:43.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f495c0cd8e0 msgr2=0x7f495c1e3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:55:43.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f495c0cd8e0 0x7f495c1e3680 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f495c074540 tx=0x7f494c004750 comp rx=0 tx=0).stop 2023-03-03T10:55:43.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 shutdown_connections 2023-03-03T10:55:43.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49400682f0 0x7f494006a7a0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:55:43.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f495c153290 0x7f495c1e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:55:43.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f495c0cd390 0x7f495c0b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:55:43.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 --2- 172.21.15.102:0/3098197537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f495c0cd8e0 0x7f495c1e3680 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:55:43.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.307+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 >> 172.21.15.102:0/3098197537 conn(0x7f495c065fb0 msgr2=0x7f495c0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:55:43.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.308+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 shutdown_connections 2023-03-03T10:55:43.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:55:43.308+0000 7f49639ff700 1 -- 172.21.15.102:0/3098197537 wait complete. 2023-03-03T10:55:43.312 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:55:43.336 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-03-03T10:55:54.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:54.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:24.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:24.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:25.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:56:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.591+0000 7fd6821ff700 1 Processor -- start 2023-03-03T10:56:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.591+0000 7fd6821ff700 1 -- start start 2023-03-03T10:56:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd6821ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd67c150c80 0x7fd67c06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd6821ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c0c2020 0x7fd67c0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd6821ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd67c0c5d60 0x7fd67c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd6821ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67c157ad0 con 0x7fd67c150c80 2023-03-03T10:56:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd6821ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67c0c5390 con 0x7fd67c0c5d60 2023-03-03T10:56:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd6821ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0c61e0 con 0x7fd67c0c2020 2023-03-03T10:56:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd67bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd67c0c5d60 0x7fd67c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd673fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c0c2020 0x7fd67c0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd67b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd67c150c80 0x7fd67c06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd673fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c0c2020 0x7fd67c0c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46304/0 (socket says 172.21.15.102:46304) 2023-03-03T10:56:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd673fff700 1 -- 172.21.15.102:0/856922111 learned_addr learned my addr 172.21.15.102:0/856922111 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:56:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.593+0000 7fd67bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd67c0c5d60 0x7fd67c0c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45202/0 (socket says 172.21.15.102:45202) 2023-03-03T10:56:25.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67b7fe700 1 -- 172.21.15.102:0/856922111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c0c2020 msgr2=0x7fd67c0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:25.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67b7fe700 1 --2- 172.21.15.102:0/856922111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c0c2020 0x7fd67c0c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67b7fe700 1 -- 172.21.15.102:0/856922111 >> v2:172.21.15.172:3300/0 conn(0x7fd67c0c5d60 msgr2=0x7fd67c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:25.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67b7fe700 1 --2- 172.21.15.102:0/856922111 >> v2:172.21.15.172:3300/0 conn(0x7fd67c0c5d60 0x7fd67c0c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67b7fe700 1 -- 172.21.15.102:0/856922111 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0c4a20 con 0x7fd67c150c80 2023-03-03T10:56:25.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67bfff700 1 --2- 172.21.15.102:0/856922111 >> v2:172.21.15.172:3300/0 conn(0x7fd67c0c5d60 0x7fd67c0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:56:25.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd673fff700 1 --2- 172.21.15.102:0/856922111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c0c2020 0x7fd67c0c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:56:25.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67b7fe700 1 --2- 172.21.15.102:0/856922111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c150c80 0x7fd67c06d100 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0c1310 tx=0x7fd66c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c142e8571cb9f7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67affd700 1 -- 172.21.15.102:0/856922111 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c011830 con 0x7fd67c150c80 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67affd700 1 -- 172.21.15.102:0/856922111 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c011990 con 0x7fd67c150c80 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.594+0000 7fd67affd700 1 -- 172.21.15.102:0/856922111 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c011d20 con 0x7fd67c150c80 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 -- 172.21.15.102:0/856922111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c150c80 msgr2=0x7fd67c06d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 --2- 172.21.15.102:0/856922111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c150c80 0x7fd67c06d100 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0c1310 tx=0x7fd66c009b20 comp rx=0 tx=0).stop 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 -- 172.21.15.102:0/856922111 shutdown_connections 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 --2- 172.21.15.102:0/856922111 >> v2:172.21.15.172:3300/0 conn(0x7fd67c0c5d60 0x7fd67c0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 --2- 172.21.15.102:0/856922111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c150c80 0x7fd67c06d100 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 --2- 172.21.15.102:0/856922111 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c0c2020 0x7fd67c0c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 -- 172.21.15.102:0/856922111 >> 172.21.15.102:0/856922111 conn(0x7fd67c06af90 msgr2=0x7fd67c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:25.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 -- 172.21.15.102:0/856922111 shutdown_connections 2023-03-03T10:56:25.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.595+0000 7fd6821ff700 1 -- 172.21.15.102:0/856922111 wait complete. 2023-03-03T10:56:25.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.598+0000 7fd6821ff700 1 Processor -- start 2023-03-03T10:56:25.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.598+0000 7fd6821ff700 1 -- start start 2023-03-03T10:56:25.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd6821ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0c2020 0x7fd67c1568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:25.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd6821ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd67c0c5d60 0x7fd67c147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:25.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd6821ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c150c80 0x7fd67c141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:25.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd6821ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c1472d0 con 0x7fd67c0c2020 2023-03-03T10:56:25.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd6821ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c148710 con 0x7fd67c0c5d60 2023-03-03T10:56:25.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd6821ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c148450 con 0x7fd67c150c80 2023-03-03T10:56:25.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd67b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0c2020 0x7fd67c1568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:25.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd67b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0c2020 0x7fd67c1568a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43804/0 (socket says 172.21.15.102:43804) 2023-03-03T10:56:25.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd67b7fe700 1 -- 172.21.15.102:0/2672696074 learned_addr learned my addr 172.21.15.102:0/2672696074 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:56:25.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd67bfff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c150c80 0x7fd67c141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:25.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.600+0000 7fd673fff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd67c0c5d60 0x7fd67c147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:25.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd67b7fe700 1 -- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c150c80 msgr2=0x7fd67c141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:25.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd67b7fe700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c150c80 0x7fd67c141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd67b7fe700 1 -- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd67c0c5d60 msgr2=0x7fd67c147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:25.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd67b7fe700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd67c0c5d60 0x7fd67c147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd67b7fe700 1 -- 172.21.15.102:0/2672696074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd664009550 con 0x7fd67c0c2020 2023-03-03T10:56:25.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd67b7fe700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0c2020 0x7fd67c1568a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd66c000c00 tx=0x7fd66c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:25.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd678ff9700 1 -- 172.21.15.102:0/2672696074 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0044d0 con 0x7fd67c0c2020 2023-03-03T10:56:25.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd678ff9700 1 -- 172.21.15.102:0/2672696074 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c01e070 con 0x7fd67c0c2020 2023-03-03T10:56:25.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd678ff9700 1 -- 172.21.15.102:0/2672696074 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0038e0 con 0x7fd67c0c2020 2023-03-03T10:56:25.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.601+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd674009a70 con 0x7fd67c0c2020 2023-03-03T10:56:25.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.602+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c0094a0 con 0x7fd67c0c2020 2023-03-03T10:56:25.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.603+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c157510 con 0x7fd67c0c2020 2023-03-03T10:56:25.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.604+0000 7fd678ff9700 1 -- 172.21.15.102:0/2672696074 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd66c00f050 con 0x7fd67c0c2020 2023-03-03T10:56:25.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.605+0000 7fd678ff9700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd65c0682f0 0x7fd65c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:25.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.605+0000 7fd678ff9700 1 -- 172.21.15.102:0/2672696074 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd66c086a20 con 0x7fd67c0c2020 2023-03-03T10:56:25.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.605+0000 7fd673fff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd65c0682f0 0x7fd65c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:25.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.606+0000 7fd673fff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd65c0682f0 0x7fd65c06a7a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd67c14aeb0 tx=0x7fd6640092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:25.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.610+0000 7fd678ff9700 1 -- 172.21.15.102:0/2672696074 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd66c0531f0 con 0x7fd67c0c2020 2023-03-03T10:56:25.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.848+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd67c1577f0 con 0x7fd67c0c2020 2023-03-03T10:56:25.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.848+0000 7fd678ff9700 1 -- 172.21.15.102:0/2672696074 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd66c015090 con 0x7fd67c0c2020 2023-03-03T10:56:25.850 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:56:25.851 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:56:25.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd65c0682f0 msgr2=0x7fd65c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:25.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd65c0682f0 0x7fd65c06a7a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd67c14aeb0 tx=0x7fd6640092c0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0c2020 msgr2=0x7fd67c1568a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:25.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0c2020 0x7fd67c1568a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd66c000c00 tx=0x7fd66c004750 comp rx=0 tx=0).stop 2023-03-03T10:56:25.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 shutdown_connections 2023-03-03T10:56:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd65c0682f0 0x7fd65c06a7a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd67c150c80 0x7fd67c141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0c2020 0x7fd67c1568a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 --2- 172.21.15.102:0/2672696074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd67c0c5d60 0x7fd67c147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 >> 172.21.15.102:0/2672696074 conn(0x7fd67c06af90 msgr2=0x7fd67c14feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 shutdown_connections 2023-03-03T10:56:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:56:25.851+0000 7fd6821ff700 1 -- 172.21.15.102:0/2672696074 wait complete. 2023-03-03T10:56:25.855 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:56:25.881 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-03-03T10:56:54.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:54.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:24.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:24.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:55.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:55.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:01.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:58:02.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.128+0000 7f4d483ff700 1 Processor -- start 2023-03-03T10:58:02.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.129+0000 7f4d483ff700 1 -- start start 2023-03-03T10:58:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.130+0000 7f4d483ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd390 0x7f4d400ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.130+0000 7f4d483ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40153660 con 0x7f4d400cd8e0 2023-03-03T10:58:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.130+0000 7f4d483ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4006c360 con 0x7f4d40153290 2023-03-03T10:58:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.130+0000 7f4d483ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4015cea0 con 0x7f4d400cd390 2023-03-03T10:58:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.130+0000 7f4d4619b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4d40153290 0x7f4d40158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37218/0 (socket says 172.21.15.102:37218) 2023-03-03T10:58:02.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.130+0000 7f4d4619b700 1 -- 172.21.15.102:0/2145179377 learned_addr learned my addr 172.21.15.102:0/2145179377 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T10:58:02.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.130+0000 7f4d4699c700 1 --2- 172.21.15.102:0/2145179377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd390 0x7f4d400ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:02.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d45199700 1 -- 172.21.15.102:0/2145179377 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996575698 0 0) 0x7f4d4006c360 con 0x7f4d40153290 2023-03-03T10:58:02.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d45199700 1 -- 172.21.15.102:0/2145179377 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c0035b0 con 0x7f4d40153290 2023-03-03T10:58:02.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d45199700 1 -- 172.21.15.102:0/2145179377 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250788986 0 0) 0x7f4d40153660 con 0x7f4d400cd8e0 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d4699c700 1 -- 172.21.15.102:0/2145179377 >> v1:172.21.15.172:6789/0 conn(0x7f4d40153290 legacy=0x7f4d40158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d4699c700 1 -- 172.21.15.102:0/2145179377 >> v1:172.21.15.102:6789/0 conn(0x7f4d400cd8e0 legacy=0x7f4d400cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d4699c700 1 -- 172.21.15.102:0/2145179377 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4006c7f0 con 0x7f4d400cd390 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d4699c700 1 --2- 172.21.15.102:0/2145179377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd390 0x7f4d400ce530 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009dc0 tx=0x7f4d3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7f51d5e7ae598e2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.131+0000 7f4d45199700 1 -- 172.21.15.102:0/2145179377 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800e070 con 0x7f4d400cd390 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d45199700 1 -- 172.21.15.102:0/2145179377 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3800bb20 con 0x7f4d400cd390 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d45199700 1 -- 172.21.15.102:0/2145179377 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800bed0 con 0x7f4d400cd390 2023-03-03T10:58:02.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d483ff700 1 -- 172.21.15.102:0/2145179377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd390 msgr2=0x7f4d400ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2145179377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd390 0x7f4d400ce530 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009dc0 tx=0x7f4d3800b330 comp rx=0 tx=0).stop 2023-03-03T10:58:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d483ff700 1 -- 172.21.15.102:0/2145179377 shutdown_connections 2023-03-03T10:58:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2145179377 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd390 0x7f4d400ce530 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d483ff700 1 -- 172.21.15.102:0/2145179377 >> 172.21.15.102:0/2145179377 conn(0x7f4d40065fb0 msgr2=0x7f4d400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d483ff700 1 -- 172.21.15.102:0/2145179377 shutdown_connections 2023-03-03T10:58:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.132+0000 7f4d483ff700 1 -- 172.21.15.102:0/2145179377 wait complete. 2023-03-03T10:58:02.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.134+0000 7f4d483ff700 1 Processor -- start 2023-03-03T10:58:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.134+0000 7f4d483ff700 1 -- start start 2023-03-03T10:58:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.135+0000 7f4d483ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d400cd390 0x7f4d401e21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.135+0000 7f4d483ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd8e0 0x7f4d401e26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.135+0000 7f4d483ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d40153290 0x7f4d4014ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.135+0000 7f4d483ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d401e4d90 con 0x7f4d400cd390 2023-03-03T10:58:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.135+0000 7f4d483ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d401e41d0 con 0x7f4d40153290 2023-03-03T10:58:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.135+0000 7f4d483ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40150f20 con 0x7f4d400cd8e0 2023-03-03T10:58:02.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4619b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d400cd390 0x7f4d401e21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:02.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4599a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd8e0 0x7f4d401e26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:02.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4619b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d400cd390 0x7f4d401e21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33666/0 (socket says 172.21.15.102:33666) 2023-03-03T10:58:02.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4619b700 1 -- 172.21.15.102:0/2222620661 learned_addr learned my addr 172.21.15.102:0/2222620661 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:58:02.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4699c700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d40153290 0x7f4d4014ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:02.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4599a700 1 -- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d40153290 msgr2=0x7f4d4014ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:02.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4599a700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d40153290 0x7f4d4014ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4599a700 1 -- 172.21.15.102:0/2222620661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d400cd390 msgr2=0x7f4d401e21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4599a700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d400cd390 0x7f4d401e21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4599a700 1 -- 172.21.15.102:0/2222620661 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d38009a70 con 0x7f4d400cd8e0 2023-03-03T10:58:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.136+0000 7f4d4599a700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd8e0 0x7f4d401e26e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000c520 tx=0x7f4d3000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.137+0000 7f4d36ffd700 1 -- 172.21.15.102:0/2222620661 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300054e0 con 0x7f4d400cd8e0 2023-03-03T10:58:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.137+0000 7f4d36ffd700 1 -- 172.21.15.102:0/2222620661 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d30003680 con 0x7f4d400cd8e0 2023-03-03T10:58:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.137+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d400ccde0 con 0x7f4d400cd8e0 2023-03-03T10:58:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.137+0000 7f4d36ffd700 1 -- 172.21.15.102:0/2222620661 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30003a30 con 0x7f4d400cd8e0 2023-03-03T10:58:02.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.137+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4015d160 con 0x7f4d400cd8e0 2023-03-03T10:58:02.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.138+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d400cd0c0 con 0x7f4d400cd8e0 2023-03-03T10:58:02.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.139+0000 7f4d36ffd700 1 -- 172.21.15.102:0/2222620661 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4d30019030 con 0x7f4d400cd8e0 2023-03-03T10:58:02.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.141+0000 7f4d36ffd700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d240682f0 0x7f4d2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:02.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.141+0000 7f4d36ffd700 1 -- 172.21.15.102:0/2222620661 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4d300852b0 con 0x7f4d400cd8e0 2023-03-03T10:58:02.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.141+0000 7f4d4619b700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d240682f0 0x7f4d2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:02.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.142+0000 7f4d4619b700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d240682f0 0x7f4d2406a7a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c001220 tx=0x7f4d3c000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.143+0000 7f4d36ffd700 1 -- 172.21.15.102:0/2222620661 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4d30051a60 con 0x7f4d400cd8e0 2023-03-03T10:58:02.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.379+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d401e4f40 con 0x7f4d400cd8e0 2023-03-03T10:58:02.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.381+0000 7f4d36ffd700 1 -- 172.21.15.102:0/2222620661 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4d401e4f40 con 0x7f4d400cd8e0 2023-03-03T10:58:02.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:58:02.466 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:58:02.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d240682f0 msgr2=0x7f4d2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:02.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d240682f0 0x7f4d2406a7a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c001220 tx=0x7f4d3c000ed0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd8e0 msgr2=0x7f4d401e26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:02.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd8e0 0x7f4d401e26e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000c520 tx=0x7f4d3000c830 comp rx=0 tx=0).stop 2023-03-03T10:58:02.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 shutdown_connections 2023-03-03T10:58:02.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d40153290 0x7f4d4014ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d400cd390 0x7f4d401e21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4d240682f0 0x7f4d2406a7a0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 --2- 172.21.15.102:0/2222620661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4d400cd8e0 0x7f4d401e26e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:02.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 >> 172.21.15.102:0/2222620661 conn(0x7f4d40065fb0 msgr2=0x7f4d40156b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:02.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.383+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 shutdown_connections 2023-03-03T10:58:02.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:02.384+0000 7f4d483ff700 1 -- 172.21.15.102:0/2222620661 wait complete. 2023-03-03T10:58:02.469 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:58:02.469 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2023-03-03T10:58:15.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:58:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.717+0000 7f37d179d700 1 Processor -- start 2023-03-03T10:58:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.718+0000 7f37d179d700 1 -- start start 2023-03-03T10:58:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37d179d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37cc152ea0 0x7f37cc158a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37d179d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0cf910 0x7f37cc0caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37d179d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37cc0ca200 0x7f37cc0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37d179d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc0c9b40 con 0x7f37cc152ea0 2023-03-03T10:58:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37d179d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc0c8b40 con 0x7f37cc0ca200 2023-03-03T10:58:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37d179d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0b4580 con 0x7f37cc0cf910 2023-03-03T10:58:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37cbfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37cc152ea0 0x7f37cc158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37c3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0cf910 0x7f37cc0caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37d0f9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37cc0ca200 0x7f37cc0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37cbfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37cc152ea0 0x7f37cc158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56928/0 (socket says 172.21.15.102:56928) 2023-03-03T10:58:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.719+0000 7f37cbfff700 1 -- 172.21.15.102:0/2844312160 learned_addr learned my addr 172.21.15.102:0/2844312160 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:58:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37c3fff700 1 -- 172.21.15.102:0/2844312160 >> v2:172.21.15.172:3300/0 conn(0x7f37cc0ca200 msgr2=0x7f37cc0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37c3fff700 1 --2- 172.21.15.102:0/2844312160 >> v2:172.21.15.172:3300/0 conn(0x7f37cc0ca200 0x7f37cc0caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37c3fff700 1 -- 172.21.15.102:0/2844312160 >> v2:172.21.15.102:3300/0 conn(0x7f37cc152ea0 msgr2=0x7f37cc158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37c3fff700 1 --2- 172.21.15.102:0/2844312160 >> v2:172.21.15.102:3300/0 conn(0x7f37cc152ea0 0x7f37cc158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37c3fff700 1 -- 172.21.15.102:0/2844312160 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc0c8fd0 con 0x7f37cc0cf910 2023-03-03T10:58:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37cbfff700 1 --2- 172.21.15.102:0/2844312160 >> v2:172.21.15.102:3300/0 conn(0x7f37cc152ea0 0x7f37cc158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:58:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37c3fff700 1 --2- 172.21.15.102:0/2844312160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0cf910 0x7f37cc0caa00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0c5c00 tx=0x7f37b4009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33748ee384de8c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.720+0000 7f37cb7fe700 1 -- 172.21.15.102:0/2844312160 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4011830 con 0x7f37cc0cf910 2023-03-03T10:58:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37cb7fe700 1 -- 172.21.15.102:0/2844312160 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b4011990 con 0x7f37cc0cf910 2023-03-03T10:58:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37cb7fe700 1 -- 172.21.15.102:0/2844312160 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4011d20 con 0x7f37cc0cf910 2023-03-03T10:58:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 -- 172.21.15.102:0/2844312160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0cf910 msgr2=0x7f37cc0caa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 --2- 172.21.15.102:0/2844312160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0cf910 0x7f37cc0caa00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0c5c00 tx=0x7f37b4009b20 comp rx=0 tx=0).stop 2023-03-03T10:58:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 -- 172.21.15.102:0/2844312160 shutdown_connections 2023-03-03T10:58:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 --2- 172.21.15.102:0/2844312160 >> v2:172.21.15.172:3300/0 conn(0x7f37cc0ca200 0x7f37cc0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 --2- 172.21.15.102:0/2844312160 >> v2:172.21.15.102:3300/0 conn(0x7f37cc152ea0 0x7f37cc158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 --2- 172.21.15.102:0/2844312160 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0cf910 0x7f37cc0caa00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 -- 172.21.15.102:0/2844312160 >> 172.21.15.102:0/2844312160 conn(0x7f37cc066c40 msgr2=0x7f37cc0cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 -- 172.21.15.102:0/2844312160 shutdown_connections 2023-03-03T10:58:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.721+0000 7f37d179d700 1 -- 172.21.15.102:0/2844312160 wait complete. 2023-03-03T10:58:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.724+0000 7f37d179d700 1 Processor -- start 2023-03-03T10:58:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.725+0000 7f37d179d700 1 -- start start 2023-03-03T10:58:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37d179d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0ca200 0x7f37cc0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37d179d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0cf910 0x7f37cc0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37d179d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc152ea0 0x7f37cc150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37d179d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0c4440 con 0x7f37cc0cf910 2023-03-03T10:58:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37d179d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc159a70 con 0x7f37cc152ea0 2023-03-03T10:58:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37d179d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc1e2130 con 0x7f37cc0ca200 2023-03-03T10:58:15.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37c3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0cf910 0x7f37cc0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37cbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0ca200 0x7f37cc0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37c3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0cf910 0x7f37cc0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56936/0 (socket says 172.21.15.102:56936) 2023-03-03T10:58:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37c3fff700 1 -- 172.21.15.102:0/3893892370 learned_addr learned my addr 172.21.15.102:0/3893892370 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T10:58:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.726+0000 7f37d0f9c700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc152ea0 0x7f37cc150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c3fff700 1 -- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0ca200 msgr2=0x7f37cc0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c3fff700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0ca200 0x7f37cc0c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c3fff700 1 -- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc152ea0 msgr2=0x7f37cc150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c3fff700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc152ea0 0x7f37cc150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c3fff700 1 -- 172.21.15.102:0/3893892370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc009550 con 0x7f37cc0cf910 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37cbfff700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0ca200 0x7f37cc0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c3fff700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0cf910 0x7f37cc0c2fe0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009b60 tx=0x7f37b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c97fa700 1 -- 172.21.15.102:0/3893892370 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4004510 con 0x7f37cc0cf910 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c97fa700 1 -- 172.21.15.102:0/3893892370 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b401f070 con 0x7f37cc0cf910 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b40094a0 con 0x7f37cc0cf910 2023-03-03T10:58:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.727+0000 7f37c97fa700 1 -- 172.21.15.102:0/3893892370 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40038e0 con 0x7f37cc0cf910 2023-03-03T10:58:15.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.728+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc0c9580 con 0x7f37cc0cf910 2023-03-03T10:58:15.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.729+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc0b4840 con 0x7f37cc0cf910 2023-03-03T10:58:15.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.730+0000 7f37c97fa700 1 -- 172.21.15.102:0/3893892370 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37b400f050 con 0x7f37cc0cf910 2023-03-03T10:58:15.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.731+0000 7f37c97fa700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:15.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.731+0000 7f37c97fa700 1 -- 172.21.15.102:0/3893892370 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f37b4021070 con 0x7f37cc0cf910 2023-03-03T10:58:15.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.731+0000 7f37cbfff700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:15.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.732+0000 7f37cbfff700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f37bc006d30 tx=0x7f37bc0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:15.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.735+0000 7f37c97fa700 1 -- 172.21.15.102:0/3893892370 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37b4053aa0 con 0x7f37cc0cf910 2023-03-03T10:58:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.967+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37cc0ca870 con 0x7f37cc0cf910 2023-03-03T10:58:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.968+0000 7f37c97fa700 1 -- 172.21.15.102:0/3893892370 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f37b4015020 con 0x7f37cc0cf910 2023-03-03T10:58:15.971 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T10:58:15.971 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T10:58:15.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 msgr2=0x7f37ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:15.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f37bc006d30 tx=0x7f37bc0092c0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0cf910 msgr2=0x7f37cc0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0cf910 0x7f37cc0c2fe0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009b60 tx=0x7f37b4004750 comp rx=0 tx=0).stop 2023-03-03T10:58:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 shutdown_connections 2023-03-03T10:58:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37ac0682f0 0x7f37ac06a7a0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37cc0ca200 0x7f37cc0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37cc152ea0 0x7f37cc150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 --2- 172.21.15.102:0/3893892370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37cc0cf910 0x7f37cc0c2fe0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:15.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.970+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 >> 172.21.15.102:0/3893892370 conn(0x7f37cc066c40 msgr2=0x7f37cc06ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:15.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.971+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 shutdown_connections 2023-03-03T10:58:15.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T10:58:15.971+0000 7f37d179d700 1 -- 172.21.15.102:0/3893892370 wait complete. 2023-03-03T10:58:15.975 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T10:58:16.000 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 118 secs before thrashing 2023-03-03T10:58:25.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:25.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:55.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:55.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:25.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:25.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:55.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:55.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:00.002 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T10:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:00:14.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:00:14.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.256+0000 7f47109fe700 1 Processor -- start 2023-03-03T11:00:14.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.256+0000 7f47109fe700 1 -- start start 2023-03-03T11:00:14.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.257+0000 7f47109fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:14.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.257+0000 7f47109fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f470c0c9610 0x7f470c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:14.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.257+0000 7f47109fe700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c0c9ca0 con 0x7f470c0ca440 2023-03-03T11:00:14.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.257+0000 7f47109fe700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470c0b4920 con 0x7f470c0c9610 2023-03-03T11:00:14.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.257+0000 7f47109fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0b4c00 con 0x7f470c1564b0 2023-03-03T11:00:14.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470b59e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f470c0ca440 0x7f470c0ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53196/0 (socket says 172.21.15.102:53196) 2023-03-03T11:00:14.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470b59e700 1 -- 172.21.15.102:0/947130117 learned_addr learned my addr 172.21.15.102:0/947130117 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:00:14.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470ad9d700 1 --2- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:14.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470a59c700 1 --2- 172.21.15.102:0/947130117 >> v2:172.21.15.172:3300/0 conn(0x7f470c0c9610 0x7f470c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:14.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f4709d9b700 1 -- 172.21.15.102:0/947130117 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575583100 0 0) 0x7f470c0c9ca0 con 0x7f470c0ca440 2023-03-03T11:00:14.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f4709d9b700 1 -- 172.21.15.102:0/947130117 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f40035b0 con 0x7f470c0ca440 2023-03-03T11:00:14.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470a59c700 1 -- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 msgr2=0x7f470c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:00:14.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470a59c700 1 --2- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470a59c700 1 -- 172.21.15.102:0/947130117 >> v1:172.21.15.102:6789/0 conn(0x7f470c0ca440 legacy=0x7f470c0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:14.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470a59c700 1 -- 172.21.15.102:0/947130117 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c0b46f0 con 0x7f470c0c9610 2023-03-03T11:00:14.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.258+0000 7f470ad9d700 1 --2- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:00:14.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.259+0000 7f470a59c700 1 --2- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0c9610 0x7f470c0cbba0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4700009840 tx=0x7f4700009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8b523953594e66f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:14.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.259+0000 7f4709d9b700 1 -- 172.21.15.102:0/947130117 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700011830 con 0x7f470c0c9610 2023-03-03T11:00:14.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.259+0000 7f4709d9b700 1 -- 172.21.15.102:0/947130117 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4700011990 con 0x7f470c0c9610 2023-03-03T11:00:14.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.259+0000 7f4709d9b700 1 -- 172.21.15.102:0/947130117 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700011d20 con 0x7f470c0c9610 2023-03-03T11:00:14.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 -- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0c9610 msgr2=0x7f470c0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:14.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 --2- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0c9610 0x7f470c0cbba0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4700009840 tx=0x7f4700009b50 comp rx=0 tx=0).stop 2023-03-03T11:00:14.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 -- 172.21.15.102:0/947130117 shutdown_connections 2023-03-03T11:00:14.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 --2- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 --2- 172.21.15.102:0/947130117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0c9610 0x7f470c0cbba0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 -- 172.21.15.102:0/947130117 >> 172.21.15.102:0/947130117 conn(0x7f470c065fb0 msgr2=0x7f470c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:00:14.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 -- 172.21.15.102:0/947130117 shutdown_connections 2023-03-03T11:00:14.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.260+0000 7f47109fe700 1 -- 172.21.15.102:0/947130117 wait complete. 2023-03-03T11:00:14.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.263+0000 7f47109fe700 1 Processor -- start 2023-03-03T11:00:14.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.263+0000 7f47109fe700 1 -- start start 2023-03-03T11:00:14.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.264+0000 7f47109fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f470c0c9610 0x7f470c14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.264+0000 7f47109fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0ca440 0x7f470c1f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.264+0000 7f47109fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c1eece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.264+0000 7f47109fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c1dfb20 con 0x7f470c0c9610 2023-03-03T11:00:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.264+0000 7f47109fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c1ee2f0 con 0x7f470c0ca440 2023-03-03T11:00:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.264+0000 7f47109fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c1e5d60 con 0x7f470c1564b0 2023-03-03T11:00:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f470c0c9610 0x7f470c14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:14.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f470c0c9610 0x7f470c14dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54592/0 (socket says 172.21.15.102:54592) 2023-03-03T11:00:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 -- 172.21.15.102:0/941409394 learned_addr learned my addr 172.21.15.102:0/941409394 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:00:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470a59c700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0ca440 0x7f470c1f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:14.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470b59e700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c1eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 -- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 msgr2=0x7f470c1eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c1eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 -- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0ca440 msgr2=0x7f470c1f2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:14.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0ca440 0x7f470c1f2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 -- 172.21.15.102:0/941409394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0094a0 con 0x7f470c0c9610 2023-03-03T11:00:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.265+0000 7f470ad9d700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f470c0c9610 0x7f470c14dcb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00ec10 tx=0x7f46fc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.266+0000 7f46f37fe700 1 -- 172.21.15.102:0/941409394 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc009760 con 0x7f470c0c9610 2023-03-03T11:00:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.266+0000 7f46f37fe700 1 -- 172.21.15.102:0/941409394 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc00c850 con 0x7f470c0c9610 2023-03-03T11:00:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.266+0000 7f46f37fe700 1 -- 172.21.15.102:0/941409394 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc00cbe0 con 0x7f470c0c9610 2023-03-03T11:00:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.266+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47000094a0 con 0x7f470c0c9610 2023-03-03T11:00:14.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.266+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c0b2210 con 0x7f470c0c9610 2023-03-03T11:00:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.267+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470c0b1f30 con 0x7f470c0c9610 2023-03-03T11:00:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.268+0000 7f46f37fe700 1 -- 172.21.15.102:0/941409394 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f46fc023070 con 0x7f470c0c9610 2023-03-03T11:00:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.269+0000 7f46f37fe700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46f80682f0 0x7f46f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:14.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.270+0000 7f46f37fe700 1 -- 172.21.15.102:0/941409394 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f46fc085fe0 con 0x7f470c0c9610 2023-03-03T11:00:14.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.270+0000 7f470a59c700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46f80682f0 0x7f46f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:14.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.271+0000 7f470a59c700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46f80682f0 0x7f46f806a7a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f4700000c00 tx=0x7f4700008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:14.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.274+0000 7f46f37fe700 1 -- 172.21.15.102:0/941409394 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f46fc052690 con 0x7f470c0c9610 2023-03-03T11:00:14.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.509+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f470c156880 con 0x7f470c0c9610 2023-03-03T11:00:14.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.509+0000 7f46f37fe700 1 -- 172.21.15.102:0/941409394 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f46fc055cb0 con 0x7f470c0c9610 2023-03-03T11:00:14.511 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:00:14.511 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:00:14.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46f80682f0 msgr2=0x7f46f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:14.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46f80682f0 0x7f46f806a7a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f4700000c00 tx=0x7f4700008030 comp rx=0 tx=0).stop 2023-03-03T11:00:14.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f470c0c9610 msgr2=0x7f470c14dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:14.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f470c0c9610 0x7f470c14dcb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00ec10 tx=0x7f46fc00ef20 comp rx=0 tx=0).stop 2023-03-03T11:00:14.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 shutdown_connections 2023-03-03T11:00:14.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f46f80682f0 0x7f46f806a7a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f470c1564b0 0x7f470c1eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f470c0c9610 0x7f470c14dcb0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 --2- 172.21.15.102:0/941409394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f470c0ca440 0x7f470c1f2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:14.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 >> 172.21.15.102:0/941409394 conn(0x7f470c065fb0 msgr2=0x7f470c0ca880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:00:14.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 shutdown_connections 2023-03-03T11:00:14.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:00:14.512+0000 7f47109fe700 1 -- 172.21.15.102:0/941409394 wait complete. 2023-03-03T11:00:14.516 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:00:14.541 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 61 secs before thrashing 2023-03-03T11:00:26.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:26.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:56.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:56.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:15.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:01:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.802+0000 7f95949ff700 1 Processor -- start 2023-03-03T11:01:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.803+0000 7f95949ff700 1 -- start start 2023-03-03T11:01:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.804+0000 7f95949ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f959014ffb0 0x7f95900c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.804+0000 7f95949ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.804+0000 7f95949ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95900c3c80 con 0x7f959014fbe0 2023-03-03T11:01:15.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.804+0000 7f95949ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95900c41a0 con 0x7f959014ffb0 2023-03-03T11:01:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.804+0000 7f95949ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9590071cd0 con 0x7f95900716c0 2023-03-03T11:01:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958dd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f959014ffb0 0x7f95900c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958e59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f959014fbe0 0x7f95900c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57202/0 (socket says 172.21.15.102:57202) 2023-03-03T11:01:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958ed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958e59c700 1 -- 172.21.15.102:0/3610545668 learned_addr learned my addr 172.21.15.102:0/3610545668 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:01:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958dd9b700 1 -- 172.21.15.102:0/3610545668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 msgr2=0x7f95900706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:01:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958dd9b700 1 --2- 172.21.15.102:0/3610545668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958dd9b700 1 -- 172.21.15.102:0/3610545668 >> v1:172.21.15.172:6789/0 conn(0x7f959014fbe0 legacy=0x7f95900c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958dd9b700 1 -- 172.21.15.102:0/3610545668 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900c3ec0 con 0x7f959014ffb0 2023-03-03T11:01:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.805+0000 7f958dd9b700 1 --2- 172.21.15.102:0/3610545668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 0x7f95900c3740 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f958400bae0 tx=0x7f958400bdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1df15346a55fc7e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f958d59a700 1 -- 172.21.15.102:0/3610545668 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958400dc70 con 0x7f959014ffb0 2023-03-03T11:01:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f958d59a700 1 -- 172.21.15.102:0/3610545668 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958400ddd0 con 0x7f959014ffb0 2023-03-03T11:01:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f958d59a700 1 -- 172.21.15.102:0/3610545668 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584010690 con 0x7f959014ffb0 2023-03-03T11:01:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 -- 172.21.15.102:0/3610545668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 msgr2=0x7f95900c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 --2- 172.21.15.102:0/3610545668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 0x7f95900c3740 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f958400bae0 tx=0x7f958400bdf0 comp rx=0 tx=0).stop 2023-03-03T11:01:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 -- 172.21.15.102:0/3610545668 shutdown_connections 2023-03-03T11:01:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 --2- 172.21.15.102:0/3610545668 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 --2- 172.21.15.102:0/3610545668 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 0x7f95900c3740 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 -- 172.21.15.102:0/3610545668 >> 172.21.15.102:0/3610545668 conn(0x7f959006af90 msgr2=0x7f959006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:01:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 -- 172.21.15.102:0/3610545668 shutdown_connections 2023-03-03T11:01:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.806+0000 7f95949ff700 1 -- 172.21.15.102:0/3610545668 wait complete. 2023-03-03T11:01:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.808+0000 7f95949ff700 1 Processor -- start 2023-03-03T11:01:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.809+0000 7f95949ff700 1 -- start start 2023-03-03T11:01:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f95949ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f95949ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f959014fbe0 0x7f9590155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f95949ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 0x7f95901edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f95949ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9590156430 con 0x7f959014fbe0 2023-03-03T11:01:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f95949ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9590156710 con 0x7f959014ffb0 2023-03-03T11:01:15.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f95949ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959014a180 con 0x7f95900716c0 2023-03-03T11:01:15.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f959014fbe0 0x7f9590155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:15.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958ed9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 0x7f95901edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41306/0 (socket says 172.21.15.102:41306) 2023-03-03T11:01:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 -- 172.21.15.102:0/3081513707 learned_addr learned my addr 172.21.15.102:0/3081513707 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:01:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 -- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 msgr2=0x7f95901edb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 0x7f95901edb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 -- 172.21.15.102:0/3081513707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f959014fbe0 msgr2=0x7f9590155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:15.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f959014fbe0 0x7f9590155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.810+0000 7f958e59c700 1 -- 172.21.15.102:0/3081513707 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958400b740 con 0x7f95900716c0 2023-03-03T11:01:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.811+0000 7f958e59c700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900c5e10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f958000c520 tx=0x7f958000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.811+0000 7f957effd700 1 -- 172.21.15.102:0/3081513707 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580005d70 con 0x7f95900716c0 2023-03-03T11:01:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.811+0000 7f957effd700 1 -- 172.21.15.102:0/3081513707 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95800070d0 con 0x7f95900716c0 2023-03-03T11:01:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.811+0000 7f957effd700 1 -- 172.21.15.102:0/3081513707 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580007440 con 0x7f95900716c0 2023-03-03T11:01:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.811+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9590072250 con 0x7f95900716c0 2023-03-03T11:01:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.811+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9590071f70 con 0x7f95900716c0 2023-03-03T11:01:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.812+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9590072530 con 0x7f95900716c0 2023-03-03T11:01:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.814+0000 7f957effd700 1 -- 172.21.15.102:0/3081513707 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9580019030 con 0x7f95900716c0 2023-03-03T11:01:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.815+0000 7f957effd700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95700682f0 0x7f957006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.815+0000 7f957effd700 1 -- 172.21.15.102:0/3081513707 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9580084df0 con 0x7f95900716c0 2023-03-03T11:01:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.815+0000 7f958dd9b700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95700682f0 0x7f957006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.816+0000 7f958dd9b700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95700682f0 0x7f957006a7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f958400be30 tx=0x7f958401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:15.819+0000 7f957effd700 1 -- 172.21.15.102:0/3081513707 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f958004b5e0 con 0x7f95900716c0 2023-03-03T11:01:16.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.051+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95900c3c80 con 0x7f95900716c0 2023-03-03T11:01:16.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.052+0000 7f957effd700 1 -- 172.21.15.102:0/3081513707 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f95900c3c80 con 0x7f95900716c0 2023-03-03T11:01:16.054 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:01:16.054 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:01:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95700682f0 msgr2=0x7f957006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95700682f0 0x7f957006a7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f958400be30 tx=0x7f958401c040 comp rx=0 tx=0).stop 2023-03-03T11:01:16.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 msgr2=0x7f95900c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900c5e10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f958000c520 tx=0x7f958000c830 comp rx=0 tx=0).stop 2023-03-03T11:01:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 shutdown_connections 2023-03-03T11:01:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95700682f0 0x7f957006a7a0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95900716c0 0x7f95900c5e10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f959014ffb0 0x7f95901edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 --2- 172.21.15.102:0/3081513707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f959014fbe0 0x7f9590155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:16.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 >> 172.21.15.102:0/3081513707 conn(0x7f959006af90 msgr2=0x7f95900cb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:01:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.054+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 shutdown_connections 2023-03-03T11:01:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:16.055+0000 7f95949ff700 1 -- 172.21.15.102:0/3081513707 wait complete. 2023-03-03T11:01:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:01:16.083 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-03-03T11:01:19.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:01:19.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.341+0000 7fa6b6dff700 1 Processor -- start 2023-03-03T11:01:19.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.341+0000 7fa6b6dff700 1 -- start start 2023-03-03T11:01:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6b6dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa6b00cd8e0 0x7fa6b00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6b6dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b0153660 con 0x7fa6b00cd390 2023-03-03T11:01:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6b6dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b006c360 con 0x7fa6b0153290 2023-03-03T11:01:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6b6dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b015cea0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6affff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa6b00cd8e0 0x7fa6b00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:19.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6b4b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa6b0153290 0x7fa6b0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60440/0 (socket says 172.21.15.102:60440) 2023-03-03T11:01:19.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6b4b9b700 1 -- 172.21.15.102:0/2049700039 learned_addr learned my addr 172.21.15.102:0/2049700039 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:01:19.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.343+0000 7fa6b539c700 1 --1- 172.21.15.102:0/2049700039 >> v1:172.21.15.102:6789/0 conn(0x7fa6b00cd390 0x7fa6b00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44046/0 (socket says 172.21.15.102:44046) 2023-03-03T11:01:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6af7fe700 1 -- 172.21.15.102:0/2049700039 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607667596 0 0) 0x7fa6b006c360 con 0x7fa6b0153290 2023-03-03T11:01:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6af7fe700 1 -- 172.21.15.102:0/2049700039 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a00035b0 con 0x7fa6b0153290 2023-03-03T11:01:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6af7fe700 1 -- 172.21.15.102:0/2049700039 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823727532 0 0) 0x7fa6b0153660 con 0x7fa6b00cd390 2023-03-03T11:01:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6affff700 1 -- 172.21.15.102:0/2049700039 >> v1:172.21.15.172:6789/0 conn(0x7fa6b0153290 legacy=0x7fa6b0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6affff700 1 -- 172.21.15.102:0/2049700039 >> v1:172.21.15.102:6789/0 conn(0x7fa6b00cd390 legacy=0x7fa6b00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6affff700 1 -- 172.21.15.102:0/2049700039 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b006c7f0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6affff700 1 --2- 172.21.15.102:0/2049700039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b00cde20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009840 tx=0x7fa69c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cda3adda336146d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:19.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6af7fe700 1 -- 172.21.15.102:0/2049700039 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c011830 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6af7fe700 1 -- 172.21.15.102:0/2049700039 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c011990 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.344+0000 7fa6af7fe700 1 -- 172.21.15.102:0/2049700039 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c011d20 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.345+0000 7fa6b6dff700 1 -- 172.21.15.102:0/2049700039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b00cd8e0 msgr2=0x7fa6b00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.345+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/2049700039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b00cde20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009840 tx=0x7fa69c009b50 comp rx=0 tx=0).stop 2023-03-03T11:01:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.345+0000 7fa6b6dff700 1 -- 172.21.15.102:0/2049700039 shutdown_connections 2023-03-03T11:01:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.345+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/2049700039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b00cde20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.345+0000 7fa6b6dff700 1 -- 172.21.15.102:0/2049700039 >> 172.21.15.102:0/2049700039 conn(0x7fa6b0065fb0 msgr2=0x7fa6b00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:01:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.345+0000 7fa6b6dff700 1 -- 172.21.15.102:0/2049700039 shutdown_connections 2023-03-03T11:01:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.345+0000 7fa6b6dff700 1 -- 172.21.15.102:0/2049700039 wait complete. 2023-03-03T11:01:19.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.347+0000 7fa6b6dff700 1 Processor -- start 2023-03-03T11:01:19.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.347+0000 7fa6b6dff700 1 -- start start 2023-03-03T11:01:19.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.348+0000 7fa6b6dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6b00cd390 0x7fa6b00c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:19.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.348+0000 7fa6b6dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b01eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:19.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.348+0000 7fa6b6dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b0153290 0x7fa6b01ef150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.348+0000 7fa6b6dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b01ee7f0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.348+0000 7fa6b6dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00c6370 con 0x7fa6b0153290 2023-03-03T11:01:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.348+0000 7fa6b6dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00c6610 con 0x7fa6b00cd390 2023-03-03T11:01:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b01eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b01eec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59578/0 (socket says 172.21.15.102:59578) 2023-03-03T11:01:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 -- 172.21.15.102:0/3851066667 learned_addr learned my addr 172.21.15.102:0/3851066667 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:01:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6b4b9b700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6b00cd390 0x7fa6b00c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6b539c700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b0153290 0x7fa6b01ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 -- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6b00cd390 msgr2=0x7fa6b00c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6b00cd390 0x7fa6b00c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 -- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b0153290 msgr2=0x7fa6b01ef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b0153290 0x7fa6b01ef150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.349+0000 7fa6affff700 1 -- 172.21.15.102:0/3851066667 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0094a0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.350+0000 7fa6affff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b01eec10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0090b0 tx=0x7fa69c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:19.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.350+0000 7fa6ad7fa700 1 -- 172.21.15.102:0/3851066667 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c01a7e0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.350+0000 7fa6ad7fa700 1 -- 172.21.15.102:0/3851066667 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c01a940 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.350+0000 7fa6ad7fa700 1 -- 172.21.15.102:0/3851066667 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c01acd0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.350+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b00ccde0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.350+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b015d160 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.351+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b00cd0c0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.352+0000 7fa6ad7fa700 1 -- 172.21.15.102:0/3851066667 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa69c00f050 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.353+0000 7fa6ad7fa700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa694068320 0x7fa69406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.353+0000 7fa6ad7fa700 1 -- 172.21.15.102:0/3851066667 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa69c086c20 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.354+0000 7fa6b4b9b700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa694068320 0x7fa69406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.355+0000 7fa6b4b9b700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa694068320 0x7fa69406a7d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4007fd0 tx=0x7fa6a4007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.358+0000 7fa6ad7fa700 1 -- 172.21.15.102:0/3851066667 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa69c053330 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.593+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6b004e6e0 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.594+0000 7fa6ad7fa700 1 -- 172.21.15.102:0/3851066667 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa69c029090 con 0x7fa6b00cd8e0 2023-03-03T11:01:19.597 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:01:19.597 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:01:19.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.596+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa694068320 msgr2=0x7fa69406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:19.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.596+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa694068320 0x7fa69406a7d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4007fd0 tx=0x7fa6a4007dc0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.596+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6b00cd8e0 msgr2=0x7fa6b01eec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:19.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.596+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b01eec10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0090b0 tx=0x7fa69c004750 comp rx=0 tx=0).stop 2023-03-03T11:01:19.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 shutdown_connections 2023-03-03T11:01:19.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa694068320 0x7fa69406a7d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa6b00cd390 0x7fa6b00c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6b0153290 0x7fa6b01ef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 --2- 172.21.15.102:0/3851066667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6b00cd8e0 0x7fa6b01eec10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:19.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 >> 172.21.15.102:0/3851066667 conn(0x7fa6b0065fb0 msgr2=0x7fa6b00cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:01:19.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 shutdown_connections 2023-03-03T11:01:19.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:01:19.597+0000 7fa6b6dff700 1 -- 172.21.15.102:0/3851066667 wait complete. 2023-03-03T11:01:19.601 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:01:19.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T11:01:26.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:26.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:56.496 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:56.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:26.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:02:26.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:26.723 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:26.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.905+0000 7f9a68dff700 1 Processor -- start 2023-03-03T11:02:26.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.905+0000 7f9a68dff700 1 -- start start 2023-03-03T11:02:26.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a68dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a64150c80 0x7f9a6406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:26.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a68dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a640c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:26.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a68dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a640c5d60 0x7f9a640c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:26.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a68dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a64157ad0 con 0x7f9a640c5d60 2023-03-03T11:02:26.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a68dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a640c5390 con 0x7f9a64150c80 2023-03-03T11:02:26.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a68dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640c61e0 con 0x7f9a640c2020 2023-03-03T11:02:26.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a62d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a640c5d60 0x7f9a640c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:26.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a6259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9a64150c80 0x7f9a6406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:26.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a61d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a640c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a62d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a640c5d60 0x7f9a640c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45444/0 (socket says 172.21.15.102:45444) 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.907+0000 7f9a62d9d700 1 -- 172.21.15.102:0/377184643 learned_addr learned my addr 172.21.15.102:0/377184643 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a62d9d700 1 -- 172.21.15.102:0/377184643 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 msgr2=0x7f9a640c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a62d9d700 1 --2- 172.21.15.102:0/377184643 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a640c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a62d9d700 1 -- 172.21.15.102:0/377184643 >> v2:172.21.15.172:3300/0 conn(0x7f9a64150c80 msgr2=0x7f9a6406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a62d9d700 1 --2- 172.21.15.102:0/377184643 >> v2:172.21.15.172:3300/0 conn(0x7f9a64150c80 0x7f9a6406d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a62d9d700 1 -- 172.21.15.102:0/377184643 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640c4a20 con 0x7f9a640c5d60 2023-03-03T11:02:26.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a62d9d700 1 --2- 172.21.15.102:0/377184643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a640c5d60 0x7f9a640c2720 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9a64072a90 tx=0x7f9a5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8692a681719abb84 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:26.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a6159a700 1 -- 172.21.15.102:0/377184643 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00e070 con 0x7f9a640c5d60 2023-03-03T11:02:26.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a6159a700 1 -- 172.21.15.102:0/377184643 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c00b910 con 0x7f9a640c5d60 2023-03-03T11:02:26.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.908+0000 7f9a6159a700 1 -- 172.21.15.102:0/377184643 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00bcc0 con 0x7f9a640c5d60 2023-03-03T11:02:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 -- 172.21.15.102:0/377184643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a640c5d60 msgr2=0x7f9a640c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 --2- 172.21.15.102:0/377184643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a640c5d60 0x7f9a640c2720 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9a64072a90 tx=0x7f9a5c00b330 comp rx=0 tx=0).stop 2023-03-03T11:02:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 -- 172.21.15.102:0/377184643 shutdown_connections 2023-03-03T11:02:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 --2- 172.21.15.102:0/377184643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a640c5d60 0x7f9a640c2720 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 --2- 172.21.15.102:0/377184643 >> v2:172.21.15.172:3300/0 conn(0x7f9a64150c80 0x7f9a6406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 --2- 172.21.15.102:0/377184643 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a640c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:26.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 -- 172.21.15.102:0/377184643 >> 172.21.15.102:0/377184643 conn(0x7f9a6406af90 msgr2=0x7f9a6406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:26.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 -- 172.21.15.102:0/377184643 shutdown_connections 2023-03-03T11:02:26.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.909+0000 7f9a68dff700 1 -- 172.21.15.102:0/377184643 wait complete. 2023-03-03T11:02:26.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.911+0000 7f9a68dff700 1 Processor -- start 2023-03-03T11:02:26.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.911+0000 7f9a68dff700 1 -- start start 2023-03-03T11:02:26.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.912+0000 7f9a68dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a64152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:26.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.912+0000 7f9a68dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640c5d60 0x7f9a64151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a68dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a64150c80 0x7f9a64155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a68dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64155060 con 0x7f9a64150c80 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a68dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641564e0 con 0x7f9a640c5d60 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a68dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64154aa0 con 0x7f9a640c2020 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a64152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a64152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46858/0 (socket says 172.21.15.102:46858) 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 -- 172.21.15.102:0/4177775395 learned_addr learned my addr 172.21.15.102:0/4177775395 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:02:26.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a61d9b700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640c5d60 0x7f9a64151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:26.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a62d9d700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a64150c80 0x7f9a64155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:26.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 -- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640c5d60 msgr2=0x7f9a64151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:26.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640c5d60 0x7f9a64151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:26.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 -- 172.21.15.102:0/4177775395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a64150c80 msgr2=0x7f9a64155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:26.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a64150c80 0x7f9a64155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:26.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.913+0000 7f9a6259c700 1 -- 172.21.15.102:0/4177775395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c009b30 con 0x7f9a640c2020 2023-03-03T11:02:26.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.914+0000 7f9a6259c700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a64152fa0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400e8a0 tx=0x7f9a5400ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:26.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.914+0000 7f9a52ffd700 1 -- 172.21.15.102:0/4177775395 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54009590 con 0x7f9a640c2020 2023-03-03T11:02:26.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.914+0000 7f9a52ffd700 1 -- 172.21.15.102:0/4177775395 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54013680 con 0x7f9a640c2020 2023-03-03T11:02:26.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.914+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64157510 con 0x7f9a640c2020 2023-03-03T11:02:26.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.914+0000 7f9a52ffd700 1 -- 172.21.15.102:0/4177775395 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a540139f0 con 0x7f9a640c2020 2023-03-03T11:02:26.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.914+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6406f010 con 0x7f9a640c2020 2023-03-03T11:02:26.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.915+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a641577f0 con 0x7f9a640c2020 2023-03-03T11:02:26.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.917+0000 7f9a52ffd700 1 -- 172.21.15.102:0/4177775395 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a54023070 con 0x7f9a640c2020 2023-03-03T11:02:26.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.918+0000 7f9a52ffd700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a440682f0 0x7f9a4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:26.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.918+0000 7f9a52ffd700 1 -- 172.21.15.102:0/4177775395 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9a54085f00 con 0x7f9a640c2020 2023-03-03T11:02:26.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.918+0000 7f9a61d9b700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a440682f0 0x7f9a4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:26.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.919+0000 7f9a61d9b700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a440682f0 0x7f9a4406a7a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cb570 tx=0x7f9a58009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:26.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:26.922+0000 7f9a52ffd700 1 -- 172.21.15.102:0/4177775395 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a54052520 con 0x7f9a640c2020 2023-03-03T11:02:27.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.158+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6404e6e0 con 0x7f9a640c2020 2023-03-03T11:02:27.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.159+0000 7f9a52ffd700 1 -- 172.21.15.102:0/4177775395 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9a54017090 con 0x7f9a640c2020 2023-03-03T11:02:27.161 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:02:27.162 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:02:27.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a440682f0 msgr2=0x7f9a4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:27.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a440682f0 0x7f9a4406a7a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cb570 tx=0x7f9a58009210 comp rx=0 tx=0).stop 2023-03-03T11:02:27.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 msgr2=0x7f9a64152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:27.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a64152fa0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400e8a0 tx=0x7f9a5400ebb0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 shutdown_connections 2023-03-03T11:02:27.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9a440682f0 0x7f9a4406a7a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9a640c2020 0x7f9a64152fa0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a64150c80 0x7f9a64155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 --2- 172.21.15.102:0/4177775395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a640c5d60 0x7f9a64151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 >> 172.21.15.102:0/4177775395 conn(0x7f9a6406af90 msgr2=0x7f9a640c89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:27.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 shutdown_connections 2023-03-03T11:02:27.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:02:27.163+0000 7f9a68dff700 1 -- 172.21.15.102:0/4177775395 wait complete. 2023-03-03T11:02:27.167 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:02:27.193 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T11:02:56.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:56.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:06.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:03:06.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.457+0000 7f6c967ff700 1 Processor -- start 2023-03-03T11:03:06.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.458+0000 7f6c967ff700 1 -- start start 2023-03-03T11:03:06.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c967ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c900cd8e0 0x7f6c900cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:06.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c967ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c90153660 con 0x7f6c900cd390 2023-03-03T11:03:06.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c967ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9006c360 con 0x7f6c90153290 2023-03-03T11:03:06.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c967ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9015cea0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c8f7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c900cd8e0 0x7f6c900cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:06.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c94d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6c900cd390 0x7f6c900ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46370/0 (socket says 172.21.15.102:46370) 2023-03-03T11:03:06.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c94d9c700 1 -- 172.21.15.102:0/3431659005 learned_addr learned my addr 172.21.15.102:0/3431659005 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:03:06.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.459+0000 7f6c8ffff700 1 --1- 172.21.15.102:0/3431659005 >> v1:172.21.15.172:6789/0 conn(0x7f6c90153290 0x7f6c90158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38870/0 (socket says 172.21.15.102:38870) 2023-03-03T11:03:06.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.460+0000 7f6c8f7fe700 1 -- 172.21.15.102:0/3431659005 >> v1:172.21.15.172:6789/0 conn(0x7f6c90153290 legacy=0x7f6c90158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:06.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.460+0000 7f6c8f7fe700 1 -- 172.21.15.102:0/3431659005 >> v1:172.21.15.102:6789/0 conn(0x7f6c900cd390 legacy=0x7f6c900ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:06.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.460+0000 7f6c8effd700 1 -- 172.21.15.102:0/3431659005 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837167526 0 0) 0x7f6c90153660 con 0x7f6c900cd390 2023-03-03T11:03:06.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.460+0000 7f6c8f7fe700 1 -- 172.21.15.102:0/3431659005 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9006c7f0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.460+0000 7f6c8f7fe700 1 --2- 172.21.15.102:0/3431659005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cde20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6c840097c0 tx=0x7f6c84009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce107a93ff9bbd6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:06.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.460+0000 7f6c8effd700 1 -- 172.21.15.102:0/3431659005 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84004230 con 0x7f6c900cd8e0 2023-03-03T11:03:06.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c8effd700 1 -- 172.21.15.102:0/3431659005 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c84004390 con 0x7f6c900cd8e0 2023-03-03T11:03:06.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c8effd700 1 -- 172.21.15.102:0/3431659005 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c840036a0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c967ff700 1 -- 172.21.15.102:0/3431659005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 msgr2=0x7f6c900cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:06.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c967ff700 1 --2- 172.21.15.102:0/3431659005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cde20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6c840097c0 tx=0x7f6c84009ad0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c967ff700 1 -- 172.21.15.102:0/3431659005 shutdown_connections 2023-03-03T11:03:06.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c967ff700 1 --2- 172.21.15.102:0/3431659005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cde20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c967ff700 1 -- 172.21.15.102:0/3431659005 >> 172.21.15.102:0/3431659005 conn(0x7f6c90065fb0 msgr2=0x7f6c900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:03:06.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c967ff700 1 -- 172.21.15.102:0/3431659005 shutdown_connections 2023-03-03T11:03:06.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.461+0000 7f6c967ff700 1 -- 172.21.15.102:0/3431659005 wait complete. 2023-03-03T11:03:06.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.463+0000 7f6c967ff700 1 Processor -- start 2023-03-03T11:03:06.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.463+0000 7f6c967ff700 1 -- start start 2023-03-03T11:03:06.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.464+0000 7f6c967ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900cd390 0x7f6c900c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:06.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.464+0000 7f6c967ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:06.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.464+0000 7f6c967ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c90153290 0x7f6c900c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:06.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.464+0000 7f6c967ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900c74e0 con 0x7f6c900cd390 2023-03-03T11:03:06.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.464+0000 7f6c967ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900b2c50 con 0x7f6c900cd8e0 2023-03-03T11:03:06.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.464+0000 7f6c967ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900c8570 con 0x7f6c90153290 2023-03-03T11:03:06.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900cd390 0x7f6c900c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:06.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:06.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47088/0 (socket says 172.21.15.102:47088) 2023-03-03T11:03:06.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 -- 172.21.15.102:0/1810188789 learned_addr learned my addr 172.21.15.102:0/1810188789 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:03:06.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c94d9c700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c90153290 0x7f6c900c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:06.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 -- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c90153290 msgr2=0x7f6c900c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:06.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c90153290 0x7f6c900c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 -- 172.21.15.102:0/1810188789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900cd390 msgr2=0x7f6c900c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:06.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900cd390 0x7f6c900c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8f7fe700 1 -- 172.21.15.102:0/1810188789 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840094a0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.465+0000 7f6c8ffff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900cd390 0x7f6c900c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:03:06.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.466+0000 7f6c8f7fe700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cf0b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6c840090b0 tx=0x7f6c84004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:06.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.466+0000 7f6c8cff9700 1 -- 172.21.15.102:0/1810188789 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84011630 con 0x7f6c900cd8e0 2023-03-03T11:03:06.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.466+0000 7f6c8cff9700 1 -- 172.21.15.102:0/1810188789 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8401f070 con 0x7f6c900cd8e0 2023-03-03T11:03:06.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.466+0000 7f6c8cff9700 1 -- 172.21.15.102:0/1810188789 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84011a10 con 0x7f6c900cd8e0 2023-03-03T11:03:06.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.466+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c900ccde0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.466+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9015d160 con 0x7f6c900cd8e0 2023-03-03T11:03:06.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.467+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c900cd0c0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.468+0000 7f6c8cff9700 1 -- 172.21.15.102:0/1810188789 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c84017020 con 0x7f6c900cd8e0 2023-03-03T11:03:06.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.470+0000 7f6c8cff9700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c7c068320 0x7f6c7c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:06.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.470+0000 7f6c8cff9700 1 -- 172.21.15.102:0/1810188789 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6c84021070 con 0x7f6c900cd8e0 2023-03-03T11:03:06.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.470+0000 7f6c8ffff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c7c068320 0x7f6c7c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:06.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.471+0000 7f6c8ffff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c7c068320 0x7f6c7c06a7d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6c80007c20 tx=0x7f6c80007b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:06.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.474+0000 7f6c8cff9700 1 -- 172.21.15.102:0/1810188789 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c84053880 con 0x7f6c900cd8e0 2023-03-03T11:03:06.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.706+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c9004e6e0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.707+0000 7f6c8cff9700 1 -- 172.21.15.102:0/1810188789 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6c84056ea0 con 0x7f6c900cd8e0 2023-03-03T11:03:06.710 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:03:06.711 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:03:06.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c7c068320 msgr2=0x7f6c7c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:06.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c7c068320 0x7f6c7c06a7d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6c80007c20 tx=0x7f6c80007b70 comp rx=0 tx=0).stop 2023-03-03T11:03:06.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 msgr2=0x7f6c900cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:06.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cf0b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6c840090b0 tx=0x7f6c84004750 comp rx=0 tx=0).stop 2023-03-03T11:03:06.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 shutdown_connections 2023-03-03T11:03:06.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c7c068320 0x7f6c7c06a7d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c90153290 0x7f6c900c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900cd390 0x7f6c900c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 --2- 172.21.15.102:0/1810188789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c900cd8e0 0x7f6c900cf0b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:06.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 >> 172.21.15.102:0/1810188789 conn(0x7f6c90065fb0 msgr2=0x7f6c900cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:03:06.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 shutdown_connections 2023-03-03T11:03:06.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:06.710+0000 7f6c967ff700 1 -- 172.21.15.102:0/1810188789 wait complete. 2023-03-03T11:03:06.715 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:03:06.739 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-03-03T11:03:27.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:27.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:42.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:03:43.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.006+0000 7f8973fff700 1 Processor -- start 2023-03-03T11:03:43.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.007+0000 7f8973fff700 1 -- start start 2023-03-03T11:03:43.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f8973fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f89741564b0 0x7f89740cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:43.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f8973fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f89740c9610 0x7f89740cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:43.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f8973fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89740c9ca0 con 0x7f89740ca440 2023-03-03T11:03:43.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f8973fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89740b4920 con 0x7f89740c9610 2023-03-03T11:03:43.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f8973fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89740b4c00 con 0x7f89741564b0 2023-03-03T11:03:43.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f89727fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f89740c9610 0x7f89740cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:43.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f89737fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f89740ca440 0x7f89740ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34898/0 (socket says 172.21.15.102:34898) 2023-03-03T11:03:43.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f89737fe700 1 -- 172.21.15.102:0/1515453451 learned_addr learned my addr 172.21.15.102:0/1515453451 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:03:43.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.008+0000 7f8972ffd700 1 --2- 172.21.15.102:0/1515453451 >> v2:172.21.15.172:3300/0 conn(0x7f89741564b0 0x7f89740cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:43.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.009+0000 7f89727fc700 1 -- 172.21.15.102:0/1515453451 >> v2:172.21.15.172:3300/0 conn(0x7f89741564b0 msgr2=0x7f89740cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:03:43.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.009+0000 7f89727fc700 1 --2- 172.21.15.102:0/1515453451 >> v2:172.21.15.172:3300/0 conn(0x7f89741564b0 0x7f89740cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.009+0000 7f89727fc700 1 -- 172.21.15.102:0/1515453451 >> v1:172.21.15.102:6789/0 conn(0x7f89740ca440 legacy=0x7f89740ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:43.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.009+0000 7f89727fc700 1 -- 172.21.15.102:0/1515453451 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89740b46f0 con 0x7f89740c9610 2023-03-03T11:03:43.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.009+0000 7f89727fc700 1 --2- 172.21.15.102:0/1515453451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740cbba0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f897408c600 tx=0x7f8968009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e661b2309676399 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:43.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.009+0000 7f8971ffb700 1 -- 172.21.15.102:0/1515453451 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968011830 con 0x7f89740c9610 2023-03-03T11:03:43.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8971ffb700 1 -- 172.21.15.102:0/1515453451 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8968011990 con 0x7f89740c9610 2023-03-03T11:03:43.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8971ffb700 1 -- 172.21.15.102:0/1515453451 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968011d20 con 0x7f89740c9610 2023-03-03T11:03:43.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 -- 172.21.15.102:0/1515453451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 msgr2=0x7f89740cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:43.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 --2- 172.21.15.102:0/1515453451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740cbba0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f897408c600 tx=0x7f8968009b20 comp rx=0 tx=0).stop 2023-03-03T11:03:43.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 -- 172.21.15.102:0/1515453451 shutdown_connections 2023-03-03T11:03:43.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 --2- 172.21.15.102:0/1515453451 >> v2:172.21.15.172:3300/0 conn(0x7f89741564b0 0x7f89740cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 --2- 172.21.15.102:0/1515453451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740cbba0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 -- 172.21.15.102:0/1515453451 >> 172.21.15.102:0/1515453451 conn(0x7f8974065fb0 msgr2=0x7f89740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:03:43.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 -- 172.21.15.102:0/1515453451 shutdown_connections 2023-03-03T11:03:43.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.010+0000 7f8973fff700 1 -- 172.21.15.102:0/1515453451 wait complete. 2023-03-03T11:03:43.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.012+0000 7f8973fff700 1 Processor -- start 2023-03-03T11:03:43.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.013+0000 7f8973fff700 1 -- start start 2023-03-03T11:03:43.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8973fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:43.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8973fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89740ca440 0x7f89740c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:43.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8973fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89741564b0 0x7f89741ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:43.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8973fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897415bac0 con 0x7f89740c9610 2023-03-03T11:03:43.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8973fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89741edf40 con 0x7f89740ca440 2023-03-03T11:03:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8973fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89741f1960 con 0x7f89741564b0 2023-03-03T11:03:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45894/0 (socket says 172.21.15.102:45894) 2023-03-03T11:03:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 -- 172.21.15.102:0/1723938585 learned_addr learned my addr 172.21.15.102:0/1723938585 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:03:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f89727fc700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89740ca440 0x7f89740c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:43.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f89737fe700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89741564b0 0x7f89741ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:43.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 -- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89741564b0 msgr2=0x7f89741ee720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89741564b0 0x7f89741ee720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 -- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89740ca440 msgr2=0x7f89740c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89740ca440 0x7f89740c7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.014+0000 7f8972ffd700 1 -- 172.21.15.102:0/1723938585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680094a0 con 0x7f89740c9610 2023-03-03T11:03:43.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.015+0000 7f8972ffd700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740c71f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f896400e870 tx=0x7f896400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.015+0000 7f895bfff700 1 -- 172.21.15.102:0/1723938585 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964009590 con 0x7f89740c9610 2023-03-03T11:03:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.015+0000 7f895bfff700 1 -- 172.21.15.102:0/1723938585 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8964013680 con 0x7f89740c9610 2023-03-03T11:03:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.015+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89740b2210 con 0x7f89740c9610 2023-03-03T11:03:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.015+0000 7f895bfff700 1 -- 172.21.15.102:0/1723938585 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89640139f0 con 0x7f89740c9610 2023-03-03T11:03:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.015+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89740b1f30 con 0x7f89740c9610 2023-03-03T11:03:43.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.016+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8974156880 con 0x7f89740c9610 2023-03-03T11:03:43.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.017+0000 7f895bfff700 1 -- 172.21.15.102:0/1723938585 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8964019030 con 0x7f89740c9610 2023-03-03T11:03:43.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.019+0000 7f895bfff700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89600682f0 0x7f896006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:43.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.019+0000 7f895bfff700 1 -- 172.21.15.102:0/1723938585 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8964085bf0 con 0x7f89740c9610 2023-03-03T11:03:43.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.019+0000 7f89727fc700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89600682f0 0x7f896006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:43.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.020+0000 7f89727fc700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89600682f0 0x7f896006a7a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f89740c9a30 tx=0x7f8968008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:43.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.023+0000 7f895bfff700 1 -- 172.21.15.102:0/1723938585 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8964052270 con 0x7f89740c9610 2023-03-03T11:03:43.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.260+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89740677f0 con 0x7f89740c9610 2023-03-03T11:03:43.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.260+0000 7f895bfff700 1 -- 172.21.15.102:0/1723938585 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8964021090 con 0x7f89740c9610 2023-03-03T11:03:43.264 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:03:43.264 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:03:43.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89600682f0 msgr2=0x7f896006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:43.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89600682f0 0x7f896006a7a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f89740c9a30 tx=0x7f8968008030 comp rx=0 tx=0).stop 2023-03-03T11:03:43.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 msgr2=0x7f89740c71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:43.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740c71f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f896400e870 tx=0x7f896400eb80 comp rx=0 tx=0).stop 2023-03-03T11:03:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 shutdown_connections 2023-03-03T11:03:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f89600682f0 0x7f896006a7a0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f89741564b0 0x7f89741ee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89740c9610 0x7f89740c71f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 --2- 172.21.15.102:0/1723938585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89740ca440 0x7f89740c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 >> 172.21.15.102:0/1723938585 conn(0x7f8974065fb0 msgr2=0x7f8974155500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:03:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 shutdown_connections 2023-03-03T11:03:43.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:03:43.263+0000 7f8973fff700 1 -- 172.21.15.102:0/1723938585 wait complete. 2023-03-03T11:03:43.268 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:03:43.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T11:03:57.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:57.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:27.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:27.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:35.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:04:35.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.557+0000 7f1484dff700 1 Processor -- start 2023-03-03T11:04:35.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.557+0000 7f1484dff700 1 -- start start 2023-03-03T11:04:35.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f1484dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f14800cd8e0 0x7f14800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:35.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f1484dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14800cd390 0x7f14800ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:35.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f1484dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1480153660 con 0x7f1480153290 2023-03-03T11:04:35.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f1484dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148006c360 con 0x7f14800cd390 2023-03-03T11:04:35.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f1484dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148015cea0 con 0x7f14800cd8e0 2023-03-03T11:04:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f147ed9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14800cd390 0x7f14800ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f147dd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f14800cd8e0 0x7f14800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f147ed9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14800cd390 0x7f14800ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37820/0 (socket says 172.21.15.102:37820) 2023-03-03T11:04:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f147e59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1480153290 0x7f1480158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47314/0 (socket says 172.21.15.102:47314) 2023-03-03T11:04:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.559+0000 7f147ed9d700 1 -- 172.21.15.102:0/696788658 learned_addr learned my addr 172.21.15.102:0/696788658 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:04:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147ed9d700 1 -- 172.21.15.102:0/696788658 >> v2:172.21.15.172:3300/0 conn(0x7f14800cd8e0 msgr2=0x7f14800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147ed9d700 1 --2- 172.21.15.102:0/696788658 >> v2:172.21.15.172:3300/0 conn(0x7f14800cd8e0 0x7f14800cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147d59a700 1 -- 172.21.15.102:0/696788658 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310349827 0 0) 0x7f1480153660 con 0x7f1480153290 2023-03-03T11:04:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147ed9d700 1 -- 172.21.15.102:0/696788658 >> v1:172.21.15.172:6789/0 conn(0x7f1480153290 legacy=0x7f1480158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147ed9d700 1 -- 172.21.15.102:0/696788658 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148006c7f0 con 0x7f14800cd390 2023-03-03T11:04:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147ed9d700 1 --2- 172.21.15.102:0/696788658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14800cd390 0x7f14800ce360 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1480159fb0 tx=0x7f147800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8386295fac301b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147d59a700 1 -- 172.21.15.102:0/696788658 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800e070 con 0x7f14800cd390 2023-03-03T11:04:35.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147d59a700 1 -- 172.21.15.102:0/696788658 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147800b910 con 0x7f14800cd390 2023-03-03T11:04:35.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.560+0000 7f147d59a700 1 -- 172.21.15.102:0/696788658 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800bcc0 con 0x7f14800cd390 2023-03-03T11:04:35.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 -- 172.21.15.102:0/696788658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14800cd390 msgr2=0x7f14800ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:35.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 --2- 172.21.15.102:0/696788658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14800cd390 0x7f14800ce360 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1480159fb0 tx=0x7f147800b330 comp rx=0 tx=0).stop 2023-03-03T11:04:35.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 -- 172.21.15.102:0/696788658 shutdown_connections 2023-03-03T11:04:35.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 --2- 172.21.15.102:0/696788658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14800cd390 0x7f14800ce360 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 --2- 172.21.15.102:0/696788658 >> v2:172.21.15.172:3300/0 conn(0x7f14800cd8e0 0x7f14800cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 -- 172.21.15.102:0/696788658 >> 172.21.15.102:0/696788658 conn(0x7f1480065fb0 msgr2=0x7f14800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:35.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 -- 172.21.15.102:0/696788658 shutdown_connections 2023-03-03T11:04:35.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.561+0000 7f1484dff700 1 -- 172.21.15.102:0/696788658 wait complete. 2023-03-03T11:04:35.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.563+0000 7f1484dff700 1 Processor -- start 2023-03-03T11:04:35.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.563+0000 7f1484dff700 1 -- start start 2023-03-03T11:04:35.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.564+0000 7f1484dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14800cd390 0x7f1480150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:35.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.564+0000 7f1484dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14800cd8e0 0x7f1480157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:35.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.564+0000 7f1484dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1480153290 0x7f14801efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:35.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.564+0000 7f1484dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14800cfd20 con 0x7f1480153290 2023-03-03T11:04:35.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.564+0000 7f1484dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14801585b0 con 0x7f14800cd8e0 2023-03-03T11:04:35.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.564+0000 7f1484dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14801582d0 con 0x7f14800cd390 2023-03-03T11:04:35.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14800cd8e0 0x7f1480157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:35.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14800cd390 0x7f1480150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:35.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14800cd8e0 0x7f1480157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36638/0 (socket says 172.21.15.102:36638) 2023-03-03T11:04:35.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147dd9b700 1 -- 172.21.15.102:0/1403213336 learned_addr learned my addr 172.21.15.102:0/1403213336 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147ed9d700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1480153290 0x7f14801efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147e59c700 1 -- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14800cd8e0 msgr2=0x7f1480157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147e59c700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14800cd8e0 0x7f1480157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147e59c700 1 -- 172.21.15.102:0/1403213336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1480153290 msgr2=0x7f14801efaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147e59c700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1480153290 0x7f14801efaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147e59c700 1 -- 172.21.15.102:0/1403213336 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478009b30 con 0x7f14800cd390 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147dd9b700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14800cd8e0 0x7f1480157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:04:35.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.565+0000 7f147e59c700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14800cd390 0x7f1480150780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f147000c410 tx=0x7f147000c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:35.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.566+0000 7f146effd700 1 -- 172.21.15.102:0/1403213336 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470004c80 con 0x7f14800cd390 2023-03-03T11:04:35.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.566+0000 7f146effd700 1 -- 172.21.15.102:0/1403213336 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1470004de0 con 0x7f14800cd390 2023-03-03T11:04:35.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.566+0000 7f146effd700 1 -- 172.21.15.102:0/1403213336 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470007310 con 0x7f14800cd390 2023-03-03T11:04:35.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.566+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14800ccde0 con 0x7f14800cd390 2023-03-03T11:04:35.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.566+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148015d160 con 0x7f14800cd390 2023-03-03T11:04:35.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.567+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14800cd0c0 con 0x7f14800cd390 2023-03-03T11:04:35.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.568+0000 7f146effd700 1 -- 172.21.15.102:0/1403213336 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f147001a020 con 0x7f14800cd390 2023-03-03T11:04:35.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.569+0000 7f146effd700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14600682f0 0x7f146006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:35.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.570+0000 7f146effd700 1 -- 172.21.15.102:0/1403213336 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1470085760 con 0x7f14800cd390 2023-03-03T11:04:35.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.570+0000 7f147dd9b700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14600682f0 0x7f146006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:35.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.570+0000 7f147dd9b700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14600682f0 0x7f146006a7a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f14801f2b00 tx=0x7f1474009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:35.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.574+0000 7f146effd700 1 -- 172.21.15.102:0/1403213336 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1470051e70 con 0x7f14800cd390 2023-03-03T11:04:35.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.808+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14801ef940 con 0x7f14800cd390 2023-03-03T11:04:35.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.809+0000 7f146effd700 1 -- 172.21.15.102:0/1403213336 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1470055490 con 0x7f14800cd390 2023-03-03T11:04:35.812 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:04:35.812 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:04:35.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14600682f0 msgr2=0x7f146006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:35.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14600682f0 0x7f146006a7a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f14801f2b00 tx=0x7f1474009210 comp rx=0 tx=0).stop 2023-03-03T11:04:35.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14800cd390 msgr2=0x7f1480150780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14800cd390 0x7f1480150780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f147000c410 tx=0x7f147000c720 comp rx=0 tx=0).stop 2023-03-03T11:04:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 shutdown_connections 2023-03-03T11:04:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14600682f0 0x7f146006a7a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14800cd390 0x7f1480150780 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1480153290 0x7f14801efaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 --2- 172.21.15.102:0/1403213336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14800cd8e0 0x7f1480157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 >> 172.21.15.102:0/1403213336 conn(0x7f1480065fb0 msgr2=0x7f1480150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.811+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 shutdown_connections 2023-03-03T11:04:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:04:35.812+0000 7f1484dff700 1 -- 172.21.15.102:0/1403213336 wait complete. 2023-03-03T11:04:35.816 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:04:35.841 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-03-03T11:04:57.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:57.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:27.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:27.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:52.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:05:53.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.104+0000 7fdcbc5ff700 1 Processor -- start 2023-03-03T11:05:53.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.105+0000 7fdcbc5ff700 1 -- start start 2023-03-03T11:05:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcbc5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcb40cd8e0 0x7fdcb40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcbc5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd390 0x7fdcb40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcbc5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb4153660 con 0x7fdcb4153290 2023-03-03T11:05:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcbc5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb406c360 con 0x7fdcb40cd8e0 2023-03-03T11:05:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcbc5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb415cea0 con 0x7fdcb40cd390 2023-03-03T11:05:53.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcba39b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdcb4153290 0x7fdcb4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39330/0 (socket says 172.21.15.102:39330) 2023-03-03T11:05:53.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcba39b700 1 -- 172.21.15.102:0/274968387 learned_addr learned my addr 172.21.15.102:0/274968387 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:05:53.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcb9b9a700 1 --2- 172.21.15.102:0/274968387 >> v2:172.21.15.172:3300/0 conn(0x7fdcb40cd8e0 0x7fdcb40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.106+0000 7fdcbab9c700 1 --2- 172.21.15.102:0/274968387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd390 0x7fdcb40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.107+0000 7fdcb9399700 1 -- 172.21.15.102:0/274968387 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003029029 0 0) 0x7fdcb4153660 con 0x7fdcb4153290 2023-03-03T11:05:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.107+0000 7fdcbab9c700 1 -- 172.21.15.102:0/274968387 >> v2:172.21.15.172:3300/0 conn(0x7fdcb40cd8e0 msgr2=0x7fdcb40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.107+0000 7fdcbab9c700 1 --2- 172.21.15.102:0/274968387 >> v2:172.21.15.172:3300/0 conn(0x7fdcb40cd8e0 0x7fdcb40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.107+0000 7fdcbab9c700 1 -- 172.21.15.102:0/274968387 >> v1:172.21.15.102:6789/0 conn(0x7fdcb4153290 legacy=0x7fdcb4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.107+0000 7fdcbab9c700 1 -- 172.21.15.102:0/274968387 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb406c7f0 con 0x7fdcb40cd390 2023-03-03T11:05:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.107+0000 7fdcbab9c700 1 --2- 172.21.15.102:0/274968387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd390 0x7fdcb40ce360 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009d90 tx=0x7fdcac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=403091036841464 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcb9399700 1 -- 172.21.15.102:0/274968387 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00e070 con 0x7fdcb40cd390 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcb9399700 1 -- 172.21.15.102:0/274968387 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac00bb20 con 0x7fdcb40cd390 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcb9399700 1 -- 172.21.15.102:0/274968387 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac012430 con 0x7fdcb40cd390 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/274968387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd390 msgr2=0x7fdcb40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/274968387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd390 0x7fdcb40ce360 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009d90 tx=0x7fdcac00b330 comp rx=0 tx=0).stop 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/274968387 shutdown_connections 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/274968387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd390 0x7fdcb40ce360 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/274968387 >> v2:172.21.15.172:3300/0 conn(0x7fdcb40cd8e0 0x7fdcb40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/274968387 >> 172.21.15.102:0/274968387 conn(0x7fdcb4065fb0 msgr2=0x7fdcb40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/274968387 shutdown_connections 2023-03-03T11:05:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.108+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/274968387 wait complete. 2023-03-03T11:05:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.111+0000 7fdcbc5ff700 1 Processor -- start 2023-03-03T11:05:53.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.112+0000 7fdcbc5ff700 1 -- start start 2023-03-03T11:05:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcbc5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcb40cd390 0x7fdcb40c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcbc5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd8e0 0x7fdcb414e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcbc5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcb4153290 0x7fdcb4156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcbc5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4155ef0 con 0x7fdcb40cd390 2023-03-03T11:05:53.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcbc5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40cfc60 con 0x7fdcb4153290 2023-03-03T11:05:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcbc5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40b2160 con 0x7fdcb40cd8e0 2023-03-03T11:05:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcba39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcb40cd390 0x7fdcb40c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcba39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcb40cd390 0x7fdcb40c7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45148/0 (socket says 172.21.15.102:45148) 2023-03-03T11:05:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcba39b700 1 -- 172.21.15.102:0/303596089 learned_addr learned my addr 172.21.15.102:0/303596089 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:05:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcb9b9a700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd8e0 0x7fdcb414e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.113+0000 7fdcbab9c700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcb4153290 0x7fdcb4156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcba39b700 1 -- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd8e0 msgr2=0x7fdcb414e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcba39b700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd8e0 0x7fdcb414e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcba39b700 1 -- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcb4153290 msgr2=0x7fdcb4156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcba39b700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcb4153290 0x7fdcb4156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcba39b700 1 -- 172.21.15.102:0/303596089 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac009a70 con 0x7fdcb40cd390 2023-03-03T11:05:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcba39b700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcb40cd390 0x7fdcb40c7190 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00011e0 tx=0x7fdcb000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcaaffd700 1 -- 172.21.15.102:0/303596089 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0004c80 con 0x7fdcb40cd390 2023-03-03T11:05:53.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcaaffd700 1 -- 172.21.15.102:0/303596089 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb0004e00 con 0x7fdcb40cd390 2023-03-03T11:05:53.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcaaffd700 1 -- 172.21.15.102:0/303596089 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0007310 con 0x7fdcb40cd390 2023-03-03T11:05:53.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.114+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb40ccde0 con 0x7fdcb40cd390 2023-03-03T11:05:53.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.115+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb415d160 con 0x7fdcb40cd390 2023-03-03T11:05:53.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.116+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb40cd0c0 con 0x7fdcb40cd390 2023-03-03T11:05:53.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.117+0000 7fdcaaffd700 1 -- 172.21.15.102:0/303596089 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdcb001a020 con 0x7fdcb40cd390 2023-03-03T11:05:53.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.118+0000 7fdcaaffd700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc980682f0 0x7fdc9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.118+0000 7fdcaaffd700 1 -- 172.21.15.102:0/303596089 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdcb0085950 con 0x7fdcb40cd390 2023-03-03T11:05:53.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.118+0000 7fdcb9b9a700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc980682f0 0x7fdc9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.119+0000 7fdcb9b9a700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc980682f0 0x7fdc9806a7a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdcb41e8760 tx=0x7fdca4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.122+0000 7fdcaaffd700 1 -- 172.21.15.102:0/303596089 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdcb0051fd0 con 0x7fdcb40cd390 2023-03-03T11:05:53.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.358+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcb41561d0 con 0x7fdcb40cd390 2023-03-03T11:05:53.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.358+0000 7fdcaaffd700 1 -- 172.21.15.102:0/303596089 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdcb00555f0 con 0x7fdcb40cd390 2023-03-03T11:05:53.361 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:05:53.361 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:05:53.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc980682f0 msgr2=0x7fdc9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc980682f0 0x7fdc9806a7a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdcb41e8760 tx=0x7fdca4009210 comp rx=0 tx=0).stop 2023-03-03T11:05:53.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcb40cd390 msgr2=0x7fdcb40c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcb40cd390 0x7fdcb40c7190 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00011e0 tx=0x7fdcb000ca30 comp rx=0 tx=0).stop 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 shutdown_connections 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcb4153290 0x7fdcb4156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcb40cd390 0x7fdcb40c7190 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc980682f0 0x7fdc9806a7a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 --2- 172.21.15.102:0/303596089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcb40cd8e0 0x7fdcb414e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 >> 172.21.15.102:0/303596089 conn(0x7fdcb4065fb0 msgr2=0x7fdcb41586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 shutdown_connections 2023-03-03T11:05:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:05:53.361+0000 7fdcbc5ff700 1 -- 172.21.15.102:0/303596089 wait complete. 2023-03-03T11:05:53.365 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:05:53.390 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T11:05:57.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:57.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:28.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:28.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:58.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:58.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:00.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:07:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.651+0000 7ff0773ff700 1 Processor -- start 2023-03-03T11:07:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.651+0000 7ff0773ff700 1 -- start start 2023-03-03T11:07:00.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.652+0000 7ff0773ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff070153290 0x7ff070158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:00.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff0773ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0700cd8e0 0x7ff0700ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:00.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff0773ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff070153660 con 0x7ff0701593c0 2023-03-03T11:07:00.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff0773ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07006c360 con 0x7ff0700cd8e0 2023-03-03T11:07:00.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff0773ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07015cea0 con 0x7ff070153290 2023-03-03T11:07:00.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff07599c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0700cd8e0 0x7ff0700ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:00.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff07519b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff070153290 0x7ff070158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:00.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff07599c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0700cd8e0 0x7ff0700ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36316/0 (socket says 172.21.15.102:36316) 2023-03-03T11:07:00.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.653+0000 7ff07599c700 1 -- 172.21.15.102:0/2825261380 learned_addr learned my addr 172.21.15.102:0/2825261380 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:07:00.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff07599c700 1 -- 172.21.15.102:0/2825261380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff070153290 msgr2=0x7ff070158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff07599c700 1 --2- 172.21.15.102:0/2825261380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff070153290 0x7ff070158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff07599c700 1 -- 172.21.15.102:0/2825261380 >> v1:172.21.15.102:6789/0 conn(0x7ff0701593c0 legacy=0x7ff0700cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff07599c700 1 -- 172.21.15.102:0/2825261380 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07006c7f0 con 0x7ff0700cd8e0 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff07519b700 1 --2- 172.21.15.102:0/2825261380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff070153290 0x7ff070158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff07599c700 1 --2- 172.21.15.102:0/2825261380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0700cd8e0 0x7ff0700ce530 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff068009e50 tx=0x7ff06800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116f81ef19b92f9f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff067fff700 1 -- 172.21.15.102:0/2825261380 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800e070 con 0x7ff0700cd8e0 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff067fff700 1 -- 172.21.15.102:0/2825261380 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06800b910 con 0x7ff0700cd8e0 2023-03-03T11:07:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.654+0000 7ff067fff700 1 -- 172.21.15.102:0/2825261380 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800bcc0 con 0x7ff0700cd8e0 2023-03-03T11:07:00.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 -- 172.21.15.102:0/2825261380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0700cd8e0 msgr2=0x7ff0700ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:00.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2825261380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0700cd8e0 0x7ff0700ce530 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff068009e50 tx=0x7ff06800b330 comp rx=0 tx=0).stop 2023-03-03T11:07:00.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 -- 172.21.15.102:0/2825261380 shutdown_connections 2023-03-03T11:07:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2825261380 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff070153290 0x7ff070158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2825261380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0700cd8e0 0x7ff0700ce530 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 -- 172.21.15.102:0/2825261380 >> 172.21.15.102:0/2825261380 conn(0x7ff070065fb0 msgr2=0x7ff0700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 -- 172.21.15.102:0/2825261380 shutdown_connections 2023-03-03T11:07:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.655+0000 7ff0773ff700 1 -- 172.21.15.102:0/2825261380 wait complete. 2023-03-03T11:07:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.657+0000 7ff0773ff700 1 Processor -- start 2023-03-03T11:07:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.657+0000 7ff0773ff700 1 -- start start 2023-03-03T11:07:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.658+0000 7ff0773ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0700cd8e0 0x7ff0700c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.658+0000 7ff0773ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff070153290 0x7ff0700cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.658+0000 7ff0773ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0701593c0 0x7ff0701f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.658+0000 7ff0773ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07014edb0 con 0x7ff0701593c0 2023-03-03T11:07:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.658+0000 7ff0773ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700c4f00 con 0x7ff070153290 2023-03-03T11:07:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.658+0000 7ff0773ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700c2c50 con 0x7ff0700cd8e0 2023-03-03T11:07:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0701593c0 0x7ff0701f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0701593c0 0x7ff0701f2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36322/0 (socket says 172.21.15.102:36322) 2023-03-03T11:07:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 -- 172.21.15.102:0/2421835302 learned_addr learned my addr 172.21.15.102:0/2421835302 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:07:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07519b700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0700cd8e0 0x7ff0700c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07499a700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff070153290 0x7ff0700cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 -- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0700cd8e0 msgr2=0x7ff0700c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0700cd8e0 0x7ff0700c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 -- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff070153290 msgr2=0x7ff0700cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff070153290 0x7ff0700cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 -- 172.21.15.102:0/2421835302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c0094a0 con 0x7ff0701593c0 2023-03-03T11:07:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.659+0000 7ff07599c700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0701593c0 0x7ff0701f2050 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff06800b680 tx=0x7ff068004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.660+0000 7ff065ffb700 1 -- 172.21.15.102:0/2421835302 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068004830 con 0x7ff0701593c0 2023-03-03T11:07:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.660+0000 7ff065ffb700 1 -- 172.21.15.102:0/2421835302 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0680099b0 con 0x7ff0701593c0 2023-03-03T11:07:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.660+0000 7ff065ffb700 1 -- 172.21.15.102:0/2421835302 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06801c680 con 0x7ff0701593c0 2023-03-03T11:07:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.660+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff068009b30 con 0x7ff0701593c0 2023-03-03T11:07:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.660+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0700ccde0 con 0x7ff0701593c0 2023-03-03T11:07:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.661+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07015d160 con 0x7ff0701593c0 2023-03-03T11:07:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.662+0000 7ff065ffb700 1 -- 172.21.15.102:0/2421835302 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff06802c070 con 0x7ff0701593c0 2023-03-03T11:07:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.664+0000 7ff065ffb700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0540682f0 0x7ff05406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.664+0000 7ff065ffb700 1 -- 172.21.15.102:0/2421835302 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff0680877b0 con 0x7ff0701593c0 2023-03-03T11:07:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.664+0000 7ff07519b700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0540682f0 0x7ff05406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.665+0000 7ff07519b700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0540682f0 0x7ff05406a7a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009d70 tx=0x7ff06c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.668+0000 7ff065ffb700 1 -- 172.21.15.102:0/2421835302 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff068053ec0 con 0x7ff0701593c0 2023-03-03T11:07:00.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.900+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0700cd0c0 con 0x7ff0701593c0 2023-03-03T11:07:00.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.901+0000 7ff065ffb700 1 -- 172.21.15.102:0/2421835302 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff0680574e0 con 0x7ff0701593c0 2023-03-03T11:07:00.903 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:07:00.904 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:07:00.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.903+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0540682f0 msgr2=0x7ff05406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:00.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.903+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0540682f0 0x7ff05406a7a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009d70 tx=0x7ff06c009210 comp rx=0 tx=0).stop 2023-03-03T11:07:00.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.903+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0701593c0 msgr2=0x7ff0701f2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:00.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.903+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0701593c0 0x7ff0701f2050 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff06800b680 tx=0x7ff068004ad0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 shutdown_connections 2023-03-03T11:07:00.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0540682f0 0x7ff05406a7a0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff0700cd8e0 0x7ff0700c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0701593c0 0x7ff0701f2050 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 --2- 172.21.15.102:0/2421835302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff070153290 0x7ff0700cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:00.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 >> 172.21.15.102:0/2421835302 conn(0x7ff070065fb0 msgr2=0x7ff070073920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:00.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 shutdown_connections 2023-03-03T11:07:00.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:07:00.904+0000 7ff0773ff700 1 -- 172.21.15.102:0/2421835302 wait complete. 2023-03-03T11:07:00.908 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:07:00.933 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T11:07:28.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:28.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:58.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:58.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:29.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:29.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:54.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:08:55.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.183+0000 7f1adbdff700 1 Processor -- start 2023-03-03T11:08:55.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.183+0000 7f1adbdff700 1 -- start start 2023-03-03T11:08:55.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1adbdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ad40cd8e0 0x7f1ad40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:55.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1adbdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:55.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1adbdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad4153660 con 0x7f1ad4153290 2023-03-03T11:08:55.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1adbdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad406c360 con 0x7f1ad40cd8e0 2023-03-03T11:08:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1adbdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad415cea0 con 0x7f1ad40cd390 2023-03-03T11:08:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1ad939a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ad40cd8e0 0x7f1ad40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1ad9b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1ad4153290 0x7f1ad4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60118/0 (socket says 172.21.15.102:60118) 2023-03-03T11:08:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1ad9b9b700 1 -- 172.21.15.102:0/629139849 learned_addr learned my addr 172.21.15.102:0/629139849 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:08:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.184+0000 7f1ada39c700 1 --2- 172.21.15.102:0/629139849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:55.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.185+0000 7f1ad939a700 1 -- 172.21.15.102:0/629139849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 msgr2=0x7f1ad40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:08:55.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.185+0000 7f1ad939a700 1 --2- 172.21.15.102:0/629139849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.185+0000 7f1ad939a700 1 -- 172.21.15.102:0/629139849 >> v1:172.21.15.172:6789/0 conn(0x7f1ad4153290 legacy=0x7f1ad4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.185+0000 7f1ad939a700 1 -- 172.21.15.102:0/629139849 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad406c7f0 con 0x7f1ad40cd8e0 2023-03-03T11:08:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.185+0000 7f1ad8b99700 1 -- 172.21.15.102:0/629139849 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879088456 0 0) 0x7f1ad4153660 con 0x7f1ad4153290 2023-03-03T11:08:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.185+0000 7f1ad939a700 1 --2- 172.21.15.102:0/629139849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad40cde20 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009840 tx=0x7f1ac4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6af5bd68abc186 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.185+0000 7f1ad8b99700 1 -- 172.21.15.102:0/629139849 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4011830 con 0x7f1ad40cd8e0 2023-03-03T11:08:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1ad8b99700 1 -- 172.21.15.102:0/629139849 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac4011990 con 0x7f1ad40cd8e0 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1ad8b99700 1 -- 172.21.15.102:0/629139849 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4011d20 con 0x7f1ad40cd8e0 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 -- 172.21.15.102:0/629139849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 msgr2=0x7f1ad40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 --2- 172.21.15.102:0/629139849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad40cde20 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009840 tx=0x7f1ac4009b50 comp rx=0 tx=0).stop 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 -- 172.21.15.102:0/629139849 shutdown_connections 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 --2- 172.21.15.102:0/629139849 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 --2- 172.21.15.102:0/629139849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad40cde20 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 -- 172.21.15.102:0/629139849 >> 172.21.15.102:0/629139849 conn(0x7f1ad4065fb0 msgr2=0x7f1ad40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 -- 172.21.15.102:0/629139849 shutdown_connections 2023-03-03T11:08:55.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.186+0000 7f1adbdff700 1 -- 172.21.15.102:0/629139849 wait complete. 2023-03-03T11:08:55.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.188+0000 7f1adbdff700 1 Processor -- start 2023-03-03T11:08:55.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.188+0000 7f1adbdff700 1 -- start start 2023-03-03T11:08:55.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.189+0000 7f1adbdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad41511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:55.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.189+0000 7f1adbdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad41f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:55.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.189+0000 7f1adbdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad4153290 0x7f1ad41f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.189+0000 7f1adbdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad41ee070 con 0x7f1ad40cd8e0 2023-03-03T11:08:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.189+0000 7f1adbdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad41f1830 con 0x7f1ad4153290 2023-03-03T11:08:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.189+0000 7f1adbdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad41ee260 con 0x7f1ad40cd390 2023-03-03T11:08:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad4153290 0x7f1ad41f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ad9b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad41511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad4153290 0x7f1ad41f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47052/0 (socket says 172.21.15.102:47052) 2023-03-03T11:08:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 -- 172.21.15.102:0/3861538330 learned_addr learned my addr 172.21.15.102:0/3861538330 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ad939a700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad41f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 -- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 msgr2=0x7f1ad41511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad41511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 -- 172.21.15.102:0/3861538330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 msgr2=0x7f1ad41f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad41f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 -- 172.21.15.102:0/3861538330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40094a0 con 0x7f1ad4153290 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ad939a700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad41f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:08:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.190+0000 7f1ada39c700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad4153290 0x7f1ad41f2200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1acc009d30 tx=0x7f1acc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:55.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.191+0000 7f1aca7fc700 1 -- 172.21.15.102:0/3861538330 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc00cc90 con 0x7f1ad4153290 2023-03-03T11:08:55.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.191+0000 7f1aca7fc700 1 -- 172.21.15.102:0/3861538330 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1acc00f040 con 0x7f1ad4153290 2023-03-03T11:08:55.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.191+0000 7f1aca7fc700 1 -- 172.21.15.102:0/3861538330 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc009e80 con 0x7f1ad4153290 2023-03-03T11:08:55.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.191+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad40ccde0 con 0x7f1ad4153290 2023-03-03T11:08:55.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.191+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad415d160 con 0x7f1ad4153290 2023-03-03T11:08:55.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.192+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad40cd0c0 con 0x7f1ad4153290 2023-03-03T11:08:55.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.193+0000 7f1aca7fc700 1 -- 172.21.15.102:0/3861538330 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1acc014480 con 0x7f1ad4153290 2023-03-03T11:08:55.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.194+0000 7f1aca7fc700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab80682f0 0x7f1ab806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:55.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.195+0000 7f1aca7fc700 1 -- 172.21.15.102:0/3861538330 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1acc086750 con 0x7f1ad4153290 2023-03-03T11:08:55.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.195+0000 7f1ad9b9b700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab80682f0 0x7f1ab806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:55.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.195+0000 7f1ad9b9b700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab80682f0 0x7f1ab806a7a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0007fd0 tx=0x7f1ad0007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.199+0000 7f1aca7fc700 1 -- 172.21.15.102:0/3861538330 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1acc052d70 con 0x7f1ad4153290 2023-03-03T11:08:55.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.434+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ad41f1620 con 0x7f1ad4153290 2023-03-03T11:08:55.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.435+0000 7f1aca7fc700 1 -- 172.21.15.102:0/3861538330 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1acc018090 con 0x7f1ad4153290 2023-03-03T11:08:55.437 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:08:55.437 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:08:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab80682f0 msgr2=0x7f1ab806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab80682f0 0x7f1ab806a7a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0007fd0 tx=0x7f1ad0007de0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad4153290 msgr2=0x7f1ad41f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad4153290 0x7f1ad41f2200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1acc009d30 tx=0x7f1acc00c3a0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 shutdown_connections 2023-03-03T11:08:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ab80682f0 0x7f1ab806a7a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ad40cd390 0x7f1ad41511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ad4153290 0x7f1ad41f2200 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 --2- 172.21.15.102:0/3861538330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ad40cd8e0 0x7f1ad41f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 >> 172.21.15.102:0/3861538330 conn(0x7f1ad4065fb0 msgr2=0x7f1ad41574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 shutdown_connections 2023-03-03T11:08:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:08:55.438+0000 7f1adbdff700 1 -- 172.21.15.102:0/3861538330 wait complete. 2023-03-03T11:08:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:08:55.466 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2023-03-03T11:08:59.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:59.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:12.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:09:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.729+0000 7f72bafff700 1 Processor -- start 2023-03-03T11:09:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.730+0000 7f72bafff700 1 -- start start 2023-03-03T11:09:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72bafff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72bafff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72bafff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b4153660 con 0x7f72b40cd8e0 2023-03-03T11:09:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72bafff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b406c360 con 0x7f72b4153290 2023-03-03T11:09:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72bafff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b415cea0 con 0x7f72b41593c0 2023-03-03T11:09:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72b8d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72b8d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45908/0 (socket says 172.21.15.102:45908) 2023-03-03T11:09:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72b959c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f72b40cd8e0 0x7f72b40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49360/0 (socket says 172.21.15.102:49360) 2023-03-03T11:09:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1215131939 learned_addr learned my addr 172.21.15.102:0/1215131939 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:09:12.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.731+0000 7f72b3fff700 1 --2- 172.21.15.102:0/1215131939 >> v2:172.21.15.172:3300/0 conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1215131939 >> v2:172.21.15.172:3300/0 conn(0x7f72b41593c0 msgr2=0x7f72b40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:09:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b8d9b700 1 --2- 172.21.15.102:0/1215131939 >> v2:172.21.15.172:3300/0 conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1215131939 >> v1:172.21.15.102:6789/0 conn(0x7f72b40cd8e0 legacy=0x7f72b40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1215131939 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b406c7f0 con 0x7f72b4153290 2023-03-03T11:09:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b37fe700 1 -- 172.21.15.102:0/1215131939 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170946614 0 0) 0x7f72b4153660 con 0x7f72b40cd8e0 2023-03-03T11:09:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b8d9b700 1 --2- 172.21.15.102:0/1215131939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 0x7f72b4158e80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f72b4159fb0 tx=0x7f72a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6d6b4fdf8fa0a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b37fe700 1 -- 172.21.15.102:0/1215131939 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8011830 con 0x7f72b4153290 2023-03-03T11:09:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.732+0000 7f72b37fe700 1 -- 172.21.15.102:0/1215131939 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a8011990 con 0x7f72b4153290 2023-03-03T11:09:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72b37fe700 1 -- 172.21.15.102:0/1215131939 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8011d20 con 0x7f72b4153290 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 -- 172.21.15.102:0/1215131939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 msgr2=0x7f72b4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 --2- 172.21.15.102:0/1215131939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 0x7f72b4158e80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f72b4159fb0 tx=0x7f72a8009bd0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 -- 172.21.15.102:0/1215131939 shutdown_connections 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 --2- 172.21.15.102:0/1215131939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b4153290 0x7f72b4158e80 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 --2- 172.21.15.102:0/1215131939 >> v2:172.21.15.172:3300/0 conn(0x7f72b41593c0 0x7f72b40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 -- 172.21.15.102:0/1215131939 >> 172.21.15.102:0/1215131939 conn(0x7f72b4065fb0 msgr2=0x7f72b40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 -- 172.21.15.102:0/1215131939 shutdown_connections 2023-03-03T11:09:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.733+0000 7f72bafff700 1 -- 172.21.15.102:0/1215131939 wait complete. 2023-03-03T11:09:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.735+0000 7f72bafff700 1 Processor -- start 2023-03-03T11:09:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.735+0000 7f72bafff700 1 -- start start 2023-03-03T11:09:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.736+0000 7f72bafff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.736+0000 7f72bafff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.736+0000 7f72bafff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b41e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.736+0000 7f72bafff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b414ea90 con 0x7f72b41593c0 2023-03-03T11:09:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.736+0000 7f72bafff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41f02b0 con 0x7f72b40cd8e0 2023-03-03T11:09:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.736+0000 7f72bafff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40b3e20 con 0x7f72b4153290 2023-03-03T11:09:12.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b959c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b41e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b414e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47326/0 (socket says 172.21.15.102:47326) 2023-03-03T11:09:12.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1597595827 learned_addr learned my addr 172.21.15.102:0/1597595827 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:09:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 msgr2=0x7f72b41e5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41e5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1597595827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 msgr2=0x7f72b41e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b41e5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 -- 172.21.15.102:0/1597595827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a8009550 con 0x7f72b40cd8e0 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.737+0000 7f72b8d9b700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b414e550 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009c10 tx=0x7f72a8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.738+0000 7f72b17fa700 1 -- 172.21.15.102:0/1597595827 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8004510 con 0x7f72b40cd8e0 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.738+0000 7f72b17fa700 1 -- 172.21.15.102:0/1597595827 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a80036a0 con 0x7f72b40cd8e0 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.738+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b40ccde0 con 0x7f72b40cd8e0 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.738+0000 7f72b17fa700 1 -- 172.21.15.102:0/1597595827 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8003a10 con 0x7f72b40cd8e0 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.738+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b415d160 con 0x7f72b40cd8e0 2023-03-03T11:09:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.739+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b40cd0c0 con 0x7f72b40cd8e0 2023-03-03T11:09:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.740+0000 7f72b17fa700 1 -- 172.21.15.102:0/1597595827 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72a802d070 con 0x7f72b40cd8e0 2023-03-03T11:09:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.741+0000 7f72b17fa700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7298068320 0x7f729806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:12.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.741+0000 7f72b17fa700 1 -- 172.21.15.102:0/1597595827 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f72a8021030 con 0x7f72b40cd8e0 2023-03-03T11:09:12.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.742+0000 7f72b3fff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7298068320 0x7f729806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:12.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.742+0000 7f72b3fff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7298068320 0x7f729806a7d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f72b40cf820 tx=0x7f72a0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:12.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.746+0000 7f72b17fa700 1 -- 172.21.15.102:0/1597595827 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72a8053ae0 con 0x7f72b40cd8e0 2023-03-03T11:09:12.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.980+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72b404e6e0 con 0x7f72b40cd8e0 2023-03-03T11:09:12.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.981+0000 7f72b17fa700 1 -- 172.21.15.102:0/1597595827 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f72a8015090 con 0x7f72b40cd8e0 2023-03-03T11:09:12.984 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:09:12.984 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:09:12.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7298068320 msgr2=0x7f729806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:12.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7298068320 0x7f729806a7d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f72b40cf820 tx=0x7f72a0009210 comp rx=0 tx=0).stop 2023-03-03T11:09:12.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 msgr2=0x7f72b414e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b414e550 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009c10 tx=0x7f72a8004750 comp rx=0 tx=0).stop 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 shutdown_connections 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72b41593c0 0x7f72b41e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72b40cd8e0 0x7f72b414e550 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7298068320 0x7f729806a7d0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 --2- 172.21.15.102:0/1597595827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72b4153290 0x7f72b41e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 >> 172.21.15.102:0/1597595827 conn(0x7f72b4065fb0 msgr2=0x7f72b415ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:12.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 shutdown_connections 2023-03-03T11:09:12.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:09:12.983+0000 7f72bafff700 1 -- 172.21.15.102:0/1597595827 wait complete. 2023-03-03T11:09:12.988 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:09:13.013 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 93 secs before thrashing 2023-03-03T11:09:29.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:29.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:59.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:59.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T11:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:10:29.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:29.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:46.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:10:46.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.264+0000 7f40971ff700 1 Processor -- start 2023-03-03T11:10:46.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.264+0000 7f40971ff700 1 -- start start 2023-03-03T11:10:46.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.265+0000 7f40971ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4090153290 0x7f4090158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:10:46.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f40971ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4090153660 con 0x7f40901593c0 2023-03-03T11:10:46.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f40971ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409006c360 con 0x7f40900cd8e0 2023-03-03T11:10:46.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f40971ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40900cec40 con 0x7f4090153290 2023-03-03T11:10:46.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f408ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f40901593c0 0x7f40900cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46394/0 (socket says 172.21.15.102:46394) 2023-03-03T11:10:46.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f408ffff700 1 -- 172.21.15.102:0/2631095737 learned_addr learned my addr 172.21.15.102:0/2631095737 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:10:46.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f4094f9b700 1 --2- 172.21.15.102:0/2631095737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4090153290 0x7f4090158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:10:46.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f409579c700 1 --1- 172.21.15.102:0/2631095737 >> v1:172.21.15.172:6789/0 conn(0x7f40900cd8e0 0x7f40900ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45098/0 (socket says 172.21.15.102:45098) 2023-03-03T11:10:46.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f4094f9b700 1 -- 172.21.15.102:0/2631095737 >> v1:172.21.15.172:6789/0 conn(0x7f40900cd8e0 legacy=0x7f40900ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:10:46.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f4094f9b700 1 -- 172.21.15.102:0/2631095737 >> v1:172.21.15.102:6789/0 conn(0x7f40901593c0 legacy=0x7f40900cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:10:46.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f4094f9b700 1 -- 172.21.15.102:0/2631095737 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409015cea0 con 0x7f4090153290 2023-03-03T11:10:46.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.266+0000 7f408f7fe700 1 -- 172.21.15.102:0/2631095737 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744915437 0 0) 0x7f409006c360 con 0x7f40900cd8e0 2023-03-03T11:10:46.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.267+0000 7f4094f9b700 1 --2- 172.21.15.102:0/2631095737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4090153290 0x7f4090158e80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4084009870 tx=0x7f4084009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43d3ec1e0d16e6d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:10:46.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.267+0000 7f408f7fe700 1 -- 172.21.15.102:0/2631095737 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084004230 con 0x7f4090153290 2023-03-03T11:10:46.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.267+0000 7f408f7fe700 1 -- 172.21.15.102:0/2631095737 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4084004390 con 0x7f4090153290 2023-03-03T11:10:46.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.267+0000 7f408f7fe700 1 -- 172.21.15.102:0/2631095737 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40840036a0 con 0x7f4090153290 2023-03-03T11:10:46.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.268+0000 7f40971ff700 1 -- 172.21.15.102:0/2631095737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4090153290 msgr2=0x7f4090158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:10:46.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.268+0000 7f40971ff700 1 --2- 172.21.15.102:0/2631095737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4090153290 0x7f4090158e80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4084009870 tx=0x7f4084009b80 comp rx=0 tx=0).stop 2023-03-03T11:10:46.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.268+0000 7f40971ff700 1 -- 172.21.15.102:0/2631095737 shutdown_connections 2023-03-03T11:10:46.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.268+0000 7f40971ff700 1 --2- 172.21.15.102:0/2631095737 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4090153290 0x7f4090158e80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:10:46.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.268+0000 7f40971ff700 1 -- 172.21.15.102:0/2631095737 >> 172.21.15.102:0/2631095737 conn(0x7f4090065fb0 msgr2=0x7f40900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:10:46.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.268+0000 7f40971ff700 1 -- 172.21.15.102:0/2631095737 shutdown_connections 2023-03-03T11:10:46.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.268+0000 7f40971ff700 1 -- 172.21.15.102:0/2631095737 wait complete. 2023-03-03T11:10:46.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.270+0000 7f40971ff700 1 Processor -- start 2023-03-03T11:10:46.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.270+0000 7f40971ff700 1 -- start start 2023-03-03T11:10:46.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f40971ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40900cd8e0 0x7f40900c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:10:46.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f40971ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4090153290 0x7f40901e33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:10:46.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f40971ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40901593c0 0x7f40901f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:10:46.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f40971ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40900c8e60 con 0x7f4090153290 2023-03-03T11:10:46.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f40971ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40901e3920 con 0x7f40900cd8e0 2023-03-03T11:10:46.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f40971ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40900c6b10 con 0x7f40901593c0 2023-03-03T11:10:46.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f408ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4090153290 0x7f40901e33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:10:46.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f408ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4090153290 0x7f40901e33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42612/0 (socket says 172.21.15.102:42612) 2023-03-03T11:10:46.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f408ffff700 1 -- 172.21.15.102:0/1654002353 learned_addr learned my addr 172.21.15.102:0/1654002353 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:10:46.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.271+0000 7f4094f9b700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40900cd8e0 0x7f40900c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:10:46.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f409579c700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40901593c0 0x7f40901f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:10:46.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408ffff700 1 -- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40901593c0 msgr2=0x7f40901f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:10:46.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408ffff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40901593c0 0x7f40901f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:10:46.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408ffff700 1 -- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40900cd8e0 msgr2=0x7f40900c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:10:46.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408ffff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40900cd8e0 0x7f40900c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:10:46.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408ffff700 1 -- 172.21.15.102:0/1654002353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40840094a0 con 0x7f4090153290 2023-03-03T11:10:46.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f4094f9b700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40900cd8e0 0x7f40900c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:10:46.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408ffff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4090153290 0x7f40901e33e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c780 tx=0x7f407c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:10:46.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408d7fa700 1 -- 172.21.15.102:0/1654002353 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c004c80 con 0x7f4090153290 2023-03-03T11:10:46.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408d7fa700 1 -- 172.21.15.102:0/1654002353 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c004de0 con 0x7f4090153290 2023-03-03T11:10:46.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.272+0000 7f408d7fa700 1 -- 172.21.15.102:0/1654002353 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c007310 con 0x7f4090153290 2023-03-03T11:10:46.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.273+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409015d160 con 0x7f4090153290 2023-03-03T11:10:46.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.273+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409006c7f0 con 0x7f4090153290 2023-03-03T11:10:46.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.274+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40900ccde0 con 0x7f4090153290 2023-03-03T11:10:46.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.275+0000 7f408d7fa700 1 -- 172.21.15.102:0/1654002353 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f407c01a020 con 0x7f4090153290 2023-03-03T11:10:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.277+0000 7f408d7fa700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f40740682f0 0x7f407406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:10:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.277+0000 7f408d7fa700 1 -- 172.21.15.102:0/1654002353 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f407c085a60 con 0x7f4090153290 2023-03-03T11:10:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.277+0000 7f4094f9b700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f40740682f0 0x7f407406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:10:46.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.278+0000 7f4094f9b700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f40740682f0 0x7f407406a7a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f40840090b0 tx=0x7f4084008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:10:46.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.282+0000 7f408d7fa700 1 -- 172.21.15.102:0/1654002353 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f407c052170 con 0x7f4090153290 2023-03-03T11:10:46.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.512+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f409004e6e0 con 0x7f4090153290 2023-03-03T11:10:46.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.513+0000 7f408d7fa700 1 -- 172.21.15.102:0/1654002353 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f407c055790 con 0x7f4090153290 2023-03-03T11:10:46.516 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:10:46.516 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:10:46.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.515+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f40740682f0 msgr2=0x7f407406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:10:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.515+0000 7f40971ff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f40740682f0 0x7f407406a7a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f40840090b0 tx=0x7f4084008030 comp rx=0 tx=0).stop 2023-03-03T11:10:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.515+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4090153290 msgr2=0x7f40901e33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:10:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.515+0000 7f40971ff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4090153290 0x7f40901e33e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c780 tx=0x7f407c00ca90 comp rx=0 tx=0).stop 2023-03-03T11:10:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 shutdown_connections 2023-03-03T11:10:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f40740682f0 0x7f407406a7a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:10:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f40901593c0 0x7f40901f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:10:46.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f40900cd8e0 0x7f40900c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:10:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 --2- 172.21.15.102:0/1654002353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4090153290 0x7f40901e33e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:10:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 >> 172.21.15.102:0/1654002353 conn(0x7f4090065fb0 msgr2=0x7f409015ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:10:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 shutdown_connections 2023-03-03T11:10:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:10:46.516+0000 7f40971ff700 1 -- 172.21.15.102:0/1654002353 wait complete. 2023-03-03T11:10:46.520 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:10:46.546 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-03-03T11:11:00.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:00.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:30.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:30.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:00.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:00.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:07.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:12:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.792+0000 7f5add9ff700 1 Processor -- start 2023-03-03T11:12:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.793+0000 7f5add9ff700 1 -- start start 2023-03-03T11:12:07.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.794+0000 7f5add9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ad8153290 0x7f5ad8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.794+0000 7f5add9ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad8153660 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.794+0000 7f5add9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad806c360 con 0x7f5ad81593c0 2023-03-03T11:12:07.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.794+0000 7f5add9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad80cec40 con 0x7f5ad8153290 2023-03-03T11:12:07.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.794+0000 7f5ad77fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5ad80cd8e0 0x7f5ad80ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52390/0 (socket says 172.21.15.102:52390) 2023-03-03T11:12:07.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.794+0000 7f5ad77fe700 1 -- 172.21.15.102:0/2386540261 learned_addr learned my addr 172.21.15.102:0/2386540261 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:12:07.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad6ffd700 1 --2- 172.21.15.102:0/2386540261 >> v2:172.21.15.172:3300/0 conn(0x7f5ad8153290 0x7f5ad8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad5ffb700 1 -- 172.21.15.102:0/2386540261 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379942330 0 0) 0x7f5ad8153660 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad5ffb700 1 -- 172.21.15.102:0/2386540261 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac00035b0 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad5ffb700 1 -- 172.21.15.102:0/2386540261 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138194053 0 0) 0x7f5ad806c360 con 0x7f5ad81593c0 2023-03-03T11:12:07.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad5ffb700 1 -- 172.21.15.102:0/2386540261 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad8153660 con 0x7f5ad81593c0 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad6ffd700 1 -- 172.21.15.102:0/2386540261 >> v1:172.21.15.172:6789/0 conn(0x7f5ad81593c0 legacy=0x7f5ad80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad6ffd700 1 -- 172.21.15.102:0/2386540261 >> v1:172.21.15.102:6789/0 conn(0x7f5ad80cd8e0 legacy=0x7f5ad80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.795+0000 7f5ad6ffd700 1 -- 172.21.15.102:0/2386540261 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad815cea0 con 0x7f5ad8153290 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5ad6ffd700 1 --2- 172.21.15.102:0/2386540261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad8153290 0x7f5ad8158e80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009840 tx=0x7f5ac8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d84484f393da7736 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5ad5ffb700 1 -- 172.21.15.102:0/2386540261 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8011830 con 0x7f5ad8153290 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5ad5ffb700 1 -- 172.21.15.102:0/2386540261 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac8011990 con 0x7f5ad8153290 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5ad5ffb700 1 -- 172.21.15.102:0/2386540261 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8011d20 con 0x7f5ad8153290 2023-03-03T11:12:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5add9ff700 1 -- 172.21.15.102:0/2386540261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad8153290 msgr2=0x7f5ad8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5add9ff700 1 --2- 172.21.15.102:0/2386540261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad8153290 0x7f5ad8158e80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009840 tx=0x7f5ac8009b50 comp rx=0 tx=0).stop 2023-03-03T11:12:07.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5add9ff700 1 -- 172.21.15.102:0/2386540261 shutdown_connections 2023-03-03T11:12:07.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5add9ff700 1 --2- 172.21.15.102:0/2386540261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad8153290 0x7f5ad8158e80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.796+0000 7f5add9ff700 1 -- 172.21.15.102:0/2386540261 >> 172.21.15.102:0/2386540261 conn(0x7f5ad8065fb0 msgr2=0x7f5ad80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:07.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.797+0000 7f5add9ff700 1 -- 172.21.15.102:0/2386540261 shutdown_connections 2023-03-03T11:12:07.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.797+0000 7f5add9ff700 1 -- 172.21.15.102:0/2386540261 wait complete. 2023-03-03T11:12:07.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.798+0000 7f5add9ff700 1 Processor -- start 2023-03-03T11:12:07.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.799+0000 7f5add9ff700 1 -- start start 2023-03-03T11:12:07.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5add9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad80cd8e0 0x7f5ad81ef340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5add9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ad8153290 0x7f5ad80caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5add9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad81593c0 0x7f5ad80c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5add9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad80c7830 con 0x7f5ad81593c0 2023-03-03T11:12:07.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5add9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad80c5730 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5add9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad80c6a80 con 0x7f5ad8153290 2023-03-03T11:12:07.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5ad77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad81593c0 0x7f5ad80c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5ad6ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad80cd8e0 0x7f5ad81ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5ad6ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad80cd8e0 0x7f5ad81ef340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34612/0 (socket says 172.21.15.102:34612) 2023-03-03T11:12:07.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5ad6ffd700 1 -- 172.21.15.102:0/3243223991 learned_addr learned my addr 172.21.15.102:0/3243223991 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:12:07.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.800+0000 7f5ad67fc700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ad8153290 0x7f5ad80caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5ad6ffd700 1 -- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ad8153290 msgr2=0x7f5ad80caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5ad6ffd700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ad8153290 0x7f5ad80caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5ad6ffd700 1 -- 172.21.15.102:0/3243223991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad81593c0 msgr2=0x7f5ad80c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5ad6ffd700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad81593c0 0x7f5ad80c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5ad6ffd700 1 -- 172.21.15.102:0/3243223991 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac80094a0 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5ad6ffd700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad80cd8e0 0x7f5ad81ef340 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80090b0 tx=0x7f5ac8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:07.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5abffff700 1 -- 172.21.15.102:0/3243223991 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac801a8f0 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5abffff700 1 -- 172.21.15.102:0/3243223991 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac801aa50 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5abffff700 1 -- 172.21.15.102:0/3243223991 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac801adc0 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.801+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad815d160 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.802+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad806c7f0 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.803+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad80ccde0 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.804+0000 7f5abffff700 1 -- 172.21.15.102:0/3243223991 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5ac800f050 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.805+0000 7f5abffff700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ac40682f0 0x7f5ac406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.805+0000 7f5abffff700 1 -- 172.21.15.102:0/3243223991 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5ac8086f40 con 0x7f5ad80cd8e0 2023-03-03T11:12:07.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.805+0000 7f5ad67fc700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ac40682f0 0x7f5ac406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.806+0000 7f5ad67fc700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ac40682f0 0x7f5ac406a7a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8152170 tx=0x7f5acc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:07.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:07.809+0000 7f5abffff700 1 -- 172.21.15.102:0/3243223991 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5ac8053650 con 0x7f5ad80cd8e0 2023-03-03T11:12:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.044+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ad80c7b10 con 0x7f5ad80cd8e0 2023-03-03T11:12:08.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.045+0000 7f5abffff700 1 -- 172.21.15.102:0/3243223991 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5ac8015070 con 0x7f5ad80cd8e0 2023-03-03T11:12:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:12:08.047 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:12:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ac40682f0 msgr2=0x7f5ac406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ac40682f0 0x7f5ac406a7a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8152170 tx=0x7f5acc009040 comp rx=0 tx=0).stop 2023-03-03T11:12:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad80cd8e0 msgr2=0x7f5ad81ef340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad80cd8e0 0x7f5ad81ef340 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80090b0 tx=0x7f5ac8004750 comp rx=0 tx=0).stop 2023-03-03T11:12:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 shutdown_connections 2023-03-03T11:12:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ad81593c0 0x7f5ad80c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ad80cd8e0 0x7f5ad81ef340 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5ac40682f0 0x7f5ac406a7a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 --2- 172.21.15.102:0/3243223991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ad8153290 0x7f5ad80caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 >> 172.21.15.102:0/3243223991 conn(0x7f5ad8065fb0 msgr2=0x7f5ad815bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:08.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.047+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 shutdown_connections 2023-03-03T11:12:08.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:08.048+0000 7f5add9ff700 1 -- 172.21.15.102:0/3243223991 wait complete. 2023-03-03T11:12:08.051 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:12:08.076 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T11:12:30.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:30.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:36.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:12:36.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.340+0000 7f2b87fff700 1 Processor -- start 2023-03-03T11:12:36.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.340+0000 7f2b87fff700 1 -- start start 2023-03-03T11:12:36.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b87fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b8814fb20 0x7f2b880721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b880c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b87fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8806b160 con 0x7f2b880cb2d0 2023-03-03T11:12:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b87fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b880c2810 con 0x7f2b8814fb20 2023-03-03T11:12:36.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b87fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88159880 con 0x7f2b88071470 2023-03-03T11:12:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b86ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b8814fb20 0x7f2b880721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b877fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b880c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b867fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2b880cb2d0 0x7f2b880c6250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:58122/0 (socket says 172.21.15.102:58122) 2023-03-03T11:12:36.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.341+0000 7f2b867fc700 1 -- 172.21.15.102:0/1626291605 learned_addr learned my addr 172.21.15.102:0/1626291605 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:12:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b86ffd700 1 -- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 msgr2=0x7f2b880c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b86ffd700 1 --2- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b880c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b86ffd700 1 -- 172.21.15.102:0/1626291605 >> v1:172.21.15.172:6789/0 conn(0x7f2b880cb2d0 legacy=0x7f2b880c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b86ffd700 1 -- 172.21.15.102:0/1626291605 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88071d40 con 0x7f2b8814fb20 2023-03-03T11:12:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b85ffb700 1 -- 172.21.15.102:0/1626291605 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445026658 0 0) 0x7f2b8806b160 con 0x7f2b880cb2d0 2023-03-03T11:12:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b877fe700 1 --2- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b880c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:12:36.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b86ffd700 1 --2- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b8814fb20 0x7f2b880721d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009840 tx=0x7f2b78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=542d0b7ae1e69f7f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:36.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b85ffb700 1 -- 172.21.15.102:0/1626291605 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78011830 con 0x7f2b8814fb20 2023-03-03T11:12:36.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b85ffb700 1 -- 172.21.15.102:0/1626291605 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b78011990 con 0x7f2b8814fb20 2023-03-03T11:12:36.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.342+0000 7f2b85ffb700 1 -- 172.21.15.102:0/1626291605 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78011d20 con 0x7f2b8814fb20 2023-03-03T11:12:36.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 -- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b8814fb20 msgr2=0x7f2b880721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:36.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b8814fb20 0x7f2b880721d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009840 tx=0x7f2b78009b50 comp rx=0 tx=0).stop 2023-03-03T11:12:36.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 -- 172.21.15.102:0/1626291605 shutdown_connections 2023-03-03T11:12:36.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b880c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1626291605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b8814fb20 0x7f2b880721d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 -- 172.21.15.102:0/1626291605 >> 172.21.15.102:0/1626291605 conn(0x7f2b880ca7c0 msgr2=0x7f2b880cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:36.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 -- 172.21.15.102:0/1626291605 shutdown_connections 2023-03-03T11:12:36.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.343+0000 7f2b87fff700 1 -- 172.21.15.102:0/1626291605 wait complete. 2023-03-03T11:12:36.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.345+0000 7f2b87fff700 1 Processor -- start 2023-03-03T11:12:36.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.345+0000 7f2b87fff700 1 -- start start 2023-03-03T11:12:36.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.346+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b88158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:36.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.346+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b880cb2d0 0x7f2b881f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.346+0000 7f2b87fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b8814fb20 0x7f2b880bba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.346+0000 7f2b87fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b881f2660 con 0x7f2b8814fb20 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.346+0000 7f2b87fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b881580a0 con 0x7f2b880cb2d0 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.346+0000 7f2b87fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c9830 con 0x7f2b88071470 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b88158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b88158530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59524/0 (socket says 172.21.15.102:59524) 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 -- 172.21.15.102:0/1862822175 learned_addr learned my addr 172.21.15.102:0/1862822175 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:12:36.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b867fc700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b880cb2d0 0x7f2b881f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b877fe700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b8814fb20 0x7f2b880bba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 -- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b880cb2d0 msgr2=0x7f2b881f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b880cb2d0 0x7f2b881f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 -- 172.21.15.102:0/1862822175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b8814fb20 msgr2=0x7f2b880bba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b8814fb20 0x7f2b880bba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 -- 172.21.15.102:0/1862822175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b80009a70 con 0x7f2b88071470 2023-03-03T11:12:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.347+0000 7f2b86ffd700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b88158530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009b90 tx=0x7f2b78004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:36.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.348+0000 7f2b6f7fe700 1 -- 172.21.15.102:0/1862822175 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b780044d0 con 0x7f2b88071470 2023-03-03T11:12:36.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.348+0000 7f2b6f7fe700 1 -- 172.21.15.102:0/1862822175 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7801e070 con 0x7f2b88071470 2023-03-03T11:12:36.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.348+0000 7f2b6f7fe700 1 -- 172.21.15.102:0/1862822175 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b780038e0 con 0x7f2b88071470 2023-03-03T11:12:36.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.348+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b780094a0 con 0x7f2b88071470 2023-03-03T11:12:36.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.348+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b88159a90 con 0x7f2b88071470 2023-03-03T11:12:36.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.349+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b880706f0 con 0x7f2b88071470 2023-03-03T11:12:36.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.350+0000 7f2b6f7fe700 1 -- 172.21.15.102:0/1862822175 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2b7800f050 con 0x7f2b88071470 2023-03-03T11:12:36.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.352+0000 7f2b6f7fe700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b740682f0 0x7f2b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:36.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.352+0000 7f2b6f7fe700 1 -- 172.21.15.102:0/1862822175 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2b78086ac0 con 0x7f2b88071470 2023-03-03T11:12:36.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.352+0000 7f2b867fc700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b740682f0 0x7f2b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:36.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.352+0000 7f2b867fc700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b740682f0 0x7f2b7406a7a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2b880c16b0 tx=0x7f2b7c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:36.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.356+0000 7f2b6f7fe700 1 -- 172.21.15.102:0/1862822175 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2b780531d0 con 0x7f2b88071470 2023-03-03T11:12:36.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.590+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b880719d0 con 0x7f2b88071470 2023-03-03T11:12:36.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.591+0000 7f2b6f7fe700 1 -- 172.21.15.102:0/1862822175 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2b78020090 con 0x7f2b88071470 2023-03-03T11:12:36.594 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:12:36.594 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:12:36.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.593+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b740682f0 msgr2=0x7f2b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b740682f0 0x7f2b7406a7a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2b880c16b0 tx=0x7f2b7c007de0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 msgr2=0x7f2b88158530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b88158530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009b90 tx=0x7f2b78004750 comp rx=0 tx=0).stop 2023-03-03T11:12:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 shutdown_connections 2023-03-03T11:12:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b740682f0 0x7f2b7406a7a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b88071470 0x7f2b88158530 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b8814fb20 0x7f2b880bba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 --2- 172.21.15.102:0/1862822175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b880cb2d0 0x7f2b881f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 >> 172.21.15.102:0/1862822175 conn(0x7f2b880ca7c0 msgr2=0x7f2b8814b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 shutdown_connections 2023-03-03T11:12:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:12:36.594+0000 7f2b87fff700 1 -- 172.21.15.102:0/1862822175 wait complete. 2023-03-03T11:12:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:12:36.623 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T11:13:00.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:00.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:30.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:30.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:55.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:13:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.886+0000 7f1bfa9ff700 1 Processor -- start 2023-03-03T11:13:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.887+0000 7f1bfa9ff700 1 -- start start 2023-03-03T11:13:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bfa9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf40cd8e0 0x7f1bf40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bfa9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1bf40cd390 0x7f1bf40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bfa9ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf4153660 con 0x7f1bf4153290 2023-03-03T11:13:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bfa9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf406c360 con 0x7f1bf40cd390 2023-03-03T11:13:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bfa9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf415cea0 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bf3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1bf4153290 0x7f1bf4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54108/0 (socket says 172.21.15.102:54108) 2023-03-03T11:13:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bf3fff700 1 -- 172.21.15.102:0/453920545 learned_addr learned my addr 172.21.15.102:0/453920545 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:13:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.888+0000 7f1bf8f9c700 1 --2- 172.21.15.102:0/453920545 >> v2:172.21.15.172:3300/0 conn(0x7f1bf40cd390 0x7f1bf40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf37fe700 1 --2- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf40cd8e0 0x7f1bf40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf2ffd700 1 -- 172.21.15.102:0/453920545 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113307066 0 0) 0x7f1bf4153660 con 0x7f1bf4153290 2023-03-03T11:13:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf2ffd700 1 -- 172.21.15.102:0/453920545 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bdc0035b0 con 0x7f1bf4153290 2023-03-03T11:13:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf8f9c700 1 -- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf40cd8e0 msgr2=0x7f1bf40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:13:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf8f9c700 1 --2- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf40cd8e0 0x7f1bf40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf8f9c700 1 -- 172.21.15.102:0/453920545 >> v1:172.21.15.102:6789/0 conn(0x7f1bf4153290 legacy=0x7f1bf4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf8f9c700 1 -- 172.21.15.102:0/453920545 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf406c7f0 con 0x7f1bf40cd390 2023-03-03T11:13:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.889+0000 7f1bf8f9c700 1 --2- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 0x7f1bf40ce360 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009d90 tx=0x7f1bec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7554e1c75a141ac4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:13:55.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bf2ffd700 1 -- 172.21.15.102:0/453920545 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00e070 con 0x7f1bf40cd390 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bf2ffd700 1 -- 172.21.15.102:0/453920545 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec00bb20 con 0x7f1bf40cd390 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bf2ffd700 1 -- 172.21.15.102:0/453920545 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec012430 con 0x7f1bf40cd390 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 msgr2=0x7f1bf40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 0x7f1bf40ce360 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009d90 tx=0x7f1bec00b330 comp rx=0 tx=0).stop 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/453920545 shutdown_connections 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 0x7f1bf40ce360 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/453920545 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf40cd8e0 0x7f1bf40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/453920545 >> 172.21.15.102:0/453920545 conn(0x7f1bf4065fb0 msgr2=0x7f1bf40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:13:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/453920545 shutdown_connections 2023-03-03T11:13:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.890+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/453920545 wait complete. 2023-03-03T11:13:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.892+0000 7f1bfa9ff700 1 Processor -- start 2023-03-03T11:13:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.893+0000 7f1bfa9ff700 1 -- start start 2023-03-03T11:13:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bfa9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 0x7f1bf41ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bfa9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bf40cd8e0 0x7f1bf41efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bfa9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf4153290 0x7f1bf4150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bfa9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40c8660 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bfa9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40c6f70 con 0x7f1bf40cd390 2023-03-03T11:13:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bfa9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40c7250 con 0x7f1bf4153290 2023-03-03T11:13:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bf37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bf40cd8e0 0x7f1bf41efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bf37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bf40cd8e0 0x7f1bf41efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59752/0 (socket says 172.21.15.102:59752) 2023-03-03T11:13:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bf37fe700 1 -- 172.21.15.102:0/1884724998 learned_addr learned my addr 172.21.15.102:0/1884724998 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:13:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bf3fff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 0x7f1bf41ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.894+0000 7f1bf8f9c700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf4153290 0x7f1bf4150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf37fe700 1 -- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf4153290 msgr2=0x7f1bf4150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf37fe700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf4153290 0x7f1bf4150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf37fe700 1 -- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 msgr2=0x7f1bf41ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf37fe700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 0x7f1bf41ef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf37fe700 1 -- 172.21.15.102:0/1884724998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec009a70 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf37fe700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bf40cd8e0 0x7f1bf41efd60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1be800e9b0 tx=0x7f1be800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:13:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf0ff9700 1 -- 172.21.15.102:0/1884724998 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8009710 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf0ff9700 1 -- 172.21.15.102:0/1884724998 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be800ee50 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bf0ff9700 1 -- 172.21.15.102:0/1884724998 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be800ca50 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.895+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf40ccde0 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.896+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf415d160 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.897+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf40cd0c0 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.898+0000 7f1bf0ff9700 1 -- 172.21.15.102:0/1884724998 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1be8019030 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.899+0000 7f1bf0ff9700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1be0068320 0x7f1be006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.899+0000 7f1bf0ff9700 1 -- 172.21.15.102:0/1884724998 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1be8085c80 con 0x7f1bf40cd8e0 2023-03-03T11:13:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.899+0000 7f1bf3fff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1be0068320 0x7f1be006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.900+0000 7f1bf3fff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1be0068320 0x7f1be006a7d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1be4007fd0 tx=0x7f1be4007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:13:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:55.903+0000 7f1bf0ff9700 1 -- 172.21.15.102:0/1884724998 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1be8052330 con 0x7f1bf40cd8e0 2023-03-03T11:13:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.135+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bf404e6e0 con 0x7f1bf40cd8e0 2023-03-03T11:13:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.136+0000 7f1bf0ff9700 1 -- 172.21.15.102:0/1884724998 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1be8055950 con 0x7f1bf40cd8e0 2023-03-03T11:13:56.139 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:13:56.139 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:13:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.138+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1be0068320 msgr2=0x7f1be006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.138+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1be0068320 0x7f1be006a7d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1be4007fd0 tx=0x7f1be4007dc0 comp rx=0 tx=0).stop 2023-03-03T11:13:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.138+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bf40cd8e0 msgr2=0x7f1bf41efd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.138+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bf40cd8e0 0x7f1bf41efd60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1be800e9b0 tx=0x7f1be800ecc0 comp rx=0 tx=0).stop 2023-03-03T11:13:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 shutdown_connections 2023-03-03T11:13:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1be0068320 0x7f1be006a7d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1bf4153290 0x7f1bf4150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1bf40cd390 0x7f1bf41ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 --2- 172.21.15.102:0/1884724998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bf40cd8e0 0x7f1bf41efd60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 >> 172.21.15.102:0/1884724998 conn(0x7f1bf4065fb0 msgr2=0x7f1bf4157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:13:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 shutdown_connections 2023-03-03T11:13:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:13:56.139+0000 7f1bfa9ff700 1 -- 172.21.15.102:0/1884724998 wait complete. 2023-03-03T11:13:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:13:56.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-03-03T11:14:01.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:01.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:25.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:14:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.422+0000 7f75ad79d700 1 Processor -- start 2023-03-03T11:14:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.422+0000 7f75ad79d700 1 -- start start 2023-03-03T11:14:25.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.423+0000 7f75ad79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a80c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.423+0000 7f75ad79d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f75a80c41b0 0x7f75a80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.423+0000 7f75ad79d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a80c3d20 con 0x7f75a80b3970 2023-03-03T11:14:25.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.423+0000 7f75ad79d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a80c4930 con 0x7f75a80c41b0 2023-03-03T11:14:25.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.423+0000 7f75ad79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80cf610 con 0x7f75a80ced00 2023-03-03T11:14:25.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75a7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a80c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75acf9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f75a80c41b0 0x7f75a80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75a77fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f75a80b3970 0x7f75a8158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44476/0 (socket says 172.21.15.102:44476) 2023-03-03T11:14:25.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75a77fe700 1 -- 172.21.15.102:0/3738997347 learned_addr learned my addr 172.21.15.102:0/3738997347 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:14:25.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75acf9c700 1 -- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 msgr2=0x7f75a80c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:14:25.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75acf9c700 1 --2- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a80c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75acf9c700 1 -- 172.21.15.102:0/3738997347 >> v1:172.21.15.172:6789/0 conn(0x7f75a80b3970 legacy=0x7f75a8158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75acf9c700 1 -- 172.21.15.102:0/3738997347 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a815cba0 con 0x7f75a80c41b0 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75a7fff700 1 --2- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a80c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.424+0000 7f75a6ffd700 1 -- 172.21.15.102:0/3738997347 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139773652 0 0) 0x7f75a80c3d20 con 0x7f75a80b3970 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.425+0000 7f75acf9c700 1 --2- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a80cf0d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009dc0 tx=0x7f75a000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d98b0427055297e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.425+0000 7f75a6ffd700 1 -- 172.21.15.102:0/3738997347 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a000e070 con 0x7f75a80c41b0 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.425+0000 7f75a6ffd700 1 -- 172.21.15.102:0/3738997347 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a000bae0 con 0x7f75a80c41b0 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.425+0000 7f75a6ffd700 1 -- 172.21.15.102:0/3738997347 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a000be90 con 0x7f75a80c41b0 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.425+0000 7f75ad79d700 1 -- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 msgr2=0x7f75a80cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.425+0000 7f75ad79d700 1 --2- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a80cf0d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009dc0 tx=0x7f75a000b330 comp rx=0 tx=0).stop 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.426+0000 7f75ad79d700 1 -- 172.21.15.102:0/3738997347 shutdown_connections 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.426+0000 7f75ad79d700 1 --2- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a80cf0d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.426+0000 7f75ad79d700 1 --2- 172.21.15.102:0/3738997347 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a80c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.426+0000 7f75ad79d700 1 -- 172.21.15.102:0/3738997347 >> 172.21.15.102:0/3738997347 conn(0x7f75a808b1c0 msgr2=0x7f75a8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.426+0000 7f75ad79d700 1 -- 172.21.15.102:0/3738997347 shutdown_connections 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.426+0000 7f75ad79d700 1 -- 172.21.15.102:0/3738997347 wait complete. 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.428+0000 7f75ad79d700 1 Processor -- start 2023-03-03T11:14:25.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.429+0000 7f75ad79d700 1 -- start start 2023-03-03T11:14:25.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.430+0000 7f75ad79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75a80b3970 0x7f75a8148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.430+0000 7f75ad79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a8159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.430+0000 7f75ad79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.430+0000 7f75ad79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a814dab0 con 0x7f75a80b3970 2023-03-03T11:14:25.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.430+0000 7f75ad79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a814ea40 con 0x7f75a80c41b0 2023-03-03T11:14:25.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.430+0000 7f75ad79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a814d110 con 0x7f75a80ced00 2023-03-03T11:14:25.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.430+0000 7f75a77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a8159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75acf9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75a80b3970 0x7f75a8148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75acf9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a815a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36574/0 (socket says 172.21.15.102:36574) 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a8159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60012/0 (socket says 172.21.15.102:60012) 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75acf9c700 1 -- 172.21.15.102:0/4052912605 learned_addr learned my addr 172.21.15.102:0/4052912605 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a77fe700 1 -- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 msgr2=0x7f75a815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a77fe700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a77fe700 1 -- 172.21.15.102:0/4052912605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75a80b3970 msgr2=0x7f75a8148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a77fe700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75a80b3970 0x7f75a8148df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a77fe700 1 -- 172.21.15.102:0/4052912605 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75980094a0 con 0x7f75a80c41b0 2023-03-03T11:14:25.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a77fe700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a8159fa0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f759c00c410 tx=0x7f759c00c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:25.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.431+0000 7f75a4ff9700 1 -- 172.21.15.102:0/4052912605 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c004c80 con 0x7f75a80c41b0 2023-03-03T11:14:25.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.432+0000 7f75a4ff9700 1 -- 172.21.15.102:0/4052912605 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c004de0 con 0x7f75a80c41b0 2023-03-03T11:14:25.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.432+0000 7f75a4ff9700 1 -- 172.21.15.102:0/4052912605 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c007310 con 0x7f75a80c41b0 2023-03-03T11:14:25.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.432+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a0009a70 con 0x7f75a80c41b0 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.432+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a806bf20 con 0x7f75a80c41b0 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.433+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a815ce80 con 0x7f75a80c41b0 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.434+0000 7f75a4ff9700 1 -- 172.21.15.102:0/4052912605 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f759c01a020 con 0x7f75a80c41b0 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.435+0000 7f75a4ff9700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7594068320 0x7f759406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.435+0000 7f75a4ff9700 1 -- 172.21.15.102:0/4052912605 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f759c085750 con 0x7f75a80c41b0 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.436+0000 7f75a7fff700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7594068320 0x7f759406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.436+0000 7f75a7fff700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7594068320 0x7f759406a7d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7598009d70 tx=0x7f7598009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:25.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.439+0000 7f75a4ff9700 1 -- 172.21.15.102:0/4052912605 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f759c051e60 con 0x7f75a80c41b0 2023-03-03T11:14:25.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.674+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a806c200 con 0x7f75a80c41b0 2023-03-03T11:14:25.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.675+0000 7f75a4ff9700 1 -- 172.21.15.102:0/4052912605 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f759c055480 con 0x7f75a80c41b0 2023-03-03T11:14:25.677 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:14:25.677 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:14:25.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.677+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7594068320 msgr2=0x7f759406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.677+0000 7f75ad79d700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7594068320 0x7f759406a7d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7598009d70 tx=0x7f7598009210 comp rx=0 tx=0).stop 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.677+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 msgr2=0x7f75a8159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.677+0000 7f75ad79d700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a8159fa0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f759c00c410 tx=0x7f759c00c720 comp rx=0 tx=0).stop 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 shutdown_connections 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7594068320 0x7f759406a7d0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75a80ced00 0x7f75a815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75a80b3970 0x7f75a8148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 --2- 172.21.15.102:0/4052912605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75a80c41b0 0x7f75a8159fa0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 >> 172.21.15.102:0/4052912605 conn(0x7f75a808b1c0 msgr2=0x7f75a8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:14:25.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 shutdown_connections 2023-03-03T11:14:25.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.678+0000 7f75ad79d700 1 -- 172.21.15.102:0/4052912605 wait complete. 2023-03-03T11:14:25.681 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:14:25.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:14:25.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.950+0000 7fb1557ff700 1 Processor -- start 2023-03-03T11:14:25.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.950+0000 7fb1557ff700 1 -- start start 2023-03-03T11:14:25.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.951+0000 7fb1557ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb150153290 0x7fb150158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.951+0000 7fb1557ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb1501593c0 0x7fb1500cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.951+0000 7fb1557ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb150153660 con 0x7fb1500cd8e0 2023-03-03T11:14:25.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.951+0000 7fb1557ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15006c360 con 0x7fb1501593c0 2023-03-03T11:14:25.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.951+0000 7fb1557ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15015cea0 con 0x7fb150153290 2023-03-03T11:14:25.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb1501593c0 0x7fb1500cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14effd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb150153290 0x7fb150158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb1501593c0 0x7fb1500cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37772/0 (socket says 172.21.15.102:37772) 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14effd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb150153290 0x7fb150158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60028/0 (socket says 172.21.15.102:60028) 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 -- 172.21.15.102:0/2013848857 learned_addr learned my addr 172.21.15.102:0/2013848857 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 -- 172.21.15.102:0/2013848857 >> v2:172.21.15.172:3300/0 conn(0x7fb150153290 msgr2=0x7fb150158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 --2- 172.21.15.102:0/2013848857 >> v2:172.21.15.172:3300/0 conn(0x7fb150153290 0x7fb150158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 -- 172.21.15.102:0/2013848857 >> v1:172.21.15.102:6789/0 conn(0x7fb1500cd8e0 legacy=0x7fb1500ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 -- 172.21.15.102:0/2013848857 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15006c7f0 con 0x7fb1501593c0 2023-03-03T11:14:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.952+0000 7fb14e7fc700 1 --2- 172.21.15.102:0/2013848857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1501593c0 0x7fb1500cde20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb15008cc70 tx=0x7fb144009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7387740bb6367a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:25.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb14dffb700 1 -- 172.21.15.102:0/2013848857 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144011830 con 0x7fb1501593c0 2023-03-03T11:14:25.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb14dffb700 1 -- 172.21.15.102:0/2013848857 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb144011990 con 0x7fb1501593c0 2023-03-03T11:14:25.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb14dffb700 1 -- 172.21.15.102:0/2013848857 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144011d20 con 0x7fb1501593c0 2023-03-03T11:14:25.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb1557ff700 1 -- 172.21.15.102:0/2013848857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1501593c0 msgr2=0x7fb1500cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb1557ff700 1 --2- 172.21.15.102:0/2013848857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1501593c0 0x7fb1500cde20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb15008cc70 tx=0x7fb144009bd0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb1557ff700 1 -- 172.21.15.102:0/2013848857 shutdown_connections 2023-03-03T11:14:25.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb1557ff700 1 --2- 172.21.15.102:0/2013848857 >> v2:172.21.15.172:3300/0 conn(0x7fb150153290 0x7fb150158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb1557ff700 1 --2- 172.21.15.102:0/2013848857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1501593c0 0x7fb1500cde20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb1557ff700 1 -- 172.21.15.102:0/2013848857 >> 172.21.15.102:0/2013848857 conn(0x7fb150065fb0 msgr2=0x7fb1500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:14:25.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.953+0000 7fb1557ff700 1 -- 172.21.15.102:0/2013848857 shutdown_connections 2023-03-03T11:14:25.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.954+0000 7fb1557ff700 1 -- 172.21.15.102:0/2013848857 wait complete. 2023-03-03T11:14:25.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.955+0000 7fb1557ff700 1 Processor -- start 2023-03-03T11:14:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.956+0000 7fb1557ff700 1 -- start start 2023-03-03T11:14:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb1557ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1500cd8e0 0x7fb1500b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb1557ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb150153290 0x7fb1500b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb1557ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1501593c0 0x7fb1501e3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb1557ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1501e5440 con 0x7fb150153290 2023-03-03T11:14:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb1557ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1501f2af0 con 0x7fb1501593c0 2023-03-03T11:14:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb1557ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1501e4bc0 con 0x7fb1500cd8e0 2023-03-03T11:14:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb150153290 0x7fb1500b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb150153290 0x7fb1500b4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37778/0 (socket says 172.21.15.102:37778) 2023-03-03T11:14:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 -- 172.21.15.102:0/1433569429 learned_addr learned my addr 172.21.15.102:0/1433569429 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:14:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14effd700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1500cd8e0 0x7fb1500b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14f7fe700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1501593c0 0x7fb1501e3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 -- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1500cd8e0 msgr2=0x7fb1500b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1500cd8e0 0x7fb1500b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 -- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1501593c0 msgr2=0x7fb1501e3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1501593c0 0x7fb1501e3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.957+0000 7fb14e7fc700 1 -- 172.21.15.102:0/1433569429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144009550 con 0x7fb150153290 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.958+0000 7fb14e7fc700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb150153290 0x7fb1500b4280 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb144000c00 tx=0x7fb144004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.958+0000 7fb1377fe700 1 -- 172.21.15.102:0/1433569429 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144004380 con 0x7fb150153290 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.958+0000 7fb1377fe700 1 -- 172.21.15.102:0/1433569429 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14401f070 con 0x7fb150153290 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.958+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1500ccde0 con 0x7fb150153290 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.958+0000 7fb1377fe700 1 -- 172.21.15.102:0/1433569429 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1440036a0 con 0x7fb150153290 2023-03-03T11:14:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.958+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15015d160 con 0x7fb150153290 2023-03-03T11:14:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.959+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1500cd0c0 con 0x7fb150153290 2023-03-03T11:14:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.961+0000 7fb1377fe700 1 -- 172.21.15.102:0/1433569429 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb14402a020 con 0x7fb150153290 2023-03-03T11:14:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.962+0000 7fb1377fe700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb13c0682f0 0x7fb13c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.962+0000 7fb1377fe700 1 -- 172.21.15.102:0/1433569429 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb144087390 con 0x7fb150153290 2023-03-03T11:14:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.962+0000 7fb14effd700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb13c0682f0 0x7fb13c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.963+0000 7fb14effd700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb13c0682f0 0x7fb13c06a7a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb140009be0 tx=0x7fb140009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:25.967+0000 7fb1377fe700 1 -- 172.21.15.102:0/1433569429 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb144053a20 con 0x7fb150153290 2023-03-03T11:14:26.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.200+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb15004e6e0 con 0x7fb150153290 2023-03-03T11:14:26.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.200+0000 7fb1377fe700 1 -- 172.21.15.102:0/1433569429 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb144057040 con 0x7fb150153290 2023-03-03T11:14:26.202 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:14:26.202 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:14:26.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb13c0682f0 msgr2=0x7fb13c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:26.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb13c0682f0 0x7fb13c06a7a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb140009be0 tx=0x7fb140009210 comp rx=0 tx=0).stop 2023-03-03T11:14:26.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb150153290 msgr2=0x7fb1500b4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:26.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb150153290 0x7fb1500b4280 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb144000c00 tx=0x7fb144004750 comp rx=0 tx=0).stop 2023-03-03T11:14:26.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 shutdown_connections 2023-03-03T11:14:26.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb13c0682f0 0x7fb13c06a7a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:26.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1500cd8e0 0x7fb1500b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:26.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1501593c0 0x7fb1501e3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:26.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 --2- 172.21.15.102:0/1433569429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb150153290 0x7fb1500b4280 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:26.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 >> 172.21.15.102:0/1433569429 conn(0x7fb150065fb0 msgr2=0x7fb1500cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:14:26.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 shutdown_connections 2023-03-03T11:14:26.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:14:26.203+0000 7fb1557ff700 1 -- 172.21.15.102:0/1433569429 wait complete. 2023-03-03T11:14:26.207 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:14:26.232 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T11:14:31.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:31.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:01.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:01.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:31.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:31.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:01.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:01.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:20.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:16:20.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.480+0000 7fc3d1b9d700 1 Processor -- start 2023-03-03T11:16:20.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.480+0000 7fc3d1b9d700 1 -- start start 2023-03-03T11:16:20.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.481+0000 7fc3d1b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0ced00 0x7fc3cc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:20.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.481+0000 7fc3d1b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc0c3d20 con 0x7fc3cc0b3970 2023-03-03T11:16:20.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.481+0000 7fc3d1b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc15cba0 con 0x7fc3cc0c41b0 2023-03-03T11:16:20.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.481+0000 7fc3d1b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc0c4930 con 0x7fc3cc0ced00 2023-03-03T11:16:20.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3cbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc3cc0b3970 0x7fc3cc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57064/0 (socket says 172.21.15.102:57064) 2023-03-03T11:16:20.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3cbfff700 1 -- 172.21.15.102:0/270766695 learned_addr learned my addr 172.21.15.102:0/270766695 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:16:20.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3d0b9b700 1 --2- 172.21.15.102:0/270766695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0ced00 0x7fc3cc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3cb7fe700 1 -- 172.21.15.102:0/270766695 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207624421 0 0) 0x7fc3cc0c3d20 con 0x7fc3cc0b3970 2023-03-03T11:16:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3cb7fe700 1 -- 172.21.15.102:0/270766695 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3bc0035b0 con 0x7fc3cc0b3970 2023-03-03T11:16:20.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3cb7fe700 1 -- 172.21.15.102:0/270766695 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091006824 0 0) 0x7fc3cc15cba0 con 0x7fc3cc0c41b0 2023-03-03T11:16:20.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3d0b9b700 1 -- 172.21.15.102:0/270766695 >> v1:172.21.15.172:6789/0 conn(0x7fc3cc0c41b0 legacy=0x7fc3cc163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:20.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3d0b9b700 1 -- 172.21.15.102:0/270766695 >> v1:172.21.15.102:6789/0 conn(0x7fc3cc0b3970 legacy=0x7fc3cc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:20.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.482+0000 7fc3d0b9b700 1 -- 172.21.15.102:0/270766695 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc15ce80 con 0x7fc3cc0ced00 2023-03-03T11:16:20.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3d0b9b700 1 --2- 172.21.15.102:0/270766695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0ced00 0x7fc3cc0c5a90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009870 tx=0x7fc3c0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a35ba6c21ed62ed5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:20.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3cb7fe700 1 -- 172.21.15.102:0/270766695 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0011830 con 0x7fc3cc0ced00 2023-03-03T11:16:20.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3cb7fe700 1 -- 172.21.15.102:0/270766695 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0011990 con 0x7fc3cc0ced00 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3cb7fe700 1 -- 172.21.15.102:0/270766695 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0011d20 con 0x7fc3cc0ced00 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/270766695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0ced00 msgr2=0x7fc3cc0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/270766695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0ced00 0x7fc3cc0c5a90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009870 tx=0x7fc3c0009b80 comp rx=0 tx=0).stop 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/270766695 shutdown_connections 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/270766695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0ced00 0x7fc3cc0c5a90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.483+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/270766695 >> 172.21.15.102:0/270766695 conn(0x7fc3cc08b1c0 msgr2=0x7fc3cc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.484+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/270766695 shutdown_connections 2023-03-03T11:16:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.484+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/270766695 wait complete. 2023-03-03T11:16:20.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.485+0000 7fc3d1b9d700 1 Processor -- start 2023-03-03T11:16:20.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.486+0000 7fc3d1b9d700 1 -- start start 2023-03-03T11:16:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d1b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3cc0b3970 0x7fc3cc1e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d1b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0c41b0 0x7fc3cc1ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d1b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3cc0ced00 0x7fc3cc1eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d1b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1f2900 con 0x7fc3cc0ced00 2023-03-03T11:16:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d1b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1f1aa0 con 0x7fc3cc0b3970 2023-03-03T11:16:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d1b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1f1d80 con 0x7fc3cc0c41b0 2023-03-03T11:16:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d139c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3cc0ced00 0x7fc3cc1eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d139c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3cc0ced00 0x7fc3cc1eed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49478/0 (socket says 172.21.15.102:49478) 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d139c700 1 -- 172.21.15.102:0/604575415 learned_addr learned my addr 172.21.15.102:0/604575415 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d0b9b700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3cc0b3970 0x7fc3cc1e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3cbfff700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0c41b0 0x7fc3cc1ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d139c700 1 -- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0c41b0 msgr2=0x7fc3cc1ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.487+0000 7fc3d139c700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0c41b0 0x7fc3cc1ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3d139c700 1 -- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3cc0b3970 msgr2=0x7fc3cc1e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3d139c700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3cc0b3970 0x7fc3cc1e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3d139c700 1 -- 172.21.15.102:0/604575415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c00094a0 con 0x7fc3cc0ced00 2023-03-03T11:16:20.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3d0b9b700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3cc0b3970 0x7fc3cc1e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:16:20.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3d139c700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3cc0ced00 0x7fc3cc1eed70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400cbc0 tx=0x7fc3c400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:20.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3c97fa700 1 -- 172.21.15.102:0/604575415 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4003e80 con 0x7fc3cc0ced00 2023-03-03T11:16:20.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3c97fa700 1 -- 172.21.15.102:0/604575415 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c4005070 con 0x7fc3cc0ced00 2023-03-03T11:16:20.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3c97fa700 1 -- 172.21.15.102:0/604575415 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4005420 con 0x7fc3cc0ced00 2023-03-03T11:16:20.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3cc06c200 con 0x7fc3cc0ced00 2023-03-03T11:16:20.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.488+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3cc06bf20 con 0x7fc3cc0ced00 2023-03-03T11:16:20.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.490+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc15c890 con 0x7fc3cc0ced00 2023-03-03T11:16:20.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.491+0000 7fc3c97fa700 1 -- 172.21.15.102:0/604575415 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3c4019030 con 0x7fc3cc0ced00 2023-03-03T11:16:20.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.492+0000 7fc3c97fa700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3b00682f0 0x7fc3b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:20.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.492+0000 7fc3c97fa700 1 -- 172.21.15.102:0/604575415 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc3c4085690 con 0x7fc3cc0ced00 2023-03-03T11:16:20.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.492+0000 7fc3d0b9b700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3b00682f0 0x7fc3b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:20.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.493+0000 7fc3d0b9b700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3b00682f0 0x7fc3b006a7a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00097c0 tx=0x7fc3c001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:20.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.496+0000 7fc3c97fa700 1 -- 172.21.15.102:0/604575415 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3c4051d10 con 0x7fc3cc0ced00 2023-03-03T11:16:20.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.735+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3cc06b4d0 con 0x7fc3cc0ced00 2023-03-03T11:16:20.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.736+0000 7fc3c97fa700 1 -- 172.21.15.102:0/604575415 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc3c4017070 con 0x7fc3cc0ced00 2023-03-03T11:16:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:16:20.739 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:16:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.738+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3b00682f0 msgr2=0x7fc3b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.738+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3b00682f0 0x7fc3b006a7a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00097c0 tx=0x7fc3c001c040 comp rx=0 tx=0).stop 2023-03-03T11:16:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.738+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3cc0ced00 msgr2=0x7fc3cc1eed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.738+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3cc0ced00 0x7fc3cc1eed70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400cbc0 tx=0x7fc3c400ced0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 shutdown_connections 2023-03-03T11:16:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3cc0ced00 0x7fc3cc1eed70 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3cc0b3970 0x7fc3cc1e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc3b00682f0 0x7fc3b006a7a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 --2- 172.21.15.102:0/604575415 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc3cc0c41b0 0x7fc3cc1ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 >> 172.21.15.102:0/604575415 conn(0x7fc3cc08b1c0 msgr2=0x7fc3cc066840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 shutdown_connections 2023-03-03T11:16:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:16:20.739+0000 7fc3d1b9d700 1 -- 172.21.15.102:0/604575415 wait complete. 2023-03-03T11:16:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:16:20.768 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T11:16:31.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:31.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:02.012 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:02.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:32.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:32.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:57.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:17:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.016+0000 7f2492fff700 1 Processor -- start 2023-03-03T11:17:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.016+0000 7f2492fff700 1 -- start start 2023-03-03T11:17:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.017+0000 7f2492fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f248c0cd8e0 0x7f248c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.017+0000 7f2492fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd390 0x7f248c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.017+0000 7f2492fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c153660 con 0x7f248c153290 2023-03-03T11:17:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.017+0000 7f2492fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f248c06c360 con 0x7f248c0cd8e0 2023-03-03T11:17:58.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.017+0000 7f2492fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248c15cea0 con 0x7f248c0cd390 2023-03-03T11:17:58.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.017+0000 7f2490d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f248c153290 0x7f248c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54676/0 (socket says 172.21.15.102:54676) 2023-03-03T11:17:58.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.017+0000 7f2490d9b700 1 -- 172.21.15.102:0/2679944202 learned_addr learned my addr 172.21.15.102:0/2679944202 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:17:58.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f248bfff700 1 --2- 172.21.15.102:0/2679944202 >> v2:172.21.15.172:3300/0 conn(0x7f248c0cd8e0 0x7f248c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:58.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f249159c700 1 --2- 172.21.15.102:0/2679944202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd390 0x7f248c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:58.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f248b7fe700 1 -- 172.21.15.102:0/2679944202 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300603955 0 0) 0x7f248c153660 con 0x7f248c153290 2023-03-03T11:17:58.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f248b7fe700 1 -- 172.21.15.102:0/2679944202 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247c0035b0 con 0x7f248c153290 2023-03-03T11:17:58.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f249159c700 1 -- 172.21.15.102:0/2679944202 >> v2:172.21.15.172:3300/0 conn(0x7f248c0cd8e0 msgr2=0x7f248c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:17:58.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f249159c700 1 --2- 172.21.15.102:0/2679944202 >> v2:172.21.15.172:3300/0 conn(0x7f248c0cd8e0 0x7f248c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f249159c700 1 -- 172.21.15.102:0/2679944202 >> v1:172.21.15.172:6789/0 conn(0x7f248c153290 legacy=0x7f248c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:58.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f249159c700 1 -- 172.21.15.102:0/2679944202 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c06c7f0 con 0x7f248c0cd390 2023-03-03T11:17:58.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.018+0000 7f248bfff700 1 --2- 172.21.15.102:0/2679944202 >> v2:172.21.15.172:3300/0 conn(0x7f248c0cd8e0 0x7f248c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:17:58.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f249159c700 1 --2- 172.21.15.102:0/2679944202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd390 0x7f248c0ce360 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2484009d90 tx=0x7f248400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d6c63be1870ef5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:58.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f248b7fe700 1 -- 172.21.15.102:0/2679944202 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248400e070 con 0x7f248c0cd390 2023-03-03T11:17:58.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f248b7fe700 1 -- 172.21.15.102:0/2679944202 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248400bb20 con 0x7f248c0cd390 2023-03-03T11:17:58.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f248b7fe700 1 -- 172.21.15.102:0/2679944202 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484012430 con 0x7f248c0cd390 2023-03-03T11:17:58.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f2492fff700 1 -- 172.21.15.102:0/2679944202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd390 msgr2=0x7f248c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:58.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f2492fff700 1 --2- 172.21.15.102:0/2679944202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd390 0x7f248c0ce360 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2484009d90 tx=0x7f248400b330 comp rx=0 tx=0).stop 2023-03-03T11:17:58.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f2492fff700 1 -- 172.21.15.102:0/2679944202 shutdown_connections 2023-03-03T11:17:58.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f2492fff700 1 --2- 172.21.15.102:0/2679944202 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd390 0x7f248c0ce360 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f2492fff700 1 --2- 172.21.15.102:0/2679944202 >> v2:172.21.15.172:3300/0 conn(0x7f248c0cd8e0 0x7f248c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.019+0000 7f2492fff700 1 -- 172.21.15.102:0/2679944202 >> 172.21.15.102:0/2679944202 conn(0x7f248c065fb0 msgr2=0x7f248c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:58.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.020+0000 7f2492fff700 1 -- 172.21.15.102:0/2679944202 shutdown_connections 2023-03-03T11:17:58.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.020+0000 7f2492fff700 1 -- 172.21.15.102:0/2679944202 wait complete. 2023-03-03T11:17:58.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.022+0000 7f2492fff700 1 Processor -- start 2023-03-03T11:17:58.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.022+0000 7f2492fff700 1 -- start start 2023-03-03T11:17:58.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f2492fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f248c0cd390 0x7f248c0b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:58.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f2492fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd8e0 0x7f248c0b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:58.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f2492fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f248c153290 0x7f248c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:58.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f2492fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0b1ed0 con 0x7f248c0cd390 2023-03-03T11:17:58.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f2492fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c155f50 con 0x7f248c153290 2023-03-03T11:17:58.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f2492fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248c158460 con 0x7f248c0cd8e0 2023-03-03T11:17:58.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f2490d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f248c0cd390 0x7f248c0b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:58.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f248bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd8e0 0x7f248c0b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:58.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f249159c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f248c153290 0x7f248c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:58.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f248bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd8e0 0x7f248c0b22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:47462/0 (socket says 172.21.15.102:47462) 2023-03-03T11:17:58.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f248bfff700 1 -- 172.21.15.102:0/4043371084 learned_addr learned my addr 172.21.15.102:0/4043371084 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:17:58.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f248bfff700 1 -- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f248c153290 msgr2=0x7f248c0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:58.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f248bfff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f248c153290 0x7f248c0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f248bfff700 1 -- 172.21.15.102:0/4043371084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f248c0cd390 msgr2=0x7f248c0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:58.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.023+0000 7f248bfff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f248c0cd390 0x7f248c0b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.024+0000 7f248bfff700 1 -- 172.21.15.102:0/4043371084 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780094a0 con 0x7f248c0cd8e0 2023-03-03T11:17:58.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.024+0000 7f248bfff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd8e0 0x7f248c0b22f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f247800ec10 tx=0x7f247800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:58.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.024+0000 7f24897fa700 1 -- 172.21.15.102:0/4043371084 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478009760 con 0x7f248c0cd8e0 2023-03-03T11:17:58.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.024+0000 7f24897fa700 1 -- 172.21.15.102:0/4043371084 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247800c850 con 0x7f248c0cd8e0 2023-03-03T11:17:58.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.024+0000 7f24897fa700 1 -- 172.21.15.102:0/4043371084 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247800cbe0 con 0x7f248c0cd8e0 2023-03-03T11:17:58.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.024+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2484009a70 con 0x7f248c0cd8e0 2023-03-03T11:17:58.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.024+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c0ccde0 con 0x7f248c0cd8e0 2023-03-03T11:17:58.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.026+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248c14ffd0 con 0x7f248c0cd8e0 2023-03-03T11:17:58.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.027+0000 7f24897fa700 1 -- 172.21.15.102:0/4043371084 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2478019030 con 0x7f248c0cd8e0 2023-03-03T11:17:58.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.028+0000 7f24897fa700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2470068320 0x7f247006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:58.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.028+0000 7f24897fa700 1 -- 172.21.15.102:0/4043371084 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2478085cd0 con 0x7f248c0cd8e0 2023-03-03T11:17:58.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.028+0000 7f2490d9b700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2470068320 0x7f247006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:58.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.029+0000 7f2490d9b700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2470068320 0x7f247006a7d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2480007fd0 tx=0x7f2480007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:58.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.032+0000 7f24897fa700 1 -- 172.21.15.102:0/4043371084 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24780522d0 con 0x7f248c0cd8e0 2023-03-03T11:17:58.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.266+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f248c0cd0c0 con 0x7f248c0cd8e0 2023-03-03T11:17:58.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.267+0000 7f24897fa700 1 -- 172.21.15.102:0/4043371084 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2478021090 con 0x7f248c0cd8e0 2023-03-03T11:17:58.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:17:58.270 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:17:58.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2470068320 msgr2=0x7f247006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:58.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2470068320 0x7f247006a7d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2480007fd0 tx=0x7f2480007dc0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd8e0 msgr2=0x7f248c0b22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd8e0 0x7f248c0b22f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f247800ec10 tx=0x7f247800ef20 comp rx=0 tx=0).stop 2023-03-03T11:17:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 shutdown_connections 2023-03-03T11:17:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f248c153290 0x7f248c0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f248c0cd390 0x7f248c0b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2470068320 0x7f247006a7d0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 --2- 172.21.15.102:0/4043371084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f248c0cd8e0 0x7f248c0b22f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 >> 172.21.15.102:0/4043371084 conn(0x7f248c065fb0 msgr2=0x7f248c1516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 shutdown_connections 2023-03-03T11:17:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:17:58.270+0000 7f2492fff700 1 -- 172.21.15.102:0/4043371084 wait complete. 2023-03-03T11:17:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:17:58.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-03-03T11:18:02.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:02.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:32.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:32.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:34.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:18:34.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.563+0000 7f8b413ff700 1 Processor -- start 2023-03-03T11:18:34.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.564+0000 7f8b413ff700 1 -- start start 2023-03-03T11:18:34.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b413ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd8e0 0x7f8b3c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:34.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b413ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b3c0cd390 0x7f8b3c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:34.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b413ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c153660 con 0x7f8b3c153290 2023-03-03T11:18:34.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b413ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c06c360 con 0x7f8b3c0cd390 2023-03-03T11:18:34.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b413ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15cea0 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b3b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b3c0cd390 0x7f8b3c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b3affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b3c153290 0x7f8b3c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43152/0 (socket says 172.21.15.102:43152) 2023-03-03T11:18:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b3b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b3c0cd390 0x7f8b3c0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55640/0 (socket says 172.21.15.102:55640) 2023-03-03T11:18:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b3affd700 1 -- 172.21.15.102:0/1158176420 learned_addr learned my addr 172.21.15.102:0/1158176420 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:18:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.565+0000 7f8b33fff700 1 --2- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd8e0 0x7f8b3c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3b7fe700 1 -- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd8e0 msgr2=0x7f8b3c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:18:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3b7fe700 1 --2- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd8e0 0x7f8b3c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3b7fe700 1 -- 172.21.15.102:0/1158176420 >> v1:172.21.15.102:6789/0 conn(0x7f8b3c153290 legacy=0x7f8b3c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3b7fe700 1 -- 172.21.15.102:0/1158176420 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c06c7f0 con 0x7f8b3c0cd390 2023-03-03T11:18:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3b7fe700 1 --2- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd390 0x7f8b3c0ce360 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009d90 tx=0x7f8b3400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b54c0dea531e917 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:18:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3a7fc700 1 -- 172.21.15.102:0/1158176420 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3400e070 con 0x7f8b3c0cd390 2023-03-03T11:18:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3a7fc700 1 -- 172.21.15.102:0/1158176420 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3400bb20 con 0x7f8b3c0cd390 2023-03-03T11:18:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.566+0000 7f8b3a7fc700 1 -- 172.21.15.102:0/1158176420 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34012430 con 0x7f8b3c0cd390 2023-03-03T11:18:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 -- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd390 msgr2=0x7f8b3c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 --2- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd390 0x7f8b3c0ce360 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009d90 tx=0x7f8b3400b330 comp rx=0 tx=0).stop 2023-03-03T11:18:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 -- 172.21.15.102:0/1158176420 shutdown_connections 2023-03-03T11:18:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 --2- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd390 0x7f8b3c0ce360 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 --2- 172.21.15.102:0/1158176420 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd8e0 0x7f8b3c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 -- 172.21.15.102:0/1158176420 >> 172.21.15.102:0/1158176420 conn(0x7f8b3c065fb0 msgr2=0x7f8b3c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:18:34.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 -- 172.21.15.102:0/1158176420 shutdown_connections 2023-03-03T11:18:34.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.567+0000 7f8b413ff700 1 -- 172.21.15.102:0/1158176420 wait complete. 2023-03-03T11:18:34.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.569+0000 7f8b413ff700 1 Processor -- start 2023-03-03T11:18:34.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.570+0000 7f8b413ff700 1 -- start start 2023-03-03T11:18:34.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b413ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd390 0x7f8b3c0c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:34.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b413ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd8e0 0x7f8b3c1f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b413ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c153290 0x7f8b3c1f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b413ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0c4460 con 0x7f8b3c153290 2023-03-03T11:18:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b413ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c1f2ce0 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b413ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c1f2a70 con 0x7f8b3c0cd390 2023-03-03T11:18:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b3b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c153290 0x7f8b3c1f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b33fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd8e0 0x7f8b3c1f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b3affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd390 0x7f8b3c0c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b33fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd8e0 0x7f8b3c1f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55646/0 (socket says 172.21.15.102:55646) 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.571+0000 7f8b33fff700 1 -- 172.21.15.102:0/937885367 learned_addr learned my addr 172.21.15.102:0/937885367 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b33fff700 1 -- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd390 msgr2=0x7f8b3c0c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b33fff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd390 0x7f8b3c0c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b33fff700 1 -- 172.21.15.102:0/937885367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c153290 msgr2=0x7f8b3c1f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b33fff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c153290 0x7f8b3c1f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b33fff700 1 -- 172.21.15.102:0/937885367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b34009a70 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b3b7fe700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c153290 0x7f8b3c1f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:18:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b33fff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd8e0 0x7f8b3c1f1ff0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400ea10 tx=0x7f8b2400ed20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:18:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b337fe700 1 -- 172.21.15.102:0/937885367 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24009710 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b337fe700 1 -- 172.21.15.102:0/937885367 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2400eeb0 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c0ccde0 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.572+0000 7f8b337fe700 1 -- 172.21.15.102:0/937885367 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2400ca50 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.573+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c15d160 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.574+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c0cd0c0 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.575+0000 7f8b337fe700 1 -- 172.21.15.102:0/937885367 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b24019030 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.576+0000 7f8b337fe700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b1c0682f0 0x7f8b1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.576+0000 7f8b337fe700 1 -- 172.21.15.102:0/937885367 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8b24085c70 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.576+0000 7f8b3affd700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b1c0682f0 0x7f8b1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.577+0000 7f8b3affd700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b1c0682f0 0x7f8b1c06a7a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c005910 tx=0x7f8b2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:18:34.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.580+0000 7f8b337fe700 1 -- 172.21.15.102:0/937885367 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b24052380 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.814+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b3c04e6e0 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.816+0000 7f8b337fe700 1 -- 172.21.15.102:0/937885367 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8b240559a0 con 0x7f8b3c0cd8e0 2023-03-03T11:18:34.818 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:18:34.818 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:18:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b1c0682f0 msgr2=0x7f8b1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b1c0682f0 0x7f8b1c06a7a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c005910 tx=0x7f8b2c009040 comp rx=0 tx=0).stop 2023-03-03T11:18:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd8e0 msgr2=0x7f8b3c1f1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd8e0 0x7f8b3c1f1ff0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400ea10 tx=0x7f8b2400ed20 comp rx=0 tx=0).stop 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 shutdown_connections 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b1c0682f0 0x7f8b1c06a7a0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b3c0cd390 0x7f8b3c0c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b3c153290 0x7f8b3c1f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 --2- 172.21.15.102:0/937885367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b3c0cd8e0 0x7f8b3c1f1ff0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 >> 172.21.15.102:0/937885367 conn(0x7f8b3c065fb0 msgr2=0x7f8b3c15b160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 shutdown_connections 2023-03-03T11:18:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:18:34.818+0000 7f8b413ff700 1 -- 172.21.15.102:0/937885367 wait complete. 2023-03-03T11:18:34.823 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:18:34.848 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-03-03T11:19:02.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:02.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:32.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:32.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:47.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:19:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.097+0000 7f14475ff700 1 Processor -- start 2023-03-03T11:19:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.097+0000 7f14475ff700 1 -- start start 2023-03-03T11:19:48.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.098+0000 7f14475ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1440153290 0x7f1440158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:48.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.098+0000 7f14475ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14401593c0 0x7f14400cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:48.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f14475ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1440153660 con 0x7f14400cd8e0 2023-03-03T11:19:48.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f14475ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144006c360 con 0x7f14401593c0 2023-03-03T11:19:48.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f14475ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144015cea0 con 0x7f1440153290 2023-03-03T11:19:48.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1444b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14401593c0 0x7f14400cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:48.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f144539b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1440153290 0x7f1440158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:48.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1445b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f14400cd8e0 0x7f14400ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45900/0 (socket says 172.21.15.102:45900) 2023-03-03T11:19:48.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1445b9c700 1 -- 172.21.15.102:0/3825340974 learned_addr learned my addr 172.21.15.102:0/3825340974 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:19:48.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1444b9a700 1 -- 172.21.15.102:0/3825340974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1440153290 msgr2=0x7f1440158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:48.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1444b9a700 1 --2- 172.21.15.102:0/3825340974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1440153290 0x7f1440158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1444b9a700 1 -- 172.21.15.102:0/3825340974 >> v1:172.21.15.102:6789/0 conn(0x7f14400cd8e0 legacy=0x7f14400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1444b9a700 1 -- 172.21.15.102:0/3825340974 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144006c7f0 con 0x7f14401593c0 2023-03-03T11:19:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.099+0000 7f1437fff700 1 -- 172.21.15.102:0/3825340974 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595002949 0 0) 0x7f1440153660 con 0x7f14400cd8e0 2023-03-03T11:19:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.100+0000 7f1444b9a700 1 --2- 172.21.15.102:0/3825340974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14401593c0 0x7f14400cde20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1430009840 tx=0x7f1430009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d587f61d4c41a88c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.100+0000 7f1437fff700 1 -- 172.21.15.102:0/3825340974 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430011830 con 0x7f14401593c0 2023-03-03T11:19:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.100+0000 7f1437fff700 1 -- 172.21.15.102:0/3825340974 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1430011990 con 0x7f14401593c0 2023-03-03T11:19:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.100+0000 7f1437fff700 1 -- 172.21.15.102:0/3825340974 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430011d20 con 0x7f14401593c0 2023-03-03T11:19:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 -- 172.21.15.102:0/3825340974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14401593c0 msgr2=0x7f14400cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 --2- 172.21.15.102:0/3825340974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14401593c0 0x7f14400cde20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1430009840 tx=0x7f1430009b50 comp rx=0 tx=0).stop 2023-03-03T11:19:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 -- 172.21.15.102:0/3825340974 shutdown_connections 2023-03-03T11:19:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 --2- 172.21.15.102:0/3825340974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1440153290 0x7f1440158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 --2- 172.21.15.102:0/3825340974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14401593c0 0x7f14400cde20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 -- 172.21.15.102:0/3825340974 >> 172.21.15.102:0/3825340974 conn(0x7f1440065fb0 msgr2=0x7f14400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 -- 172.21.15.102:0/3825340974 shutdown_connections 2023-03-03T11:19:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.101+0000 7f14475ff700 1 -- 172.21.15.102:0/3825340974 wait complete. 2023-03-03T11:19:48.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.103+0000 7f14475ff700 1 Processor -- start 2023-03-03T11:19:48.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.103+0000 7f14475ff700 1 -- start start 2023-03-03T11:19:48.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f14475ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14400cd8e0 0x7f14400cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:48.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f14475ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1440153290 0x7f1440154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:48.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f14475ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14401593c0 0x7f144014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:48.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f14475ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1440153f50 con 0x7f1440153290 2023-03-03T11:19:48.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f14475ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14401501a0 con 0x7f14401593c0 2023-03-03T11:19:48.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f14475ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14400b2740 con 0x7f14400cd8e0 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f1444b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1440153290 0x7f1440154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f144539b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14400cd8e0 0x7f14400cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.104+0000 7f1444b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1440153290 0x7f1440154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39954/0 (socket says 172.21.15.102:39954) 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1444b9a700 1 -- 172.21.15.102:0/2357654353 learned_addr learned my addr 172.21.15.102:0/2357654353 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1445b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14401593c0 0x7f144014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1444b9a700 1 -- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14400cd8e0 msgr2=0x7f14400cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1444b9a700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14400cd8e0 0x7f14400cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1444b9a700 1 -- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14401593c0 msgr2=0x7f144014fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:48.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1444b9a700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14401593c0 0x7f144014fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1444b9a700 1 -- 172.21.15.102:0/2357654353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14300094a0 con 0x7f1440153290 2023-03-03T11:19:48.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f144539b700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14400cd8e0 0x7f14400cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:19:48.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.105+0000 7f1444b9a700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1440153290 0x7f1440154370 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1430009b90 tx=0x7f1430004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:48.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.106+0000 7f1435ffb700 1 -- 172.21.15.102:0/2357654353 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14300036a0 con 0x7f1440153290 2023-03-03T11:19:48.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.106+0000 7f1435ffb700 1 -- 172.21.15.102:0/2357654353 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143001f070 con 0x7f1440153290 2023-03-03T11:19:48.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.106+0000 7f1435ffb700 1 -- 172.21.15.102:0/2357654353 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430003a40 con 0x7f1440153290 2023-03-03T11:19:48.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.106+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14400ccde0 con 0x7f1440153290 2023-03-03T11:19:48.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.106+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144015d160 con 0x7f1440153290 2023-03-03T11:19:48.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.107+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14400cd0c0 con 0x7f1440153290 2023-03-03T11:19:48.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.108+0000 7f1435ffb700 1 -- 172.21.15.102:0/2357654353 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f143000f050 con 0x7f1440153290 2023-03-03T11:19:48.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.109+0000 7f1435ffb700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14240682f0 0x7f142406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:48.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.109+0000 7f1435ffb700 1 -- 172.21.15.102:0/2357654353 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1430021070 con 0x7f1440153290 2023-03-03T11:19:48.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.110+0000 7f144539b700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14240682f0 0x7f142406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:48.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.110+0000 7f144539b700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14240682f0 0x7f142406a7a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f143c009fd0 tx=0x7f143c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:48.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.114+0000 7f1435ffb700 1 -- 172.21.15.102:0/2357654353 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1430053d60 con 0x7f1440153290 2023-03-03T11:19:48.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.344+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f144004e6e0 con 0x7f1440153290 2023-03-03T11:19:48.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.345+0000 7f1435ffb700 1 -- 172.21.15.102:0/2357654353 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1430057380 con 0x7f1440153290 2023-03-03T11:19:48.347 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:19:48.347 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:19:48.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.347+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14240682f0 msgr2=0x7f142406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14240682f0 0x7f142406a7a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f143c009fd0 tx=0x7f143c009210 comp rx=0 tx=0).stop 2023-03-03T11:19:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1440153290 msgr2=0x7f1440154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1440153290 0x7f1440154370 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1430009b90 tx=0x7f1430004750 comp rx=0 tx=0).stop 2023-03-03T11:19:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 shutdown_connections 2023-03-03T11:19:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f14240682f0 0x7f142406a7a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f14400cd8e0 0x7f14400cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f14401593c0 0x7f144014fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 --2- 172.21.15.102:0/2357654353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1440153290 0x7f1440154370 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:48.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 >> 172.21.15.102:0/2357654353 conn(0x7f1440065fb0 msgr2=0x7f14401e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:48.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 shutdown_connections 2023-03-03T11:19:48.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:19:48.348+0000 7f14475ff700 1 -- 172.21.15.102:0/2357654353 wait complete. 2023-03-03T11:19:48.351 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:19:48.377 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T11:20:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T11:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:20:03.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:03.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:16.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:20:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.629+0000 7fe59119d700 1 Processor -- start 2023-03-03T11:20:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.629+0000 7fe59119d700 1 -- start start 2023-03-03T11:20:16.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.631+0000 7fe59119d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe58c0ced00 0x7fe58c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:20:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.631+0000 7fe59119d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58c0c3d20 con 0x7fe58c0b3970 2023-03-03T11:20:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.631+0000 7fe59119d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58c15cba0 con 0x7fe58c0c41b0 2023-03-03T11:20:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.631+0000 7fe59119d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58c0c4930 con 0x7fe58c0ced00 2023-03-03T11:20:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.631+0000 7fe58b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe58c0b3970 0x7fe58c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56964/0 (socket says 172.21.15.102:56964) 2023-03-03T11:20:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.631+0000 7fe58b7fe700 1 -- 172.21.15.102:0/3499528090 learned_addr learned my addr 172.21.15.102:0/3499528090 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:20:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.631+0000 7fe58bfff700 1 --2- 172.21.15.102:0/3499528090 >> v2:172.21.15.172:3300/0 conn(0x7fe58c0ced00 0x7fe58c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:20:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58affd700 1 -- 172.21.15.102:0/3499528090 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122024075 0 0) 0x7fe58c0c3d20 con 0x7fe58c0b3970 2023-03-03T11:20:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58affd700 1 -- 172.21.15.102:0/3499528090 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5740035b0 con 0x7fe58c0b3970 2023-03-03T11:20:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58affd700 1 -- 172.21.15.102:0/3499528090 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252479625 0 0) 0x7fe58c15cba0 con 0x7fe58c0c41b0 2023-03-03T11:20:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58affd700 1 -- 172.21.15.102:0/3499528090 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c0c3d20 con 0x7fe58c0c41b0 2023-03-03T11:20:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58bfff700 1 -- 172.21.15.102:0/3499528090 >> v1:172.21.15.172:6789/0 conn(0x7fe58c0c41b0 legacy=0x7fe58c163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:20:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58bfff700 1 -- 172.21.15.102:0/3499528090 >> v1:172.21.15.102:6789/0 conn(0x7fe58c0b3970 legacy=0x7fe58c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:20:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58bfff700 1 -- 172.21.15.102:0/3499528090 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c15ce80 con 0x7fe58c0ced00 2023-03-03T11:20:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58bfff700 1 --2- 172.21.15.102:0/3499528090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 0x7fe58c0c5a90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009840 tx=0x7fe57c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6bb190c8fe96bbd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:20:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58affd700 1 -- 172.21.15.102:0/3499528090 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c011830 con 0x7fe58c0ced00 2023-03-03T11:20:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58affd700 1 -- 172.21.15.102:0/3499528090 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57c011990 con 0x7fe58c0ced00 2023-03-03T11:20:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.632+0000 7fe58affd700 1 -- 172.21.15.102:0/3499528090 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c011d20 con 0x7fe58c0ced00 2023-03-03T11:20:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.633+0000 7fe59119d700 1 -- 172.21.15.102:0/3499528090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 msgr2=0x7fe58c0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:20:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.633+0000 7fe59119d700 1 --2- 172.21.15.102:0/3499528090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 0x7fe58c0c5a90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009840 tx=0x7fe57c009b50 comp rx=0 tx=0).stop 2023-03-03T11:20:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.633+0000 7fe59119d700 1 -- 172.21.15.102:0/3499528090 shutdown_connections 2023-03-03T11:20:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.633+0000 7fe59119d700 1 --2- 172.21.15.102:0/3499528090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 0x7fe58c0c5a90 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:20:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.633+0000 7fe59119d700 1 -- 172.21.15.102:0/3499528090 >> 172.21.15.102:0/3499528090 conn(0x7fe58c08b1c0 msgr2=0x7fe58c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:20:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.633+0000 7fe59119d700 1 -- 172.21.15.102:0/3499528090 shutdown_connections 2023-03-03T11:20:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.633+0000 7fe59119d700 1 -- 172.21.15.102:0/3499528090 wait complete. 2023-03-03T11:20:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.635+0000 7fe59119d700 1 Processor -- start 2023-03-03T11:20:16.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.635+0000 7fe59119d700 1 -- start start 2023-03-03T11:20:16.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.636+0000 7fe59119d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe58c0b3970 0x7fe58c0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:20:16.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.636+0000 7fe59119d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe58c0c41b0 0x7fe58c1f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:20:16.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.636+0000 7fe59119d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 0x7fe58c1f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:20:16.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.636+0000 7fe59119d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c14cef0 con 0x7fe58c0b3970 2023-03-03T11:20:16.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.636+0000 7fe59119d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c149460 con 0x7fe58c0ced00 2023-03-03T11:20:16.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.636+0000 7fe59119d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c14e3c0 con 0x7fe58c0c41b0 2023-03-03T11:20:16.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe58c0b3970 0x7fe58c0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:20:16.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe58c0b3970 0x7fe58c0ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60148/0 (socket says 172.21.15.102:60148) 2023-03-03T11:20:16.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe59099c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 0x7fe58c1f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:20:16.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 -- 172.21.15.102:0/1997539168 learned_addr learned my addr 172.21.15.102:0/1997539168 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:20:16.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58b7fe700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe58c0c41b0 0x7fe58c1f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:20:16.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 -- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe58c0c41b0 msgr2=0x7fe58c1f17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:20:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe58c0c41b0 0x7fe58c1f17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:20:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 -- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 msgr2=0x7fe58c1f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:20:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 0x7fe58c1f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:20:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 -- 172.21.15.102:0/1997539168 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c0094a0 con 0x7fe58c0b3970 2023-03-03T11:20:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.637+0000 7fe58bfff700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe58c0b3970 0x7fe58c0ce000 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0090b0 tx=0x7fe57c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:20:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.638+0000 7fe588ff9700 1 -- 172.21.15.102:0/1997539168 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c01a6e0 con 0x7fe58c0b3970 2023-03-03T11:20:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.638+0000 7fe588ff9700 1 -- 172.21.15.102:0/1997539168 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57c01a840 con 0x7fe58c0b3970 2023-03-03T11:20:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.638+0000 7fe588ff9700 1 -- 172.21.15.102:0/1997539168 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c01abb0 con 0x7fe58c0b3970 2023-03-03T11:20:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.638+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c06c200 con 0x7fe58c0b3970 2023-03-03T11:20:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.638+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c06bf20 con 0x7fe58c0b3970 2023-03-03T11:20:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.639+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c15c890 con 0x7fe58c0b3970 2023-03-03T11:20:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.640+0000 7fe588ff9700 1 -- 172.21.15.102:0/1997539168 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe57c00f050 con 0x7fe58c0b3970 2023-03-03T11:20:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.642+0000 7fe588ff9700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe578068320 0x7fe57806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:20:16.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.642+0000 7fe588ff9700 1 -- 172.21.15.102:0/1997539168 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe57c086a60 con 0x7fe58c0b3970 2023-03-03T11:20:16.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.642+0000 7fe58b7fe700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe578068320 0x7fe57806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:20:16.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.643+0000 7fe58b7fe700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe578068320 0x7fe57806a7d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe58c151260 tx=0x7fe580007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:20:16.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.646+0000 7fe588ff9700 1 -- 172.21.15.102:0/1997539168 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe57c053220 con 0x7fe58c0b3970 2023-03-03T11:20:16.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.882+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe58c0ca200 con 0x7fe58c0b3970 2023-03-03T11:20:16.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.883+0000 7fe588ff9700 1 -- 172.21.15.102:0/1997539168 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe57c029090 con 0x7fe58c0b3970 2023-03-03T11:20:16.886 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:20:16.886 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:20:16.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe578068320 msgr2=0x7fe57806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:20:16.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe578068320 0x7fe57806a7d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe58c151260 tx=0x7fe580007e00 comp rx=0 tx=0).stop 2023-03-03T11:20:16.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe58c0b3970 msgr2=0x7fe58c0ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:20:16.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe58c0b3970 0x7fe58c0ce000 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0090b0 tx=0x7fe57c004750 comp rx=0 tx=0).stop 2023-03-03T11:20:16.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 shutdown_connections 2023-03-03T11:20:16.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe58c0ced00 0x7fe58c1f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:20:16.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe58c0b3970 0x7fe58c0ce000 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:20:16.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe578068320 0x7fe57806a7d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:20:16.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 --2- 172.21.15.102:0/1997539168 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe58c0c41b0 0x7fe58c1f17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:20:16.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 >> 172.21.15.102:0/1997539168 conn(0x7fe58c08b1c0 msgr2=0x7fe58c0cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:20:16.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 shutdown_connections 2023-03-03T11:20:16.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:20:16.886+0000 7fe59119d700 1 -- 172.21.15.102:0/1997539168 wait complete. 2023-03-03T11:20:16.891 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:20:16.916 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-03-03T11:20:33.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:33.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:03.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:03.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:09.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:21:10.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.180+0000 7fa4de3ff700 1 Processor -- start 2023-03-03T11:21:10.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.180+0000 7fa4de3ff700 1 -- start start 2023-03-03T11:21:10.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.181+0000 7fa4de3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa4d8153290 0x7fa4d8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:10.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.181+0000 7fa4de3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa4d81593c0 0x7fa4d80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:10.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4de3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d8153660 con 0x7fa4d80cd8e0 2023-03-03T11:21:10.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4de3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d806c360 con 0x7fa4d8153290 2023-03-03T11:21:10.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4de3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d815cea0 con 0x7fa4d81593c0 2023-03-03T11:21:10.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4d7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa4d8153290 0x7fa4d8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:10.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4d77fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa4d81593c0 0x7fa4d80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:10.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4dc99c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa4d80cd8e0 0x7fa4d80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50408/0 (socket says 172.21.15.102:50408) 2023-03-03T11:21:10.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4dc99c700 1 -- 172.21.15.102:0/3446598709 learned_addr learned my addr 172.21.15.102:0/3446598709 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:21:10.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4d77fe700 1 -- 172.21.15.102:0/3446598709 >> v2:172.21.15.102:3300/0 conn(0x7fa4d8153290 msgr2=0x7fa4d8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:21:10.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4d77fe700 1 --2- 172.21.15.102:0/3446598709 >> v2:172.21.15.102:3300/0 conn(0x7fa4d8153290 0x7fa4d8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4d77fe700 1 -- 172.21.15.102:0/3446598709 >> v1:172.21.15.172:6789/0 conn(0x7fa4d80cd8e0 legacy=0x7fa4d80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:10.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.182+0000 7fa4d77fe700 1 -- 172.21.15.102:0/3446598709 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d806c7f0 con 0x7fa4d81593c0 2023-03-03T11:21:10.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.183+0000 7fa4d77fe700 1 --2- 172.21.15.102:0/3446598709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 0x7fa4d80cde20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8159fb0 tx=0x7fa4cc009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd0b32e06048657f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:10.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.183+0000 7fa4d6ffd700 1 -- 172.21.15.102:0/3446598709 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc011830 con 0x7fa4d81593c0 2023-03-03T11:21:10.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.183+0000 7fa4d6ffd700 1 -- 172.21.15.102:0/3446598709 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc011990 con 0x7fa4d81593c0 2023-03-03T11:21:10.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.183+0000 7fa4d6ffd700 1 -- 172.21.15.102:0/3446598709 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc011d20 con 0x7fa4d81593c0 2023-03-03T11:21:10.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.183+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3446598709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 msgr2=0x7fa4d80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:10.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.183+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3446598709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 0x7fa4d80cde20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8159fb0 tx=0x7fa4cc009b20 comp rx=0 tx=0).stop 2023-03-03T11:21:10.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.184+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3446598709 shutdown_connections 2023-03-03T11:21:10.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.184+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3446598709 >> v2:172.21.15.102:3300/0 conn(0x7fa4d8153290 0x7fa4d8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.184+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3446598709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 0x7fa4d80cde20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.184+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3446598709 >> 172.21.15.102:0/3446598709 conn(0x7fa4d8065fb0 msgr2=0x7fa4d80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:10.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.184+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3446598709 shutdown_connections 2023-03-03T11:21:10.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.184+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3446598709 wait complete. 2023-03-03T11:21:10.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.186+0000 7fa4de3ff700 1 Processor -- start 2023-03-03T11:21:10.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.186+0000 7fa4de3ff700 1 -- start start 2023-03-03T11:21:10.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4de3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4d80cd8e0 0x7fa4d8152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:10.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4de3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d8153290 0x7fa4d80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4de3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 0x7fa4d80b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4de3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8156620 con 0x7fa4d8153290 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4de3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8154e70 con 0x7fa4d81593c0 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4de3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8155a10 con 0x7fa4d80cd8e0 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d8153290 0x7fa4d80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d8153290 0x7fa4d80b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36406/0 (socket says 172.21.15.102:36406) 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4d77fe700 1 -- 172.21.15.102:0/3558059864 learned_addr learned my addr 172.21.15.102:0/3558059864 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:21:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4dc99c700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 0x7fa4d80b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:10.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.187+0000 7fa4d7fff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4d80cd8e0 0x7fa4d8152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:10.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.188+0000 7fa4d77fe700 1 -- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4d80cd8e0 msgr2=0x7fa4d8152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:10.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.188+0000 7fa4d77fe700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4d80cd8e0 0x7fa4d8152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.188+0000 7fa4d77fe700 1 -- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 msgr2=0x7fa4d80b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:10.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.188+0000 7fa4d77fe700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 0x7fa4d80b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.188+0000 7fa4d77fe700 1 -- 172.21.15.102:0/3558059864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc0094a0 con 0x7fa4d8153290 2023-03-03T11:21:10.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.188+0000 7fa4d77fe700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d8153290 0x7fa4d80b2cc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa4d808cd80 tx=0x7fa4cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:10.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.188+0000 7fa4d4ff9700 1 -- 172.21.15.102:0/3558059864 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc0036a0 con 0x7fa4d8153290 2023-03-03T11:21:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.189+0000 7fa4d4ff9700 1 -- 172.21.15.102:0/3558059864 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc01f070 con 0x7fa4d8153290 2023-03-03T11:21:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.189+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d80ccde0 con 0x7fa4d8153290 2023-03-03T11:21:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.189+0000 7fa4d4ff9700 1 -- 172.21.15.102:0/3558059864 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc003a60 con 0x7fa4d8153290 2023-03-03T11:21:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.189+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d815d160 con 0x7fa4d8153290 2023-03-03T11:21:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.190+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d80cd0c0 con 0x7fa4d8153290 2023-03-03T11:21:10.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.191+0000 7fa4d4ff9700 1 -- 172.21.15.102:0/3558059864 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa4cc00f050 con 0x7fa4d8153290 2023-03-03T11:21:10.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.193+0000 7fa4d4ff9700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4c40682f0 0x7fa4c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:10.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.193+0000 7fa4d4ff9700 1 -- 172.21.15.102:0/3558059864 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa4cc021070 con 0x7fa4d8153290 2023-03-03T11:21:10.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.193+0000 7fa4d7fff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4c40682f0 0x7fa4c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:10.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.194+0000 7fa4d7fff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4c40682f0 0x7fa4c406a7a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009be0 tx=0x7fa4c8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:10.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.197+0000 7fa4d4ff9700 1 -- 172.21.15.102:0/3558059864 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa4cc053c40 con 0x7fa4d8153290 2023-03-03T11:21:10.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.432+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4d804e6e0 con 0x7fa4d8153290 2023-03-03T11:21:10.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.434+0000 7fa4d4ff9700 1 -- 172.21.15.102:0/3558059864 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa4cc015090 con 0x7fa4d8153290 2023-03-03T11:21:10.437 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:21:10.437 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:21:10.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4c40682f0 msgr2=0x7fa4c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:10.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4c40682f0 0x7fa4c406a7a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009be0 tx=0x7fa4c8009210 comp rx=0 tx=0).stop 2023-03-03T11:21:10.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d8153290 msgr2=0x7fa4d80b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:10.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d8153290 0x7fa4d80b2cc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa4d808cd80 tx=0x7fa4cc004750 comp rx=0 tx=0).stop 2023-03-03T11:21:10.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 shutdown_connections 2023-03-03T11:21:10.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4c40682f0 0x7fa4c406a7a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa4d80cd8e0 0x7fa4d8152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4d81593c0 0x7fa4d80b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 --2- 172.21.15.102:0/3558059864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4d8153290 0x7fa4d80b2cc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:10.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.436+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 >> 172.21.15.102:0/3558059864 conn(0x7fa4d8065fb0 msgr2=0x7fa4d80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:10.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.437+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 shutdown_connections 2023-03-03T11:21:10.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:10.437+0000 7fa4de3ff700 1 -- 172.21.15.102:0/3558059864 wait complete. 2023-03-03T11:21:10.441 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:21:10.467 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2023-03-03T11:21:33.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:33.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:41.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:21:41.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.732+0000 7fb7423ff700 1 Processor -- start 2023-03-03T11:21:41.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.732+0000 7fb7423ff700 1 -- start start 2023-03-03T11:21:41.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb7423ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c153290 0x7fb73c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:41.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb7423ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb73c1593c0 0x7fb73c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:41.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb7423ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c153660 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb7423ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73c06c360 con 0x7fb73c1593c0 2023-03-03T11:21:41.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb7423ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c15cea0 con 0x7fb73c153290 2023-03-03T11:21:41.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb73bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c153290 0x7fb73c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:41.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb733fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb73c1593c0 0x7fb73c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:41.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb74099c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb73c0cd8e0 0x7fb73c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38832/0 (socket says 172.21.15.102:38832) 2023-03-03T11:21:41.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.734+0000 7fb74099c700 1 -- 172.21.15.102:0/1206145734 learned_addr learned my addr 172.21.15.102:0/1206145734 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:21:41.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb733fff700 1 -- 172.21.15.102:0/1206145734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c153290 msgr2=0x7fb73c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:21:41.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb733fff700 1 --2- 172.21.15.102:0/1206145734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c153290 0x7fb73c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb733fff700 1 -- 172.21.15.102:0/1206145734 >> v1:172.21.15.172:6789/0 conn(0x7fb73c0cd8e0 legacy=0x7fb73c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:41.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb733fff700 1 -- 172.21.15.102:0/1206145734 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c06c7f0 con 0x7fb73c1593c0 2023-03-03T11:21:41.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb73b7fe700 1 -- 172.21.15.102:0/1206145734 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939056139 0 0) 0x7fb73c153660 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb733fff700 1 --2- 172.21.15.102:0/1206145734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c1593c0 0x7fb73c0cde20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb724009840 tx=0x7fb724009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee4e163357f28bf7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:41.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb73b7fe700 1 -- 172.21.15.102:0/1206145734 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011830 con 0x7fb73c1593c0 2023-03-03T11:21:41.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb73b7fe700 1 -- 172.21.15.102:0/1206145734 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724011990 con 0x7fb73c1593c0 2023-03-03T11:21:41.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.735+0000 7fb73b7fe700 1 -- 172.21.15.102:0/1206145734 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011d20 con 0x7fb73c1593c0 2023-03-03T11:21:41.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 -- 172.21.15.102:0/1206145734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c1593c0 msgr2=0x7fb73c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:41.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 --2- 172.21.15.102:0/1206145734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c1593c0 0x7fb73c0cde20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb724009840 tx=0x7fb724009b50 comp rx=0 tx=0).stop 2023-03-03T11:21:41.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 -- 172.21.15.102:0/1206145734 shutdown_connections 2023-03-03T11:21:41.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 --2- 172.21.15.102:0/1206145734 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c153290 0x7fb73c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 --2- 172.21.15.102:0/1206145734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c1593c0 0x7fb73c0cde20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 -- 172.21.15.102:0/1206145734 >> 172.21.15.102:0/1206145734 conn(0x7fb73c065fb0 msgr2=0x7fb73c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:41.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 -- 172.21.15.102:0/1206145734 shutdown_connections 2023-03-03T11:21:41.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.736+0000 7fb7423ff700 1 -- 172.21.15.102:0/1206145734 wait complete. 2023-03-03T11:21:41.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.738+0000 7fb7423ff700 1 Processor -- start 2023-03-03T11:21:41.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.739+0000 7fb7423ff700 1 -- start start 2023-03-03T11:21:41.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb7423ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c0cd8e0 0x7fb73c0cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:41.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb7423ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb73c153290 0x7fb73c154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:41.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb7423ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c1593c0 0x7fb73c14fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:41.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb7423ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c153f50 con 0x7fb73c153290 2023-03-03T11:21:41.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb7423ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c1501a0 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb7423ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0b2740 con 0x7fb73c1593c0 2023-03-03T11:21:41.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb73bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c0cd8e0 0x7fb73c0cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:41.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb73bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c0cd8e0 0x7fb73c0cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50290/0 (socket says 172.21.15.102:50290) 2023-03-03T11:21:41.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb73bfff700 1 -- 172.21.15.102:0/2475591112 learned_addr learned my addr 172.21.15.102:0/2475591112 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:21:41.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb733fff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb73c153290 0x7fb73c154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:41.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb74099c700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c1593c0 0x7fb73c14fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:41.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb73bfff700 1 -- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c1593c0 msgr2=0x7fb73c14fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:41.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.740+0000 7fb73bfff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c1593c0 0x7fb73c14fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb73bfff700 1 -- 172.21.15.102:0/2475591112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb73c153290 msgr2=0x7fb73c154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:41.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb73bfff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb73c153290 0x7fb73c154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb73bfff700 1 -- 172.21.15.102:0/2475591112 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7240094a0 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb73bfff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c0cd8e0 0x7fb73c0cad00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00e9b0 tx=0x7fb72c00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:41.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb7397fa700 1 -- 172.21.15.102:0/2475591112 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c009710 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb7397fa700 1 -- 172.21.15.102:0/2475591112 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c00ee50 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb7397fa700 1 -- 172.21.15.102:0/2475591112 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c00ca50 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c0ccde0 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.741+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c15d160 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.742+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73c0cd0c0 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.744+0000 7fb7397fa700 1 -- 172.21.15.102:0/2475591112 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb72c019030 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.745+0000 7fb7397fa700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb71c0682f0 0x7fb71c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:41.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.745+0000 7fb7397fa700 1 -- 172.21.15.102:0/2475591112 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb72c085c60 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.745+0000 7fb733fff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb71c0682f0 0x7fb71c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:41.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.746+0000 7fb733fff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb71c0682f0 0x7fb71c06a7a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb7240090b0 tx=0x7fb724008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:41.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.749+0000 7fb7397fa700 1 -- 172.21.15.102:0/2475591112 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb72c052370 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.985+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb73c04e6e0 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.987+0000 7fb7397fa700 1 -- 172.21.15.102:0/2475591112 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb72c055990 con 0x7fb73c0cd8e0 2023-03-03T11:21:41.988 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:21:41.989 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:21:41.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb71c0682f0 msgr2=0x7fb71c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:41.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb71c0682f0 0x7fb71c06a7a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb7240090b0 tx=0x7fb724008030 comp rx=0 tx=0).stop 2023-03-03T11:21:41.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c0cd8e0 msgr2=0x7fb73c0cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:41.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c0cd8e0 0x7fb73c0cad00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00e9b0 tx=0x7fb72c00ecc0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 shutdown_connections 2023-03-03T11:21:41.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb71c0682f0 0x7fb71c06a7a0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb73c1593c0 0x7fb73c14fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb73c0cd8e0 0x7fb73c0cad00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 --2- 172.21.15.102:0/2475591112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb73c153290 0x7fb73c154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:41.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.989+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 >> 172.21.15.102:0/2475591112 conn(0x7fb73c065fb0 msgr2=0x7fb73c1e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:41.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.990+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 shutdown_connections 2023-03-03T11:21:41.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:41.990+0000 7fb7423ff700 1 -- 172.21.15.102:0/2475591112 wait complete. 2023-03-03T11:21:41.993 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:21:42.018 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2023-03-03T11:21:44.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:21:44.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.269+0000 7f31437ff700 1 Processor -- start 2023-03-03T11:21:44.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.270+0000 7f31437ff700 1 -- start start 2023-03-03T11:21:44.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f31437ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f313c153290 0x7f313c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f31437ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c1593c0 0x7f313c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f31437ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313c153660 con 0x7f313c0cd8e0 2023-03-03T11:21:44.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f31437ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313c06c360 con 0x7f313c153290 2023-03-03T11:21:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f31437ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c15cea0 con 0x7f313c1593c0 2023-03-03T11:21:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f314159b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f313c153290 0x7f313c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f3141d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f313c0cd8e0 0x7f313c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58000/0 (socket says 172.21.15.102:58000) 2023-03-03T11:21:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f3141d9c700 1 -- 172.21.15.102:0/3334167586 learned_addr learned my addr 172.21.15.102:0/3334167586 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:21:44.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.271+0000 7f3140d9a700 1 --2- 172.21.15.102:0/3334167586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c1593c0 0x7f313c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:44.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f3133fff700 1 -- 172.21.15.102:0/3334167586 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747770085 0 0) 0x7f313c153660 con 0x7f313c0cd8e0 2023-03-03T11:21:44.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f3133fff700 1 -- 172.21.15.102:0/3334167586 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31280035b0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f3140d9a700 1 -- 172.21.15.102:0/3334167586 >> v2:172.21.15.102:3300/0 conn(0x7f313c153290 msgr2=0x7f313c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:21:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f3140d9a700 1 --2- 172.21.15.102:0/3334167586 >> v2:172.21.15.102:3300/0 conn(0x7f313c153290 0x7f313c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f3140d9a700 1 -- 172.21.15.102:0/3334167586 >> v1:172.21.15.102:6789/0 conn(0x7f313c0cd8e0 legacy=0x7f313c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f3140d9a700 1 -- 172.21.15.102:0/3334167586 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c06c7f0 con 0x7f313c1593c0 2023-03-03T11:21:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f314159b700 1 --2- 172.21.15.102:0/3334167586 >> v2:172.21.15.102:3300/0 conn(0x7f313c153290 0x7f313c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:21:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.272+0000 7f3140d9a700 1 --2- 172.21.15.102:0/3334167586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c1593c0 0x7f313c0cde20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f312c009840 tx=0x7f312c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a059341a02be2f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f3133fff700 1 -- 172.21.15.102:0/3334167586 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c011830 con 0x7f313c1593c0 2023-03-03T11:21:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f3133fff700 1 -- 172.21.15.102:0/3334167586 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c011990 con 0x7f313c1593c0 2023-03-03T11:21:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f3133fff700 1 -- 172.21.15.102:0/3334167586 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c011d20 con 0x7f313c1593c0 2023-03-03T11:21:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 -- 172.21.15.102:0/3334167586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c1593c0 msgr2=0x7f313c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 --2- 172.21.15.102:0/3334167586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c1593c0 0x7f313c0cde20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f312c009840 tx=0x7f312c009b50 comp rx=0 tx=0).stop 2023-03-03T11:21:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 -- 172.21.15.102:0/3334167586 shutdown_connections 2023-03-03T11:21:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 --2- 172.21.15.102:0/3334167586 >> v2:172.21.15.102:3300/0 conn(0x7f313c153290 0x7f313c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 --2- 172.21.15.102:0/3334167586 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c1593c0 0x7f313c0cde20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 -- 172.21.15.102:0/3334167586 >> 172.21.15.102:0/3334167586 conn(0x7f313c065fb0 msgr2=0x7f313c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:44.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 -- 172.21.15.102:0/3334167586 shutdown_connections 2023-03-03T11:21:44.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.273+0000 7f31437ff700 1 -- 172.21.15.102:0/3334167586 wait complete. 2023-03-03T11:21:44.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.275+0000 7f31437ff700 1 Processor -- start 2023-03-03T11:21:44.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.276+0000 7f31437ff700 1 -- start start 2023-03-03T11:21:44.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.276+0000 7f31437ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c0cd8e0 0x7f313c154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:44.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.276+0000 7f31437ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f313c153290 0x7f313c1e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:44.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.276+0000 7f31437ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f313c1593c0 0x7f313c1f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.276+0000 7f31437ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c155b70 con 0x7f313c1593c0 2023-03-03T11:21:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.276+0000 7f31437ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c150150 con 0x7f313c153290 2023-03-03T11:21:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.276+0000 7f31437ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c156630 con 0x7f313c0cd8e0 2023-03-03T11:21:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f3141d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f313c1593c0 0x7f313c1f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c0cd8e0 0x7f313c154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f3140d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f313c153290 0x7f313c1e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c0cd8e0 0x7f313c154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46892/0 (socket says 172.21.15.102:46892) 2023-03-03T11:21:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f3140d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f313c153290 0x7f313c1e5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50302/0 (socket says 172.21.15.102:50302) 2023-03-03T11:21:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 -- 172.21.15.102:0/4256486553 learned_addr learned my addr 172.21.15.102:0/4256486553 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:21:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 -- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f313c153290 msgr2=0x7f313c1e5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f313c153290 0x7f313c1e5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 -- 172.21.15.102:0/4256486553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f313c1593c0 msgr2=0x7f313c1f1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f313c1593c0 0x7f313c1f1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.277+0000 7f314159b700 1 -- 172.21.15.102:0/4256486553 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31380094a0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.278+0000 7f314159b700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c0cd8e0 0x7f313c154980 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f313800ebe0 tx=0x7f313800eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:44.283 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.278+0000 7f3131ffb700 1 -- 172.21.15.102:0/4256486553 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313800c850 con 0x7f313c0cd8e0 2023-03-03T11:21:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.278+0000 7f3131ffb700 1 -- 172.21.15.102:0/4256486553 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313800c9b0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.278+0000 7f3131ffb700 1 -- 172.21.15.102:0/4256486553 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313800cd40 con 0x7f313c0cd8e0 2023-03-03T11:21:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.278+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c0094a0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.278+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c0ccde0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.279+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c15d160 con 0x7f313c0cd8e0 2023-03-03T11:21:44.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.281+0000 7f3131ffb700 1 -- 172.21.15.102:0/4256486553 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3138019030 con 0x7f313c0cd8e0 2023-03-03T11:21:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.282+0000 7f3131ffb700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31200682f0 0x7f312006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.282+0000 7f3131ffb700 1 -- 172.21.15.102:0/4256486553 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3138085d90 con 0x7f313c0cd8e0 2023-03-03T11:21:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.282+0000 7f3140d9a700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31200682f0 0x7f312006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:44.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.283+0000 7f3140d9a700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31200682f0 0x7f312006a7a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f312c000c00 tx=0x7f312c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:44.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.286+0000 7f3131ffb700 1 -- 172.21.15.102:0/4256486553 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31380524a0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.523+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f313c0cd0c0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.525+0000 7f3131ffb700 1 -- 172.21.15.102:0/4256486553 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f313800cea0 con 0x7f313c0cd8e0 2023-03-03T11:21:44.527 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:21:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:21:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.526+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31200682f0 msgr2=0x7f312006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31200682f0 0x7f312006a7a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f312c000c00 tx=0x7f312c008030 comp rx=0 tx=0).stop 2023-03-03T11:21:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c0cd8e0 msgr2=0x7f313c154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c0cd8e0 0x7f313c154980 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f313800ebe0 tx=0x7f313800eef0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 shutdown_connections 2023-03-03T11:21:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f31200682f0 0x7f312006a7a0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f313c0cd8e0 0x7f313c154980 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f313c1593c0 0x7f313c1f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 --2- 172.21.15.102:0/4256486553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f313c153290 0x7f313c1e5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 >> 172.21.15.102:0/4256486553 conn(0x7f313c065fb0 msgr2=0x7f313c15aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 shutdown_connections 2023-03-03T11:21:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:21:44.527+0000 7f31437ff700 1 -- 172.21.15.102:0/4256486553 wait complete. 2023-03-03T11:21:44.531 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:21:44.556 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T11:22:03.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:03.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:22.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:22:22.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.807+0000 7f777ffff700 1 Processor -- start 2023-03-03T11:22:22.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.807+0000 7f777ffff700 1 -- start start 2023-03-03T11:22:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.808+0000 7f777ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f77780cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.808+0000 7f777ffff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7778153660 con 0x7f77780cd390 2023-03-03T11:22:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.808+0000 7f777ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777806c360 con 0x7f7778153290 2023-03-03T11:22:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.808+0000 7f777ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777815cea0 con 0x7f77780cd8e0 2023-03-03T11:22:22.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777d59a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f77780cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777dd9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7778153290 0x7f7778158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:43462/0 (socket says 172.21.15.102:43462) 2023-03-03T11:22:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777dd9b700 1 -- 172.21.15.102:0/305876919 learned_addr learned my addr 172.21.15.102:0/305876919 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:22:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777e59c700 1 --1- 172.21.15.102:0/305876919 >> v1:172.21.15.102:6789/0 conn(0x7f77780cd390 0x7f77780ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53976/0 (socket says 172.21.15.102:53976) 2023-03-03T11:22:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777cd99700 1 -- 172.21.15.102:0/305876919 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620865323 0 0) 0x7f777806c360 con 0x7f7778153290 2023-03-03T11:22:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777d59a700 1 -- 172.21.15.102:0/305876919 >> v1:172.21.15.172:6789/0 conn(0x7f7778153290 legacy=0x7f7778158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:22.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777d59a700 1 -- 172.21.15.102:0/305876919 >> v1:172.21.15.102:6789/0 conn(0x7f77780cd390 legacy=0x7f77780ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.809+0000 7f777d59a700 1 -- 172.21.15.102:0/305876919 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777806c7f0 con 0x7f77780cd8e0 2023-03-03T11:22:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777d59a700 1 --2- 172.21.15.102:0/305876919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f77780cde20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7768009870 tx=0x7f7768009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef9a4ee6b8fe8a09 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777cd99700 1 -- 172.21.15.102:0/305876919 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768004230 con 0x7f77780cd8e0 2023-03-03T11:22:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777cd99700 1 -- 172.21.15.102:0/305876919 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7768004390 con 0x7f77780cd8e0 2023-03-03T11:22:22.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777cd99700 1 -- 172.21.15.102:0/305876919 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77680036a0 con 0x7f77780cd8e0 2023-03-03T11:22:22.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777ffff700 1 -- 172.21.15.102:0/305876919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 msgr2=0x7f77780cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:22.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777ffff700 1 --2- 172.21.15.102:0/305876919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f77780cde20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7768009870 tx=0x7f7768009b80 comp rx=0 tx=0).stop 2023-03-03T11:22:22.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777ffff700 1 -- 172.21.15.102:0/305876919 shutdown_connections 2023-03-03T11:22:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777ffff700 1 --2- 172.21.15.102:0/305876919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f77780cde20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.810+0000 7f777ffff700 1 -- 172.21.15.102:0/305876919 >> 172.21.15.102:0/305876919 conn(0x7f7778065fb0 msgr2=0x7f77780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:22:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.811+0000 7f777ffff700 1 -- 172.21.15.102:0/305876919 shutdown_connections 2023-03-03T11:22:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.811+0000 7f777ffff700 1 -- 172.21.15.102:0/305876919 wait complete. 2023-03-03T11:22:22.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.812+0000 7f777ffff700 1 Processor -- start 2023-03-03T11:22:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.813+0000 7f777ffff700 1 -- start start 2023-03-03T11:22:22.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77780cd390 0x7f7778158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:22.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f777814d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:22.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7778153290 0x7f777814dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:22.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7778159230 con 0x7f7778153290 2023-03-03T11:22:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b4bf0 con 0x7f77780cd390 2023-03-03T11:22:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77781f25b0 con 0x7f77780cd8e0 2023-03-03T11:22:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7778153290 0x7f777814dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777d59a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f777814d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7778153290 0x7f777814dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56538/0 (socket says 172.21.15.102:56538) 2023-03-03T11:22:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777e59c700 1 -- 172.21.15.102:0/3275930786 learned_addr learned my addr 172.21.15.102:0/3275930786 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:22:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777dd9b700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77780cd390 0x7f7778158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:22.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777d59a700 1 -- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77780cd390 msgr2=0x7f7778158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:22.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777d59a700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77780cd390 0x7f7778158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:22.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777d59a700 1 -- 172.21.15.102:0/3275930786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7778153290 msgr2=0x7f777814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:22.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777d59a700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7778153290 0x7f777814dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:22.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777d59a700 1 -- 172.21.15.102:0/3275930786 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77680094a0 con 0x7f77780cd8e0 2023-03-03T11:22:22.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.814+0000 7f777e59c700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7778153290 0x7f777814dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:22:22.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.815+0000 7f777d59a700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f777814d680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7768015000 tx=0x7f7768004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:22.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.815+0000 7f776e7fc700 1 -- 172.21.15.102:0/3275930786 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768011630 con 0x7f77780cd8e0 2023-03-03T11:22:22.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.815+0000 7f776e7fc700 1 -- 172.21.15.102:0/3275930786 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776801f070 con 0x7f77780cd8e0 2023-03-03T11:22:22.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.815+0000 7f776e7fc700 1 -- 172.21.15.102:0/3275930786 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77680119f0 con 0x7f77780cd8e0 2023-03-03T11:22:22.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.815+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77780ccde0 con 0x7f77780cd8e0 2023-03-03T11:22:22.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.815+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777815d160 con 0x7f77780cd8e0 2023-03-03T11:22:22.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.817+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77780cd0c0 con 0x7f77780cd8e0 2023-03-03T11:22:22.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.818+0000 7f776e7fc700 1 -- 172.21.15.102:0/3275930786 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7768017020 con 0x7f77780cd8e0 2023-03-03T11:22:22.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.819+0000 7f776e7fc700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f775c0682f0 0x7f775c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:22.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.819+0000 7f776e7fc700 1 -- 172.21.15.102:0/3275930786 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7768021070 con 0x7f77780cd8e0 2023-03-03T11:22:22.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.819+0000 7f777dd9b700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f775c0682f0 0x7f775c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:22.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.820+0000 7f777dd9b700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f775c0682f0 0x7f775c06a7a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7774000f60 tx=0x7f7774004a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:22.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:22.823+0000 7f776e7fc700 1 -- 172.21.15.102:0/3275930786 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7768053ec0 con 0x7f77780cd8e0 2023-03-03T11:22:23.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.058+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f777804e6e0 con 0x7f77780cd8e0 2023-03-03T11:22:23.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.058+0000 7f776e7fc700 1 -- 172.21.15.102:0/3275930786 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f77680574e0 con 0x7f77780cd8e0 2023-03-03T11:22:23.060 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:22:23.060 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:22:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f775c0682f0 msgr2=0x7f775c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f775c0682f0 0x7f775c06a7a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7774000f60 tx=0x7f7774004a50 comp rx=0 tx=0).stop 2023-03-03T11:22:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 msgr2=0x7f777814d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f777814d680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7768015000 tx=0x7f7768004750 comp rx=0 tx=0).stop 2023-03-03T11:22:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 shutdown_connections 2023-03-03T11:22:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7778153290 0x7f777814dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77780cd390 0x7f7778158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f775c0682f0 0x7f775c06a7a0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 --2- 172.21.15.102:0/3275930786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77780cd8e0 0x7f777814d680 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 >> 172.21.15.102:0/3275930786 conn(0x7f7778065fb0 msgr2=0x7f77781e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:22:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 shutdown_connections 2023-03-03T11:22:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:22:23.061+0000 7f777ffff700 1 -- 172.21.15.102:0/3275930786 wait complete. 2023-03-03T11:22:23.065 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:22:23.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T11:22:33.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:33.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:04.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:04.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:34.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:34.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:55.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:23:55.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.355+0000 7f1d1a79d700 1 Processor -- start 2023-03-03T11:23:55.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.355+0000 7f1d1a79d700 1 -- start start 2023-03-03T11:23:55.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1a79d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d140ced00 0x7f1d140c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1a79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1a79d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1d140cf0d0 0x7f1d14158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1a79d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d140c4930 con 0x7f1d140ced00 2023-03-03T11:23:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1a79d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d140c3d20 con 0x7f1d140cf0d0 2023-03-03T11:23:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1a79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1415cba0 con 0x7f1d140b3970 2023-03-03T11:23:55.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1979b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d140ced00 0x7f1d140c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:55.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1979b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d140ced00 0x7f1d140c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42654/0 (socket says 172.21.15.102:42654) 2023-03-03T11:23:55.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d1979b700 1 -- 172.21.15.102:0/3790000873 learned_addr learned my addr 172.21.15.102:0/3790000873 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:23:55.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d19f9c700 1 --2- 172.21.15.102:0/3790000873 >> v2:172.21.15.172:3300/0 conn(0x7f1d140cf0d0 0x7f1d14158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:55.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.357+0000 7f1d18f9a700 1 --2- 172.21.15.102:0/3790000873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:55.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.358+0000 7f1d1979b700 1 -- 172.21.15.102:0/3790000873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 msgr2=0x7f1d140c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:23:55.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.358+0000 7f1d1979b700 1 --2- 172.21.15.102:0/3790000873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.358+0000 7f1d1979b700 1 -- 172.21.15.102:0/3790000873 >> v2:172.21.15.172:3300/0 conn(0x7f1d140cf0d0 msgr2=0x7f1d14158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.358+0000 7f1d1979b700 1 --2- 172.21.15.102:0/3790000873 >> v2:172.21.15.172:3300/0 conn(0x7f1d140cf0d0 0x7f1d14158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.358+0000 7f1d1979b700 1 -- 172.21.15.102:0/3790000873 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1415ce80 con 0x7f1d140ced00 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.358+0000 7f1d1979b700 1 --2- 172.21.15.102:0/3790000873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140ced00 0x7f1d140c5a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1d140b3920 tx=0x7f1d10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f02ac7e95588d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d0bfff700 1 -- 172.21.15.102:0/3790000873 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10011830 con 0x7f1d140ced00 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d0bfff700 1 -- 172.21.15.102:0/3790000873 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d10011990 con 0x7f1d140ced00 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d0bfff700 1 -- 172.21.15.102:0/3790000873 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10011d20 con 0x7f1d140ced00 2023-03-03T11:23:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 -- 172.21.15.102:0/3790000873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140ced00 msgr2=0x7f1d140c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/3790000873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140ced00 0x7f1d140c5a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1d140b3920 tx=0x7f1d10009bd0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 -- 172.21.15.102:0/3790000873 shutdown_connections 2023-03-03T11:23:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/3790000873 >> v2:172.21.15.172:3300/0 conn(0x7f1d140cf0d0 0x7f1d14158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/3790000873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140ced00 0x7f1d140c5a90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/3790000873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 -- 172.21.15.102:0/3790000873 >> 172.21.15.102:0/3790000873 conn(0x7f1d1408b1c0 msgr2=0x7f1d14071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:23:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.359+0000 7f1d1a79d700 1 -- 172.21.15.102:0/3790000873 shutdown_connections 2023-03-03T11:23:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.360+0000 7f1d1a79d700 1 -- 172.21.15.102:0/3790000873 wait complete. 2023-03-03T11:23:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.361+0000 7f1d1a79d700 1 Processor -- start 2023-03-03T11:23:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.362+0000 7f1d1a79d700 1 -- start start 2023-03-03T11:23:55.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.362+0000 7f1d1a79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:55.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1a79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d140ced00 0x7f1d141f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:55.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1a79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140cf0d0 0x7f1d141f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:55.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1a79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14153990 con 0x7f1d140cf0d0 2023-03-03T11:23:55.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1a79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d140c6fb0 con 0x7f1d140ced00 2023-03-03T11:23:55.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1a79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d140c6a30 con 0x7f1d140b3970 2023-03-03T11:23:55.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d19f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140cf0d0 0x7f1d141f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:55.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:55.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d18f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d140ced00 0x7f1d141f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:55.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57182/0 (socket says 172.21.15.102:57182) 2023-03-03T11:23:55.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 -- 172.21.15.102:0/1541420023 learned_addr learned my addr 172.21.15.102:0/1541420023 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:23:55.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 -- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d140ced00 msgr2=0x7f1d141f1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:55.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d140ced00 0x7f1d141f1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 -- 172.21.15.102:0/1541420023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140cf0d0 msgr2=0x7f1d141f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140cf0d0 0x7f1d141f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d1979b700 1 -- 172.21.15.102:0/1541420023 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10009550 con 0x7f1d140b3970 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.363+0000 7f1d19f9c700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140cf0d0 0x7f1d141f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.364+0000 7f1d1979b700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c8c10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1d10009c10 tx=0x7f1d10004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.364+0000 7f1d09ffb700 1 -- 172.21.15.102:0/1541420023 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10004450 con 0x7f1d140b3970 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.364+0000 7f1d09ffb700 1 -- 172.21.15.102:0/1541420023 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1001e070 con 0x7f1d140b3970 2023-03-03T11:23:55.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.364+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1406c200 con 0x7f1d140b3970 2023-03-03T11:23:55.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.364+0000 7f1d09ffb700 1 -- 172.21.15.102:0/1541420023 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d100038e0 con 0x7f1d140b3970 2023-03-03T11:23:55.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.364+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1406bf20 con 0x7f1d140b3970 2023-03-03T11:23:55.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.365+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d14159040 con 0x7f1d140b3970 2023-03-03T11:23:55.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.367+0000 7f1d09ffb700 1 -- 172.21.15.102:0/1541420023 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1d1000f050 con 0x7f1d140b3970 2023-03-03T11:23:55.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.368+0000 7f1d09ffb700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cf8068320 0x7f1cf806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:55.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.368+0000 7f1d09ffb700 1 -- 172.21.15.102:0/1541420023 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1d10086ab0 con 0x7f1d140b3970 2023-03-03T11:23:55.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.368+0000 7f1d18f9a700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cf8068320 0x7f1cf806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:55.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.369+0000 7f1d18f9a700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cf8068320 0x7f1cf806a7d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1d141e31f0 tx=0x7f1d04009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:55.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.373+0000 7f1d09ffb700 1 -- 172.21.15.102:0/1541420023 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1d100531c0 con 0x7f1d140b3970 2023-03-03T11:23:55.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.608+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d141f1970 con 0x7f1d140b3970 2023-03-03T11:23:55.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.609+0000 7f1d09ffb700 1 -- 172.21.15.102:0/1541420023 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1d10020090 con 0x7f1d140b3970 2023-03-03T11:23:55.611 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:23:55.611 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:23:55.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cf8068320 msgr2=0x7f1cf806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:55.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cf8068320 0x7f1cf806a7d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1d141e31f0 tx=0x7f1d04009210 comp rx=0 tx=0).stop 2023-03-03T11:23:55.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 msgr2=0x7f1d140c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:55.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c8c10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1d10009c10 tx=0x7f1d10004750 comp rx=0 tx=0).stop 2023-03-03T11:23:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 shutdown_connections 2023-03-03T11:23:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1cf8068320 0x7f1cf806a7d0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d140b3970 0x7f1d140c8c10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140cf0d0 0x7f1d141f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 --2- 172.21.15.102:0/1541420023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d140ced00 0x7f1d141f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 >> 172.21.15.102:0/1541420023 conn(0x7f1d1408b1c0 msgr2=0x7f1d140cd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:23:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 shutdown_connections 2023-03-03T11:23:55.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:23:55.611+0000 7f1d1a79d700 1 -- 172.21.15.102:0/1541420023 wait complete. 2023-03-03T11:23:55.615 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:23:55.640 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-03-03T11:24:04.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:04.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:34.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:34.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:41.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:24:41.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.903+0000 7f90fadff700 1 Processor -- start 2023-03-03T11:24:41.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.903+0000 7f90fadff700 1 -- start start 2023-03-03T11:24:41.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.904+0000 7f90fadff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90f4153290 0x7f90f4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:41.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.904+0000 7f90fadff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90f40cd8e0 0x7f90f40ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:41.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.904+0000 7f90fadff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f4153660 con 0x7f90f41593c0 2023-03-03T11:24:41.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.904+0000 7f90fadff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f406c360 con 0x7f90f4153290 2023-03-03T11:24:41.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.904+0000 7f90fadff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f415cea0 con 0x7f90f40cd8e0 2023-03-03T11:24:41.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f8b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90f4153290 0x7f90f4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f939c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90f40cd8e0 0x7f90f40ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f939c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90f40cd8e0 0x7f90f40ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56554/0 (socket says 172.21.15.102:56554) 2023-03-03T11:24:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f8b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90f4153290 0x7f90f4158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40384/0 (socket says 172.21.15.102:40384) 2023-03-03T11:24:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f939c700 1 -- 172.21.15.102:0/2882309156 learned_addr learned my addr 172.21.15.102:0/2882309156 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:24:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f939c700 1 -- 172.21.15.102:0/2882309156 >> v2:172.21.15.102:3300/0 conn(0x7f90f4153290 msgr2=0x7f90f4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:24:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f939c700 1 --2- 172.21.15.102:0/2882309156 >> v2:172.21.15.102:3300/0 conn(0x7f90f4153290 0x7f90f4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f939c700 1 -- 172.21.15.102:0/2882309156 >> v1:172.21.15.172:6789/0 conn(0x7f90f41593c0 legacy=0x7f90f40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f939c700 1 -- 172.21.15.102:0/2882309156 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f406c7f0 con 0x7f90f40cd8e0 2023-03-03T11:24:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f8b9b700 1 --2- 172.21.15.102:0/2882309156 >> v2:172.21.15.102:3300/0 conn(0x7f90f4153290 0x7f90f4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:24:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.905+0000 7f90f37fe700 1 -- 172.21.15.102:0/2882309156 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914702130 0 0) 0x7f90f4153660 con 0x7f90f41593c0 2023-03-03T11:24:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.906+0000 7f90f939c700 1 --2- 172.21.15.102:0/2882309156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f40cd8e0 0x7f90f40ce530 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f90f40c9390 tx=0x7f90ec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ce90025568ab5c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.906+0000 7f90f37fe700 1 -- 172.21.15.102:0/2882309156 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec00e070 con 0x7f90f40cd8e0 2023-03-03T11:24:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.906+0000 7f90f37fe700 1 -- 172.21.15.102:0/2882309156 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec00b910 con 0x7f90f40cd8e0 2023-03-03T11:24:41.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.906+0000 7f90f37fe700 1 -- 172.21.15.102:0/2882309156 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec00bcc0 con 0x7f90f40cd8e0 2023-03-03T11:24:41.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.906+0000 7f90fadff700 1 -- 172.21.15.102:0/2882309156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f40cd8e0 msgr2=0x7f90f40ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.906+0000 7f90fadff700 1 --2- 172.21.15.102:0/2882309156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f40cd8e0 0x7f90f40ce530 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f90f40c9390 tx=0x7f90ec00b330 comp rx=0 tx=0).stop 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.907+0000 7f90fadff700 1 -- 172.21.15.102:0/2882309156 shutdown_connections 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.907+0000 7f90fadff700 1 --2- 172.21.15.102:0/2882309156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f40cd8e0 0x7f90f40ce530 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.907+0000 7f90fadff700 1 --2- 172.21.15.102:0/2882309156 >> v2:172.21.15.102:3300/0 conn(0x7f90f4153290 0x7f90f4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.907+0000 7f90fadff700 1 -- 172.21.15.102:0/2882309156 >> 172.21.15.102:0/2882309156 conn(0x7f90f4065fb0 msgr2=0x7f90f40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.907+0000 7f90fadff700 1 -- 172.21.15.102:0/2882309156 shutdown_connections 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.907+0000 7f90fadff700 1 -- 172.21.15.102:0/2882309156 wait complete. 2023-03-03T11:24:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.909+0000 7f90fadff700 1 Processor -- start 2023-03-03T11:24:41.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.909+0000 7f90fadff700 1 -- start start 2023-03-03T11:24:41.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90fadff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f40cd8e0 0x7f90f40c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:41.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90fadff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f4153290 0x7f90f4155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:41.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90fadff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f41593c0 0x7f90f4155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:41.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90fadff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40b3be0 con 0x7f90f41593c0 2023-03-03T11:24:41.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90fadff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40b3e10 con 0x7f90f4153290 2023-03-03T11:24:41.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90fadff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40cf940 con 0x7f90f40cd8e0 2023-03-03T11:24:41.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90f8b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f40cd8e0 0x7f90f40c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:41.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90f8b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f40cd8e0 0x7f90f40c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46324/0 (socket says 172.21.15.102:46324) 2023-03-03T11:24:41.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90f8b9b700 1 -- 172.21.15.102:0/2996575359 learned_addr learned my addr 172.21.15.102:0/2996575359 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:24:41.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90f3fff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f4153290 0x7f90f4155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:41.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.910+0000 7f90f939c700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f41593c0 0x7f90f4155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f8b9b700 1 -- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f4153290 msgr2=0x7f90f4155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f8b9b700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f4153290 0x7f90f4155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f8b9b700 1 -- 172.21.15.102:0/2996575359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f41593c0 msgr2=0x7f90f4155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f8b9b700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f41593c0 0x7f90f4155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f8b9b700 1 -- 172.21.15.102:0/2996575359 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e80094a0 con 0x7f90f40cd8e0 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f939c700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f41593c0 0x7f90f4155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f8b9b700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f40cd8e0 0x7f90f40c6320 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f90e800ebe0 tx=0x7f90e800eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:41.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f17fa700 1 -- 172.21.15.102:0/2996575359 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800c850 con 0x7f90f40cd8e0 2023-03-03T11:24:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f17fa700 1 -- 172.21.15.102:0/2996575359 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e800c9b0 con 0x7f90f40cd8e0 2023-03-03T11:24:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec009b30 con 0x7f90f40cd8e0 2023-03-03T11:24:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90f17fa700 1 -- 172.21.15.102:0/2996575359 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800cd40 con 0x7f90f40cd8e0 2023-03-03T11:24:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.911+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f40ccde0 con 0x7f90f40cd8e0 2023-03-03T11:24:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.912+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f415d160 con 0x7f90f40cd8e0 2023-03-03T11:24:41.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.914+0000 7f90f17fa700 1 -- 172.21.15.102:0/2996575359 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90e8019030 con 0x7f90f40cd8e0 2023-03-03T11:24:41.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.915+0000 7f90f17fa700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d8068320 0x7f90d806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:41.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.915+0000 7f90f17fa700 1 -- 172.21.15.102:0/2996575359 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f90e8085d30 con 0x7f90f40cd8e0 2023-03-03T11:24:41.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.916+0000 7f90f3fff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d8068320 0x7f90d806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:41.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.916+0000 7f90f3fff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d8068320 0x7f90d806a7d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f90e0007fd0 tx=0x7f90e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:41.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:41.918+0000 7f90f17fa700 1 -- 172.21.15.102:0/2996575359 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90e8052430 con 0x7f90f40cd8e0 2023-03-03T11:24:42.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.151+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f40cd0c0 con 0x7f90f40cd8e0 2023-03-03T11:24:42.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.152+0000 7f90f17fa700 1 -- 172.21.15.102:0/2996575359 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f90e8017090 con 0x7f90f40cd8e0 2023-03-03T11:24:42.154 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:24:42.154 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:24:42.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d8068320 msgr2=0x7f90d806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:42.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d8068320 0x7f90d806a7d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f90e0007fd0 tx=0x7f90e0009040 comp rx=0 tx=0).stop 2023-03-03T11:24:42.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f40cd8e0 msgr2=0x7f90f40c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:42.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f40cd8e0 0x7f90f40c6320 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f90e800ebe0 tx=0x7f90e800eef0 comp rx=0 tx=0).stop 2023-03-03T11:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 shutdown_connections 2023-03-03T11:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90d8068320 0x7f90d806a7d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f40cd8e0 0x7f90f40c6320 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f41593c0 0x7f90f4155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 --2- 172.21.15.102:0/2996575359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f4153290 0x7f90f4155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.154+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 >> 172.21.15.102:0/2996575359 conn(0x7f90f4065fb0 msgr2=0x7f90f415ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.155+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 shutdown_connections 2023-03-03T11:24:42.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:42.155+0000 7f90fadff700 1 -- 172.21.15.102:0/2996575359 wait complete. 2023-03-03T11:24:42.158 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:24:42.184 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-03-03T11:24:49.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:24:49.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.434+0000 7f9d1f59d700 1 Processor -- start 2023-03-03T11:24:49.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.434+0000 7f9d1f59d700 1 -- start start 2023-03-03T11:24:49.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.436+0000 7f9d1f59d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9d180c5a90 0x7f9d180c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:49.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.436+0000 7f9d1f59d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d180c3d20 con 0x7f9d180b3970 2023-03-03T11:24:49.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.436+0000 7f9d1f59d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1815cba0 con 0x7f9d180ced00 2023-03-03T11:24:49.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.436+0000 7f9d1f59d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d180c4930 con 0x7f9d180c5a90 2023-03-03T11:24:49.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.436+0000 7f9d1dd9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9d180b3970 0x7f9d18161470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50320/0 (socket says 172.21.15.102:50320) 2023-03-03T11:24:49.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.436+0000 7f9d1dd9a700 1 -- 172.21.15.102:0/3655575659 learned_addr learned my addr 172.21.15.102:0/3655575659 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:24:49.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.436+0000 7f9d1ed9c700 1 --2- 172.21.15.102:0/3655575659 >> v2:172.21.15.172:3300/0 conn(0x7f9d180c5a90 0x7f9d180c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:49.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1d599700 1 -- 172.21.15.102:0/3655575659 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734990375 0 0) 0x7f9d180c3d20 con 0x7f9d180b3970 2023-03-03T11:24:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1d599700 1 -- 172.21.15.102:0/3655575659 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d040035b0 con 0x7f9d180b3970 2023-03-03T11:24:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1d599700 1 -- 172.21.15.102:0/3655575659 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177919907 0 0) 0x7f9d1815cba0 con 0x7f9d180ced00 2023-03-03T11:24:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1ed9c700 1 -- 172.21.15.102:0/3655575659 >> v1:172.21.15.172:6789/0 conn(0x7f9d180ced00 legacy=0x7f9d18158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1ed9c700 1 -- 172.21.15.102:0/3655575659 >> v1:172.21.15.102:6789/0 conn(0x7f9d180b3970 legacy=0x7f9d18161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1ed9c700 1 -- 172.21.15.102:0/3655575659 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1815ce80 con 0x7f9d180c5a90 2023-03-03T11:24:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1ed9c700 1 --2- 172.21.15.102:0/3655575659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180c5a90 0x7f9d180c41b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d10009d90 tx=0x7f9d1000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18c832d81be39582 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1d599700 1 -- 172.21.15.102:0/3655575659 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1000e070 con 0x7f9d180c5a90 2023-03-03T11:24:49.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1d599700 1 -- 172.21.15.102:0/3655575659 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1000bb20 con 0x7f9d180c5a90 2023-03-03T11:24:49.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.437+0000 7f9d1d599700 1 -- 172.21.15.102:0/3655575659 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1000bed0 con 0x7f9d180c5a90 2023-03-03T11:24:49.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.438+0000 7f9d1f59d700 1 -- 172.21.15.102:0/3655575659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180c5a90 msgr2=0x7f9d180c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.438+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/3655575659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180c5a90 0x7f9d180c41b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d10009d90 tx=0x7f9d1000b330 comp rx=0 tx=0).stop 2023-03-03T11:24:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.438+0000 7f9d1f59d700 1 -- 172.21.15.102:0/3655575659 shutdown_connections 2023-03-03T11:24:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.438+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/3655575659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180c5a90 0x7f9d180c41b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.438+0000 7f9d1f59d700 1 -- 172.21.15.102:0/3655575659 >> 172.21.15.102:0/3655575659 conn(0x7f9d1808b1c0 msgr2=0x7f9d18071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:24:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.438+0000 7f9d1f59d700 1 -- 172.21.15.102:0/3655575659 shutdown_connections 2023-03-03T11:24:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.438+0000 7f9d1f59d700 1 -- 172.21.15.102:0/3655575659 wait complete. 2023-03-03T11:24:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.440+0000 7f9d1f59d700 1 Processor -- start 2023-03-03T11:24:49.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.440+0000 7f9d1f59d700 1 -- start start 2023-03-03T11:24:49.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.441+0000 7f9d1f59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180b3970 0x7f9d180c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:49.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.441+0000 7f9d1f59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d180c5a90 0x7f9d1815b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:49.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.441+0000 7f9d1f59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d180ced00 0x7f9d1814d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:49.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.441+0000 7f9d1f59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180ce600 con 0x7f9d180ced00 2023-03-03T11:24:49.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.441+0000 7f9d1f59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180c6070 con 0x7f9d180b3970 2023-03-03T11:24:49.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.441+0000 7f9d1f59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d18066b20 con 0x7f9d180c5a90 2023-03-03T11:24:49.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.441+0000 7f9d1e59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180b3970 0x7f9d180c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:49.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1ed9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d180ced00 0x7f9d1814d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:49.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180b3970 0x7f9d180c8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35200/0 (socket says 172.21.15.102:35200) 2023-03-03T11:24:49.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 -- 172.21.15.102:0/2776559099 learned_addr learned my addr 172.21.15.102:0/2776559099 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:24:49.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1dd9a700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d180c5a90 0x7f9d1815b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:49.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 -- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d180c5a90 msgr2=0x7f9d1815b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:49.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d180c5a90 0x7f9d1815b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:49.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 -- 172.21.15.102:0/2776559099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d180ced00 msgr2=0x7f9d1814d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:49.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d180ced00 0x7f9d1814d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:49.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 -- 172.21.15.102:0/2776559099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10009a70 con 0x7f9d180b3970 2023-03-03T11:24:49.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.442+0000 7f9d1e59b700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180b3970 0x7f9d180c8bc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400c780 tx=0x7f9d1400ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:49.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.443+0000 7f9d0effd700 1 -- 172.21.15.102:0/2776559099 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d140054e0 con 0x7f9d180b3970 2023-03-03T11:24:49.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.443+0000 7f9d0effd700 1 -- 172.21.15.102:0/2776559099 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d14003680 con 0x7f9d180b3970 2023-03-03T11:24:49.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.443+0000 7f9d0effd700 1 -- 172.21.15.102:0/2776559099 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14003a30 con 0x7f9d180b3970 2023-03-03T11:24:49.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.443+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1806c200 con 0x7f9d180b3970 2023-03-03T11:24:49.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.443+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1806bf20 con 0x7f9d180b3970 2023-03-03T11:24:49.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.444+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1815c890 con 0x7f9d180b3970 2023-03-03T11:24:49.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.445+0000 7f9d0effd700 1 -- 172.21.15.102:0/2776559099 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9d14019030 con 0x7f9d180b3970 2023-03-03T11:24:49.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.447+0000 7f9d0effd700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9cfc068320 0x7f9cfc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:49.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.447+0000 7f9d0effd700 1 -- 172.21.15.102:0/2776559099 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9d14084fe0 con 0x7f9d180b3970 2023-03-03T11:24:49.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.447+0000 7f9d1dd9a700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9cfc068320 0x7f9cfc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:49.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.447+0000 7f9d1dd9a700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9cfc068320 0x7f9cfc06a7d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9d180c9fb0 tx=0x7f9d08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:49.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.451+0000 7f9d0effd700 1 -- 172.21.15.102:0/2776559099 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9d14051750 con 0x7f9d180b3970 2023-03-03T11:24:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.684+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d180ccf80 con 0x7f9d180b3970 2023-03-03T11:24:49.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.685+0000 7f9d0effd700 1 -- 172.21.15.102:0/2776559099 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9d14017070 con 0x7f9d180b3970 2023-03-03T11:24:49.687 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:24:49.688 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:24:49.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9cfc068320 msgr2=0x7f9cfc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:49.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9cfc068320 0x7f9cfc06a7d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9d180c9fb0 tx=0x7f9d08009040 comp rx=0 tx=0).stop 2023-03-03T11:24:49.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180b3970 msgr2=0x7f9d180c8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:49.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180b3970 0x7f9d180c8bc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400c780 tx=0x7f9d1400ca90 comp rx=0 tx=0).stop 2023-03-03T11:24:49.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 shutdown_connections 2023-03-03T11:24:49.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d180ced00 0x7f9d1814d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:49.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d180b3970 0x7f9d180c8bc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:49.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9cfc068320 0x7f9cfc06a7d0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:49.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 --2- 172.21.15.102:0/2776559099 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9d180c5a90 0x7f9d1815b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:49.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.688+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 >> 172.21.15.102:0/2776559099 conn(0x7f9d1808b1c0 msgr2=0x7f9d180cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:24:49.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.689+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 shutdown_connections 2023-03-03T11:24:49.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:24:49.689+0000 7f9d1f59d700 1 -- 172.21.15.102:0/2776559099 wait complete. 2023-03-03T11:24:49.692 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:24:49.717 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T11:25:04.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:04.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:34.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:34.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:41.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:25:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.969+0000 7f7c609ff700 1 Processor -- start 2023-03-03T11:25:41.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.969+0000 7f7c609ff700 1 -- start start 2023-03-03T11:25:41.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c609ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c150c80 0x7f7c5c06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:41.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c609ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7c5c0c5f30 0x7f7c5c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:41.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c609ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c0c5390 con 0x7f7c5c0c2020 2023-03-03T11:25:41.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c609ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c157ad0 con 0x7f7c5c0c5f30 2023-03-03T11:25:41.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c609ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0c4a20 con 0x7f7c5c150c80 2023-03-03T11:25:41.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c5ad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7c5c0c5f30 0x7f7c5c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:41.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c5a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c150c80 0x7f7c5c06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:41.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c5ad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7c5c0c5f30 0x7f7c5c0c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36834/0 (socket says 172.21.15.102:36834) 2023-03-03T11:25:41.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c5a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c150c80 0x7f7c5c06d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34656/0 (socket says 172.21.15.102:34656) 2023-03-03T11:25:41.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.971+0000 7f7c5ad9d700 1 -- 172.21.15.102:0/869092728 learned_addr learned my addr 172.21.15.102:0/869092728 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:25:41.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5ad9d700 1 -- 172.21.15.102:0/869092728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c150c80 msgr2=0x7f7c5c06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:25:41.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5ad9d700 1 --2- 172.21.15.102:0/869092728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c150c80 0x7f7c5c06d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:41.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5ad9d700 1 -- 172.21.15.102:0/869092728 >> v1:172.21.15.102:6789/0 conn(0x7f7c5c0c2020 legacy=0x7f7c5c0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:41.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5ad9d700 1 -- 172.21.15.102:0/869092728 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c06f010 con 0x7f7c5c0c5f30 2023-03-03T11:25:41.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5959a700 1 -- 172.21.15.102:0/869092728 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360328942 0 0) 0x7f7c5c0c5390 con 0x7f7c5c0c2020 2023-03-03T11:25:41.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5ad9d700 1 --2- 172.21.15.102:0/869092728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c0c5f30 0x7f7c5c0c2720 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009e50 tx=0x7f7c5400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbaa698c73493c65 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:41.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5959a700 1 -- 172.21.15.102:0/869092728 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400e070 con 0x7f7c5c0c5f30 2023-03-03T11:25:41.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5959a700 1 -- 172.21.15.102:0/869092728 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5400b910 con 0x7f7c5c0c5f30 2023-03-03T11:25:41.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.972+0000 7f7c5959a700 1 -- 172.21.15.102:0/869092728 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400bcc0 con 0x7f7c5c0c5f30 2023-03-03T11:25:41.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 -- 172.21.15.102:0/869092728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c0c5f30 msgr2=0x7f7c5c0c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:41.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 --2- 172.21.15.102:0/869092728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c0c5f30 0x7f7c5c0c2720 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009e50 tx=0x7f7c5400b330 comp rx=0 tx=0).stop 2023-03-03T11:25:41.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 -- 172.21.15.102:0/869092728 shutdown_connections 2023-03-03T11:25:41.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 --2- 172.21.15.102:0/869092728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c0c5f30 0x7f7c5c0c2720 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:41.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 --2- 172.21.15.102:0/869092728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c150c80 0x7f7c5c06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:41.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 -- 172.21.15.102:0/869092728 >> 172.21.15.102:0/869092728 conn(0x7f7c5c06af90 msgr2=0x7f7c5c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:25:41.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 -- 172.21.15.102:0/869092728 shutdown_connections 2023-03-03T11:25:41.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.973+0000 7f7c609ff700 1 -- 172.21.15.102:0/869092728 wait complete. 2023-03-03T11:25:41.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.975+0000 7f7c609ff700 1 Processor -- start 2023-03-03T11:25:41.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.975+0000 7f7c609ff700 1 -- start start 2023-03-03T11:25:41.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.976+0000 7f7c609ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c5c0c2020 0x7f7c5c1f0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:41.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.976+0000 7f7c609ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c0c5f30 0x7f7c5c1ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:41.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.976+0000 7f7c609ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c150c80 0x7f7c5c14cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:41.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.976+0000 7f7c609ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c149aa0 con 0x7f7c5c0c2020 2023-03-03T11:25:41.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.976+0000 7f7c609ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c156e10 con 0x7f7c5c150c80 2023-03-03T11:25:41.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.976+0000 7f7c609ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c151520 con 0x7f7c5c0c5f30 2023-03-03T11:25:41.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c5c0c2020 0x7f7c5c1f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:41.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c59d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c0c5f30 0x7f7c5c1ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:41.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c5c0c2020 0x7f7c5c1f0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40860/0 (socket says 172.21.15.102:40860) 2023-03-03T11:25:41.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 -- 172.21.15.102:0/481398069 learned_addr learned my addr 172.21.15.102:0/481398069 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:25:41.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5ad9d700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c150c80 0x7f7c5c14cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:41.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 -- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c0c5f30 msgr2=0x7f7c5c1ed0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:41.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c0c5f30 0x7f7c5c1ed0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:41.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 -- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c150c80 msgr2=0x7f7c5c14cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:41.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c150c80 0x7f7c5c14cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:41.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 -- 172.21.15.102:0/481398069 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c54009b30 con 0x7f7c5c0c2020 2023-03-03T11:25:41.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.977+0000 7f7c5a59c700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c5c0c2020 0x7f7c5c1f0700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00e870 tx=0x7f7c4c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:41.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.978+0000 7f7c4affd700 1 -- 172.21.15.102:0/481398069 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c009590 con 0x7f7c5c0c2020 2023-03-03T11:25:41.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.978+0000 7f7c4affd700 1 -- 172.21.15.102:0/481398069 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c013680 con 0x7f7c5c0c2020 2023-03-03T11:25:41.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.978+0000 7f7c4affd700 1 -- 172.21.15.102:0/481398069 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0139f0 con 0x7f7c5c0c2020 2023-03-03T11:25:41.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.978+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c1577f0 con 0x7f7c5c0c2020 2023-03-03T11:25:41.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.978+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c157510 con 0x7f7c5c0c2020 2023-03-03T11:25:41.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.979+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0c4cc0 con 0x7f7c5c0c2020 2023-03-03T11:25:41.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.980+0000 7f7c4affd700 1 -- 172.21.15.102:0/481398069 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c4c019030 con 0x7f7c5c0c2020 2023-03-03T11:25:41.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.981+0000 7f7c4affd700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7c3c068320 0x7f7c3c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:41.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.981+0000 7f7c4affd700 1 -- 172.21.15.102:0/481398069 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7c4c085ab0 con 0x7f7c5c0c2020 2023-03-03T11:25:41.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.982+0000 7f7c59d9b700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7c3c068320 0x7f7c3c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:41.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.982+0000 7f7c59d9b700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7c3c068320 0x7f7c3c06a7d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c15b0f0 tx=0x7f7c50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:41.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:41.985+0000 7f7c4affd700 1 -- 172.21.15.102:0/481398069 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c4c052130 con 0x7f7c5c0c2020 2023-03-03T11:25:42.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.221+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c5c06f990 con 0x7f7c5c0c2020 2023-03-03T11:25:42.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.222+0000 7f7c4affd700 1 -- 172.21.15.102:0/481398069 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7c4c021090 con 0x7f7c5c0c2020 2023-03-03T11:25:42.224 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:25:42.225 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:25:42.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.224+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7c3c068320 msgr2=0x7f7c3c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:42.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.224+0000 7f7c609ff700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7c3c068320 0x7f7c3c06a7d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c15b0f0 tx=0x7f7c50009040 comp rx=0 tx=0).stop 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.224+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c5c0c2020 msgr2=0x7f7c5c1f0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.224+0000 7f7c609ff700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c5c0c2020 0x7f7c5c1f0700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00e870 tx=0x7f7c4c00eb80 comp rx=0 tx=0).stop 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 shutdown_connections 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c5c150c80 0x7f7c5c14cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c5c0c2020 0x7f7c5c1f0700 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7c3c068320 0x7f7c3c06a7d0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 --2- 172.21.15.102:0/481398069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7c5c0c5f30 0x7f7c5c1ed0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:42.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 >> 172.21.15.102:0/481398069 conn(0x7f7c5c06af90 msgr2=0x7f7c5c14e680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:25:42.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 shutdown_connections 2023-03-03T11:25:42.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:25:42.225+0000 7f7c609ff700 1 -- 172.21.15.102:0/481398069 wait complete. 2023-03-03T11:25:42.229 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:25:42.254 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2023-03-03T11:26:05.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:05.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:35.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:35.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:05.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:05.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:16.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:27:16.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.505+0000 7f95c0bff700 1 Processor -- start 2023-03-03T11:27:16.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.506+0000 7f95c0bff700 1 -- start start 2023-03-03T11:27:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95c0bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95bc153290 0x7f95bc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:16.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95c0bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f95bc1593c0 0x7f95bc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:16.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95c0bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc153660 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95c0bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95bc06c360 con 0x7f95bc153290 2023-03-03T11:27:16.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95c0bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95bc15cea0 con 0x7f95bc1593c0 2023-03-03T11:27:16.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95ba59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95bc153290 0x7f95bc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:16.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95ba59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95bc153290 0x7f95bc158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48256/0 (socket says 172.21.15.102:48256) 2023-03-03T11:27:16.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95ba59c700 1 -- 172.21.15.102:0/743019154 learned_addr learned my addr 172.21.15.102:0/743019154 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95bad9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f95bc0cd8e0 0x7f95bc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50922/0 (socket says 172.21.15.102:50922) 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.507+0000 7f95b9d9b700 1 --2- 172.21.15.102:0/743019154 >> v2:172.21.15.172:3300/0 conn(0x7f95bc1593c0 0x7f95bc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.508+0000 7f95ba59c700 1 -- 172.21.15.102:0/743019154 >> v2:172.21.15.172:3300/0 conn(0x7f95bc1593c0 msgr2=0x7f95bc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.508+0000 7f95ba59c700 1 --2- 172.21.15.102:0/743019154 >> v2:172.21.15.172:3300/0 conn(0x7f95bc1593c0 0x7f95bc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.508+0000 7f95ba59c700 1 -- 172.21.15.102:0/743019154 >> v1:172.21.15.102:6789/0 conn(0x7f95bc0cd8e0 legacy=0x7f95bc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.508+0000 7f95ba59c700 1 -- 172.21.15.102:0/743019154 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc06c7f0 con 0x7f95bc153290 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.508+0000 7f95ba59c700 1 --2- 172.21.15.102:0/743019154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 0x7f95bc158e80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f95bc159fb0 tx=0x7f95ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5506583c52fdd4cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:27:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.508+0000 7f95b959a700 1 -- 172.21.15.102:0/743019154 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac011830 con 0x7f95bc153290 2023-03-03T11:27:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95b959a700 1 -- 172.21.15.102:0/743019154 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac011990 con 0x7f95bc153290 2023-03-03T11:27:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95b959a700 1 -- 172.21.15.102:0/743019154 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac011d20 con 0x7f95bc153290 2023-03-03T11:27:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 -- 172.21.15.102:0/743019154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 msgr2=0x7f95bc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 --2- 172.21.15.102:0/743019154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 0x7f95bc158e80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f95bc159fb0 tx=0x7f95ac009bd0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 -- 172.21.15.102:0/743019154 shutdown_connections 2023-03-03T11:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 --2- 172.21.15.102:0/743019154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 0x7f95bc158e80 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 --2- 172.21.15.102:0/743019154 >> v2:172.21.15.172:3300/0 conn(0x7f95bc1593c0 0x7f95bc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 -- 172.21.15.102:0/743019154 >> 172.21.15.102:0/743019154 conn(0x7f95bc065fb0 msgr2=0x7f95bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 -- 172.21.15.102:0/743019154 shutdown_connections 2023-03-03T11:27:16.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.509+0000 7f95c0bff700 1 -- 172.21.15.102:0/743019154 wait complete. 2023-03-03T11:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.511+0000 7f95c0bff700 1 Processor -- start 2023-03-03T11:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.512+0000 7f95c0bff700 1 -- start start 2023-03-03T11:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95c0bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95bc0cd8e0 0x7f95bc14c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95c0bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 0x7f95bc155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95c0bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95bc1593c0 0x7f95bc156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95c0bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1ee930 con 0x7f95bc153290 2023-03-03T11:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95c0bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc0b45f0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95c0bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc0b1d10 con 0x7f95bc1593c0 2023-03-03T11:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95bc0cd8e0 0x7f95bc14c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95bc0cd8e0 0x7f95bc14c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33330/0 (socket says 172.21.15.102:33330) 2023-03-03T11:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 -- 172.21.15.102:0/216173940 learned_addr learned my addr 172.21.15.102:0/216173940 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95b9d9b700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 0x7f95bc155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95bad9d700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95bc1593c0 0x7f95bc156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:16.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 -- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95bc1593c0 msgr2=0x7f95bc156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:16.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95bc1593c0 0x7f95bc156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 -- 172.21.15.102:0/216173940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 msgr2=0x7f95bc155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 0x7f95bc155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.513+0000 7f95ba59c700 1 -- 172.21.15.102:0/216173940 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac009550 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.514+0000 7f95ba59c700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95bc0cd8e0 0x7f95bc14c770 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009c10 tx=0x7f95ac004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.514+0000 7f95aaffd700 1 -- 172.21.15.102:0/216173940 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac004380 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.514+0000 7f95aaffd700 1 -- 172.21.15.102:0/216173940 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac01f070 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.514+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc0ccde0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.514+0000 7f95aaffd700 1 -- 172.21.15.102:0/216173940 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0036a0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.514+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc15d160 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.515+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc0cd0c0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.517+0000 7f95aaffd700 1 -- 172.21.15.102:0/216173940 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f95ac02a020 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.518+0000 7f95aaffd700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f959c0682f0 0x7f959c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:16.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.518+0000 7f95aaffd700 1 -- 172.21.15.102:0/216173940 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f95ac0872c0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.518+0000 7f95b9d9b700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f959c0682f0 0x7f959c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:16.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.519+0000 7f95b9d9b700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f959c0682f0 0x7f959c06a7a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f95bc15a4d0 tx=0x7f95b0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.523+0000 7f95aaffd700 1 -- 172.21.15.102:0/216173940 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95ac0539d0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.757+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95bc04e6e0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.758+0000 7f95aaffd700 1 -- 172.21.15.102:0/216173940 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f95ac056ff0 con 0x7f95bc0cd8e0 2023-03-03T11:27:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:27:16.761 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:27:16.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f959c0682f0 msgr2=0x7f959c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f959c0682f0 0x7f959c06a7a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f95bc15a4d0 tx=0x7f95b0009210 comp rx=0 tx=0).stop 2023-03-03T11:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95bc0cd8e0 msgr2=0x7f95bc14c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95bc0cd8e0 0x7f95bc14c770 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009c10 tx=0x7f95ac004750 comp rx=0 tx=0).stop 2023-03-03T11:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 shutdown_connections 2023-03-03T11:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f959c0682f0 0x7f959c06a7a0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95bc1593c0 0x7f95bc156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95bc0cd8e0 0x7f95bc14c770 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 --2- 172.21.15.102:0/216173940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95bc153290 0x7f95bc155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 >> 172.21.15.102:0/216173940 conn(0x7f95bc065fb0 msgr2=0x7f95bc0cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 shutdown_connections 2023-03-03T11:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:27:16.761+0000 7f95c0bff700 1 -- 172.21.15.102:0/216173940 wait complete. 2023-03-03T11:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:27:16.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 100 secs before thrashing 2023-03-03T11:27:35.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:35.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:05.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:05.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:35.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:35.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:56.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:28:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.038+0000 7f2213bff700 1 Processor -- start 2023-03-03T11:28:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.038+0000 7f2213bff700 1 -- start start 2023-03-03T11:28:57.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.039+0000 7f2213bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f220c153290 0x7f220c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:57.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.039+0000 7f2213bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220c153660 con 0x7f220c1593c0 2023-03-03T11:28:57.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.039+0000 7f2213bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220c06c360 con 0x7f220c0cd8e0 2023-03-03T11:28:57.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.039+0000 7f2213bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220c0cec40 con 0x7f220c153290 2023-03-03T11:28:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.040+0000 7f221199b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f220c153290 0x7f220c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.040+0000 7f221119a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f220c1593c0 0x7f220c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50390/0 (socket says 172.21.15.102:50390) 2023-03-03T11:28:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.040+0000 7f221119a700 1 -- 172.21.15.102:0/2544158418 learned_addr learned my addr 172.21.15.102:0/2544158418 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:28:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.040+0000 7f221199b700 1 -- 172.21.15.102:0/2544158418 >> v1:172.21.15.172:6789/0 conn(0x7f220c0cd8e0 legacy=0x7f220c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.040+0000 7f2210999700 1 -- 172.21.15.102:0/2544158418 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506990 0 0) 0x7f220c06c360 con 0x7f220c0cd8e0 2023-03-03T11:28:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.040+0000 7f221199b700 1 -- 172.21.15.102:0/2544158418 >> v1:172.21.15.102:6789/0 conn(0x7f220c1593c0 legacy=0x7f220c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.040+0000 7f221199b700 1 -- 172.21.15.102:0/2544158418 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c15cea0 con 0x7f220c153290 2023-03-03T11:28:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.041+0000 7f221199b700 1 --2- 172.21.15.102:0/2544158418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c158e80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2208009840 tx=0x7f2208009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43333ffea6910698 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:57.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.041+0000 7f2210999700 1 -- 172.21.15.102:0/2544158418 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208011830 con 0x7f220c153290 2023-03-03T11:28:57.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.041+0000 7f2210999700 1 -- 172.21.15.102:0/2544158418 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2208011990 con 0x7f220c153290 2023-03-03T11:28:57.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.041+0000 7f2210999700 1 -- 172.21.15.102:0/2544158418 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208011d20 con 0x7f220c153290 2023-03-03T11:28:57.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.041+0000 7f2213bff700 1 -- 172.21.15.102:0/2544158418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 msgr2=0x7f220c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.041+0000 7f2213bff700 1 --2- 172.21.15.102:0/2544158418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c158e80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2208009840 tx=0x7f2208009b50 comp rx=0 tx=0).stop 2023-03-03T11:28:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.042+0000 7f2213bff700 1 -- 172.21.15.102:0/2544158418 shutdown_connections 2023-03-03T11:28:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.042+0000 7f2213bff700 1 --2- 172.21.15.102:0/2544158418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c158e80 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.042+0000 7f2213bff700 1 -- 172.21.15.102:0/2544158418 >> 172.21.15.102:0/2544158418 conn(0x7f220c065fb0 msgr2=0x7f220c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:28:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.042+0000 7f2213bff700 1 -- 172.21.15.102:0/2544158418 shutdown_connections 2023-03-03T11:28:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.042+0000 7f2213bff700 1 -- 172.21.15.102:0/2544158418 wait complete. 2023-03-03T11:28:57.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.044+0000 7f2213bff700 1 Processor -- start 2023-03-03T11:28:57.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.044+0000 7f2213bff700 1 -- start start 2023-03-03T11:28:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f2213bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f220c0cd8e0 0x7f220c1ef490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f2213bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f2213bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f220c1593c0 0x7f220c15b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f2213bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c15ad20 con 0x7f220c153290 2023-03-03T11:28:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f2213bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c15b000 con 0x7f220c0cd8e0 2023-03-03T11:28:57.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f2213bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c1546b0 con 0x7f220c1593c0 2023-03-03T11:28:57.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f221119a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:57.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f221119a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52036/0 (socket says 172.21.15.102:52036) 2023-03-03T11:28:57.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f221119a700 1 -- 172.21.15.102:0/3467035242 learned_addr learned my addr 172.21.15.102:0/3467035242 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:28:57.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f221199b700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f220c0cd8e0 0x7f220c1ef490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:57.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.045+0000 7f221219c700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f220c1593c0 0x7f220c15b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:57.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f221119a700 1 -- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f220c1593c0 msgr2=0x7f220c15b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f221119a700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f220c1593c0 0x7f220c15b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f221119a700 1 -- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f220c0cd8e0 msgr2=0x7f220c1ef490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f221119a700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f220c0cd8e0 0x7f220c1ef490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f221119a700 1 -- 172.21.15.102:0/3467035242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22080094a0 con 0x7f220c153290 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f221119a700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c157ba0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c550 tx=0x7f21fc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f22027fc700 1 -- 172.21.15.102:0/3467035242 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc005d70 con 0x7f220c153290 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f22027fc700 1 -- 172.21.15.102:0/3467035242 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc0070d0 con 0x7f220c153290 2023-03-03T11:28:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c15d160 con 0x7f220c153290 2023-03-03T11:28:57.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.046+0000 7f22027fc700 1 -- 172.21.15.102:0/3467035242 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc007440 con 0x7f220c153290 2023-03-03T11:28:57.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.047+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c06c7f0 con 0x7f220c153290 2023-03-03T11:28:57.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.048+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c0ccde0 con 0x7f220c153290 2023-03-03T11:28:57.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.049+0000 7f22027fc700 1 -- 172.21.15.102:0/3467035242 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f21fc019030 con 0x7f220c153290 2023-03-03T11:28:57.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.050+0000 7f22027fc700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f21f0068320 0x7f21f006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:57.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.050+0000 7f22027fc700 1 -- 172.21.15.102:0/3467035242 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f21fc084f30 con 0x7f220c153290 2023-03-03T11:28:57.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.051+0000 7f221199b700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f21f0068320 0x7f21f006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:57.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.051+0000 7f221199b700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f21f0068320 0x7f21f006a7d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f22080090b0 tx=0x7f220801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:57.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.054+0000 7f22027fc700 1 -- 172.21.15.102:0/3467035242 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f21fc04b5e0 con 0x7f220c153290 2023-03-03T11:28:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.283+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f220c04e6e0 con 0x7f220c153290 2023-03-03T11:28:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.284+0000 7f22027fc700 1 -- 172.21.15.102:0/3467035242 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f21fc017090 con 0x7f220c153290 2023-03-03T11:28:57.287 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:28:57.287 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:28:57.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.286+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f21f0068320 msgr2=0x7f21f006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:57.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.286+0000 7f2213bff700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f21f0068320 0x7f21f006a7d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f22080090b0 tx=0x7f220801c040 comp rx=0 tx=0).stop 2023-03-03T11:28:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.286+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 msgr2=0x7f220c157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.286+0000 7f2213bff700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c157ba0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c550 tx=0x7f21fc00c860 comp rx=0 tx=0).stop 2023-03-03T11:28:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 shutdown_connections 2023-03-03T11:28:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f21f0068320 0x7f21f006a7d0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f220c1593c0 0x7f220c15b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f220c0cd8e0 0x7f220c1ef490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:57.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 --2- 172.21.15.102:0/3467035242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f220c153290 0x7f220c157ba0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:57.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 >> 172.21.15.102:0/3467035242 conn(0x7f220c065fb0 msgr2=0x7f220c0b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:28:57.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 shutdown_connections 2023-03-03T11:28:57.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:28:57.287+0000 7f2213bff700 1 -- 172.21.15.102:0/3467035242 wait complete. 2023-03-03T11:28:57.291 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:28:57.316 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 78 secs before thrashing 2023-03-03T11:29:06.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:06.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:36.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:36.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T11:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:30:06.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:06.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:15.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:30:15.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.568+0000 7f93bb9ff700 1 Processor -- start 2023-03-03T11:30:15.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.568+0000 7f93bb9ff700 1 -- start start 2023-03-03T11:30:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.569+0000 7f93bb9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93b4153290 0x7f93b4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.569+0000 7f93bb9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93b41593c0 0x7f93b40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.569+0000 7f93bb9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b40cd8e0 0x7f93b40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.569+0000 7f93bb9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b4153660 con 0x7f93b41593c0 2023-03-03T11:30:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.569+0000 7f93bb9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b406c360 con 0x7f93b4153290 2023-03-03T11:30:15.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.569+0000 7f93bb9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b415cea0 con 0x7f93b40cd8e0 2023-03-03T11:30:15.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b40cd8e0 0x7f93b40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b979b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93b4153290 0x7f93b4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b40cd8e0 0x7f93b40ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55590/0 (socket says 172.21.15.102:55590) 2023-03-03T11:30:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b979b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93b4153290 0x7f93b4158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35498/0 (socket says 172.21.15.102:35498) 2023-03-03T11:30:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 -- 172.21.15.102:0/1455482340 learned_addr learned my addr 172.21.15.102:0/1455482340 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:30:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b8f9a700 1 --2- 172.21.15.102:0/1455482340 >> v2:172.21.15.102:3300/0 conn(0x7f93b41593c0 0x7f93b40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:15.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 -- 172.21.15.102:0/1455482340 >> v2:172.21.15.172:3300/0 conn(0x7f93b4153290 msgr2=0x7f93b4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:30:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 --2- 172.21.15.102:0/1455482340 >> v2:172.21.15.172:3300/0 conn(0x7f93b4153290 0x7f93b4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 -- 172.21.15.102:0/1455482340 >> v2:172.21.15.102:3300/0 conn(0x7f93b41593c0 msgr2=0x7f93b40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:30:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 --2- 172.21.15.102:0/1455482340 >> v2:172.21.15.102:3300/0 conn(0x7f93b41593c0 0x7f93b40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 -- 172.21.15.102:0/1455482340 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b406c7f0 con 0x7f93b40cd8e0 2023-03-03T11:30:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b979b700 1 --2- 172.21.15.102:0/1455482340 >> v2:172.21.15.172:3300/0 conn(0x7f93b4153290 0x7f93b4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:30:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.570+0000 7f93b9f9c700 1 --2- 172.21.15.102:0/1455482340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b40cd8e0 0x7f93b40ce360 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f93b408cc70 tx=0x7f93ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9243bfc6f2baa619 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.571+0000 7f93abfff700 1 -- 172.21.15.102:0/1455482340 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00e070 con 0x7f93b40cd8e0 2023-03-03T11:30:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.571+0000 7f93abfff700 1 -- 172.21.15.102:0/1455482340 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac00b910 con 0x7f93b40cd8e0 2023-03-03T11:30:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.571+0000 7f93abfff700 1 -- 172.21.15.102:0/1455482340 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00bcc0 con 0x7f93b40cd8e0 2023-03-03T11:30:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.571+0000 7f93bb9ff700 1 -- 172.21.15.102:0/1455482340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b40cd8e0 msgr2=0x7f93b40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.571+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/1455482340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b40cd8e0 0x7f93b40ce360 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f93b408cc70 tx=0x7f93ac00b330 comp rx=0 tx=0).stop 2023-03-03T11:30:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.572+0000 7f93bb9ff700 1 -- 172.21.15.102:0/1455482340 shutdown_connections 2023-03-03T11:30:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.572+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/1455482340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b40cd8e0 0x7f93b40ce360 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.572+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/1455482340 >> v2:172.21.15.172:3300/0 conn(0x7f93b4153290 0x7f93b4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.572+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/1455482340 >> v2:172.21.15.102:3300/0 conn(0x7f93b41593c0 0x7f93b40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.572+0000 7f93bb9ff700 1 -- 172.21.15.102:0/1455482340 >> 172.21.15.102:0/1455482340 conn(0x7f93b4065fb0 msgr2=0x7f93b40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:15.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.572+0000 7f93bb9ff700 1 -- 172.21.15.102:0/1455482340 shutdown_connections 2023-03-03T11:30:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.572+0000 7f93bb9ff700 1 -- 172.21.15.102:0/1455482340 wait complete. 2023-03-03T11:30:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.575+0000 7f93bb9ff700 1 Processor -- start 2023-03-03T11:30:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.575+0000 7f93bb9ff700 1 -- start start 2023-03-03T11:30:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.576+0000 7f93bb9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b40cd8e0 0x7f93b41efd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.576+0000 7f93bb9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b4153290 0x7f93b41f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.576+0000 7f93bb9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93b41593c0 0x7f93b40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.576+0000 7f93bb9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40c5950 con 0x7f93b40cd8e0 2023-03-03T11:30:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.576+0000 7f93bb9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b41e50c0 con 0x7f93b41593c0 2023-03-03T11:30:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.576+0000 7f93bb9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b41e3be0 con 0x7f93b4153290 2023-03-03T11:30:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b4153290 0x7f93b41f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b4153290 0x7f93b41f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55606/0 (socket says 172.21.15.102:55606) 2023-03-03T11:30:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 -- 172.21.15.102:0/377191856 learned_addr learned my addr 172.21.15.102:0/377191856 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:30:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b9f9c700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93b41593c0 0x7f93b40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:15.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b979b700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b40cd8e0 0x7f93b41efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:15.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 -- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93b41593c0 msgr2=0x7f93b40cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93b41593c0 0x7f93b40cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 -- 172.21.15.102:0/377191856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b40cd8e0 msgr2=0x7f93b41efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b40cd8e0 0x7f93b41efd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 -- 172.21.15.102:0/377191856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00094a0 con 0x7f93b4153290 2023-03-03T11:30:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b9f9c700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93b41593c0 0x7f93b40cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:30:15.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b979b700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b40cd8e0 0x7f93b41efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:30:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.577+0000 7f93b8f9a700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b4153290 0x7f93b41f02d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f93a400e870 tx=0x7f93a400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.578+0000 7f93a9ffb700 1 -- 172.21.15.102:0/377191856 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a4009590 con 0x7f93b4153290 2023-03-03T11:30:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.578+0000 7f93a9ffb700 1 -- 172.21.15.102:0/377191856 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a4013680 con 0x7f93b4153290 2023-03-03T11:30:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.578+0000 7f93a9ffb700 1 -- 172.21.15.102:0/377191856 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a40139f0 con 0x7f93b4153290 2023-03-03T11:30:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.578+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ac009b30 con 0x7f93b4153290 2023-03-03T11:30:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.578+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b40ccde0 con 0x7f93b4153290 2023-03-03T11:30:15.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.579+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b415d160 con 0x7f93b4153290 2023-03-03T11:30:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.580+0000 7f93a9ffb700 1 -- 172.21.15.102:0/377191856 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f93a4023070 con 0x7f93b4153290 2023-03-03T11:30:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.581+0000 7f93a9ffb700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f93980682f0 0x7f939806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.581+0000 7f93a9ffb700 1 -- 172.21.15.102:0/377191856 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f93a4085e60 con 0x7f93b4153290 2023-03-03T11:30:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.582+0000 7f93b979b700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f93980682f0 0x7f939806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:15.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.582+0000 7f93b979b700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f93980682f0 0x7f939806a7a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f93b0011380 tx=0x7f93b0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.585+0000 7f93a9ffb700 1 -- 172.21.15.102:0/377191856 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f93a4052480 con 0x7f93b4153290 2023-03-03T11:30:15.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.819+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93b40cd0c0 con 0x7f93b4153290 2023-03-03T11:30:15.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.820+0000 7f93a9ffb700 1 -- 172.21.15.102:0/377191856 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f93a4017090 con 0x7f93b4153290 2023-03-03T11:30:15.822 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:30:15.823 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:30:15.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f93980682f0 msgr2=0x7f939806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:15.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f93980682f0 0x7f939806a7a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f93b0011380 tx=0x7f93b0009210 comp rx=0 tx=0).stop 2023-03-03T11:30:15.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b4153290 msgr2=0x7f93b41f02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:15.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b4153290 0x7f93b41f02d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f93a400e870 tx=0x7f93a400eb80 comp rx=0 tx=0).stop 2023-03-03T11:30:15.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 shutdown_connections 2023-03-03T11:30:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93b41593c0 0x7f93b40cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b40cd8e0 0x7f93b41efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f93980682f0 0x7f939806a7a0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 --2- 172.21.15.102:0/377191856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93b4153290 0x7f93b41f02d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 >> 172.21.15.102:0/377191856 conn(0x7f93b4065fb0 msgr2=0x7f93b415b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 shutdown_connections 2023-03-03T11:30:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:30:15.823+0000 7f93bb9ff700 1 -- 172.21.15.102:0/377191856 wait complete. 2023-03-03T11:30:15.827 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:30:15.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2023-03-03T11:30:36.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:36.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:06.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:06.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:19.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:31:20.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.102+0000 7f80a2bff700 1 Processor -- start 2023-03-03T11:31:20.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.102+0000 7f80a2bff700 1 -- start start 2023-03-03T11:31:20.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.104+0000 7f80a2bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f809c153290 0x7f809c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:31:20.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.104+0000 7f80a2bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c0cd8e0 0x7f809c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:31:20.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.104+0000 7f80a2bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809c153660 con 0x7f809c1593c0 2023-03-03T11:31:20.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.104+0000 7f80a2bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809c06c360 con 0x7f809c153290 2023-03-03T11:31:20.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.104+0000 7f80a2bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c15cea0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f80a099b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f809c153290 0x7f809c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:31:20.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f809bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f809c1593c0 0x7f809c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39470/0 (socket says 172.21.15.102:39470) 2023-03-03T11:31:20.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f809bfff700 1 -- 172.21.15.102:0/3864356851 learned_addr learned my addr 172.21.15.102:0/3864356851 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:31:20.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f80a119c700 1 --2- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c0cd8e0 0x7f809c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:31:20.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f80a099b700 1 -- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c0cd8e0 msgr2=0x7f809c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:31:20.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f80a099b700 1 --2- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c0cd8e0 0x7f809c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f80a099b700 1 -- 172.21.15.102:0/3864356851 >> v1:172.21.15.102:6789/0 conn(0x7f809c1593c0 legacy=0x7f809c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:31:20.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f809b7fe700 1 -- 172.21.15.102:0/3864356851 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093586310 0 0) 0x7f809c153660 con 0x7f809c1593c0 2023-03-03T11:31:20.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.105+0000 7f80a099b700 1 -- 172.21.15.102:0/3864356851 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c06c7f0 con 0x7f809c153290 2023-03-03T11:31:20.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.106+0000 7f80a099b700 1 --2- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c153290 0x7f809c158e80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8090009840 tx=0x7f8090009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c2771a9b2bbfa9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:31:20.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.106+0000 7f809b7fe700 1 -- 172.21.15.102:0/3864356851 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090011830 con 0x7f809c153290 2023-03-03T11:31:20.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.106+0000 7f809b7fe700 1 -- 172.21.15.102:0/3864356851 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8090011990 con 0x7f809c153290 2023-03-03T11:31:20.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.106+0000 7f809b7fe700 1 -- 172.21.15.102:0/3864356851 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090011d20 con 0x7f809c153290 2023-03-03T11:31:20.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.106+0000 7f80a2bff700 1 -- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c153290 msgr2=0x7f809c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:31:20.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.106+0000 7f80a2bff700 1 --2- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c153290 0x7f809c158e80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8090009840 tx=0x7f8090009b50 comp rx=0 tx=0).stop 2023-03-03T11:31:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.106+0000 7f80a2bff700 1 -- 172.21.15.102:0/3864356851 shutdown_connections 2023-03-03T11:31:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.107+0000 7f80a2bff700 1 --2- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c0cd8e0 0x7f809c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.107+0000 7f80a2bff700 1 --2- 172.21.15.102:0/3864356851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c153290 0x7f809c158e80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.107+0000 7f80a2bff700 1 -- 172.21.15.102:0/3864356851 >> 172.21.15.102:0/3864356851 conn(0x7f809c065fb0 msgr2=0x7f809c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:31:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.107+0000 7f80a2bff700 1 -- 172.21.15.102:0/3864356851 shutdown_connections 2023-03-03T11:31:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.107+0000 7f80a2bff700 1 -- 172.21.15.102:0/3864356851 wait complete. 2023-03-03T11:31:20.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.109+0000 7f80a2bff700 1 Processor -- start 2023-03-03T11:31:20.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.110+0000 7f80a2bff700 1 -- start start 2023-03-03T11:31:20.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a2bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c0cd8e0 0x7f809c0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:31:20.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a2bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c153290 0x7f809c14e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:31:20.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a2bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f809c1593c0 0x7f809c14e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:31:20.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a2bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0b1f90 con 0x7f809c1593c0 2023-03-03T11:31:20.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a2bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0b2860 con 0x7f809c0cd8e0 2023-03-03T11:31:20.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a2bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0b2180 con 0x7f809c153290 2023-03-03T11:31:20.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a099b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c0cd8e0 0x7f809c0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:31:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f809bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c153290 0x7f809c14e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:31:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a099b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c0cd8e0 0x7f809c0c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42692/0 (socket says 172.21.15.102:42692) 2023-03-03T11:31:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.111+0000 7f80a099b700 1 -- 172.21.15.102:0/830547588 learned_addr learned my addr 172.21.15.102:0/830547588 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:31:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.112+0000 7f80a119c700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f809c1593c0 0x7f809c14e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:31:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.112+0000 7f80a099b700 1 -- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c153290 msgr2=0x7f809c14e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:31:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.112+0000 7f80a099b700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c153290 0x7f809c14e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.112+0000 7f80a099b700 1 -- 172.21.15.102:0/830547588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f809c1593c0 msgr2=0x7f809c14e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:31:20.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.112+0000 7f80a099b700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f809c1593c0 0x7f809c14e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.112+0000 7f80a099b700 1 -- 172.21.15.102:0/830547588 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900094a0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.112+0000 7f80a099b700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c0cd8e0 0x7f809c0c4750 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8090009fd0 tx=0x7f8090004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:31:20.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.113+0000 7f80997fa700 1 -- 172.21.15.102:0/830547588 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80900036a0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.113+0000 7f80997fa700 1 -- 172.21.15.102:0/830547588 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809001e070 con 0x7f809c0cd8e0 2023-03-03T11:31:20.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.113+0000 7f80997fa700 1 -- 172.21.15.102:0/830547588 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090003a40 con 0x7f809c0cd8e0 2023-03-03T11:31:20.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.113+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809c0ccde0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.113+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809c15d160 con 0x7f809c0cd8e0 2023-03-03T11:31:20.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.114+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809c0cd0c0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.115+0000 7f80997fa700 1 -- 172.21.15.102:0/830547588 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f809000f050 con 0x7f809c0cd8e0 2023-03-03T11:31:20.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.117+0000 7f80997fa700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8080068320 0x7f808006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:31:20.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.117+0000 7f80997fa700 1 -- 172.21.15.102:0/830547588 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f80900871c0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.117+0000 7f809bfff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8080068320 0x7f808006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:31:20.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.118+0000 7f809bfff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8080068320 0x7f808006a7d0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f809c15bdd0 tx=0x7f8088007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:31:20.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.119+0000 7f80997fa700 1 -- 172.21.15.102:0/830547588 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80900538d0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.353+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f809c04e6e0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.354+0000 7f80997fa700 1 -- 172.21.15.102:0/830547588 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8090056ef0 con 0x7f809c0cd8e0 2023-03-03T11:31:20.357 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:31:20.357 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:31:20.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8080068320 msgr2=0x7f808006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:31:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8080068320 0x7f808006a7d0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f809c15bdd0 tx=0x7f8088007de0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c0cd8e0 msgr2=0x7f809c0c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:31:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c0cd8e0 0x7f809c0c4750 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8090009fd0 tx=0x7f8090004750 comp rx=0 tx=0).stop 2023-03-03T11:31:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 shutdown_connections 2023-03-03T11:31:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f809c1593c0 0x7f809c14e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f809c0cd8e0 0x7f809c0c4750 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8080068320 0x7f808006a7d0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 --2- 172.21.15.102:0/830547588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f809c153290 0x7f809c14e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:31:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 >> 172.21.15.102:0/830547588 conn(0x7f809c065fb0 msgr2=0x7f809c150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:31:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 shutdown_connections 2023-03-03T11:31:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:31:20.357+0000 7f80a2bff700 1 -- 172.21.15.102:0/830547588 wait complete. 2023-03-03T11:31:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:31:20.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2023-03-03T11:31:37.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:37.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:07.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:07.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:37.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:37.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:54.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:32:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.636+0000 7f48643ff700 1 Processor -- start 2023-03-03T11:32:54.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.636+0000 7f48643ff700 1 -- start start 2023-03-03T11:32:54.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.638+0000 7f48643ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f485c0cd8e0 0x7f485c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:54.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.638+0000 7f48643ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c0cd390 0x7f485c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:54.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.638+0000 7f48643ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c153660 con 0x7f485c153290 2023-03-03T11:32:54.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.638+0000 7f48643ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c06c360 con 0x7f485c0cd8e0 2023-03-03T11:32:54.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.638+0000 7f48643ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c15cea0 con 0x7f485c0cd390 2023-03-03T11:32:54.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486219b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f485c153290 0x7f485c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45698/0 (socket says 172.21.15.102:45698) 2023-03-03T11:32:54.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486219b700 1 -- 172.21.15.102:0/2454484678 learned_addr learned my addr 172.21.15.102:0/2454484678 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:32:54.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486199a700 1 --2- 172.21.15.102:0/2454484678 >> v2:172.21.15.102:3300/0 conn(0x7f485c0cd8e0 0x7f485c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:54.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486299c700 1 --2- 172.21.15.102:0/2454484678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c0cd390 0x7f485c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:54.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f4861199700 1 -- 172.21.15.102:0/2454484678 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205859139 0 0) 0x7f485c153660 con 0x7f485c153290 2023-03-03T11:32:54.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f4861199700 1 -- 172.21.15.102:0/2454484678 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48480035b0 con 0x7f485c153290 2023-03-03T11:32:54.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486199a700 1 -- 172.21.15.102:0/2454484678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c0cd390 msgr2=0x7f485c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:54.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486199a700 1 --2- 172.21.15.102:0/2454484678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c0cd390 0x7f485c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486199a700 1 -- 172.21.15.102:0/2454484678 >> v1:172.21.15.172:6789/0 conn(0x7f485c153290 legacy=0x7f485c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:54.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.639+0000 7f486199a700 1 -- 172.21.15.102:0/2454484678 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c06c7f0 con 0x7f485c0cd8e0 2023-03-03T11:32:54.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.640+0000 7f486199a700 1 --2- 172.21.15.102:0/2454484678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd8e0 0x7f485c0cde20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f484c009840 tx=0x7f484c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6aded2f0ed317a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:54.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.640+0000 7f4861199700 1 -- 172.21.15.102:0/2454484678 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c011830 con 0x7f485c0cd8e0 2023-03-03T11:32:54.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.640+0000 7f4861199700 1 -- 172.21.15.102:0/2454484678 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c011990 con 0x7f485c0cd8e0 2023-03-03T11:32:54.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.640+0000 7f4861199700 1 -- 172.21.15.102:0/2454484678 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c011d20 con 0x7f485c0cd8e0 2023-03-03T11:32:54.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.640+0000 7f48643ff700 1 -- 172.21.15.102:0/2454484678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd8e0 msgr2=0x7f485c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:54.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.640+0000 7f48643ff700 1 --2- 172.21.15.102:0/2454484678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd8e0 0x7f485c0cde20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f484c009840 tx=0x7f484c009b50 comp rx=0 tx=0).stop 2023-03-03T11:32:54.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.641+0000 7f48643ff700 1 -- 172.21.15.102:0/2454484678 shutdown_connections 2023-03-03T11:32:54.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.641+0000 7f48643ff700 1 --2- 172.21.15.102:0/2454484678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c0cd390 0x7f485c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.641+0000 7f48643ff700 1 --2- 172.21.15.102:0/2454484678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd8e0 0x7f485c0cde20 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.641+0000 7f48643ff700 1 -- 172.21.15.102:0/2454484678 >> 172.21.15.102:0/2454484678 conn(0x7f485c065fb0 msgr2=0x7f485c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.641+0000 7f48643ff700 1 -- 172.21.15.102:0/2454484678 shutdown_connections 2023-03-03T11:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.641+0000 7f48643ff700 1 -- 172.21.15.102:0/2454484678 wait complete. 2023-03-03T11:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.643+0000 7f48643ff700 1 Processor -- start 2023-03-03T11:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.644+0000 7f48643ff700 1 -- start start 2023-03-03T11:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f48643ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd390 0x7f485c153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f48643ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cd8e0 0x7f485c14f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:54.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f48643ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c153290 0x7f485c14fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:54.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f48643ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c155360 con 0x7f485c0cd390 2023-03-03T11:32:54.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f48643ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c155b60 con 0x7f485c0cd8e0 2023-03-03T11:32:54.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f48643ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0b4b10 con 0x7f485c153290 2023-03-03T11:32:54.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f486219b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd390 0x7f485c153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:54.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f486199a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cd8e0 0x7f485c14f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:54.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f486299c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c153290 0x7f485c14fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:54.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f486219b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd390 0x7f485c153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42308/0 (socket says 172.21.15.102:42308) 2023-03-03T11:32:54.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486219b700 1 -- 172.21.15.102:0/3503198326 learned_addr learned my addr 172.21.15.102:0/3503198326 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:32:54.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.645+0000 7f486199a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cd8e0 0x7f485c14f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39382/0 (socket says 172.21.15.102:39382) 2023-03-03T11:32:54.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486199a700 1 -- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c153290 msgr2=0x7f485c14fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:54.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486199a700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c153290 0x7f485c14fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486199a700 1 -- 172.21.15.102:0/3503198326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd390 msgr2=0x7f485c153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:54.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486199a700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd390 0x7f485c153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486199a700 1 -- 172.21.15.102:0/3503198326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0094a0 con 0x7f485c0cd8e0 2023-03-03T11:32:54.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486219b700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd390 0x7f485c153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:32:54.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.646+0000 7f486199a700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cd8e0 0x7f485c14f610 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f484c016000 tx=0x7f484c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:54.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.647+0000 7f4852ffd700 1 -- 172.21.15.102:0/3503198326 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c004590 con 0x7f485c0cd8e0 2023-03-03T11:32:54.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.647+0000 7f4852ffd700 1 -- 172.21.15.102:0/3503198326 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c01e070 con 0x7f485c0cd8e0 2023-03-03T11:32:54.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.647+0000 7f4852ffd700 1 -- 172.21.15.102:0/3503198326 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0038e0 con 0x7f485c0cd8e0 2023-03-03T11:32:54.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.647+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c0ccde0 con 0x7f485c0cd8e0 2023-03-03T11:32:54.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.647+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c15d160 con 0x7f485c0cd8e0 2023-03-03T11:32:54.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.648+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c0cd0c0 con 0x7f485c0cd8e0 2023-03-03T11:32:54.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.649+0000 7f4852ffd700 1 -- 172.21.15.102:0/3503198326 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f484c00f050 con 0x7f485c0cd8e0 2023-03-03T11:32:54.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.650+0000 7f4852ffd700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f48400682f0 0x7f484006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:54.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.650+0000 7f4852ffd700 1 -- 172.21.15.102:0/3503198326 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f484c086d10 con 0x7f485c0cd8e0 2023-03-03T11:32:54.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.650+0000 7f486219b700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f48400682f0 0x7f484006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:54.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.651+0000 7f486219b700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f48400682f0 0x7f484006a7a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f48580011e0 tx=0x7f4858009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:54.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.654+0000 7f4852ffd700 1 -- 172.21.15.102:0/3503198326 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f484c0533a0 con 0x7f485c0cd8e0 2023-03-03T11:32:54.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.887+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f485c0ca380 con 0x7f485c0cd8e0 2023-03-03T11:32:54.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.889+0000 7f4852ffd700 1 -- 172.21.15.102:0/3503198326 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f484c0569c0 con 0x7f485c0cd8e0 2023-03-03T11:32:54.891 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:32:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:32:54.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f48400682f0 msgr2=0x7f484006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:54.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f48400682f0 0x7f484006a7a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f48580011e0 tx=0x7f4858009040 comp rx=0 tx=0).stop 2023-03-03T11:32:54.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cd8e0 msgr2=0x7f485c14f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cd8e0 0x7f485c14f610 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f484c016000 tx=0x7f484c004750 comp rx=0 tx=0).stop 2023-03-03T11:32:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 shutdown_connections 2023-03-03T11:32:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f48400682f0 0x7f484006a7a0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f485c153290 0x7f485c14fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f485c0cd390 0x7f485c153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 --2- 172.21.15.102:0/3503198326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f485c0cd8e0 0x7f485c14f610 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 >> 172.21.15.102:0/3503198326 conn(0x7f485c065fb0 msgr2=0x7f485c158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 shutdown_connections 2023-03-03T11:32:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:32:54.891+0000 7f48643ff700 1 -- 172.21.15.102:0/3503198326 wait complete. 2023-03-03T11:32:54.896 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:32:54.921 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-03-03T11:33:07.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:07.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:27.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:33:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.172+0000 7f67cedff700 1 Processor -- start 2023-03-03T11:33:28.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.172+0000 7f67cedff700 1 -- start start 2023-03-03T11:33:28.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.173+0000 7f67cedff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:28.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.173+0000 7f67cedff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f67c81593c0 0x7f67c80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:28.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.173+0000 7f67cedff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c8153660 con 0x7f67c80cd8e0 2023-03-03T11:33:28.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.173+0000 7f67cedff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c806c360 con 0x7f67c81593c0 2023-03-03T11:33:28.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.173+0000 7f67cedff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c815cea0 con 0x7f67c8153290 2023-03-03T11:33:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67ccb9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67c7fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f67c81593c0 0x7f67c80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67cd39c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f67c80cd8e0 0x7f67c80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51742/0 (socket says 172.21.15.102:51742) 2023-03-03T11:33:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67ccb9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39914/0 (socket says 172.21.15.102:39914) 2023-03-03T11:33:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67cd39c700 1 -- 172.21.15.102:0/1279391924 learned_addr learned my addr 172.21.15.102:0/1279391924 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:33:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67c7fff700 1 -- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 msgr2=0x7f67c8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67c7fff700 1 --2- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67c7fff700 1 -- 172.21.15.102:0/1279391924 >> v1:172.21.15.172:6789/0 conn(0x7f67c80cd8e0 legacy=0x7f67c80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67c7fff700 1 -- 172.21.15.102:0/1279391924 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c806c7f0 con 0x7f67c81593c0 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67ccb9b700 1 --2- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.174+0000 7f67c77fe700 1 -- 172.21.15.102:0/1279391924 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779792117 0 0) 0x7f67c8153660 con 0x7f67c80cd8e0 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.175+0000 7f67c7fff700 1 --2- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 0x7f67c80cde20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009840 tx=0x7f67b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63e8627d4b941ddb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.175+0000 7f67c77fe700 1 -- 172.21.15.102:0/1279391924 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4011830 con 0x7f67c81593c0 2023-03-03T11:33:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.175+0000 7f67c77fe700 1 -- 172.21.15.102:0/1279391924 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b4011990 con 0x7f67c81593c0 2023-03-03T11:33:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.175+0000 7f67c77fe700 1 -- 172.21.15.102:0/1279391924 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4011d20 con 0x7f67c81593c0 2023-03-03T11:33:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 -- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 msgr2=0x7f67c80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 --2- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 0x7f67c80cde20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009840 tx=0x7f67b4009b50 comp rx=0 tx=0).stop 2023-03-03T11:33:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 -- 172.21.15.102:0/1279391924 shutdown_connections 2023-03-03T11:33:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 --2- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 --2- 172.21.15.102:0/1279391924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 0x7f67c80cde20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 -- 172.21.15.102:0/1279391924 >> 172.21.15.102:0/1279391924 conn(0x7f67c8065fb0 msgr2=0x7f67c80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 -- 172.21.15.102:0/1279391924 shutdown_connections 2023-03-03T11:33:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.176+0000 7f67cedff700 1 -- 172.21.15.102:0/1279391924 wait complete. 2023-03-03T11:33:28.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.178+0000 7f67cedff700 1 Processor -- start 2023-03-03T11:33:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.178+0000 7f67cedff700 1 -- start start 2023-03-03T11:33:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.179+0000 7f67cedff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67c80cd8e0 0x7f67c81ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.179+0000 7f67cedff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c81f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.179+0000 7f67cedff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 0x7f67c814c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.179+0000 7f67cedff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c81f29f0 con 0x7f67c80cd8e0 2023-03-03T11:33:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.179+0000 7f67cedff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80cc680 con 0x7f67c81593c0 2023-03-03T11:33:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.179+0000 7f67cedff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c81f2710 con 0x7f67c8153290 2023-03-03T11:33:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67ccb9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67c80cd8e0 0x7f67c81ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c81f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c81f1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39916/0 (socket says 172.21.15.102:39916) 2023-03-03T11:33:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67ccb9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67c80cd8e0 0x7f67c81ef9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47790/0 (socket says 172.21.15.102:47790) 2023-03-03T11:33:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 -- 172.21.15.102:0/824961692 learned_addr learned my addr 172.21.15.102:0/824961692 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:33:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67cd39c700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 0x7f67c814c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:28.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 -- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 msgr2=0x7f67c814c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:28.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 0x7f67c814c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 -- 172.21.15.102:0/824961692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67c80cd8e0 msgr2=0x7f67c81ef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67c80cd8e0 0x7f67c81ef9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67c7fff700 1 -- 172.21.15.102:0/824961692 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc0094a0 con 0x7f67c8153290 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.180+0000 7f67ccb9b700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67c80cd8e0 0x7f67c81ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.181+0000 7f67c7fff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c81f1d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009b90 tx=0x7f67b4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.181+0000 7f67c57fa700 1 -- 172.21.15.102:0/824961692 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b40036a0 con 0x7f67c8153290 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.181+0000 7f67c57fa700 1 -- 172.21.15.102:0/824961692 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b401f070 con 0x7f67c8153290 2023-03-03T11:33:28.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.181+0000 7f67c57fa700 1 -- 172.21.15.102:0/824961692 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4003a40 con 0x7f67c8153290 2023-03-03T11:33:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.181+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b40094a0 con 0x7f67c8153290 2023-03-03T11:33:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.181+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c80ccde0 con 0x7f67c8153290 2023-03-03T11:33:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.182+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c815d160 con 0x7f67c8153290 2023-03-03T11:33:28.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.183+0000 7f67c57fa700 1 -- 172.21.15.102:0/824961692 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f67b400f050 con 0x7f67c8153290 2023-03-03T11:33:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.185+0000 7f67c57fa700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67ac0682f0 0x7f67ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.185+0000 7f67c57fa700 1 -- 172.21.15.102:0/824961692 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f67b4021070 con 0x7f67c8153290 2023-03-03T11:33:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.185+0000 7f67ccb9b700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67ac0682f0 0x7f67ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:28.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.186+0000 7f67ccb9b700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67ac0682f0 0x7f67ac06a7a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f67bc011380 tx=0x7f67bc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:28.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.190+0000 7f67c57fa700 1 -- 172.21.15.102:0/824961692 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f67b4053cf0 con 0x7f67c8153290 2023-03-03T11:33:28.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.423+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67c80cd0c0 con 0x7f67c8153290 2023-03-03T11:33:28.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.424+0000 7f67c57fa700 1 -- 172.21.15.102:0/824961692 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f67b4057310 con 0x7f67c8153290 2023-03-03T11:33:28.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:33:28.427 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:33:28.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.426+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67ac0682f0 msgr2=0x7f67ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:28.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.426+0000 7f67cedff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67ac0682f0 0x7f67ac06a7a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f67bc011380 tx=0x7f67bc009210 comp rx=0 tx=0).stop 2023-03-03T11:33:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.426+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 msgr2=0x7f67c81f1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.426+0000 7f67cedff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c81f1d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009b90 tx=0x7f67b4004750 comp rx=0 tx=0).stop 2023-03-03T11:33:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 shutdown_connections 2023-03-03T11:33:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f67c81593c0 0x7f67c814c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67c80cd8e0 0x7f67c81ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f67ac0682f0 0x7f67ac06a7a0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 --2- 172.21.15.102:0/824961692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f67c8153290 0x7f67c81f1d50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:28.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 >> 172.21.15.102:0/824961692 conn(0x7f67c8065fb0 msgr2=0x7f67c80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:28.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 shutdown_connections 2023-03-03T11:33:28.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:33:28.427+0000 7f67cedff700 1 -- 172.21.15.102:0/824961692 wait complete. 2023-03-03T11:33:28.431 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:33:28.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-03-03T11:33:37.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:37.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:08.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:08.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:38.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:38.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:08.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:08.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:15.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:35:15.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.711+0000 7f9c811ff700 1 Processor -- start 2023-03-03T11:35:15.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.711+0000 7f9c811ff700 1 -- start start 2023-03-03T11:35:15.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.712+0000 7f9c811ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9c7c153290 0x7f9c7c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:15.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.712+0000 7f9c811ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9c7c0cd8e0 0x7f9c7c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:15.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c811ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c153660 con 0x7f9c7c1593c0 2023-03-03T11:35:15.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c811ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7c06c360 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c811ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7c15cea0 con 0x7f9c7c153290 2023-03-03T11:35:15.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7b59e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9c7c0cd8e0 0x7f9c7c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:15.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7b59e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9c7c0cd8e0 0x7f9c7c0ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52298/0 (socket says 172.21.15.102:52298) 2023-03-03T11:35:15.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7b59e700 1 -- 172.21.15.102:0/6197868 learned_addr learned my addr 172.21.15.102:0/6197868 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:35:15.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7ad9d700 1 --2- 172.21.15.102:0/6197868 >> v2:172.21.15.172:3300/0 conn(0x7f9c7c153290 0x7f9c7c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:15.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7b59e700 1 -- 172.21.15.102:0/6197868 >> v2:172.21.15.172:3300/0 conn(0x7f9c7c153290 msgr2=0x7f9c7c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:35:15.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7b59e700 1 --2- 172.21.15.102:0/6197868 >> v2:172.21.15.172:3300/0 conn(0x7f9c7c153290 0x7f9c7c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7b59e700 1 -- 172.21.15.102:0/6197868 >> v1:172.21.15.172:6789/0 conn(0x7f9c7c1593c0 legacy=0x7f9c7c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:15.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.713+0000 7f9c7b59e700 1 -- 172.21.15.102:0/6197868 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c06c7f0 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.714+0000 7f9c7b59e700 1 --2- 172.21.15.102:0/6197868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0ce530 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c08cc70 tx=0x7f9c7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68e5e621f777f1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:15.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.714+0000 7f9c79d9b700 1 -- 172.21.15.102:0/6197868 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7400e070 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.714+0000 7f9c79d9b700 1 -- 172.21.15.102:0/6197868 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7400b910 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.714+0000 7f9c79d9b700 1 -- 172.21.15.102:0/6197868 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7400bcc0 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.714+0000 7f9c811ff700 1 -- 172.21.15.102:0/6197868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c0cd8e0 msgr2=0x7f9c7c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.714+0000 7f9c811ff700 1 --2- 172.21.15.102:0/6197868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0ce530 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c08cc70 tx=0x7f9c7400b330 comp rx=0 tx=0).stop 2023-03-03T11:35:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.715+0000 7f9c811ff700 1 -- 172.21.15.102:0/6197868 shutdown_connections 2023-03-03T11:35:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.715+0000 7f9c811ff700 1 --2- 172.21.15.102:0/6197868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0ce530 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.715+0000 7f9c811ff700 1 --2- 172.21.15.102:0/6197868 >> v2:172.21.15.172:3300/0 conn(0x7f9c7c153290 0x7f9c7c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.715+0000 7f9c811ff700 1 -- 172.21.15.102:0/6197868 >> 172.21.15.102:0/6197868 conn(0x7f9c7c065fb0 msgr2=0x7f9c7c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:35:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.715+0000 7f9c811ff700 1 -- 172.21.15.102:0/6197868 shutdown_connections 2023-03-03T11:35:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.715+0000 7f9c811ff700 1 -- 172.21.15.102:0/6197868 wait complete. 2023-03-03T11:35:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.718+0000 7f9c811ff700 1 Processor -- start 2023-03-03T11:35:15.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.718+0000 7f9c811ff700 1 -- start start 2023-03-03T11:35:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.719+0000 7f9c811ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.719+0000 7f9c811ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c153290 0x7f9c7c1501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.719+0000 7f9c811ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c7c1593c0 0x7f9c7c1f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.719+0000 7f9c811ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1567f0 con 0x7f9c7c153290 2023-03-03T11:35:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.719+0000 7f9c811ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c0cf920 con 0x7f9c7c1593c0 2023-03-03T11:35:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.719+0000 7f9c811ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c0cf3e0 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45702/0 (socket says 172.21.15.102:45702) 2023-03-03T11:35:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 -- 172.21.15.102:0/1006284591 learned_addr learned my addr 172.21.15.102:0/1006284591 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:35:15.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7b59e700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c7c1593c0 0x7f9c7c1f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7a59c700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c153290 0x7f9c7c1501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 -- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c7c1593c0 msgr2=0x7f9c7c1f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c7c1593c0 0x7f9c7c1f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 -- 172.21.15.102:0/1006284591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c153290 msgr2=0x7f9c7c1501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c153290 0x7f9c7c1501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 -- 172.21.15.102:0/1006284591 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c74009b30 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.720+0000 7f9c7ad9d700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0c87b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00e9b0 tx=0x7f9c6c00ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.721+0000 7f9c637fe700 1 -- 172.21.15.102:0/1006284591 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c009710 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.721+0000 7f9c637fe700 1 -- 172.21.15.102:0/1006284591 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c00ee50 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.721+0000 7f9c637fe700 1 -- 172.21.15.102:0/1006284591 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c00ca50 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.721+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7c0ccde0 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.721+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7c15d160 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.722+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7c0cd0c0 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.723+0000 7f9c637fe700 1 -- 172.21.15.102:0/1006284591 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c6c019030 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.724+0000 7f9c637fe700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c68068320 0x7f9c6806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.724+0000 7f9c637fe700 1 -- 172.21.15.102:0/1006284591 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9c6c085c20 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.725+0000 7f9c7a59c700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c68068320 0x7f9c6806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.725+0000 7f9c7a59c700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c68068320 0x7f9c6806a7d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c15a1c0 tx=0x7f9c70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.729+0000 7f9c637fe700 1 -- 172.21.15.102:0/1006284591 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c6c052330 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.963+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c7c04e6e0 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.965+0000 7f9c637fe700 1 -- 172.21.15.102:0/1006284591 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9c6c055950 con 0x7f9c7c0cd8e0 2023-03-03T11:35:15.967 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:35:15.968 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:35:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c68068320 msgr2=0x7f9c6806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c68068320 0x7f9c6806a7d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c15a1c0 tx=0x7f9c70009040 comp rx=0 tx=0).stop 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c7c0cd8e0 msgr2=0x7f9c7c0c87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0c87b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00e9b0 tx=0x7f9c6c00ecc0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 shutdown_connections 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c68068320 0x7f9c6806a7d0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c7c0cd8e0 0x7f9c7c0c87b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c7c1593c0 0x7f9c7c1f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 --2- 172.21.15.102:0/1006284591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c153290 0x7f9c7c1501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.967+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 >> 172.21.15.102:0/1006284591 conn(0x7f9c7c065fb0 msgr2=0x7f9c7c15ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:35:15.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.968+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 shutdown_connections 2023-03-03T11:35:15.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:35:15.968+0000 7f9c811ff700 1 -- 172.21.15.102:0/1006284591 wait complete. 2023-03-03T11:35:15.972 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:35:15.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 100 secs before thrashing 2023-03-03T11:35:38.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:38.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:08.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:08.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:38.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:38.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:55.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:36:56.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.256+0000 7fcd38dff700 1 Processor -- start 2023-03-03T11:36:56.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.256+0000 7fcd38dff700 1 -- start start 2023-03-03T11:36:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.257+0000 7fcd38dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcd34153290 0x7fcd34158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.257+0000 7fcd38dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd341593c0 0x7fcd340cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd38dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd34153660 con 0x7fcd340cd8e0 2023-03-03T11:36:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd38dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3406c360 con 0x7fcd34153290 2023-03-03T11:36:56.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd38dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3415cea0 con 0x7fcd341593c0 2023-03-03T11:36:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd3259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcd34153290 0x7fcd34158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd31d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd341593c0 0x7fcd340cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd32d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcd340cd8e0 0x7fcd340ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50774/0 (socket says 172.21.15.102:50774) 2023-03-03T11:36:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd32d9d700 1 -- 172.21.15.102:0/3226640525 learned_addr learned my addr 172.21.15.102:0/3226640525 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:36:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd3159a700 1 -- 172.21.15.102:0/3226640525 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782562280 0 0) 0x7fcd34153660 con 0x7fcd340cd8e0 2023-03-03T11:36:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd31d9b700 1 -- 172.21.15.102:0/3226640525 >> v2:172.21.15.172:3300/0 conn(0x7fcd34153290 msgr2=0x7fcd34158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:36:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd31d9b700 1 --2- 172.21.15.102:0/3226640525 >> v2:172.21.15.172:3300/0 conn(0x7fcd34153290 0x7fcd34158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd31d9b700 1 -- 172.21.15.102:0/3226640525 >> v1:172.21.15.102:6789/0 conn(0x7fcd340cd8e0 legacy=0x7fcd340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd31d9b700 1 -- 172.21.15.102:0/3226640525 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3406c7f0 con 0x7fcd341593c0 2023-03-03T11:36:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.258+0000 7fcd3259c700 1 --2- 172.21.15.102:0/3226640525 >> v2:172.21.15.172:3300/0 conn(0x7fcd34153290 0x7fcd34158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:36:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.259+0000 7fcd31d9b700 1 --2- 172.21.15.102:0/3226640525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd341593c0 0x7fcd340cde20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009840 tx=0x7fcd28009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c92cc6bc10b6eb7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.259+0000 7fcd3159a700 1 -- 172.21.15.102:0/3226640525 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28011830 con 0x7fcd341593c0 2023-03-03T11:36:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.259+0000 7fcd3159a700 1 -- 172.21.15.102:0/3226640525 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28011990 con 0x7fcd341593c0 2023-03-03T11:36:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.259+0000 7fcd3159a700 1 -- 172.21.15.102:0/3226640525 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28011d20 con 0x7fcd341593c0 2023-03-03T11:36:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 -- 172.21.15.102:0/3226640525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd341593c0 msgr2=0x7fcd340cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3226640525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd341593c0 0x7fcd340cde20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009840 tx=0x7fcd28009b50 comp rx=0 tx=0).stop 2023-03-03T11:36:56.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 -- 172.21.15.102:0/3226640525 shutdown_connections 2023-03-03T11:36:56.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3226640525 >> v2:172.21.15.172:3300/0 conn(0x7fcd34153290 0x7fcd34158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3226640525 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd341593c0 0x7fcd340cde20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 -- 172.21.15.102:0/3226640525 >> 172.21.15.102:0/3226640525 conn(0x7fcd34065fb0 msgr2=0x7fcd340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:56.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 -- 172.21.15.102:0/3226640525 shutdown_connections 2023-03-03T11:36:56.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.260+0000 7fcd38dff700 1 -- 172.21.15.102:0/3226640525 wait complete. 2023-03-03T11:36:56.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.263+0000 7fcd38dff700 1 Processor -- start 2023-03-03T11:36:56.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.263+0000 7fcd38dff700 1 -- start start 2023-03-03T11:36:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.264+0000 7fcd38dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd340cd8e0 0x7fcd3414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.264+0000 7fcd38dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34153290 0x7fcd340c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.264+0000 7fcd38dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd341593c0 0x7fcd341eee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.264+0000 7fcd38dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3414ea90 con 0x7fcd341593c0 2023-03-03T11:36:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.264+0000 7fcd38dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340c8bc0 con 0x7fcd34153290 2023-03-03T11:36:56.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.264+0000 7fcd38dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340c7be0 con 0x7fcd340cd8e0 2023-03-03T11:36:56.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd31d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34153290 0x7fcd340c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:56.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd3259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd340cd8e0 0x7fcd3414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:56.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd32d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd341593c0 0x7fcd341eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:56.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd31d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34153290 0x7fcd340c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53932/0 (socket says 172.21.15.102:53932) 2023-03-03T11:36:56.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd3259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd340cd8e0 0x7fcd3414e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55928/0 (socket says 172.21.15.102:55928) 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd31d9b700 1 -- 172.21.15.102:0/3077155738 learned_addr learned my addr 172.21.15.102:0/3077155738 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd32d9d700 1 -- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd340cd8e0 msgr2=0x7fcd3414e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd32d9d700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd340cd8e0 0x7fcd3414e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd32d9d700 1 -- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34153290 msgr2=0x7fcd340c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd32d9d700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34153290 0x7fcd340c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd32d9d700 1 -- 172.21.15.102:0/3077155738 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd240094a0 con 0x7fcd341593c0 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd31d9b700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34153290 0x7fcd340c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:36:56.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.265+0000 7fcd3259c700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd340cd8e0 0x7fcd3414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:36:56.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.266+0000 7fcd32d9d700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd341593c0 0x7fcd341eee30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00cad0 tx=0x7fcd2c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:56.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.266+0000 7fcd22ffd700 1 -- 172.21.15.102:0/3077155738 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c00a940 con 0x7fcd341593c0 2023-03-03T11:36:56.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.266+0000 7fcd22ffd700 1 -- 172.21.15.102:0/3077155738 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c00aaa0 con 0x7fcd341593c0 2023-03-03T11:36:56.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.266+0000 7fcd22ffd700 1 -- 172.21.15.102:0/3077155738 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c00ae50 con 0x7fcd341593c0 2023-03-03T11:36:56.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.266+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd280094a0 con 0x7fcd341593c0 2023-03-03T11:36:56.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.266+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd340ccde0 con 0x7fcd341593c0 2023-03-03T11:36:56.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.267+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3415d160 con 0x7fcd341593c0 2023-03-03T11:36:56.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.269+0000 7fcd22ffd700 1 -- 172.21.15.102:0/3077155738 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd2c010070 con 0x7fcd341593c0 2023-03-03T11:36:56.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.270+0000 7fcd22ffd700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd140682f0 0x7fcd1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:56.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.270+0000 7fcd22ffd700 1 -- 172.21.15.102:0/3077155738 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcd2c085680 con 0x7fcd341593c0 2023-03-03T11:36:56.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.270+0000 7fcd3259c700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd140682f0 0x7fcd1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:56.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.271+0000 7fcd3259c700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd140682f0 0x7fcd1406a7a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fcd24011380 tx=0x7fcd24009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:56.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.274+0000 7fcd22ffd700 1 -- 172.21.15.102:0/3077155738 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcd2c051ca0 con 0x7fcd341593c0 2023-03-03T11:36:56.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.506+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd340cd0c0 con 0x7fcd341593c0 2023-03-03T11:36:56.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.507+0000 7fcd22ffd700 1 -- 172.21.15.102:0/3077155738 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcd2c018070 con 0x7fcd341593c0 2023-03-03T11:36:56.509 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:36:56.509 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:36:56.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd140682f0 msgr2=0x7fcd1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:56.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd140682f0 0x7fcd1406a7a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fcd24011380 tx=0x7fcd24009210 comp rx=0 tx=0).stop 2023-03-03T11:36:56.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd341593c0 msgr2=0x7fcd341eee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd341593c0 0x7fcd341eee30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00cad0 tx=0x7fcd2c00cde0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 shutdown_connections 2023-03-03T11:36:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd140682f0 0x7fcd1406a7a0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd340cd8e0 0x7fcd3414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd341593c0 0x7fcd341eee30 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 --2- 172.21.15.102:0/3077155738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd34153290 0x7fcd340c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.509+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 >> 172.21.15.102:0/3077155738 conn(0x7fcd34065fb0 msgr2=0x7fcd340cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:56.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.510+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 shutdown_connections 2023-03-03T11:36:56.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:36:56.510+0000 7fcd38dff700 1 -- 172.21.15.102:0/3077155738 wait complete. 2023-03-03T11:36:56.513 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:36:56.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 41 secs before thrashing 2023-03-03T11:37:09.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:09.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:37.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:37:37.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.787+0000 7f5d9b1ff700 1 Processor -- start 2023-03-03T11:37:37.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.787+0000 7f5d9b1ff700 1 -- start start 2023-03-03T11:37:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.788+0000 7f5d9b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.788+0000 7f5d9b1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5d941593c0 0x7f5d940cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.788+0000 7f5d9b1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d94153660 con 0x7f5d940cd8e0 2023-03-03T11:37:37.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.788+0000 7f5d9b1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d9406c360 con 0x7f5d941593c0 2023-03-03T11:37:37.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.788+0000 7f5d9b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9415cea0 con 0x7f5d94153290 2023-03-03T11:37:37.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d93fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5d941593c0 0x7f5d940cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d98f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d9979c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5d940cd8e0 0x7f5d940ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45682/0 (socket says 172.21.15.102:45682) 2023-03-03T11:37:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d9979c700 1 -- 172.21.15.102:0/589758838 learned_addr learned my addr 172.21.15.102:0/589758838 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:37:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d93fff700 1 -- 172.21.15.102:0/589758838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 msgr2=0x7f5d94158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:37:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d93fff700 1 --2- 172.21.15.102:0/589758838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:37.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d93fff700 1 -- 172.21.15.102:0/589758838 >> v1:172.21.15.102:6789/0 conn(0x7f5d940cd8e0 legacy=0x7f5d940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d93fff700 1 -- 172.21.15.102:0/589758838 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9406c7f0 con 0x7f5d941593c0 2023-03-03T11:37:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d937fe700 1 -- 172.21.15.102:0/589758838 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140909393 0 0) 0x7f5d94153660 con 0x7f5d940cd8e0 2023-03-03T11:37:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.789+0000 7f5d98f9b700 1 --2- 172.21.15.102:0/589758838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:37:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.790+0000 7f5d93fff700 1 --2- 172.21.15.102:0/589758838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d941593c0 0x7f5d940cde20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009840 tx=0x7f5d80009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628ff9102a12dbf8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.790+0000 7f5d937fe700 1 -- 172.21.15.102:0/589758838 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80011830 con 0x7f5d941593c0 2023-03-03T11:37:37.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.790+0000 7f5d937fe700 1 -- 172.21.15.102:0/589758838 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d80011990 con 0x7f5d941593c0 2023-03-03T11:37:37.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.790+0000 7f5d937fe700 1 -- 172.21.15.102:0/589758838 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80011d20 con 0x7f5d941593c0 2023-03-03T11:37:37.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/589758838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d941593c0 msgr2=0x7f5d940cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/589758838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d941593c0 0x7f5d940cde20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009840 tx=0x7f5d80009b50 comp rx=0 tx=0).stop 2023-03-03T11:37:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/589758838 shutdown_connections 2023-03-03T11:37:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/589758838 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/589758838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d941593c0 0x7f5d940cde20 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/589758838 >> 172.21.15.102:0/589758838 conn(0x7f5d94065fb0 msgr2=0x7f5d940663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:37:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/589758838 shutdown_connections 2023-03-03T11:37:37.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.791+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/589758838 wait complete. 2023-03-03T11:37:37.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.793+0000 7f5d9b1ff700 1 Processor -- start 2023-03-03T11:37:37.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.793+0000 7f5d9b1ff700 1 -- start start 2023-03-03T11:37:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d9b1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d940cd8e0 0x7f5d940cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d9b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d9b1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d941593c0 0x7f5d9414fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d9b1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94153f50 con 0x7f5d940cd8e0 2023-03-03T11:37:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d9b1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d941501a0 con 0x7f5d941593c0 2023-03-03T11:37:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d9b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940b2740 con 0x7f5d94153290 2023-03-03T11:37:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d98f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d940cd8e0 0x7f5d940cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d98f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d940cd8e0 0x7f5d940cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58916/0 (socket says 172.21.15.102:58916) 2023-03-03T11:37:37.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d98f9b700 1 -- 172.21.15.102:0/2680714970 learned_addr learned my addr 172.21.15.102:0/2680714970 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.794+0000 7f5d93fff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d9979c700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d941593c0 0x7f5d9414fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d98f9b700 1 -- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 msgr2=0x7f5d94154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d98f9b700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d98f9b700 1 -- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d941593c0 msgr2=0x7f5d9414fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d98f9b700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d941593c0 0x7f5d9414fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d98f9b700 1 -- 172.21.15.102:0/2680714970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880094a0 con 0x7f5d940cd8e0 2023-03-03T11:37:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d98f9b700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d940cd8e0 0x7f5d940cad00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5d8800ebe0 tx=0x7f5d8800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.795+0000 7f5d917fa700 1 -- 172.21.15.102:0/2680714970 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8800c850 con 0x7f5d940cd8e0 2023-03-03T11:37:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.796+0000 7f5d917fa700 1 -- 172.21.15.102:0/2680714970 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8800c9b0 con 0x7f5d940cd8e0 2023-03-03T11:37:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.796+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d800094a0 con 0x7f5d940cd8e0 2023-03-03T11:37:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.796+0000 7f5d917fa700 1 -- 172.21.15.102:0/2680714970 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8800cd40 con 0x7f5d940cd8e0 2023-03-03T11:37:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.796+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d940ccde0 con 0x7f5d940cd8e0 2023-03-03T11:37:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.797+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d9415d160 con 0x7f5d940cd8e0 2023-03-03T11:37:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.798+0000 7f5d917fa700 1 -- 172.21.15.102:0/2680714970 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d88019030 con 0x7f5d940cd8e0 2023-03-03T11:37:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.799+0000 7f5d917fa700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d780682f0 0x7f5d7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:37.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.799+0000 7f5d917fa700 1 -- 172.21.15.102:0/2680714970 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5d88085d10 con 0x7f5d940cd8e0 2023-03-03T11:37:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.800+0000 7f5d93fff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d780682f0 0x7f5d7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.800+0000 7f5d93fff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d780682f0 0x7f5d7806a7a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5d800090b0 tx=0x7f5d80008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:37.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:37.803+0000 7f5d917fa700 1 -- 172.21.15.102:0/2680714970 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d8808b020 con 0x7f5d940cd8e0 2023-03-03T11:37:38.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.043+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d940cd0c0 con 0x7f5d940cd8e0 2023-03-03T11:37:38.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.043+0000 7f5d917fa700 1 -- 172.21.15.102:0/2680714970 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5d88052420 con 0x7f5d940cd8e0 2023-03-03T11:37:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:37:38.046 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:37:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d780682f0 msgr2=0x7f5d7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d780682f0 0x7f5d7806a7a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5d800090b0 tx=0x7f5d80008030 comp rx=0 tx=0).stop 2023-03-03T11:37:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d940cd8e0 msgr2=0x7f5d940cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d940cd8e0 0x7f5d940cad00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5d8800ebe0 tx=0x7f5d8800eef0 comp rx=0 tx=0).stop 2023-03-03T11:37:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 shutdown_connections 2023-03-03T11:37:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d941593c0 0x7f5d9414fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d940cd8e0 0x7f5d940cad00 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d780682f0 0x7f5d7806a7a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 --2- 172.21.15.102:0/2680714970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d94153290 0x7f5d94154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 >> 172.21.15.102:0/2680714970 conn(0x7f5d94065fb0 msgr2=0x7f5d941e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:37:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 shutdown_connections 2023-03-03T11:37:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:38.046+0000 7f5d9b1ff700 1 -- 172.21.15.102:0/2680714970 wait complete. 2023-03-03T11:37:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:37:38.076 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-03-03T11:37:39.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:39.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:57.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:37:57.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.340+0000 7f878f9ff700 1 Processor -- start 2023-03-03T11:37:57.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.340+0000 7f878f9ff700 1 -- start start 2023-03-03T11:37:57.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.341+0000 7f878f9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f8788158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:57.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.341+0000 7f878f9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f87881593c0 0x7f87880cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:57.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.341+0000 7f878f9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f87880cd8e0 0x7f87880ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:57.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.341+0000 7f878f9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8788153660 con 0x7f87880cd8e0 2023-03-03T11:37:57.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.341+0000 7f878f9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878806c360 con 0x7f87881593c0 2023-03-03T11:37:57.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.341+0000 7f878f9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878815cea0 con 0x7f8788153290 2023-03-03T11:37:57.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878df9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f87880cd8e0 0x7f87880ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:57.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878cf9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f87881593c0 0x7f87880cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:57.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878d79b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f8788158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:57.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878df9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f87880cd8e0 0x7f87880ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45878/0 (socket says 172.21.15.102:45878) 2023-03-03T11:37:57.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878cf9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f87881593c0 0x7f87880cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40184/0 (socket says 172.21.15.102:40184) 2023-03-03T11:37:57.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878df9c700 1 -- 172.21.15.102:0/1364358329 learned_addr learned my addr 172.21.15.102:0/1364358329 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:37:57.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878d79b700 1 -- 172.21.15.102:0/1364358329 >> v2:172.21.15.172:3300/0 conn(0x7f87881593c0 msgr2=0x7f87880cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:37:57.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878d79b700 1 --2- 172.21.15.102:0/1364358329 >> v2:172.21.15.172:3300/0 conn(0x7f87881593c0 0x7f87880cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878d79b700 1 -- 172.21.15.102:0/1364358329 >> v2:172.21.15.102:3300/0 conn(0x7f87880cd8e0 msgr2=0x7f87880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878d79b700 1 --2- 172.21.15.102:0/1364358329 >> v2:172.21.15.102:3300/0 conn(0x7f87880cd8e0 0x7f87880ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878d79b700 1 -- 172.21.15.102:0/1364358329 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878806c7f0 con 0x7f8788153290 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878cf9a700 1 --2- 172.21.15.102:0/1364358329 >> v2:172.21.15.172:3300/0 conn(0x7f87881593c0 0x7f87880cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.342+0000 7f878df9c700 1 --2- 172.21.15.102:0/1364358329 >> v2:172.21.15.102:3300/0 conn(0x7f87880cd8e0 0x7f87880ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.343+0000 7f878d79b700 1 --2- 172.21.15.102:0/1364358329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f8788158e80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87880b1970 tx=0x7f8784009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c76693ff217e400 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.343+0000 7f877ffff700 1 -- 172.21.15.102:0/1364358329 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784011830 con 0x7f8788153290 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.343+0000 7f877ffff700 1 -- 172.21.15.102:0/1364358329 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8784011990 con 0x7f8788153290 2023-03-03T11:37:57.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.343+0000 7f877ffff700 1 -- 172.21.15.102:0/1364358329 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784011d20 con 0x7f8788153290 2023-03-03T11:37:57.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 -- 172.21.15.102:0/1364358329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 msgr2=0x7f8788158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:57.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 --2- 172.21.15.102:0/1364358329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f8788158e80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87880b1970 tx=0x7f8784009b20 comp rx=0 tx=0).stop 2023-03-03T11:37:57.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 -- 172.21.15.102:0/1364358329 shutdown_connections 2023-03-03T11:37:57.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 --2- 172.21.15.102:0/1364358329 >> v2:172.21.15.102:3300/0 conn(0x7f87880cd8e0 0x7f87880ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 --2- 172.21.15.102:0/1364358329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f8788158e80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 --2- 172.21.15.102:0/1364358329 >> v2:172.21.15.172:3300/0 conn(0x7f87881593c0 0x7f87880cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 -- 172.21.15.102:0/1364358329 >> 172.21.15.102:0/1364358329 conn(0x7f8788065fb0 msgr2=0x7f87880663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:37:57.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 -- 172.21.15.102:0/1364358329 shutdown_connections 2023-03-03T11:37:57.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.344+0000 7f878f9ff700 1 -- 172.21.15.102:0/1364358329 wait complete. 2023-03-03T11:37:57.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.347+0000 7f878f9ff700 1 Processor -- start 2023-03-03T11:37:57.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.347+0000 7f878f9ff700 1 -- start start 2023-03-03T11:37:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.348+0000 7f878f9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87880cd8e0 0x7f87881f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.348+0000 7f878f9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f87880cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.348+0000 7f878f9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87881593c0 0x7f87880c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.348+0000 7f878f9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87880b2e90 con 0x7f87880cd8e0 2023-03-03T11:37:57.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.348+0000 7f878f9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87880b3170 con 0x7f87881593c0 2023-03-03T11:37:57.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.348+0000 7f878f9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87880b3450 con 0x7f8788153290 2023-03-03T11:37:57.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87880cd8e0 0x7f87881f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:57.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878cf9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f87880cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:57.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87880cd8e0 0x7f87881f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45892/0 (socket says 172.21.15.102:45892) 2023-03-03T11:37:57.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 -- 172.21.15.102:0/3139328195 learned_addr learned my addr 172.21.15.102:0/3139328195 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:37:57.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878df9c700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87881593c0 0x7f87880c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:57.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 -- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 msgr2=0x7f87880cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f87880cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 -- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87881593c0 msgr2=0x7f87880c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87881593c0 0x7f87880c6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 -- 172.21.15.102:0/3139328195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780009b30 con 0x7f87880cd8e0 2023-03-03T11:37:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878cf9a700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f87880cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:37:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.349+0000 7f878d79b700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87880cd8e0 0x7f87881f2660 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8784000c00 tx=0x7f8784004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:57.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.350+0000 7f877dffb700 1 -- 172.21.15.102:0/3139328195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784004510 con 0x7f87880cd8e0 2023-03-03T11:37:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.350+0000 7f877dffb700 1 -- 172.21.15.102:0/3139328195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878401e070 con 0x7f87880cd8e0 2023-03-03T11:37:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.350+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87780094a0 con 0x7f87880cd8e0 2023-03-03T11:37:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.350+0000 7f877dffb700 1 -- 172.21.15.102:0/3139328195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87840038e0 con 0x7f87880cd8e0 2023-03-03T11:37:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.350+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87840094a0 con 0x7f87880cd8e0 2023-03-03T11:37:57.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.351+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87880ccde0 con 0x7f87880cd8e0 2023-03-03T11:37:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.352+0000 7f877dffb700 1 -- 172.21.15.102:0/3139328195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f878400f050 con 0x7f87880cd8e0 2023-03-03T11:37:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.353+0000 7f877dffb700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f876c0682f0 0x7f876c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.354+0000 7f877dffb700 1 -- 172.21.15.102:0/3139328195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8784086a70 con 0x7f87880cd8e0 2023-03-03T11:37:57.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.354+0000 7f878cf9a700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f876c0682f0 0x7f876c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:57.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.354+0000 7f878cf9a700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f876c0682f0 0x7f876c06a7a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f87881584f0 tx=0x7f8778009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.358+0000 7f877dffb700 1 -- 172.21.15.102:0/3139328195 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8784053100 con 0x7f87880cd8e0 2023-03-03T11:37:57.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.590+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87880cd0c0 con 0x7f87880cd8e0 2023-03-03T11:37:57.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.590+0000 7f877dffb700 1 -- 172.21.15.102:0/3139328195 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8784020090 con 0x7f87880cd8e0 2023-03-03T11:37:57.592 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:37:57.593 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:37:57.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f876c0682f0 msgr2=0x7f876c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:57.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f876c0682f0 0x7f876c06a7a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f87881584f0 tx=0x7f8778009210 comp rx=0 tx=0).stop 2023-03-03T11:37:57.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87880cd8e0 msgr2=0x7f87881f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:57.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87880cd8e0 0x7f87881f2660 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8784000c00 tx=0x7f8784004750 comp rx=0 tx=0).stop 2023-03-03T11:37:57.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 shutdown_connections 2023-03-03T11:37:57.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87881593c0 0x7f87880c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87880cd8e0 0x7f87881f2660 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f876c0682f0 0x7f876c06a7a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 --2- 172.21.15.102:0/3139328195 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8788153290 0x7f87880cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 >> 172.21.15.102:0/3139328195 conn(0x7f8788065fb0 msgr2=0x7f8788157790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:37:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 shutdown_connections 2023-03-03T11:37:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:37:57.593+0000 7f878f9ff700 1 -- 172.21.15.102:0/3139328195 wait complete. 2023-03-03T11:37:57.598 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:37:57.621 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-03-03T11:38:09.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:09.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:26.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:38:26.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.867+0000 7f2c1cfff700 1 Processor -- start 2023-03-03T11:38:26.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.867+0000 7f2c1cfff700 1 -- start start 2023-03-03T11:38:26.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1cfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c181531c0 0x7f2c1815c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:26.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1cfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c1815c890 0x7f2c180c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:26.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180c86f0 0x7f2c180c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:26.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1cfff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c180723d0 con 0x7f2c181531c0 2023-03-03T11:38:26.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1cfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c18153590 con 0x7f2c1815c890 2023-03-03T11:38:26.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18072860 con 0x7f2c180c86f0 2023-03-03T11:38:26.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c181531c0 0x7f2c1815c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:26.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c15d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c1815c890 0x7f2c180c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:26.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c181531c0 0x7f2c1815c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59244/0 (socket says 172.21.15.102:59244) 2023-03-03T11:38:26.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c1659c700 1 -- 172.21.15.102:0/1724944591 learned_addr learned my addr 172.21.15.102:0/1724944591 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.869+0000 7f2c16d9d700 1 --2- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180c86f0 0x7f2c180c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c15d9b700 1 -- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180c86f0 msgr2=0x7f2c180c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c15d9b700 1 --2- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180c86f0 0x7f2c180c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c15d9b700 1 -- 172.21.15.102:0/1724944591 >> v2:172.21.15.102:3300/0 conn(0x7f2c181531c0 msgr2=0x7f2c1815c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c15d9b700 1 --2- 172.21.15.102:0/1724944591 >> v2:172.21.15.102:3300/0 conn(0x7f2c181531c0 0x7f2c1815c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c15d9b700 1 -- 172.21.15.102:0/1724944591 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180c8c50 con 0x7f2c1815c890 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c1659c700 1 --2- 172.21.15.102:0/1724944591 >> v2:172.21.15.102:3300/0 conn(0x7f2c181531c0 0x7f2c1815c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:38:26.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c15d9b700 1 --2- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815c890 0x7f2c180c81b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2c180c3170 tx=0x7f2c0c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64700bfa5e861388 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:26.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c1559a700 1 -- 172.21.15.102:0/1724944591 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c011830 con 0x7f2c1815c890 2023-03-03T11:38:26.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c1559a700 1 -- 172.21.15.102:0/1724944591 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c011990 con 0x7f2c1815c890 2023-03-03T11:38:26.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.870+0000 7f2c1559a700 1 -- 172.21.15.102:0/1724944591 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c011d20 con 0x7f2c1815c890 2023-03-03T11:38:26.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 -- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815c890 msgr2=0x7f2c180c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:26.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815c890 0x7f2c180c81b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2c180c3170 tx=0x7f2c0c009b20 comp rx=0 tx=0).stop 2023-03-03T11:38:26.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 -- 172.21.15.102:0/1724944591 shutdown_connections 2023-03-03T11:38:26.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c180c86f0 0x7f2c180c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:26.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/1724944591 >> v2:172.21.15.102:3300/0 conn(0x7f2c181531c0 0x7f2c1815c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:26.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/1724944591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c1815c890 0x7f2c180c81b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:26.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 -- 172.21.15.102:0/1724944591 >> 172.21.15.102:0/1724944591 conn(0x7f2c1806b410 msgr2=0x7f2c1806b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:38:26.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 -- 172.21.15.102:0/1724944591 shutdown_connections 2023-03-03T11:38:26.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.871+0000 7f2c1cfff700 1 -- 172.21.15.102:0/1724944591 wait complete. 2023-03-03T11:38:26.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.873+0000 7f2c1cfff700 1 Processor -- start 2023-03-03T11:38:26.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.873+0000 7f2c1cfff700 1 -- start start 2023-03-03T11:38:26.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.874+0000 7f2c1cfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c180c86f0 0x7f2c180c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:26.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.874+0000 7f2c1cfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c181531c0 0x7f2c180cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:26.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815c890 0x7f2c180cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:26.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1cfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c181e4890 con 0x7f2c180c86f0 2023-03-03T11:38:26.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1cfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180c41f0 con 0x7f2c181531c0 2023-03-03T11:38:26.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18159f40 con 0x7f2c1815c890 2023-03-03T11:38:26.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c180c86f0 0x7f2c180c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:26.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c180c86f0 0x7f2c180c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59250/0 (socket says 172.21.15.102:59250) 2023-03-03T11:38:26.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 -- 172.21.15.102:0/4165955430 learned_addr learned my addr 172.21.15.102:0/4165955430 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:38:26.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c15d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c181531c0 0x7f2c180cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:26.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c16d9d700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815c890 0x7f2c180cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:26.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 -- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815c890 msgr2=0x7f2c180cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:26.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815c890 0x7f2c180cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:26.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 -- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c181531c0 msgr2=0x7f2c180cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:26.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c181531c0 0x7f2c180cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:26.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.875+0000 7f2c1659c700 1 -- 172.21.15.102:0/4165955430 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08009550 con 0x7f2c180c86f0 2023-03-03T11:38:26.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.876+0000 7f2c1659c700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c180c86f0 0x7f2c180c5490 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800ec10 tx=0x7f2c0800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:26.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.876+0000 7f2c06ffd700 1 -- 172.21.15.102:0/4165955430 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0800c850 con 0x7f2c180c86f0 2023-03-03T11:38:26.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.876+0000 7f2c06ffd700 1 -- 172.21.15.102:0/4165955430 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0800c9b0 con 0x7f2c180c86f0 2023-03-03T11:38:26.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.876+0000 7f2c06ffd700 1 -- 172.21.15.102:0/4165955430 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0800cd40 con 0x7f2c180c86f0 2023-03-03T11:38:26.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.876+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c0094a0 con 0x7f2c180c86f0 2023-03-03T11:38:26.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.876+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18071aa0 con 0x7f2c180c86f0 2023-03-03T11:38:26.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.877+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c180c8f30 con 0x7f2c180c86f0 2023-03-03T11:38:26.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.878+0000 7f2c06ffd700 1 -- 172.21.15.102:0/4165955430 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c08023070 con 0x7f2c180c86f0 2023-03-03T11:38:26.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.879+0000 7f2c06ffd700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf8068320 0x7f2bf806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:26.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.880+0000 7f2c06ffd700 1 -- 172.21.15.102:0/4165955430 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2c08086140 con 0x7f2c180c86f0 2023-03-03T11:38:26.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.880+0000 7f2c15d9b700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf8068320 0x7f2bf806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:26.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.881+0000 7f2c15d9b700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf8068320 0x7f2bf806a7d0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2c180c3170 tx=0x7f2c0c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:26.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:26.884+0000 7f2c06ffd700 1 -- 172.21.15.102:0/4165955430 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c080527c0 con 0x7f2c180c86f0 2023-03-03T11:38:27.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.117+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c180c5f50 con 0x7f2c180c86f0 2023-03-03T11:38:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.117+0000 7f2c06ffd700 1 -- 172.21.15.102:0/4165955430 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2c0800cea0 con 0x7f2c180c86f0 2023-03-03T11:38:27.120 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:38:27.120 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:38:27.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf8068320 msgr2=0x7f2bf806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:27.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf8068320 0x7f2bf806a7d0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2c180c3170 tx=0x7f2c0c01c040 comp rx=0 tx=0).stop 2023-03-03T11:38:27.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c180c86f0 msgr2=0x7f2c180c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:27.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c180c86f0 0x7f2c180c5490 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800ec10 tx=0x7f2c0800ef20 comp rx=0 tx=0).stop 2023-03-03T11:38:27.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 shutdown_connections 2023-03-03T11:38:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2bf8068320 0x7f2bf806a7d0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c1815c890 0x7f2c180cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c180c86f0 0x7f2c180c5490 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 --2- 172.21.15.102:0/4165955430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c181531c0 0x7f2c180cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 >> 172.21.15.102:0/4165955430 conn(0x7f2c1806b410 msgr2=0x7f2c180c6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:38:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 shutdown_connections 2023-03-03T11:38:27.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:38:27.120+0000 7f2c1cfff700 1 -- 172.21.15.102:0/4165955430 wait complete. 2023-03-03T11:38:27.125 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:38:27.150 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 100 secs before thrashing 2023-03-03T11:38:39.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:39.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:09.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:09.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:40.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:40.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T11:39:59.998+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:40:07.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:40:07.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.405+0000 7f90d53ff700 1 Processor -- start 2023-03-03T11:40:07.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.405+0000 7f90d53ff700 1 -- start start 2023-03-03T11:40:07.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.406+0000 7f90d53ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:07.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90d53ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d0153660 con 0x7f90d00cd390 2023-03-03T11:40:07.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90d53ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d006c360 con 0x7f90d0153290 2023-03-03T11:40:07.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90d53ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d015cea0 con 0x7f90d00cd8e0 2023-03-03T11:40:07.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90ce7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:07.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90ce7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d00cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58230/0 (socket says 172.21.15.102:58230) 2023-03-03T11:40:07.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90ce7fc700 1 -- 172.21.15.102:0/866930162 learned_addr learned my addr 172.21.15.102:0/866930162 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:40:07.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90ceffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f90d0153290 0x7f90d0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59798/0 (socket says 172.21.15.102:59798) 2023-03-03T11:40:07.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90ce7fc700 1 -- 172.21.15.102:0/866930162 >> v1:172.21.15.172:6789/0 conn(0x7f90d0153290 legacy=0x7f90d0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:07.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90ce7fc700 1 -- 172.21.15.102:0/866930162 >> v1:172.21.15.102:6789/0 conn(0x7f90d00cd390 legacy=0x7f90d00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:07.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.407+0000 7f90ce7fc700 1 -- 172.21.15.102:0/866930162 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d006c7f0 con 0x7f90d00cd8e0 2023-03-03T11:40:07.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.408+0000 7f90ce7fc700 1 --2- 172.21.15.102:0/866930162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d00cde20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009920 tx=0x7f90c4009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef7ac8740e151550 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:40:07.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.408+0000 7f90cdffb700 1 -- 172.21.15.102:0/866930162 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c400ba60 con 0x7f90d00cd8e0 2023-03-03T11:40:07.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.408+0000 7f90cdffb700 1 -- 172.21.15.102:0/866930162 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c400bbc0 con 0x7f90d00cd8e0 2023-03-03T11:40:07.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.408+0000 7f90cdffb700 1 -- 172.21.15.102:0/866930162 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4009e70 con 0x7f90d00cd8e0 2023-03-03T11:40:07.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.408+0000 7f90d53ff700 1 -- 172.21.15.102:0/866930162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 msgr2=0x7f90d00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:07.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.408+0000 7f90d53ff700 1 --2- 172.21.15.102:0/866930162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d00cde20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009920 tx=0x7f90c4009c30 comp rx=0 tx=0).stop 2023-03-03T11:40:07.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.409+0000 7f90d53ff700 1 -- 172.21.15.102:0/866930162 shutdown_connections 2023-03-03T11:40:07.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.409+0000 7f90d53ff700 1 --2- 172.21.15.102:0/866930162 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d00cde20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:07.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.409+0000 7f90d53ff700 1 -- 172.21.15.102:0/866930162 >> 172.21.15.102:0/866930162 conn(0x7f90d0065fb0 msgr2=0x7f90d00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:40:07.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.409+0000 7f90d53ff700 1 -- 172.21.15.102:0/866930162 shutdown_connections 2023-03-03T11:40:07.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.409+0000 7f90d53ff700 1 -- 172.21.15.102:0/866930162 wait complete. 2023-03-03T11:40:07.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.411+0000 7f90d53ff700 1 Processor -- start 2023-03-03T11:40:07.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.411+0000 7f90d53ff700 1 -- start start 2023-03-03T11:40:07.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90d53ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90d00cd390 0x7f90d014fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:07.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90d53ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d0150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:07.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90d53ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d0153290 0x7f90d00b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:07.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90d53ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00b1fa0 con 0x7f90d0153290 2023-03-03T11:40:07.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90d53ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d01ef270 con 0x7f90d00cd390 2023-03-03T11:40:07.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90d53ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d01ef550 con 0x7f90d00cd8e0 2023-03-03T11:40:07.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90ceffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90d00cd390 0x7f90d014fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:07.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90cf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d0153290 0x7f90d00b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:07.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90ce7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d0150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:07.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90ceffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90d00cd390 0x7f90d014fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43884/0 (socket says 172.21.15.102:43884) 2023-03-03T11:40:07.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.412+0000 7f90ceffd700 1 -- 172.21.15.102:0/2243095037 learned_addr learned my addr 172.21.15.102:0/2243095037 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:40:07.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90ceffd700 1 -- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 msgr2=0x7f90d0150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:07.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90ceffd700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d0150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:07.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90ceffd700 1 -- 172.21.15.102:0/2243095037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d0153290 msgr2=0x7f90d00b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:07.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90ceffd700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d0153290 0x7f90d00b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:07.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90ceffd700 1 -- 172.21.15.102:0/2243095037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c4009550 con 0x7f90d00cd390 2023-03-03T11:40:07.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90ceffd700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90d00cd390 0x7f90d014fd40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c730 tx=0x7f90c000ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:40:07.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90b77fe700 1 -- 172.21.15.102:0/2243095037 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0005cb0 con 0x7f90d00cd390 2023-03-03T11:40:07.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90b77fe700 1 -- 172.21.15.102:0/2243095037 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c0005e10 con 0x7f90d00cd390 2023-03-03T11:40:07.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d00ccde0 con 0x7f90d00cd390 2023-03-03T11:40:07.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.413+0000 7f90b77fe700 1 -- 172.21.15.102:0/2243095037 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c00072d0 con 0x7f90d00cd390 2023-03-03T11:40:07.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.414+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d015d160 con 0x7f90d00cd390 2023-03-03T11:40:07.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.414+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d00cd0c0 con 0x7f90d00cd390 2023-03-03T11:40:07.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.416+0000 7f90b77fe700 1 -- 172.21.15.102:0/2243095037 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90c0019030 con 0x7f90d00cd390 2023-03-03T11:40:07.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.417+0000 7f90b77fe700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90bc0682f0 0x7f90bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:07.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.417+0000 7f90b77fe700 1 -- 172.21.15.102:0/2243095037 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f90c0084fa0 con 0x7f90d00cd390 2023-03-03T11:40:07.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.417+0000 7f90ce7fc700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90bc0682f0 0x7f90bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:07.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.418+0000 7f90ce7fc700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90bc0682f0 0x7f90bc06a7a0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f90c4000c00 tx=0x7f90c4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:40:07.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.421+0000 7f90b77fe700 1 -- 172.21.15.102:0/2243095037 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90c004b5e0 con 0x7f90d00cd390 2023-03-03T11:40:07.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.658+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90d004e6e0 con 0x7f90d00cd390 2023-03-03T11:40:07.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.659+0000 7f90b77fe700 1 -- 172.21.15.102:0/2243095037 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f90c0017020 con 0x7f90d00cd390 2023-03-03T11:40:07.661 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:40:07.661 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:40:07.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.661+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90bc0682f0 msgr2=0x7f90bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:07.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.661+0000 7f90d53ff700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90bc0682f0 0x7f90bc06a7a0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f90c4000c00 tx=0x7f90c4008030 comp rx=0 tx=0).stop 2023-03-03T11:40:07.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.661+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90d00cd390 msgr2=0x7f90d014fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:07.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.661+0000 7f90d53ff700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90d00cd390 0x7f90d014fd40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c730 tx=0x7f90c000ca40 comp rx=0 tx=0).stop 2023-03-03T11:40:07.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 shutdown_connections 2023-03-03T11:40:07.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d0153290 0x7f90d00b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:07.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90d00cd390 0x7f90d014fd40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:07.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90bc0682f0 0x7f90bc06a7a0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:07.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 --2- 172.21.15.102:0/2243095037 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90d00cd8e0 0x7f90d0150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:07.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 >> 172.21.15.102:0/2243095037 conn(0x7f90d0065fb0 msgr2=0x7f90d00cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:40:07.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 shutdown_connections 2023-03-03T11:40:07.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:07.662+0000 7f90d53ff700 1 -- 172.21.15.102:0/2243095037 wait complete. 2023-03-03T11:40:07.665 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:40:07.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T11:40:10.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:10.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:40.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:40.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:42.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:40:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.944+0000 7fcb4179d700 1 Processor -- start 2023-03-03T11:40:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.944+0000 7fcb4179d700 1 -- start start 2023-03-03T11:40:42.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb4179d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcb3c1564b0 0x7fcb3c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:42.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb4179d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:42.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb4179d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3c0c9ca0 con 0x7fcb3c0ca440 2023-03-03T11:40:42.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb4179d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3c0b4920 con 0x7fcb3c1564b0 2023-03-03T11:40:42.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb4179d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0b4c00 con 0x7fcb3c0c9610 2023-03-03T11:40:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb3bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcb3c1564b0 0x7fcb3c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb3b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb40f9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcb3c0ca440 0x7fcb3c0ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33102/0 (socket says 172.21.15.102:33102) 2023-03-03T11:40:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.946+0000 7fcb40f9c700 1 -- 172.21.15.102:0/1823200074 learned_addr learned my addr 172.21.15.102:0/1823200074 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:40:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3bfff700 1 -- 172.21.15.102:0/1823200074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 msgr2=0x7fcb3c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:40:42.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3bfff700 1 --2- 172.21.15.102:0/1823200074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3bfff700 1 -- 172.21.15.102:0/1823200074 >> v1:172.21.15.172:6789/0 conn(0x7fcb3c0ca440 legacy=0x7fcb3c0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3bfff700 1 -- 172.21.15.102:0/1823200074 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c0b46f0 con 0x7fcb3c1564b0 2023-03-03T11:40:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3affd700 1 -- 172.21.15.102:0/1823200074 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034675698 0 0) 0x7fcb3c0c9ca0 con 0x7fcb3c0ca440 2023-03-03T11:40:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3bfff700 1 --2- 172.21.15.102:0/1823200074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 0x7fcb3c0cb660 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009840 tx=0x7fcb2c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9825c0414c4d7f0b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:40:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3affd700 1 -- 172.21.15.102:0/1823200074 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c011830 con 0x7fcb3c1564b0 2023-03-03T11:40:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3affd700 1 -- 172.21.15.102:0/1823200074 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c011990 con 0x7fcb3c1564b0 2023-03-03T11:40:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.947+0000 7fcb3affd700 1 -- 172.21.15.102:0/1823200074 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c011d20 con 0x7fcb3c1564b0 2023-03-03T11:40:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 -- 172.21.15.102:0/1823200074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 msgr2=0x7fcb3c0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 --2- 172.21.15.102:0/1823200074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 0x7fcb3c0cb660 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009840 tx=0x7fcb2c009b50 comp rx=0 tx=0).stop 2023-03-03T11:40:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 -- 172.21.15.102:0/1823200074 shutdown_connections 2023-03-03T11:40:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 --2- 172.21.15.102:0/1823200074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 0x7fcb3c0cb660 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 --2- 172.21.15.102:0/1823200074 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 -- 172.21.15.102:0/1823200074 >> 172.21.15.102:0/1823200074 conn(0x7fcb3c065fb0 msgr2=0x7fcb3c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:40:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 -- 172.21.15.102:0/1823200074 shutdown_connections 2023-03-03T11:40:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.948+0000 7fcb4179d700 1 -- 172.21.15.102:0/1823200074 wait complete. 2023-03-03T11:40:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.951+0000 7fcb4179d700 1 Processor -- start 2023-03-03T11:40:42.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.951+0000 7fcb4179d700 1 -- start start 2023-03-03T11:40:42.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.952+0000 7fcb4179d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:42.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.952+0000 7fcb4179d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb3c0ca440 0x7fcb3c158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.952+0000 7fcb4179d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 0x7fcb3c1efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.952+0000 7fcb4179d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c150e30 con 0x7fcb3c0ca440 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.952+0000 7fcb4179d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c1ef6b0 con 0x7fcb3c1564b0 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.952+0000 7fcb4179d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c1594f0 con 0x7fcb3c0c9610 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:32844/0 (socket says 172.21.15.102:32844) 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 -- 172.21.15.102:0/14385124 learned_addr learned my addr 172.21.15.102:0/14385124 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:40:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb40f9c700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 0x7fcb3c1efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:42.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3b7fe700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb3c0ca440 0x7fcb3c158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:42.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 -- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 msgr2=0x7fcb3c1efb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:42.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 0x7fcb3c1efb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:42.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 -- 172.21.15.102:0/14385124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb3c0ca440 msgr2=0x7fcb3c158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:42.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb3c0ca440 0x7fcb3c158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:42.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 -- 172.21.15.102:0/14385124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c0094a0 con 0x7fcb3c0c9610 2023-03-03T11:40:42.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.953+0000 7fcb3bfff700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0ce1c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009fd0 tx=0x7fcb2c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:40:42.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.954+0000 7fcb38ff9700 1 -- 172.21.15.102:0/14385124 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c0036a0 con 0x7fcb3c0c9610 2023-03-03T11:40:42.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.954+0000 7fcb38ff9700 1 -- 172.21.15.102:0/14385124 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c01e070 con 0x7fcb3c0c9610 2023-03-03T11:40:42.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.954+0000 7fcb38ff9700 1 -- 172.21.15.102:0/14385124 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c003a40 con 0x7fcb3c0c9610 2023-03-03T11:40:42.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.954+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c0b2210 con 0x7fcb3c0c9610 2023-03-03T11:40:42.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.954+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c0b1f30 con 0x7fcb3c0c9610 2023-03-03T11:40:42.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.955+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3c156880 con 0x7fcb3c0c9610 2023-03-03T11:40:42.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.956+0000 7fcb38ff9700 1 -- 172.21.15.102:0/14385124 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb2c00f050 con 0x7fcb3c0c9610 2023-03-03T11:40:42.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.957+0000 7fcb38ff9700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcb280682f0 0x7fcb2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:40:42.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.957+0000 7fcb38ff9700 1 -- 172.21.15.102:0/14385124 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcb2c086df0 con 0x7fcb3c0c9610 2023-03-03T11:40:42.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.957+0000 7fcb3b7fe700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcb280682f0 0x7fcb2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:40:42.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.958+0000 7fcb3b7fe700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcb280682f0 0x7fcb2806a7a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c1514c0 tx=0x7fcb30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:40:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:42.961+0000 7fcb38ff9700 1 -- 172.21.15.102:0/14385124 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb2c053470 con 0x7fcb3c0c9610 2023-03-03T11:40:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.194+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb3c1ef990 con 0x7fcb3c0c9610 2023-03-03T11:40:43.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.196+0000 7fcb38ff9700 1 -- 172.21.15.102:0/14385124 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcb2c015020 con 0x7fcb3c0c9610 2023-03-03T11:40:43.198 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:40:43.198 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:40:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcb280682f0 msgr2=0x7fcb2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcb280682f0 0x7fcb2806a7a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c1514c0 tx=0x7fcb30008040 comp rx=0 tx=0).stop 2023-03-03T11:40:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 msgr2=0x7fcb3c0ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:40:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0ce1c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009fd0 tx=0x7fcb2c004750 comp rx=0 tx=0).stop 2023-03-03T11:40:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 shutdown_connections 2023-03-03T11:40:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcb280682f0 0x7fcb2806a7a0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcb3c0c9610 0x7fcb3c0ce1c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb3c1564b0 0x7fcb3c1efb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 --2- 172.21.15.102:0/14385124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb3c0ca440 0x7fcb3c158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:40:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 >> 172.21.15.102:0/14385124 conn(0x7fcb3c065fb0 msgr2=0x7fcb3c0ccd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:40:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 shutdown_connections 2023-03-03T11:40:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:40:43.198+0000 7fcb4179d700 1 -- 172.21.15.102:0/14385124 wait complete. 2023-03-03T11:40:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:40:43.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2023-03-03T11:41:10.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:10.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:13.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:41:13.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.473+0000 7f7f8e79d700 1 Processor -- start 2023-03-03T11:41:13.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.473+0000 7f7f8e79d700 1 -- start start 2023-03-03T11:41:13.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8e79d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f880ced00 0x7f7f880c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:13.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880b3970 0x7f7f880c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:13.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8e79d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f880c3d20 con 0x7f7f880cf0d0 2023-03-03T11:41:13.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8e79d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f880c4930 con 0x7f7f880ced00 2023-03-03T11:41:13.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8815cba0 con 0x7f7f880b3970 2023-03-03T11:41:13.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8d79b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f880ced00 0x7f7f880c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:13.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8df9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7f880cf0d0 0x7f7f88158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38126/0 (socket says 172.21.15.102:38126) 2023-03-03T11:41:13.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8df9c700 1 -- 172.21.15.102:0/1935462376 learned_addr learned my addr 172.21.15.102:0/1935462376 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:41:13.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.475+0000 7f7f8cf9a700 1 --2- 172.21.15.102:0/1935462376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880b3970 0x7f7f880c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:13.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f7ffff700 1 -- 172.21.15.102:0/1935462376 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905713461 0 0) 0x7f7f880c3d20 con 0x7f7f880cf0d0 2023-03-03T11:41:13.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f7ffff700 1 -- 172.21.15.102:0/1935462376 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f740035b0 con 0x7f7f880cf0d0 2023-03-03T11:41:13.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f8d79b700 1 -- 172.21.15.102:0/1935462376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880b3970 msgr2=0x7f7f880c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:41:13.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f8d79b700 1 --2- 172.21.15.102:0/1935462376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880b3970 0x7f7f880c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f8d79b700 1 -- 172.21.15.102:0/1935462376 >> v1:172.21.15.172:6789/0 conn(0x7f7f880cf0d0 legacy=0x7f7f88158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:13.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f8d79b700 1 -- 172.21.15.102:0/1935462376 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8815ce80 con 0x7f7f880ced00 2023-03-03T11:41:13.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f8d79b700 1 --2- 172.21.15.102:0/1935462376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880ced00 0x7f7f880c5a90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009840 tx=0x7f7f84009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73b5474d36111f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:13.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f7ffff700 1 -- 172.21.15.102:0/1935462376 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f84011830 con 0x7f7f880ced00 2023-03-03T11:41:13.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f7ffff700 1 -- 172.21.15.102:0/1935462376 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f84011990 con 0x7f7f880ced00 2023-03-03T11:41:13.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.476+0000 7f7f7ffff700 1 -- 172.21.15.102:0/1935462376 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f84011d20 con 0x7f7f880ced00 2023-03-03T11:41:13.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 -- 172.21.15.102:0/1935462376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880ced00 msgr2=0x7f7f880c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:13.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/1935462376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880ced00 0x7f7f880c5a90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009840 tx=0x7f7f84009b50 comp rx=0 tx=0).stop 2023-03-03T11:41:13.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 -- 172.21.15.102:0/1935462376 shutdown_connections 2023-03-03T11:41:13.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/1935462376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880ced00 0x7f7f880c5a90 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/1935462376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880b3970 0x7f7f880c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 -- 172.21.15.102:0/1935462376 >> 172.21.15.102:0/1935462376 conn(0x7f7f8808b1c0 msgr2=0x7f7f88071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:13.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 -- 172.21.15.102:0/1935462376 shutdown_connections 2023-03-03T11:41:13.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.477+0000 7f7f8e79d700 1 -- 172.21.15.102:0/1935462376 wait complete. 2023-03-03T11:41:13.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.479+0000 7f7f8e79d700 1 Processor -- start 2023-03-03T11:41:13.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.479+0000 7f7f8e79d700 1 -- start start 2023-03-03T11:41:13.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.480+0000 7f7f8e79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f880b3970 0x7f7f8815a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:13.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.480+0000 7f7f8e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880ced00 0x7f7f881f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:13.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.480+0000 7f7f8e79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880cf0d0 0x7f7f881f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:13.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.480+0000 7f7f8e79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f881515d0 con 0x7f7f880cf0d0 2023-03-03T11:41:13.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.480+0000 7f7f8e79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8815b410 con 0x7f7f880b3970 2023-03-03T11:41:13.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.480+0000 7f7f8e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8814f3f0 con 0x7f7f880ced00 2023-03-03T11:41:13.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f880b3970 0x7f7f8815a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:13.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8cf9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880ced00 0x7f7f881f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8df9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880cf0d0 0x7f7f881f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f880b3970 0x7f7f8815a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37040/0 (socket says 172.21.15.102:37040) 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 -- 172.21.15.102:0/3970784803 learned_addr learned my addr 172.21.15.102:0/3970784803 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 -- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880ced00 msgr2=0x7f7f881f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880ced00 0x7f7f881f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 -- 172.21.15.102:0/3970784803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880cf0d0 msgr2=0x7f7f881f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880cf0d0 0x7f7f881f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 -- 172.21.15.102:0/3970784803 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840094a0 con 0x7f7f880b3970 2023-03-03T11:41:13.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.481+0000 7f7f8d79b700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f880b3970 0x7f7f8815a540 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7f840090b0 tx=0x7f7f84004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:13.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.482+0000 7f7f7dffb700 1 -- 172.21.15.102:0/3970784803 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f840036a0 con 0x7f7f880b3970 2023-03-03T11:41:13.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.482+0000 7f7f7dffb700 1 -- 172.21.15.102:0/3970784803 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8401f070 con 0x7f7f880b3970 2023-03-03T11:41:13.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.482+0000 7f7f7dffb700 1 -- 172.21.15.102:0/3970784803 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f84003a40 con 0x7f7f880b3970 2023-03-03T11:41:13.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.482+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8806c200 con 0x7f7f880b3970 2023-03-03T11:41:13.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.482+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8806bf20 con 0x7f7f880b3970 2023-03-03T11:41:13.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.483+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f880cf540 con 0x7f7f880b3970 2023-03-03T11:41:13.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.484+0000 7f7f7dffb700 1 -- 172.21.15.102:0/3970784803 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f84048070 con 0x7f7f880b3970 2023-03-03T11:41:13.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.485+0000 7f7f7dffb700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f6c0682f0 0x7f7f6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:13.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.486+0000 7f7f7dffb700 1 -- 172.21.15.102:0/3970784803 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7f84021070 con 0x7f7f880b3970 2023-03-03T11:41:13.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.486+0000 7f7f8cf9a700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f6c0682f0 0x7f7f6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:13.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.486+0000 7f7f8cf9a700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f6c0682f0 0x7f7f6c06a7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7f88067fb0 tx=0x7f7f78009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.488+0000 7f7f7dffb700 1 -- 172.21.15.102:0/3970784803 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f84054840 con 0x7f7f880b3970 2023-03-03T11:41:13.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.721+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f8804e6e0 con 0x7f7f880b3970 2023-03-03T11:41:13.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.722+0000 7f7f7dffb700 1 -- 172.21.15.102:0/3970784803 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7f8402b090 con 0x7f7f880b3970 2023-03-03T11:41:13.724 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:41:13.724 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:41:13.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f6c0682f0 msgr2=0x7f7f6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:13.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f6c0682f0 0x7f7f6c06a7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7f88067fb0 tx=0x7f7f78009210 comp rx=0 tx=0).stop 2023-03-03T11:41:13.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f880b3970 msgr2=0x7f7f8815a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f880b3970 0x7f7f8815a540 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7f840090b0 tx=0x7f7f84004750 comp rx=0 tx=0).stop 2023-03-03T11:41:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 shutdown_connections 2023-03-03T11:41:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f880cf0d0 0x7f7f881f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f880b3970 0x7f7f8815a540 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7f6c0682f0 0x7f7f6c06a7a0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 --2- 172.21.15.102:0/3970784803 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7f880ced00 0x7f7f881f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.724+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 >> 172.21.15.102:0/3970784803 conn(0x7f7f8808b1c0 msgr2=0x7f7f880ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:13.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.725+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 shutdown_connections 2023-03-03T11:41:13.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:13.725+0000 7f7f8e79d700 1 -- 172.21.15.102:0/3970784803 wait complete. 2023-03-03T11:41:13.728 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:41:13.755 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 20 secs before thrashing 2023-03-03T11:41:33.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:41:34.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.007+0000 7f4405bff700 1 Processor -- start 2023-03-03T11:41:34.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.007+0000 7f4405bff700 1 -- start start 2023-03-03T11:41:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f4405bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4400152e50 0x7f44000c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f4405bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f44000c9f60 0x7f44000cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f4405bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000c9590 con 0x7f44000c8eb0 2023-03-03T11:41:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f4405bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44000ca3d0 con 0x7f44000c9f60 2023-03-03T11:41:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f4405bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4400159880 con 0x7f4400152e50 2023-03-03T11:41:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f43f7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f44000c8eb0 0x7f44000ccc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49538/0 (socket says 172.21.15.102:49538) 2023-03-03T11:41:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f43f7fff700 1 -- 172.21.15.102:0/3966242465 learned_addr learned my addr 172.21.15.102:0/3966242465 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:41:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f43ff7fe700 1 --2- 172.21.15.102:0/3966242465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4400152e50 0x7f44000c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.009+0000 7f43fffff700 1 --2- 172.21.15.102:0/3966242465 >> v2:172.21.15.172:3300/0 conn(0x7f44000c9f60 0x7f44000cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43ff7fe700 1 -- 172.21.15.102:0/3966242465 >> v2:172.21.15.172:3300/0 conn(0x7f44000c9f60 msgr2=0x7f44000cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:41:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43ff7fe700 1 --2- 172.21.15.102:0/3966242465 >> v2:172.21.15.172:3300/0 conn(0x7f44000c9f60 0x7f44000cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43feffd700 1 -- 172.21.15.102:0/3966242465 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966539160 0 0) 0x7f44000c9590 con 0x7f44000c8eb0 2023-03-03T11:41:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43ff7fe700 1 -- 172.21.15.102:0/3966242465 >> v1:172.21.15.102:6789/0 conn(0x7f44000c8eb0 legacy=0x7f44000ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43ff7fe700 1 -- 172.21.15.102:0/3966242465 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000c7ae0 con 0x7f4400152e50 2023-03-03T11:41:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43ff7fe700 1 --2- 172.21.15.102:0/3966242465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4400152e50 0x7f44000c9a20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009840 tx=0x7f43f0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c020a1da3c44ac94 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43feffd700 1 -- 172.21.15.102:0/3966242465 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0011830 con 0x7f4400152e50 2023-03-03T11:41:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43feffd700 1 -- 172.21.15.102:0/3966242465 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f0011990 con 0x7f4400152e50 2023-03-03T11:41:34.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.010+0000 7f43feffd700 1 -- 172.21.15.102:0/3966242465 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0011d20 con 0x7f4400152e50 2023-03-03T11:41:34.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 -- 172.21.15.102:0/3966242465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4400152e50 msgr2=0x7f44000c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 --2- 172.21.15.102:0/3966242465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4400152e50 0x7f44000c9a20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009840 tx=0x7f43f0009b50 comp rx=0 tx=0).stop 2023-03-03T11:41:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 -- 172.21.15.102:0/3966242465 shutdown_connections 2023-03-03T11:41:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 --2- 172.21.15.102:0/3966242465 >> v2:172.21.15.172:3300/0 conn(0x7f44000c9f60 0x7f44000cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 --2- 172.21.15.102:0/3966242465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4400152e50 0x7f44000c9a20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 -- 172.21.15.102:0/3966242465 >> 172.21.15.102:0/3966242465 conn(0x7f44000704c0 msgr2=0x7f44000708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 -- 172.21.15.102:0/3966242465 shutdown_connections 2023-03-03T11:41:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.011+0000 7f4405bff700 1 -- 172.21.15.102:0/3966242465 wait complete. 2023-03-03T11:41:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.013+0000 7f4405bff700 1 Processor -- start 2023-03-03T11:41:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.013+0000 7f4405bff700 1 -- start start 2023-03-03T11:41:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f4405bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44000c8eb0 0x7f4400157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f4405bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44000c9f60 0x7f44001ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f4405bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4400152e50 0x7f44001efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f4405bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400158850 con 0x7f44000c9f60 2023-03-03T11:41:34.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f4405bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44001ef930 con 0x7f4400152e50 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f4405bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44001e54b0 con 0x7f44000c8eb0 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44000c8eb0 0x7f4400157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f43fffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4400152e50 0x7f44001efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44000c8eb0 0x7f4400157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59256/0 (socket says 172.21.15.102:59256) 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.014+0000 7f43ff7fe700 1 -- 172.21.15.102:0/1342403252 learned_addr learned my addr 172.21.15.102:0/1342403252 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43f7fff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44000c9f60 0x7f44001ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fffff700 1 -- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44000c8eb0 msgr2=0x7f4400157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:34.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fffff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44000c8eb0 0x7f4400157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fffff700 1 -- 172.21.15.102:0/1342403252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44000c9f60 msgr2=0x7f44001ef3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:34.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fffff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44000c9f60 0x7f44001ef3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fffff700 1 -- 172.21.15.102:0/1342403252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f00094a0 con 0x7f4400152e50 2023-03-03T11:41:34.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43ff7fe700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44000c8eb0 0x7f4400157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:41:34.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fffff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4400152e50 0x7f44001efde0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f43f800ef90 tx=0x7f43f800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fcff9700 1 -- 172.21.15.102:0/1342403252 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f800cce0 con 0x7f4400152e50 2023-03-03T11:41:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fcff9700 1 -- 172.21.15.102:0/1342403252 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f800f040 con 0x7f4400152e50 2023-03-03T11:41:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.015+0000 7f43fcff9700 1 -- 172.21.15.102:0/1342403252 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f80146c0 con 0x7f4400152e50 2023-03-03T11:41:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.016+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44000c80a0 con 0x7f4400152e50 2023-03-03T11:41:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.016+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44000c7dc0 con 0x7f4400152e50 2023-03-03T11:41:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.017+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400159b60 con 0x7f4400152e50 2023-03-03T11:41:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.018+0000 7f43fcff9700 1 -- 172.21.15.102:0/1342403252 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f43f8024070 con 0x7f4400152e50 2023-03-03T11:41:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.019+0000 7f43fcff9700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43e00682f0 0x7f43e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:34.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.019+0000 7f43fcff9700 1 -- 172.21.15.102:0/1342403252 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f43f8086910 con 0x7f4400152e50 2023-03-03T11:41:34.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.020+0000 7f43ff7fe700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43e00682f0 0x7f43e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:34.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.020+0000 7f43ff7fe700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43e00682f0 0x7f43e006a7a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f43f0000c00 tx=0x7f43f001d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.024+0000 7f43fcff9700 1 -- 172.21.15.102:0/1342403252 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f43f8052f90 con 0x7f4400152e50 2023-03-03T11:41:34.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.253+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44000673a0 con 0x7f4400152e50 2023-03-03T11:41:34.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.254+0000 7f43fcff9700 1 -- 172.21.15.102:0/1342403252 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f43f8018090 con 0x7f4400152e50 2023-03-03T11:41:34.257 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:41:34.257 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:41:34.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43e00682f0 msgr2=0x7f43e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:34.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43e00682f0 0x7f43e006a7a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f43f0000c00 tx=0x7f43f001d040 comp rx=0 tx=0).stop 2023-03-03T11:41:34.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4400152e50 msgr2=0x7f44001efde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:34.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4400152e50 0x7f44001efde0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f43f800ef90 tx=0x7f43f800c3a0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 shutdown_connections 2023-03-03T11:41:34.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43e00682f0 0x7f43e006a7a0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44000c8eb0 0x7f4400157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4400152e50 0x7f44001efde0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 --2- 172.21.15.102:0/1342403252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44000c9f60 0x7f44001ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:34.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 >> 172.21.15.102:0/1342403252 conn(0x7f44000704c0 msgr2=0x7f440014d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:34.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 shutdown_connections 2023-03-03T11:41:34.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:41:34.257+0000 7f4405bff700 1 -- 172.21.15.102:0/1342403252 wait complete. 2023-03-03T11:41:34.262 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:41:34.287 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T11:41:40.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:40.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:11.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:11.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:41.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:41.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:01.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:43:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.551+0000 7f19857ff700 1 Processor -- start 2023-03-03T11:43:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.551+0000 7f19857ff700 1 -- start start 2023-03-03T11:43:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f19857ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1980153260 0x7f19800ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f19857ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f19800cac20 0x7f1980163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f19857ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800cf940 con 0x7f19800c9cb0 2023-03-03T11:43:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f19857ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19800cfe70 con 0x7f1980153260 2023-03-03T11:43:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f19857ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1980158d70 con 0x7f19800cac20 2023-03-03T11:43:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f197effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1980153260 0x7f19800ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f197effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1980153260 0x7f19800ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40158/0 (socket says 172.21.15.102:40158) 2023-03-03T11:43:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f197effd700 1 -- 172.21.15.102:0/1510331797 learned_addr learned my addr 172.21.15.102:0/1510331797 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:43:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.553+0000 7f197f7fe700 1 --2- 172.21.15.102:0/1510331797 >> v2:172.21.15.172:3300/0 conn(0x7f19800cac20 0x7f1980163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197effd700 1 -- 172.21.15.102:0/1510331797 >> v2:172.21.15.172:3300/0 conn(0x7f19800cac20 msgr2=0x7f1980163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:43:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197effd700 1 --2- 172.21.15.102:0/1510331797 >> v2:172.21.15.172:3300/0 conn(0x7f19800cac20 0x7f1980163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197effd700 1 -- 172.21.15.102:0/1510331797 >> v1:172.21.15.172:6789/0 conn(0x7f19800c9cb0 legacy=0x7f19800cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197effd700 1 -- 172.21.15.102:0/1510331797 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19800cfb90 con 0x7f1980153260 2023-03-03T11:43:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197effd700 1 --2- 172.21.15.102:0/1510331797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153260 0x7f19800ca6e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f198008cdd0 tx=0x7f1970009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa622c349d2d860 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197dffb700 1 -- 172.21.15.102:0/1510331797 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970011830 con 0x7f1980153260 2023-03-03T11:43:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197dffb700 1 -- 172.21.15.102:0/1510331797 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970011990 con 0x7f1980153260 2023-03-03T11:43:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.554+0000 7f197dffb700 1 -- 172.21.15.102:0/1510331797 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970011d20 con 0x7f1980153260 2023-03-03T11:43:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 -- 172.21.15.102:0/1510331797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153260 msgr2=0x7f19800ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 --2- 172.21.15.102:0/1510331797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153260 0x7f19800ca6e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f198008cdd0 tx=0x7f1970009bd0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 -- 172.21.15.102:0/1510331797 shutdown_connections 2023-03-03T11:43:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 --2- 172.21.15.102:0/1510331797 >> v2:172.21.15.172:3300/0 conn(0x7f19800cac20 0x7f1980163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 --2- 172.21.15.102:0/1510331797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153260 0x7f19800ca6e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 -- 172.21.15.102:0/1510331797 >> 172.21.15.102:0/1510331797 conn(0x7f1980066c00 msgr2=0x7f19800ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 -- 172.21.15.102:0/1510331797 shutdown_connections 2023-03-03T11:43:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.555+0000 7f19857ff700 1 -- 172.21.15.102:0/1510331797 wait complete. 2023-03-03T11:43:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.558+0000 7f19857ff700 1 Processor -- start 2023-03-03T11:43:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.558+0000 7f19857ff700 1 -- start start 2023-03-03T11:43:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f19857ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19800cac20 0x7f19801f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f19857ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1980153260 0x7f1980151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f19857ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801ee990 0x7f19800cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f19857ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1980150c90 con 0x7f19800cac20 2023-03-03T11:43:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f19857ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19801f2410 con 0x7f1980153260 2023-03-03T11:43:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f19857ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19801f2de0 con 0x7f19801ee990 2023-03-03T11:43:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f197effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19800cac20 0x7f19801f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f197effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19800cac20 0x7f19801f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40168/0 (socket says 172.21.15.102:40168) 2023-03-03T11:43:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f197f7fe700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801ee990 0x7f19800cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f197e7fc700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1980153260 0x7f1980151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.560+0000 7f197effd700 1 -- 172.21.15.102:0/3778516144 learned_addr learned my addr 172.21.15.102:0/3778516144 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:43:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197effd700 1 -- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801ee990 msgr2=0x7f19800cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197effd700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801ee990 0x7f19800cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197effd700 1 -- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1980153260 msgr2=0x7f1980151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:01.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197effd700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1980153260 0x7f1980151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197effd700 1 -- 172.21.15.102:0/3778516144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970009550 con 0x7f19800cac20 2023-03-03T11:43:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197f7fe700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801ee990 0x7f19800cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:43:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197e7fc700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1980153260 0x7f1980151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:43:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f197effd700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19800cac20 0x7f19801f28a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1970000c00 tx=0x7f1970004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f19677fe700 1 -- 172.21.15.102:0/3778516144 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970004400 con 0x7f19800cac20 2023-03-03T11:43:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f19677fe700 1 -- 172.21.15.102:0/3778516144 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197001f070 con 0x7f19800cac20 2023-03-03T11:43:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f19677fe700 1 -- 172.21.15.102:0/3778516144 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19700038c0 con 0x7f19800cac20 2023-03-03T11:43:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.561+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1980159260 con 0x7f19800cac20 2023-03-03T11:43:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.562+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1980158f80 con 0x7f19800cac20 2023-03-03T11:43:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.563+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1980159540 con 0x7f19800cac20 2023-03-03T11:43:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.564+0000 7f19677fe700 1 -- 172.21.15.102:0/3778516144 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f197002a020 con 0x7f19800cac20 2023-03-03T11:43:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.565+0000 7f19677fe700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.565+0000 7f19677fe700 1 -- 172.21.15.102:0/3778516144 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f19700872d0 con 0x7f19800cac20 2023-03-03T11:43:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.565+0000 7f197e7fc700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.566+0000 7f197e7fc700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f19800b17f0 tx=0x7f1974009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.569+0000 7f19677fe700 1 -- 172.21.15.102:0/3778516144 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f19700539e0 con 0x7f19800cac20 2023-03-03T11:43:01.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.804+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f198004e6e0 con 0x7f19800cac20 2023-03-03T11:43:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.805+0000 7f19677fe700 1 -- 172.21.15.102:0/3778516144 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1970057000 con 0x7f19800cac20 2023-03-03T11:43:01.808 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:43:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:43:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 msgr2=0x7f196c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f19800b17f0 tx=0x7f1974009040 comp rx=0 tx=0).stop 2023-03-03T11:43:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19800cac20 msgr2=0x7f19801f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19800cac20 0x7f19801f28a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1970000c00 tx=0x7f1970004750 comp rx=0 tx=0).stop 2023-03-03T11:43:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 shutdown_connections 2023-03-03T11:43:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801ee990 0x7f19800cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19800cac20 0x7f19801f28a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 --2- 172.21.15.102:0/3778516144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1980153260 0x7f1980151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 >> 172.21.15.102:0/3778516144 conn(0x7f19800c9cb0 msgr2=0x7f1980158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 shutdown_connections 2023-03-03T11:43:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:01.808+0000 7f19857ff700 1 -- 172.21.15.102:0/3778516144 wait complete. 2023-03-03T11:43:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:43:01.838 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T11:43:11.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:11.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:41.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:41.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:57.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:43:58.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.102+0000 7ff7c27ff700 1 Processor -- start 2023-03-03T11:43:58.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.102+0000 7ff7c27ff700 1 -- start start 2023-03-03T11:43:58.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.103+0000 7ff7c27ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc153290 0x7ff7bc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.103+0000 7ff7c27ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7bc1593c0 0x7ff7bc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7c27ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc153660 con 0x7ff7bc0cd8e0 2023-03-03T11:43:58.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7c27ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7bc06c360 con 0x7ff7bc1593c0 2023-03-03T11:43:58.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7c27ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc15cea0 con 0x7ff7bc153290 2023-03-03T11:43:58.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7b3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7bc1593c0 0x7ff7bc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7c0d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff7bc0cd8e0 0x7ff7bc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48848/0 (socket says 172.21.15.102:48848) 2023-03-03T11:43:58.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7c0d9c700 1 -- 172.21.15.102:0/3939302105 learned_addr learned my addr 172.21.15.102:0/3939302105 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:43:58.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7bbfff700 1 --2- 172.21.15.102:0/3939302105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc153290 0x7ff7bc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7b3fff700 1 -- 172.21.15.102:0/3939302105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc153290 msgr2=0x7ff7bc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:43:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7b3fff700 1 --2- 172.21.15.102:0/3939302105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc153290 0x7ff7bc158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7b3fff700 1 -- 172.21.15.102:0/3939302105 >> v1:172.21.15.102:6789/0 conn(0x7ff7bc0cd8e0 legacy=0x7ff7bc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.104+0000 7ff7b3fff700 1 -- 172.21.15.102:0/3939302105 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc06c7f0 con 0x7ff7bc1593c0 2023-03-03T11:43:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.105+0000 7ff7b3fff700 1 --2- 172.21.15.102:0/3939302105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc1593c0 0x7ff7bc0cde20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009840 tx=0x7ff7a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a8e3b27759eae9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.105+0000 7ff7bb7fe700 1 -- 172.21.15.102:0/3939302105 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4011830 con 0x7ff7bc1593c0 2023-03-03T11:43:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.105+0000 7ff7bb7fe700 1 -- 172.21.15.102:0/3939302105 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a4011990 con 0x7ff7bc1593c0 2023-03-03T11:43:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.105+0000 7ff7bb7fe700 1 -- 172.21.15.102:0/3939302105 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4011d20 con 0x7ff7bc1593c0 2023-03-03T11:43:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 -- 172.21.15.102:0/3939302105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc1593c0 msgr2=0x7ff7bc0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/3939302105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc1593c0 0x7ff7bc0cde20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009840 tx=0x7ff7a4009b50 comp rx=0 tx=0).stop 2023-03-03T11:43:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 -- 172.21.15.102:0/3939302105 shutdown_connections 2023-03-03T11:43:58.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/3939302105 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc153290 0x7ff7bc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/3939302105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc1593c0 0x7ff7bc0cde20 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 -- 172.21.15.102:0/3939302105 >> 172.21.15.102:0/3939302105 conn(0x7ff7bc065fb0 msgr2=0x7ff7bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:58.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 -- 172.21.15.102:0/3939302105 shutdown_connections 2023-03-03T11:43:58.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.106+0000 7ff7c27ff700 1 -- 172.21.15.102:0/3939302105 wait complete. 2023-03-03T11:43:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.108+0000 7ff7c27ff700 1 Processor -- start 2023-03-03T11:43:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.108+0000 7ff7c27ff700 1 -- start start 2023-03-03T11:43:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.109+0000 7ff7c27ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7bc0cd8e0 0x7ff7bc0b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.109+0000 7ff7c27ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc153290 0x7ff7bc0b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.109+0000 7ff7c27ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc1593c0 0x7ff7bc0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.109+0000 7ff7c27ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc1ee500 con 0x7ff7bc153290 2023-03-03T11:43:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.109+0000 7ff7c27ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc0b4750 con 0x7ff7bc0cd8e0 2023-03-03T11:43:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.109+0000 7ff7c27ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc0b4a10 con 0x7ff7bc1593c0 2023-03-03T11:43:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7b3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc153290 0x7ff7bc0b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7bc0cd8e0 0x7ff7bc0b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7bc0cd8e0 0x7ff7bc0b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:41266/0 (socket says 172.21.15.102:41266) 2023-03-03T11:43:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7bbfff700 1 -- 172.21.15.102:0/2110676558 learned_addr learned my addr 172.21.15.102:0/2110676558 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:43:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7c0d9c700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc1593c0 0x7ff7bc0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7b3fff700 1 -- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc1593c0 msgr2=0x7ff7bc0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7b3fff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc1593c0 0x7ff7bc0b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7b3fff700 1 -- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7bc0cd8e0 msgr2=0x7ff7bc0b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7b3fff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7bc0cd8e0 0x7ff7bc0b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7b3fff700 1 -- 172.21.15.102:0/2110676558 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40094a0 con 0x7ff7bc153290 2023-03-03T11:43:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7bbfff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7bc0cd8e0 0x7ff7bc0b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:43:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.110+0000 7ff7b3fff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc153290 0x7ff7bc0b1c70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4016000 tx=0x7ff7a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.111+0000 7ff7b97fa700 1 -- 172.21.15.102:0/2110676558 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a40036a0 con 0x7ff7bc153290 2023-03-03T11:43:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.111+0000 7ff7b97fa700 1 -- 172.21.15.102:0/2110676558 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a401f070 con 0x7ff7bc153290 2023-03-03T11:43:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.111+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc0ccde0 con 0x7ff7bc153290 2023-03-03T11:43:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.111+0000 7ff7b97fa700 1 -- 172.21.15.102:0/2110676558 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4003a40 con 0x7ff7bc153290 2023-03-03T11:43:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.111+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc15d160 con 0x7ff7bc153290 2023-03-03T11:43:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.112+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc0cd0c0 con 0x7ff7bc153290 2023-03-03T11:43:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.113+0000 7ff7b97fa700 1 -- 172.21.15.102:0/2110676558 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff7a400f050 con 0x7ff7bc153290 2023-03-03T11:43:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.114+0000 7ff7b97fa700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff79c068320 0x7ff79c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.115+0000 7ff7b97fa700 1 -- 172.21.15.102:0/2110676558 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff7a4021070 con 0x7ff7bc153290 2023-03-03T11:43:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.115+0000 7ff7bbfff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff79c068320 0x7ff79c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.116+0000 7ff7bbfff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff79c068320 0x7ff79c06a7d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0113d0 tx=0x7ff7ac009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:58.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.119+0000 7ff7b97fa700 1 -- 172.21.15.102:0/2110676558 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7a4053ca0 con 0x7ff7bc153290 2023-03-03T11:43:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.354+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7bc04e6e0 con 0x7ff7bc153290 2023-03-03T11:43:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.354+0000 7ff7b97fa700 1 -- 172.21.15.102:0/2110676558 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff7a40572c0 con 0x7ff7bc153290 2023-03-03T11:43:58.357 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:43:58.357 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:43:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff79c068320 msgr2=0x7ff79c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff79c068320 0x7ff79c06a7d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0113d0 tx=0x7ff7ac009210 comp rx=0 tx=0).stop 2023-03-03T11:43:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc153290 msgr2=0x7ff7bc0b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc153290 0x7ff7bc0b1c70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4016000 tx=0x7ff7a4004750 comp rx=0 tx=0).stop 2023-03-03T11:43:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 shutdown_connections 2023-03-03T11:43:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff79c068320 0x7ff79c06a7d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7bc1593c0 0x7ff7bc0b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7bc0cd8e0 0x7ff7bc0b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 --2- 172.21.15.102:0/2110676558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7bc153290 0x7ff7bc0b1c70 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 >> 172.21.15.102:0/2110676558 conn(0x7ff7bc065fb0 msgr2=0x7ff7bc15c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 shutdown_connections 2023-03-03T11:43:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.357+0000 7ff7c27ff700 1 -- 172.21.15.102:0/2110676558 wait complete. 2023-03-03T11:43:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:43:58.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:43:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.627+0000 7fe2085ff700 1 Processor -- start 2023-03-03T11:43:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.627+0000 7fe2085ff700 1 -- start start 2023-03-03T11:43:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.629+0000 7fe2085ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe200153290 0x7fe200158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.629+0000 7fe2085ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe200153660 con 0x7fe2000cd8e0 2023-03-03T11:43:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.629+0000 7fe2085ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20006c360 con 0x7fe2001593c0 2023-03-03T11:43:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.629+0000 7fe2085ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2000cec40 con 0x7fe200153290 2023-03-03T11:43:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.629+0000 7fe20639b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe200153290 0x7fe200158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.629+0000 7fe206b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe2000cd8e0 0x7fe2000ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48864/0 (socket says 172.21.15.102:48864) 2023-03-03T11:43:58.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.629+0000 7fe206b9c700 1 -- 172.21.15.102:0/1911711664 learned_addr learned my addr 172.21.15.102:0/1911711664 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:43:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.630+0000 7fe20639b700 1 -- 172.21.15.102:0/1911711664 >> v1:172.21.15.172:6789/0 conn(0x7fe2001593c0 legacy=0x7fe2000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.630+0000 7fe205399700 1 -- 172.21.15.102:0/1911711664 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021063681 0 0) 0x7fe200153660 con 0x7fe2000cd8e0 2023-03-03T11:43:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.630+0000 7fe20639b700 1 -- 172.21.15.102:0/1911711664 >> v1:172.21.15.102:6789/0 conn(0x7fe2000cd8e0 legacy=0x7fe2000ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.630+0000 7fe20639b700 1 -- 172.21.15.102:0/1911711664 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20015cea0 con 0x7fe200153290 2023-03-03T11:43:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.630+0000 7fe20639b700 1 --2- 172.21.15.102:0/1911711664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe200158e80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009840 tx=0x7fe1fc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd4d3c513e34fa05 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.630+0000 7fe205399700 1 -- 172.21.15.102:0/1911711664 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc011830 con 0x7fe200153290 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe205399700 1 -- 172.21.15.102:0/1911711664 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc011990 con 0x7fe200153290 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe205399700 1 -- 172.21.15.102:0/1911711664 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc011d20 con 0x7fe200153290 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe2085ff700 1 -- 172.21.15.102:0/1911711664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 msgr2=0x7fe200158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1911711664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe200158e80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009840 tx=0x7fe1fc009b50 comp rx=0 tx=0).stop 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe2085ff700 1 -- 172.21.15.102:0/1911711664 shutdown_connections 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1911711664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe200158e80 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe2085ff700 1 -- 172.21.15.102:0/1911711664 >> 172.21.15.102:0/1911711664 conn(0x7fe200065fb0 msgr2=0x7fe2000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe2085ff700 1 -- 172.21.15.102:0/1911711664 shutdown_connections 2023-03-03T11:43:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.631+0000 7fe2085ff700 1 -- 172.21.15.102:0/1911711664 wait complete. 2023-03-03T11:43:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.633+0000 7fe2085ff700 1 Processor -- start 2023-03-03T11:43:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.633+0000 7fe2085ff700 1 -- start start 2023-03-03T11:43:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2000cd8e0 0x7fe20014f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe2001f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2001593c0 0x7fe2000cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20014f800 con 0x7fe2000cd8e0 2023-03-03T11:43:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2001ef810 con 0x7fe200153290 2023-03-03T11:43:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2001f2ca0 con 0x7fe2001593c0 2023-03-03T11:43:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe20639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2000cd8e0 0x7fe20014f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe205b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe2001f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe20639b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2000cd8e0 0x7fe20014f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59986/0 (socket says 172.21.15.102:59986) 2023-03-03T11:43:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe20639b700 1 -- 172.21.15.102:0/1141357173 learned_addr learned my addr 172.21.15.102:0/1141357173 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:43:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe206b9c700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2001593c0 0x7fe2000cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe205b9a700 1 -- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2001593c0 msgr2=0x7fe2000cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe205b9a700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2001593c0 0x7fe2000cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe205b9a700 1 -- 172.21.15.102:0/1141357173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2000cd8e0 msgr2=0x7fe20014f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe205b9a700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2000cd8e0 0x7fe20014f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe205b9a700 1 -- 172.21.15.102:0/1141357173 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc0094a0 con 0x7fe200153290 2023-03-03T11:43:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe205b9a700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe2001f2760 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0000f60 tx=0x7fe1f000c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe1f6ffd700 1 -- 172.21.15.102:0/1141357173 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00052d0 con 0x7fe200153290 2023-03-03T11:43:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe1f6ffd700 1 -- 172.21.15.102:0/1141357173 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f0005430 con 0x7fe200153290 2023-03-03T11:43:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20015d160 con 0x7fe200153290 2023-03-03T11:43:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe1f6ffd700 1 -- 172.21.15.102:0/1141357173 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00038c0 con 0x7fe200153290 2023-03-03T11:43:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.636+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20006c7f0 con 0x7fe200153290 2023-03-03T11:43:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.637+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2000ccde0 con 0x7fe200153290 2023-03-03T11:43:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.639+0000 7fe1f6ffd700 1 -- 172.21.15.102:0/1141357173 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe1f0019030 con 0x7fe200153290 2023-03-03T11:43:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.640+0000 7fe1f6ffd700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1e4068320 0x7fe1e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.640+0000 7fe1f6ffd700 1 -- 172.21.15.102:0/1141357173 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe1f0084fa0 con 0x7fe200153290 2023-03-03T11:43:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.640+0000 7fe20639b700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1e4068320 0x7fe1e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.641+0000 7fe20639b700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1e4068320 0x7fe1e406a7d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009b90 tx=0x7fe1fc01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.645+0000 7fe1f6ffd700 1 -- 172.21.15.102:0/1141357173 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1f00516b0 con 0x7fe200153290 2023-03-03T11:43:58.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.880+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2000c9a80 con 0x7fe200153290 2023-03-03T11:43:58.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.882+0000 7fe1f6ffd700 1 -- 172.21.15.102:0/1141357173 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe1f0017090 con 0x7fe200153290 2023-03-03T11:43:58.884 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:43:58.884 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:43:58.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1e4068320 msgr2=0x7fe1e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1e4068320 0x7fe1e406a7d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009b90 tx=0x7fe1fc01c040 comp rx=0 tx=0).stop 2023-03-03T11:43:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 msgr2=0x7fe2001f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe2001f2760 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0000f60 tx=0x7fe1f000c730 comp rx=0 tx=0).stop 2023-03-03T11:43:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 shutdown_connections 2023-03-03T11:43:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1e4068320 0x7fe1e406a7d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2001593c0 0x7fe2000cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2000cd8e0 0x7fe20014f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 --2- 172.21.15.102:0/1141357173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe200153290 0x7fe2001f2760 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 >> 172.21.15.102:0/1141357173 conn(0x7fe200065fb0 msgr2=0x7fe20015c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 shutdown_connections 2023-03-03T11:43:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:43:58.884+0000 7fe2085ff700 1 -- 172.21.15.102:0/1141357173 wait complete. 2023-03-03T11:43:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:43:58.913 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-03-03T11:44:11.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:11.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:27.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:44:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.159+0000 7f64417ff700 1 Processor -- start 2023-03-03T11:44:28.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.160+0000 7f64417ff700 1 -- start start 2023-03-03T11:44:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f64417ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c153290 0x7f643c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f64417ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f643c1593c0 0x7f643c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f64417ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f643c0cd8e0 0x7f643c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f64417ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f643c153660 con 0x7f643c1593c0 2023-03-03T11:44:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f64417ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f643c06c360 con 0x7f643c0cd8e0 2023-03-03T11:44:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f64417ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643c15cea0 con 0x7f643c153290 2023-03-03T11:44:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f6433fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f643c1593c0 0x7f643c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f6433fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f643c1593c0 0x7f643c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56670/0 (socket says 172.21.15.102:56670) 2023-03-03T11:44:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.161+0000 7f6433fff700 1 -- 172.21.15.102:0/149737214 learned_addr learned my addr 172.21.15.102:0/149737214 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:44:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f643b7fe700 1 --2- 172.21.15.102:0/149737214 >> v2:172.21.15.172:3300/0 conn(0x7f643c0cd8e0 0x7f643c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f643affd700 1 --2- 172.21.15.102:0/149737214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c153290 0x7f643c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f6433fff700 1 -- 172.21.15.102:0/149737214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c153290 msgr2=0x7f643c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f6433fff700 1 --2- 172.21.15.102:0/149737214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c153290 0x7f643c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f6433fff700 1 -- 172.21.15.102:0/149737214 >> v2:172.21.15.172:3300/0 conn(0x7f643c0cd8e0 msgr2=0x7f643c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f6433fff700 1 --2- 172.21.15.102:0/149737214 >> v2:172.21.15.172:3300/0 conn(0x7f643c0cd8e0 0x7f643c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f6433fff700 1 -- 172.21.15.102:0/149737214 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c06c7f0 con 0x7f643c1593c0 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f6433fff700 1 --2- 172.21.15.102:0/149737214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 0x7f643c0cde20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f643c06d0e0 tx=0x7f6424009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d4ad3b2e235506 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.162+0000 7f643a7fc700 1 -- 172.21.15.102:0/149737214 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424011830 con 0x7f643c1593c0 2023-03-03T11:44:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f643a7fc700 1 -- 172.21.15.102:0/149737214 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424011990 con 0x7f643c1593c0 2023-03-03T11:44:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f643a7fc700 1 -- 172.21.15.102:0/149737214 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424011d20 con 0x7f643c1593c0 2023-03-03T11:44:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 -- 172.21.15.102:0/149737214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 msgr2=0x7f643c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 --2- 172.21.15.102:0/149737214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 0x7f643c0cde20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f643c06d0e0 tx=0x7f6424009bd0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 -- 172.21.15.102:0/149737214 shutdown_connections 2023-03-03T11:44:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 --2- 172.21.15.102:0/149737214 >> v2:172.21.15.172:3300/0 conn(0x7f643c0cd8e0 0x7f643c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 --2- 172.21.15.102:0/149737214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c153290 0x7f643c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 --2- 172.21.15.102:0/149737214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 0x7f643c0cde20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 -- 172.21.15.102:0/149737214 >> 172.21.15.102:0/149737214 conn(0x7f643c065fb0 msgr2=0x7f643c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:44:28.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 -- 172.21.15.102:0/149737214 shutdown_connections 2023-03-03T11:44:28.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.163+0000 7f64417ff700 1 -- 172.21.15.102:0/149737214 wait complete. 2023-03-03T11:44:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.165+0000 7f64417ff700 1 Processor -- start 2023-03-03T11:44:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.165+0000 7f64417ff700 1 -- start start 2023-03-03T11:44:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.166+0000 7f64417ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c0cd8e0 0x7f643c1f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.166+0000 7f64417ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f643c153290 0x7f643c14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.166+0000 7f64417ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 0x7f643c14fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.166+0000 7f64417ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c1e6410 con 0x7f643c1593c0 2023-03-03T11:44:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.166+0000 7f64417ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c0b2010 con 0x7f643c153290 2023-03-03T11:44:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.166+0000 7f64417ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643c0cf550 con 0x7f643c0cd8e0 2023-03-03T11:44:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f6433fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f643c153290 0x7f643c14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f643affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c0cd8e0 0x7f643c1f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f643b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 0x7f643c14fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f643affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c0cd8e0 0x7f643c1f2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:52442/0 (socket says 172.21.15.102:52442) 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f643affd700 1 -- 172.21.15.102:0/819981127 learned_addr learned my addr 172.21.15.102:0/819981127 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f6433fff700 1 -- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c0cd8e0 msgr2=0x7f643c1f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f6433fff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c0cd8e0 0x7f643c1f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f6433fff700 1 -- 172.21.15.102:0/819981127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 msgr2=0x7f643c14fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f6433fff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 0x7f643c14fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f6433fff700 1 -- 172.21.15.102:0/819981127 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6424009550 con 0x7f643c153290 2023-03-03T11:44:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.167+0000 7f6433fff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f643c153290 0x7f643c14f8d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6424016000 tx=0x7f6424004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:44:28.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.168+0000 7f64337fe700 1 -- 172.21.15.102:0/819981127 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424004240 con 0x7f643c153290 2023-03-03T11:44:28.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.168+0000 7f64337fe700 1 -- 172.21.15.102:0/819981127 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642401f070 con 0x7f643c153290 2023-03-03T11:44:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.168+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c0ccde0 con 0x7f643c153290 2023-03-03T11:44:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.168+0000 7f64337fe700 1 -- 172.21.15.102:0/819981127 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240045e0 con 0x7f643c153290 2023-03-03T11:44:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.168+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c15d160 con 0x7f643c153290 2023-03-03T11:44:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.169+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c0cd0c0 con 0x7f643c153290 2023-03-03T11:44:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.170+0000 7f64337fe700 1 -- 172.21.15.102:0/819981127 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f642402a020 con 0x7f643c153290 2023-03-03T11:44:28.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.171+0000 7f64337fe700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f641c068320 0x7f641c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:28.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.172+0000 7f64337fe700 1 -- 172.21.15.102:0/819981127 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f64240870d0 con 0x7f643c153290 2023-03-03T11:44:28.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.172+0000 7f643affd700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f641c068320 0x7f641c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:28.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.172+0000 7f643affd700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f641c068320 0x7f641c06a7d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f642c007a70 tx=0x7f642c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:44:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.176+0000 7f64337fe700 1 -- 172.21.15.102:0/819981127 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f64240537e0 con 0x7f643c153290 2023-03-03T11:44:28.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.410+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f643c04e6e0 con 0x7f643c153290 2023-03-03T11:44:28.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.411+0000 7f64337fe700 1 -- 172.21.15.102:0/819981127 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6424056e00 con 0x7f643c153290 2023-03-03T11:44:28.414 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:44:28.414 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:44:28.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f641c068320 msgr2=0x7f641c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:28.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f641c068320 0x7f641c06a7d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f642c007a70 tx=0x7f642c008040 comp rx=0 tx=0).stop 2023-03-03T11:44:28.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f643c153290 msgr2=0x7f643c14f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:28.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f643c153290 0x7f643c14f8d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6424016000 tx=0x7f6424004750 comp rx=0 tx=0).stop 2023-03-03T11:44:28.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 shutdown_connections 2023-03-03T11:44:28.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f641c068320 0x7f641c06a7d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f643c0cd8e0 0x7f643c1f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1593c0 0x7f643c14fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 --2- 172.21.15.102:0/819981127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f643c153290 0x7f643c14f8d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:28.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.413+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 >> 172.21.15.102:0/819981127 conn(0x7f643c065fb0 msgr2=0x7f643c0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:44:28.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.414+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 shutdown_connections 2023-03-03T11:44:28.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:44:28.414+0000 7f64417ff700 1 -- 172.21.15.102:0/819981127 wait complete. 2023-03-03T11:44:28.418 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:44:28.443 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T11:44:41.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:41.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:11.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:11.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:42.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:42.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:06.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:46:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.689+0000 7f3a07dff700 1 Processor -- start 2023-03-03T11:46:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.690+0000 7f3a07dff700 1 -- start start 2023-03-03T11:46:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.691+0000 7f3a07dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd8e0 0x7f3a000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.691+0000 7f3a07dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3a000cd390 0x7f3a000ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.691+0000 7f3a07dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a00153660 con 0x7f3a00153290 2023-03-03T11:46:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.691+0000 7f3a07dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0006c360 con 0x7f3a000cd390 2023-03-03T11:46:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.691+0000 7f3a07dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0015cea0 con 0x7f3a000cd8e0 2023-03-03T11:46:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.691+0000 7f3a05b9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3a00153290 0x7f3a00158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58346/0 (socket says 172.21.15.102:58346) 2023-03-03T11:46:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.691+0000 7f3a05b9b700 1 -- 172.21.15.102:0/1903729045 learned_addr learned my addr 172.21.15.102:0/1903729045 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:46:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a0639c700 1 --2- 172.21.15.102:0/1903729045 >> v2:172.21.15.102:3300/0 conn(0x7f3a000cd390 0x7f3a000ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a0539a700 1 --2- 172.21.15.102:0/1903729045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd8e0 0x7f3a000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a04b99700 1 -- 172.21.15.102:0/1903729045 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689817051 0 0) 0x7f3a00153660 con 0x7f3a00153290 2023-03-03T11:46:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a04b99700 1 -- 172.21.15.102:0/1903729045 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ec0035b0 con 0x7f3a00153290 2023-03-03T11:46:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a0639c700 1 -- 172.21.15.102:0/1903729045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd8e0 msgr2=0x7f3a000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:46:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a0639c700 1 --2- 172.21.15.102:0/1903729045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd8e0 0x7f3a000cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a0639c700 1 -- 172.21.15.102:0/1903729045 >> v1:172.21.15.102:6789/0 conn(0x7f3a00153290 legacy=0x7f3a00158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a0639c700 1 -- 172.21.15.102:0/1903729045 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0006c7f0 con 0x7f3a000cd390 2023-03-03T11:46:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.692+0000 7f3a0639c700 1 --2- 172.21.15.102:0/1903729045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a000cd390 0x7f3a000ce360 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f39f8009d90 tx=0x7f39f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3143c3e1ead3fb29 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a04b99700 1 -- 172.21.15.102:0/1903729045 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f800e070 con 0x7f3a000cd390 2023-03-03T11:46:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a04b99700 1 -- 172.21.15.102:0/1903729045 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f800bb20 con 0x7f3a000cd390 2023-03-03T11:46:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a04b99700 1 -- 172.21.15.102:0/1903729045 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f8012430 con 0x7f3a000cd390 2023-03-03T11:46:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a07dff700 1 -- 172.21.15.102:0/1903729045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a000cd390 msgr2=0x7f3a000ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a07dff700 1 --2- 172.21.15.102:0/1903729045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a000cd390 0x7f3a000ce360 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f39f8009d90 tx=0x7f39f800b330 comp rx=0 tx=0).stop 2023-03-03T11:46:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a07dff700 1 -- 172.21.15.102:0/1903729045 shutdown_connections 2023-03-03T11:46:06.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a07dff700 1 --2- 172.21.15.102:0/1903729045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a000cd390 0x7f3a000ce360 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a07dff700 1 --2- 172.21.15.102:0/1903729045 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd8e0 0x7f3a000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a07dff700 1 -- 172.21.15.102:0/1903729045 >> 172.21.15.102:0/1903729045 conn(0x7f3a00065fb0 msgr2=0x7f3a000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:06.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.693+0000 7f3a07dff700 1 -- 172.21.15.102:0/1903729045 shutdown_connections 2023-03-03T11:46:06.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.694+0000 7f3a07dff700 1 -- 172.21.15.102:0/1903729045 wait complete. 2023-03-03T11:46:06.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.695+0000 7f3a07dff700 1 Processor -- start 2023-03-03T11:46:06.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.696+0000 7f3a07dff700 1 -- start start 2023-03-03T11:46:06.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a07dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd390 0x7f3a001ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:06.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a07dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a000cd8e0 0x7f3a001efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:06.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a07dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a00153290 0x7f3a00150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:06.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a07dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a000c8660 con 0x7f3a00153290 2023-03-03T11:46:06.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a07dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a000c6f70 con 0x7f3a000cd8e0 2023-03-03T11:46:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a07dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a000c7250 con 0x7f3a000cd390 2023-03-03T11:46:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a05b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd390 0x7f3a001ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0639c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a00153290 0x7f3a00150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0539a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a000cd8e0 0x7f3a001efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0539a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a000cd8e0 0x7f3a001efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46828/0 (socket says 172.21.15.102:46828) 2023-03-03T11:46:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a05b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd390 0x7f3a001ef820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54350/0 (socket says 172.21.15.102:54350) 2023-03-03T11:46:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0539a700 1 -- 172.21.15.102:0/2378573007 learned_addr learned my addr 172.21.15.102:0/2378573007 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0639c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a00153290 0x7f3a00150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45870/0 (socket says 172.21.15.102:45870) 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0639c700 1 -- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd390 msgr2=0x7f3a001ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0639c700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd390 0x7f3a001ef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0639c700 1 -- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a000cd8e0 msgr2=0x7f3a001efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0639c700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a000cd8e0 0x7f3a001efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.697+0000 7f3a0639c700 1 -- 172.21.15.102:0/2378573007 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f8009a70 con 0x7f3a00153290 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.698+0000 7f3a0639c700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a00153290 0x7f3a00150fe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f39f8016000 tx=0x7f39f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:06.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.698+0000 7f39f67fc700 1 -- 172.21.15.102:0/2378573007 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f8012430 con 0x7f3a00153290 2023-03-03T11:46:06.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.698+0000 7f39f67fc700 1 -- 172.21.15.102:0/2378573007 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f8020070 con 0x7f3a00153290 2023-03-03T11:46:06.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.698+0000 7f39f67fc700 1 -- 172.21.15.102:0/2378573007 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f80098f0 con 0x7f3a00153290 2023-03-03T11:46:06.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.698+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a000ccde0 con 0x7f3a00153290 2023-03-03T11:46:06.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.698+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0015d160 con 0x7f3a00153290 2023-03-03T11:46:06.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.700+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a000cd0c0 con 0x7f3a00153290 2023-03-03T11:46:06.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.701+0000 7f39f67fc700 1 -- 172.21.15.102:0/2378573007 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f39f802e070 con 0x7f3a00153290 2023-03-03T11:46:06.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.702+0000 7f39f67fc700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39e4068320 0x7f39e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:06.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.702+0000 7f39f67fc700 1 -- 172.21.15.102:0/2378573007 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f39f8022070 con 0x7f3a00153290 2023-03-03T11:46:06.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.702+0000 7f3a05b9b700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39e4068320 0x7f39e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:06.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.703+0000 7f3a05b9b700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39e4068320 0x7f39e406a7d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f39fc007fd0 tx=0x7f39fc007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:06.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.706+0000 7f39f67fc700 1 -- 172.21.15.102:0/2378573007 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39f8053fa0 con 0x7f3a00153290 2023-03-03T11:46:06.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.940+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a0004e6e0 con 0x7f3a00153290 2023-03-03T11:46:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.941+0000 7f39f67fc700 1 -- 172.21.15.102:0/2378573007 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f39f80575c0 con 0x7f3a00153290 2023-03-03T11:46:06.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:46:06.944 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:46:06.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.943+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39e4068320 msgr2=0x7f39e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.943+0000 7f3a07dff700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39e4068320 0x7f39e406a7d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f39fc007fd0 tx=0x7f39fc007dc0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a00153290 msgr2=0x7f3a00150fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a00153290 0x7f3a00150fe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f39f8016000 tx=0x7f39f8004ad0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 shutdown_connections 2023-03-03T11:46:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39e4068320 0x7f39e406a7d0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3a000cd390 0x7f3a001ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a00153290 0x7f3a00150fe0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 --2- 172.21.15.102:0/2378573007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a000cd8e0 0x7f3a001efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 >> 172.21.15.102:0/2378573007 conn(0x7f3a00065fb0 msgr2=0x7f3a00157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 shutdown_connections 2023-03-03T11:46:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:46:06.944+0000 7f3a07dff700 1 -- 172.21.15.102:0/2378573007 wait complete. 2023-03-03T11:46:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:46:06.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 91 secs before thrashing 2023-03-03T11:46:12.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:12.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:42.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:42.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:12.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:12.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:37.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:47:38.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.237+0000 7fe5d0d9d700 1 Processor -- start 2023-03-03T11:47:38.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.237+0000 7fe5d0d9d700 1 -- start start 2023-03-03T11:47:38.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5d0d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe5cc0ced00 0x7fe5cc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:38.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5d0d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5cc0b3970 0x7fe5cc0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:38.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5d0d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5cc0c3d20 con 0x7fe5cc0cf0d0 2023-03-03T11:47:38.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5d0d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c4930 con 0x7fe5cc0b3970 2023-03-03T11:47:38.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5d0d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5cc0cf570 con 0x7fe5cc0ced00 2023-03-03T11:47:38.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5c2ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5cc0b3970 0x7fe5cc0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:38.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5cb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe5cc0ced00 0x7fe5cc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5cbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe5cc0cf0d0 0x7fe5cc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33298/0 (socket says 172.21.15.102:33298) 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5c2ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5cc0b3970 0x7fe5cc0c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47626/0 (socket says 172.21.15.102:47626) 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.239+0000 7fe5cbfff700 1 -- 172.21.15.102:0/1063694286 learned_addr learned my addr 172.21.15.102:0/1063694286 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5c2ffd700 1 -- 172.21.15.102:0/1063694286 >> v2:172.21.15.172:3300/0 conn(0x7fe5cc0ced00 msgr2=0x7fe5cc0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5c2ffd700 1 --2- 172.21.15.102:0/1063694286 >> v2:172.21.15.172:3300/0 conn(0x7fe5cc0ced00 0x7fe5cc0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5c2ffd700 1 -- 172.21.15.102:0/1063694286 >> v1:172.21.15.102:6789/0 conn(0x7fe5cc0cf0d0 legacy=0x7fe5cc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5c2ffd700 1 -- 172.21.15.102:0/1063694286 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc15cba0 con 0x7fe5cc0b3970 2023-03-03T11:47:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5c2ffd700 1 --2- 172.21.15.102:0/1063694286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc0c41b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0b1b40 tx=0x7fe5b4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d210c1624bd0b5f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:47:38.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5caffd700 1 -- 172.21.15.102:0/1063694286 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4011830 con 0x7fe5cc0b3970 2023-03-03T11:47:38.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5caffd700 1 -- 172.21.15.102:0/1063694286 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b4011990 con 0x7fe5cc0b3970 2023-03-03T11:47:38.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.240+0000 7fe5caffd700 1 -- 172.21.15.102:0/1063694286 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4011d20 con 0x7fe5cc0b3970 2023-03-03T11:47:38.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/1063694286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0b3970 msgr2=0x7fe5cc0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:38.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/1063694286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc0c41b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0b1b40 tx=0x7fe5b4009b20 comp rx=0 tx=0).stop 2023-03-03T11:47:38.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/1063694286 shutdown_connections 2023-03-03T11:47:38.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/1063694286 >> v2:172.21.15.172:3300/0 conn(0x7fe5cc0ced00 0x7fe5cc0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/1063694286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc0c41b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/1063694286 >> 172.21.15.102:0/1063694286 conn(0x7fe5cc08b1c0 msgr2=0x7fe5cc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:47:38.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/1063694286 shutdown_connections 2023-03-03T11:47:38.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.241+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/1063694286 wait complete. 2023-03-03T11:47:38.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.243+0000 7fe5d0d9d700 1 Processor -- start 2023-03-03T11:47:38.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.244+0000 7fe5d0d9d700 1 -- start start 2023-03-03T11:47:38.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5d0d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc1e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:38.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5d0d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0ced00 0x7fe5cc0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:38.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5d0d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5cc0cf0d0 0x7fe5cc0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:38.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5d0d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1f2cd0 con 0x7fe5cc0ced00 2023-03-03T11:47:38.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5d0d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c82b0 con 0x7fe5cc0b3970 2023-03-03T11:47:38.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5d0d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1f20e0 con 0x7fe5cc0cf0d0 2023-03-03T11:47:38.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0ced00 0x7fe5cc0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:38.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0ced00 0x7fe5cc0b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47632/0 (socket says 172.21.15.102:47632) 2023-03-03T11:47:38.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 -- 172.21.15.102:0/2387081214 learned_addr learned my addr 172.21.15.102:0/2387081214 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:47:38.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5cb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc1e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:38.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5cbfff700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5cc0cf0d0 0x7fe5cc0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:38.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 -- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5cc0cf0d0 msgr2=0x7fe5cc0c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:38.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5cc0cf0d0 0x7fe5cc0c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 -- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5cc0b3970 msgr2=0x7fe5cc1e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc1e5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5c2ffd700 1 -- 172.21.15.102:0/2387081214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b40094a0 con 0x7fe5cc0ced00 2023-03-03T11:47:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.245+0000 7fe5cb7fe700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc1e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:47:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.246+0000 7fe5c2ffd700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0ced00 0x7fe5cc0b2100 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40090b0 tx=0x7fe5b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:47:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.246+0000 7fe5c8ff9700 1 -- 172.21.15.102:0/2387081214 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40036a0 con 0x7fe5cc0ced00 2023-03-03T11:47:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.246+0000 7fe5c8ff9700 1 -- 172.21.15.102:0/2387081214 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b401e070 con 0x7fe5cc0ced00 2023-03-03T11:47:38.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.246+0000 7fe5c8ff9700 1 -- 172.21.15.102:0/2387081214 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4003a60 con 0x7fe5cc0ced00 2023-03-03T11:47:38.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.246+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc06bf20 con 0x7fe5cc0ced00 2023-03-03T11:47:38.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.246+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc15ce80 con 0x7fe5cc0ced00 2023-03-03T11:47:38.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.247+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc06c200 con 0x7fe5cc0ced00 2023-03-03T11:47:38.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.249+0000 7fe5c8ff9700 1 -- 172.21.15.102:0/2387081214 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5b400f050 con 0x7fe5cc0ced00 2023-03-03T11:47:38.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.250+0000 7fe5c8ff9700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5ac0682f0 0x7fe5ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:38.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.250+0000 7fe5c8ff9700 1 -- 172.21.15.102:0/2387081214 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe5b4086c70 con 0x7fe5cc0ced00 2023-03-03T11:47:38.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.250+0000 7fe5cb7fe700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5ac0682f0 0x7fe5ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:38.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.251+0000 7fe5cb7fe700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5ac0682f0 0x7fe5ac06a7a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009f30 tx=0x7fe5bc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:47:38.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.254+0000 7fe5c8ff9700 1 -- 172.21.15.102:0/2387081214 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5b4053380 con 0x7fe5cc0ced00 2023-03-03T11:47:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.491+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5cc04e6e0 con 0x7fe5cc0ced00 2023-03-03T11:47:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.491+0000 7fe5c8ff9700 1 -- 172.21.15.102:0/2387081214 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe5b4020090 con 0x7fe5cc0ced00 2023-03-03T11:47:38.494 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:47:38.494 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:47:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5ac0682f0 msgr2=0x7fe5ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5ac0682f0 0x7fe5ac06a7a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009f30 tx=0x7fe5bc009210 comp rx=0 tx=0).stop 2023-03-03T11:47:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0ced00 msgr2=0x7fe5cc0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0ced00 0x7fe5cc0b2100 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40090b0 tx=0x7fe5b4004750 comp rx=0 tx=0).stop 2023-03-03T11:47:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 shutdown_connections 2023-03-03T11:47:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5ac0682f0 0x7fe5ac06a7a0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5cc0cf0d0 0x7fe5cc0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5cc0b3970 0x7fe5cc1e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 --2- 172.21.15.102:0/2387081214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc0ced00 0x7fe5cc0b2100 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 >> 172.21.15.102:0/2387081214 conn(0x7fe5cc08b1c0 msgr2=0x7fe5cc0c4100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:47:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 shutdown_connections 2023-03-03T11:47:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:47:38.494+0000 7fe5d0d9d700 1 -- 172.21.15.102:0/2387081214 wait complete. 2023-03-03T11:47:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:47:38.524 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2023-03-03T11:47:42.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:42.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:12.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:12.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:43.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:43.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:13.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:13.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:30.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:49:30.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.769+0000 7f1ec53ff700 1 Processor -- start 2023-03-03T11:49:30.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.769+0000 7f1ec53ff700 1 -- start start 2023-03-03T11:49:30.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.770+0000 7f1ec53ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ec00cd8e0 0x7f1ec00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.770+0000 7f1ec53ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec0153660 con 0x7f1ec00cd390 2023-03-03T11:49:30.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.770+0000 7f1ec53ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec006c360 con 0x7f1ec0153290 2023-03-03T11:49:30.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.770+0000 7f1ec53ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec015cea0 con 0x7f1ec00cd8e0 2023-03-03T11:49:30.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.771+0000 7f1ebeffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1ec0153290 0x7f1ec0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52486/0 (socket says 172.21.15.102:52486) 2023-03-03T11:49:30.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.771+0000 7f1ebeffd700 1 -- 172.21.15.102:0/3153162465 learned_addr learned my addr 172.21.15.102:0/3153162465 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:49:30.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.771+0000 7f1ebe7fc700 1 --2- 172.21.15.102:0/3153162465 >> v2:172.21.15.102:3300/0 conn(0x7f1ec00cd8e0 0x7f1ec00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.771+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836138730 0 0) 0x7f1ec006c360 con 0x7f1ec0153290 2023-03-03T11:49:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.771+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea80035b0 con 0x7f1ec0153290 2023-03-03T11:49:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.771+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384637903 0 0) 0x7f1ec0153660 con 0x7f1ec00cd390 2023-03-03T11:49:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.771+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec006c360 con 0x7f1ec00cd390 2023-03-03T11:49:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 84928119 0 0) 0x7f1ea80035b0 con 0x7f1ec0153290 2023-03-03T11:49:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 >> v2:172.21.15.102:3300/0 conn(0x7f1ec00cd8e0 msgr2=0x7f1ec00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:49:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 --2- 172.21.15.102:0/3153162465 >> v2:172.21.15.102:3300/0 conn(0x7f1ec00cd8e0 0x7f1ec00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 >> v1:172.21.15.102:6789/0 conn(0x7f1ec00cd390 legacy=0x7f1ec00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec006c7f0 con 0x7f1ec0153290 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1894587018 0 0) 0x7f1eb0001670 con 0x7f1ec0153290 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebe7fc700 1 --2- 172.21.15.102:0/3153162465 >> v2:172.21.15.102:3300/0 conn(0x7f1ec00cd8e0 0x7f1ec00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 >> v1:172.21.15.172:6789/0 conn(0x7f1ec0153290 legacy=0x7f1ec0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ea8003c00 0x7f1ea8003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8004850 0x7f1ea8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ea8005200 0x7f1ea800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec015cea0 con 0x7f1ea8004850 2023-03-03T11:49:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea800c030 con 0x7f1ea8003c00 2023-03-03T11:49:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea800c1f0 con 0x7f1ea8005200 2023-03-03T11:49:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebeffd700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ea8003c00 0x7f1ea8003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebe7fc700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8004850 0x7f1ea8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.772+0000 7f1ebf7fe700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ea8005200 0x7f1ea800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebeffd700 1 -- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ea8005200 msgr2=0x7f1ea800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:49:30.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebeffd700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ea8005200 0x7f1ea800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebeffd700 1 -- 172.21.15.102:0/3153162465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8004850 msgr2=0x7f1ea8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:49:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebeffd700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8004850 0x7f1ea8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebeffd700 1 -- 172.21.15.102:0/3153162465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80035b0 con 0x7f1ea8003c00 2023-03-03T11:49:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebe7fc700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8004850 0x7f1ea8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:49:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebf7fe700 1 -- 172.21.15.102:0/3153162465 reap_dead start 2023-03-03T11:49:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebeffd700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ea8003c00 0x7f1ea8003ff0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000c210 tx=0x7f1ea8005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38cd0d19d8fad514 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:49:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000cd50 con 0x7f1ea8003c00 2023-03-03T11:49:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb0005960 con 0x7f1ea8003c00 2023-03-03T11:49:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.773+0000 7f1ebdffb700 1 -- 172.21.15.102:0/3153162465 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0005d10 con 0x7f1ea8003c00 2023-03-03T11:49:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.774+0000 7f1ec53ff700 1 -- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ea8003c00 msgr2=0x7f1ea8003ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.774+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ea8003c00 0x7f1ea8003ff0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000c210 tx=0x7f1ea8005880 comp rx=0 tx=0).stop 2023-03-03T11:49:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.774+0000 7f1ec53ff700 1 -- 172.21.15.102:0/3153162465 shutdown_connections 2023-03-03T11:49:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.774+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/3153162465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ea8003c00 0x7f1ea8003ff0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.774+0000 7f1ec53ff700 1 -- 172.21.15.102:0/3153162465 >> 172.21.15.102:0/3153162465 conn(0x7f1ec0065fb0 msgr2=0x7f1ec00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:49:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.774+0000 7f1ec53ff700 1 -- 172.21.15.102:0/3153162465 shutdown_connections 2023-03-03T11:49:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.774+0000 7f1ec53ff700 1 -- 172.21.15.102:0/3153162465 wait complete. 2023-03-03T11:49:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.776+0000 7f1ec53ff700 1 Processor -- start 2023-03-03T11:49:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.776+0000 7f1ec53ff700 1 -- start start 2023-03-03T11:49:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ec53ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8003c00 0x7f1ec00c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ec53ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ec00b3fd0 0x7f1ec01efb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ec53ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec01f1ec0 0x7f1ec014fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ec53ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0151ca0 con 0x7f1ea8003c00 2023-03-03T11:49:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ec53ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00b3bb0 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ec53ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00c6730 con 0x7f1ec00b3fd0 2023-03-03T11:49:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ebeffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8003c00 0x7f1ec00c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ebf7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec01f1ec0 0x7f1ec014fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.777+0000 7f1ebeffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8003c00 0x7f1ec00c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39550/0 (socket says 172.21.15.102:39550) 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebeffd700 1 -- 172.21.15.102:0/703639814 learned_addr learned my addr 172.21.15.102:0/703639814 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebe7fc700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ec00b3fd0 0x7f1ec01efb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebf7fe700 1 -- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ec00b3fd0 msgr2=0x7f1ec01efb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebf7fe700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ec00b3fd0 0x7f1ec01efb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebf7fe700 1 -- 172.21.15.102:0/703639814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8003c00 msgr2=0x7f1ec00c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebf7fe700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8003c00 0x7f1ec00c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:30.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebf7fe700 1 -- 172.21.15.102:0/703639814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec00ccde0 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebeffd700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8003c00 0x7f1ec00c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:49:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ebf7fe700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec01f1ec0 0x7f1ec014fed0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00b1940 tx=0x7f1eb800d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:49:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ea77fe700 1 -- 172.21.15.102:0/703639814 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8003c10 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.778+0000 7f1ea77fe700 1 -- 172.21.15.102:0/703639814 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8003d70 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.779+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec015d160 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.779+0000 7f1ea77fe700 1 -- 172.21.15.102:0/703639814 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800db20 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.779+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec00cd0c0 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.780+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec01f2b60 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.781+0000 7f1ea77fe700 1 -- 172.21.15.102:0/703639814 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1eb800dc80 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.782+0000 7f1ea77fe700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1eac0682f0 0x7f1eac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.782+0000 7f1ea77fe700 1 -- 172.21.15.102:0/703639814 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1eb8086060 con 0x7f1ec01f1ec0 2023-03-03T11:49:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.783+0000 7f1ebeffd700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1eac0682f0 0x7f1eac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.783+0000 7f1ebeffd700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1eac0682f0 0x7f1eac06a7a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0001820 tx=0x7f1eb0007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:49:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:30.786+0000 7f1ea77fe700 1 -- 172.21.15.102:0/703639814 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1eb8052770 con 0x7f1ec01f1ec0 2023-03-03T11:49:31.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.018+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ec004e6e0 con 0x7f1ec01f1ec0 2023-03-03T11:49:31.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.019+0000 7f1ea77fe700 1 -- 172.21.15.102:0/703639814 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1eb8055d90 con 0x7f1ec01f1ec0 2023-03-03T11:49:31.021 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:49:31.021 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:49:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1eac0682f0 msgr2=0x7f1eac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1eac0682f0 0x7f1eac06a7a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0001820 tx=0x7f1eb0007f90 comp rx=0 tx=0).stop 2023-03-03T11:49:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec01f1ec0 msgr2=0x7f1ec014fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec01f1ec0 0x7f1ec014fed0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00b1940 tx=0x7f1eb800d430 comp rx=0 tx=0).stop 2023-03-03T11:49:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 shutdown_connections 2023-03-03T11:49:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec01f1ec0 0x7f1ec014fed0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea8003c00 0x7f1ec00c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1eac0682f0 0x7f1eac06a7a0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 --2- 172.21.15.102:0/703639814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1ec00b3fd0 0x7f1ec01efb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 >> 172.21.15.102:0/703639814 conn(0x7f1ec0065fb0 msgr2=0x7f1ec0158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:49:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 shutdown_connections 2023-03-03T11:49:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:49:31.022+0000 7f1ec53ff700 1 -- 172.21.15.102:0/703639814 wait complete. 2023-03-03T11:49:31.026 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:49:31.051 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T11:49:43.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:43.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T11:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:50:06.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:50:06.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.320+0000 7f1669dff700 1 Processor -- start 2023-03-03T11:50:06.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.320+0000 7f1669dff700 1 -- start start 2023-03-03T11:50:06.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.321+0000 7f1669dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f16640cd8e0 0x7f16640cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:06.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.321+0000 7f1669dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16640ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:06.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.321+0000 7f1669dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1664153660 con 0x7f1664153290 2023-03-03T11:50:06.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.321+0000 7f1669dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166406c360 con 0x7f16640cd8e0 2023-03-03T11:50:06.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.321+0000 7f1669dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166415cea0 con 0x7f16640cd390 2023-03-03T11:50:06.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.321+0000 7f16637fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1664153290 0x7f1664158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44276/0 (socket says 172.21.15.102:44276) 2023-03-03T11:50:06.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f16637fe700 1 -- 172.21.15.102:0/824499396 learned_addr learned my addr 172.21.15.102:0/824499396 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:50:06.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f1662ffd700 1 --2- 172.21.15.102:0/824499396 >> v2:172.21.15.102:3300/0 conn(0x7f16640cd8e0 0x7f16640cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:06.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f1663fff700 1 --2- 172.21.15.102:0/824499396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16640ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f16627fc700 1 -- 172.21.15.102:0/824499396 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038710071 0 0) 0x7f1664153660 con 0x7f1664153290 2023-03-03T11:50:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f16627fc700 1 -- 172.21.15.102:0/824499396 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f164c0035b0 con 0x7f1664153290 2023-03-03T11:50:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f1663fff700 1 -- 172.21.15.102:0/824499396 >> v2:172.21.15.102:3300/0 conn(0x7f16640cd8e0 msgr2=0x7f16640cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f1663fff700 1 --2- 172.21.15.102:0/824499396 >> v2:172.21.15.102:3300/0 conn(0x7f16640cd8e0 0x7f16640cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f1663fff700 1 -- 172.21.15.102:0/824499396 >> v1:172.21.15.102:6789/0 conn(0x7f1664153290 legacy=0x7f1664158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f1663fff700 1 -- 172.21.15.102:0/824499396 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166406c7f0 con 0x7f16640cd390 2023-03-03T11:50:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.322+0000 7f1662ffd700 1 --2- 172.21.15.102:0/824499396 >> v2:172.21.15.102:3300/0 conn(0x7f16640cd8e0 0x7f16640cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:50:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f1663fff700 1 --2- 172.21.15.102:0/824499396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16640ce360 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f165c009d90 tx=0x7f165c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f298afc74219b126 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f16627fc700 1 -- 172.21.15.102:0/824499396 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c00e070 con 0x7f16640cd390 2023-03-03T11:50:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f16627fc700 1 -- 172.21.15.102:0/824499396 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c00bb20 con 0x7f16640cd390 2023-03-03T11:50:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f16627fc700 1 -- 172.21.15.102:0/824499396 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c012430 con 0x7f16640cd390 2023-03-03T11:50:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f1669dff700 1 -- 172.21.15.102:0/824499396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 msgr2=0x7f16640ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f1669dff700 1 --2- 172.21.15.102:0/824499396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16640ce360 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f165c009d90 tx=0x7f165c00b330 comp rx=0 tx=0).stop 2023-03-03T11:50:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f1669dff700 1 -- 172.21.15.102:0/824499396 shutdown_connections 2023-03-03T11:50:06.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f1669dff700 1 --2- 172.21.15.102:0/824499396 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16640ce360 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f1669dff700 1 --2- 172.21.15.102:0/824499396 >> v2:172.21.15.102:3300/0 conn(0x7f16640cd8e0 0x7f16640cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.323+0000 7f1669dff700 1 -- 172.21.15.102:0/824499396 >> 172.21.15.102:0/824499396 conn(0x7f1664065fb0 msgr2=0x7f16640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:06.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.324+0000 7f1669dff700 1 -- 172.21.15.102:0/824499396 shutdown_connections 2023-03-03T11:50:06.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.324+0000 7f1669dff700 1 -- 172.21.15.102:0/824499396 wait complete. 2023-03-03T11:50:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.326+0000 7f1669dff700 1 Processor -- start 2023-03-03T11:50:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.326+0000 7f1669dff700 1 -- start start 2023-03-03T11:50:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1669dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16641f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1669dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16640cd8e0 0x7f1664154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1669dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1664153290 0x7f166414d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1669dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641f25e0 con 0x7f16640cd8e0 2023-03-03T11:50:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1669dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641f2370 con 0x7f1664153290 2023-03-03T11:50:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1669dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1664158b50 con 0x7f16640cd390 2023-03-03T11:50:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1662ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16640cd8e0 0x7f1664154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1662ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16640cd8e0 0x7f1664154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59402/0 (socket says 172.21.15.102:59402) 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1662ffd700 1 -- 172.21.15.102:0/4217104673 learned_addr learned my addr 172.21.15.102:0/4217104673 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f16637fe700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16641f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.327+0000 7f1663fff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1664153290 0x7f166414d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f1662ffd700 1 -- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 msgr2=0x7f16641f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f1662ffd700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16641f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f1662ffd700 1 -- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1664153290 msgr2=0x7f166414d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f1662ffd700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1664153290 0x7f166414d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f1662ffd700 1 -- 172.21.15.102:0/4217104673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16580094a0 con 0x7f16640cd8e0 2023-03-03T11:50:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f1662ffd700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16640cd8e0 0x7f1664154170 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f165800ebe0 tx=0x7f165800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f164bfff700 1 -- 172.21.15.102:0/4217104673 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658009760 con 0x7f16640cd8e0 2023-03-03T11:50:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f164bfff700 1 -- 172.21.15.102:0/4217104673 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165800c850 con 0x7f16640cd8e0 2023-03-03T11:50:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f164bfff700 1 -- 172.21.15.102:0/4217104673 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165800cbe0 con 0x7f16640cd8e0 2023-03-03T11:50:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.328+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c009a70 con 0x7f16640cd8e0 2023-03-03T11:50:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.329+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16640ccde0 con 0x7f16640cd8e0 2023-03-03T11:50:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.330+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166415d160 con 0x7f16640cd8e0 2023-03-03T11:50:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.331+0000 7f164bfff700 1 -- 172.21.15.102:0/4217104673 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1658019030 con 0x7f16640cd8e0 2023-03-03T11:50:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.332+0000 7f164bfff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.332+0000 7f164bfff700 1 -- 172.21.15.102:0/4217104673 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1658085c20 con 0x7f16640cd8e0 2023-03-03T11:50:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.332+0000 7f16637fe700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.333+0000 7f16637fe700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1654007fd0 tx=0x7f1654007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.337+0000 7f164bfff700 1 -- 172.21.15.102:0/4217104673 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1658052330 con 0x7f16640cd8e0 2023-03-03T11:50:06.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.571+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16640cd0c0 con 0x7f16640cd8e0 2023-03-03T11:50:06.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.571+0000 7f164bfff700 1 -- 172.21.15.102:0/4217104673 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1658055950 con 0x7f16640cd8e0 2023-03-03T11:50:06.574 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:50:06.574 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:50:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 msgr2=0x7f165006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1654007fd0 tx=0x7f1654007dc0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16640cd8e0 msgr2=0x7f1664154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16640cd8e0 0x7f1664154170 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f165800ebe0 tx=0x7f165800eef0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 shutdown_connections 2023-03-03T11:50:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16500682f0 0x7f165006a7a0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16640cd390 0x7f16641f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1664153290 0x7f166414d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 --2- 172.21.15.102:0/4217104673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16640cd8e0 0x7f1664154170 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 >> 172.21.15.102:0/4217104673 conn(0x7f1664065fb0 msgr2=0x7f1664157560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 shutdown_connections 2023-03-03T11:50:06.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:06.574+0000 7f1669dff700 1 -- 172.21.15.102:0/4217104673 wait complete. 2023-03-03T11:50:06.579 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:50:06.604 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before thrashing 2023-03-03T11:50:10.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:50:10.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.852+0000 7f6447fff700 1 Processor -- start 2023-03-03T11:50:10.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.853+0000 7f6447fff700 1 -- start start 2023-03-03T11:50:10.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f6447fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6440153290 0x7f6440158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:10.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f6447fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f64400ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:10.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f6447fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6440153660 con 0x7f64401593c0 2023-03-03T11:50:10.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f6447fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644006c360 con 0x7f6440153290 2023-03-03T11:50:10.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f6447fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644015cea0 con 0x7f64400cd8e0 2023-03-03T11:50:10.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f6445d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6440153290 0x7f6440158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:10.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f644559a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f64401593c0 0x7f64400cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54378/0 (socket says 172.21.15.102:54378) 2023-03-03T11:50:10.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f644559a700 1 -- 172.21.15.102:0/880705808 learned_addr learned my addr 172.21.15.102:0/880705808 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:50:10.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.854+0000 7f644659c700 1 --2- 172.21.15.102:0/880705808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f64400ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.855+0000 7f644659c700 1 -- 172.21.15.102:0/880705808 >> v2:172.21.15.102:3300/0 conn(0x7f6440153290 msgr2=0x7f6440158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.855+0000 7f6444d99700 1 -- 172.21.15.102:0/880705808 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075433716 0 0) 0x7f6440153660 con 0x7f64401593c0 2023-03-03T11:50:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.855+0000 7f644659c700 1 --2- 172.21.15.102:0/880705808 >> v2:172.21.15.102:3300/0 conn(0x7f6440153290 0x7f6440158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.855+0000 7f644659c700 1 -- 172.21.15.102:0/880705808 >> v1:172.21.15.102:6789/0 conn(0x7f64401593c0 legacy=0x7f64400cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.855+0000 7f644659c700 1 -- 172.21.15.102:0/880705808 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644006c7f0 con 0x7f64400cd8e0 2023-03-03T11:50:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.855+0000 7f644659c700 1 --2- 172.21.15.102:0/880705808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f64400ce530 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6438009d90 tx=0x7f643800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc4f86f2612a7fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6444d99700 1 -- 172.21.15.102:0/880705808 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643800e070 con 0x7f64400cd8e0 2023-03-03T11:50:10.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6444d99700 1 -- 172.21.15.102:0/880705808 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643800bb20 con 0x7f64400cd8e0 2023-03-03T11:50:10.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6444d99700 1 -- 172.21.15.102:0/880705808 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438012430 con 0x7f64400cd8e0 2023-03-03T11:50:10.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 -- 172.21.15.102:0/880705808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 msgr2=0x7f64400ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:10.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 --2- 172.21.15.102:0/880705808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f64400ce530 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6438009d90 tx=0x7f643800b330 comp rx=0 tx=0).stop 2023-03-03T11:50:10.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 -- 172.21.15.102:0/880705808 shutdown_connections 2023-03-03T11:50:10.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 --2- 172.21.15.102:0/880705808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f64400ce530 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:10.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 --2- 172.21.15.102:0/880705808 >> v2:172.21.15.102:3300/0 conn(0x7f6440153290 0x7f6440158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:10.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 -- 172.21.15.102:0/880705808 >> 172.21.15.102:0/880705808 conn(0x7f6440065fb0 msgr2=0x7f64400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:10.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 -- 172.21.15.102:0/880705808 shutdown_connections 2023-03-03T11:50:10.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.856+0000 7f6447fff700 1 -- 172.21.15.102:0/880705808 wait complete. 2023-03-03T11:50:10.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.858+0000 7f6447fff700 1 Processor -- start 2023-03-03T11:50:10.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.859+0000 7f6447fff700 1 -- start start 2023-03-03T11:50:10.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6447fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f6440154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:10.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6447fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6440153290 0x7f64400c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:10.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6447fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64401593c0 0x7f64400c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:10.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6447fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64401541b0 con 0x7f6440153290 2023-03-03T11:50:10.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6447fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64400c9170 con 0x7f64401593c0 2023-03-03T11:50:10.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6447fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64401ef300 con 0x7f64400cd8e0 2023-03-03T11:50:10.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644559a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6440153290 0x7f64400c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:10.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6445d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f6440154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:10.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64401593c0 0x7f64400c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:10.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6445d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f6440154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51374/0 (socket says 172.21.15.102:51374) 2023-03-03T11:50:10.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f6445d9b700 1 -- 172.21.15.102:0/2270473807 learned_addr learned my addr 172.21.15.102:0/2270473807 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:50:10.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644659c700 1 -- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 msgr2=0x7f6440154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:10.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644659c700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f6440154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:10.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644659c700 1 -- 172.21.15.102:0/2270473807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6440153290 msgr2=0x7f64400c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644659c700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6440153290 0x7f64400c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644659c700 1 -- 172.21.15.102:0/2270473807 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6438009a70 con 0x7f64401593c0 2023-03-03T11:50:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.860+0000 7f644559a700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6440153290 0x7f64400c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:50:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.861+0000 7f644659c700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64401593c0 0x7f64400c4f30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6438016000 tx=0x7f6438004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.861+0000 7f64367fc700 1 -- 172.21.15.102:0/2270473807 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438012430 con 0x7f64401593c0 2023-03-03T11:50:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.861+0000 7f64367fc700 1 -- 172.21.15.102:0/2270473807 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6438020070 con 0x7f64401593c0 2023-03-03T11:50:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.861+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64400ccde0 con 0x7f64401593c0 2023-03-03T11:50:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.861+0000 7f64367fc700 1 -- 172.21.15.102:0/2270473807 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438004280 con 0x7f64401593c0 2023-03-03T11:50:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.861+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644015d160 con 0x7f64401593c0 2023-03-03T11:50:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.862+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64400cd0c0 con 0x7f64401593c0 2023-03-03T11:50:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.864+0000 7f64367fc700 1 -- 172.21.15.102:0/2270473807 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f643802e070 con 0x7f64401593c0 2023-03-03T11:50:10.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.865+0000 7f64367fc700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64240682f0 0x7f642406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:10.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.865+0000 7f64367fc700 1 -- 172.21.15.102:0/2270473807 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6438022070 con 0x7f64401593c0 2023-03-03T11:50:10.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.865+0000 7f6445d9b700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64240682f0 0x7f642406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:10.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.866+0000 7f6445d9b700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64240682f0 0x7f642406a7a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f643c009c00 tx=0x7f643c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:10.869+0000 7f64367fc700 1 -- 172.21.15.102:0/2270473807 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f64380540a0 con 0x7f64401593c0 2023-03-03T11:50:11.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.105+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64400c54f0 con 0x7f64401593c0 2023-03-03T11:50:11.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.106+0000 7f64367fc700 1 -- 172.21.15.102:0/2270473807 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f64380576c0 con 0x7f64401593c0 2023-03-03T11:50:11.109 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:50:11.109 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:50:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64240682f0 msgr2=0x7f642406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64240682f0 0x7f642406a7a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f643c009c00 tx=0x7f643c009210 comp rx=0 tx=0).stop 2023-03-03T11:50:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64401593c0 msgr2=0x7f64400c4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64401593c0 0x7f64400c4f30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6438016000 tx=0x7f6438004ad0 comp rx=0 tx=0).stop 2023-03-03T11:50:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 shutdown_connections 2023-03-03T11:50:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64240682f0 0x7f642406a7a0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64400cd8e0 0x7f6440154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64401593c0 0x7f64400c4f30 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 --2- 172.21.15.102:0/2270473807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6440153290 0x7f64400c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 >> 172.21.15.102:0/2270473807 conn(0x7f6440065fb0 msgr2=0x7f644015ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.109+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 shutdown_connections 2023-03-03T11:50:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:11.110+0000 7f6447fff700 1 -- 172.21.15.102:0/2270473807 wait complete. 2023-03-03T11:50:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:50:11.139 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-03-03T11:50:13.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:13.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:23.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:50:23.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.387+0000 7f2c6d1ff700 1 Processor -- start 2023-03-03T11:50:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.387+0000 7f2c6d1ff700 1 -- start start 2023-03-03T11:50:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.388+0000 7f2c6d1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c680cd390 0x7f2c680ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.388+0000 7f2c6d1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c68153660 con 0x7f2c68153290 2023-03-03T11:50:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.388+0000 7f2c6d1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6806c360 con 0x7f2c680cd8e0 2023-03-03T11:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.388+0000 7f2c6d1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6815cea0 con 0x7f2c680cd390 2023-03-03T11:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c66d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2c68153290 0x7f2c68158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46394/0 (socket says 172.21.15.102:46394) 2023-03-03T11:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c66d9d700 1 -- 172.21.15.102:0/998678869 learned_addr learned my addr 172.21.15.102:0/998678869 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c6759e700 1 --2- 172.21.15.102:0/998678869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c680cd390 0x7f2c680ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c65d9b700 1 -- 172.21.15.102:0/998678869 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548065205 0 0) 0x7f2c6806c360 con 0x7f2c680cd8e0 2023-03-03T11:50:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c65d9b700 1 -- 172.21.15.102:0/998678869 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c500035b0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c65d9b700 1 -- 172.21.15.102:0/998678869 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263161429 0 0) 0x7f2c68153660 con 0x7f2c68153290 2023-03-03T11:50:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c6759e700 1 -- 172.21.15.102:0/998678869 >> v1:172.21.15.172:6789/0 conn(0x7f2c680cd8e0 legacy=0x7f2c680cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c6759e700 1 -- 172.21.15.102:0/998678869 >> v1:172.21.15.102:6789/0 conn(0x7f2c68153290 legacy=0x7f2c68158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c6759e700 1 -- 172.21.15.102:0/998678869 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6806c7f0 con 0x7f2c680cd390 2023-03-03T11:50:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.389+0000 7f2c6759e700 1 --2- 172.21.15.102:0/998678869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c680cd390 0x7f2c680ce530 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009dc0 tx=0x7f2c6000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a9e0e67b32559f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c65d9b700 1 -- 172.21.15.102:0/998678869 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6000e070 con 0x7f2c680cd390 2023-03-03T11:50:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c65d9b700 1 -- 172.21.15.102:0/998678869 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6000bb20 con 0x7f2c680cd390 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c65d9b700 1 -- 172.21.15.102:0/998678869 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6000bed0 con 0x7f2c680cd390 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/998678869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c680cd390 msgr2=0x7f2c680ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/998678869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c680cd390 0x7f2c680ce530 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009dc0 tx=0x7f2c6000b330 comp rx=0 tx=0).stop 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/998678869 shutdown_connections 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/998678869 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c680cd390 0x7f2c680ce530 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/998678869 >> 172.21.15.102:0/998678869 conn(0x7f2c68065fb0 msgr2=0x7f2c680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.390+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/998678869 shutdown_connections 2023-03-03T11:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.391+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/998678869 wait complete. 2023-03-03T11:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.392+0000 7f2c6d1ff700 1 Processor -- start 2023-03-03T11:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.393+0000 7f2c6d1ff700 1 -- start start 2023-03-03T11:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6d1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c680cd390 0x7f2c681579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6d1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c680cd8e0 0x7f2c68155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6d1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c68153290 0x7f2c681558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6d1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b4b30 con 0x7f2c680cd8e0 2023-03-03T11:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6d1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b3800 con 0x7f2c680cd390 2023-03-03T11:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6d1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b3aa0 con 0x7f2c68153290 2023-03-03T11:50:23.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c680cd8e0 0x7f2c68155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c680cd8e0 0x7f2c68155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59478/0 (socket says 172.21.15.102:59478) 2023-03-03T11:50:23.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6659c700 1 -- 172.21.15.102:0/51675241 learned_addr learned my addr 172.21.15.102:0/51675241 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c66d9d700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c680cd390 0x7f2c681579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6759e700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c68153290 0x7f2c681558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6659c700 1 -- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c68153290 msgr2=0x7f2c681558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6659c700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c68153290 0x7f2c681558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6659c700 1 -- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c680cd390 msgr2=0x7f2c681579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.394+0000 7f2c6659c700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c680cd390 0x7f2c681579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.395+0000 7f2c6659c700 1 -- 172.21.15.102:0/51675241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60009a70 con 0x7f2c680cd8e0 2023-03-03T11:50:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.395+0000 7f2c6659c700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c680cd8e0 0x7f2c68155360 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00c410 tx=0x7f2c5c00c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.395+0000 7f2c4f7fe700 1 -- 172.21.15.102:0/51675241 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0052d0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.395+0000 7f2c4f7fe700 1 -- 172.21.15.102:0/51675241 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c005450 con 0x7f2c680cd8e0 2023-03-03T11:50:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.395+0000 7f2c4f7fe700 1 -- 172.21.15.102:0/51675241 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0038c0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.395+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c680ccde0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.395+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6815d160 con 0x7f2c680cd8e0 2023-03-03T11:50:23.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.397+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c680cd0c0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.398+0000 7f2c4f7fe700 1 -- 172.21.15.102:0/51675241 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c5c019030 con 0x7f2c680cd8e0 2023-03-03T11:50:23.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.399+0000 7f2c4f7fe700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c540682f0 0x7f2c5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.399+0000 7f2c4f7fe700 1 -- 172.21.15.102:0/51675241 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2c5c084fe0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.399+0000 7f2c66d9d700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c540682f0 0x7f2c5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.400+0000 7f2c66d9d700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c540682f0 0x7f2c5406a7a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2c58007fd0 tx=0x7f2c58007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:23.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.403+0000 7f2c4f7fe700 1 -- 172.21.15.102:0/51675241 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c5c0516f0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.636+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c681549a0 con 0x7f2c680cd8e0 2023-03-03T11:50:23.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.637+0000 7f2c4f7fe700 1 -- 172.21.15.102:0/51675241 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2c5c017090 con 0x7f2c680cd8e0 2023-03-03T11:50:23.639 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:50:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:50:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c540682f0 msgr2=0x7f2c5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c540682f0 0x7f2c5406a7a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2c58007fd0 tx=0x7f2c58007de0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c680cd8e0 msgr2=0x7f2c68155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c680cd8e0 0x7f2c68155360 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00c410 tx=0x7f2c5c00c720 comp rx=0 tx=0).stop 2023-03-03T11:50:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 shutdown_connections 2023-03-03T11:50:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c540682f0 0x7f2c5406a7a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2c68153290 0x7f2c681558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c680cd390 0x7f2c681579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 --2- 172.21.15.102:0/51675241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c680cd8e0 0x7f2c68155360 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 >> 172.21.15.102:0/51675241 conn(0x7f2c68065fb0 msgr2=0x7f2c681512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.639+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 shutdown_connections 2023-03-03T11:50:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:50:23.640+0000 7f2c6d1ff700 1 -- 172.21.15.102:0/51675241 wait complete. 2023-03-03T11:50:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:50:23.668 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 90 secs before thrashing 2023-03-03T11:50:43.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:43.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:14.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:14.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:44.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:44.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:53.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:51:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.918+0000 7f0f1759e700 1 Processor -- start 2023-03-03T11:51:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.918+0000 7f0f1759e700 1 -- start start 2023-03-03T11:51:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f1759e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f180b3970 0x7f0f180c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f1759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180c41b0 0x7f0f180cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f1759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f180c3d20 con 0x7f0f180ced00 2023-03-03T11:51:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f1759e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f180c4930 con 0x7f0f180b3970 2023-03-03T11:51:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f1759e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180cf610 con 0x7f0f180c41b0 2023-03-03T11:51:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f15d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f180b3970 0x7f0f180c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f15d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f180b3970 0x7f0f180c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41540/0 (socket says 172.21.15.102:41540) 2023-03-03T11:51:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f1659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0f180ced00 0x7f0f18158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48910/0 (socket says 172.21.15.102:48910) 2023-03-03T11:51:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.920+0000 7f0f15d9b700 1 -- 172.21.15.102:0/531404647 learned_addr learned my addr 172.21.15.102:0/531404647 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:51:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.921+0000 7f0f16d9d700 1 --2- 172.21.15.102:0/531404647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180c41b0 0x7f0f180cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.921+0000 7f0f15d9b700 1 -- 172.21.15.102:0/531404647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180c41b0 msgr2=0x7f0f180cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:51:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.921+0000 7f0f15d9b700 1 --2- 172.21.15.102:0/531404647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180c41b0 0x7f0f180cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.921+0000 7f0f15d9b700 1 -- 172.21.15.102:0/531404647 >> v1:172.21.15.102:6789/0 conn(0x7f0f180ced00 legacy=0x7f0f18158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.921+0000 7f0f15d9b700 1 -- 172.21.15.102:0/531404647 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1815cba0 con 0x7f0f180b3970 2023-03-03T11:51:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.921+0000 7f0f1559a700 1 -- 172.21.15.102:0/531404647 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307777211 0 0) 0x7f0f180c3d20 con 0x7f0f180ced00 2023-03-03T11:51:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.921+0000 7f0f15d9b700 1 --2- 172.21.15.102:0/531404647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 0x7f0f180c5a90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009920 tx=0x7f0f0c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b79ca2843e3058 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:51:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1559a700 1 -- 172.21.15.102:0/531404647 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c011830 con 0x7f0f180b3970 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1559a700 1 -- 172.21.15.102:0/531404647 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c011990 con 0x7f0f180b3970 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1559a700 1 -- 172.21.15.102:0/531404647 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c011d20 con 0x7f0f180b3970 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 -- 172.21.15.102:0/531404647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 msgr2=0x7f0f180c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 --2- 172.21.15.102:0/531404647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 0x7f0f180c5a90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009920 tx=0x7f0f0c009c30 comp rx=0 tx=0).stop 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 -- 172.21.15.102:0/531404647 shutdown_connections 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 --2- 172.21.15.102:0/531404647 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180c41b0 0x7f0f180cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 --2- 172.21.15.102:0/531404647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 0x7f0f180c5a90 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:53.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 -- 172.21.15.102:0/531404647 >> 172.21.15.102:0/531404647 conn(0x7f0f1808b1c0 msgr2=0x7f0f18071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:51:53.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 -- 172.21.15.102:0/531404647 shutdown_connections 2023-03-03T11:51:53.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.922+0000 7f0f1759e700 1 -- 172.21.15.102:0/531404647 wait complete. 2023-03-03T11:51:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.925+0000 7f0f1759e700 1 Processor -- start 2023-03-03T11:51:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.925+0000 7f0f1759e700 1 -- start start 2023-03-03T11:51:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f1759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 0x7f0f180c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f1759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f180c41b0 0x7f0f180c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f1759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180ced00 0x7f0f181ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f1759e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180c8c10 con 0x7f0f180b3970 2023-03-03T11:51:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f1759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180c8f90 con 0x7f0f180c41b0 2023-03-03T11:51:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f1759e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18071c50 con 0x7f0f180ced00 2023-03-03T11:51:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f1659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 0x7f0f180c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f15d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f180c41b0 0x7f0f180c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f16d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180ced00 0x7f0f181ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f15d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f180c41b0 0x7f0f180c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52736/0 (socket says 172.21.15.102:52736) 2023-03-03T11:51:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.926+0000 7f0f15d9b700 1 -- 172.21.15.102:0/2101830217 learned_addr learned my addr 172.21.15.102:0/2101830217 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:51:53.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f15d9b700 1 -- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180ced00 msgr2=0x7f0f181ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f15d9b700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180ced00 0x7f0f181ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f15d9b700 1 -- 172.21.15.102:0/2101830217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 msgr2=0x7f0f180c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f15d9b700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 0x7f0f180c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f15d9b700 1 -- 172.21.15.102:0/2101830217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c009550 con 0x7f0f180c41b0 2023-03-03T11:51:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f16d9d700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180ced00 0x7f0f181ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:51:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f15d9b700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f180c41b0 0x7f0f180c62e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c014040 tx=0x7f0f0c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:51:53.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f06ffd700 1 -- 172.21.15.102:0/2101830217 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c004450 con 0x7f0f180c41b0 2023-03-03T11:51:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f06ffd700 1 -- 172.21.15.102:0/2101830217 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c01e070 con 0x7f0f180c41b0 2023-03-03T11:51:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f06ffd700 1 -- 172.21.15.102:0/2101830217 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c0038e0 con 0x7f0f180c41b0 2023-03-03T11:51:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.927+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1806bf20 con 0x7f0f180c41b0 2023-03-03T11:51:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.928+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1815ce80 con 0x7f0f180c41b0 2023-03-03T11:51:53.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.929+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1806c200 con 0x7f0f180c41b0 2023-03-03T11:51:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.930+0000 7f0f06ffd700 1 -- 172.21.15.102:0/2101830217 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f0c00f050 con 0x7f0f180c41b0 2023-03-03T11:51:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.931+0000 7f0f06ffd700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0ef80682f0 0x7f0ef806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.931+0000 7f0f06ffd700 1 -- 172.21.15.102:0/2101830217 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0f0c086cf0 con 0x7f0f180c41b0 2023-03-03T11:51:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.931+0000 7f0f1659c700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0ef80682f0 0x7f0ef806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:53.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.932+0000 7f0f1659c700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0ef80682f0 0x7f0ef806a7a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007fd0 tx=0x7f0f08007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:51:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:53.935+0000 7f0f06ffd700 1 -- 172.21.15.102:0/2101830217 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f0c053400 con 0x7f0f180c41b0 2023-03-03T11:51:54.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.167+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f181ef660 con 0x7f0f180c41b0 2023-03-03T11:51:54.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.168+0000 7f0f06ffd700 1 -- 172.21.15.102:0/2101830217 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0f181ef660 con 0x7f0f180c41b0 2023-03-03T11:51:54.170 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:51:54.170 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:51:54.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0ef80682f0 msgr2=0x7f0ef806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0ef80682f0 0x7f0ef806a7a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007fd0 tx=0x7f0f08007de0 comp rx=0 tx=0).stop 2023-03-03T11:51:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f180c41b0 msgr2=0x7f0f180c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f180c41b0 0x7f0f180c62e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c014040 tx=0x7f0f0c004750 comp rx=0 tx=0).stop 2023-03-03T11:51:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 shutdown_connections 2023-03-03T11:51:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0ef80682f0 0x7f0ef806a7a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f180ced00 0x7f0f181ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f180b3970 0x7f0f180c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 --2- 172.21.15.102:0/2101830217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f180c41b0 0x7f0f180c62e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 >> 172.21.15.102:0/2101830217 conn(0x7f0f1808b1c0 msgr2=0x7f0f180ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:51:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 shutdown_connections 2023-03-03T11:51:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:51:54.171+0000 7f0f1759e700 1 -- 172.21.15.102:0/2101830217 wait complete. 2023-03-03T11:51:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:51:54.199 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T11:52:14.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:14.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:44.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:44.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:04.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:53:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.462+0000 7f4b94fff700 1 Processor -- start 2023-03-03T11:53:04.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.463+0000 7f4b94fff700 1 -- start start 2023-03-03T11:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.464+0000 7f4b94fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4b90153290 0x7f4b90158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.464+0000 7f4b94fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b901593c0 0x7f4b900cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.464+0000 7f4b94fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b900cd8e0 0x7f4b900ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.464+0000 7f4b94fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b90153660 con 0x7f4b901593c0 2023-03-03T11:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.464+0000 7f4b94fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9006c360 con 0x7f4b90153290 2023-03-03T11:53:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.464+0000 7f4b94fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9015cea0 con 0x7f4b900cd8e0 2023-03-03T11:53:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.464+0000 7f4b8dd9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b901593c0 0x7f4b900cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b901593c0 0x7f4b900cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57236/0 (socket says 172.21.15.102:57236) 2023-03-03T11:53:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 -- 172.21.15.102:0/3758530676 learned_addr learned my addr 172.21.15.102:0/3758530676 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:53:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8ed9d700 1 --2- 172.21.15.102:0/3758530676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b900cd8e0 0x7f4b900ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8e59c700 1 --2- 172.21.15.102:0/3758530676 >> v2:172.21.15.172:3300/0 conn(0x7f4b90153290 0x7f4b90158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 -- 172.21.15.102:0/3758530676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b900cd8e0 msgr2=0x7f4b900ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 --2- 172.21.15.102:0/3758530676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b900cd8e0 0x7f4b900ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 -- 172.21.15.102:0/3758530676 >> v2:172.21.15.172:3300/0 conn(0x7f4b90153290 msgr2=0x7f4b90158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 --2- 172.21.15.102:0/3758530676 >> v2:172.21.15.172:3300/0 conn(0x7f4b90153290 0x7f4b90158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 -- 172.21.15.102:0/3758530676 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9006c7f0 con 0x7f4b901593c0 2023-03-03T11:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8e59c700 1 --2- 172.21.15.102:0/3758530676 >> v2:172.21.15.172:3300/0 conn(0x7f4b90153290 0x7f4b90158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.465+0000 7f4b8dd9b700 1 --2- 172.21.15.102:0/3758530676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b900cde20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4b9008cc70 tx=0x7f4b84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec64ae31da4d3f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b8d59a700 1 -- 172.21.15.102:0/3758530676 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b84011830 con 0x7f4b901593c0 2023-03-03T11:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b8d59a700 1 -- 172.21.15.102:0/3758530676 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b84011990 con 0x7f4b901593c0 2023-03-03T11:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b8d59a700 1 -- 172.21.15.102:0/3758530676 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b84011d20 con 0x7f4b901593c0 2023-03-03T11:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b94fff700 1 -- 172.21.15.102:0/3758530676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 msgr2=0x7f4b900cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3758530676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b900cde20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4b9008cc70 tx=0x7f4b84009bd0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b94fff700 1 -- 172.21.15.102:0/3758530676 shutdown_connections 2023-03-03T11:53:04.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3758530676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b900cd8e0 0x7f4b900ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3758530676 >> v2:172.21.15.172:3300/0 conn(0x7f4b90153290 0x7f4b90158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3758530676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b900cde20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.466+0000 7f4b94fff700 1 -- 172.21.15.102:0/3758530676 >> 172.21.15.102:0/3758530676 conn(0x7f4b90065fb0 msgr2=0x7f4b900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:53:04.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.467+0000 7f4b94fff700 1 -- 172.21.15.102:0/3758530676 shutdown_connections 2023-03-03T11:53:04.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.467+0000 7f4b94fff700 1 -- 172.21.15.102:0/3758530676 wait complete. 2023-03-03T11:53:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.469+0000 7f4b94fff700 1 Processor -- start 2023-03-03T11:53:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.469+0000 7f4b94fff700 1 -- start start 2023-03-03T11:53:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b94fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b900cd8e0 0x7f4b9014f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b94fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b90153290 0x7f4b901f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b94fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b901f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b94fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90156560 con 0x7f4b901593c0 2023-03-03T11:53:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b94fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b901f2c60 con 0x7f4b900cd8e0 2023-03-03T11:53:04.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b94fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b901e2c70 con 0x7f4b90153290 2023-03-03T11:53:04.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b8ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b901f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:04.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b8e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b900cd8e0 0x7f4b9014f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:04.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b8e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b900cd8e0 0x7f4b9014f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46830/0 (socket says 172.21.15.102:46830) 2023-03-03T11:53:04.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b8e59c700 1 -- 172.21.15.102:0/3410646468 learned_addr learned my addr 172.21.15.102:0/3410646468 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:53:04.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.470+0000 7f4b8dd9b700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b90153290 0x7f4b901f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:04.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b8ed9d700 1 -- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b90153290 msgr2=0x7f4b901f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:04.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b8ed9d700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b90153290 0x7f4b901f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b8ed9d700 1 -- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b900cd8e0 msgr2=0x7f4b9014f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:04.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b8ed9d700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b900cd8e0 0x7f4b9014f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b8ed9d700 1 -- 172.21.15.102:0/3410646468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b84009550 con 0x7f4b901593c0 2023-03-03T11:53:04.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b8e59c700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b900cd8e0 0x7f4b9014f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:53:04.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b8ed9d700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b901f25c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800eab0 tx=0x7f4b8800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b7effd700 1 -- 172.21.15.102:0/3410646468 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88010040 con 0x7f4b901593c0 2023-03-03T11:53:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b7effd700 1 -- 172.21.15.102:0/3410646468 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8800b970 con 0x7f4b901593c0 2023-03-03T11:53:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b7effd700 1 -- 172.21.15.102:0/3410646468 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88009a80 con 0x7f4b901593c0 2023-03-03T11:53:04.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.471+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b900ccde0 con 0x7f4b901593c0 2023-03-03T11:53:04.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.472+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9015d160 con 0x7f4b901593c0 2023-03-03T11:53:04.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.473+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b900cd0c0 con 0x7f4b901593c0 2023-03-03T11:53:04.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.474+0000 7f4b7effd700 1 -- 172.21.15.102:0/3410646468 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b8800bd20 con 0x7f4b901593c0 2023-03-03T11:53:04.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.476+0000 7f4b7effd700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b700682f0 0x7f4b7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:04.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.476+0000 7f4b7effd700 1 -- 172.21.15.102:0/3410646468 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4b88086130 con 0x7f4b901593c0 2023-03-03T11:53:04.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.476+0000 7f4b8e59c700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b700682f0 0x7f4b7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:04.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.477+0000 7f4b8e59c700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b700682f0 0x7f4b7006a7a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009ea0 tx=0x7f4b80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:04.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.480+0000 7f4b7effd700 1 -- 172.21.15.102:0/3410646468 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b88052840 con 0x7f4b901593c0 2023-03-03T11:53:04.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.719+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b901573b0 con 0x7f4b901593c0 2023-03-03T11:53:04.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.720+0000 7f4b7effd700 1 -- 172.21.15.102:0/3410646468 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4b88055e60 con 0x7f4b901593c0 2023-03-03T11:53:04.723 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:53:04.724 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:53:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b700682f0 msgr2=0x7f4b7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b700682f0 0x7f4b7006a7a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009ea0 tx=0x7f4b80008040 comp rx=0 tx=0).stop 2023-03-03T11:53:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 msgr2=0x7f4b901f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b901f25c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800eab0 tx=0x7f4b8800edc0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 shutdown_connections 2023-03-03T11:53:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b901593c0 0x7f4b901f25c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b900cd8e0 0x7f4b9014f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b700682f0 0x7f4b7006a7a0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 --2- 172.21.15.102:0/3410646468 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b90153290 0x7f4b901f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:04.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 >> 172.21.15.102:0/3410646468 conn(0x7f4b90065fb0 msgr2=0x7f4b9015ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:53:04.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 shutdown_connections 2023-03-03T11:53:04.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:04.723+0000 7f4b94fff700 1 -- 172.21.15.102:0/3410646468 wait complete. 2023-03-03T11:53:04.728 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:53:04.755 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2023-03-03T11:53:14.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:14.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:17.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:53:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.005+0000 7f27e49ff700 1 Processor -- start 2023-03-03T11:53:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.005+0000 7f27e49ff700 1 -- start start 2023-03-03T11:53:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.007+0000 7f27e49ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f27e014fbe0 0x7f27e00c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.007+0000 7f27e49ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f27e014ffb0 0x7f27e00c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.007+0000 7f27e49ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e00c3c80 con 0x7f27e00716c0 2023-03-03T11:53:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27e49ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e00c41a0 con 0x7f27e014fbe0 2023-03-03T11:53:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27e49ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e0071cd0 con 0x7f27e014ffb0 2023-03-03T11:53:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27ddd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f27e014ffb0 0x7f27e00c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:18.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27de59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f27e014fbe0 0x7f27e00c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:18.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27ded9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f27e00716c0 0x7f27e00706f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52418/0 (socket says 172.21.15.102:52418) 2023-03-03T11:53:18.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27ded9d700 1 -- 172.21.15.102:0/419462009 learned_addr learned my addr 172.21.15.102:0/419462009 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:53:18.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27dd59a700 1 -- 172.21.15.102:0/419462009 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261300963 0 0) 0x7f27d8001490 con 0x7f27e00716c0 2023-03-03T11:53:18.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27ddd9b700 1 -- 172.21.15.102:0/419462009 >> v2:172.21.15.102:3300/0 conn(0x7f27e014fbe0 msgr2=0x7f27e00c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:53:18.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27ddd9b700 1 --2- 172.21.15.102:0/419462009 >> v2:172.21.15.102:3300/0 conn(0x7f27e014fbe0 0x7f27e00c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27ddd9b700 1 -- 172.21.15.102:0/419462009 >> v1:172.21.15.102:6789/0 conn(0x7f27e00716c0 legacy=0x7f27e00706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27ddd9b700 1 -- 172.21.15.102:0/419462009 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e00c3ec0 con 0x7f27e014ffb0 2023-03-03T11:53:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.008+0000 7f27de59c700 1 --2- 172.21.15.102:0/419462009 >> v2:172.21.15.102:3300/0 conn(0x7f27e014fbe0 0x7f27e00c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:53:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.009+0000 7f27ddd9b700 1 --2- 172.21.15.102:0/419462009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e00c45a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f27e00c2850 tx=0x7f27d4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2b483f987b9971e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.009+0000 7f27dd59a700 1 -- 172.21.15.102:0/419462009 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4011830 con 0x7f27e014ffb0 2023-03-03T11:53:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.009+0000 7f27dd59a700 1 -- 172.21.15.102:0/419462009 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d4011990 con 0x7f27e014ffb0 2023-03-03T11:53:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.009+0000 7f27dd59a700 1 -- 172.21.15.102:0/419462009 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4011d20 con 0x7f27e014ffb0 2023-03-03T11:53:18.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.009+0000 7f27e49ff700 1 -- 172.21.15.102:0/419462009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 msgr2=0x7f27e00c45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:18.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.009+0000 7f27e49ff700 1 --2- 172.21.15.102:0/419462009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e00c45a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f27e00c2850 tx=0x7f27d4009b20 comp rx=0 tx=0).stop 2023-03-03T11:53:18.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.010+0000 7f27e49ff700 1 -- 172.21.15.102:0/419462009 shutdown_connections 2023-03-03T11:53:18.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.010+0000 7f27e49ff700 1 --2- 172.21.15.102:0/419462009 >> v2:172.21.15.102:3300/0 conn(0x7f27e014fbe0 0x7f27e00c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.010+0000 7f27e49ff700 1 --2- 172.21.15.102:0/419462009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e00c45a0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.010+0000 7f27e49ff700 1 -- 172.21.15.102:0/419462009 >> 172.21.15.102:0/419462009 conn(0x7f27e006af90 msgr2=0x7f27e006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:53:18.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.010+0000 7f27e49ff700 1 -- 172.21.15.102:0/419462009 shutdown_connections 2023-03-03T11:53:18.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.010+0000 7f27e49ff700 1 -- 172.21.15.102:0/419462009 wait complete. 2023-03-03T11:53:18.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.012+0000 7f27e49ff700 1 Processor -- start 2023-03-03T11:53:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.012+0000 7f27e49ff700 1 -- start start 2023-03-03T11:53:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27e49ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27e00716c0 0x7f27e00be070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27e49ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e014fbe0 0x7f27e00bc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27e49ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e014e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27e49ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00bc290 con 0x7f27e014fbe0 2023-03-03T11:53:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27e49ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e01f1db0 con 0x7f27e014ffb0 2023-03-03T11:53:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27e49ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00bcbf0 con 0x7f27e00716c0 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27ddd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e014fbe0 0x7f27e00bc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27ddd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e014fbe0 0x7f27e00bc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37066/0 (socket says 172.21.15.102:37066) 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27ddd9b700 1 -- 172.21.15.102:0/3658057170 learned_addr learned my addr 172.21.15.102:0/3658057170 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27de59c700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27e00716c0 0x7f27e00be070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.013+0000 7f27ded9d700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e014e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ddd9b700 1 -- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27e00716c0 msgr2=0x7f27e00be070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ddd9b700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27e00716c0 0x7f27e00be070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ddd9b700 1 -- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 msgr2=0x7f27e014e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:18.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ddd9b700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e014e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ddd9b700 1 -- 172.21.15.102:0/3658057170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d00094a0 con 0x7f27e014fbe0 2023-03-03T11:53:18.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ded9d700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e014e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:53:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ddd9b700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e014fbe0 0x7f27e00bc6b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f27d40090b0 tx=0x7f27d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ceffd700 1 -- 172.21.15.102:0/3658057170 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d40036a0 con 0x7f27e014fbe0 2023-03-03T11:53:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ceffd700 1 -- 172.21.15.102:0/3658057170 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d401e070 con 0x7f27e014fbe0 2023-03-03T11:53:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27ceffd700 1 -- 172.21.15.102:0/3658057170 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4003a20 con 0x7f27e014fbe0 2023-03-03T11:53:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.014+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d40094a0 con 0x7f27e014fbe0 2023-03-03T11:53:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.015+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e0072250 con 0x7f27e014fbe0 2023-03-03T11:53:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.016+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e0071f70 con 0x7f27e014fbe0 2023-03-03T11:53:18.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.017+0000 7f27ceffd700 1 -- 172.21.15.102:0/3658057170 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f27d400f050 con 0x7f27e014fbe0 2023-03-03T11:53:18.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.018+0000 7f27ceffd700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f27c0068320 0x7f27c006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:18.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.018+0000 7f27ceffd700 1 -- 172.21.15.102:0/3658057170 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f27d4086db0 con 0x7f27e014fbe0 2023-03-03T11:53:18.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.019+0000 7f27de59c700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f27c0068320 0x7f27c006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:18.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.019+0000 7f27de59c700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f27c0068320 0x7f27c006a7d0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009d70 tx=0x7f27d0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:18.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.023+0000 7f27ceffd700 1 -- 172.21.15.102:0/3658057170 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f27d408b070 con 0x7f27e014fbe0 2023-03-03T11:53:18.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.257+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27e0072530 con 0x7f27e014fbe0 2023-03-03T11:53:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.258+0000 7f27ceffd700 1 -- 172.21.15.102:0/3658057170 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f27d40534c0 con 0x7f27e014fbe0 2023-03-03T11:53:18.261 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:53:18.261 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:53:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f27c0068320 msgr2=0x7f27c006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:18.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f27c0068320 0x7f27c006a7d0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009d70 tx=0x7f27d0009210 comp rx=0 tx=0).stop 2023-03-03T11:53:18.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e014fbe0 msgr2=0x7f27e00bc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:18.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e014fbe0 0x7f27e00bc6b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f27d40090b0 tx=0x7f27d4004750 comp rx=0 tx=0).stop 2023-03-03T11:53:18.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 shutdown_connections 2023-03-03T11:53:18.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f27c0068320 0x7f27c006a7d0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f27e00716c0 0x7f27e00be070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27e014ffb0 0x7f27e014e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 --2- 172.21.15.102:0/3658057170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e014fbe0 0x7f27e00bc6b0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:18.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 >> 172.21.15.102:0/3658057170 conn(0x7f27e006af90 msgr2=0x7f27e00caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:53:18.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 shutdown_connections 2023-03-03T11:53:18.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:53:18.261+0000 7f27e49ff700 1 -- 172.21.15.102:0/3658057170 wait complete. 2023-03-03T11:53:18.265 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:53:18.291 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T11:53:44.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:44.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:14.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:14.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:45.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:45.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:13.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:55:13.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.540+0000 7fc60ed9d700 1 Processor -- start 2023-03-03T11:55:13.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.540+0000 7fc60ed9d700 1 -- start start 2023-03-03T11:55:13.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60ed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6081564b0 0x7fc6080cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:55:13.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60ed9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6080cab50 0x7fc6080cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:55:13.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60ed9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6080c9ca0 con 0x7fc6080c9610 2023-03-03T11:55:13.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60ed9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6080b4920 con 0x7fc6080cab50 2023-03-03T11:55:13.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60ed9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6080b4c00 con 0x7fc6081564b0 2023-03-03T11:55:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6080cab50 0x7fc6080cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:55:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6081564b0 0x7fc6080cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:55:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60d59a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc6080c9610 0x7fc6080ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34942/0 (socket says 172.21.15.102:34942) 2023-03-03T11:55:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.542+0000 7fc60d59a700 1 -- 172.21.15.102:0/3705895787 learned_addr learned my addr 172.21.15.102:0/3705895787 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:55:13.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60e59c700 1 -- 172.21.15.102:0/3705895787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6081564b0 msgr2=0x7fc6080cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:55:13.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60e59c700 1 --2- 172.21.15.102:0/3705895787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6081564b0 0x7fc6080cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60e59c700 1 -- 172.21.15.102:0/3705895787 >> v1:172.21.15.172:6789/0 conn(0x7fc6080c9610 legacy=0x7fc6080ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:55:13.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60e59c700 1 -- 172.21.15.102:0/3705895787 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6080b46f0 con 0x7fc6080cab50 2023-03-03T11:55:13.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60e59c700 1 --2- 172.21.15.102:0/3705895787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6080cab50 0x7fc6080cbba0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc600009d90 tx=0x7fc60000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49092f3acfd3e7ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:55:13.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60cd99700 1 -- 172.21.15.102:0/3705895787 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60000e070 con 0x7fc6080cab50 2023-03-03T11:55:13.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60cd99700 1 -- 172.21.15.102:0/3705895787 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60000bb20 con 0x7fc6080cab50 2023-03-03T11:55:13.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.543+0000 7fc60cd99700 1 -- 172.21.15.102:0/3705895787 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600012430 con 0x7fc6080cab50 2023-03-03T11:55:13.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 -- 172.21.15.102:0/3705895787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6080cab50 msgr2=0x7fc6080cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:55:13.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/3705895787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6080cab50 0x7fc6080cbba0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc600009d90 tx=0x7fc60000b330 comp rx=0 tx=0).stop 2023-03-03T11:55:13.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 -- 172.21.15.102:0/3705895787 shutdown_connections 2023-03-03T11:55:13.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/3705895787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6080cab50 0x7fc6080cbba0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/3705895787 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6081564b0 0x7fc6080cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 -- 172.21.15.102:0/3705895787 >> 172.21.15.102:0/3705895787 conn(0x7fc608065fb0 msgr2=0x7fc6080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:55:13.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 -- 172.21.15.102:0/3705895787 shutdown_connections 2023-03-03T11:55:13.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.544+0000 7fc60ed9d700 1 -- 172.21.15.102:0/3705895787 wait complete. 2023-03-03T11:55:13.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.546+0000 7fc60ed9d700 1 Processor -- start 2023-03-03T11:55:13.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.546+0000 7fc60ed9d700 1 -- start start 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.547+0000 7fc60ed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6080c9610 0x7fc6081ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.547+0000 7fc60ed9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6080cab50 0x7fc60815cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.547+0000 7fc60ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6081564b0 0x7fc608159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.547+0000 7fc60ed9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6081ef5e0 con 0x7fc6081564b0 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.547+0000 7fc60ed9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6080c7b10 con 0x7fc6080cab50 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.547+0000 7fc60ed9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6080c7270 con 0x7fc6080c9610 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6080c9610 0x7fc6081ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:55:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6081564b0 0x7fc608159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:55:13.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60d59a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6080cab50 0x7fc60815cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:55:13.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6080c9610 0x7fc6081ef0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:56968/0 (socket says 172.21.15.102:56968) 2023-03-03T11:55:13.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 -- 172.21.15.102:0/145311800 learned_addr learned my addr 172.21.15.102:0/145311800 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:55:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 -- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6080cab50 msgr2=0x7fc60815cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:55:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6080cab50 0x7fc60815cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 -- 172.21.15.102:0/145311800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6081564b0 msgr2=0x7fc608159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:55:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6081564b0 0x7fc608159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 -- 172.21.15.102:0/145311800 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc600009a70 con 0x7fc6080c9610 2023-03-03T11:55:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.548+0000 7fc60dd9b700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6080c9610 0x7fc6081ef0a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc60400e870 tx=0x7fc60400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:55:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.549+0000 7fc5fe7fc700 1 -- 172.21.15.102:0/145311800 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604009590 con 0x7fc6080c9610 2023-03-03T11:55:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.549+0000 7fc5fe7fc700 1 -- 172.21.15.102:0/145311800 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc604013680 con 0x7fc6080c9610 2023-03-03T11:55:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.549+0000 7fc5fe7fc700 1 -- 172.21.15.102:0/145311800 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6040139f0 con 0x7fc6080c9610 2023-03-03T11:55:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.549+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6080b2210 con 0x7fc6080c9610 2023-03-03T11:55:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.549+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6080b1f30 con 0x7fc6080c9610 2023-03-03T11:55:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.550+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc608156880 con 0x7fc6080c9610 2023-03-03T11:55:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.551+0000 7fc5fe7fc700 1 -- 172.21.15.102:0/145311800 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc604023070 con 0x7fc6080c9610 2023-03-03T11:55:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.552+0000 7fc5fe7fc700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5ec0682f0 0x7fc5ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:55:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.552+0000 7fc5fe7fc700 1 -- 172.21.15.102:0/145311800 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc604085e80 con 0x7fc6080c9610 2023-03-03T11:55:13.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.553+0000 7fc60d59a700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5ec0682f0 0x7fc5ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:55:13.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.553+0000 7fc60d59a700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5ec0682f0 0x7fc5ec06a7a0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc6081554b0 tx=0x7fc5f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:55:13.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.556+0000 7fc5fe7fc700 1 -- 172.21.15.102:0/145311800 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc604052530 con 0x7fc6080c9610 2023-03-03T11:55:13.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.789+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6080677f0 con 0x7fc6080c9610 2023-03-03T11:55:13.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.790+0000 7fc5fe7fc700 1 -- 172.21.15.102:0/145311800 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc604055b50 con 0x7fc6080c9610 2023-03-03T11:55:13.792 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:55:13.792 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:55:13.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5ec0682f0 msgr2=0x7fc5ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:55:13.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5ec0682f0 0x7fc5ec06a7a0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc6081554b0 tx=0x7fc5f8009040 comp rx=0 tx=0).stop 2023-03-03T11:55:13.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6080c9610 msgr2=0x7fc6081ef0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:55:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6080c9610 0x7fc6081ef0a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc60400e870 tx=0x7fc60400eb80 comp rx=0 tx=0).stop 2023-03-03T11:55:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 shutdown_connections 2023-03-03T11:55:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5ec0682f0 0x7fc5ec06a7a0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6080c9610 0x7fc6081ef0a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6081564b0 0x7fc608159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 --2- 172.21.15.102:0/145311800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6080cab50 0x7fc60815cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:55:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.792+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 >> 172.21.15.102:0/145311800 conn(0x7fc608065fb0 msgr2=0x7fc608155760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:55:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.793+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 shutdown_connections 2023-03-03T11:55:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:55:13.793+0000 7fc60ed9d700 1 -- 172.21.15.102:0/145311800 wait complete. 2023-03-03T11:55:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:55:13.821 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T11:55:15.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:15.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:45.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:45.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:11.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:56:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.072+0000 7fcd681ff700 1 Processor -- start 2023-03-03T11:56:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.072+0000 7fcd681ff700 1 -- start start 2023-03-03T11:56:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.073+0000 7fcd681ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcd600cd8e0 0x7fcd600cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.073+0000 7fcd681ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.073+0000 7fcd681ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd60153660 con 0x7fcd60153290 2023-03-03T11:56:12.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.073+0000 7fcd681ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6006c360 con 0x7fcd600cd8e0 2023-03-03T11:56:12.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.073+0000 7fcd681ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6015cea0 con 0x7fcd600cd390 2023-03-03T11:56:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcd600cd8e0 0x7fcd600cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6679c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:12.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcd600cd8e0 0x7fcd600cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43418/0 (socket says 172.21.15.102:43418) 2023-03-03T11:56:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 -- 172.21.15.102:0/1016666191 learned_addr learned my addr 172.21.15.102:0/1016666191 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:56:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd65f9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcd60153290 0x7fcd60158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:39006/0 (socket says 172.21.15.102:39006) 2023-03-03T11:56:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 -- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 msgr2=0x7fcd600ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:56:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 --2- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 -- 172.21.15.102:0/1016666191 >> v1:172.21.15.172:6789/0 conn(0x7fcd60153290 legacy=0x7fcd60158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 -- 172.21.15.102:0/1016666191 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6006c7f0 con 0x7fcd600cd8e0 2023-03-03T11:56:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6679c700 1 --2- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:56:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd64f99700 1 -- 172.21.15.102:0/1016666191 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403943091 0 0) 0x7fcd60153660 con 0x7fcd60153290 2023-03-03T11:56:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.074+0000 7fcd6579a700 1 --2- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600cde20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcd500098f0 tx=0x7fcd50009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e20c9260c68d6b66 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd64f99700 1 -- 172.21.15.102:0/1016666191 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd50011830 con 0x7fcd600cd8e0 2023-03-03T11:56:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd64f99700 1 -- 172.21.15.102:0/1016666191 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd50011990 con 0x7fcd600cd8e0 2023-03-03T11:56:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd64f99700 1 -- 172.21.15.102:0/1016666191 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd50011d20 con 0x7fcd600cd8e0 2023-03-03T11:56:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd681ff700 1 -- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 msgr2=0x7fcd600cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:12.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd681ff700 1 --2- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600cde20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcd500098f0 tx=0x7fcd50009c00 comp rx=0 tx=0).stop 2023-03-03T11:56:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd681ff700 1 -- 172.21.15.102:0/1016666191 shutdown_connections 2023-03-03T11:56:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd681ff700 1 --2- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd681ff700 1 --2- 172.21.15.102:0/1016666191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600cde20 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd681ff700 1 -- 172.21.15.102:0/1016666191 >> 172.21.15.102:0/1016666191 conn(0x7fcd60065fb0 msgr2=0x7fcd600663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.075+0000 7fcd681ff700 1 -- 172.21.15.102:0/1016666191 shutdown_connections 2023-03-03T11:56:12.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.076+0000 7fcd681ff700 1 -- 172.21.15.102:0/1016666191 wait complete. 2023-03-03T11:56:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.078+0000 7fcd681ff700 1 Processor -- start 2023-03-03T11:56:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.078+0000 7fcd681ff700 1 -- start start 2023-03-03T11:56:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd681ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd681ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd681ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd60153290 0x7fcd600b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd681ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600c6630 con 0x7fcd60153290 2023-03-03T11:56:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd681ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600c6140 con 0x7fcd600cd8e0 2023-03-03T11:56:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd681ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600b4180 con 0x7fcd600cd390 2023-03-03T11:56:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd6579a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd65f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd6579a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600b34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43432/0 (socket says 172.21.15.102:43432) 2023-03-03T11:56:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd6579a700 1 -- 172.21.15.102:0/2704323943 learned_addr learned my addr 172.21.15.102:0/2704323943 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:56:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.079+0000 7fcd6679c700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd60153290 0x7fcd600b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd6579a700 1 -- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 msgr2=0x7fcd600b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd6579a700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd6579a700 1 -- 172.21.15.102:0/2704323943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd60153290 msgr2=0x7fcd600b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd6579a700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd60153290 0x7fcd600b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd6579a700 1 -- 172.21.15.102:0/2704323943 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd58009a70 con 0x7fcd600cd8e0 2023-03-03T11:56:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd6579a700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600b34b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009c40 tx=0x7fcd50004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd567fc700 1 -- 172.21.15.102:0/2704323943 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500044d0 con 0x7fcd600cd8e0 2023-03-03T11:56:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd567fc700 1 -- 172.21.15.102:0/2704323943 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5001e070 con 0x7fcd600cd8e0 2023-03-03T11:56:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd50009550 con 0x7fcd600cd8e0 2023-03-03T11:56:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.080+0000 7fcd567fc700 1 -- 172.21.15.102:0/2704323943 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500038c0 con 0x7fcd600cd8e0 2023-03-03T11:56:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.081+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd600ccde0 con 0x7fcd600cd8e0 2023-03-03T11:56:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.082+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6015d160 con 0x7fcd600cd8e0 2023-03-03T11:56:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.083+0000 7fcd567fc700 1 -- 172.21.15.102:0/2704323943 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd5002a020 con 0x7fcd600cd8e0 2023-03-03T11:56:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.084+0000 7fcd567fc700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd44068320 0x7fcd4406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.084+0000 7fcd567fc700 1 -- 172.21.15.102:0/2704323943 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcd50086c50 con 0x7fcd600cd8e0 2023-03-03T11:56:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.084+0000 7fcd65f9b700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd44068320 0x7fcd4406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.085+0000 7fcd65f9b700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd44068320 0x7fcd4406a7d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c007fd0 tx=0x7fcd5c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.089+0000 7fcd567fc700 1 -- 172.21.15.102:0/2704323943 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcd50053360 con 0x7fcd600cd8e0 2023-03-03T11:56:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.320+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd600cd0c0 con 0x7fcd600cd8e0 2023-03-03T11:56:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.322+0000 7fcd567fc700 1 -- 172.21.15.102:0/2704323943 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcd50056980 con 0x7fcd600cd8e0 2023-03-03T11:56:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:56:12.324 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:56:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd44068320 msgr2=0x7fcd4406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd44068320 0x7fcd4406a7d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c007fd0 tx=0x7fcd5c007de0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 msgr2=0x7fcd600b34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600b34b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009c40 tx=0x7fcd50004750 comp rx=0 tx=0).stop 2023-03-03T11:56:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 shutdown_connections 2023-03-03T11:56:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcd44068320 0x7fcd4406a7d0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcd600cd390 0x7fcd600b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd60153290 0x7fcd600b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 --2- 172.21.15.102:0/2704323943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd600cd8e0 0x7fcd600b34b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 >> 172.21.15.102:0/2704323943 conn(0x7fcd60065fb0 msgr2=0x7fcd600cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 shutdown_connections 2023-03-03T11:56:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:56:12.324+0000 7fcd681ff700 1 -- 172.21.15.102:0/2704323943 wait complete. 2023-03-03T11:56:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:56:12.352 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T11:56:15.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:15.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:45.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:45.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:16.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:16.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:46.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:46.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:07.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:58:07.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.603+0000 7f076abff700 1 Processor -- start 2023-03-03T11:58:07.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.603+0000 7f076abff700 1 -- start start 2023-03-03T11:58:07.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076abff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07640cd8e0 0x7f07640cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:07.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076abff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd390 0x7f07640ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:07.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076abff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0764153660 con 0x7f0764153290 2023-03-03T11:58:07.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076abff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076406c360 con 0x7f07640cd8e0 2023-03-03T11:58:07.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076abff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076415cea0 con 0x7f07640cd390 2023-03-03T11:58:07.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f0763fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07640cd8e0 0x7f07640cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:07.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076899b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0764153290 0x7f0764158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38864/0 (socket says 172.21.15.102:38864) 2023-03-03T11:58:07.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076899b700 1 -- 172.21.15.102:0/2344167780 learned_addr learned my addr 172.21.15.102:0/2344167780 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T11:58:07.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f076919c700 1 --2- 172.21.15.102:0/2344167780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd390 0x7f07640ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f0763fff700 1 -- 172.21.15.102:0/2344167780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd390 msgr2=0x7f07640ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f0763fff700 1 --2- 172.21.15.102:0/2344167780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd390 0x7f07640ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.606+0000 7f0763fff700 1 -- 172.21.15.102:0/2344167780 >> v1:172.21.15.172:6789/0 conn(0x7f0764153290 legacy=0x7f0764158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.606+0000 7f0763fff700 1 -- 172.21.15.102:0/2344167780 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076406c7f0 con 0x7f07640cd8e0 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.605+0000 7f07637fe700 1 -- 172.21.15.102:0/2344167780 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058627336 0 0) 0x7f0764153660 con 0x7f0764153290 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.606+0000 7f0763fff700 1 --2- 172.21.15.102:0/2344167780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07640cde20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0750009840 tx=0x7f0750009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa716cc2cf73b3be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.606+0000 7f07637fe700 1 -- 172.21.15.102:0/2344167780 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750011830 con 0x7f07640cd8e0 2023-03-03T11:58:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.606+0000 7f07637fe700 1 -- 172.21.15.102:0/2344167780 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0750011990 con 0x7f07640cd8e0 2023-03-03T11:58:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.606+0000 7f07637fe700 1 -- 172.21.15.102:0/2344167780 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750011d20 con 0x7f07640cd8e0 2023-03-03T11:58:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 -- 172.21.15.102:0/2344167780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 msgr2=0x7f07640cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 --2- 172.21.15.102:0/2344167780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07640cde20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0750009840 tx=0x7f0750009b50 comp rx=0 tx=0).stop 2023-03-03T11:58:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 -- 172.21.15.102:0/2344167780 shutdown_connections 2023-03-03T11:58:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 --2- 172.21.15.102:0/2344167780 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd390 0x7f07640ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 --2- 172.21.15.102:0/2344167780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07640cde20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 -- 172.21.15.102:0/2344167780 >> 172.21.15.102:0/2344167780 conn(0x7f0764065fb0 msgr2=0x7f07640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:58:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 -- 172.21.15.102:0/2344167780 shutdown_connections 2023-03-03T11:58:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.607+0000 7f076abff700 1 -- 172.21.15.102:0/2344167780 wait complete. 2023-03-03T11:58:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.610+0000 7f076abff700 1 Processor -- start 2023-03-03T11:58:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.610+0000 7f076abff700 1 -- start start 2023-03-03T11:58:07.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.611+0000 7f076abff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07640cd390 0x7f07641511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.611+0000 7f076abff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07641f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.611+0000 7f076abff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0764153290 0x7f07641f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.611+0000 7f076abff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641ee070 con 0x7f07640cd8e0 2023-03-03T11:58:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.611+0000 7f076abff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f1830 con 0x7f07640cd390 2023-03-03T11:58:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.611+0000 7f076abff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07641ee260 con 0x7f0764153290 2023-03-03T11:58:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f0763fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07641f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076899b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07640cd390 0x7f07641511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:07.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076919c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0764153290 0x7f07641f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:07.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076899b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07640cd390 0x7f07641511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38294/0 (socket says 172.21.15.102:38294) 2023-03-03T11:58:07.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076899b700 1 -- 172.21.15.102:0/2426790839 learned_addr learned my addr 172.21.15.102:0/2426790839 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:58:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076919c700 1 -- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07640cd390 msgr2=0x7f07641511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076919c700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07640cd390 0x7f07641511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076919c700 1 -- 172.21.15.102:0/2426790839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 msgr2=0x7f07641f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076919c700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07641f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f076919c700 1 -- 172.21.15.102:0/2426790839 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07500094a0 con 0x7f0764153290 2023-03-03T11:58:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.612+0000 7f0763fff700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07641f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:58:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.613+0000 7f076919c700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0764153290 0x7f07641f2200 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f075c00ebd0 tx=0x7f075c00eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:58:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.613+0000 7f07617fa700 1 -- 172.21.15.102:0/2426790839 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00ca00 con 0x7f0764153290 2023-03-03T11:58:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.613+0000 7f07617fa700 1 -- 172.21.15.102:0/2426790839 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c00f040 con 0x7f0764153290 2023-03-03T11:58:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.613+0000 7f07617fa700 1 -- 172.21.15.102:0/2426790839 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00cda0 con 0x7f0764153290 2023-03-03T11:58:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.613+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07640ccde0 con 0x7f0764153290 2023-03-03T11:58:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.613+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076415d160 con 0x7f0764153290 2023-03-03T11:58:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.614+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07640cd0c0 con 0x7f0764153290 2023-03-03T11:58:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.615+0000 7f07617fa700 1 -- 172.21.15.102:0/2426790839 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f075c014480 con 0x7f0764153290 2023-03-03T11:58:07.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.617+0000 7f07617fa700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07480682f0 0x7f074806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:07.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.617+0000 7f07617fa700 1 -- 172.21.15.102:0/2426790839 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f075c086970 con 0x7f0764153290 2023-03-03T11:58:07.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.617+0000 7f076899b700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07480682f0 0x7f074806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:07.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.617+0000 7f076899b700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07480682f0 0x7f074806a7a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0758000f60 tx=0x7f0758007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:58:07.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.621+0000 7f07617fa700 1 -- 172.21.15.102:0/2426790839 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f075c052f90 con 0x7f0764153290 2023-03-03T11:58:07.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.856+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07641f1680 con 0x7f0764153290 2023-03-03T11:58:07.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.857+0000 7f07617fa700 1 -- 172.21.15.102:0/2426790839 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f075c018070 con 0x7f0764153290 2023-03-03T11:58:07.859 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:58:07.860 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:58:07.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07480682f0 msgr2=0x7f074806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:07.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07480682f0 0x7f074806a7a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0758000f60 tx=0x7f0758007ea0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0764153290 msgr2=0x7f07641f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:07.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0764153290 0x7f07641f2200 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f075c00ebd0 tx=0x7f075c00eee0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 shutdown_connections 2023-03-03T11:58:07.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07480682f0 0x7f074806a7a0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0764153290 0x7f07641f2200 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07640cd390 0x7f07641511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 --2- 172.21.15.102:0/2426790839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07641f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:07.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 >> 172.21.15.102:0/2426790839 conn(0x7f0764065fb0 msgr2=0x7f07641574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:58:07.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 shutdown_connections 2023-03-03T11:58:07.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:07.860+0000 7f076abff700 1 -- 172.21.15.102:0/2426790839 wait complete. 2023-03-03T11:58:07.864 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:58:07.889 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 22 secs before thrashing 2023-03-03T11:58:16.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:16.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:29.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:58:30.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.139+0000 7f9537fff700 1 Processor -- start 2023-03-03T11:58:30.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.140+0000 7f9537fff700 1 -- start start 2023-03-03T11:58:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f9537fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95380c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f9537fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f95380b1e90 0x7f953815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f9537fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95380b3f30 0x7f95380cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f9537fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953806c780 con 0x7f95380b3f30 2023-03-03T11:58:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f9537fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953815c620 con 0x7f95380b1e90 2023-03-03T11:58:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f9537fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953815c050 con 0x7f9538153030 2023-03-03T11:58:30.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f95377fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f95380b3f30 0x7f95380cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f952ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f95380b1e90 0x7f953815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.141+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95380c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95380c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54198/0 (socket says 172.21.15.102:54198) 2023-03-03T11:58:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 -- 172.21.15.102:0/574641596 learned_addr learned my addr 172.21.15.102:0/574641596 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:58:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 -- 172.21.15.102:0/574641596 >> v2:172.21.15.172:3300/0 conn(0x7f95380b1e90 msgr2=0x7f953815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:58:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 --2- 172.21.15.102:0/574641596 >> v2:172.21.15.172:3300/0 conn(0x7f95380b1e90 0x7f953815ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 -- 172.21.15.102:0/574641596 >> v2:172.21.15.102:3300/0 conn(0x7f95380b3f30 msgr2=0x7f95380cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:58:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 --2- 172.21.15.102:0/574641596 >> v2:172.21.15.102:3300/0 conn(0x7f95380b3f30 0x7f95380cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 -- 172.21.15.102:0/574641596 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953806bee0 con 0x7f9538153030 2023-03-03T11:58:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.142+0000 7f9536ffd700 1 --2- 172.21.15.102:0/574641596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95380c5b00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9538072650 tx=0x7f9528009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdabda97648c19c0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:58:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f95367fc700 1 -- 172.21.15.102:0/574641596 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528011830 con 0x7f9538153030 2023-03-03T11:58:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f95367fc700 1 -- 172.21.15.102:0/574641596 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9528011990 con 0x7f9538153030 2023-03-03T11:58:30.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f95367fc700 1 -- 172.21.15.102:0/574641596 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528011d20 con 0x7f9538153030 2023-03-03T11:58:30.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 -- 172.21.15.102:0/574641596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 msgr2=0x7f95380c5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:30.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 --2- 172.21.15.102:0/574641596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95380c5b00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9538072650 tx=0x7f9528009bd0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 -- 172.21.15.102:0/574641596 shutdown_connections 2023-03-03T11:58:30.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 --2- 172.21.15.102:0/574641596 >> v2:172.21.15.102:3300/0 conn(0x7f95380b3f30 0x7f95380cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 --2- 172.21.15.102:0/574641596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95380c5b00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 --2- 172.21.15.102:0/574641596 >> v2:172.21.15.172:3300/0 conn(0x7f95380b1e90 0x7f953815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 -- 172.21.15.102:0/574641596 >> 172.21.15.102:0/574641596 conn(0x7f953808b1c0 msgr2=0x7f9538071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:58:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 -- 172.21.15.102:0/574641596 shutdown_connections 2023-03-03T11:58:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.143+0000 7f9537fff700 1 -- 172.21.15.102:0/574641596 wait complete. 2023-03-03T11:58:30.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.145+0000 7f9537fff700 1 Processor -- start 2023-03-03T11:58:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.146+0000 7f9537fff700 1 -- start start 2023-03-03T11:58:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9537fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b1e90 0x7f9538150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9537fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95380b3f30 0x7f95381ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9537fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95381efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:30.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9537fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95380cbae0 con 0x7f95380b3f30 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9537fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95380cc3c0 con 0x7f95380b1e90 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9537fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953814de50 con 0x7f9538153030 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f952ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95380b3f30 0x7f95381ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b1e90 0x7f9538150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f95377fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95381efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b1e90 0x7f9538150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:51498/0 (socket says 172.21.15.102:51498) 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f952ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95380b3f30 0x7f95381ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50438/0 (socket says 172.21.15.102:50438) 2023-03-03T11:58:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.147+0000 7f9536ffd700 1 -- 172.21.15.102:0/1677265215 learned_addr learned my addr 172.21.15.102:0/1677265215 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9536ffd700 1 -- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 msgr2=0x7f95381efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9536ffd700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95381efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9536ffd700 1 -- 172.21.15.102:0/1677265215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95380b3f30 msgr2=0x7f95381ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9536ffd700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95380b3f30 0x7f95381ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9536ffd700 1 -- 172.21.15.102:0/1677265215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528009550 con 0x7f95380b1e90 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f95377fe700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95381efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9536ffd700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b1e90 0x7f9538150a20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9528009c10 tx=0x7f9528004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:58:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f952f7fe700 1 -- 172.21.15.102:0/1677265215 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528004450 con 0x7f95380b1e90 2023-03-03T11:58:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f952f7fe700 1 -- 172.21.15.102:0/1677265215 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95280045b0 con 0x7f95380b1e90 2023-03-03T11:58:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f952f7fe700 1 -- 172.21.15.102:0/1677265215 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95280038a0 con 0x7f95380b1e90 2023-03-03T11:58:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f953806c4a0 con 0x7f95380b1e90 2023-03-03T11:58:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.148+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f953806c1c0 con 0x7f95380b1e90 2023-03-03T11:58:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.150+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f953815c240 con 0x7f95380b1e90 2023-03-03T11:58:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.151+0000 7f952f7fe700 1 -- 172.21.15.102:0/1677265215 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f952802d070 con 0x7f95380b1e90 2023-03-03T11:58:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.152+0000 7f952f7fe700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95180682f0 0x7f951806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:58:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.152+0000 7f952f7fe700 1 -- 172.21.15.102:0/1677265215 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9528021030 con 0x7f95380b1e90 2023-03-03T11:58:30.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.152+0000 7f952ffff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95180682f0 0x7f951806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:58:30.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.153+0000 7f952ffff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95180682f0 0x7f951806a7a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f95380ceb50 tx=0x7f9520009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:58:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.157+0000 7f952f7fe700 1 -- 172.21.15.102:0/1677265215 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95280538b0 con 0x7f95380b1e90 2023-03-03T11:58:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.390+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f953814e130 con 0x7f95380b1e90 2023-03-03T11:58:30.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.392+0000 7f952f7fe700 1 -- 172.21.15.102:0/1677265215 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9528056ed0 con 0x7f95380b1e90 2023-03-03T11:58:30.393 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:58:30.394 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:58:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95180682f0 msgr2=0x7f951806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95180682f0 0x7f951806a7a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f95380ceb50 tx=0x7f9520009210 comp rx=0 tx=0).stop 2023-03-03T11:58:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b1e90 msgr2=0x7f9538150a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:58:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b1e90 0x7f9538150a20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9528009c10 tx=0x7f9528004750 comp rx=0 tx=0).stop 2023-03-03T11:58:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 shutdown_connections 2023-03-03T11:58:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f95180682f0 0x7f951806a7a0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9538153030 0x7f95381efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b1e90 0x7f9538150a20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 --2- 172.21.15.102:0/1677265215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95380b3f30 0x7f95381ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:58:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 >> 172.21.15.102:0/1677265215 conn(0x7f953808b1c0 msgr2=0x7f953814ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:58:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 shutdown_connections 2023-03-03T11:58:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:58:30.394+0000 7f9537fff700 1 -- 172.21.15.102:0/1677265215 wait complete. 2023-03-03T11:58:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:58:30.423 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T11:58:46.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:46.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:16.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:16.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:47.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:47.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:54.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:59:54.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.674+0000 7f19855ff700 1 Processor -- start 2023-03-03T11:59:54.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.675+0000 7f19855ff700 1 -- start start 2023-03-03T11:59:54.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f19855ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1980153290 0x7f1980158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:54.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f19855ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f19801593c0 0x7f19800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:54.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f19855ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1980153660 con 0x7f19800cd8e0 2023-03-03T11:59:54.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f19855ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198006c360 con 0x7f1980153290 2023-03-03T11:59:54.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f19855ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198015cea0 con 0x7f19801593c0 2023-03-03T11:59:54.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f197effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1980153290 0x7f1980158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:54.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f197effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1980153290 0x7f1980158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37504/0 (socket says 172.21.15.102:37504) 2023-03-03T11:59:54.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f197effd700 1 -- 172.21.15.102:0/1483715126 learned_addr learned my addr 172.21.15.102:0/1483715126 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:59:54.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.677+0000 7f197e7fc700 1 --2- 172.21.15.102:0/1483715126 >> v2:172.21.15.172:3300/0 conn(0x7f19801593c0 0x7f19800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:54.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197effd700 1 -- 172.21.15.102:0/1483715126 >> v2:172.21.15.172:3300/0 conn(0x7f19801593c0 msgr2=0x7f19800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:59:54.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197effd700 1 --2- 172.21.15.102:0/1483715126 >> v2:172.21.15.172:3300/0 conn(0x7f19801593c0 0x7f19800cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197effd700 1 -- 172.21.15.102:0/1483715126 >> v1:172.21.15.172:6789/0 conn(0x7f19800cd8e0 legacy=0x7f19800ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:54.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197effd700 1 -- 172.21.15.102:0/1483715126 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198006c7f0 con 0x7f1980153290 2023-03-03T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197effd700 1 --2- 172.21.15.102:0/1483715126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980158e80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1980159fb0 tx=0x7f1970009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780e096c62e30241 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197dffb700 1 -- 172.21.15.102:0/1483715126 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970011830 con 0x7f1980153290 2023-03-03T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197dffb700 1 -- 172.21.15.102:0/1483715126 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970011990 con 0x7f1980153290 2023-03-03T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.678+0000 7f197dffb700 1 -- 172.21.15.102:0/1483715126 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970011d20 con 0x7f1980153290 2023-03-03T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 -- 172.21.15.102:0/1483715126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 msgr2=0x7f1980158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:54.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 --2- 172.21.15.102:0/1483715126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980158e80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1980159fb0 tx=0x7f1970009bd0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 -- 172.21.15.102:0/1483715126 shutdown_connections 2023-03-03T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 --2- 172.21.15.102:0/1483715126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980158e80 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 --2- 172.21.15.102:0/1483715126 >> v2:172.21.15.172:3300/0 conn(0x7f19801593c0 0x7f19800cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 -- 172.21.15.102:0/1483715126 >> 172.21.15.102:0/1483715126 conn(0x7f1980065fb0 msgr2=0x7f19800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 -- 172.21.15.102:0/1483715126 shutdown_connections 2023-03-03T11:59:54.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.679+0000 7f19855ff700 1 -- 172.21.15.102:0/1483715126 wait complete. 2023-03-03T11:59:54.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.682+0000 7f19855ff700 1 Processor -- start 2023-03-03T11:59:54.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.682+0000 7f19855ff700 1 -- start start 2023-03-03T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.683+0000 7f19855ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19800cd8e0 0x7f198014c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.683+0000 7f19855ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f19855ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801593c0 0x7f1980156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f19855ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19801ee930 con 0x7f1980153290 2023-03-03T11:59:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f19855ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800b45f0 con 0x7f19800cd8e0 2023-03-03T11:59:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f19855ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19800b1d10 con 0x7f19801593c0 2023-03-03T11:59:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197effd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19800cd8e0 0x7f198014c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37518/0 (socket says 172.21.15.102:37518) 2023-03-03T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 -- 172.21.15.102:0/2729067543 learned_addr learned my addr 172.21.15.102:0/2729067543 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197f7fe700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801593c0 0x7f1980156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 -- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801593c0 msgr2=0x7f1980156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801593c0 0x7f1980156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 -- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19800cd8e0 msgr2=0x7f198014c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19800cd8e0 0x7f198014c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197e7fc700 1 -- 172.21.15.102:0/2729067543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970009550 con 0x7f1980153290 2023-03-03T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.684+0000 7f197effd700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19800cd8e0 0x7f198014c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.685+0000 7f197e7fc700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980155cc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f197400e9e0 tx=0x7f197400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.685+0000 7f19677fe700 1 -- 172.21.15.102:0/2729067543 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974009710 con 0x7f1980153290 2023-03-03T11:59:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.685+0000 7f19677fe700 1 -- 172.21.15.102:0/2729067543 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197400ee80 con 0x7f1980153290 2023-03-03T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.685+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19800ccde0 con 0x7f1980153290 2023-03-03T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.685+0000 7f19677fe700 1 -- 172.21.15.102:0/2729067543 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197400ca50 con 0x7f1980153290 2023-03-03T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.685+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198015d160 con 0x7f1980153290 2023-03-03T11:59:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.686+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19800cd0c0 con 0x7f1980153290 2023-03-03T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.688+0000 7f19677fe700 1 -- 172.21.15.102:0/2729067543 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1974023070 con 0x7f1980153290 2023-03-03T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.689+0000 7f19677fe700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.689+0000 7f19677fe700 1 -- 172.21.15.102:0/2729067543 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1974085fa0 con 0x7f1980153290 2023-03-03T11:59:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.689+0000 7f197effd700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.690+0000 7f197effd700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1970009c10 tx=0x7f1970008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.693+0000 7f19677fe700 1 -- 172.21.15.102:0/2729067543 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1974052650 con 0x7f1980153290 2023-03-03T11:59:54.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.926+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19801eea70 con 0x7f1980153290 2023-03-03T11:59:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.926+0000 7f19677fe700 1 -- 172.21.15.102:0/2729067543 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1974055c70 con 0x7f1980153290 2023-03-03T11:59:54.928 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T11:59:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T11:59:54.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 msgr2=0x7f196c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:54.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1970009c10 tx=0x7f1970008030 comp rx=0 tx=0).stop 2023-03-03T11:59:54.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 msgr2=0x7f1980155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:54.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980155cc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f197400e9e0 tx=0x7f197400ecf0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 shutdown_connections 2023-03-03T11:59:54.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f196c0682f0 0x7f196c06a7a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19801593c0 0x7f1980156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19800cd8e0 0x7f198014c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 --2- 172.21.15.102:0/2729067543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1980153290 0x7f1980155cc0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:54.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 >> 172.21.15.102:0/2729067543 conn(0x7f1980065fb0 msgr2=0x7f19800cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:59:54.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.929+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 shutdown_connections 2023-03-03T11:59:54.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T11:59:54.930+0000 7f19855ff700 1 -- 172.21.15.102:0/2729067543 wait complete. 2023-03-03T11:59:54.933 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T11:59:54.958 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T12:00:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T11:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:00:17.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:17.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:47.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:47.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:03.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:01:04.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.211+0000 7f1294fff700 1 Processor -- start 2023-03-03T12:01:04.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.212+0000 7f1294fff700 1 -- start start 2023-03-03T12:01:04.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.213+0000 7f1294fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f129014ffb0 0x7f12900c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:04.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.213+0000 7f1294fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12900716c0 0x7f12900706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:04.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.213+0000 7f1294fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12900c3c80 con 0x7f129014fbe0 2023-03-03T12:01:04.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.213+0000 7f1294fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12900c41a0 con 0x7f129014ffb0 2023-03-03T12:01:04.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.213+0000 7f1294fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1290071cd0 con 0x7f12900716c0 2023-03-03T12:01:04.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.213+0000 7f1287fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f129014ffb0 0x7f12900c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:04.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f1287fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f129014ffb0 0x7f12900c3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44932/0 (socket says 172.21.15.102:44932) 2023-03-03T12:01:04.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f1287fff700 1 -- 172.21.15.102:0/2698351743 learned_addr learned my addr 172.21.15.102:0/2698351743 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:01:04.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f128ed9d700 1 --2- 172.21.15.102:0/2698351743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12900716c0 0x7f12900706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:04.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f128ed9d700 1 -- 172.21.15.102:0/2698351743 >> v2:172.21.15.102:3300/0 conn(0x7f129014ffb0 msgr2=0x7f12900c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:01:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f128ed9d700 1 --2- 172.21.15.102:0/2698351743 >> v2:172.21.15.102:3300/0 conn(0x7f129014ffb0 0x7f12900c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f128dd9b700 1 -- 172.21.15.102:0/2698351743 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312686019 0 0) 0x7f1280001110 con 0x7f129014fbe0 2023-03-03T12:01:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f128ed9d700 1 -- 172.21.15.102:0/2698351743 >> v1:172.21.15.172:6789/0 conn(0x7f129014fbe0 legacy=0x7f12900c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f128ed9d700 1 -- 172.21.15.102:0/2698351743 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12900c3ec0 con 0x7f12900716c0 2023-03-03T12:01:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f1287fff700 1 --2- 172.21.15.102:0/2698351743 >> v2:172.21.15.102:3300/0 conn(0x7f129014ffb0 0x7f12900c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:01:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.214+0000 7f128ed9d700 1 --2- 172.21.15.102:0/2698351743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12900716c0 0x7f12900706f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1288009d90 tx=0x7f128800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9c09d09220bbaea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f128dd9b700 1 -- 172.21.15.102:0/2698351743 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128800e070 con 0x7f12900716c0 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f128dd9b700 1 -- 172.21.15.102:0/2698351743 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128800bb20 con 0x7f12900716c0 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f128dd9b700 1 -- 172.21.15.102:0/2698351743 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1288012430 con 0x7f12900716c0 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 -- 172.21.15.102:0/2698351743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12900716c0 msgr2=0x7f12900706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 --2- 172.21.15.102:0/2698351743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12900716c0 0x7f12900706f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1288009d90 tx=0x7f128800b330 comp rx=0 tx=0).stop 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 -- 172.21.15.102:0/2698351743 shutdown_connections 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 --2- 172.21.15.102:0/2698351743 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12900716c0 0x7f12900706f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 --2- 172.21.15.102:0/2698351743 >> v2:172.21.15.102:3300/0 conn(0x7f129014ffb0 0x7f12900c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 -- 172.21.15.102:0/2698351743 >> 172.21.15.102:0/2698351743 conn(0x7f129006af90 msgr2=0x7f129006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:01:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 -- 172.21.15.102:0/2698351743 shutdown_connections 2023-03-03T12:01:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.215+0000 7f1294fff700 1 -- 172.21.15.102:0/2698351743 wait complete. 2023-03-03T12:01:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.217+0000 7f1294fff700 1 Processor -- start 2023-03-03T12:01:04.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.218+0000 7f1294fff700 1 -- start start 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1294fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12900716c0 0x7f12901dce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1294fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129014fbe0 0x7f12901ee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1294fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f129014ffb0 0x7f1290147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1294fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12901f2030 con 0x7f129014fbe0 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1294fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12900cb4e0 con 0x7f12900716c0 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1294fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12900cb200 con 0x7f129014ffb0 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129014fbe0 0x7f12901ee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:04.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129014fbe0 0x7f12901ee0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44948/0 (socket says 172.21.15.102:44948) 2023-03-03T12:01:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 -- 172.21.15.102:0/3916971560 learned_addr learned my addr 172.21.15.102:0/3916971560 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:01:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f128ed9d700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f129014ffb0 0x7f1290147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f128e59c700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12900716c0 0x7f12901dce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 -- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f129014ffb0 msgr2=0x7f1290147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f129014ffb0 0x7f1290147170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 -- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12900716c0 msgr2=0x7f12901dce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:04.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12900716c0 0x7f12901dce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.219+0000 7f1287fff700 1 -- 172.21.15.102:0/3916971560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278009550 con 0x7f129014fbe0 2023-03-03T12:01:04.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.220+0000 7f1287fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129014fbe0 0x7f12901ee0f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f127800ebe0 tx=0x7f127800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:04.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.220+0000 7f1286ffd700 1 -- 172.21.15.102:0/3916971560 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127800c850 con 0x7f129014fbe0 2023-03-03T12:01:04.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.220+0000 7f1286ffd700 1 -- 172.21.15.102:0/3916971560 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127800c9b0 con 0x7f129014fbe0 2023-03-03T12:01:04.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.220+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1288009a70 con 0x7f129014fbe0 2023-03-03T12:01:04.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.220+0000 7f1286ffd700 1 -- 172.21.15.102:0/3916971560 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127800cd40 con 0x7f129014fbe0 2023-03-03T12:01:04.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.220+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1290072250 con 0x7f129014fbe0 2023-03-03T12:01:04.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.221+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1290071f70 con 0x7f129014fbe0 2023-03-03T12:01:04.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.223+0000 7f1286ffd700 1 -- 172.21.15.102:0/3916971560 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1278019030 con 0x7f129014fbe0 2023-03-03T12:01:04.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.225+0000 7f1286ffd700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1270068320 0x7f127006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:04.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.225+0000 7f1286ffd700 1 -- 172.21.15.102:0/3916971560 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1278085e50 con 0x7f129014fbe0 2023-03-03T12:01:04.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.225+0000 7f128e59c700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1270068320 0x7f127006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:04.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.226+0000 7f128e59c700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1270068320 0x7f127006a7d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f1280000f60 tx=0x7f1280009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:04.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.229+0000 7f1286ffd700 1 -- 172.21.15.102:0/3916971560 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12780524d0 con 0x7f129014fbe0 2023-03-03T12:01:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.463+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1290072530 con 0x7f129014fbe0 2023-03-03T12:01:04.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.463+0000 7f1286ffd700 1 -- 172.21.15.102:0/3916971560 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1278021090 con 0x7f129014fbe0 2023-03-03T12:01:04.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:01:04.466 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:01:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1270068320 msgr2=0x7f127006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1270068320 0x7f127006a7d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f1280000f60 tx=0x7f1280009040 comp rx=0 tx=0).stop 2023-03-03T12:01:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129014fbe0 msgr2=0x7f12901ee0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129014fbe0 0x7f12901ee0f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f127800ebe0 tx=0x7f127800eef0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 shutdown_connections 2023-03-03T12:01:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1270068320 0x7f127006a7d0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f129014ffb0 0x7f1290147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12900716c0 0x7f12901dce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 --2- 172.21.15.102:0/3916971560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129014fbe0 0x7f12901ee0f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 >> 172.21.15.102:0/3916971560 conn(0x7f129006af90 msgr2=0x7f12900c79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:01:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.466+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 shutdown_connections 2023-03-03T12:01:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:04.467+0000 7f1294fff700 1 -- 172.21.15.102:0/3916971560 wait complete. 2023-03-03T12:01:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:01:04.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 16 secs before thrashing 2023-03-03T12:01:17.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:17.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:20.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:01:20.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.762+0000 7f7d4539d700 1 Processor -- start 2023-03-03T12:01:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.762+0000 7f7d4539d700 1 -- start start 2023-03-03T12:01:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d4539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d400cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d4539d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7d400c9610 0x7f7d400cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d4539d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d400c9ca0 con 0x7f7d400ca440 2023-03-03T12:01:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d4539d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d400b4920 con 0x7f7d400c9610 2023-03-03T12:01:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d4539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400b4c00 con 0x7f7d401564b0 2023-03-03T12:01:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d3f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7d400c9610 0x7f7d400cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d400cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d44b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7d400ca440 0x7f7d400ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53454/0 (socket says 172.21.15.102:53454) 2023-03-03T12:01:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.764+0000 7f7d44b9c700 1 -- 172.21.15.102:0/2835798113 learned_addr learned my addr 172.21.15.102:0/2835798113 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:01:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3f7fe700 1 -- 172.21.15.102:0/2835798113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 msgr2=0x7f7d400cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:01:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3f7fe700 1 --2- 172.21.15.102:0/2835798113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d400cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3f7fe700 1 -- 172.21.15.102:0/2835798113 >> v1:172.21.15.102:6789/0 conn(0x7f7d400ca440 legacy=0x7f7d400ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3f7fe700 1 -- 172.21.15.102:0/2835798113 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d400b46f0 con 0x7f7d400c9610 2023-03-03T12:01:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3f7fe700 1 --2- 172.21.15.102:0/2835798113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400c9610 0x7f7d400cbba0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009840 tx=0x7f7d34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e477a55b444741b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3effd700 1 -- 172.21.15.102:0/2835798113 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34011830 con 0x7f7d400c9610 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3effd700 1 -- 172.21.15.102:0/2835798113 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34011990 con 0x7f7d400c9610 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.765+0000 7f7d3effd700 1 -- 172.21.15.102:0/2835798113 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34011d20 con 0x7f7d400c9610 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 -- 172.21.15.102:0/2835798113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400c9610 msgr2=0x7f7d400cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 --2- 172.21.15.102:0/2835798113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400c9610 0x7f7d400cbba0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009840 tx=0x7f7d34009b50 comp rx=0 tx=0).stop 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 -- 172.21.15.102:0/2835798113 shutdown_connections 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 --2- 172.21.15.102:0/2835798113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d400cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 --2- 172.21.15.102:0/2835798113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400c9610 0x7f7d400cbba0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 -- 172.21.15.102:0/2835798113 >> 172.21.15.102:0/2835798113 conn(0x7f7d40065fb0 msgr2=0x7f7d400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:01:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 -- 172.21.15.102:0/2835798113 shutdown_connections 2023-03-03T12:01:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.766+0000 7f7d4539d700 1 -- 172.21.15.102:0/2835798113 wait complete. 2023-03-03T12:01:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.768+0000 7f7d4539d700 1 Processor -- start 2023-03-03T12:01:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.768+0000 7f7d4539d700 1 -- start start 2023-03-03T12:01:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.769+0000 7f7d4539d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d400c9610 0x7f7d401e4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.769+0000 7f7d4539d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ca440 0x7f7d40159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.769+0000 7f7d4539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d4014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.769+0000 7f7d4539d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400c5f30 con 0x7f7d400ca440 2023-03-03T12:01:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.769+0000 7f7d4539d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40158910 con 0x7f7d400c9610 2023-03-03T12:01:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.769+0000 7f7d4539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40157be0 con 0x7f7d401564b0 2023-03-03T12:01:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ca440 0x7f7d40159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d400c9610 0x7f7d401e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ca440 0x7f7d40159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42188/0 (socket says 172.21.15.102:42188) 2023-03-03T12:01:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 -- 172.21.15.102:0/1259748863 learned_addr learned my addr 172.21.15.102:0/1259748863 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:01:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d44b9c700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d4014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 -- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 msgr2=0x7f7d4014d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d4014d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 -- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d400c9610 msgr2=0x7f7d401e4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d400c9610 0x7f7d401e4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 -- 172.21.15.102:0/1259748863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d340094a0 con 0x7f7d400ca440 2023-03-03T12:01:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3ffff700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d400c9610 0x7f7d401e4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:01:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.770+0000 7f7d3f7fe700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ca440 0x7f7d40159360 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7d340090b0 tx=0x7f7d34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.771+0000 7f7d3cff9700 1 -- 172.21.15.102:0/1259748863 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d340036a0 con 0x7f7d400ca440 2023-03-03T12:01:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.771+0000 7f7d3cff9700 1 -- 172.21.15.102:0/1259748863 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3401f070 con 0x7f7d400ca440 2023-03-03T12:01:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.771+0000 7f7d3cff9700 1 -- 172.21.15.102:0/1259748863 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34003a60 con 0x7f7d400ca440 2023-03-03T12:01:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.771+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d400b2210 con 0x7f7d400ca440 2023-03-03T12:01:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.771+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d400b1f30 con 0x7f7d400ca440 2023-03-03T12:01:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.772+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d40156880 con 0x7f7d400ca440 2023-03-03T12:01:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.773+0000 7f7d3cff9700 1 -- 172.21.15.102:0/1259748863 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7d3400f050 con 0x7f7d400ca440 2023-03-03T12:01:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.774+0000 7f7d3cff9700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7d2c0682f0 0x7f7d2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.774+0000 7f7d3cff9700 1 -- 172.21.15.102:0/1259748863 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7d34021070 con 0x7f7d400ca440 2023-03-03T12:01:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.775+0000 7f7d3ffff700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7d2c0682f0 0x7f7d2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.775+0000 7f7d3ffff700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7d2c0682f0 0x7f7d2c06a7a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009f30 tx=0x7f7d30009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:20.779+0000 7f7d3cff9700 1 -- 172.21.15.102:0/1259748863 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7d34053d10 con 0x7f7d400ca440 2023-03-03T12:01:21.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.014+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d40150150 con 0x7f7d400ca440 2023-03-03T12:01:21.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.014+0000 7f7d3cff9700 1 -- 172.21.15.102:0/1259748863 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7d34057330 con 0x7f7d400ca440 2023-03-03T12:01:21.016 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:01:21.016 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:01:21.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7d2c0682f0 msgr2=0x7f7d2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:21.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7d2c0682f0 0x7f7d2c06a7a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009f30 tx=0x7f7d30009210 comp rx=0 tx=0).stop 2023-03-03T12:01:21.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ca440 msgr2=0x7f7d40159360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:21.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ca440 0x7f7d40159360 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7d340090b0 tx=0x7f7d34004750 comp rx=0 tx=0).stop 2023-03-03T12:01:21.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 shutdown_connections 2023-03-03T12:01:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7d2c0682f0 0x7f7d2c06a7a0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7d401564b0 0x7f7d4014d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d400c9610 0x7f7d401e4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 --2- 172.21.15.102:0/1259748863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d400ca440 0x7f7d40159360 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 >> 172.21.15.102:0/1259748863 conn(0x7f7d40065fb0 msgr2=0x7f7d400ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:01:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 shutdown_connections 2023-03-03T12:01:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:21.017+0000 7f7d4539d700 1 -- 172.21.15.102:0/1259748863 wait complete. 2023-03-03T12:01:21.021 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:01:21.046 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-03-03T12:01:40.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:01:40.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.310+0000 7fd204bff700 1 Processor -- start 2023-03-03T12:01:40.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.310+0000 7fd204bff700 1 -- start start 2023-03-03T12:01:40.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.311+0000 7fd204bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd200153290 0x7fd200158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:40.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.311+0000 7fd204bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2001593c0 0x7fd2000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:40.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.311+0000 7fd204bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd200153660 con 0x7fd2000cd8e0 2023-03-03T12:01:40.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.311+0000 7fd204bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd20006c360 con 0x7fd200153290 2023-03-03T12:01:40.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.311+0000 7fd204bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd20015cea0 con 0x7fd2001593c0 2023-03-03T12:01:40.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fdd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd2001593c0 0x7fd2000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:40.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fed9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd2000cd8e0 0x7fd2000ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52740/0 (socket says 172.21.15.102:52740) 2023-03-03T12:01:40.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fed9d700 1 -- 172.21.15.102:0/1961560457 learned_addr learned my addr 172.21.15.102:0/1961560457 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:01:40.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fe59c700 1 --2- 172.21.15.102:0/1961560457 >> v2:172.21.15.102:3300/0 conn(0x7fd200153290 0x7fd200158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:40.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fd59a700 1 -- 172.21.15.102:0/1961560457 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386977044 0 0) 0x7fd200153660 con 0x7fd2000cd8e0 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fd59a700 1 -- 172.21.15.102:0/1961560457 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e80035b0 con 0x7fd2000cd8e0 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fdd9b700 1 -- 172.21.15.102:0/1961560457 >> v2:172.21.15.102:3300/0 conn(0x7fd200153290 msgr2=0x7fd200158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fdd9b700 1 --2- 172.21.15.102:0/1961560457 >> v2:172.21.15.102:3300/0 conn(0x7fd200153290 0x7fd200158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fdd9b700 1 -- 172.21.15.102:0/1961560457 >> v1:172.21.15.172:6789/0 conn(0x7fd2000cd8e0 legacy=0x7fd2000ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fdd9b700 1 -- 172.21.15.102:0/1961560457 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20006c7f0 con 0x7fd2001593c0 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.312+0000 7fd1fe59c700 1 --2- 172.21.15.102:0/1961560457 >> v2:172.21.15.102:3300/0 conn(0x7fd200153290 0x7fd200158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.313+0000 7fd1fdd9b700 1 --2- 172.21.15.102:0/1961560457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2001593c0 0x7fd2000cde20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd200159fb0 tx=0x7fd1f4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d47950d77f85445 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:40.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.313+0000 7fd1fd59a700 1 -- 172.21.15.102:0/1961560457 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4011830 con 0x7fd2001593c0 2023-03-03T12:01:40.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.313+0000 7fd1fd59a700 1 -- 172.21.15.102:0/1961560457 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f4011990 con 0x7fd2001593c0 2023-03-03T12:01:40.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.313+0000 7fd1fd59a700 1 -- 172.21.15.102:0/1961560457 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4011d20 con 0x7fd2001593c0 2023-03-03T12:01:40.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 -- 172.21.15.102:0/1961560457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2001593c0 msgr2=0x7fd2000cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:40.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 --2- 172.21.15.102:0/1961560457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2001593c0 0x7fd2000cde20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd200159fb0 tx=0x7fd1f4009b20 comp rx=0 tx=0).stop 2023-03-03T12:01:40.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 -- 172.21.15.102:0/1961560457 shutdown_connections 2023-03-03T12:01:40.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 --2- 172.21.15.102:0/1961560457 >> v2:172.21.15.102:3300/0 conn(0x7fd200153290 0x7fd200158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 --2- 172.21.15.102:0/1961560457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2001593c0 0x7fd2000cde20 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 -- 172.21.15.102:0/1961560457 >> 172.21.15.102:0/1961560457 conn(0x7fd200065fb0 msgr2=0x7fd2000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:01:40.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 -- 172.21.15.102:0/1961560457 shutdown_connections 2023-03-03T12:01:40.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.314+0000 7fd204bff700 1 -- 172.21.15.102:0/1961560457 wait complete. 2023-03-03T12:01:40.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.316+0000 7fd204bff700 1 Processor -- start 2023-03-03T12:01:40.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.316+0000 7fd204bff700 1 -- start start 2023-03-03T12:01:40.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.317+0000 7fd204bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2000cd8e0 0x7fd2001e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:40.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.317+0000 7fd204bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd200153290 0x7fd2001f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:40.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.317+0000 7fd204bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2001593c0 0x7fd2000c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:40.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.317+0000 7fd204bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2001e3900 con 0x7fd2001593c0 2023-03-03T12:01:40.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.317+0000 7fd204bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2001e6130 con 0x7fd2000cd8e0 2023-03-03T12:01:40.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.317+0000 7fd204bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000c81e0 con 0x7fd200153290 2023-03-03T12:01:40.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.317+0000 7fd1fe59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2000cd8e0 0x7fd2001e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2001593c0 0x7fd2000c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fdd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd200153290 0x7fd2001f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fe59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2000cd8e0 0x7fd2001e5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60944/0 (socket says 172.21.15.102:60944) 2023-03-03T12:01:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fe59c700 1 -- 172.21.15.102:0/3399229733 learned_addr learned my addr 172.21.15.102:0/3399229733 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:01:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fdd9b700 1 -- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2000cd8e0 msgr2=0x7fd2001e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fdd9b700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2000cd8e0 0x7fd2001e5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fdd9b700 1 -- 172.21.15.102:0/3399229733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2001593c0 msgr2=0x7fd2000c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fdd9b700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2001593c0 0x7fd2000c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fdd9b700 1 -- 172.21.15.102:0/3399229733 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f00094a0 con 0x7fd200153290 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fed9d700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2001593c0 0x7fd2000c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.318+0000 7fd1fdd9b700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd200153290 0x7fd2001f1d00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2000c9390 tx=0x7fd1f4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.319+0000 7fd1eeffd700 1 -- 172.21.15.102:0/3399229733 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4004590 con 0x7fd200153290 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.319+0000 7fd1eeffd700 1 -- 172.21.15.102:0/3399229733 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f401f070 con 0x7fd200153290 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.319+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f40094a0 con 0x7fd200153290 2023-03-03T12:01:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.319+0000 7fd1eeffd700 1 -- 172.21.15.102:0/3399229733 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40038e0 con 0x7fd200153290 2023-03-03T12:01:40.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.319+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2000ccde0 con 0x7fd200153290 2023-03-03T12:01:40.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.320+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20015d160 con 0x7fd200153290 2023-03-03T12:01:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.321+0000 7fd1eeffd700 1 -- 172.21.15.102:0/3399229733 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd1f400f050 con 0x7fd200153290 2023-03-03T12:01:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.322+0000 7fd1eeffd700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd1e0068320 0x7fd1e006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:01:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.322+0000 7fd1eeffd700 1 -- 172.21.15.102:0/3399229733 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd1f4021070 con 0x7fd200153290 2023-03-03T12:01:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.323+0000 7fd1fe59c700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd1e0068320 0x7fd1e006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:01:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.323+0000 7fd1fe59c700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd1e0068320 0x7fd1e006a7d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0009e70 tx=0x7fd1f0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:01:40.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.326+0000 7fd1eeffd700 1 -- 172.21.15.102:0/3399229733 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd1f4053d70 con 0x7fd200153290 2023-03-03T12:01:40.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.557+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2000cd0c0 con 0x7fd200153290 2023-03-03T12:01:40.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.558+0000 7fd1eeffd700 1 -- 172.21.15.102:0/3399229733 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd1f4015090 con 0x7fd200153290 2023-03-03T12:01:40.560 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:01:40.560 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:01:40.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd1e0068320 msgr2=0x7fd1e006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:40.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd1e0068320 0x7fd1e006a7d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0009e70 tx=0x7fd1f0009210 comp rx=0 tx=0).stop 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd200153290 msgr2=0x7fd2001f1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd200153290 0x7fd2001f1d00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2000c9390 tx=0x7fd1f4004750 comp rx=0 tx=0).stop 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 shutdown_connections 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2001593c0 0x7fd2000c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2000cd8e0 0x7fd2001e5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd1e0068320 0x7fd1e006a7d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 --2- 172.21.15.102:0/3399229733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd200153290 0x7fd2001f1d00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:01:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.560+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 >> 172.21.15.102:0/3399229733 conn(0x7fd200065fb0 msgr2=0x7fd2001510e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:01:40.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.561+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 shutdown_connections 2023-03-03T12:01:40.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:01:40.561+0000 7fd204bff700 1 -- 172.21.15.102:0/3399229733 wait complete. 2023-03-03T12:01:40.564 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:01:40.589 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T12:01:47.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:47.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:17.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:17.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:48.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:48.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:18.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:18.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:35.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:03:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.841+0000 7fa91d1ff700 1 Processor -- start 2023-03-03T12:03:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.841+0000 7fa91d1ff700 1 -- start start 2023-03-03T12:03:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.842+0000 7fa91d1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9180cd8e0 0x7fa9180cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.842+0000 7fa91d1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa9180cd390 0x7fa9180ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.842+0000 7fa91d1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa918153660 con 0x7fa918153290 2023-03-03T12:03:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.842+0000 7fa91d1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91806c360 con 0x7fa9180cd8e0 2023-03-03T12:03:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.842+0000 7fa91d1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91815cea0 con 0x7fa9180cd390 2023-03-03T12:03:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9180cd8e0 0x7fa9180cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9180cd8e0 0x7fa9180cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35132/0 (socket says 172.21.15.102:35132) 2023-03-03T12:03:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91659c700 1 -- 172.21.15.102:0/3708582765 learned_addr learned my addr 172.21.15.102:0/3708582765 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:03:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91759e700 1 --2- 172.21.15.102:0/3708582765 >> v2:172.21.15.172:3300/0 conn(0x7fa9180cd390 0x7fa9180ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91659c700 1 -- 172.21.15.102:0/3708582765 >> v2:172.21.15.172:3300/0 conn(0x7fa9180cd390 msgr2=0x7fa9180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:03:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91659c700 1 --2- 172.21.15.102:0/3708582765 >> v2:172.21.15.172:3300/0 conn(0x7fa9180cd390 0x7fa9180ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91659c700 1 -- 172.21.15.102:0/3708582765 >> v1:172.21.15.172:6789/0 conn(0x7fa918153290 legacy=0x7fa918158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91659c700 1 -- 172.21.15.102:0/3708582765 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91806c7f0 con 0x7fa9180cd8e0 2023-03-03T12:03:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa91759e700 1 --2- 172.21.15.102:0/3708582765 >> v2:172.21.15.172:3300/0 conn(0x7fa9180cd390 0x7fa9180ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:03:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.843+0000 7fa915d9b700 1 -- 172.21.15.102:0/3708582765 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621502176 0 0) 0x7fa918153660 con 0x7fa918153290 2023-03-03T12:03:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa91659c700 1 --2- 172.21.15.102:0/3708582765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd8e0 0x7fa9180cde20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa9180c9390 tx=0x7fa90c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66e12f18f86dd38 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:03:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa915d9b700 1 -- 172.21.15.102:0/3708582765 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c00b870 con 0x7fa9180cd8e0 2023-03-03T12:03:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa915d9b700 1 -- 172.21.15.102:0/3708582765 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c00b9d0 con 0x7fa9180cd8e0 2023-03-03T12:03:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa915d9b700 1 -- 172.21.15.102:0/3708582765 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c00bd60 con 0x7fa9180cd8e0 2023-03-03T12:03:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa91d1ff700 1 -- 172.21.15.102:0/3708582765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd8e0 msgr2=0x7fa9180cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/3708582765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd8e0 0x7fa9180cde20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa9180c9390 tx=0x7fa90c009bd0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa91d1ff700 1 -- 172.21.15.102:0/3708582765 shutdown_connections 2023-03-03T12:03:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/3708582765 >> v2:172.21.15.172:3300/0 conn(0x7fa9180cd390 0x7fa9180ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.844+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/3708582765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd8e0 0x7fa9180cde20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.845+0000 7fa91d1ff700 1 -- 172.21.15.102:0/3708582765 >> 172.21.15.102:0/3708582765 conn(0x7fa918065fb0 msgr2=0x7fa9180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:03:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.845+0000 7fa91d1ff700 1 -- 172.21.15.102:0/3708582765 shutdown_connections 2023-03-03T12:03:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.845+0000 7fa91d1ff700 1 -- 172.21.15.102:0/3708582765 wait complete. 2023-03-03T12:03:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.847+0000 7fa91d1ff700 1 Processor -- start 2023-03-03T12:03:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.847+0000 7fa91d1ff700 1 -- start start 2023-03-03T12:03:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91d1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd390 0x7fa9180c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91d1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9180cd8e0 0x7fa9180b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91d1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa918153290 0x7fa9181ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91d1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180c9200 con 0x7fa9180cd390 2023-03-03T12:03:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91d1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9181ee720 con 0x7fa9180cd8e0 2023-03-03T12:03:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91d1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9181eeca0 con 0x7fa918153290 2023-03-03T12:03:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa916d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd390 0x7fa9180c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa916d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd390 0x7fa9180c81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35134/0 (socket says 172.21.15.102:35134) 2023-03-03T12:03:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa916d9d700 1 -- 172.21.15.102:0/2029558970 learned_addr learned my addr 172.21.15.102:0/2029558970 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:03:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91659c700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9180cd8e0 0x7fa9180b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:35.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.848+0000 7fa91759e700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa918153290 0x7fa9181ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa916d9d700 1 -- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa918153290 msgr2=0x7fa9181ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa916d9d700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa918153290 0x7fa9181ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa916d9d700 1 -- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9180cd8e0 msgr2=0x7fa9180b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:35.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa916d9d700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9180cd8e0 0x7fa9180b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa916d9d700 1 -- 172.21.15.102:0/2029558970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c009550 con 0x7fa9180cd390 2023-03-03T12:03:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa916d9d700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd390 0x7fa9180c81d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c580 tx=0x7fa90800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:03:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa8ff7fe700 1 -- 172.21.15.102:0/2029558970 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9080054e0 con 0x7fa9180cd390 2023-03-03T12:03:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa8ff7fe700 1 -- 172.21.15.102:0/2029558970 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa908003680 con 0x7fa9180cd390 2023-03-03T12:03:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa8ff7fe700 1 -- 172.21.15.102:0/2029558970 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908003a30 con 0x7fa9180cd390 2023-03-03T12:03:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.849+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9180ccde0 con 0x7fa9180cd390 2023-03-03T12:03:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.850+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91815d160 con 0x7fa9180cd390 2023-03-03T12:03:35.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.851+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9180cd0c0 con 0x7fa9180cd390 2023-03-03T12:03:35.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.852+0000 7fa8ff7fe700 1 -- 172.21.15.102:0/2029558970 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa908019030 con 0x7fa9180cd390 2023-03-03T12:03:35.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.853+0000 7fa8ff7fe700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9040682f0 0x7fa90406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:35.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.853+0000 7fa8ff7fe700 1 -- 172.21.15.102:0/2029558970 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa908085080 con 0x7fa9180cd390 2023-03-03T12:03:35.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.854+0000 7fa91659c700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9040682f0 0x7fa90406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:35.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.854+0000 7fa91659c700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9040682f0 0x7fa90406a7a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009c10 tx=0x7fa90c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:03:35.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:35.857+0000 7fa8ff7fe700 1 -- 172.21.15.102:0/2029558970 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa908051700 con 0x7fa9180cd390 2023-03-03T12:03:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.092+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9181eef80 con 0x7fa9180cd390 2023-03-03T12:03:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.093+0000 7fa8ff7fe700 1 -- 172.21.15.102:0/2029558970 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa908017030 con 0x7fa9180cd390 2023-03-03T12:03:36.095 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:03:36.095 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:03:36.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9040682f0 msgr2=0x7fa90406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9040682f0 0x7fa90406a7a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009c10 tx=0x7fa90c008030 comp rx=0 tx=0).stop 2023-03-03T12:03:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd390 msgr2=0x7fa9180c81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd390 0x7fa9180c81d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c580 tx=0x7fa90800c890 comp rx=0 tx=0).stop 2023-03-03T12:03:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 shutdown_connections 2023-03-03T12:03:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9040682f0 0x7fa90406a7a0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa918153290 0x7fa9181ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9180cd390 0x7fa9180c81d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 --2- 172.21.15.102:0/2029558970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9180cd8e0 0x7fa9180b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 >> 172.21.15.102:0/2029558970 conn(0x7fa918065fb0 msgr2=0x7fa9180cbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:03:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 shutdown_connections 2023-03-03T12:03:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:03:36.096+0000 7fa91d1ff700 1 -- 172.21.15.102:0/2029558970 wait complete. 2023-03-03T12:03:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:03:36.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2023-03-03T12:03:48.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:48.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:18.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:18.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:48.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:48.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:18.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:18.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:28.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:05:28.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.376+0000 7fa26559d700 1 Processor -- start 2023-03-03T12:05:28.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.376+0000 7fa26559d700 1 -- start start 2023-03-03T12:05:28.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa26559d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa2600ced00 0x7fa2600c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:28.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa26559d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2600b3970 0x7fa2600c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:28.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa26559d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600c3d20 con 0x7fa2600cf0d0 2023-03-03T12:05:28.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa26559d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2600c4930 con 0x7fa2600b3970 2023-03-03T12:05:28.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa26559d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2600cf570 con 0x7fa2600ced00 2023-03-03T12:05:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa257fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2600b3970 0x7fa2600c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa257fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2600b3970 0x7fa2600c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50344/0 (socket says 172.21.15.102:50344) 2023-03-03T12:05:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa257fff700 1 -- 172.21.15.102:0/557339149 learned_addr learned my addr 172.21.15.102:0/557339149 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:05:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.378+0000 7fa25ffff700 1 --2- 172.21.15.102:0/557339149 >> v2:172.21.15.172:3300/0 conn(0x7fa2600ced00 0x7fa2600c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:28.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa257fff700 1 -- 172.21.15.102:0/557339149 >> v2:172.21.15.172:3300/0 conn(0x7fa2600ced00 msgr2=0x7fa2600c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:05:28.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa257fff700 1 --2- 172.21.15.102:0/557339149 >> v2:172.21.15.172:3300/0 conn(0x7fa2600ced00 0x7fa2600c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa257fff700 1 -- 172.21.15.102:0/557339149 >> v1:172.21.15.172:6789/0 conn(0x7fa2600cf0d0 legacy=0x7fa260158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:28.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa257fff700 1 -- 172.21.15.102:0/557339149 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26015cba0 con 0x7fa2600b3970 2023-03-03T12:05:28.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa257fff700 1 --2- 172.21.15.102:0/557339149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600b3970 0x7fa2600c41b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa2600b1b40 tx=0x7fa248009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e2cbca0fd7eb6c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:28.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa25f7fe700 1 -- 172.21.15.102:0/557339149 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248011830 con 0x7fa2600b3970 2023-03-03T12:05:28.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa25f7fe700 1 -- 172.21.15.102:0/557339149 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa248011990 con 0x7fa2600b3970 2023-03-03T12:05:28.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.379+0000 7fa25f7fe700 1 -- 172.21.15.102:0/557339149 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248011d20 con 0x7fa2600b3970 2023-03-03T12:05:28.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 -- 172.21.15.102:0/557339149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600b3970 msgr2=0x7fa2600c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:28.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 --2- 172.21.15.102:0/557339149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600b3970 0x7fa2600c41b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa2600b1b40 tx=0x7fa248009bd0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 -- 172.21.15.102:0/557339149 shutdown_connections 2023-03-03T12:05:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 --2- 172.21.15.102:0/557339149 >> v2:172.21.15.172:3300/0 conn(0x7fa2600ced00 0x7fa2600c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 --2- 172.21.15.102:0/557339149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600b3970 0x7fa2600c41b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 -- 172.21.15.102:0/557339149 >> 172.21.15.102:0/557339149 conn(0x7fa26008b1c0 msgr2=0x7fa260071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 -- 172.21.15.102:0/557339149 shutdown_connections 2023-03-03T12:05:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.380+0000 7fa26559d700 1 -- 172.21.15.102:0/557339149 wait complete. 2023-03-03T12:05:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.382+0000 7fa26559d700 1 Processor -- start 2023-03-03T12:05:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.382+0000 7fa26559d700 1 -- start start 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.383+0000 7fa26559d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2600b3970 0x7fa2601e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.383+0000 7fa26559d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2600ced00 0x7fa2600b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.383+0000 7fa26559d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600cf0d0 0x7fa2600c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.383+0000 7fa26559d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2601f2cd0 con 0x7fa2600cf0d0 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.383+0000 7fa26559d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600c82b0 con 0x7fa2600ced00 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.383+0000 7fa26559d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2601f20e0 con 0x7fa2600b3970 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600cf0d0 0x7fa2600c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:28.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600cf0d0 0x7fa2600c6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50356/0 (socket says 172.21.15.102:50356) 2023-03-03T12:05:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 -- 172.21.15.102:0/3540867424 learned_addr learned my addr 172.21.15.102:0/3540867424 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:05:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa25ffff700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2600b3970 0x7fa2601e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa257fff700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2600ced00 0x7fa2600b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 -- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2600b3970 msgr2=0x7fa2601e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2600b3970 0x7fa2601e5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 -- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2600ced00 msgr2=0x7fa2600b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2600ced00 0x7fa2600b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 -- 172.21.15.102:0/3540867424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248009550 con 0x7fa2600cf0d0 2023-03-03T12:05:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.384+0000 7fa264d9c700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600cf0d0 0x7fa2600c6070 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa25800cb90 tx=0x7fa25800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.385+0000 7fa25d7fa700 1 -- 172.21.15.102:0/3540867424 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258003c50 con 0x7fa2600cf0d0 2023-03-03T12:05:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.385+0000 7fa25d7fa700 1 -- 172.21.15.102:0/3540867424 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa258003dd0 con 0x7fa2600cf0d0 2023-03-03T12:05:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.385+0000 7fa25d7fa700 1 -- 172.21.15.102:0/3540867424 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25800aae0 con 0x7fa2600cf0d0 2023-03-03T12:05:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.385+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26006bf20 con 0x7fa2600cf0d0 2023-03-03T12:05:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.385+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26015ce80 con 0x7fa2600cf0d0 2023-03-03T12:05:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.386+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26006c200 con 0x7fa2600cf0d0 2023-03-03T12:05:28.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.387+0000 7fa25d7fa700 1 -- 172.21.15.102:0/3540867424 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa258010050 con 0x7fa2600cf0d0 2023-03-03T12:05:28.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.389+0000 7fa25d7fa700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2400682f0 0x7fa24006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:28.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.389+0000 7fa25d7fa700 1 -- 172.21.15.102:0/3540867424 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa258085c50 con 0x7fa2600cf0d0 2023-03-03T12:05:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.389+0000 7fa25ffff700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2400682f0 0x7fa24006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.390+0000 7fa25ffff700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2400682f0 0x7fa24006a7a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa250009b40 tx=0x7fa250009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.393+0000 7fa25d7fa700 1 -- 172.21.15.102:0/3540867424 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa258052360 con 0x7fa2600cf0d0 2023-03-03T12:05:28.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.630+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2601f2e10 con 0x7fa2600cf0d0 2023-03-03T12:05:28.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.630+0000 7fa25d7fa700 1 -- 172.21.15.102:0/3540867424 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa2601f2e10 con 0x7fa2600cf0d0 2023-03-03T12:05:28.632 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:05:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:05:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2400682f0 msgr2=0x7fa24006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2400682f0 0x7fa24006a7a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa250009b40 tx=0x7fa250009210 comp rx=0 tx=0).stop 2023-03-03T12:05:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600cf0d0 msgr2=0x7fa2600c6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600cf0d0 0x7fa2600c6070 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa25800cb90 tx=0x7fa25800cea0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 shutdown_connections 2023-03-03T12:05:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2400682f0 0x7fa24006a7a0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2600b3970 0x7fa2601e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2600cf0d0 0x7fa2600c6070 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 --2- 172.21.15.102:0/3540867424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2600ced00 0x7fa2600b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.633+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 >> 172.21.15.102:0/3540867424 conn(0x7fa26008b1c0 msgr2=0x7fa2600c4100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.634+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 shutdown_connections 2023-03-03T12:05:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:05:28.634+0000 7fa26559d700 1 -- 172.21.15.102:0/3540867424 wait complete. 2023-03-03T12:05:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:05:28.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 93 secs before thrashing 2023-03-03T12:05:49.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:49.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:19.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:19.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:49.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:49.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:01.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:07:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.926+0000 7f217efff700 1 Processor -- start 2023-03-03T12:07:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.926+0000 7f217efff700 1 -- start start 2023-03-03T12:07:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.927+0000 7f217efff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21780cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.927+0000 7f217efff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2178153660 con 0x7f21780cd390 2023-03-03T12:07:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.927+0000 7f217efff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217806c360 con 0x7f2178153290 2023-03-03T12:07:01.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.927+0000 7f217efff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217815cea0 con 0x7f21780cd8e0 2023-03-03T12:07:01.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.928+0000 7f2177fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21780cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.928+0000 7f217cd9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2178153290 0x7f2178158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57690/0 (socket says 172.21.15.102:57690) 2023-03-03T12:07:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.928+0000 7f217cd9b700 1 -- 172.21.15.102:0/223518329 learned_addr learned my addr 172.21.15.102:0/223518329 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:07:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.928+0000 7f2177fff700 1 -- 172.21.15.102:0/223518329 >> v1:172.21.15.172:6789/0 conn(0x7f2178153290 legacy=0x7f2178158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.928+0000 7f2177fff700 1 -- 172.21.15.102:0/223518329 >> v1:172.21.15.102:6789/0 conn(0x7f21780cd390 legacy=0x7f21780ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.928+0000 7f21777fe700 1 -- 172.21.15.102:0/223518329 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900185802 0 0) 0x7f217806c360 con 0x7f2178153290 2023-03-03T12:07:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.928+0000 7f2177fff700 1 -- 172.21.15.102:0/223518329 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217806c7f0 con 0x7f21780cd8e0 2023-03-03T12:07:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.929+0000 7f2177fff700 1 --2- 172.21.15.102:0/223518329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21780cde20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2164009870 tx=0x7f2164009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6940f948ac94d7d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.929+0000 7f21777fe700 1 -- 172.21.15.102:0/223518329 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164011830 con 0x7f21780cd8e0 2023-03-03T12:07:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.929+0000 7f21777fe700 1 -- 172.21.15.102:0/223518329 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2164011990 con 0x7f21780cd8e0 2023-03-03T12:07:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.929+0000 7f21777fe700 1 -- 172.21.15.102:0/223518329 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164011d20 con 0x7f21780cd8e0 2023-03-03T12:07:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.929+0000 7f217efff700 1 -- 172.21.15.102:0/223518329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 msgr2=0x7f21780cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.929+0000 7f217efff700 1 --2- 172.21.15.102:0/223518329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21780cde20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2164009870 tx=0x7f2164009b80 comp rx=0 tx=0).stop 2023-03-03T12:07:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.930+0000 7f217efff700 1 -- 172.21.15.102:0/223518329 shutdown_connections 2023-03-03T12:07:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.930+0000 7f217efff700 1 --2- 172.21.15.102:0/223518329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21780cde20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.930+0000 7f217efff700 1 -- 172.21.15.102:0/223518329 >> 172.21.15.102:0/223518329 conn(0x7f2178065fb0 msgr2=0x7f21780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:07:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.930+0000 7f217efff700 1 -- 172.21.15.102:0/223518329 shutdown_connections 2023-03-03T12:07:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.930+0000 7f217efff700 1 -- 172.21.15.102:0/223518329 wait complete. 2023-03-03T12:07:01.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.932+0000 7f217efff700 1 Processor -- start 2023-03-03T12:07:01.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.932+0000 7f217efff700 1 -- start start 2023-03-03T12:07:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217efff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21780cd390 0x7f21781ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217efff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21781eee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217efff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2178153290 0x7f21780b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217efff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21780cf8c0 con 0x7f2178153290 2023-03-03T12:07:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217efff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21780b22c0 con 0x7f21780cd390 2023-03-03T12:07:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217efff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21780b1e30 con 0x7f21780cd8e0 2023-03-03T12:07:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217d59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2178153290 0x7f21780b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217cd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21780cd390 0x7f21781ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f2177fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21781eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217d59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2178153290 0x7f21780b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50658/0 (socket says 172.21.15.102:50658) 2023-03-03T12:07:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f2177fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21781eee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55602/0 (socket says 172.21.15.102:55602) 2023-03-03T12:07:01.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.933+0000 7f217d59c700 1 -- 172.21.15.102:0/3723144961 learned_addr learned my addr 172.21.15.102:0/3723144961 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:07:01.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f2177fff700 1 -- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21780cd390 msgr2=0x7f21781ee8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:01.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f2177fff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21780cd390 0x7f21781ee8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:01.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f2177fff700 1 -- 172.21.15.102:0/3723144961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2178153290 msgr2=0x7f21780b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:01.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f2177fff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2178153290 0x7f21780b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:01.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f2177fff700 1 -- 172.21.15.102:0/3723144961 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21640094a0 con 0x7f21780cd8e0 2023-03-03T12:07:01.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f217d59c700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2178153290 0x7f21780b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:07:01.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f2177fff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21781eee30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2164016000 tx=0x7f2164004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:01.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f21757fa700 1 -- 172.21.15.102:0/3723144961 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21640036a0 con 0x7f21780cd8e0 2023-03-03T12:07:01.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f21757fa700 1 -- 172.21.15.102:0/3723144961 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216401f070 con 0x7f21780cd8e0 2023-03-03T12:07:01.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f21757fa700 1 -- 172.21.15.102:0/3723144961 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164003a20 con 0x7f21780cd8e0 2023-03-03T12:07:01.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.934+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21780ccde0 con 0x7f21780cd8e0 2023-03-03T12:07:01.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.935+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217815d160 con 0x7f21780cd8e0 2023-03-03T12:07:01.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.935+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21780cd0c0 con 0x7f21780cd8e0 2023-03-03T12:07:01.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.937+0000 7f21757fa700 1 -- 172.21.15.102:0/3723144961 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2164047070 con 0x7f21780cd8e0 2023-03-03T12:07:01.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.938+0000 7f21757fa700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f215c068320 0x7f215c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:01.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.938+0000 7f21757fa700 1 -- 172.21.15.102:0/3723144961 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2164021070 con 0x7f21780cd8e0 2023-03-03T12:07:01.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.938+0000 7f217cd9b700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f215c068320 0x7f215c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:01.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.939+0000 7f217cd9b700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f215c068320 0x7f215c06a7d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f216c000f60 tx=0x7f216c004a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:01.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:01.942+0000 7f21757fa700 1 -- 172.21.15.102:0/3723144961 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f21640546b0 con 0x7f21780cd8e0 2023-03-03T12:07:02.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.175+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21780b1fe0 con 0x7f21780cd8e0 2023-03-03T12:07:02.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.176+0000 7f21757fa700 1 -- 172.21.15.102:0/3723144961 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f21780b1fe0 con 0x7f21780cd8e0 2023-03-03T12:07:02.178 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:07:02.179 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:07:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f215c068320 msgr2=0x7f215c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f215c068320 0x7f215c06a7d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f216c000f60 tx=0x7f216c004a50 comp rx=0 tx=0).stop 2023-03-03T12:07:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 msgr2=0x7f21781eee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:02.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21781eee30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2164016000 tx=0x7f2164004750 comp rx=0 tx=0).stop 2023-03-03T12:07:02.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 shutdown_connections 2023-03-03T12:07:02.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2178153290 0x7f21780b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:02.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21780cd390 0x7f21781ee8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:02.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f215c068320 0x7f215c06a7d0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:02.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 --2- 172.21.15.102:0/3723144961 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f21780cd8e0 0x7f21781eee30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:02.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 >> 172.21.15.102:0/3723144961 conn(0x7f2178065fb0 msgr2=0x7f21781589d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:07:02.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 shutdown_connections 2023-03-03T12:07:02.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:07:02.179+0000 7f217efff700 1 -- 172.21.15.102:0/3723144961 wait complete. 2023-03-03T12:07:02.183 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:07:02.208 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2023-03-03T12:07:19.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:19.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:49.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:49.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:20.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:20.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:41.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:08:41.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.458+0000 7f754c3ff700 1 Processor -- start 2023-03-03T12:08:41.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.458+0000 7f754c3ff700 1 -- start start 2023-03-03T12:08:41.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.459+0000 7f754c3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7544153290 0x7f7544158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:41.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f754c3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7544153660 con 0x7f75440cd8e0 2023-03-03T12:08:41.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f754c3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754406c360 con 0x7f75441593c0 2023-03-03T12:08:41.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f754c3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75440cec40 con 0x7f7544153290 2023-03-03T12:08:41.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f754a99c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f75440cd8e0 0x7f75440ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42956/0 (socket says 172.21.15.102:42956) 2023-03-03T12:08:41.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f754a99c700 1 -- 172.21.15.102:0/2211455320 learned_addr learned my addr 172.21.15.102:0/2211455320 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:08:41.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f754a19b700 1 --2- 172.21.15.102:0/2211455320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7544153290 0x7f7544158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:41.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f7549199700 1 -- 172.21.15.102:0/2211455320 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251086994 0 0) 0x7f754406c360 con 0x7f75441593c0 2023-03-03T12:08:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f7549199700 1 -- 172.21.15.102:0/2211455320 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75300035b0 con 0x7f75441593c0 2023-03-03T12:08:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f754a19b700 1 -- 172.21.15.102:0/2211455320 >> v1:172.21.15.172:6789/0 conn(0x7f75441593c0 legacy=0x7f75440cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.460+0000 7f7549199700 1 -- 172.21.15.102:0/2211455320 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536496712 0 0) 0x7f7544153660 con 0x7f75440cd8e0 2023-03-03T12:08:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.461+0000 7f754a19b700 1 -- 172.21.15.102:0/2211455320 >> v1:172.21.15.102:6789/0 conn(0x7f75440cd8e0 legacy=0x7f75440ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.461+0000 7f754a19b700 1 -- 172.21.15.102:0/2211455320 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754415cea0 con 0x7f7544153290 2023-03-03T12:08:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.461+0000 7f754a19b700 1 --2- 172.21.15.102:0/2211455320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7544153290 0x7f7544158e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7540009870 tx=0x7f7540009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eccc3cb1f61c667c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:08:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.461+0000 7f7549199700 1 -- 172.21.15.102:0/2211455320 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540011830 con 0x7f7544153290 2023-03-03T12:08:41.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.461+0000 7f7549199700 1 -- 172.21.15.102:0/2211455320 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7540011990 con 0x7f7544153290 2023-03-03T12:08:41.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.461+0000 7f7549199700 1 -- 172.21.15.102:0/2211455320 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540011d20 con 0x7f7544153290 2023-03-03T12:08:41.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.462+0000 7f754c3ff700 1 -- 172.21.15.102:0/2211455320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7544153290 msgr2=0x7f7544158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:41.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.462+0000 7f754c3ff700 1 --2- 172.21.15.102:0/2211455320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7544153290 0x7f7544158e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7540009870 tx=0x7f7540009b80 comp rx=0 tx=0).stop 2023-03-03T12:08:41.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.462+0000 7f754c3ff700 1 -- 172.21.15.102:0/2211455320 shutdown_connections 2023-03-03T12:08:41.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.462+0000 7f754c3ff700 1 --2- 172.21.15.102:0/2211455320 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7544153290 0x7f7544158e80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:41.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.462+0000 7f754c3ff700 1 -- 172.21.15.102:0/2211455320 >> 172.21.15.102:0/2211455320 conn(0x7f7544065fb0 msgr2=0x7f75440663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:08:41.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.462+0000 7f754c3ff700 1 -- 172.21.15.102:0/2211455320 shutdown_connections 2023-03-03T12:08:41.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.462+0000 7f754c3ff700 1 -- 172.21.15.102:0/2211455320 wait complete. 2023-03-03T12:08:41.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.464+0000 7f754c3ff700 1 Processor -- start 2023-03-03T12:08:41.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.464+0000 7f754c3ff700 1 -- start start 2023-03-03T12:08:41.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754c3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75440cd8e0 0x7f75441f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754c3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7544153290 0x7f75441ede20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754c3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75441593c0 0x7f7544155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754c3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75441ee360 con 0x7f75441593c0 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754c3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75440c8870 con 0x7f7544153290 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754c3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75440c8b30 con 0x7f75440cd8e0 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754a99c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75441593c0 0x7f7544155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7544153290 0x7f75441ede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:41.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75440cd8e0 0x7f75441f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:41.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7544153290 0x7f75441ede20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33550/0 (socket says 172.21.15.102:33550) 2023-03-03T12:08:41.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75440cd8e0 0x7f75441f1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38690/0 (socket says 172.21.15.102:38690) 2023-03-03T12:08:41.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754a19b700 1 -- 172.21.15.102:0/117871892 learned_addr learned my addr 172.21.15.102:0/117871892 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:08:41.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.465+0000 7f754a99c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75441593c0 0x7f7544155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48098/0 (socket says 172.21.15.102:48098) 2023-03-03T12:08:41.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f754a19b700 1 -- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7544153290 msgr2=0x7f75441ede20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:41.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f754a19b700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7544153290 0x7f75441ede20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:41.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f754a19b700 1 -- 172.21.15.102:0/117871892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75441593c0 msgr2=0x7f7544155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:41.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f754a19b700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75441593c0 0x7f7544155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:41.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f754a19b700 1 -- 172.21.15.102:0/117871892 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75400094a0 con 0x7f75440cd8e0 2023-03-03T12:08:41.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f754999a700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7544153290 0x7f75441ede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:08:41.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f754a19b700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75440cd8e0 0x7f75441f1e70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7540014040 tx=0x7f7540004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:08:41.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.466+0000 7f753affd700 1 -- 172.21.15.102:0/117871892 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754001a7e0 con 0x7f75440cd8e0 2023-03-03T12:08:41.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.467+0000 7f753affd700 1 -- 172.21.15.102:0/117871892 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754001a940 con 0x7f75440cd8e0 2023-03-03T12:08:41.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.467+0000 7f753affd700 1 -- 172.21.15.102:0/117871892 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754001acd0 con 0x7f75440cd8e0 2023-03-03T12:08:41.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.467+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754415d160 con 0x7f75440cd8e0 2023-03-03T12:08:41.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.467+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754406c7f0 con 0x7f75440cd8e0 2023-03-03T12:08:41.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.468+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75440ccde0 con 0x7f75440cd8e0 2023-03-03T12:08:41.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.469+0000 7f753affd700 1 -- 172.21.15.102:0/117871892 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f754000f050 con 0x7f75440cd8e0 2023-03-03T12:08:41.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.470+0000 7f753affd700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75280682f0 0x7f752806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:41.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.470+0000 7f753affd700 1 -- 172.21.15.102:0/117871892 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7540086c00 con 0x7f75440cd8e0 2023-03-03T12:08:41.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.470+0000 7f754999a700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75280682f0 0x7f752806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:41.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.471+0000 7f754999a700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75280682f0 0x7f752806a7a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f75440cc8b0 tx=0x7f7534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:08:41.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.475+0000 7f753affd700 1 -- 172.21.15.102:0/117871892 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7540053310 con 0x7f75440cd8e0 2023-03-03T12:08:41.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.707+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f754404e6e0 con 0x7f75440cd8e0 2023-03-03T12:08:41.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.707+0000 7f753affd700 1 -- 172.21.15.102:0/117871892 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7540029090 con 0x7f75440cd8e0 2023-03-03T12:08:41.709 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:08:41.710 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:08:41.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75280682f0 msgr2=0x7f752806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:41.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75280682f0 0x7f752806a7a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f75440cc8b0 tx=0x7f7534009040 comp rx=0 tx=0).stop 2023-03-03T12:08:41.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75440cd8e0 msgr2=0x7f75441f1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:41.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75440cd8e0 0x7f75441f1e70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7540014040 tx=0x7f7540004750 comp rx=0 tx=0).stop 2023-03-03T12:08:41.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 shutdown_connections 2023-03-03T12:08:41.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75280682f0 0x7f752806a7a0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:41.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75440cd8e0 0x7f75441f1e70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:41.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75441593c0 0x7f7544155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:41.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 --2- 172.21.15.102:0/117871892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7544153290 0x7f75441ede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:41.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 >> 172.21.15.102:0/117871892 conn(0x7f7544065fb0 msgr2=0x7f7544156b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:08:41.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.710+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 shutdown_connections 2023-03-03T12:08:41.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:08:41.711+0000 7f754c3ff700 1 -- 172.21.15.102:0/117871892 wait complete. 2023-03-03T12:08:41.714 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:08:41.740 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 26 secs before thrashing 2023-03-03T12:08:50.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:50.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:07.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:09:07.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.991+0000 7f9dd2fff700 1 Processor -- start 2023-03-03T12:09:07.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.991+0000 7f9dd2fff700 1 -- start start 2023-03-03T12:09:07.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.992+0000 7f9dd2fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9dcc153290 0x7f9dcc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:07.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.992+0000 7f9dd2fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc1593c0 0x7f9dcc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:07.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.992+0000 7f9dd2fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc153660 con 0x7f9dcc0cd8e0 2023-03-03T12:09:07.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.992+0000 7f9dd2fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dcc06c360 con 0x7f9dcc153290 2023-03-03T12:09:07.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.992+0000 7f9dd2fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc15cea0 con 0x7f9dcc1593c0 2023-03-03T12:09:07.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dd159c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9dcc0cd8e0 0x7f9dcc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41034/0 (socket says 172.21.15.102:41034) 2023-03-03T12:09:07.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dd159c700 1 -- 172.21.15.102:0/3098418999 learned_addr learned my addr 172.21.15.102:0/3098418999 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:09:07.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dd0d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9dcc153290 0x7f9dcc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:07.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dcbfff700 1 --2- 172.21.15.102:0/3098418999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc1593c0 0x7f9dcc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:07.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dcb7fe700 1 -- 172.21.15.102:0/3098418999 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222931306 0 0) 0x7f9dcc153660 con 0x7f9dcc0cd8e0 2023-03-03T12:09:07.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dcb7fe700 1 -- 172.21.15.102:0/3098418999 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dbc0035b0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:07.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dd0d9b700 1 -- 172.21.15.102:0/3098418999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc1593c0 msgr2=0x7f9dcc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:09:07.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dd0d9b700 1 --2- 172.21.15.102:0/3098418999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc1593c0 0x7f9dcc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:07.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dd0d9b700 1 -- 172.21.15.102:0/3098418999 >> v1:172.21.15.102:6789/0 conn(0x7f9dcc0cd8e0 legacy=0x7f9dcc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dd0d9b700 1 -- 172.21.15.102:0/3098418999 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc06c7f0 con 0x7f9dcc153290 2023-03-03T12:09:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.993+0000 7f9dcbfff700 1 --2- 172.21.15.102:0/3098418999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc1593c0 0x7f9dcc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:09:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.994+0000 7f9dd0d9b700 1 --2- 172.21.15.102:0/3098418999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc153290 0x7f9dcc158e80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009840 tx=0x7f9dc0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1821e5d9526b17c7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.994+0000 7f9dcb7fe700 1 -- 172.21.15.102:0/3098418999 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0011830 con 0x7f9dcc153290 2023-03-03T12:09:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.994+0000 7f9dcb7fe700 1 -- 172.21.15.102:0/3098418999 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0011990 con 0x7f9dcc153290 2023-03-03T12:09:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.994+0000 7f9dcb7fe700 1 -- 172.21.15.102:0/3098418999 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0011d20 con 0x7f9dcc153290 2023-03-03T12:09:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.994+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3098418999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc153290 msgr2=0x7f9dcc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:07.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.994+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3098418999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc153290 0x7f9dcc158e80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009840 tx=0x7f9dc0009b50 comp rx=0 tx=0).stop 2023-03-03T12:09:07.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.995+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3098418999 shutdown_connections 2023-03-03T12:09:07.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.995+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3098418999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc153290 0x7f9dcc158e80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:07.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.995+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3098418999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc1593c0 0x7f9dcc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:07.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.995+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3098418999 >> 172.21.15.102:0/3098418999 conn(0x7f9dcc065fb0 msgr2=0x7f9dcc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:07.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.995+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3098418999 shutdown_connections 2023-03-03T12:09:07.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.995+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3098418999 wait complete. 2023-03-03T12:09:08.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.997+0000 7f9dd2fff700 1 Processor -- start 2023-03-03T12:09:08.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.997+0000 7f9dd2fff700 1 -- start start 2023-03-03T12:09:08.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd2fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dcc0cd8e0 0x7f9dcc149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:08.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd2fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc153290 0x7f9dcc1f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:08.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd2fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc1593c0 0x7f9dcc1f1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:08.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd2fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc14c8c0 con 0x7f9dcc1593c0 2023-03-03T12:09:08.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd2fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc14bca0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd2fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc149d60 con 0x7f9dcc153290 2023-03-03T12:09:08.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd0d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dcc0cd8e0 0x7f9dcc149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:08.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dcbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc153290 0x7f9dcc1f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:08.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd0d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dcc0cd8e0 0x7f9dcc149820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33550/0 (socket says 172.21.15.102:33550) 2023-03-03T12:09:08.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd0d9b700 1 -- 172.21.15.102:0/3700004360 learned_addr learned my addr 172.21.15.102:0/3700004360 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:09:08.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.998+0000 7f9dd159c700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc1593c0 0x7f9dcc1f1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:08.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dd0d9b700 1 -- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc153290 msgr2=0x7f9dcc1f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:08.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dd0d9b700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc153290 0x7f9dcc1f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:08.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dd0d9b700 1 -- 172.21.15.102:0/3700004360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc1593c0 msgr2=0x7f9dcc1f1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:08.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dd0d9b700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc1593c0 0x7f9dcc1f1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:08.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dd0d9b700 1 -- 172.21.15.102:0/3700004360 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80094a0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dd0d9b700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dcc0cd8e0 0x7f9dcc149820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0000c00 tx=0x7f9dc0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:08.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dc97fa700 1 -- 172.21.15.102:0/3700004360 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0004590 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dc97fa700 1 -- 172.21.15.102:0/3700004360 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc001f070 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc00094a0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:07.999+0000 7f9dc97fa700 1 -- 172.21.15.102:0/3700004360 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc00038e0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.000+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc0ccde0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.001+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc15d160 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.002+0000 7f9dc97fa700 1 -- 172.21.15.102:0/3700004360 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9dc000f050 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.003+0000 7f9dc97fa700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9db0068320 0x7f9db006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:08.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.003+0000 7f9dc97fa700 1 -- 172.21.15.102:0/3700004360 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9dc0021070 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.003+0000 7f9dcbfff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9db0068320 0x7f9db006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:08.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.004+0000 7f9dcbfff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9db0068320 0x7f9db006a7d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc15b9b0 tx=0x7f9db8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:08.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.007+0000 7f9dc97fa700 1 -- 172.21.15.102:0/3700004360 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9dc0053cc0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.240+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dcc0cd0c0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.241+0000 7f9dc97fa700 1 -- 172.21.15.102:0/3700004360 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9dc00572e0 con 0x7f9dcc0cd8e0 2023-03-03T12:09:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:09:08.243 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:09:08.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9db0068320 msgr2=0x7f9db006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:08.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9db0068320 0x7f9db006a7d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc15b9b0 tx=0x7f9db8009210 comp rx=0 tx=0).stop 2023-03-03T12:09:08.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dcc0cd8e0 msgr2=0x7f9dcc149820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:08.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dcc0cd8e0 0x7f9dcc149820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0000c00 tx=0x7f9dc0004750 comp rx=0 tx=0).stop 2023-03-03T12:09:08.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 shutdown_connections 2023-03-03T12:09:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9dcc1593c0 0x7f9dcc1f1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dcc0cd8e0 0x7f9dcc149820 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9db0068320 0x7f9db006a7d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 --2- 172.21.15.102:0/3700004360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9dcc153290 0x7f9dcc1f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 >> 172.21.15.102:0/3700004360 conn(0x7f9dcc065fb0 msgr2=0x7f9dcc15ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 shutdown_connections 2023-03-03T12:09:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:09:08.244+0000 7f9dd2fff700 1 -- 172.21.15.102:0/3700004360 wait complete. 2023-03-03T12:09:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:09:08.274 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T12:09:20.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:20.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:50.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:50.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T12:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:10:00.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:10:00.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.545+0000 7f6830fff700 1 Processor -- start 2023-03-03T12:10:00.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.545+0000 7f6830fff700 1 -- start start 2023-03-03T12:10:00.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.546+0000 7f6830fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c153290 0x7f682c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:00.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.546+0000 7f6830fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f682c0cd8e0 0x7f682c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:00.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f6830fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682c153660 con 0x7f682c1593c0 2023-03-03T12:10:00.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f6830fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682c06c360 con 0x7f682c0cd8e0 2023-03-03T12:10:00.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f6830fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c15cea0 con 0x7f682c153290 2023-03-03T12:10:00.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682ad9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f682c0cd8e0 0x7f682c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:00.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682ad9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f682c0cd8e0 0x7f682c0ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45992/0 (socket says 172.21.15.102:45992) 2023-03-03T12:10:00.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682ad9d700 1 -- 172.21.15.102:0/333279339 learned_addr learned my addr 172.21.15.102:0/333279339 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:10:00.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f6829d9b700 1 --1- 172.21.15.102:0/333279339 >> v1:172.21.15.172:6789/0 conn(0x7f682c1593c0 0x7f682c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49710/0 (socket says 172.21.15.102:49710) 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682a59c700 1 --2- 172.21.15.102:0/333279339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c153290 0x7f682c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682ad9d700 1 -- 172.21.15.102:0/333279339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c153290 msgr2=0x7f682c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682ad9d700 1 --2- 172.21.15.102:0/333279339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c153290 0x7f682c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682959a700 1 -- 172.21.15.102:0/333279339 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627306668 0 0) 0x7f682c153660 con 0x7f682c1593c0 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682ad9d700 1 -- 172.21.15.102:0/333279339 >> v1:172.21.15.172:6789/0 conn(0x7f682c1593c0 legacy=0x7f682c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.547+0000 7f682ad9d700 1 -- 172.21.15.102:0/333279339 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c06c7f0 con 0x7f682c0cd8e0 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.548+0000 7f682ad9d700 1 --2- 172.21.15.102:0/333279339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c0cd8e0 0x7f682c0ce530 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6824009e50 tx=0x7f682400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4b2d203bd5cc44 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:10:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.548+0000 7f682959a700 1 -- 172.21.15.102:0/333279339 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682400e070 con 0x7f682c0cd8e0 2023-03-03T12:10:00.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.548+0000 7f682959a700 1 -- 172.21.15.102:0/333279339 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682400b910 con 0x7f682c0cd8e0 2023-03-03T12:10:00.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.548+0000 7f682959a700 1 -- 172.21.15.102:0/333279339 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682400bcc0 con 0x7f682c0cd8e0 2023-03-03T12:10:00.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 -- 172.21.15.102:0/333279339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c0cd8e0 msgr2=0x7f682c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:00.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 --2- 172.21.15.102:0/333279339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c0cd8e0 0x7f682c0ce530 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6824009e50 tx=0x7f682400b330 comp rx=0 tx=0).stop 2023-03-03T12:10:00.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 -- 172.21.15.102:0/333279339 shutdown_connections 2023-03-03T12:10:00.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 --2- 172.21.15.102:0/333279339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c0cd8e0 0x7f682c0ce530 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 --2- 172.21.15.102:0/333279339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c153290 0x7f682c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 -- 172.21.15.102:0/333279339 >> 172.21.15.102:0/333279339 conn(0x7f682c065fb0 msgr2=0x7f682c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:10:00.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 -- 172.21.15.102:0/333279339 shutdown_connections 2023-03-03T12:10:00.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.549+0000 7f6830fff700 1 -- 172.21.15.102:0/333279339 wait complete. 2023-03-03T12:10:00.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.551+0000 7f6830fff700 1 Processor -- start 2023-03-03T12:10:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.551+0000 7f6830fff700 1 -- start start 2023-03-03T12:10:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.552+0000 7f6830fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c0cd8e0 0x7f682c1e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.552+0000 7f6830fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f682c153290 0x7f682c0c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.552+0000 7f6830fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c1593c0 0x7f682c0c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.552+0000 7f6830fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0c8e90 con 0x7f682c1593c0 2023-03-03T12:10:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.552+0000 7f6830fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0c6d00 con 0x7f682c153290 2023-03-03T12:10:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.552+0000 7f6830fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0c7880 con 0x7f682c0cd8e0 2023-03-03T12:10:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c1593c0 0x7f682c0c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682a59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c0cd8e0 0x7f682c1e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c1593c0 0x7f682c0c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45994/0 (socket says 172.21.15.102:45994) 2023-03-03T12:10:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 -- 172.21.15.102:0/1605390470 learned_addr learned my addr 172.21.15.102:0/1605390470 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:10:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f6829d9b700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f682c153290 0x7f682c0c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 -- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c0cd8e0 msgr2=0x7f682c1e21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c0cd8e0 0x7f682c1e21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 -- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f682c153290 msgr2=0x7f682c0c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f682c153290 0x7f682c0c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.553+0000 7f682ad9d700 1 -- 172.21.15.102:0/1605390470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824009b30 con 0x7f682c1593c0 2023-03-03T12:10:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.554+0000 7f682ad9d700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c1593c0 0x7f682c0c6680 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f682400b680 tx=0x7f6824004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:10:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.554+0000 7f681affd700 1 -- 172.21.15.102:0/1605390470 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68240048b0 con 0x7f682c1593c0 2023-03-03T12:10:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.554+0000 7f681affd700 1 -- 172.21.15.102:0/1605390470 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68240099b0 con 0x7f682c1593c0 2023-03-03T12:10:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.554+0000 7f681affd700 1 -- 172.21.15.102:0/1605390470 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682401c680 con 0x7f682c1593c0 2023-03-03T12:10:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.554+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682c0ccde0 con 0x7f682c1593c0 2023-03-03T12:10:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.554+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682c15d160 con 0x7f682c1593c0 2023-03-03T12:10:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.555+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682c0cd0c0 con 0x7f682c1593c0 2023-03-03T12:10:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.556+0000 7f681affd700 1 -- 172.21.15.102:0/1605390470 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f682402c070 con 0x7f682c1593c0 2023-03-03T12:10:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.558+0000 7f681affd700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f680c0682f0 0x7f680c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.558+0000 7f681affd700 1 -- 172.21.15.102:0/1605390470 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6824087820 con 0x7f682c1593c0 2023-03-03T12:10:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.558+0000 7f682a59c700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f680c0682f0 0x7f680c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.559+0000 7f682a59c700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f680c0682f0 0x7f680c06a7a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f681c009b40 tx=0x7f681c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:10:00.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.562+0000 7f681affd700 1 -- 172.21.15.102:0/1605390470 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6824053f30 con 0x7f682c1593c0 2023-03-03T12:10:00.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.800+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f682c0c79f0 con 0x7f682c1593c0 2023-03-03T12:10:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.801+0000 7f681affd700 1 -- 172.21.15.102:0/1605390470 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6824057550 con 0x7f682c1593c0 2023-03-03T12:10:00.803 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:10:00.803 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:10:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.803+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f680c0682f0 msgr2=0x7f680c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.803+0000 7f6830fff700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f680c0682f0 0x7f680c06a7a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f681c009b40 tx=0x7f681c009210 comp rx=0 tx=0).stop 2023-03-03T12:10:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.803+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c1593c0 msgr2=0x7f682c0c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.803+0000 7f6830fff700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c1593c0 0x7f682c0c6680 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f682400b680 tx=0x7f6824004ad0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 shutdown_connections 2023-03-03T12:10:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f680c0682f0 0x7f680c06a7a0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f682c0cd8e0 0x7f682c1e21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f682c1593c0 0x7f682c0c6680 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 --2- 172.21.15.102:0/1605390470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f682c153290 0x7f682c0c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 >> 172.21.15.102:0/1605390470 conn(0x7f682c065fb0 msgr2=0x7f682c0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:10:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 shutdown_connections 2023-03-03T12:10:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:10:00.804+0000 7f6830fff700 1 -- 172.21.15.102:0/1605390470 wait complete. 2023-03-03T12:10:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:10:00.833 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-03T12:10:20.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:21.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:51.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:51.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:11.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:11:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.082+0000 7f70acd9d700 1 Processor -- start 2023-03-03T12:11:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.082+0000 7f70acd9d700 1 -- start start 2023-03-03T12:11:12.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70acd9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70a80c9610 0x7f70a80ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70acd9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80c9ca0 con 0x7f70a80ca980 2023-03-03T12:11:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70acd9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80b46f0 con 0x7f70a81564b0 2023-03-03T12:11:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70acd9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a80b4920 con 0x7f70a80c9610 2023-03-03T12:11:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70a6ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70a80c9610 0x7f70a80ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70a7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f70a80ca980 0x7f70a81636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34640/0 (socket says 172.21.15.102:34640) 2023-03-03T12:11:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70a7fff700 1 -- 172.21.15.102:0/1438878793 learned_addr learned my addr 172.21.15.102:0/1438878793 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:11:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.084+0000 7f70a77fe700 1 --1- 172.21.15.102:0/1438878793 >> v1:172.21.15.172:6789/0 conn(0x7f70a81564b0 0x7f70a80cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56854/0 (socket says 172.21.15.102:56854) 2023-03-03T12:11:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a6ffd700 1 -- 172.21.15.102:0/1438878793 >> v1:172.21.15.172:6789/0 conn(0x7f70a81564b0 legacy=0x7f70a80cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a67fc700 1 -- 172.21.15.102:0/1438878793 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750019993 0 0) 0x7f70a80b46f0 con 0x7f70a81564b0 2023-03-03T12:11:12.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a6ffd700 1 -- 172.21.15.102:0/1438878793 >> v1:172.21.15.102:6789/0 conn(0x7f70a80ca980 legacy=0x7f70a81636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a6ffd700 1 -- 172.21.15.102:0/1438878793 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a80b4c00 con 0x7f70a80c9610 2023-03-03T12:11:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a6ffd700 1 --2- 172.21.15.102:0/1438878793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a80ca440 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f709c009840 tx=0x7f709c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc19698f806053d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a67fc700 1 -- 172.21.15.102:0/1438878793 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c004230 con 0x7f70a80c9610 2023-03-03T12:11:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a67fc700 1 -- 172.21.15.102:0/1438878793 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c004390 con 0x7f70a80c9610 2023-03-03T12:11:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.085+0000 7f70a67fc700 1 -- 172.21.15.102:0/1438878793 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c0036a0 con 0x7f70a80c9610 2023-03-03T12:11:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.086+0000 7f70acd9d700 1 -- 172.21.15.102:0/1438878793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 msgr2=0x7f70a80ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.086+0000 7f70acd9d700 1 --2- 172.21.15.102:0/1438878793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a80ca440 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f709c009840 tx=0x7f709c009b50 comp rx=0 tx=0).stop 2023-03-03T12:11:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.086+0000 7f70acd9d700 1 -- 172.21.15.102:0/1438878793 shutdown_connections 2023-03-03T12:11:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.086+0000 7f70acd9d700 1 --2- 172.21.15.102:0/1438878793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a80ca440 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.086+0000 7f70acd9d700 1 -- 172.21.15.102:0/1438878793 >> 172.21.15.102:0/1438878793 conn(0x7f70a8065fb0 msgr2=0x7f70a80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.086+0000 7f70acd9d700 1 -- 172.21.15.102:0/1438878793 shutdown_connections 2023-03-03T12:11:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.086+0000 7f70acd9d700 1 -- 172.21.15.102:0/1438878793 wait complete. 2023-03-03T12:11:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.088+0000 7f70acd9d700 1 Processor -- start 2023-03-03T12:11:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.088+0000 7f70acd9d700 1 -- start start 2023-03-03T12:11:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.089+0000 7f70acd9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a81ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.089+0000 7f70acd9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a80ca980 0x7f70a815cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.089+0000 7f70acd9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a81564b0 0x7f70a8159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.089+0000 7f70acd9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80c6f80 con 0x7f70a80c9610 2023-03-03T12:11:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.089+0000 7f70acd9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a81ef5c0 con 0x7f70a80ca980 2023-03-03T12:11:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.089+0000 7f70acd9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80c71b0 con 0x7f70a81564b0 2023-03-03T12:11:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a81ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a81ef080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57834/0 (socket says 172.21.15.102:57834) 2023-03-03T12:11:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 -- 172.21.15.102:0/3163576010 learned_addr learned my addr 172.21.15.102:0/3163576010 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:11:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a6ffd700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a80ca980 0x7f70a815cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:12.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a7fff700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a81564b0 0x7f70a8159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 -- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a81564b0 msgr2=0x7f70a8159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a81564b0 0x7f70a8159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 -- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a80ca980 msgr2=0x7f70a815cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a80ca980 0x7f70a815cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 -- 172.21.15.102:0/3163576010 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c0094a0 con 0x7f70a80c9610 2023-03-03T12:11:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.090+0000 7f70a77fe700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a81ef080 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f709800c520 tx=0x7f709800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.091+0000 7f708ffff700 1 -- 172.21.15.102:0/3163576010 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098004c80 con 0x7f70a80c9610 2023-03-03T12:11:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.091+0000 7f708ffff700 1 -- 172.21.15.102:0/3163576010 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098004e00 con 0x7f70a80c9610 2023-03-03T12:11:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.091+0000 7f708ffff700 1 -- 172.21.15.102:0/3163576010 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098007310 con 0x7f70a80c9610 2023-03-03T12:11:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.091+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a80b2210 con 0x7f70a80c9610 2023-03-03T12:11:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.091+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a80b1f30 con 0x7f70a80c9610 2023-03-03T12:11:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.092+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a8156880 con 0x7f70a80c9610 2023-03-03T12:11:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.093+0000 7f708ffff700 1 -- 172.21.15.102:0/3163576010 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f709801a020 con 0x7f70a80c9610 2023-03-03T12:11:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.094+0000 7f708ffff700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70940682f0 0x7f709406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.094+0000 7f708ffff700 1 -- 172.21.15.102:0/3163576010 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f70980858f0 con 0x7f70a80c9610 2023-03-03T12:11:12.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.095+0000 7f70a6ffd700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70940682f0 0x7f709406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:12.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.095+0000 7f70a6ffd700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70940682f0 0x7f709406a7a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f70a80c9a30 tx=0x7f709c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:12.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.099+0000 7f708ffff700 1 -- 172.21.15.102:0/3163576010 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7098052000 con 0x7f70a80c9610 2023-03-03T12:11:12.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.337+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70a80677f0 con 0x7f70a80c9610 2023-03-03T12:11:12.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.337+0000 7f708ffff700 1 -- 172.21.15.102:0/3163576010 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7098055620 con 0x7f70a80c9610 2023-03-03T12:11:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:11:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:11:12.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70940682f0 msgr2=0x7f709406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:12.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70940682f0 0x7f709406a7a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f70a80c9a30 tx=0x7f709c008030 comp rx=0 tx=0).stop 2023-03-03T12:11:12.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 msgr2=0x7f70a81ef080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:12.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a81ef080 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f709800c520 tx=0x7f709800c830 comp rx=0 tx=0).stop 2023-03-03T12:11:12.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 shutdown_connections 2023-03-03T12:11:12.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70940682f0 0x7f709406a7a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:12.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70a81564b0 0x7f70a8159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:12.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a80c9610 0x7f70a81ef080 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:12.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 --2- 172.21.15.102:0/3163576010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70a80ca980 0x7f70a815cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:12.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 >> 172.21.15.102:0/3163576010 conn(0x7f70a8065fb0 msgr2=0x7f70a8066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:12.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 shutdown_connections 2023-03-03T12:11:12.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:12.340+0000 7f70acd9d700 1 -- 172.21.15.102:0/3163576010 wait complete. 2023-03-03T12:11:12.345 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:11:12.370 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-03-03T12:11:21.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:21.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:48.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:11:48.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.618+0000 7f001e1ff700 1 Processor -- start 2023-03-03T12:11:48.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.618+0000 7f001e1ff700 1 -- start start 2023-03-03T12:11:48.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.619+0000 7f001e1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0018153290 0x7f0018158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:48.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.619+0000 7f001e1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f00180cd8e0 0x7f00180ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:48.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.619+0000 7f001e1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0018153660 con 0x7f00181593c0 2023-03-03T12:11:48.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.619+0000 7f001e1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001806c360 con 0x7f0018153290 2023-03-03T12:11:48.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.619+0000 7f001e1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001815cea0 con 0x7f00180cd8e0 2023-03-03T12:11:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0018153290 0x7f0018158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f0017fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f00180cd8e0 0x7f00180ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0018153290 0x7f0018158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36468/0 (socket says 172.21.15.102:36468) 2023-03-03T12:11:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 -- 172.21.15.102:0/2267508589 learned_addr learned my addr 172.21.15.102:0/2267508589 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:11:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 -- 172.21.15.102:0/2267508589 >> v2:172.21.15.172:3300/0 conn(0x7f00180cd8e0 msgr2=0x7f00180ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 --2- 172.21.15.102:0/2267508589 >> v2:172.21.15.172:3300/0 conn(0x7f00180cd8e0 0x7f00180ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 -- 172.21.15.102:0/2267508589 >> v1:172.21.15.102:6789/0 conn(0x7f00181593c0 legacy=0x7f00180cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 -- 172.21.15.102:0/2267508589 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001806c7f0 con 0x7f0018153290 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.620+0000 7f00177fe700 1 --2- 172.21.15.102:0/2267508589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 0x7f0018158e80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f00180c9390 tx=0x7f0008009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193a16100414ec27 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f00167fc700 1 -- 172.21.15.102:0/2267508589 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008011830 con 0x7f0018153290 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f00167fc700 1 -- 172.21.15.102:0/2267508589 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0008011990 con 0x7f0018153290 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f00167fc700 1 -- 172.21.15.102:0/2267508589 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008011d20 con 0x7f0018153290 2023-03-03T12:11:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f001e1ff700 1 -- 172.21.15.102:0/2267508589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 msgr2=0x7f0018158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:48.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f001e1ff700 1 --2- 172.21.15.102:0/2267508589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 0x7f0018158e80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f00180c9390 tx=0x7f0008009bd0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f001e1ff700 1 -- 172.21.15.102:0/2267508589 shutdown_connections 2023-03-03T12:11:48.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f001e1ff700 1 --2- 172.21.15.102:0/2267508589 >> v2:172.21.15.172:3300/0 conn(0x7f00180cd8e0 0x7f00180ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f001e1ff700 1 --2- 172.21.15.102:0/2267508589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 0x7f0018158e80 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f001e1ff700 1 -- 172.21.15.102:0/2267508589 >> 172.21.15.102:0/2267508589 conn(0x7f0018065fb0 msgr2=0x7f00180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:48.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.621+0000 7f001e1ff700 1 -- 172.21.15.102:0/2267508589 shutdown_connections 2023-03-03T12:11:48.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.622+0000 7f001e1ff700 1 -- 172.21.15.102:0/2267508589 wait complete. 2023-03-03T12:11:48.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.623+0000 7f001e1ff700 1 Processor -- start 2023-03-03T12:11:48.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.624+0000 7f001e1ff700 1 -- start start 2023-03-03T12:11:48.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f001e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00180cd8e0 0x7f00181524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:48.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f001e1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 0x7f00181f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:48.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f001e1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00181593c0 0x7f00180c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:48.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f001e1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180b43a0 con 0x7f0018153290 2023-03-03T12:11:48.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f001e1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180c8cf0 con 0x7f00181593c0 2023-03-03T12:11:48.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f001e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00180cfc30 con 0x7f00180cd8e0 2023-03-03T12:11:48.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f00177fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00180cd8e0 0x7f00181524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:48.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f0016ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 0x7f00181f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:48.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f0017fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00181593c0 0x7f00180c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f00177fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00180cd8e0 0x7f00181524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37382/0 (socket says 172.21.15.102:37382) 2023-03-03T12:11:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f00177fe700 1 -- 172.21.15.102:0/3660459352 learned_addr learned my addr 172.21.15.102:0/3660459352 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:11:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f0017fff700 1 -- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00180cd8e0 msgr2=0x7f00181524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f0017fff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00180cd8e0 0x7f00181524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f0017fff700 1 -- 172.21.15.102:0/3660459352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 msgr2=0x7f00181f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f0017fff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 0x7f00181f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.625+0000 7f0017fff700 1 -- 172.21.15.102:0/3660459352 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008009550 con 0x7f00181593c0 2023-03-03T12:11:48.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.626+0000 7f0017fff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00181593c0 0x7f00180c7780 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f001000eba0 tx=0x7f001000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:48.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.626+0000 7efffffff700 1 -- 172.21.15.102:0/3660459352 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010009a80 con 0x7f00181593c0 2023-03-03T12:11:48.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.626+0000 7efffffff700 1 -- 172.21.15.102:0/3660459352 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001000f040 con 0x7f00181593c0 2023-03-03T12:11:48.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.626+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00180ccde0 con 0x7f00181593c0 2023-03-03T12:11:48.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.626+0000 7efffffff700 1 -- 172.21.15.102:0/3660459352 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001000cc40 con 0x7f00181593c0 2023-03-03T12:11:48.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.626+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001815d160 con 0x7f00181593c0 2023-03-03T12:11:48.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.627+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00180cd0c0 con 0x7f00181593c0 2023-03-03T12:11:48.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.629+0000 7efffffff700 1 -- 172.21.15.102:0/3660459352 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0010014480 con 0x7f00181593c0 2023-03-03T12:11:48.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.630+0000 7efffffff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00040682f0 0x7f000406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:48.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.630+0000 7efffffff700 1 -- 172.21.15.102:0/3660459352 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0010086880 con 0x7f00181593c0 2023-03-03T12:11:48.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.630+0000 7f00177fe700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00040682f0 0x7f000406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:48.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.631+0000 7f00177fe700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00040682f0 0x7f000406a7a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0008009fd0 tx=0x7f0008008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:48.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.634+0000 7efffffff700 1 -- 172.21.15.102:0/3660459352 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0010052f30 con 0x7f00181593c0 2023-03-03T12:11:48.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.868+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00180b44e0 con 0x7f00181593c0 2023-03-03T12:11:48.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.869+0000 7efffffff700 1 -- 172.21.15.102:0/3660459352 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f00180b44e0 con 0x7f00181593c0 2023-03-03T12:11:48.871 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:11:48.871 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:11:48.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00040682f0 msgr2=0x7f000406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:48.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00040682f0 0x7f000406a7a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0008009fd0 tx=0x7f0008008030 comp rx=0 tx=0).stop 2023-03-03T12:11:48.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00181593c0 msgr2=0x7f00180c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00181593c0 0x7f00180c7780 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f001000eba0 tx=0x7f001000eeb0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 shutdown_connections 2023-03-03T12:11:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00040682f0 0x7f000406a7a0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f00180cd8e0 0x7f00181524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00181593c0 0x7f00180c7780 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 --2- 172.21.15.102:0/3660459352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0018153290 0x7f00181f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 >> 172.21.15.102:0/3660459352 conn(0x7f0018065fb0 msgr2=0x7f0018157840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:48.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 shutdown_connections 2023-03-03T12:11:48.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:11:48.871+0000 7f001e1ff700 1 -- 172.21.15.102:0/3660459352 wait complete. 2023-03-03T12:11:48.875 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:11:48.900 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T12:11:51.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:51.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:16.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:12:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.148+0000 7ffb5c5ff700 1 Processor -- start 2023-03-03T12:12:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.148+0000 7ffb5c5ff700 1 -- start start 2023-03-03T12:12:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5c5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb540cd8e0 0x7ffb540cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5c5ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb540cd390 0x7ffb540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5c5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb54153660 con 0x7ffb54153290 2023-03-03T12:12:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5c5ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5406c360 con 0x7ffb540cd390 2023-03-03T12:12:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5c5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5415cea0 con 0x7ffb540cd8e0 2023-03-03T12:12:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5a39b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ffb54153290 0x7ffb54158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50192/0 (socket says 172.21.15.102:50192) 2023-03-03T12:12:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5a39b700 1 -- 172.21.15.102:0/1326951090 learned_addr learned my addr 172.21.15.102:0/1326951090 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:12:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5ab9c700 1 --2- 172.21.15.102:0/1326951090 >> v2:172.21.15.172:3300/0 conn(0x7ffb540cd390 0x7ffb540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb59b9a700 1 --2- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb540cd8e0 0x7ffb540cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb59399700 1 -- 172.21.15.102:0/1326951090 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804033828 0 0) 0x7ffb54153660 con 0x7ffb54153290 2023-03-03T12:12:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb59399700 1 -- 172.21.15.102:0/1326951090 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb400035b0 con 0x7ffb54153290 2023-03-03T12:12:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5ab9c700 1 -- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb540cd8e0 msgr2=0x7ffb540cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:12:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.150+0000 7ffb5ab9c700 1 --2- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb540cd8e0 0x7ffb540cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.151+0000 7ffb5ab9c700 1 -- 172.21.15.102:0/1326951090 >> v1:172.21.15.172:6789/0 conn(0x7ffb54153290 legacy=0x7ffb54158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.151+0000 7ffb5ab9c700 1 -- 172.21.15.102:0/1326951090 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5406c7f0 con 0x7ffb540cd390 2023-03-03T12:12:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.151+0000 7ffb5ab9c700 1 --2- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb540ce360 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009d90 tx=0x7ffb4c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e8643b72776eaeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:12:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.151+0000 7ffb59399700 1 -- 172.21.15.102:0/1326951090 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00e070 con 0x7ffb540cd390 2023-03-03T12:12:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.151+0000 7ffb59399700 1 -- 172.21.15.102:0/1326951090 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c00bb20 con 0x7ffb540cd390 2023-03-03T12:12:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.151+0000 7ffb59399700 1 -- 172.21.15.102:0/1326951090 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c012430 con 0x7ffb540cd390 2023-03-03T12:12:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 msgr2=0x7ffb540ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:17.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb540ce360 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009d90 tx=0x7ffb4c00b330 comp rx=0 tx=0).stop 2023-03-03T12:12:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1326951090 shutdown_connections 2023-03-03T12:12:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb540ce360 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1326951090 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb540cd8e0 0x7ffb540cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1326951090 >> 172.21.15.102:0/1326951090 conn(0x7ffb54065fb0 msgr2=0x7ffb540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:12:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1326951090 shutdown_connections 2023-03-03T12:12:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.152+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1326951090 wait complete. 2023-03-03T12:12:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.154+0000 7ffb5c5ff700 1 Processor -- start 2023-03-03T12:12:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.154+0000 7ffb5c5ff700 1 -- start start 2023-03-03T12:12:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.155+0000 7ffb5c5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb541ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.155+0000 7ffb5c5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb540cd8e0 0x7ffb541efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.155+0000 7ffb5c5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb54153290 0x7ffb54150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.155+0000 7ffb5c5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb540c8660 con 0x7ffb540cd8e0 2023-03-03T12:12:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.155+0000 7ffb5c5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb540c6f70 con 0x7ffb540cd390 2023-03-03T12:12:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.155+0000 7ffb5c5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb540c7250 con 0x7ffb54153290 2023-03-03T12:12:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb541ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb59b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb540cd8e0 0x7ffb541efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5ab9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb54153290 0x7ffb54150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb541ef820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48192/0 (socket says 172.21.15.102:48192) 2023-03-03T12:12:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 -- 172.21.15.102:0/1863155717 learned_addr learned my addr 172.21.15.102:0/1863155717 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:12:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 -- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb54153290 msgr2=0x7ffb54150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb54153290 0x7ffb54150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 -- 172.21.15.102:0/1863155717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb540cd8e0 msgr2=0x7ffb541efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb540cd8e0 0x7ffb541efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 -- 172.21.15.102:0/1863155717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c009a70 con 0x7ffb540cd390 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb59b9a700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb540cd8e0 0x7ffb541efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.156+0000 7ffb5a39b700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb541ef820 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c700 tx=0x7ffb5000ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.157+0000 7ffb4affd700 1 -- 172.21.15.102:0/1863155717 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb50005d70 con 0x7ffb540cd390 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.157+0000 7ffb4affd700 1 -- 172.21.15.102:0/1863155717 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb500070d0 con 0x7ffb540cd390 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.157+0000 7ffb4affd700 1 -- 172.21.15.102:0/1863155717 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb50007460 con 0x7ffb540cd390 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.157+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb540ccde0 con 0x7ffb540cd390 2023-03-03T12:12:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.157+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5415d160 con 0x7ffb540cd390 2023-03-03T12:12:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.158+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb540cd0c0 con 0x7ffb540cd390 2023-03-03T12:12:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.159+0000 7ffb4affd700 1 -- 172.21.15.102:0/1863155717 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb50019030 con 0x7ffb540cd390 2023-03-03T12:12:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.161+0000 7ffb4affd700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb38068320 0x7ffb3806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.161+0000 7ffb4affd700 1 -- 172.21.15.102:0/1863155717 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ffb500850a0 con 0x7ffb540cd390 2023-03-03T12:12:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.161+0000 7ffb59b9a700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb38068320 0x7ffb3806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.162+0000 7ffb59b9a700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb38068320 0x7ffb3806a7d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ffb54073980 tx=0x7ffb44009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:12:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.165+0000 7ffb4affd700 1 -- 172.21.15.102:0/1863155717 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb500517b0 con 0x7ffb540cd390 2023-03-03T12:12:17.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.399+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb5404e6e0 con 0x7ffb540cd390 2023-03-03T12:12:17.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.400+0000 7ffb4affd700 1 -- 172.21.15.102:0/1863155717 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ffb50017070 con 0x7ffb540cd390 2023-03-03T12:12:17.402 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:12:17.402 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:12:17.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb38068320 msgr2=0x7ffb3806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:17.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb38068320 0x7ffb3806a7d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ffb54073980 tx=0x7ffb44009210 comp rx=0 tx=0).stop 2023-03-03T12:12:17.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 msgr2=0x7ffb541ef820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:17.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb541ef820 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c700 tx=0x7ffb5000ca10 comp rx=0 tx=0).stop 2023-03-03T12:12:17.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 shutdown_connections 2023-03-03T12:12:17.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb38068320 0x7ffb3806a7d0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb54153290 0x7ffb54150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb540cd390 0x7ffb541ef820 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 --2- 172.21.15.102:0/1863155717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb540cd8e0 0x7ffb541efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:17.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 >> 172.21.15.102:0/1863155717 conn(0x7ffb54065fb0 msgr2=0x7ffb54157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:12:17.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 shutdown_connections 2023-03-03T12:12:17.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:12:17.403+0000 7ffb5c5ff700 1 -- 172.21.15.102:0/1863155717 wait complete. 2023-03-03T12:12:17.407 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:12:17.432 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T12:12:21.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:21.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:51.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:51.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:15.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:13:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.699+0000 7fb5855ff700 1 Processor -- start 2023-03-03T12:13:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.699+0000 7fb5855ff700 1 -- start start 2023-03-03T12:13:15.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.700+0000 7fb5855ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb580153290 0x7fb580158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.700+0000 7fb5855ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5800cd8e0 0x7fb5800ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb5855ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb580153660 con 0x7fb5801593c0 2023-03-03T12:13:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb5855ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb58006c360 con 0x7fb580153290 2023-03-03T12:13:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb5855ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58015cea0 con 0x7fb5800cd8e0 2023-03-03T12:13:15.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb57effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb580153290 0x7fb580158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:15.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb577fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb5801593c0 0x7fb5800cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42114/0 (socket says 172.21.15.102:42114) 2023-03-03T12:13:15.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb577fff700 1 -- 172.21.15.102:0/2040114239 learned_addr learned my addr 172.21.15.102:0/2040114239 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:13:15.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb57f7fe700 1 --2- 172.21.15.102:0/2040114239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5800cd8e0 0x7fb5800ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:15.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb57effd700 1 -- 172.21.15.102:0/2040114239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5800cd8e0 msgr2=0x7fb5800ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:13:15.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb57effd700 1 --2- 172.21.15.102:0/2040114239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5800cd8e0 0x7fb5800ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb57effd700 1 -- 172.21.15.102:0/2040114239 >> v1:172.21.15.102:6789/0 conn(0x7fb5801593c0 legacy=0x7fb5800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.702+0000 7fb57effd700 1 -- 172.21.15.102:0/2040114239 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58006c7f0 con 0x7fb580153290 2023-03-03T12:13:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.701+0000 7fb57e7fc700 1 -- 172.21.15.102:0/2040114239 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801871803 0 0) 0x7fb580153660 con 0x7fb5801593c0 2023-03-03T12:13:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.702+0000 7fb57f7fe700 1 --2- 172.21.15.102:0/2040114239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5800cd8e0 0x7fb5800ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:13:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.702+0000 7fb57effd700 1 --2- 172.21.15.102:0/2040114239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb580153290 0x7fb580158e80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb570009840 tx=0x7fb570009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dff1562f3afa11a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.702+0000 7fb57e7fc700 1 -- 172.21.15.102:0/2040114239 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570011830 con 0x7fb580153290 2023-03-03T12:13:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.702+0000 7fb57e7fc700 1 -- 172.21.15.102:0/2040114239 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb570011990 con 0x7fb580153290 2023-03-03T12:13:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.702+0000 7fb57e7fc700 1 -- 172.21.15.102:0/2040114239 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570011d20 con 0x7fb580153290 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 -- 172.21.15.102:0/2040114239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb580153290 msgr2=0x7fb580158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 --2- 172.21.15.102:0/2040114239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb580153290 0x7fb580158e80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb570009840 tx=0x7fb570009b50 comp rx=0 tx=0).stop 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 -- 172.21.15.102:0/2040114239 shutdown_connections 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 --2- 172.21.15.102:0/2040114239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5800cd8e0 0x7fb5800ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 --2- 172.21.15.102:0/2040114239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb580153290 0x7fb580158e80 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 -- 172.21.15.102:0/2040114239 >> 172.21.15.102:0/2040114239 conn(0x7fb580065fb0 msgr2=0x7fb5800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 -- 172.21.15.102:0/2040114239 shutdown_connections 2023-03-03T12:13:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.703+0000 7fb5855ff700 1 -- 172.21.15.102:0/2040114239 wait complete. 2023-03-03T12:13:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.705+0000 7fb5855ff700 1 Processor -- start 2023-03-03T12:13:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.705+0000 7fb5855ff700 1 -- start start 2023-03-03T12:13:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.706+0000 7fb5855ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5800cd8e0 0x7fb58014fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.706+0000 7fb5855ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb580153290 0x7fb5801512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.706+0000 7fb5855ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5801593c0 0x7fb5801517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.706+0000 7fb5855ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580150710 con 0x7fb5800cd8e0 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.706+0000 7fb5855ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5801564c0 con 0x7fb580153290 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.706+0000 7fb5855ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb580150450 con 0x7fb5801593c0 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.706+0000 7fb57effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5800cd8e0 0x7fb58014fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5800cd8e0 0x7fb58014fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42660/0 (socket says 172.21.15.102:42660) 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 -- 172.21.15.102:0/3451981299 learned_addr learned my addr 172.21.15.102:0/3451981299 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57f7fe700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5801593c0 0x7fb5801517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb577fff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb580153290 0x7fb5801512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 -- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5801593c0 msgr2=0x7fb5801517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:15.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5801593c0 0x7fb5801517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 -- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb580153290 msgr2=0x7fb5801512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:15.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb580153290 0x7fb5801512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 -- 172.21.15.102:0/3451981299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5700094a0 con 0x7fb5800cd8e0 2023-03-03T12:13:15.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.707+0000 7fb57effd700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5800cd8e0 0x7fb58014fdf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb570014040 tx=0x7fb570004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:15.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.708+0000 7fb5777fe700 1 -- 172.21.15.102:0/3451981299 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700036a0 con 0x7fb5800cd8e0 2023-03-03T12:13:15.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.708+0000 7fb5777fe700 1 -- 172.21.15.102:0/3451981299 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57001e070 con 0x7fb5800cd8e0 2023-03-03T12:13:15.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.708+0000 7fb5777fe700 1 -- 172.21.15.102:0/3451981299 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570003a20 con 0x7fb5800cd8e0 2023-03-03T12:13:15.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.708+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5800ccde0 con 0x7fb5800cd8e0 2023-03-03T12:13:15.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.708+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58015d160 con 0x7fb5800cd8e0 2023-03-03T12:13:15.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.709+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5800cd0c0 con 0x7fb5800cd8e0 2023-03-03T12:13:15.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.710+0000 7fb5777fe700 1 -- 172.21.15.102:0/3451981299 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb57000f050 con 0x7fb5800cd8e0 2023-03-03T12:13:15.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.711+0000 7fb5777fe700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb5600682f0 0x7fb56006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:15.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.711+0000 7fb5777fe700 1 -- 172.21.15.102:0/3451981299 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb570086da0 con 0x7fb5800cd8e0 2023-03-03T12:13:15.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.712+0000 7fb577fff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb5600682f0 0x7fb56006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:15.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.712+0000 7fb577fff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb5600682f0 0x7fb56006a7a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fb580151df0 tx=0x7fb568007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:15.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.716+0000 7fb5777fe700 1 -- 172.21.15.102:0/3451981299 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb5700534b0 con 0x7fb5800cd8e0 2023-03-03T12:13:15.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.950+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb580156600 con 0x7fb5800cd8e0 2023-03-03T12:13:15.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.950+0000 7fb5777fe700 1 -- 172.21.15.102:0/3451981299 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb570015090 con 0x7fb5800cd8e0 2023-03-03T12:13:15.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:13:15.953 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:13:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb5600682f0 msgr2=0x7fb56006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb5600682f0 0x7fb56006a7a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fb580151df0 tx=0x7fb568007de0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5800cd8e0 msgr2=0x7fb58014fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5800cd8e0 0x7fb58014fdf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb570014040 tx=0x7fb570004750 comp rx=0 tx=0).stop 2023-03-03T12:13:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 shutdown_connections 2023-03-03T12:13:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb5600682f0 0x7fb56006a7a0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb5801593c0 0x7fb5801517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5800cd8e0 0x7fb58014fdf0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 --2- 172.21.15.102:0/3451981299 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb580153290 0x7fb5801512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 >> 172.21.15.102:0/3451981299 conn(0x7fb580065fb0 msgr2=0x7fb58015ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 shutdown_connections 2023-03-03T12:13:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:13:15.953+0000 7fb5855ff700 1 -- 172.21.15.102:0/3451981299 wait complete. 2023-03-03T12:13:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:13:15.983 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 101 secs before thrashing 2023-03-03T12:13:22.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:22.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:52.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:52.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:22.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:22.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:52.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:52.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:56.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:14:57.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.237+0000 7fb4c539d700 1 Processor -- start 2023-03-03T12:14:57.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.237+0000 7fb4c539d700 1 -- start start 2023-03-03T12:14:57.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4c539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00ced00 0x7fb4c00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:57.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4c539d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c00c3d20 con 0x7fb4c00c41b0 2023-03-03T12:14:57.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4c539d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c015cba0 con 0x7fb4c00b3970 2023-03-03T12:14:57.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4c539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00c4930 con 0x7fb4c00ced00 2023-03-03T12:14:57.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4bf7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb4c00b3970 0x7fb4c0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59536/0 (socket says 172.21.15.102:59536) 2023-03-03T12:14:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4c4b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb4c00c41b0 0x7fb4c0163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58830/0 (socket says 172.21.15.102:58830) 2023-03-03T12:14:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4bffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00ced00 0x7fb4c00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:57.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.239+0000 7fb4bf7fe700 1 -- 172.21.15.102:0/2069676222 learned_addr learned my addr 172.21.15.102:0/2069676222 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:14:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.240+0000 7fb4bffff700 1 -- 172.21.15.102:0/2069676222 >> v1:172.21.15.172:6789/0 conn(0x7fb4c00b3970 legacy=0x7fb4c0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.240+0000 7fb4bffff700 1 -- 172.21.15.102:0/2069676222 >> v1:172.21.15.102:6789/0 conn(0x7fb4c00c41b0 legacy=0x7fb4c0163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.240+0000 7fb4bffff700 1 -- 172.21.15.102:0/2069676222 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c015ce80 con 0x7fb4c00ced00 2023-03-03T12:14:57.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.240+0000 7fb4beffd700 1 -- 172.21.15.102:0/2069676222 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252481688 0 0) 0x7fb4c015cba0 con 0x7fb4c00b3970 2023-03-03T12:14:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.240+0000 7fb4bffff700 1 --2- 172.21.15.102:0/2069676222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00ced00 0x7fb4c00c5a90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009870 tx=0x7fb4b0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea47f25a8a0b843 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:14:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.240+0000 7fb4beffd700 1 -- 172.21.15.102:0/2069676222 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0011830 con 0x7fb4c00ced00 2023-03-03T12:14:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.240+0000 7fb4beffd700 1 -- 172.21.15.102:0/2069676222 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b0011990 con 0x7fb4c00ced00 2023-03-03T12:14:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4beffd700 1 -- 172.21.15.102:0/2069676222 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0011d20 con 0x7fb4c00ced00 2023-03-03T12:14:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4c539d700 1 -- 172.21.15.102:0/2069676222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00ced00 msgr2=0x7fb4c00c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2069676222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00ced00 0x7fb4c00c5a90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009870 tx=0x7fb4b0009b80 comp rx=0 tx=0).stop 2023-03-03T12:14:57.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4c539d700 1 -- 172.21.15.102:0/2069676222 shutdown_connections 2023-03-03T12:14:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2069676222 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00ced00 0x7fb4c00c5a90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4c539d700 1 -- 172.21.15.102:0/2069676222 >> 172.21.15.102:0/2069676222 conn(0x7fb4c008b1c0 msgr2=0x7fb4c0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:14:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4c539d700 1 -- 172.21.15.102:0/2069676222 shutdown_connections 2023-03-03T12:14:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.241+0000 7fb4c539d700 1 -- 172.21.15.102:0/2069676222 wait complete. 2023-03-03T12:14:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.243+0000 7fb4c539d700 1 Processor -- start 2023-03-03T12:14:57.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.243+0000 7fb4c539d700 1 -- start start 2023-03-03T12:14:57.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.244+0000 7fb4c539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00b3970 0x7fb4c01ee940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.244+0000 7fb4c539d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4c00c41b0 0x7fb4c00c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.244+0000 7fb4c539d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4c00ced00 0x7fb4c014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.244+0000 7fb4c539d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01ee4b0 con 0x7fb4c00c41b0 2023-03-03T12:14:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.244+0000 7fb4c539d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01f1b50 con 0x7fb4c00ced00 2023-03-03T12:14:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.244+0000 7fb4c539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01f1e30 con 0x7fb4c00b3970 2023-03-03T12:14:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4c00c41b0 0x7fb4c00c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:57.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00b3970 0x7fb4c01ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4c4b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4c00ced00 0x7fb4c014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4c00c41b0 0x7fb4c00c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41258/0 (socket says 172.21.15.102:41258) 2023-03-03T12:14:57.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bf7fe700 1 -- 172.21.15.102:0/2736907489 learned_addr learned my addr 172.21.15.102:0/2736907489 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:14:57.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bffff700 1 -- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4c00ced00 msgr2=0x7fb4c014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:57.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bffff700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4c00ced00 0x7fb4c014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:57.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bffff700 1 -- 172.21.15.102:0/2736907489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4c00c41b0 msgr2=0x7fb4c00c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:57.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bffff700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4c00c41b0 0x7fb4c00c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:57.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bffff700 1 -- 172.21.15.102:0/2736907489 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00094a0 con 0x7fb4c00b3970 2023-03-03T12:14:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bf7fe700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4c00c41b0 0x7fb4c00c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:14:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.245+0000 7fb4bffff700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00b3970 0x7fb4c01ee940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0016000 tx=0x7fb4b0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:14:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.246+0000 7fb4bcff9700 1 -- 172.21.15.102:0/2736907489 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b00036a0 con 0x7fb4c00b3970 2023-03-03T12:14:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.246+0000 7fb4bcff9700 1 -- 172.21.15.102:0/2736907489 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b001f070 con 0x7fb4c00b3970 2023-03-03T12:14:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.246+0000 7fb4bcff9700 1 -- 172.21.15.102:0/2736907489 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0003a20 con 0x7fb4c00b3970 2023-03-03T12:14:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.246+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c006c200 con 0x7fb4c00b3970 2023-03-03T12:14:57.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.246+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c006bf20 con 0x7fb4c00b3970 2023-03-03T12:14:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.247+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c015c890 con 0x7fb4c00b3970 2023-03-03T12:14:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.248+0000 7fb4bcff9700 1 -- 172.21.15.102:0/2736907489 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb4b000f050 con 0x7fb4c00b3970 2023-03-03T12:14:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.249+0000 7fb4bcff9700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb4ac068320 0x7fb4ac06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:57.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.250+0000 7fb4bcff9700 1 -- 172.21.15.102:0/2736907489 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb4b0021070 con 0x7fb4c00b3970 2023-03-03T12:14:57.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.250+0000 7fb4bf7fe700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb4ac068320 0x7fb4ac06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:57.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.250+0000 7fb4bf7fe700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb4ac068320 0x7fb4ac06a7d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0066930 tx=0x7fb4b4007f30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:14:57.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.254+0000 7fb4bcff9700 1 -- 172.21.15.102:0/2736907489 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb4b0053aa0 con 0x7fb4c00b3970 2023-03-03T12:14:57.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.488+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4c004e6e0 con 0x7fb4c00b3970 2023-03-03T12:14:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.490+0000 7fb4bcff9700 1 -- 172.21.15.102:0/2736907489 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb4b00570c0 con 0x7fb4c00b3970 2023-03-03T12:14:57.491 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:14:57.491 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:14:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb4ac068320 msgr2=0x7fb4ac06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb4ac068320 0x7fb4ac06a7d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0066930 tx=0x7fb4b4007f30 comp rx=0 tx=0).stop 2023-03-03T12:14:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00b3970 msgr2=0x7fb4c01ee940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00b3970 0x7fb4c01ee940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0016000 tx=0x7fb4b0004750 comp rx=0 tx=0).stop 2023-03-03T12:14:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 shutdown_connections 2023-03-03T12:14:57.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb4ac068320 0x7fb4ac06a7d0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb4c00b3970 0x7fb4c01ee940 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4c00ced00 0x7fb4c014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 --2- 172.21.15.102:0/2736907489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4c00c41b0 0x7fb4c00c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 >> 172.21.15.102:0/2736907489 conn(0x7fb4c008b1c0 msgr2=0x7fb4c0065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:14:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 shutdown_connections 2023-03-03T12:14:57.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:14:57.492+0000 7fb4c539d700 1 -- 172.21.15.102:0/2736907489 wait complete. 2023-03-03T12:14:57.496 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:14:57.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2023-03-03T12:15:11.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:15:11.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.781+0000 7f54137ff700 1 Processor -- start 2023-03-03T12:15:11.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.781+0000 7f54137ff700 1 -- start start 2023-03-03T12:15:11.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.782+0000 7f54137ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f540c0c9cb0 0x7f540c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:11.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f54137ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c0cf940 con 0x7f540c153260 2023-03-03T12:15:11.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f54137ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c0cfb90 con 0x7f540c0ca190 2023-03-03T12:15:11.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f54137ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540c0cfe70 con 0x7f540c0c9cb0 2023-03-03T12:15:11.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5410d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f540c0c9cb0 0x7f540c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:11.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5410d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f540c0c9cb0 0x7f540c0cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41790/0 (socket says 172.21.15.102:41790) 2023-03-03T12:15:11.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5410d9a700 1 -- 172.21.15.102:0/3827824263 learned_addr learned my addr 172.21.15.102:0/3827824263 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:15:11.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f541159b700 1 --1- 172.21.15.102:0/3827824263 >> v1:172.21.15.102:6789/0 conn(0x7f540c153260 0x7f540c0ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45482/0 (socket says 172.21.15.102:45482) 2023-03-03T12:15:11.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5411d9c700 1 --1- 172.21.15.102:0/3827824263 >> v1:172.21.15.172:6789/0 conn(0x7f540c0ca190 0x7f540c163610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59812/0 (socket says 172.21.15.102:59812) 2023-03-03T12:15:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5403fff700 1 -- 172.21.15.102:0/3827824263 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124529948 0 0) 0x7f540c0cfb90 con 0x7f540c0ca190 2023-03-03T12:15:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5410d9a700 1 -- 172.21.15.102:0/3827824263 >> v1:172.21.15.172:6789/0 conn(0x7f540c0ca190 legacy=0x7f540c163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5410d9a700 1 -- 172.21.15.102:0/3827824263 >> v1:172.21.15.102:6789/0 conn(0x7f540c153260 legacy=0x7f540c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.783+0000 7f5410d9a700 1 -- 172.21.15.102:0/3827824263 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c158f80 con 0x7f540c0c9cb0 2023-03-03T12:15:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.784+0000 7f5410d9a700 1 --2- 172.21.15.102:0/3827824263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c0c9cb0 0x7f540c0cb880 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0098f0 tx=0x7f53fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8f445827662aba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.784+0000 7f5403fff700 1 -- 172.21.15.102:0/3827824263 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc00b8a0 con 0x7f540c0c9cb0 2023-03-03T12:15:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.784+0000 7f5403fff700 1 -- 172.21.15.102:0/3827824263 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc00ba00 con 0x7f540c0c9cb0 2023-03-03T12:15:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.784+0000 7f5403fff700 1 -- 172.21.15.102:0/3827824263 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc00bd90 con 0x7f540c0c9cb0 2023-03-03T12:15:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.785+0000 7f54137ff700 1 -- 172.21.15.102:0/3827824263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c0c9cb0 msgr2=0x7f540c0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.785+0000 7f54137ff700 1 --2- 172.21.15.102:0/3827824263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c0c9cb0 0x7f540c0cb880 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0098f0 tx=0x7f53fc009c00 comp rx=0 tx=0).stop 2023-03-03T12:15:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.785+0000 7f54137ff700 1 -- 172.21.15.102:0/3827824263 shutdown_connections 2023-03-03T12:15:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.785+0000 7f54137ff700 1 --2- 172.21.15.102:0/3827824263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c0c9cb0 0x7f540c0cb880 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.785+0000 7f54137ff700 1 -- 172.21.15.102:0/3827824263 >> 172.21.15.102:0/3827824263 conn(0x7f540c066c00 msgr2=0x7f540c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.785+0000 7f54137ff700 1 -- 172.21.15.102:0/3827824263 shutdown_connections 2023-03-03T12:15:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.785+0000 7f54137ff700 1 -- 172.21.15.102:0/3827824263 wait complete. 2023-03-03T12:15:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.787+0000 7f54137ff700 1 Processor -- start 2023-03-03T12:15:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.787+0000 7f54137ff700 1 -- start start 2023-03-03T12:15:11.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.788+0000 7f54137ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c0ca190 0x7f540c1e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.788+0000 7f54137ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c153260 0x7f540c1ef660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.788+0000 7f54137ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f540c1efba0 0x7f540c14d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.788+0000 7f54137ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c1560b0 con 0x7f540c153260 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.788+0000 7f54137ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c1e4b00 con 0x7f540c0ca190 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.788+0000 7f54137ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c1e4f90 con 0x7f540c1efba0 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f541159b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c0ca190 0x7f540c1e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5410d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c153260 0x7f540c1ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5411d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f540c1efba0 0x7f540c14d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:11.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f541159b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c0ca190 0x7f540c1e5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39056/0 (socket says 172.21.15.102:39056) 2023-03-03T12:15:11.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f541159b700 1 -- 172.21.15.102:0/2659491127 learned_addr learned my addr 172.21.15.102:0/2659491127 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:15:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5410d9a700 1 -- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f540c1efba0 msgr2=0x7f540c14d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5410d9a700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f540c1efba0 0x7f540c14d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5410d9a700 1 -- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c0ca190 msgr2=0x7f540c1e5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5410d9a700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c0ca190 0x7f540c1e5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5410d9a700 1 -- 172.21.15.102:0/2659491127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc009550 con 0x7f540c153260 2023-03-03T12:15:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.789+0000 7f5410d9a700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c153260 0x7f540c1ef660 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f53fc016000 tx=0x7f53fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.790+0000 7f5401ffb700 1 -- 172.21.15.102:0/2659491127 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc01a900 con 0x7f540c153260 2023-03-03T12:15:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.790+0000 7f5401ffb700 1 -- 172.21.15.102:0/2659491127 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc01f070 con 0x7f540c153260 2023-03-03T12:15:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.790+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540c159540 con 0x7f540c153260 2023-03-03T12:15:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.790+0000 7f5401ffb700 1 -- 172.21.15.102:0/2659491127 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc01ac60 con 0x7f540c153260 2023-03-03T12:15:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.790+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c159260 con 0x7f540c153260 2023-03-03T12:15:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.791+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540c159820 con 0x7f540c153260 2023-03-03T12:15:11.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.792+0000 7f5401ffb700 1 -- 172.21.15.102:0/2659491127 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f53fc00f050 con 0x7f540c153260 2023-03-03T12:15:11.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.793+0000 7f5401ffb700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53f00682f0 0x7f53f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:11.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.793+0000 7f5401ffb700 1 -- 172.21.15.102:0/2659491127 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f53fc021070 con 0x7f540c153260 2023-03-03T12:15:11.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.794+0000 7f541159b700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53f00682f0 0x7f53f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:11.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.794+0000 7f541159b700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53f00682f0 0x7f53f006a7a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f5408000f60 tx=0x7f5408007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:11.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:11.798+0000 7f5401ffb700 1 -- 172.21.15.102:0/2659491127 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53fc053dc0 con 0x7f540c153260 2023-03-03T12:15:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.031+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f540c1e4c70 con 0x7f540c153260 2023-03-03T12:15:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.032+0000 7f5401ffb700 1 -- 172.21.15.102:0/2659491127 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f53fc0573e0 con 0x7f540c153260 2023-03-03T12:15:12.035 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:15:12.035 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:15:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53f00682f0 msgr2=0x7f53f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53f00682f0 0x7f53f006a7a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f5408000f60 tx=0x7f5408007f90 comp rx=0 tx=0).stop 2023-03-03T12:15:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c153260 msgr2=0x7f540c1ef660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c153260 0x7f540c1ef660 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f53fc016000 tx=0x7f53fc004750 comp rx=0 tx=0).stop 2023-03-03T12:15:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 shutdown_connections 2023-03-03T12:15:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53f00682f0 0x7f53f006a7a0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f540c1efba0 0x7f540c14d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c0ca190 0x7f540c1e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 --2- 172.21.15.102:0/2659491127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540c153260 0x7f540c1ef660 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.034+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 >> 172.21.15.102:0/2659491127 conn(0x7f540c0c9cb0 msgr2=0x7f540c158830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.035+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 shutdown_connections 2023-03-03T12:15:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:12.035+0000 7f54137ff700 1 -- 172.21.15.102:0/2659491127 wait complete. 2023-03-03T12:15:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:15:12.064 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2023-03-03T12:15:22.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:22.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:42.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:15:42.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.315+0000 7f2d8ddff700 1 Processor -- start 2023-03-03T12:15:42.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.315+0000 7f2d8ddff700 1 -- start start 2023-03-03T12:15:42.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d8ddff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2d880cd8e0 0x7f2d880cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:42.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d8ddff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d880cd390 0x7f2d880ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:42.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d8ddff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d88153660 con 0x7f2d88153290 2023-03-03T12:15:42.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d8ddff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8806c360 con 0x7f2d880cd8e0 2023-03-03T12:15:42.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d8ddff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8815cea0 con 0x7f2d880cd390 2023-03-03T12:15:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d87fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d880cd390 0x7f2d880ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d877fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2d88153290 0x7f2d88158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40750/0 (socket says 172.21.15.102:40750) 2023-03-03T12:15:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d86ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2d880cd8e0 0x7f2d880cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.316+0000 7f2d877fe700 1 -- 172.21.15.102:0/2448616375 learned_addr learned my addr 172.21.15.102:0/2448616375 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:15:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d867fc700 1 -- 172.21.15.102:0/2448616375 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256957933 0 0) 0x7f2d88153660 con 0x7f2d88153290 2023-03-03T12:15:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d867fc700 1 -- 172.21.15.102:0/2448616375 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d700035b0 con 0x7f2d88153290 2023-03-03T12:15:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d87fff700 1 -- 172.21.15.102:0/2448616375 >> v2:172.21.15.102:3300/0 conn(0x7f2d880cd8e0 msgr2=0x7f2d880cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:15:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d87fff700 1 --2- 172.21.15.102:0/2448616375 >> v2:172.21.15.102:3300/0 conn(0x7f2d880cd8e0 0x7f2d880cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d87fff700 1 -- 172.21.15.102:0/2448616375 >> v1:172.21.15.172:6789/0 conn(0x7f2d88153290 legacy=0x7f2d88158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d87fff700 1 -- 172.21.15.102:0/2448616375 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8806c7f0 con 0x7f2d880cd390 2023-03-03T12:15:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d86ffd700 1 --2- 172.21.15.102:0/2448616375 >> v2:172.21.15.102:3300/0 conn(0x7f2d880cd8e0 0x7f2d880cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:15:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.317+0000 7f2d87fff700 1 --2- 172.21.15.102:0/2448616375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880ce360 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2d880c9390 tx=0x7f2d8000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdcb38745aa4bf0c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d867fc700 1 -- 172.21.15.102:0/2448616375 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000e070 con 0x7f2d880cd390 2023-03-03T12:15:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d867fc700 1 -- 172.21.15.102:0/2448616375 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8000bb20 con 0x7f2d880cd390 2023-03-03T12:15:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d867fc700 1 -- 172.21.15.102:0/2448616375 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000be90 con 0x7f2d880cd390 2023-03-03T12:15:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 -- 172.21.15.102:0/2448616375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 msgr2=0x7f2d880ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/2448616375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880ce360 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2d880c9390 tx=0x7f2d8000b330 comp rx=0 tx=0).stop 2023-03-03T12:15:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 -- 172.21.15.102:0/2448616375 shutdown_connections 2023-03-03T12:15:42.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/2448616375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880ce360 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/2448616375 >> v2:172.21.15.102:3300/0 conn(0x7f2d880cd8e0 0x7f2d880cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 -- 172.21.15.102:0/2448616375 >> 172.21.15.102:0/2448616375 conn(0x7f2d88065fb0 msgr2=0x7f2d880663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:42.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 -- 172.21.15.102:0/2448616375 shutdown_connections 2023-03-03T12:15:42.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.318+0000 7f2d8ddff700 1 -- 172.21.15.102:0/2448616375 wait complete. 2023-03-03T12:15:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.320+0000 7f2d8ddff700 1 Processor -- start 2023-03-03T12:15:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.321+0000 7f2d8ddff700 1 -- start start 2023-03-03T12:15:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.321+0000 7f2d8ddff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.321+0000 7f2d8ddff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d880cd8e0 0x7f2d881e2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:42.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.321+0000 7f2d8ddff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d88153290 0x7f2d8814d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:42.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.321+0000 7f2d8ddff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8814eba0 con 0x7f2d88153290 2023-03-03T12:15:42.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.321+0000 7f2d8ddff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d880b3110 con 0x7f2d880cd390 2023-03-03T12:15:42.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.321+0000 7f2d8ddff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d880b36d0 con 0x7f2d880cd8e0 2023-03-03T12:15:42.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:42.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d86ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d880cd8e0 0x7f2d881e2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:42.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d87fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d88153290 0x7f2d8814d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42488/0 (socket says 172.21.15.102:42488) 2023-03-03T12:15:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 -- 172.21.15.102:0/1496996552 learned_addr learned my addr 172.21.15.102:0/1496996552 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:15:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 -- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d880cd8e0 msgr2=0x7f2d881e2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d880cd8e0 0x7f2d881e2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 -- 172.21.15.102:0/1496996552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d88153290 msgr2=0x7f2d8814d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d88153290 0x7f2d8814d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d877fe700 1 -- 172.21.15.102:0/1496996552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c0094a0 con 0x7f2d880cd390 2023-03-03T12:15:42.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.322+0000 7f2d87fff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d88153290 0x7f2d8814d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:15:42.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.323+0000 7f2d877fe700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880b1ee0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800c420 tx=0x7f2d7800c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:42.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.323+0000 7f2d6ffff700 1 -- 172.21.15.102:0/1496996552 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78005d70 con 0x7f2d880cd390 2023-03-03T12:15:42.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.323+0000 7f2d6ffff700 1 -- 172.21.15.102:0/1496996552 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7800ce30 con 0x7f2d880cd390 2023-03-03T12:15:42.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.323+0000 7f2d6ffff700 1 -- 172.21.15.102:0/1496996552 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d780072d0 con 0x7f2d880cd390 2023-03-03T12:15:42.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.323+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d80009a70 con 0x7f2d880cd390 2023-03-03T12:15:42.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.323+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d880ccde0 con 0x7f2d880cd390 2023-03-03T12:15:42.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.324+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8815d160 con 0x7f2d880cd390 2023-03-03T12:15:42.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.326+0000 7f2d6ffff700 1 -- 172.21.15.102:0/1496996552 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d78019030 con 0x7f2d880cd390 2023-03-03T12:15:42.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.327+0000 7f2d6ffff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d74068320 0x7f2d7406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:42.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.327+0000 7f2d6ffff700 1 -- 172.21.15.102:0/1496996552 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2d78084ec0 con 0x7f2d880cd390 2023-03-03T12:15:42.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.327+0000 7f2d86ffd700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d74068320 0x7f2d7406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:42.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.328+0000 7f2d86ffd700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d74068320 0x7f2d7406a7d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2d8808c0c0 tx=0x7f2d7c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:42.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.331+0000 7f2d6ffff700 1 -- 172.21.15.102:0/1496996552 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d7804b5e0 con 0x7f2d880cd390 2023-03-03T12:15:42.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.569+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d880cd0c0 con 0x7f2d880cd390 2023-03-03T12:15:42.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.570+0000 7f2d6ffff700 1 -- 172.21.15.102:0/1496996552 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2d78017070 con 0x7f2d880cd390 2023-03-03T12:15:42.572 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:15:42.572 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:15:42.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d74068320 msgr2=0x7f2d7406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d74068320 0x7f2d7406a7d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2d8808c0c0 tx=0x7f2d7c009210 comp rx=0 tx=0).stop 2023-03-03T12:15:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 msgr2=0x7f2d880b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880b1ee0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800c420 tx=0x7f2d7800c730 comp rx=0 tx=0).stop 2023-03-03T12:15:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 shutdown_connections 2023-03-03T12:15:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d88153290 0x7f2d8814d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d880cd390 0x7f2d880b1ee0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2d74068320 0x7f2d7406a7d0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 --2- 172.21.15.102:0/1496996552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d880cd8e0 0x7f2d881e2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 >> 172.21.15.102:0/1496996552 conn(0x7f2d88065fb0 msgr2=0x7f2d88151070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.572+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 shutdown_connections 2023-03-03T12:15:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:15:42.573+0000 7f2d8ddff700 1 -- 172.21.15.102:0/1496996552 wait complete. 2023-03-03T12:15:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:15:42.602 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 101 secs before thrashing 2023-03-03T12:15:52.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:52.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:23.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:23.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:53.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:53.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:23.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:23.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:23.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:17:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.845+0000 7ff7991ff700 1 Processor -- start 2023-03-03T12:17:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.845+0000 7ff7991ff700 1 -- start start 2023-03-03T12:17:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.846+0000 7ff7991ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff794153290 0x7ff794158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.846+0000 7ff7991ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7940cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.846+0000 7ff7991ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7940cd8e0 0x7ff7940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.846+0000 7ff7991ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff794153660 con 0x7ff7940cd8e0 2023-03-03T12:17:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff7991ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79406c360 con 0x7ff794153290 2023-03-03T12:17:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff7991ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79415cea0 con 0x7ff7941593c0 2023-03-03T12:17:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7940cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff792d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff794153290 0x7ff794158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7940cd8e0 0x7ff7940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7940cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60198/0 (socket says 172.21.15.102:60198) 2023-03-03T12:17:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 -- 172.21.15.102:0/4109804046 learned_addr learned my addr 172.21.15.102:0/4109804046 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:17:23.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 -- 172.21.15.102:0/4109804046 >> v2:172.21.15.172:3300/0 conn(0x7ff794153290 msgr2=0x7ff794158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:23.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 --2- 172.21.15.102:0/4109804046 >> v2:172.21.15.172:3300/0 conn(0x7ff794153290 0x7ff794158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 -- 172.21.15.102:0/4109804046 >> v2:172.21.15.102:3300/0 conn(0x7ff7940cd8e0 msgr2=0x7ff7940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 --2- 172.21.15.102:0/4109804046 >> v2:172.21.15.102:3300/0 conn(0x7ff7940cd8e0 0x7ff7940ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.847+0000 7ff79259c700 1 -- 172.21.15.102:0/4109804046 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79406c7f0 con 0x7ff7941593c0 2023-03-03T12:17:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.848+0000 7ff79259c700 1 --2- 172.21.15.102:0/4109804046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7940cde20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff79408cc70 tx=0x7ff788009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adebe42d7bf5fd3e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.848+0000 7ff791d9b700 1 -- 172.21.15.102:0/4109804046 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788011830 con 0x7ff7941593c0 2023-03-03T12:17:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.848+0000 7ff791d9b700 1 -- 172.21.15.102:0/4109804046 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff788011990 con 0x7ff7941593c0 2023-03-03T12:17:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.848+0000 7ff791d9b700 1 -- 172.21.15.102:0/4109804046 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788011d20 con 0x7ff7941593c0 2023-03-03T12:17:23.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 -- 172.21.15.102:0/4109804046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 msgr2=0x7ff7940cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:23.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 --2- 172.21.15.102:0/4109804046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7940cde20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff79408cc70 tx=0x7ff788009bd0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 -- 172.21.15.102:0/4109804046 shutdown_connections 2023-03-03T12:17:23.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 --2- 172.21.15.102:0/4109804046 >> v2:172.21.15.102:3300/0 conn(0x7ff7940cd8e0 0x7ff7940ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 --2- 172.21.15.102:0/4109804046 >> v2:172.21.15.172:3300/0 conn(0x7ff794153290 0x7ff794158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 --2- 172.21.15.102:0/4109804046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7940cde20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 -- 172.21.15.102:0/4109804046 >> 172.21.15.102:0/4109804046 conn(0x7ff794065fb0 msgr2=0x7ff7940663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:23.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 -- 172.21.15.102:0/4109804046 shutdown_connections 2023-03-03T12:17:23.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.849+0000 7ff7991ff700 1 -- 172.21.15.102:0/4109804046 wait complete. 2023-03-03T12:17:23.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.852+0000 7ff7991ff700 1 Processor -- start 2023-03-03T12:17:23.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.852+0000 7ff7991ff700 1 -- start start 2023-03-03T12:17:23.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.853+0000 7ff7991ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7940cd8e0 0x7ff7940cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:23.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.853+0000 7ff7991ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff794153290 0x7ff7941eef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:23.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.853+0000 7ff7991ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7941ef440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:23.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.853+0000 7ff7991ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7941f3000 con 0x7ff794153290 2023-03-03T12:17:23.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.853+0000 7ff7991ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7941f23f0 con 0x7ff7940cd8e0 2023-03-03T12:17:23.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.853+0000 7ff7991ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7941f26d0 con 0x7ff7941593c0 2023-03-03T12:17:23.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff794153290 0x7ff7941eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:23.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff792d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7940cd8e0 0x7ff7940cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:23.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7941ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:23.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff792d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7940cd8e0 0x7ff7940cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56960/0 (socket says 172.21.15.102:56960) 2023-03-03T12:17:23.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff792d9d700 1 -- 172.21.15.102:0/529560897 learned_addr learned my addr 172.21.15.102:0/529560897 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:17:23.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79259c700 1 -- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 msgr2=0x7ff7941ef440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:23.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79259c700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7941ef440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79259c700 1 -- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7940cd8e0 msgr2=0x7ff7940cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:23.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79259c700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7940cd8e0 0x7ff7940cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:23.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79259c700 1 -- 172.21.15.102:0/529560897 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff788009550 con 0x7ff794153290 2023-03-03T12:17:23.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff792d9d700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7940cd8e0 0x7ff7940cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79359e700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7941ef440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.854+0000 7ff79259c700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff794153290 0x7ff7941eef00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff788016000 tx=0x7ff788004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.855+0000 7ff77b7fe700 1 -- 172.21.15.102:0/529560897 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788004240 con 0x7ff794153290 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.855+0000 7ff77b7fe700 1 -- 172.21.15.102:0/529560897 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78801f070 con 0x7ff794153290 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.855+0000 7ff77b7fe700 1 -- 172.21.15.102:0/529560897 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7880045e0 con 0x7ff794153290 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.855+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7940ccde0 con 0x7ff794153290 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.855+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79415d160 con 0x7ff794153290 2023-03-03T12:17:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.856+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7940cd0c0 con 0x7ff794153290 2023-03-03T12:17:23.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.857+0000 7ff77b7fe700 1 -- 172.21.15.102:0/529560897 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff78802a020 con 0x7ff794153290 2023-03-03T12:17:23.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.859+0000 7ff77b7fe700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7800682f0 0x7ff78006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:23.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.859+0000 7ff77b7fe700 1 -- 172.21.15.102:0/529560897 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff788086ff0 con 0x7ff794153290 2023-03-03T12:17:23.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.860+0000 7ff792d9d700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7800682f0 0x7ff78006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:23.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.860+0000 7ff792d9d700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7800682f0 0x7ff78006a7a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff784009f70 tx=0x7ff784009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:23.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:23.863+0000 7ff77b7fe700 1 -- 172.21.15.102:0/529560897 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7880b85e0 con 0x7ff794153290 2023-03-03T12:17:24.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.095+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7941f3140 con 0x7ff794153290 2023-03-03T12:17:24.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.096+0000 7ff77b7fe700 1 -- 172.21.15.102:0/529560897 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff7941f3140 con 0x7ff794153290 2023-03-03T12:17:24.099 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:17:24.099 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:17:24.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7800682f0 msgr2=0x7ff78006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:24.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7800682f0 0x7ff78006a7a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff784009f70 tx=0x7ff784009210 comp rx=0 tx=0).stop 2023-03-03T12:17:24.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff794153290 msgr2=0x7ff7941eef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:24.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff794153290 0x7ff7941eef00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff788016000 tx=0x7ff788004750 comp rx=0 tx=0).stop 2023-03-03T12:17:24.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 shutdown_connections 2023-03-03T12:17:24.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7800682f0 0x7ff78006a7a0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:24.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7941593c0 0x7ff7941ef440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:24.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7940cd8e0 0x7ff7940cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:24.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 --2- 172.21.15.102:0/529560897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff794153290 0x7ff7941eef00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:24.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 >> 172.21.15.102:0/529560897 conn(0x7ff794065fb0 msgr2=0x7ff7940cac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:24.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 shutdown_connections 2023-03-03T12:17:24.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:17:24.099+0000 7ff7991ff700 1 -- 172.21.15.102:0/529560897 wait complete. 2023-03-03T12:17:24.104 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:17:24.129 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T12:17:53.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:53.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:23.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:23.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:53.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:53.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:18.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:19:18.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.394+0000 7f6aedfff700 1 Processor -- start 2023-03-03T12:19:18.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.395+0000 7f6aedfff700 1 -- start start 2023-03-03T12:19:18.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.396+0000 7f6aedfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.396+0000 7f6aedfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae8153660 con 0x7f6ae81593c0 2023-03-03T12:19:18.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.396+0000 7f6aedfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae806c360 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.396+0000 7f6aedfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80cec40 con 0x7f6ae8153290 2023-03-03T12:19:18.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.396+0000 7f6adffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6ae81593c0 0x7f6ae80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47740/0 (socket says 172.21.15.102:47740) 2023-03-03T12:19:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6adffff700 1 -- 172.21.15.102:0/694991089 learned_addr learned my addr 172.21.15.102:0/694991089 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:19:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae77fe700 1 --2- 172.21.15.102:0/694991089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae6ffd700 1 -- 172.21.15.102:0/694991089 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979976007 0 0) 0x7f6ae8153660 con 0x7f6ae81593c0 2023-03-03T12:19:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae6ffd700 1 -- 172.21.15.102:0/694991089 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad40035b0 con 0x7f6ae81593c0 2023-03-03T12:19:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae6ffd700 1 -- 172.21.15.102:0/694991089 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212698953 0 0) 0x7f6ae806c360 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae6ffd700 1 -- 172.21.15.102:0/694991089 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae8153660 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae77fe700 1 -- 172.21.15.102:0/694991089 >> v1:172.21.15.172:6789/0 conn(0x7f6ae80cd8e0 legacy=0x7f6ae80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae77fe700 1 -- 172.21.15.102:0/694991089 >> v1:172.21.15.102:6789/0 conn(0x7f6ae81593c0 legacy=0x7f6ae80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.397+0000 7f6ae77fe700 1 -- 172.21.15.102:0/694991089 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae815cea0 con 0x7f6ae8153290 2023-03-03T12:19:18.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.398+0000 7f6ae77fe700 1 --2- 172.21.15.102:0/694991089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae8158e80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e78d9d106664081 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:18.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.398+0000 7f6ae6ffd700 1 -- 172.21.15.102:0/694991089 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8011830 con 0x7f6ae8153290 2023-03-03T12:19:18.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.398+0000 7f6ae6ffd700 1 -- 172.21.15.102:0/694991089 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad8011990 con 0x7f6ae8153290 2023-03-03T12:19:18.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.398+0000 7f6ae6ffd700 1 -- 172.21.15.102:0/694991089 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8011d20 con 0x7f6ae8153290 2023-03-03T12:19:18.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.398+0000 7f6aedfff700 1 -- 172.21.15.102:0/694991089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 msgr2=0x7f6ae8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.398+0000 7f6aedfff700 1 --2- 172.21.15.102:0/694991089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae8158e80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad8009b80 comp rx=0 tx=0).stop 2023-03-03T12:19:18.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.399+0000 7f6aedfff700 1 -- 172.21.15.102:0/694991089 shutdown_connections 2023-03-03T12:19:18.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.399+0000 7f6aedfff700 1 --2- 172.21.15.102:0/694991089 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae8158e80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.399+0000 7f6aedfff700 1 -- 172.21.15.102:0/694991089 >> 172.21.15.102:0/694991089 conn(0x7f6ae8065fb0 msgr2=0x7f6ae80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:18.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.399+0000 7f6aedfff700 1 -- 172.21.15.102:0/694991089 shutdown_connections 2023-03-03T12:19:18.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.399+0000 7f6aedfff700 1 -- 172.21.15.102:0/694991089 wait complete. 2023-03-03T12:19:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.401+0000 7f6aedfff700 1 Processor -- start 2023-03-03T12:19:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.401+0000 7f6aedfff700 1 -- start start 2023-03-03T12:19:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6aedfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ae80cd8e0 0x7f6ae8154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6aedfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae81f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6aedfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ae81593c0 0x7f6ae80c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6aedfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b21a0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6aedfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b1c70 con 0x7f6ae81593c0 2023-03-03T12:19:18.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6aedfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80c8f40 con 0x7f6ae8153290 2023-03-03T12:19:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6ae77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ae80cd8e0 0x7f6ae8154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6adffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae81f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6ae77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ae80cd8e0 0x7f6ae8154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42154/0 (socket says 172.21.15.102:42154) 2023-03-03T12:19:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6ae77fe700 1 -- 172.21.15.102:0/2465413239 learned_addr learned my addr 172.21.15.102:0/2465413239 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:19:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.402+0000 7f6ae7fff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ae81593c0 0x7f6ae80c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae77fe700 1 -- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 msgr2=0x7f6ae81f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae77fe700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae81f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae77fe700 1 -- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ae81593c0 msgr2=0x7f6ae80c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae77fe700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ae81593c0 0x7f6ae80c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae77fe700 1 -- 172.21.15.102:0/2465413239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad80094a0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae7fff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ae81593c0 0x7f6ae80c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae77fe700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ae80cd8e0 0x7f6ae8154430 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80090b0 tx=0x7f6ad8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.403+0000 7f6ae4ff9700 1 -- 172.21.15.102:0/2465413239 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad801a8f0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.404+0000 7f6ae4ff9700 1 -- 172.21.15.102:0/2465413239 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad801aa50 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.404+0000 7f6ae4ff9700 1 -- 172.21.15.102:0/2465413239 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad801adc0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.404+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae815d160 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.404+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae806c7f0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.405+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae80ccde0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.406+0000 7f6ae4ff9700 1 -- 172.21.15.102:0/2465413239 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6ad800f050 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.407+0000 7f6ae4ff9700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ac8068320 0x7f6ac806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.407+0000 7f6ae4ff9700 1 -- 172.21.15.102:0/2465413239 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6ad8086fa0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.407+0000 7f6adffff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ac8068320 0x7f6ac806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.408+0000 7f6adffff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ac8068320 0x7f6ac806a7d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80c89c0 tx=0x7f6ad0007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.411+0000 7f6ae4ff9700 1 -- 172.21.15.102:0/2465413239 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ad80536b0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.644+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ae804e6e0 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.644+0000 7f6ae4ff9700 1 -- 172.21.15.102:0/2465413239 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6ad8015020 con 0x7f6ae80cd8e0 2023-03-03T12:19:18.646 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:19:18.646 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:19:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ac8068320 msgr2=0x7f6ac806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ac8068320 0x7f6ac806a7d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80c89c0 tx=0x7f6ad0007de0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ae80cd8e0 msgr2=0x7f6ae8154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ae80cd8e0 0x7f6ae8154430 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80090b0 tx=0x7f6ad8004750 comp rx=0 tx=0).stop 2023-03-03T12:19:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 shutdown_connections 2023-03-03T12:19:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ae81593c0 0x7f6ae80c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ae80cd8e0 0x7f6ae8154430 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ac8068320 0x7f6ac806a7d0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 --2- 172.21.15.102:0/2465413239 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ae8153290 0x7f6ae81f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 >> 172.21.15.102:0/2465413239 conn(0x7f6ae8065fb0 msgr2=0x7f6ae80cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.647+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 shutdown_connections 2023-03-03T12:19:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.648+0000 7f6aedfff700 1 -- 172.21.15.102:0/2465413239 wait complete. 2023-03-03T12:19:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:19:18.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:19:18.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.919+0000 7f7e7759d700 1 Processor -- start 2023-03-03T12:19:18.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.919+0000 7f7e7759d700 1 -- start start 2023-03-03T12:19:18.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e7759d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7e700ced00 0x7f7e700c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e7759d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700c3d20 con 0x7f7e700b3970 2023-03-03T12:19:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e7759d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7015cba0 con 0x7f7e700c41b0 2023-03-03T12:19:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e7759d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700c4930 con 0x7f7e700ced00 2023-03-03T12:19:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e7659b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7e700ced00 0x7f7e700c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e7659b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7e700ced00 0x7f7e700c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42164/0 (socket says 172.21.15.102:42164) 2023-03-03T12:19:18.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e7659b700 1 -- 172.21.15.102:0/4145376966 learned_addr learned my addr 172.21.15.102:0/4145376966 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:19:18.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.921+0000 7f7e75d9a700 1 --1- 172.21.15.102:0/4145376966 >> v1:172.21.15.102:6789/0 conn(0x7f7e700b3970 0x7f7e70158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47748/0 (socket says 172.21.15.102:47748) 2023-03-03T12:19:18.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.922+0000 7f7e7659b700 1 -- 172.21.15.102:0/4145376966 >> v1:172.21.15.172:6789/0 conn(0x7f7e700c41b0 legacy=0x7f7e70163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.922+0000 7f7e7659b700 1 -- 172.21.15.102:0/4145376966 >> v1:172.21.15.102:6789/0 conn(0x7f7e700b3970 legacy=0x7f7e70158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.922+0000 7f7e7659b700 1 -- 172.21.15.102:0/4145376966 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7015ce80 con 0x7f7e700ced00 2023-03-03T12:19:18.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.922+0000 7f7e7659b700 1 --2- 172.21.15.102:0/4145376966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700ced00 0x7f7e700c5a90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0098f0 tx=0x7f7e6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da666f0b8c726c56 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:18.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.922+0000 7f7e75599700 1 -- 172.21.15.102:0/4145376966 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00b8a0 con 0x7f7e700ced00 2023-03-03T12:19:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.922+0000 7f7e75599700 1 -- 172.21.15.102:0/4145376966 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c00ba00 con 0x7f7e700ced00 2023-03-03T12:19:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.922+0000 7f7e75599700 1 -- 172.21.15.102:0/4145376966 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00bd90 con 0x7f7e700ced00 2023-03-03T12:19:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.923+0000 7f7e7759d700 1 -- 172.21.15.102:0/4145376966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700ced00 msgr2=0x7f7e700c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.923+0000 7f7e7759d700 1 --2- 172.21.15.102:0/4145376966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700ced00 0x7f7e700c5a90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0098f0 tx=0x7f7e6c009c00 comp rx=0 tx=0).stop 2023-03-03T12:19:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.923+0000 7f7e7759d700 1 -- 172.21.15.102:0/4145376966 shutdown_connections 2023-03-03T12:19:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.923+0000 7f7e7759d700 1 --2- 172.21.15.102:0/4145376966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700ced00 0x7f7e700c5a90 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.923+0000 7f7e7759d700 1 -- 172.21.15.102:0/4145376966 >> 172.21.15.102:0/4145376966 conn(0x7f7e7008b1c0 msgr2=0x7f7e70071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.923+0000 7f7e7759d700 1 -- 172.21.15.102:0/4145376966 shutdown_connections 2023-03-03T12:19:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.923+0000 7f7e7759d700 1 -- 172.21.15.102:0/4145376966 wait complete. 2023-03-03T12:19:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.925+0000 7f7e7759d700 1 Processor -- start 2023-03-03T12:19:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.925+0000 7f7e7759d700 1 -- start start 2023-03-03T12:19:18.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.926+0000 7f7e7759d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e700b3970 0x7f7e701ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.926+0000 7f7e7759d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700c41b0 0x7f7e70159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.926+0000 7f7e7759d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e700ced00 0x7f7e7015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.926+0000 7f7e7759d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70152790 con 0x7f7e700c41b0 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.926+0000 7f7e7759d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7015a9b0 con 0x7f7e700b3970 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.926+0000 7f7e7759d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e701521d0 con 0x7f7e700ced00 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e700b3970 0x7f7e701ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e75d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700c41b0 0x7f7e70159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e76d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e700ced00 0x7f7e7015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e700b3970 0x7f7e701ef890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37598/0 (socket says 172.21.15.102:37598) 2023-03-03T12:19:18.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 -- 172.21.15.102:0/3656393183 learned_addr learned my addr 172.21.15.102:0/3656393183 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:19:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 -- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e700ced00 msgr2=0x7f7e7015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e700ced00 0x7f7e7015a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 -- 172.21.15.102:0/3656393183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700c41b0 msgr2=0x7f7e70159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700c41b0 0x7f7e70159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 -- 172.21.15.102:0/3656393183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c009550 con 0x7f7e700b3970 2023-03-03T12:19:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e76d9c700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e700ced00 0x7f7e7015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e75d9a700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700c41b0 0x7f7e70159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.927+0000 7f7e7659b700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e700b3970 0x7f7e701ef890 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009c40 tx=0x7f7e6c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.928+0000 7f7e66ffd700 1 -- 172.21.15.102:0/3656393183 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c004380 con 0x7f7e700b3970 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.928+0000 7f7e66ffd700 1 -- 172.21.15.102:0/3656393183 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c01f070 con 0x7f7e700b3970 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.928+0000 7f7e66ffd700 1 -- 172.21.15.102:0/3656393183 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c0036a0 con 0x7f7e700b3970 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.928+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7006c200 con 0x7f7e700b3970 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.928+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7006bf20 con 0x7f7e700b3970 2023-03-03T12:19:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.929+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7015c890 con 0x7f7e700b3970 2023-03-03T12:19:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.930+0000 7f7e66ffd700 1 -- 172.21.15.102:0/3656393183 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e6c02a020 con 0x7f7e700b3970 2023-03-03T12:19:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.931+0000 7f7e66ffd700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e54068320 0x7f7e5406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.931+0000 7f7e66ffd700 1 -- 172.21.15.102:0/3656393183 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7e6c087310 con 0x7f7e700b3970 2023-03-03T12:19:18.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.932+0000 7f7e75d9a700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e54068320 0x7f7e5406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:18.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.932+0000 7f7e75d9a700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e54068320 0x7f7e5406a7d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7e700b41c0 tx=0x7f7e60007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:18.936+0000 7f7e66ffd700 1 -- 172.21.15.102:0/3656393183 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e6c053a20 con 0x7f7e700b3970 2023-03-03T12:19:19.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.170+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e7004e6e0 con 0x7f7e700b3970 2023-03-03T12:19:19.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.171+0000 7f7e66ffd700 1 -- 172.21.15.102:0/3656393183 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7e6c057040 con 0x7f7e700b3970 2023-03-03T12:19:19.173 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:19:19.173 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:19:19.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e54068320 msgr2=0x7f7e5406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:19.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e54068320 0x7f7e5406a7d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7e700b41c0 tx=0x7f7e60007ed0 comp rx=0 tx=0).stop 2023-03-03T12:19:19.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e700b3970 msgr2=0x7f7e701ef890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:19.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e700b3970 0x7f7e701ef890 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009c40 tx=0x7f7e6c004750 comp rx=0 tx=0).stop 2023-03-03T12:19:19.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 shutdown_connections 2023-03-03T12:19:19.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e54068320 0x7f7e5406a7d0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:19.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e700ced00 0x7f7e7015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e700b3970 0x7f7e701ef890 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 --2- 172.21.15.102:0/3656393183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e700c41b0 0x7f7e70159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 >> 172.21.15.102:0/3656393183 conn(0x7f7e7008b1c0 msgr2=0x7f7e70066930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 shutdown_connections 2023-03-03T12:19:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:19.174+0000 7f7e7759d700 1 -- 172.21.15.102:0/3656393183 wait complete. 2023-03-03T12:19:19.177 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:19:19.202 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T12:19:24.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:24.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:53.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:19:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.465+0000 7fd026d9d700 1 Processor -- start 2023-03-03T12:19:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.465+0000 7fd026d9d700 1 -- start start 2023-03-03T12:19:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd026d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd0200ced00 0x7fd0200c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd026d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200c41b0 0x7fd0200cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd026d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0200c3d20 con 0x7fd0200b3970 2023-03-03T12:19:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd026d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0200c4930 con 0x7fd0200ced00 2023-03-03T12:19:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd026d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200cf610 con 0x7fd0200c41b0 2023-03-03T12:19:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd025d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd0200ced00 0x7fd0200c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd02559a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd0200b3970 0x7fd020158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54084/0 (socket says 172.21.15.102:54084) 2023-03-03T12:19:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd02559a700 1 -- 172.21.15.102:0/1939233106 learned_addr learned my addr 172.21.15.102:0/1939233106 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:19:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.467+0000 7fd02659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200c41b0 0x7fd0200cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd02659c700 1 -- 172.21.15.102:0/1939233106 >> v2:172.21.15.172:3300/0 conn(0x7fd0200ced00 msgr2=0x7fd0200c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:19:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd02659c700 1 --2- 172.21.15.102:0/1939233106 >> v2:172.21.15.172:3300/0 conn(0x7fd0200ced00 0x7fd0200c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd024d99700 1 -- 172.21.15.102:0/1939233106 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778291409 0 0) 0x7fd0200c3d20 con 0x7fd0200b3970 2023-03-03T12:19:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd02659c700 1 -- 172.21.15.102:0/1939233106 >> v1:172.21.15.102:6789/0 conn(0x7fd0200b3970 legacy=0x7fd020158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd02659c700 1 -- 172.21.15.102:0/1939233106 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02015cba0 con 0x7fd0200c41b0 2023-03-03T12:19:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd025d9b700 1 --2- 172.21.15.102:0/1939233106 >> v2:172.21.15.172:3300/0 conn(0x7fd0200ced00 0x7fd0200c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:19:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd02659c700 1 --2- 172.21.15.102:0/1939233106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200c41b0 0x7fd0200cf0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd018009d90 tx=0x7fd01800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3389166e0c2437 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd024d99700 1 -- 172.21.15.102:0/1939233106 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01800e070 con 0x7fd0200c41b0 2023-03-03T12:19:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd024d99700 1 -- 172.21.15.102:0/1939233106 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01800bb20 con 0x7fd0200c41b0 2023-03-03T12:19:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.468+0000 7fd024d99700 1 -- 172.21.15.102:0/1939233106 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018012430 con 0x7fd0200c41b0 2023-03-03T12:19:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 -- 172.21.15.102:0/1939233106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200c41b0 msgr2=0x7fd0200cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 --2- 172.21.15.102:0/1939233106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200c41b0 0x7fd0200cf0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd018009d90 tx=0x7fd01800b330 comp rx=0 tx=0).stop 2023-03-03T12:19:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 -- 172.21.15.102:0/1939233106 shutdown_connections 2023-03-03T12:19:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 --2- 172.21.15.102:0/1939233106 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200c41b0 0x7fd0200cf0d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 --2- 172.21.15.102:0/1939233106 >> v2:172.21.15.172:3300/0 conn(0x7fd0200ced00 0x7fd0200c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 -- 172.21.15.102:0/1939233106 >> 172.21.15.102:0/1939233106 conn(0x7fd02008b1c0 msgr2=0x7fd020071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 -- 172.21.15.102:0/1939233106 shutdown_connections 2023-03-03T12:19:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.469+0000 7fd026d9d700 1 -- 172.21.15.102:0/1939233106 wait complete. 2023-03-03T12:19:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.471+0000 7fd026d9d700 1 Processor -- start 2023-03-03T12:19:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.472+0000 7fd026d9d700 1 -- start start 2023-03-03T12:19:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.472+0000 7fd026d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0200b3970 0x7fd02014e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.472+0000 7fd026d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0200c41b0 0x7fd0201f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.472+0000 7fd026d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200ced00 0x7fd0201f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.472+0000 7fd026d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200c77a0 con 0x7fd0200c41b0 2023-03-03T12:19:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.472+0000 7fd026d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0201f18b0 con 0x7fd0200b3970 2023-03-03T12:19:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.472+0000 7fd026d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200c88f0 con 0x7fd0200ced00 2023-03-03T12:19:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0200b3970 0x7fd02014e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd02559a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0200c41b0 0x7fd0201f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0200b3970 0x7fd02014e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:41268/0 (socket says 172.21.15.102:41268) 2023-03-03T12:19:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 -- 172.21.15.102:0/3395531930 learned_addr learned my addr 172.21.15.102:0/3395531930 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:19:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd02659c700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200ced00 0x7fd0201f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:53.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 -- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200ced00 msgr2=0x7fd0201f2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:53.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200ced00 0x7fd0201f2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 -- 172.21.15.102:0/3395531930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0200c41b0 msgr2=0x7fd0201f1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0200c41b0 0x7fd0201f1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd025d9b700 1 -- 172.21.15.102:0/3395531930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c0094a0 con 0x7fd0200b3970 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.473+0000 7fd02559a700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0200c41b0 0x7fd0201f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.474+0000 7fd025d9b700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0200b3970 0x7fd02014e350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00ec10 tx=0x7fd01c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.474+0000 7fd0167fc700 1 -- 172.21.15.102:0/3395531930 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00c850 con 0x7fd0200b3970 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.474+0000 7fd0167fc700 1 -- 172.21.15.102:0/3395531930 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c00c9b0 con 0x7fd0200b3970 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.474+0000 7fd0167fc700 1 -- 172.21.15.102:0/3395531930 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00cd20 con 0x7fd0200b3970 2023-03-03T12:19:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.474+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd018009a70 con 0x7fd0200b3970 2023-03-03T12:19:53.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.474+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02006bf20 con 0x7fd0200b3970 2023-03-03T12:19:53.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.475+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02015ce80 con 0x7fd0200b3970 2023-03-03T12:19:53.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.476+0000 7fd0167fc700 1 -- 172.21.15.102:0/3395531930 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd01c023070 con 0x7fd0200b3970 2023-03-03T12:19:53.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.478+0000 7fd0167fc700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd0040682f0 0x7fd00406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:53.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.478+0000 7fd0167fc700 1 -- 172.21.15.102:0/3395531930 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd01c085fe0 con 0x7fd0200b3970 2023-03-03T12:19:53.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.478+0000 7fd02559a700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd0040682f0 0x7fd00406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:53.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.479+0000 7fd02559a700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd0040682f0 0x7fd00406a7a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd020151b20 tx=0x7fd010007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:53.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.482+0000 7fd0167fc700 1 -- 172.21.15.102:0/3395531930 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd01c0526f0 con 0x7fd0200b3970 2023-03-03T12:19:53.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.716+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd02006c200 con 0x7fd0200b3970 2023-03-03T12:19:53.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.717+0000 7fd0167fc700 1 -- 172.21.15.102:0/3395531930 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd01c055d10 con 0x7fd0200b3970 2023-03-03T12:19:53.720 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:19:53.720 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:19:53.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.719+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd0040682f0 msgr2=0x7fd00406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.719+0000 7fd026d9d700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd0040682f0 0x7fd00406a7a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd020151b20 tx=0x7fd010007e70 comp rx=0 tx=0).stop 2023-03-03T12:19:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.719+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0200b3970 msgr2=0x7fd02014e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.719+0000 7fd026d9d700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0200b3970 0x7fd02014e350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00ec10 tx=0x7fd01c00ef20 comp rx=0 tx=0).stop 2023-03-03T12:19:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 shutdown_connections 2023-03-03T12:19:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd0040682f0 0x7fd00406a7a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd0200ced00 0x7fd0201f2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0200b3970 0x7fd02014e350 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 --2- 172.21.15.102:0/3395531930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0200c41b0 0x7fd0201f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 >> 172.21.15.102:0/3395531930 conn(0x7fd02008b1c0 msgr2=0x7fd0200cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 shutdown_connections 2023-03-03T12:19:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:19:53.720+0000 7fd026d9d700 1 -- 172.21.15.102:0/3395531930 wait complete. 2023-03-03T12:19:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:19:53.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 101 secs before thrashing 2023-03-03T12:19:54.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:54.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T12:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:20:24.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:24.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:54.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:54.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:24.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:24.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:34.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:21:35.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.997+0000 7f83441ff700 1 Processor -- start 2023-03-03T12:21:35.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.997+0000 7f83441ff700 1 -- start start 2023-03-03T12:21:35.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.998+0000 7f83441ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f833c0cd8e0 0x7f833c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:35.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.998+0000 7f83441ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f833c0cd390 0x7f833c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:35.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.998+0000 7f83441ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c153660 con 0x7f833c153290 2023-03-03T12:21:35.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f83441ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c06c360 con 0x7f833c0cd8e0 2023-03-03T12:21:35.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f83441ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c15cea0 con 0x7f833c0cd390 2023-03-03T12:21:35.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834179a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f833c0cd8e0 0x7f833c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:35.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834179a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f833c0cd8e0 0x7f833c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46712/0 (socket says 172.21.15.102:46712) 2023-03-03T12:21:35.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834179a700 1 -- 172.21.15.102:0/441153352 learned_addr learned my addr 172.21.15.102:0/441153352 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:21:35.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834279c700 1 --2- 172.21.15.102:0/441153352 >> v2:172.21.15.172:3300/0 conn(0x7f833c0cd390 0x7f833c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:35.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834179a700 1 -- 172.21.15.102:0/441153352 >> v2:172.21.15.172:3300/0 conn(0x7f833c0cd390 msgr2=0x7f833c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:21:35.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834179a700 1 --2- 172.21.15.102:0/441153352 >> v2:172.21.15.172:3300/0 conn(0x7f833c0cd390 0x7f833c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834179a700 1 -- 172.21.15.102:0/441153352 >> v1:172.21.15.172:6789/0 conn(0x7f833c153290 legacy=0x7f833c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:35.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f834179a700 1 -- 172.21.15.102:0/441153352 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c06c7f0 con 0x7f833c0cd8e0 2023-03-03T12:21:35.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:34.999+0000 7f8340f99700 1 -- 172.21.15.102:0/441153352 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206414692 0 0) 0x7f833c153660 con 0x7f833c153290 2023-03-03T12:21:35.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.000+0000 7f834179a700 1 --2- 172.21.15.102:0/441153352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c0cde20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f833c0c9390 tx=0x7f832c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff4dcd029c7ff35 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:21:35.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.000+0000 7f8340f99700 1 -- 172.21.15.102:0/441153352 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c011830 con 0x7f833c0cd8e0 2023-03-03T12:21:35.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.000+0000 7f8340f99700 1 -- 172.21.15.102:0/441153352 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c011990 con 0x7f833c0cd8e0 2023-03-03T12:21:35.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.000+0000 7f8340f99700 1 -- 172.21.15.102:0/441153352 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c011d20 con 0x7f833c0cd8e0 2023-03-03T12:21:35.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.000+0000 7f83441ff700 1 -- 172.21.15.102:0/441153352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 msgr2=0x7f833c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:35.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.000+0000 7f83441ff700 1 --2- 172.21.15.102:0/441153352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c0cde20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f833c0c9390 tx=0x7f832c009bd0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.001+0000 7f83441ff700 1 -- 172.21.15.102:0/441153352 shutdown_connections 2023-03-03T12:21:35.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.001+0000 7f83441ff700 1 --2- 172.21.15.102:0/441153352 >> v2:172.21.15.172:3300/0 conn(0x7f833c0cd390 0x7f833c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.001+0000 7f83441ff700 1 --2- 172.21.15.102:0/441153352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c0cde20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.001+0000 7f83441ff700 1 -- 172.21.15.102:0/441153352 >> 172.21.15.102:0/441153352 conn(0x7f833c065fb0 msgr2=0x7f833c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:21:35.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.001+0000 7f83441ff700 1 -- 172.21.15.102:0/441153352 shutdown_connections 2023-03-03T12:21:35.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.001+0000 7f83441ff700 1 -- 172.21.15.102:0/441153352 wait complete. 2023-03-03T12:21:35.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.003+0000 7f83441ff700 1 Processor -- start 2023-03-03T12:21:35.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.003+0000 7f83441ff700 1 -- start start 2023-03-03T12:21:35.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f83441ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f833c0cd390 0x7f833c1f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:35.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f83441ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c1e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:35.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f83441ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f833c153290 0x7f833c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:35.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f83441ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c1e4260 con 0x7f833c0cd8e0 2023-03-03T12:21:35.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f83441ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c1e4cd0 con 0x7f833c153290 2023-03-03T12:21:35.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f83441ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0c9020 con 0x7f833c0cd390 2023-03-03T12:21:35.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f834279c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f833c153290 0x7f833c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:35.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f8341f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f833c0cd390 0x7f833c1f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:35.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f834279c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f833c153290 0x7f833c1570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56504/0 (socket says 172.21.15.102:56504) 2023-03-03T12:21:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f834279c700 1 -- 172.21.15.102:0/3978234336 learned_addr learned my addr 172.21.15.102:0/3978234336 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:21:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.004+0000 7f834179a700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c1e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f834279c700 1 -- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f833c0cd390 msgr2=0x7f833c1f2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f834279c700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f833c0cd390 0x7f833c1f2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f834279c700 1 -- 172.21.15.102:0/3978234336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 msgr2=0x7f833c1e5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f834279c700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c1e5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f834279c700 1 -- 172.21.15.102:0/3978234336 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c009550 con 0x7f833c153290 2023-03-03T12:21:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f834179a700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c1e5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:21:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f834279c700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f833c153290 0x7f833c1570a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f833400efc0 tx=0x7f833400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:21:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f83327fc700 1 -- 172.21.15.102:0/3978234336 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334009ea0 con 0x7f833c153290 2023-03-03T12:21:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f83327fc700 1 -- 172.21.15.102:0/3978234336 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833400f040 con 0x7f833c153290 2023-03-03T12:21:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c0ccde0 con 0x7f833c153290 2023-03-03T12:21:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f83327fc700 1 -- 172.21.15.102:0/3978234336 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334014480 con 0x7f833c153290 2023-03-03T12:21:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.005+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c15d160 con 0x7f833c153290 2023-03-03T12:21:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.006+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c0cd0c0 con 0x7f833c153290 2023-03-03T12:21:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.008+0000 7f83327fc700 1 -- 172.21.15.102:0/3978234336 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f83340145e0 con 0x7f833c153290 2023-03-03T12:21:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.009+0000 7f83327fc700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8320068320 0x7f832006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.009+0000 7f83327fc700 1 -- 172.21.15.102:0/3978234336 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8334086cc0 con 0x7f833c153290 2023-03-03T12:21:35.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.009+0000 7f8341f9b700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8320068320 0x7f832006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:35.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.010+0000 7f8341f9b700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8320068320 0x7f832006a7d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f8338000f60 tx=0x7f8338009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:21:35.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.012+0000 7f83327fc700 1 -- 172.21.15.102:0/3978234336 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8334053450 con 0x7f833c153290 2023-03-03T12:21:35.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.246+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f833c0cc2d0 con 0x7f833c153290 2023-03-03T12:21:35.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.248+0000 7f83327fc700 1 -- 172.21.15.102:0/3978234336 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8334018020 con 0x7f833c153290 2023-03-03T12:21:35.250 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:21:35.250 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:21:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8320068320 msgr2=0x7f832006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8320068320 0x7f832006a7d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f8338000f60 tx=0x7f8338009040 comp rx=0 tx=0).stop 2023-03-03T12:21:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f833c153290 msgr2=0x7f833c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f833c153290 0x7f833c1570a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f833400efc0 tx=0x7f833400c3a0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 shutdown_connections 2023-03-03T12:21:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8320068320 0x7f832006a7d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f833c0cd390 0x7f833c1f2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f833c153290 0x7f833c1570a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 --2- 172.21.15.102:0/3978234336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f833c0cd8e0 0x7f833c1e5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 >> 172.21.15.102:0/3978234336 conn(0x7f833c065fb0 msgr2=0x7f833c0cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:21:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 shutdown_connections 2023-03-03T12:21:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:21:35.250+0000 7f83441ff700 1 -- 172.21.15.102:0/3978234336 wait complete. 2023-03-03T12:21:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:21:35.280 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-03-03T12:21:55.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:55.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:25.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:25.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:52.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:22:52.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.532+0000 7f257299d700 1 Processor -- start 2023-03-03T12:22:52.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.532+0000 7f257299d700 1 -- start start 2023-03-03T12:22:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257299d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c1564b0 0x7f256c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257299d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f256c0c9610 0x7f256c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257299d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f256c0ca440 0x7f256c0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257299d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256c0c9ca0 con 0x7f256c0c9610 2023-03-03T12:22:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257299d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256c0b46f0 con 0x7f256c0ca440 2023-03-03T12:22:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257299d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0cadc0 con 0x7f256c1564b0 2023-03-03T12:22:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257119a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f256c0c9610 0x7f256c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257199b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c1564b0 0x7f256c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257119a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f256c0c9610 0x7f256c0cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37346/0 (socket says 172.21.15.102:37346) 2023-03-03T12:22:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257119a700 1 -- 172.21.15.102:0/2310746256 learned_addr learned my addr 172.21.15.102:0/2310746256 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:22:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.534+0000 7f257219c700 1 --2- 172.21.15.102:0/2310746256 >> v2:172.21.15.172:3300/0 conn(0x7f256c0ca440 0x7f256c0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f257119a700 1 -- 172.21.15.102:0/2310746256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c1564b0 msgr2=0x7f256c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:22:52.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f257119a700 1 --2- 172.21.15.102:0/2310746256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c1564b0 0x7f256c0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f257119a700 1 -- 172.21.15.102:0/2310746256 >> v2:172.21.15.172:3300/0 conn(0x7f256c0ca440 msgr2=0x7f256c0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:22:52.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f257119a700 1 --2- 172.21.15.102:0/2310746256 >> v2:172.21.15.172:3300/0 conn(0x7f256c0ca440 0x7f256c0ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f257119a700 1 -- 172.21.15.102:0/2310746256 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c0b4920 con 0x7f256c0c9610 2023-03-03T12:22:52.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f257119a700 1 --2- 172.21.15.102:0/2310746256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c0c9610 0x7f256c0cbba0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f256c0b1c10 tx=0x7f255c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bf8bb1b7ddf4a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:52.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f2570999700 1 -- 172.21.15.102:0/2310746256 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c011830 con 0x7f256c0c9610 2023-03-03T12:22:52.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f2570999700 1 -- 172.21.15.102:0/2310746256 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c011990 con 0x7f256c0c9610 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.535+0000 7f2570999700 1 -- 172.21.15.102:0/2310746256 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c011d20 con 0x7f256c0c9610 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 -- 172.21.15.102:0/2310746256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c0c9610 msgr2=0x7f256c0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 --2- 172.21.15.102:0/2310746256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c0c9610 0x7f256c0cbba0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f256c0b1c10 tx=0x7f255c009bd0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 -- 172.21.15.102:0/2310746256 shutdown_connections 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 --2- 172.21.15.102:0/2310746256 >> v2:172.21.15.172:3300/0 conn(0x7f256c0ca440 0x7f256c0ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 --2- 172.21.15.102:0/2310746256 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c1564b0 0x7f256c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 --2- 172.21.15.102:0/2310746256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c0c9610 0x7f256c0cbba0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 -- 172.21.15.102:0/2310746256 >> 172.21.15.102:0/2310746256 conn(0x7f256c065fb0 msgr2=0x7f256c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:22:52.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 -- 172.21.15.102:0/2310746256 shutdown_connections 2023-03-03T12:22:52.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.536+0000 7f257299d700 1 -- 172.21.15.102:0/2310746256 wait complete. 2023-03-03T12:22:52.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.538+0000 7f257299d700 1 Processor -- start 2023-03-03T12:22:52.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.538+0000 7f257299d700 1 -- start start 2023-03-03T12:22:52.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.539+0000 7f257299d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f256c0c9610 0x7f256c1ef710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:52.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.539+0000 7f257299d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c0ca440 0x7f256c0ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:52.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.539+0000 7f257299d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c1564b0 0x7f256c1f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:52.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.539+0000 7f257299d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c1f2fe0 con 0x7f256c1564b0 2023-03-03T12:22:52.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.539+0000 7f257299d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0b1900 con 0x7f256c0c9610 2023-03-03T12:22:52.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.539+0000 7f257299d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c158e20 con 0x7f256c0ca440 2023-03-03T12:22:52.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f256c0c9610 0x7f256c1ef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:52.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257119a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c0ca440 0x7f256c0ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:52.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f256c0c9610 0x7f256c1ef710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54778/0 (socket says 172.21.15.102:54778) 2023-03-03T12:22:52.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 -- 172.21.15.102:0/4218062019 learned_addr learned my addr 172.21.15.102:0/4218062019 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:22:52.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257219c700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c1564b0 0x7f256c1f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:52.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 -- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c0ca440 msgr2=0x7f256c0ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:52.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c0ca440 0x7f256c0ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 -- 172.21.15.102:0/4218062019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c1564b0 msgr2=0x7f256c1f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:52.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c1564b0 0x7f256c1f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.540+0000 7f257199b700 1 -- 172.21.15.102:0/4218062019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c009550 con 0x7f256c0c9610 2023-03-03T12:22:52.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.541+0000 7f257199b700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f256c0c9610 0x7f256c1ef710 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f256800e870 tx=0x7f256800eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:52.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.541+0000 7f25627fc700 1 -- 172.21.15.102:0/4218062019 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568009590 con 0x7f256c0c9610 2023-03-03T12:22:52.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.541+0000 7f25627fc700 1 -- 172.21.15.102:0/4218062019 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2568013680 con 0x7f256c0c9610 2023-03-03T12:22:52.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.541+0000 7f25627fc700 1 -- 172.21.15.102:0/4218062019 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25680139f0 con 0x7f256c0c9610 2023-03-03T12:22:52.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.541+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256c0b1f30 con 0x7f256c0c9610 2023-03-03T12:22:52.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.541+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c0b4c00 con 0x7f256c0c9610 2023-03-03T12:22:52.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.542+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c0b2210 con 0x7f256c0c9610 2023-03-03T12:22:52.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.543+0000 7f25627fc700 1 -- 172.21.15.102:0/4218062019 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2568019030 con 0x7f256c0c9610 2023-03-03T12:22:52.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.544+0000 7f25627fc700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2550068320 0x7f255006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:52.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.545+0000 7f25627fc700 1 -- 172.21.15.102:0/4218062019 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2568085a70 con 0x7f256c0c9610 2023-03-03T12:22:52.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.545+0000 7f257119a700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2550068320 0x7f255006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:52.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.546+0000 7f257119a700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2550068320 0x7f255006a7d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f255c009c10 tx=0x7f255c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:52.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.549+0000 7f25627fc700 1 -- 172.21.15.102:0/4218062019 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2568052180 con 0x7f256c0c9610 2023-03-03T12:22:52.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.782+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f256c0b1a40 con 0x7f256c0c9610 2023-03-03T12:22:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.783+0000 7f25627fc700 1 -- 172.21.15.102:0/4218062019 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f25680557a0 con 0x7f256c0c9610 2023-03-03T12:22:52.786 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:22:52.787 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:22:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2550068320 msgr2=0x7f255006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2550068320 0x7f255006a7d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f255c009c10 tx=0x7f255c008030 comp rx=0 tx=0).stop 2023-03-03T12:22:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f256c0c9610 msgr2=0x7f256c1ef710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f256c0c9610 0x7f256c1ef710 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f256800e870 tx=0x7f256800eb80 comp rx=0 tx=0).stop 2023-03-03T12:22:52.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 shutdown_connections 2023-03-03T12:22:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f256c1564b0 0x7f256c1f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f256c0c9610 0x7f256c1ef710 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2550068320 0x7f255006a7d0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 --2- 172.21.15.102:0/4218062019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f256c0ca440 0x7f256c0ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 >> 172.21.15.102:0/4218062019 conn(0x7f256c065fb0 msgr2=0x7f256c155550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:22:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 shutdown_connections 2023-03-03T12:22:52.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:22:52.786+0000 7f257299d700 1 -- 172.21.15.102:0/4218062019 wait complete. 2023-03-03T12:22:52.791 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:22:52.816 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-03-03T12:22:55.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:55.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:25.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:25.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:56.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:56.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:26.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:26.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:28.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:24:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.064+0000 7f36705ff700 1 Processor -- start 2023-03-03T12:24:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.064+0000 7f36705ff700 1 -- start start 2023-03-03T12:24:29.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.065+0000 7f36705ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:29.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.065+0000 7f36705ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f36680cd390 0x7f36680ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f36705ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3668153660 con 0x7f3668153290 2023-03-03T12:24:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f36705ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366806c360 con 0x7f36680cd390 2023-03-03T12:24:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f36705ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366815cea0 con 0x7f36680cd8e0 2023-03-03T12:24:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366db9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366e39b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3668153290 0x7f3668158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46088/0 (socket says 172.21.15.102:46088) 2023-03-03T12:24:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366eb9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f36680cd390 0x7f36680ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366e39b700 1 -- 172.21.15.102:0/2360487497 learned_addr learned my addr 172.21.15.102:0/2360487497 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:24:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366db9a700 1 -- 172.21.15.102:0/2360487497 >> v2:172.21.15.172:3300/0 conn(0x7f36680cd390 msgr2=0x7f36680ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:24:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366db9a700 1 --2- 172.21.15.102:0/2360487497 >> v2:172.21.15.172:3300/0 conn(0x7f36680cd390 0x7f36680ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366db9a700 1 -- 172.21.15.102:0/2360487497 >> v1:172.21.15.172:6789/0 conn(0x7f3668153290 legacy=0x7f3668158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366db9a700 1 -- 172.21.15.102:0/2360487497 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366806c7f0 con 0x7f36680cd8e0 2023-03-03T12:24:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.066+0000 7f366d399700 1 -- 172.21.15.102:0/2360487497 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59422872 0 0) 0x7f3668153660 con 0x7f3668153290 2023-03-03T12:24:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.067+0000 7f366db9a700 1 --2- 172.21.15.102:0/2360487497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680cde20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3658009840 tx=0x7f3658009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0481439d7334f5f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.067+0000 7f366d399700 1 -- 172.21.15.102:0/2360487497 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658011830 con 0x7f36680cd8e0 2023-03-03T12:24:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.067+0000 7f366d399700 1 -- 172.21.15.102:0/2360487497 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658011990 con 0x7f36680cd8e0 2023-03-03T12:24:29.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.067+0000 7f366d399700 1 -- 172.21.15.102:0/2360487497 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658011d20 con 0x7f36680cd8e0 2023-03-03T12:24:29.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 -- 172.21.15.102:0/2360487497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 msgr2=0x7f36680cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:29.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 --2- 172.21.15.102:0/2360487497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680cde20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3658009840 tx=0x7f3658009b50 comp rx=0 tx=0).stop 2023-03-03T12:24:29.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 -- 172.21.15.102:0/2360487497 shutdown_connections 2023-03-03T12:24:29.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 --2- 172.21.15.102:0/2360487497 >> v2:172.21.15.172:3300/0 conn(0x7f36680cd390 0x7f36680ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 --2- 172.21.15.102:0/2360487497 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680cde20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 -- 172.21.15.102:0/2360487497 >> 172.21.15.102:0/2360487497 conn(0x7f3668065fb0 msgr2=0x7f36680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:24:29.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 -- 172.21.15.102:0/2360487497 shutdown_connections 2023-03-03T12:24:29.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.068+0000 7f36705ff700 1 -- 172.21.15.102:0/2360487497 wait complete. 2023-03-03T12:24:29.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.070+0000 7f36705ff700 1 Processor -- start 2023-03-03T12:24:29.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.070+0000 7f36705ff700 1 -- start start 2023-03-03T12:24:29.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f36705ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36680cd390 0x7f36681537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:29.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f36705ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:29.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f36705ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668153290 0x7f36681549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:29.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f36705ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3668153e80 con 0x7f3668153290 2023-03-03T12:24:29.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f36705ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36680b2d80 con 0x7f36680cd390 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f36705ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36680b3060 con 0x7f36680cd8e0 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f366e39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36680cd390 0x7f36681537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f366db9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f366eb9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668153290 0x7f36681549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f366e39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36680cd390 0x7f36681537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:44528/0 (socket says 172.21.15.102:44528) 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f366db9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:53732/0 (socket says 172.21.15.102:53732) 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.071+0000 7f366e39b700 1 -- 172.21.15.102:0/1424189576 learned_addr learned my addr 172.21.15.102:0/1424189576 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:24:29.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f366db9a700 1 -- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36680cd390 msgr2=0x7f36681537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:29.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f366db9a700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36680cd390 0x7f36681537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f366db9a700 1 -- 172.21.15.102:0/1424189576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668153290 msgr2=0x7f36681549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:29.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f366db9a700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668153290 0x7f36681549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f366db9a700 1 -- 172.21.15.102:0/1424189576 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36580094a0 con 0x7f36680cd8e0 2023-03-03T12:24:29.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f366e39b700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36680cd390 0x7f36681537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:24:29.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f366db9a700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680b20a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3658016000 tx=0x7f3658004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:29.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.072+0000 7f365effd700 1 -- 172.21.15.102:0/1424189576 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36580036a0 con 0x7f36680cd8e0 2023-03-03T12:24:29.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.073+0000 7f365effd700 1 -- 172.21.15.102:0/1424189576 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365801e070 con 0x7f36680cd8e0 2023-03-03T12:24:29.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.073+0000 7f365effd700 1 -- 172.21.15.102:0/1424189576 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658003a40 con 0x7f36680cd8e0 2023-03-03T12:24:29.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.073+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36680ccde0 con 0x7f36680cd8e0 2023-03-03T12:24:29.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.073+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366815d160 con 0x7f36680cd8e0 2023-03-03T12:24:29.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.074+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36680cd0c0 con 0x7f36680cd8e0 2023-03-03T12:24:29.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.075+0000 7f365effd700 1 -- 172.21.15.102:0/1424189576 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f365800f050 con 0x7f36680cd8e0 2023-03-03T12:24:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.076+0000 7f365effd700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f364c0682f0 0x7f364c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.076+0000 7f365effd700 1 -- 172.21.15.102:0/1424189576 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3658086d60 con 0x7f36680cd8e0 2023-03-03T12:24:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.077+0000 7f366e39b700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f364c0682f0 0x7f364c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.077+0000 7f366e39b700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f364c0682f0 0x7f364c06a7a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f3664000f60 tx=0x7f3664007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.081+0000 7f365effd700 1 -- 172.21.15.102:0/1424189576 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3658053470 con 0x7f36680cd8e0 2023-03-03T12:24:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.313+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36680b31a0 con 0x7f36680cd8e0 2023-03-03T12:24:29.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.315+0000 7f365effd700 1 -- 172.21.15.102:0/1424189576 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f36680b31a0 con 0x7f36680cd8e0 2023-03-03T12:24:29.317 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:24:29.318 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:24:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f364c0682f0 msgr2=0x7f364c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f364c0682f0 0x7f364c06a7a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f3664000f60 tx=0x7f3664007ea0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 msgr2=0x7f36680b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680b20a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3658016000 tx=0x7f3658004750 comp rx=0 tx=0).stop 2023-03-03T12:24:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 shutdown_connections 2023-03-03T12:24:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668153290 0x7f36681549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36680cd390 0x7f36681537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f364c0682f0 0x7f364c06a7a0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 --2- 172.21.15.102:0/1424189576 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f36680cd8e0 0x7f36680b20a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 >> 172.21.15.102:0/1424189576 conn(0x7f3668065fb0 msgr2=0x7f3668156b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:24:29.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 shutdown_connections 2023-03-03T12:24:29.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:24:29.317+0000 7f36705ff700 1 -- 172.21.15.102:0/1424189576 wait complete. 2023-03-03T12:24:29.322 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:24:29.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T12:24:56.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:56.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:26.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:26.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:53.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:25:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.602+0000 7f19c91ff700 1 Processor -- start 2023-03-03T12:25:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.602+0000 7f19c91ff700 1 -- start start 2023-03-03T12:25:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.603+0000 7f19c91ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19c4153290 0x7f19c4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c91ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f19c41593c0 0x7f19c40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c91ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c40cd8e0 0x7f19c40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c91ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c4153660 con 0x7f19c4153290 2023-03-03T12:25:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c91ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c406c360 con 0x7f19c41593c0 2023-03-03T12:25:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c91ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c415cea0 con 0x7f19c40cd8e0 2023-03-03T12:25:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19c4153290 0x7f19c4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19c4153290 0x7f19c4158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54490/0 (socket says 172.21.15.102:54490) 2023-03-03T12:25:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 -- 172.21.15.102:0/4164065858 learned_addr learned my addr 172.21.15.102:0/4164065858 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:25:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c259c700 1 --2- 172.21.15.102:0/4164065858 >> v2:172.21.15.172:3300/0 conn(0x7f19c41593c0 0x7f19c40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c359e700 1 --2- 172.21.15.102:0/4164065858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c40cd8e0 0x7f19c40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 -- 172.21.15.102:0/4164065858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c40cd8e0 msgr2=0x7f19c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:25:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 --2- 172.21.15.102:0/4164065858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c40cd8e0 0x7f19c40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 -- 172.21.15.102:0/4164065858 >> v2:172.21.15.172:3300/0 conn(0x7f19c41593c0 msgr2=0x7f19c40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:25:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 --2- 172.21.15.102:0/4164065858 >> v2:172.21.15.172:3300/0 conn(0x7f19c41593c0 0x7f19c40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.604+0000 7f19c2d9d700 1 -- 172.21.15.102:0/4164065858 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c406c7f0 con 0x7f19c4153290 2023-03-03T12:25:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.605+0000 7f19c2d9d700 1 --2- 172.21.15.102:0/4164065858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c4153290 0x7f19c4158e80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f19c4159fb0 tx=0x7f19b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a9b55418d9134a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.605+0000 7f19c1d9b700 1 -- 172.21.15.102:0/4164065858 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4011830 con 0x7f19c4153290 2023-03-03T12:25:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.605+0000 7f19c1d9b700 1 -- 172.21.15.102:0/4164065858 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b4011990 con 0x7f19c4153290 2023-03-03T12:25:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.605+0000 7f19c1d9b700 1 -- 172.21.15.102:0/4164065858 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4011d20 con 0x7f19c4153290 2023-03-03T12:25:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 -- 172.21.15.102:0/4164065858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c4153290 msgr2=0x7f19c4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 --2- 172.21.15.102:0/4164065858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c4153290 0x7f19c4158e80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f19c4159fb0 tx=0x7f19b4009bd0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 -- 172.21.15.102:0/4164065858 shutdown_connections 2023-03-03T12:25:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 --2- 172.21.15.102:0/4164065858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c40cd8e0 0x7f19c40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 --2- 172.21.15.102:0/4164065858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c4153290 0x7f19c4158e80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 --2- 172.21.15.102:0/4164065858 >> v2:172.21.15.172:3300/0 conn(0x7f19c41593c0 0x7f19c40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 -- 172.21.15.102:0/4164065858 >> 172.21.15.102:0/4164065858 conn(0x7f19c4065fb0 msgr2=0x7f19c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:25:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 -- 172.21.15.102:0/4164065858 shutdown_connections 2023-03-03T12:25:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.606+0000 7f19c91ff700 1 -- 172.21.15.102:0/4164065858 wait complete. 2023-03-03T12:25:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.608+0000 7f19c91ff700 1 Processor -- start 2023-03-03T12:25:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.608+0000 7f19c91ff700 1 -- start start 2023-03-03T12:25:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.609+0000 7f19c91ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c40cd8e0 0x7f19c40c3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.609+0000 7f19c91ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19c4153290 0x7f19c41500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.609+0000 7f19c91ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c41593c0 0x7f19c4150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.609+0000 7f19c91ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c41eec40 con 0x7f19c40cd8e0 2023-03-03T12:25:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.609+0000 7f19c91ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40c4140 con 0x7f19c4153290 2023-03-03T12:25:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.609+0000 7f19c91ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40c4420 con 0x7f19c41593c0 2023-03-03T12:25:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.609+0000 7f19c2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c40cd8e0 0x7f19c40c3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c40cd8e0 0x7f19c40c3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54504/0 (socket says 172.21.15.102:54504) 2023-03-03T12:25:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 -- 172.21.15.102:0/18493095 learned_addr learned my addr 172.21.15.102:0/18493095 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:25:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c259c700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19c4153290 0x7f19c41500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c359e700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c41593c0 0x7f19c4150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 -- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c41593c0 msgr2=0x7f19c4150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c41593c0 0x7f19c4150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 -- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19c4153290 msgr2=0x7f19c41500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19c4153290 0x7f19c41500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 -- 172.21.15.102:0/18493095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b4009550 con 0x7f19c40cd8e0 2023-03-03T12:25:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.610+0000 7f19c2d9d700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c40cd8e0 0x7f19c40c3220 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009c10 tx=0x7f19b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.611+0000 7f19ab7fe700 1 -- 172.21.15.102:0/18493095 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4004450 con 0x7f19c40cd8e0 2023-03-03T12:25:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.611+0000 7f19ab7fe700 1 -- 172.21.15.102:0/18493095 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b401e070 con 0x7f19c40cd8e0 2023-03-03T12:25:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.611+0000 7f19ab7fe700 1 -- 172.21.15.102:0/18493095 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b40038e0 con 0x7f19c40cd8e0 2023-03-03T12:25:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.611+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c40ccde0 con 0x7f19c40cd8e0 2023-03-03T12:25:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.611+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c415d160 con 0x7f19c40cd8e0 2023-03-03T12:25:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.612+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c40cd0c0 con 0x7f19c40cd8e0 2023-03-03T12:25:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.613+0000 7f19ab7fe700 1 -- 172.21.15.102:0/18493095 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f19b400f050 con 0x7f19c40cd8e0 2023-03-03T12:25:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.615+0000 7f19ab7fe700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19b0068320 0x7f19b006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.615+0000 7f19ab7fe700 1 -- 172.21.15.102:0/18493095 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f19b4086a90 con 0x7f19c40cd8e0 2023-03-03T12:25:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.615+0000 7f19c259c700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19b0068320 0x7f19b006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.616+0000 7f19c259c700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19b0068320 0x7f19b006a7d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f19c40b49e0 tx=0x7f19b8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:53.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.620+0000 7f19ab7fe700 1 -- 172.21.15.102:0/18493095 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f19b40531a0 con 0x7f19c40cd8e0 2023-03-03T12:25:53.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.855+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19c404e6e0 con 0x7f19c40cd8e0 2023-03-03T12:25:53.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.856+0000 7f19ab7fe700 1 -- 172.21.15.102:0/18493095 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f19b4020090 con 0x7f19c40cd8e0 2023-03-03T12:25:53.858 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:25:53.858 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:25:53.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.858+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19b0068320 msgr2=0x7f19b006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19b0068320 0x7f19b006a7d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f19c40b49e0 tx=0x7f19b8009210 comp rx=0 tx=0).stop 2023-03-03T12:25:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c40cd8e0 msgr2=0x7f19c40c3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c40cd8e0 0x7f19c40c3220 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009c10 tx=0x7f19b4004750 comp rx=0 tx=0).stop 2023-03-03T12:25:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 shutdown_connections 2023-03-03T12:25:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19b0068320 0x7f19b006a7d0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f19c41593c0 0x7f19c4150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19c40cd8e0 0x7f19c40c3220 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 --2- 172.21.15.102:0/18493095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19c4153290 0x7f19c41500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 >> 172.21.15.102:0/18493095 conn(0x7f19c4065fb0 msgr2=0x7f19c415bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:25:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 shutdown_connections 2023-03-03T12:25:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:25:53.859+0000 7f19c91ff700 1 -- 172.21.15.102:0/18493095 wait complete. 2023-03-03T12:25:53.862 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:25:53.888 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 37 secs before thrashing 2023-03-03T12:25:56.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:56.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:26.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:26.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:30.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:26:31.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.135+0000 7f2d183ff700 1 Processor -- start 2023-03-03T12:26:31.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.136+0000 7f2d183ff700 1 -- start start 2023-03-03T12:26:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.137+0000 7f2d183ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d10153290 0x7f2d10158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.137+0000 7f2d183ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.137+0000 7f2d183ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d10153660 con 0x7f2d101593c0 2023-03-03T12:26:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.137+0000 7f2d183ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1006c360 con 0x7f2d10153290 2023-03-03T12:26:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.137+0000 7f2d183ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1015cea0 con 0x7f2d100cd8e0 2023-03-03T12:26:31.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1699c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1619b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2d10153290 0x7f2d10158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1599a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2d101593c0 0x7f2d100cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52986/0 (socket says 172.21.15.102:52986) 2023-03-03T12:26:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1599a700 1 -- 172.21.15.102:0/3844990127 learned_addr learned my addr 172.21.15.102:0/3844990127 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:26:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d15199700 1 -- 172.21.15.102:0/3844990127 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256238071 0 0) 0x7f2d10153660 con 0x7f2d101593c0 2023-03-03T12:26:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d15199700 1 -- 172.21.15.102:0/3844990127 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cfc0035b0 con 0x7f2d101593c0 2023-03-03T12:26:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1699c700 1 -- 172.21.15.102:0/3844990127 >> v2:172.21.15.172:3300/0 conn(0x7f2d10153290 msgr2=0x7f2d10158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:26:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1699c700 1 --2- 172.21.15.102:0/3844990127 >> v2:172.21.15.172:3300/0 conn(0x7f2d10153290 0x7f2d10158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1699c700 1 -- 172.21.15.102:0/3844990127 >> v1:172.21.15.172:6789/0 conn(0x7f2d101593c0 legacy=0x7f2d100cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1699c700 1 -- 172.21.15.102:0/3844990127 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1006c7f0 con 0x7f2d100cd8e0 2023-03-03T12:26:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1619b700 1 --2- 172.21.15.102:0/3844990127 >> v2:172.21.15.172:3300/0 conn(0x7f2d10153290 0x7f2d10158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:26:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.138+0000 7f2d1699c700 1 --2- 172.21.15.102:0/3844990127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100ce530 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009d90 tx=0x7f2d0800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=545d56a52652d7d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:26:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d15199700 1 -- 172.21.15.102:0/3844990127 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800e070 con 0x7f2d100cd8e0 2023-03-03T12:26:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d15199700 1 -- 172.21.15.102:0/3844990127 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0800bb20 con 0x7f2d100cd8e0 2023-03-03T12:26:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d15199700 1 -- 172.21.15.102:0/3844990127 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08012430 con 0x7f2d100cd8e0 2023-03-03T12:26:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 -- 172.21.15.102:0/3844990127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 msgr2=0x7f2d100ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3844990127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100ce530 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009d90 tx=0x7f2d0800b330 comp rx=0 tx=0).stop 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 -- 172.21.15.102:0/3844990127 shutdown_connections 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3844990127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100ce530 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3844990127 >> v2:172.21.15.172:3300/0 conn(0x7f2d10153290 0x7f2d10158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 -- 172.21.15.102:0/3844990127 >> 172.21.15.102:0/3844990127 conn(0x7f2d10065fb0 msgr2=0x7f2d100663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 -- 172.21.15.102:0/3844990127 shutdown_connections 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.139+0000 7f2d183ff700 1 -- 172.21.15.102:0/3844990127 wait complete. 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.141+0000 7f2d183ff700 1 Processor -- start 2023-03-03T12:26:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.142+0000 7f2d183ff700 1 -- start start 2023-03-03T12:26:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.142+0000 7f2d183ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.142+0000 7f2d183ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d10153290 0x7f2d101f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d183ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d101593c0 0x7f2d100cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d183ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d101537f0 con 0x7f2d101593c0 2023-03-03T12:26:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d183ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d101f1a90 con 0x7f2d10153290 2023-03-03T12:26:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d183ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d101f17b0 con 0x7f2d100cd8e0 2023-03-03T12:26:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d101593c0 0x7f2d100cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1599a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d10153290 0x7f2d101f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d101593c0 0x7f2d100cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55922/0 (socket says 172.21.15.102:55922) 2023-03-03T12:26:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1619b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 -- 172.21.15.102:0/3706847629 learned_addr learned my addr 172.21.15.102:0/3706847629 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:26:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 -- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 msgr2=0x7f2d100c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 -- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d10153290 msgr2=0x7f2d101f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d10153290 0x7f2d101f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1699c700 1 -- 172.21.15.102:0/3706847629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c0094a0 con 0x7f2d101593c0 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.143+0000 7f2d1599a700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d10153290 0x7f2d101f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.144+0000 7f2d1699c700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d101593c0 0x7f2d100cb400 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2d1015cfe0 tx=0x7f2d08004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.144+0000 7f2d06ffd700 1 -- 172.21.15.102:0/3706847629 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d080129f0 con 0x7f2d101593c0 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.144+0000 7f2d06ffd700 1 -- 172.21.15.102:0/3706847629 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d08020070 con 0x7f2d101593c0 2023-03-03T12:26:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.144+0000 7f2d06ffd700 1 -- 172.21.15.102:0/3706847629 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08012db0 con 0x7f2d101593c0 2023-03-03T12:26:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.144+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d08009a70 con 0x7f2d101593c0 2023-03-03T12:26:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.144+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d100ccde0 con 0x7f2d101593c0 2023-03-03T12:26:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.146+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1015d160 con 0x7f2d101593c0 2023-03-03T12:26:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.146+0000 7f2d06ffd700 1 -- 172.21.15.102:0/3706847629 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d0802e070 con 0x7f2d101593c0 2023-03-03T12:26:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.148+0000 7f2d06ffd700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2cf40682f0 0x7f2cf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.148+0000 7f2d06ffd700 1 -- 172.21.15.102:0/3706847629 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2d08022070 con 0x7f2d101593c0 2023-03-03T12:26:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.148+0000 7f2d1619b700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2cf40682f0 0x7f2cf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.149+0000 7f2d1619b700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2cf40682f0 0x7f2cf406a7a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c009cd0 tx=0x7f2d0c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:26:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.152+0000 7f2d06ffd700 1 -- 172.21.15.102:0/3706847629 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d08053fe0 con 0x7f2d101593c0 2023-03-03T12:26:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.388+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d100cd0c0 con 0x7f2d101593c0 2023-03-03T12:26:31.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.389+0000 7f2d06ffd700 1 -- 172.21.15.102:0/3706847629 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2d08057600 con 0x7f2d101593c0 2023-03-03T12:26:31.391 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:26:31.391 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:26:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2cf40682f0 msgr2=0x7f2cf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2cf40682f0 0x7f2cf406a7a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c009cd0 tx=0x7f2d0c009210 comp rx=0 tx=0).stop 2023-03-03T12:26:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d101593c0 msgr2=0x7f2d100cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d101593c0 0x7f2d100cb400 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2d1015cfe0 tx=0x7f2d08004ad0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 shutdown_connections 2023-03-03T12:26:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2cf40682f0 0x7f2cf406a7a0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2d100cd8e0 0x7f2d100c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d101593c0 0x7f2d100cb400 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 --2- 172.21.15.102:0/3706847629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d10153290 0x7f2d101f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.391+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 >> 172.21.15.102:0/3706847629 conn(0x7f2d10065fb0 msgr2=0x7f2d10156b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:26:31.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.392+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 shutdown_connections 2023-03-03T12:26:31.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:26:31.392+0000 7f2d183ff700 1 -- 172.21.15.102:0/3706847629 wait complete. 2023-03-03T12:26:31.395 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:26:31.421 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T12:26:57.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:57.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:11.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:27:11.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.670+0000 7f810e1ff700 1 Processor -- start 2023-03-03T12:27:11.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.671+0000 7f810e1ff700 1 -- start start 2023-03-03T12:27:11.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f810e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f8108158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:11.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f810e1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81081593c0 0x7f81080cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:11.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f810e1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108153660 con 0x7f81080cd8e0 2023-03-03T12:27:11.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f810e1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810806c360 con 0x7f81081593c0 2023-03-03T12:27:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f810e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810815cea0 con 0x7f8108153290 2023-03-03T12:27:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f8107fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f81080cd8e0 0x7f81080ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34702/0 (socket says 172.21.15.102:34702) 2023-03-03T12:27:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f8107fff700 1 -- 172.21.15.102:0/3314224152 learned_addr learned my addr 172.21.15.102:0/3314224152 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:27:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f8106ffd700 1 --2- 172.21.15.102:0/3314224152 >> v2:172.21.15.102:3300/0 conn(0x7f81081593c0 0x7f81080cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.672+0000 7f81077fe700 1 --2- 172.21.15.102:0/3314224152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f8108158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f81067fc700 1 -- 172.21.15.102:0/3314224152 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440352648 0 0) 0x7f8108153660 con 0x7f81080cd8e0 2023-03-03T12:27:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f81067fc700 1 -- 172.21.15.102:0/3314224152 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f00035b0 con 0x7f81080cd8e0 2023-03-03T12:27:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f81077fe700 1 -- 172.21.15.102:0/3314224152 >> v2:172.21.15.102:3300/0 conn(0x7f81081593c0 msgr2=0x7f81080cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:27:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f81077fe700 1 --2- 172.21.15.102:0/3314224152 >> v2:172.21.15.102:3300/0 conn(0x7f81081593c0 0x7f81080cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f81077fe700 1 -- 172.21.15.102:0/3314224152 >> v1:172.21.15.172:6789/0 conn(0x7f81080cd8e0 legacy=0x7f81080ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f81077fe700 1 -- 172.21.15.102:0/3314224152 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810806c7f0 con 0x7f8108153290 2023-03-03T12:27:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f8106ffd700 1 --2- 172.21.15.102:0/3314224152 >> v2:172.21.15.102:3300/0 conn(0x7f81081593c0 0x7f81080cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:27:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.673+0000 7f81077fe700 1 --2- 172.21.15.102:0/3314224152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f8108158e80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009840 tx=0x7f80f8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f77b559da91ea5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f81067fc700 1 -- 172.21.15.102:0/3314224152 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8011830 con 0x7f8108153290 2023-03-03T12:27:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f81067fc700 1 -- 172.21.15.102:0/3314224152 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8011990 con 0x7f8108153290 2023-03-03T12:27:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f81067fc700 1 -- 172.21.15.102:0/3314224152 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8011d20 con 0x7f8108153290 2023-03-03T12:27:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 -- 172.21.15.102:0/3314224152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 msgr2=0x7f8108158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 --2- 172.21.15.102:0/3314224152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f8108158e80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009840 tx=0x7f80f8009b50 comp rx=0 tx=0).stop 2023-03-03T12:27:11.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 -- 172.21.15.102:0/3314224152 shutdown_connections 2023-03-03T12:27:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 --2- 172.21.15.102:0/3314224152 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f8108158e80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 --2- 172.21.15.102:0/3314224152 >> v2:172.21.15.102:3300/0 conn(0x7f81081593c0 0x7f81080cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 -- 172.21.15.102:0/3314224152 >> 172.21.15.102:0/3314224152 conn(0x7f8108065fb0 msgr2=0x7f81080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 -- 172.21.15.102:0/3314224152 shutdown_connections 2023-03-03T12:27:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.674+0000 7f810e1ff700 1 -- 172.21.15.102:0/3314224152 wait complete. 2023-03-03T12:27:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.676+0000 7f810e1ff700 1 Processor -- start 2023-03-03T12:27:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.677+0000 7f810e1ff700 1 -- start start 2023-03-03T12:27:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f810e1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81080cd8e0 0x7f81080caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f810e1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f81080b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f810e1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81081593c0 0x7f81080b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f810e1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81080c7080 con 0x7f81081593c0 2023-03-03T12:27:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f810e1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81080c8260 con 0x7f81080cd8e0 2023-03-03T12:27:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f810e1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81080c6140 con 0x7f8108153290 2023-03-03T12:27:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f8107fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81081593c0 0x7f81080b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f81077fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81080cd8e0 0x7f81080caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f8106ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f81080b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f8107fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81081593c0 0x7f81080b22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53408/0 (socket says 172.21.15.102:53408) 2023-03-03T12:27:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.678+0000 7f8107fff700 1 -- 172.21.15.102:0/1952671506 learned_addr learned my addr 172.21.15.102:0/1952671506 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:27:11.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f81077fe700 1 -- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 msgr2=0x7f81080b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f81077fe700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f81080b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f81077fe700 1 -- 172.21.15.102:0/1952671506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81081593c0 msgr2=0x7f81080b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f81077fe700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81081593c0 0x7f81080b22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f81077fe700 1 -- 172.21.15.102:0/1952671506 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc0094a0 con 0x7f81080cd8e0 2023-03-03T12:27:11.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f81077fe700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81080cd8e0 0x7f81080caf70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f80f8016000 tx=0x7f80f8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f80effff700 1 -- 172.21.15.102:0/1952671506 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80036a0 con 0x7f81080cd8e0 2023-03-03T12:27:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f80effff700 1 -- 172.21.15.102:0/1952671506 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f801e070 con 0x7f81080cd8e0 2023-03-03T12:27:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f80effff700 1 -- 172.21.15.102:0/1952671506 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8003a20 con 0x7f81080cd8e0 2023-03-03T12:27:11.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f80094a0 con 0x7f81080cd8e0 2023-03-03T12:27:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.679+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81080ccde0 con 0x7f81080cd8e0 2023-03-03T12:27:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.680+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810815d160 con 0x7f81080cd8e0 2023-03-03T12:27:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.682+0000 7f80effff700 1 -- 172.21.15.102:0/1952671506 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80f802a020 con 0x7f81080cd8e0 2023-03-03T12:27:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.683+0000 7f80effff700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80f4068320 0x7f80f406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.683+0000 7f80effff700 1 -- 172.21.15.102:0/1952671506 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f80f8086b90 con 0x7f81080cd8e0 2023-03-03T12:27:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.683+0000 7f8106ffd700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80f4068320 0x7f80f406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.684+0000 7f8106ffd700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80f4068320 0x7f80f406a7d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f810808c980 tx=0x7f80fc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:11.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.687+0000 7f80effff700 1 -- 172.21.15.102:0/1952671506 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80f80532a0 con 0x7f81080cd8e0 2023-03-03T12:27:11.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.917+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81080cd0c0 con 0x7f81080cd8e0 2023-03-03T12:27:11.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.918+0000 7f80effff700 1 -- 172.21.15.102:0/1952671506 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f80f80568c0 con 0x7f81080cd8e0 2023-03-03T12:27:11.920 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:27:11.921 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:27:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80f4068320 msgr2=0x7f80f406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80f4068320 0x7f80f406a7d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f810808c980 tx=0x7f80fc009210 comp rx=0 tx=0).stop 2023-03-03T12:27:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81080cd8e0 msgr2=0x7f81080caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:11.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81080cd8e0 0x7f81080caf70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f80f8016000 tx=0x7f80f8004750 comp rx=0 tx=0).stop 2023-03-03T12:27:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 shutdown_connections 2023-03-03T12:27:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81081593c0 0x7f81080b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81080cd8e0 0x7f81080caf70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80f4068320 0x7f80f406a7d0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 --2- 172.21.15.102:0/1952671506 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8108153290 0x7f81080b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 >> 172.21.15.102:0/1952671506 conn(0x7f8108065fb0 msgr2=0x7f810815ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:11.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 shutdown_connections 2023-03-03T12:27:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:11.921+0000 7f810e1ff700 1 -- 172.21.15.102:0/1952671506 wait complete. 2023-03-03T12:27:11.925 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:27:11.950 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-03-03T12:27:27.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:27.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:53.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:27:54.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.217+0000 7fe03c1ff700 1 Processor -- start 2023-03-03T12:27:54.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.218+0000 7fe03c1ff700 1 -- start start 2023-03-03T12:27:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.219+0000 7fe03c1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe03414fbe0 0x7fe0340c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.219+0000 7fe03c1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe0340716c0 0x7fe0340706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.219+0000 7fe03c1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340c3c80 con 0x7fe03414ffb0 2023-03-03T12:27:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.219+0000 7fe03c1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340c41a0 con 0x7fe03414fbe0 2023-03-03T12:27:54.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.219+0000 7fe03c1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe034071cd0 con 0x7fe0340716c0 2023-03-03T12:27:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03a79c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe0340716c0 0x7fe0340706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03979a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe03414ffb0 0x7fe0340c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59768/0 (socket says 172.21.15.102:59768) 2023-03-03T12:27:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03979a700 1 -- 172.21.15.102:0/1473047457 learned_addr learned my addr 172.21.15.102:0/1473047457 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:27:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe039f9b700 1 --2- 172.21.15.102:0/1473047457 >> v2:172.21.15.102:3300/0 conn(0x7fe03414fbe0 0x7fe0340c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03a79c700 1 -- 172.21.15.102:0/1473047457 >> v2:172.21.15.102:3300/0 conn(0x7fe03414fbe0 msgr2=0x7fe0340c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:27:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03a79c700 1 --2- 172.21.15.102:0/1473047457 >> v2:172.21.15.102:3300/0 conn(0x7fe03414fbe0 0x7fe0340c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03a79c700 1 -- 172.21.15.102:0/1473047457 >> v1:172.21.15.102:6789/0 conn(0x7fe03414ffb0 legacy=0x7fe0340c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03a79c700 1 -- 172.21.15.102:0/1473047457 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0340c3ec0 con 0x7fe0340716c0 2023-03-03T12:27:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe039f9b700 1 --2- 172.21.15.102:0/1473047457 >> v2:172.21.15.102:3300/0 conn(0x7fe03414fbe0 0x7fe0340c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:27:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe038f99700 1 -- 172.21.15.102:0/1473047457 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325198852 0 0) 0x7fe024001110 con 0x7fe03414ffb0 2023-03-03T12:27:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.220+0000 7fe03a79c700 1 --2- 172.21.15.102:0/1473047457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340716c0 0x7fe0340706f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe03406be90 tx=0x7fe02c00d330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b38cac1b62be97d0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe038f99700 1 -- 172.21.15.102:0/1473047457 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c010070 con 0x7fe0340716c0 2023-03-03T12:27:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe038f99700 1 -- 172.21.15.102:0/1473047457 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c00b2e0 con 0x7fe0340716c0 2023-03-03T12:27:54.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe038f99700 1 -- 172.21.15.102:0/1473047457 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c00dc10 con 0x7fe0340716c0 2023-03-03T12:27:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe03c1ff700 1 -- 172.21.15.102:0/1473047457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340716c0 msgr2=0x7fe0340706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/1473047457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340716c0 0x7fe0340706f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe03406be90 tx=0x7fe02c00d330 comp rx=0 tx=0).stop 2023-03-03T12:27:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe03c1ff700 1 -- 172.21.15.102:0/1473047457 shutdown_connections 2023-03-03T12:27:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/1473047457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340716c0 0x7fe0340706f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/1473047457 >> v2:172.21.15.102:3300/0 conn(0x7fe03414fbe0 0x7fe0340c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.221+0000 7fe03c1ff700 1 -- 172.21.15.102:0/1473047457 >> 172.21.15.102:0/1473047457 conn(0x7fe03406af90 msgr2=0x7fe03406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.222+0000 7fe03c1ff700 1 -- 172.21.15.102:0/1473047457 shutdown_connections 2023-03-03T12:27:54.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.222+0000 7fe03c1ff700 1 -- 172.21.15.102:0/1473047457 wait complete. 2023-03-03T12:27:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.224+0000 7fe03c1ff700 1 Processor -- start 2023-03-03T12:27:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.224+0000 7fe03c1ff700 1 -- start start 2023-03-03T12:27:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03c1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0340716c0 0x7fe0341de640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:54.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03c1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe03414fbe0 0x7fe0340c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03c1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe03414ffb0 0x7fe0340c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03c1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe034151e30 con 0x7fe03414fbe0 2023-03-03T12:27:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03c1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0341f6b50 con 0x7fe03414ffb0 2023-03-03T12:27:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03c1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe034151590 con 0x7fe0340716c0 2023-03-03T12:27:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe039f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0340716c0 0x7fe0341de640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe03414fbe0 0x7fe0340c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:54.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03a79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe03414ffb0 0x7fe0340c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe03414fbe0 0x7fe0340c8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36830/0 (socket says 172.21.15.102:36830) 2023-03-03T12:27:54.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.225+0000 7fe03979a700 1 -- 172.21.15.102:0/821351348 learned_addr learned my addr 172.21.15.102:0/821351348 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:27:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03a79c700 1 -- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0340716c0 msgr2=0x7fe0341de640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03a79c700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0340716c0 0x7fe0341de640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03a79c700 1 -- 172.21.15.102:0/821351348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe03414fbe0 msgr2=0x7fe0340c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03a79c700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe03414fbe0 0x7fe0340c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03a79c700 1 -- 172.21.15.102:0/821351348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0300094a0 con 0x7fe03414ffb0 2023-03-03T12:27:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03979a700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe03414fbe0 0x7fe0340c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:27:54.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03a79c700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe03414ffb0 0x7fe0340c93f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe02c017000 tx=0x7fe02c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe02a7fc700 1 -- 172.21.15.102:0/821351348 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c004450 con 0x7fe03414ffb0 2023-03-03T12:27:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe02a7fc700 1 -- 172.21.15.102:0/821351348 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c0045b0 con 0x7fe03414ffb0 2023-03-03T12:27:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c00bc80 con 0x7fe03414ffb0 2023-03-03T12:27:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.226+0000 7fe02a7fc700 1 -- 172.21.15.102:0/821351348 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c004940 con 0x7fe03414ffb0 2023-03-03T12:27:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.227+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe034072250 con 0x7fe03414ffb0 2023-03-03T12:27:54.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.228+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe034071f70 con 0x7fe03414ffb0 2023-03-03T12:27:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.229+0000 7fe02a7fc700 1 -- 172.21.15.102:0/821351348 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe02c023030 con 0x7fe03414ffb0 2023-03-03T12:27:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.230+0000 7fe02a7fc700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe018068320 0x7fe01806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:54.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.231+0000 7fe02a7fc700 1 -- 172.21.15.102:0/821351348 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe02c0874a0 con 0x7fe03414ffb0 2023-03-03T12:27:54.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.231+0000 7fe039f9b700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe018068320 0x7fe01806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:54.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.231+0000 7fe039f9b700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe018068320 0x7fe01806a7d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe030009d70 tx=0x7fe030009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:54.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.235+0000 7fe02a7fc700 1 -- 172.21.15.102:0/821351348 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe02c053bb0 con 0x7fe03414ffb0 2023-03-03T12:27:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.469+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe034072530 con 0x7fe03414ffb0 2023-03-03T12:27:54.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.470+0000 7fe02a7fc700 1 -- 172.21.15.102:0/821351348 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe02c0571d0 con 0x7fe03414ffb0 2023-03-03T12:27:54.473 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:27:54.473 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:27:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe018068320 msgr2=0x7fe01806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe018068320 0x7fe01806a7d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe030009d70 tx=0x7fe030009210 comp rx=0 tx=0).stop 2023-03-03T12:27:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe03414ffb0 msgr2=0x7fe0340c93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe03414ffb0 0x7fe0340c93f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe02c017000 tx=0x7fe02c004ad0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 shutdown_connections 2023-03-03T12:27:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe018068320 0x7fe01806a7d0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe0340716c0 0x7fe0341de640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe03414ffb0 0x7fe0340c93f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 --2- 172.21.15.102:0/821351348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe03414fbe0 0x7fe0340c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 >> 172.21.15.102:0/821351348 conn(0x7fe03406af90 msgr2=0x7fe0340c7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 shutdown_connections 2023-03-03T12:27:54.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:27:54.473+0000 7fe03c1ff700 1 -- 172.21.15.102:0/821351348 wait complete. 2023-03-03T12:27:54.478 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:27:54.503 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-03-03T12:27:57.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:57.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:27.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:27.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:57.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:57.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:15.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:29:15.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.750+0000 7f3c7c9ff700 1 Processor -- start 2023-03-03T12:29:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.751+0000 7f3c7c9ff700 1 -- start start 2023-03-03T12:29:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.752+0000 7f3c7c9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c78153290 0x7f3c78158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:29:15.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.752+0000 7f3c7c9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c781593c0 0x7f3c780cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:29:15.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.752+0000 7f3c7c9ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c78153660 con 0x7f3c780cd8e0 2023-03-03T12:29:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.752+0000 7f3c7c9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7806c360 con 0x7f3c781593c0 2023-03-03T12:29:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.752+0000 7f3c7c9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7815cea0 con 0x7f3c78153290 2023-03-03T12:29:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c78153290 0x7f3c78158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:29:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c78153290 0x7f3c78158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60654/0 (socket says 172.21.15.102:60654) 2023-03-03T12:29:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 -- 172.21.15.102:0/1938288582 learned_addr learned my addr 172.21.15.102:0/1938288582 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:29:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c75d9b700 1 --2- 172.21.15.102:0/1938288582 >> v2:172.21.15.102:3300/0 conn(0x7f3c781593c0 0x7f3c780cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:29:15.757 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 -- 172.21.15.102:0/1938288582 >> v2:172.21.15.102:3300/0 conn(0x7f3c781593c0 msgr2=0x7f3c780cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:29:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 --2- 172.21.15.102:0/1938288582 >> v2:172.21.15.102:3300/0 conn(0x7f3c781593c0 0x7f3c780cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7559a700 1 -- 172.21.15.102:0/1938288582 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124834080 0 0) 0x7f3c78153660 con 0x7f3c780cd8e0 2023-03-03T12:29:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 -- 172.21.15.102:0/1938288582 >> v1:172.21.15.102:6789/0 conn(0x7f3c780cd8e0 legacy=0x7f3c780ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:29:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 -- 172.21.15.102:0/1938288582 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7806c7f0 con 0x7f3c78153290 2023-03-03T12:29:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.753+0000 7f3c7659c700 1 --2- 172.21.15.102:0/1938288582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c78158e80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3c7808cc70 tx=0x7f3c68009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93275b9fad480da server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:29:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7559a700 1 -- 172.21.15.102:0/1938288582 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68011830 con 0x7f3c78153290 2023-03-03T12:29:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7559a700 1 -- 172.21.15.102:0/1938288582 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c68011990 con 0x7f3c78153290 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7559a700 1 -- 172.21.15.102:0/1938288582 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68011d20 con 0x7f3c78153290 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/1938288582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 msgr2=0x7f3c78158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/1938288582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c78158e80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3c7808cc70 tx=0x7f3c68009bd0 comp rx=0 tx=0).stop 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/1938288582 shutdown_connections 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/1938288582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c78158e80 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/1938288582 >> v2:172.21.15.102:3300/0 conn(0x7f3c781593c0 0x7f3c780cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/1938288582 >> 172.21.15.102:0/1938288582 conn(0x7f3c78065fb0 msgr2=0x7f3c780663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:29:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.754+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/1938288582 shutdown_connections 2023-03-03T12:29:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.755+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/1938288582 wait complete. 2023-03-03T12:29:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.756+0000 7f3c7c9ff700 1 Processor -- start 2023-03-03T12:29:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.757+0000 7f3c7c9ff700 1 -- start start 2023-03-03T12:29:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7c9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c780cd8e0 0x7f3c781ef690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:29:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7c9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c780b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:29:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7c9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c781593c0 0x7f3c780cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:29:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7c9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780c65a0 con 0x7f3c781593c0 2023-03-03T12:29:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7c9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7815b3b0 con 0x7f3c78153290 2023-03-03T12:29:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7c9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780b2360 con 0x7f3c780cd8e0 2023-03-03T12:29:15.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c76d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c781593c0 0x7f3c780cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:29:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c75d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c780b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:29:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c780cd8e0 0x7f3c781ef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:29:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c780cd8e0 0x7f3c781ef690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:44712/0 (socket says 172.21.15.102:44712) 2023-03-03T12:29:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c75d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c780b27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60662/0 (socket says 172.21.15.102:60662) 2023-03-03T12:29:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c7659c700 1 -- 172.21.15.102:0/4242208242 learned_addr learned my addr 172.21.15.102:0/4242208242 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:29:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c76d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c781593c0 0x7f3c780cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48522/0 (socket says 172.21.15.102:48522) 2023-03-03T12:29:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c76d9d700 1 -- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c780cd8e0 msgr2=0x7f3c781ef690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:29:15.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c76d9d700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c780cd8e0 0x7f3c781ef690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:15.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c76d9d700 1 -- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 msgr2=0x7f3c780b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:29:15.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c76d9d700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c780b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:15.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.758+0000 7f3c76d9d700 1 -- 172.21.15.102:0/4242208242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c68009550 con 0x7f3c781593c0 2023-03-03T12:29:15.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.759+0000 7f3c76d9d700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c781593c0 0x7f3c780cf0b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000c7b0 tx=0x7f3c7000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:29:15.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.759+0000 7f3c66ffd700 1 -- 172.21.15.102:0/4242208242 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70003e30 con 0x7f3c781593c0 2023-03-03T12:29:15.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.759+0000 7f3c66ffd700 1 -- 172.21.15.102:0/4242208242 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c700050d0 con 0x7f3c781593c0 2023-03-03T12:29:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.759+0000 7f3c66ffd700 1 -- 172.21.15.102:0/4242208242 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70005460 con 0x7f3c781593c0 2023-03-03T12:29:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.759+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c780ccde0 con 0x7f3c781593c0 2023-03-03T12:29:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.759+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7815d160 con 0x7f3c781593c0 2023-03-03T12:29:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.760+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c780cd0c0 con 0x7f3c781593c0 2023-03-03T12:29:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.761+0000 7f3c66ffd700 1 -- 172.21.15.102:0/4242208242 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c70019030 con 0x7f3c781593c0 2023-03-03T12:29:15.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.763+0000 7f3c66ffd700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c58068320 0x7f3c5806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:29:15.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.763+0000 7f3c66ffd700 1 -- 172.21.15.102:0/4242208242 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3c70085200 con 0x7f3c781593c0 2023-03-03T12:29:15.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.764+0000 7f3c7659c700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c58068320 0x7f3c5806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:29:15.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.764+0000 7f3c7659c700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c58068320 0x7f3c5806a7d0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009130 tx=0x7f3c68008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:29:15.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:15.768+0000 7f3c66ffd700 1 -- 172.21.15.102:0/4242208242 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c70051910 con 0x7f3c781593c0 2023-03-03T12:29:16.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.002+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c78150570 con 0x7f3c781593c0 2023-03-03T12:29:16.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.003+0000 7f3c66ffd700 1 -- 172.21.15.102:0/4242208242 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3c70054f30 con 0x7f3c781593c0 2023-03-03T12:29:16.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:29:16.006 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:29:16.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.005+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c58068320 msgr2=0x7f3c5806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:29:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.005+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c58068320 0x7f3c5806a7d0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009130 tx=0x7f3c68008030 comp rx=0 tx=0).stop 2023-03-03T12:29:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.005+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c781593c0 msgr2=0x7f3c780cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:29:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.005+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c781593c0 0x7f3c780cf0b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000c7b0 tx=0x7f3c7000cac0 comp rx=0 tx=0).stop 2023-03-03T12:29:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 shutdown_connections 2023-03-03T12:29:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3c58068320 0x7f3c5806a7d0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c780cd8e0 0x7f3c781ef690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c781593c0 0x7f3c780cf0b0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:16.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 --2- 172.21.15.102:0/4242208242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c78153290 0x7f3c780b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:29:16.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 >> 172.21.15.102:0/4242208242 conn(0x7f3c78065fb0 msgr2=0x7f3c78150e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:29:16.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 shutdown_connections 2023-03-03T12:29:16.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:29:16.006+0000 7f3c7c9ff700 1 -- 172.21.15.102:0/4242208242 wait complete. 2023-03-03T12:29:16.010 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:29:16.036 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T12:29:28.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:28.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:58.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:58.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T12:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:30:28.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:28.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:43.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:30:43.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.293+0000 7fe7d679d700 1 Processor -- start 2023-03-03T12:30:43.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.293+0000 7fe7d679d700 1 -- start start 2023-03-03T12:30:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d679d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe7d00ced00 0x7fe7d00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d679d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00b3970 0x7fe7d00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d679d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7d00cf0d0 0x7fe7d0158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d679d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d00c4930 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d679d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d00c3d20 con 0x7fe7d00ced00 2023-03-03T12:30:43.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d679d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d015cba0 con 0x7fe7d00b3970 2023-03-03T12:30:43.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d5f9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7d00cf0d0 0x7fe7d0158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:43.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d4f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00b3970 0x7fe7d00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:43.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d5f9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7d00cf0d0 0x7fe7d0158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58124/0 (socket says 172.21.15.102:58124) 2023-03-03T12:30:43.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d5f9c700 1 -- 172.21.15.102:0/1832903419 learned_addr learned my addr 172.21.15.102:0/1832903419 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:30:43.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.295+0000 7fe7d579b700 1 --2- 172.21.15.102:0/1832903419 >> v2:172.21.15.172:3300/0 conn(0x7fe7d00ced00 0x7fe7d00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7d579b700 1 -- 172.21.15.102:0/1832903419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00b3970 msgr2=0x7fe7d00c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7d579b700 1 --2- 172.21.15.102:0/1832903419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00b3970 0x7fe7d00c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7d579b700 1 -- 172.21.15.102:0/1832903419 >> v2:172.21.15.102:3300/0 conn(0x7fe7d00cf0d0 msgr2=0x7fe7d0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7d579b700 1 --2- 172.21.15.102:0/1832903419 >> v2:172.21.15.102:3300/0 conn(0x7fe7d00cf0d0 0x7fe7d0158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7d579b700 1 -- 172.21.15.102:0/1832903419 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d015ce80 con 0x7fe7d00ced00 2023-03-03T12:30:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7d579b700 1 --2- 172.21.15.102:0/1832903419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00ced00 0x7fe7d00c5a90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00b1b40 tx=0x7fe7cc009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b3b0484d297c34 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7c7fff700 1 -- 172.21.15.102:0/1832903419 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011830 con 0x7fe7d00ced00 2023-03-03T12:30:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7c7fff700 1 -- 172.21.15.102:0/1832903419 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc011990 con 0x7fe7d00ced00 2023-03-03T12:30:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.296+0000 7fe7c7fff700 1 -- 172.21.15.102:0/1832903419 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011d20 con 0x7fe7d00ced00 2023-03-03T12:30:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 -- 172.21.15.102:0/1832903419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00ced00 msgr2=0x7fe7d00c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 --2- 172.21.15.102:0/1832903419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00ced00 0x7fe7d00c5a90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00b1b40 tx=0x7fe7cc009b20 comp rx=0 tx=0).stop 2023-03-03T12:30:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 -- 172.21.15.102:0/1832903419 shutdown_connections 2023-03-03T12:30:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 --2- 172.21.15.102:0/1832903419 >> v2:172.21.15.102:3300/0 conn(0x7fe7d00cf0d0 0x7fe7d0158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 --2- 172.21.15.102:0/1832903419 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00ced00 0x7fe7d00c5a90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 --2- 172.21.15.102:0/1832903419 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00b3970 0x7fe7d00c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 -- 172.21.15.102:0/1832903419 >> 172.21.15.102:0/1832903419 conn(0x7fe7d008b1c0 msgr2=0x7fe7d0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 -- 172.21.15.102:0/1832903419 shutdown_connections 2023-03-03T12:30:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.297+0000 7fe7d679d700 1 -- 172.21.15.102:0/1832903419 wait complete. 2023-03-03T12:30:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.300+0000 7fe7d679d700 1 Processor -- start 2023-03-03T12:30:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.300+0000 7fe7d679d700 1 -- start start 2023-03-03T12:30:43.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d679d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00b3970 0x7fe7d01ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:43.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d679d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00ced00 0x7fe7d0152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:43.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d679d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7d00cf0d0 0x7fe7d00c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:43.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d679d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d014e740 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d679d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d014dee0 con 0x7fe7d00b3970 2023-03-03T12:30:43.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d679d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d014e110 con 0x7fe7d00ced00 2023-03-03T12:30:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7d00cf0d0 0x7fe7d00c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7d00cf0d0 0x7fe7d00c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58126/0 (socket says 172.21.15.102:58126) 2023-03-03T12:30:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 -- 172.21.15.102:0/2050121784 learned_addr learned my addr 172.21.15.102:0/2050121784 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:30:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d579b700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00b3970 0x7fe7d01ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d4f9a700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00ced00 0x7fe7d0152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 -- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00ced00 msgr2=0x7fe7d0152320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00ced00 0x7fe7d0152320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 -- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00b3970 msgr2=0x7fe7d01ee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00b3970 0x7fe7d01ee990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.302+0000 7fe7d5f9c700 1 -- 172.21.15.102:0/2050121784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc0094a0 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.303+0000 7fe7d5f9c700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7d00cf0d0 0x7fe7d00c8570 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800efc0 tx=0x7fe7c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.303+0000 7fe7c5ffb700 1 -- 172.21.15.102:0/2050121784 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8003c70 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.303+0000 7fe7c5ffb700 1 -- 172.21.15.102:0/2050121784 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c800f040 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.303+0000 7fe7c5ffb700 1 -- 172.21.15.102:0/2050121784 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8009e60 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.303+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d006c200 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.303+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d006bf20 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.304+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d0159040 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.305+0000 7fe7c5ffb700 1 -- 172.21.15.102:0/2050121784 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe7c800c880 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.307+0000 7fe7c5ffb700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7b40682f0 0x7fe7b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:43.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.308+0000 7fe7c5ffb700 1 -- 172.21.15.102:0/2050121784 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe7c8086200 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.308+0000 7fe7d579b700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7b40682f0 0x7fe7b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:43.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.309+0000 7fe7d579b700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7b40682f0 0x7fe7b406a7a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0090b0 tx=0x7fe7cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:43.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.312+0000 7fe7c5ffb700 1 -- 172.21.15.102:0/2050121784 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7c8052910 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.545+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7d004e6e0 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.546+0000 7fe7c5ffb700 1 -- 172.21.15.102:0/2050121784 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe7c8012340 con 0x7fe7d00cf0d0 2023-03-03T12:30:43.548 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:30:43.548 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:30:43.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.548+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7b40682f0 msgr2=0x7fe7b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.548+0000 7fe7d679d700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7b40682f0 0x7fe7b406a7a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0090b0 tx=0x7fe7cc008030 comp rx=0 tx=0).stop 2023-03-03T12:30:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.548+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7d00cf0d0 msgr2=0x7fe7d00c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.548+0000 7fe7d679d700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7d00cf0d0 0x7fe7d00c8570 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800efc0 tx=0x7fe7c800c3a0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 shutdown_connections 2023-03-03T12:30:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7d00cf0d0 0x7fe7d00c8570 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7d00b3970 0x7fe7d01ee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7b40682f0 0x7fe7b406a7a0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 --2- 172.21.15.102:0/2050121784 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7d00ced00 0x7fe7d0152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 >> 172.21.15.102:0/2050121784 conn(0x7fe7d008b1c0 msgr2=0x7fe7d0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 shutdown_connections 2023-03-03T12:30:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:30:43.549+0000 7fe7d679d700 1 -- 172.21.15.102:0/2050121784 wait complete. 2023-03-03T12:30:43.552 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:30:43.577 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T12:30:58.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:58.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:26.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:31:26.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.843+0000 7fb7267ff700 1 Processor -- start 2023-03-03T12:31:26.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.843+0000 7fb7267ff700 1 -- start start 2023-03-03T12:31:26.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb7267ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb720158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:26.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb7267ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7200cd8e0 0x7fb7200ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:26.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb7267ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb720153660 con 0x7fb7201593c0 2023-03-03T12:31:26.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb7267ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72006c360 con 0x7fb7200cd8e0 2023-03-03T12:31:26.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb7267ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72015cea0 con 0x7fb720153290 2023-03-03T12:31:26.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb71ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb720158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:26.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb71f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb7201593c0 0x7fb7200cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60066/0 (socket says 172.21.15.102:60066) 2023-03-03T12:31:26.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb724d9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7200cd8e0 0x7fb7200ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:26.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.845+0000 7fb71f7fe700 1 -- 172.21.15.102:0/3517633505 learned_addr learned my addr 172.21.15.102:0/3517633505 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:31:26.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71effd700 1 -- 172.21.15.102:0/3517633505 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262066628 0 0) 0x7fb720153660 con 0x7fb7201593c0 2023-03-03T12:31:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71ffff700 1 -- 172.21.15.102:0/3517633505 >> v2:172.21.15.102:3300/0 conn(0x7fb7200cd8e0 msgr2=0x7fb7200ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:31:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71ffff700 1 --2- 172.21.15.102:0/3517633505 >> v2:172.21.15.102:3300/0 conn(0x7fb7200cd8e0 0x7fb7200ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71ffff700 1 -- 172.21.15.102:0/3517633505 >> v1:172.21.15.172:6789/0 conn(0x7fb7201593c0 legacy=0x7fb7200cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71ffff700 1 -- 172.21.15.102:0/3517633505 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72006c7f0 con 0x7fb720153290 2023-03-03T12:31:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb724d9c700 1 --2- 172.21.15.102:0/3517633505 >> v2:172.21.15.102:3300/0 conn(0x7fb7200cd8e0 0x7fb7200ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:31:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71ffff700 1 --2- 172.21.15.102:0/3517633505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb720158e80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb710009840 tx=0x7fb710009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=238be0165a1ab01e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:31:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71effd700 1 -- 172.21.15.102:0/3517633505 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb710011830 con 0x7fb720153290 2023-03-03T12:31:26.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71effd700 1 -- 172.21.15.102:0/3517633505 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb710011990 con 0x7fb720153290 2023-03-03T12:31:26.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.846+0000 7fb71effd700 1 -- 172.21.15.102:0/3517633505 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb710011d20 con 0x7fb720153290 2023-03-03T12:31:26.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 -- 172.21.15.102:0/3517633505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 msgr2=0x7fb720158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:26.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3517633505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb720158e80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb710009840 tx=0x7fb710009b50 comp rx=0 tx=0).stop 2023-03-03T12:31:26.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 -- 172.21.15.102:0/3517633505 shutdown_connections 2023-03-03T12:31:26.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3517633505 >> v2:172.21.15.102:3300/0 conn(0x7fb7200cd8e0 0x7fb7200ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:26.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3517633505 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb720158e80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:26.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 -- 172.21.15.102:0/3517633505 >> 172.21.15.102:0/3517633505 conn(0x7fb720065fb0 msgr2=0x7fb7200663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:31:26.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 -- 172.21.15.102:0/3517633505 shutdown_connections 2023-03-03T12:31:26.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.847+0000 7fb7267ff700 1 -- 172.21.15.102:0/3517633505 wait complete. 2023-03-03T12:31:26.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.850+0000 7fb7267ff700 1 Processor -- start 2023-03-03T12:31:26.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.850+0000 7fb7267ff700 1 -- start start 2023-03-03T12:31:26.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb7267ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7200cd8e0 0x7fb7200c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:26.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb7267ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb7200c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:26.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb7267ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7201593c0 0x7fb7201f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:26.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb7267ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200c7ba0 con 0x7fb7201593c0 2023-03-03T12:31:26.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb7267ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200c7e40 con 0x7fb7200cd8e0 2023-03-03T12:31:26.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb7267ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7201ef900 con 0x7fb720153290 2023-03-03T12:31:26.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb724d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7201593c0 0x7fb7201f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:26.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb724d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7201593c0 0x7fb7201f06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54972/0 (socket says 172.21.15.102:54972) 2023-03-03T12:31:26.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb71ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7200cd8e0 0x7fb7200c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:26.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb724d9c700 1 -- 172.21.15.102:0/3060589513 learned_addr learned my addr 172.21.15.102:0/3060589513 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:31:26.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb71f7fe700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb7200c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:26.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb724d9c700 1 -- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 msgr2=0x7fb7200c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:26.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.852+0000 7fb724d9c700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb7200c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:26.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb724d9c700 1 -- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7200cd8e0 msgr2=0x7fb7200c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:26.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb724d9c700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7200cd8e0 0x7fb7200c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:26.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb724d9c700 1 -- 172.21.15.102:0/3060589513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb718009a70 con 0x7fb7201593c0 2023-03-03T12:31:26.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb71ffff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7200cd8e0 0x7fb7200c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:31:26.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb724d9c700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7201593c0 0x7fb7201f06c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb71800efc0 tx=0x7fb71800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:31:26.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb71cff9700 1 -- 172.21.15.102:0/3060589513 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71800ca90 con 0x7fb7201593c0 2023-03-03T12:31:26.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb71cff9700 1 -- 172.21.15.102:0/3060589513 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71800f040 con 0x7fb7201593c0 2023-03-03T12:31:26.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb71cff9700 1 -- 172.21.15.102:0/3060589513 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71800ce30 con 0x7fb7201593c0 2023-03-03T12:31:26.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7100094a0 con 0x7fb7201593c0 2023-03-03T12:31:26.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.853+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7200ccde0 con 0x7fb7201593c0 2023-03-03T12:31:26.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.855+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72015d160 con 0x7fb7201593c0 2023-03-03T12:31:26.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.856+0000 7fb71cff9700 1 -- 172.21.15.102:0/3060589513 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb718014480 con 0x7fb7201593c0 2023-03-03T12:31:26.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.857+0000 7fb71cff9700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c0682f0 0x7fb70c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:26.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.857+0000 7fb71cff9700 1 -- 172.21.15.102:0/3060589513 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb718086530 con 0x7fb7201593c0 2023-03-03T12:31:26.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.857+0000 7fb71ffff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c0682f0 0x7fb70c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:26.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.858+0000 7fb71ffff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c0682f0 0x7fb70c06a7a0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb7100090b0 tx=0x7fb710008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:31:26.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:26.861+0000 7fb71cff9700 1 -- 172.21.15.102:0/3060589513 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb718052be0 con 0x7fb7201593c0 2023-03-03T12:31:27.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.097+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7200cd0c0 con 0x7fb7201593c0 2023-03-03T12:31:27.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.098+0000 7fb71cff9700 1 -- 172.21.15.102:0/3060589513 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb718009cd0 con 0x7fb7201593c0 2023-03-03T12:31:27.099 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:31:27.100 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:31:27.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c0682f0 msgr2=0x7fb70c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:27.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c0682f0 0x7fb70c06a7a0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb7100090b0 tx=0x7fb710008030 comp rx=0 tx=0).stop 2023-03-03T12:31:27.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7201593c0 msgr2=0x7fb7201f06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:27.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7201593c0 0x7fb7201f06c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb71800efc0 tx=0x7fb71800c3a0 comp rx=0 tx=0).stop 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 shutdown_connections 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7201593c0 0x7fb7201f06c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7200cd8e0 0x7fb7200c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb70c0682f0 0x7fb70c06a7a0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 --2- 172.21.15.102:0/3060589513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb720153290 0x7fb7200c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 >> 172.21.15.102:0/3060589513 conn(0x7fb720065fb0 msgr2=0x7fb72015bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 shutdown_connections 2023-03-03T12:31:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:31:27.100+0000 7fb7267ff700 1 -- 172.21.15.102:0/3060589513 wait complete. 2023-03-03T12:31:27.104 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:31:27.128 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2023-03-03T12:31:28.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:28.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:58.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:58.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:29.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:29.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:59.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:59.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:24.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:33:24.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.376+0000 7fe652dff700 1 Processor -- start 2023-03-03T12:33:24.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.376+0000 7fe652dff700 1 -- start start 2023-03-03T12:33:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.378+0000 7fe652dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe64c150c80 0x7fe64c06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.378+0000 7fe652dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c5f30 0x7fe64c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.378+0000 7fe652dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64c0c5390 con 0x7fe64c0c2020 2023-03-03T12:33:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.378+0000 7fe652dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64c157ad0 con 0x7fe64c150c80 2023-03-03T12:33:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.378+0000 7fe652dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0c4a20 con 0x7fe64c0c5f30 2023-03-03T12:33:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.378+0000 7fe650b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe64c150c80 0x7fe64c06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:24.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.378+0000 7fe65139c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c5f30 0x7fe64c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:24.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe65139c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c5f30 0x7fe64c0c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:40120/0 (socket says 172.21.15.102:40120) 2023-03-03T12:33:24.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe65139c700 1 -- 172.21.15.102:0/221579885 learned_addr learned my addr 172.21.15.102:0/221579885 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:33:24.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe64bfff700 1 --1- 172.21.15.102:0/221579885 >> v1:172.21.15.102:6789/0 conn(0x7fe64c0c2020 0x7fe64c0c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38798/0 (socket says 172.21.15.102:38798) 2023-03-03T12:33:24.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe65139c700 1 -- 172.21.15.102:0/221579885 >> v2:172.21.15.172:3300/0 conn(0x7fe64c150c80 msgr2=0x7fe64c06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:33:24.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe65139c700 1 --2- 172.21.15.102:0/221579885 >> v2:172.21.15.172:3300/0 conn(0x7fe64c150c80 0x7fe64c06d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe65139c700 1 -- 172.21.15.102:0/221579885 >> v1:172.21.15.102:6789/0 conn(0x7fe64c0c2020 legacy=0x7fe64c0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe65139c700 1 -- 172.21.15.102:0/221579885 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c06f010 con 0x7fe64c0c5f30 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe650b9b700 1 --2- 172.21.15.102:0/221579885 >> v2:172.21.15.172:3300/0 conn(0x7fe64c150c80 0x7fe64c06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe65139c700 1 --2- 172.21.15.102:0/221579885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c5f30 0x7fe64c0c2720 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe644009e50 tx=0x7fe64400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e810af649e96e0ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.379+0000 7fe64b7fe700 1 -- 172.21.15.102:0/221579885 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400e070 con 0x7fe64c0c5f30 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe64b7fe700 1 -- 172.21.15.102:0/221579885 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64400b910 con 0x7fe64c0c5f30 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe64b7fe700 1 -- 172.21.15.102:0/221579885 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400bcc0 con 0x7fe64c0c5f30 2023-03-03T12:33:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 -- 172.21.15.102:0/221579885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c5f30 msgr2=0x7fe64c0c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 --2- 172.21.15.102:0/221579885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c5f30 0x7fe64c0c2720 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe644009e50 tx=0x7fe64400b330 comp rx=0 tx=0).stop 2023-03-03T12:33:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 -- 172.21.15.102:0/221579885 shutdown_connections 2023-03-03T12:33:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 --2- 172.21.15.102:0/221579885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c5f30 0x7fe64c0c2720 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 --2- 172.21.15.102:0/221579885 >> v2:172.21.15.172:3300/0 conn(0x7fe64c150c80 0x7fe64c06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 -- 172.21.15.102:0/221579885 >> 172.21.15.102:0/221579885 conn(0x7fe64c06af90 msgr2=0x7fe64c06c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:33:24.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 -- 172.21.15.102:0/221579885 shutdown_connections 2023-03-03T12:33:24.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.380+0000 7fe652dff700 1 -- 172.21.15.102:0/221579885 wait complete. 2023-03-03T12:33:24.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.382+0000 7fe652dff700 1 Processor -- start 2023-03-03T12:33:24.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.383+0000 7fe652dff700 1 -- start start 2023-03-03T12:33:24.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe652dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c2020 0x7fe64c1429e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:24.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe652dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe64c0c5f30 0x7fe64c14b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe652dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe64c150c80 0x7fe64c14bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe652dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c154a30 con 0x7fe64c150c80 2023-03-03T12:33:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe652dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0ca080 con 0x7fe64c0c5f30 2023-03-03T12:33:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe652dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0ca360 con 0x7fe64c0c2020 2023-03-03T12:33:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe650b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c2020 0x7fe64c1429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe65139c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe64c150c80 0x7fe64c14bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe64bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe64c0c5f30 0x7fe64c14b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe650b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c2020 0x7fe64c1429e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:40134/0 (socket says 172.21.15.102:40134) 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe650b9b700 1 -- 172.21.15.102:0/1758550527 learned_addr learned my addr 172.21.15.102:0/1758550527 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe650b9b700 1 -- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe64c0c5f30 msgr2=0x7fe64c14b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.384+0000 7fe650b9b700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe64c0c5f30 0x7fe64c14b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe650b9b700 1 -- 172.21.15.102:0/1758550527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe64c150c80 msgr2=0x7fe64c14bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe650b9b700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe64c150c80 0x7fe64c14bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe650b9b700 1 -- 172.21.15.102:0/1758550527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6400094a0 con 0x7fe64c0c2020 2023-03-03T12:33:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe650b9b700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c2020 0x7fe64c1429e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe64000ebe0 tx=0x7fe64000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:24.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe6497fa700 1 -- 172.21.15.102:0/1758550527 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640009760 con 0x7fe64c0c2020 2023-03-03T12:33:24.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe6497fa700 1 -- 172.21.15.102:0/1758550527 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64000c850 con 0x7fe64c0c2020 2023-03-03T12:33:24.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe6497fa700 1 -- 172.21.15.102:0/1758550527 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64000cbe0 con 0x7fe64c0c2020 2023-03-03T12:33:24.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe644009b30 con 0x7fe64c0c2020 2023-03-03T12:33:24.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.385+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64c1577f0 con 0x7fe64c0c2020 2023-03-03T12:33:24.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.386+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64c0c9e50 con 0x7fe64c0c2020 2023-03-03T12:33:24.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.388+0000 7fe6497fa700 1 -- 172.21.15.102:0/1758550527 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe640023070 con 0x7fe64c0c2020 2023-03-03T12:33:24.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.389+0000 7fe6497fa700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe630068320 0x7fe63006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:24.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.389+0000 7fe6497fa700 1 -- 172.21.15.102:0/1758550527 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe640085fb0 con 0x7fe64c0c2020 2023-03-03T12:33:24.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.389+0000 7fe64bfff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe630068320 0x7fe63006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:24.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.390+0000 7fe64bfff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe630068320 0x7fe63006a7d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fe64c1502b0 tx=0x7fe638007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.394+0000 7fe6497fa700 1 -- 172.21.15.102:0/1758550527 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6400526c0 con 0x7fe64c0c2020 2023-03-03T12:33:24.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.627+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe64c071bd0 con 0x7fe64c0c2020 2023-03-03T12:33:24.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.628+0000 7fe6497fa700 1 -- 172.21.15.102:0/1758550527 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe640055ce0 con 0x7fe64c0c2020 2023-03-03T12:33:24.631 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:33:24.632 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:33:24.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe630068320 msgr2=0x7fe63006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:24.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe630068320 0x7fe63006a7d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fe64c1502b0 tx=0x7fe638007ed0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c2020 msgr2=0x7fe64c1429e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c2020 0x7fe64c1429e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe64000ebe0 tx=0x7fe64000eef0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 shutdown_connections 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe630068320 0x7fe63006a7d0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe64c0c2020 0x7fe64c1429e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe64c150c80 0x7fe64c14bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 --2- 172.21.15.102:0/1758550527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe64c0c5f30 0x7fe64c14b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 >> 172.21.15.102:0/1758550527 conn(0x7fe64c06af90 msgr2=0x7fe64c14eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:33:24.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 shutdown_connections 2023-03-03T12:33:24.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:33:24.631+0000 7fe652dff700 1 -- 172.21.15.102:0/1758550527 wait complete. 2023-03-03T12:33:24.636 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:33:24.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-03-03T12:33:29.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:29.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:59.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:59.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:30.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:30.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:49.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:34:49.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.910+0000 7f6cb69ff700 1 Processor -- start 2023-03-03T12:34:49.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.910+0000 7f6cb69ff700 1 -- start start 2023-03-03T12:34:49.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.911+0000 7f6cb69ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6cb0153290 0x7f6cb0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:49.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.911+0000 7f6cb69ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0153660 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.911+0000 7f6cb69ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb006c360 con 0x7f6cb01593c0 2023-03-03T12:34:49.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.911+0000 7f6cb69ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb00cec40 con 0x7f6cb0153290 2023-03-03T12:34:49.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6cb4f9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6cb00cd8e0 0x7f6cb00ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40736/0 (socket says 172.21.15.102:40736) 2023-03-03T12:34:49.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6cb4f9c700 1 -- 172.21.15.102:0/10609066 learned_addr learned my addr 172.21.15.102:0/10609066 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:34:49.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6cb0153290 0x7f6cb0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:49.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caf7fe700 1 --1- 172.21.15.102:0/10609066 >> v1:172.21.15.172:6789/0 conn(0x7f6cb01593c0 0x7f6cb00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51450/0 (socket says 172.21.15.102:51450) 2023-03-03T12:34:49.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caeffd700 1 -- 172.21.15.102:0/10609066 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434422572 0 0) 0x7f6cb0153660 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caeffd700 1 -- 172.21.15.102:0/10609066 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c980035b0 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caeffd700 1 -- 172.21.15.102:0/10609066 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963297031 0 0) 0x7f6cb006c360 con 0x7f6cb01593c0 2023-03-03T12:34:49.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caffff700 1 -- 172.21.15.102:0/10609066 >> v1:172.21.15.172:6789/0 conn(0x7f6cb01593c0 legacy=0x7f6cb00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:49.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caffff700 1 -- 172.21.15.102:0/10609066 >> v1:172.21.15.102:6789/0 conn(0x7f6cb00cd8e0 legacy=0x7f6cb00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:49.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.912+0000 7f6caffff700 1 -- 172.21.15.102:0/10609066 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb015cea0 con 0x7f6cb0153290 2023-03-03T12:34:49.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.913+0000 7f6caffff700 1 --2- 172.21.15.102:0/10609066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 0x7f6cb0158e80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009840 tx=0x7f6ca0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1f9c9498438339e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:34:49.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.913+0000 7f6caeffd700 1 -- 172.21.15.102:0/10609066 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0011830 con 0x7f6cb0153290 2023-03-03T12:34:49.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.913+0000 7f6caeffd700 1 -- 172.21.15.102:0/10609066 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca0011990 con 0x7f6cb0153290 2023-03-03T12:34:49.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.913+0000 7f6caeffd700 1 -- 172.21.15.102:0/10609066 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0011d20 con 0x7f6cb0153290 2023-03-03T12:34:49.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.913+0000 7f6cb69ff700 1 -- 172.21.15.102:0/10609066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 msgr2=0x7f6cb0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:49.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.913+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/10609066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 0x7f6cb0158e80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009840 tx=0x7f6ca0009b50 comp rx=0 tx=0).stop 2023-03-03T12:34:49.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.914+0000 7f6cb69ff700 1 -- 172.21.15.102:0/10609066 shutdown_connections 2023-03-03T12:34:49.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.914+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/10609066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 0x7f6cb0158e80 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:49.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.914+0000 7f6cb69ff700 1 -- 172.21.15.102:0/10609066 >> 172.21.15.102:0/10609066 conn(0x7f6cb0065fb0 msgr2=0x7f6cb00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:34:49.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.914+0000 7f6cb69ff700 1 -- 172.21.15.102:0/10609066 shutdown_connections 2023-03-03T12:34:49.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.914+0000 7f6cb69ff700 1 -- 172.21.15.102:0/10609066 wait complete. 2023-03-03T12:34:49.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.916+0000 7f6cb69ff700 1 Processor -- start 2023-03-03T12:34:49.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.916+0000 7f6cb69ff700 1 -- start start 2023-03-03T12:34:49.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6cb69ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cb00cd8e0 0x7f6cb014e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:49.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6cb69ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 0x7f6cb00b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:49.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6cb69ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cb01593c0 0x7f6cb00b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:49.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6cb69ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb01f2810 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6cb69ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb014ea90 con 0x7f6cb0153290 2023-03-03T12:34:49.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6cb69ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb01f1c90 con 0x7f6cb01593c0 2023-03-03T12:34:49.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cb00cd8e0 0x7f6cb014e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:49.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cb00cd8e0 0x7f6cb014e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49940/0 (socket says 172.21.15.102:49940) 2023-03-03T12:34:49.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 -- 172.21.15.102:0/1262161938 learned_addr learned my addr 172.21.15.102:0/1262161938 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:34:49.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caf7fe700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 0x7f6cb00b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:49.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6cb4f9c700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cb01593c0 0x7f6cb00b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:49.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 -- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cb01593c0 msgr2=0x7f6cb00b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:49.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cb01593c0 0x7f6cb00b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:49.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 -- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 msgr2=0x7f6cb00b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:49.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 0x7f6cb00b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:49.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.917+0000 7f6caffff700 1 -- 172.21.15.102:0/1262161938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca00094a0 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.918+0000 7f6caffff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cb00cd8e0 0x7f6cb014e550 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0014040 tx=0x7f6ca0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:34:49.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.918+0000 7f6cacff9700 1 -- 172.21.15.102:0/1262161938 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca001a7e0 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.918+0000 7f6cacff9700 1 -- 172.21.15.102:0/1262161938 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca001a940 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.918+0000 7f6cacff9700 1 -- 172.21.15.102:0/1262161938 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca001acd0 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.918+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb015d160 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.918+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb006c7f0 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.919+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb00ccde0 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.920+0000 7f6cacff9700 1 -- 172.21.15.102:0/1262161938 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6ca000f050 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.922+0000 7f6cacff9700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c9c0682f0 0x7f6c9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:49.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.922+0000 7f6cacff9700 1 -- 172.21.15.102:0/1262161938 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6ca0086c00 con 0x7f6cb00cd8e0 2023-03-03T12:34:49.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.922+0000 7f6caf7fe700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c9c0682f0 0x7f6c9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:49.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.923+0000 7f6caf7fe700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c9c0682f0 0x7f6c9c06a7a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4001110 tx=0x7f6ca4001080 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:34:49.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:49.926+0000 7f6cacff9700 1 -- 172.21.15.102:0/1262161938 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ca0053310 con 0x7f6cb00cd8e0 2023-03-03T12:34:50.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.162+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cb004e6e0 con 0x7f6cb00cd8e0 2023-03-03T12:34:50.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.162+0000 7f6cacff9700 1 -- 172.21.15.102:0/1262161938 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6ca0029090 con 0x7f6cb00cd8e0 2023-03-03T12:34:50.165 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:34:50.166 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:34:50.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c9c0682f0 msgr2=0x7f6c9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:50.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c9c0682f0 0x7f6c9c06a7a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4001110 tx=0x7f6ca4001080 comp rx=0 tx=0).stop 2023-03-03T12:34:50.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cb00cd8e0 msgr2=0x7f6cb014e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:50.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cb00cd8e0 0x7f6cb014e550 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0014040 tx=0x7f6ca0004750 comp rx=0 tx=0).stop 2023-03-03T12:34:50.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 shutdown_connections 2023-03-03T12:34:50.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c9c0682f0 0x7f6c9c06a7a0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:50.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cb01593c0 0x7f6cb00b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:50.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cb00cd8e0 0x7f6cb014e550 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:50.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 --2- 172.21.15.102:0/1262161938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb0153290 0x7f6cb00b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:50.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 >> 172.21.15.102:0/1262161938 conn(0x7f6cb0065fb0 msgr2=0x7f6cb0151070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:34:50.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 shutdown_connections 2023-03-03T12:34:50.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:34:50.165+0000 7f6cb69ff700 1 -- 172.21.15.102:0/1262161938 wait complete. 2023-03-03T12:34:50.170 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:34:50.195 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T12:35:00.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:00.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:30.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:30.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:00.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:00.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:22.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:36:22.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.441+0000 7f6ce0fff700 1 Processor -- start 2023-03-03T12:36:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.442+0000 7f6ce0fff700 1 -- start start 2023-03-03T12:36:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.443+0000 7f6ce0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc153290 0x7f6cdc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:36:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.443+0000 7f6ce0fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6cdc1593c0 0x7f6cdc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:36:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.443+0000 7f6ce0fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc153660 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.443+0000 7f6ce0fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc06c360 con 0x7f6cdc1593c0 2023-03-03T12:36:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.443+0000 7f6ce0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc15cea0 con 0x7f6cdc153290 2023-03-03T12:36:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.443+0000 7f6cd9d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6cdc1593c0 0x7f6cdc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:36:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cda59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc153290 0x7f6cdc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:36:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd9d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6cdc1593c0 0x7f6cdc0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48126/0 (socket says 172.21.15.102:48126) 2023-03-03T12:36:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd9d9b700 1 -- 172.21.15.102:0/18198291 learned_addr learned my addr 172.21.15.102:0/18198291 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:36:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cdad9d700 1 --1- 172.21.15.102:0/18198291 >> v1:172.21.15.102:6789/0 conn(0x7f6cdc0cd8e0 0x7f6cdc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52646/0 (socket says 172.21.15.102:52646) 2023-03-03T12:36:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd9d9b700 1 -- 172.21.15.102:0/18198291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc153290 msgr2=0x7f6cdc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:36:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd9d9b700 1 --2- 172.21.15.102:0/18198291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc153290 0x7f6cdc158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd9d9b700 1 -- 172.21.15.102:0/18198291 >> v1:172.21.15.102:6789/0 conn(0x7f6cdc0cd8e0 legacy=0x7f6cdc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:36:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd9d9b700 1 -- 172.21.15.102:0/18198291 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc06c7f0 con 0x7f6cdc1593c0 2023-03-03T12:36:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd959a700 1 -- 172.21.15.102:0/18198291 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908374004 0 0) 0x7f6cdc153660 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.444+0000 7f6cd9d9b700 1 --2- 172.21.15.102:0/18198291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc1593c0 0x7f6cdc0cde20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00098f0 tx=0x7f6cd0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a744c80e8ff991 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:36:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6cd959a700 1 -- 172.21.15.102:0/18198291 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0011830 con 0x7f6cdc1593c0 2023-03-03T12:36:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6cd959a700 1 -- 172.21.15.102:0/18198291 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd0011990 con 0x7f6cdc1593c0 2023-03-03T12:36:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6cd959a700 1 -- 172.21.15.102:0/18198291 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0011d20 con 0x7f6cdc1593c0 2023-03-03T12:36:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 -- 172.21.15.102:0/18198291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc1593c0 msgr2=0x7f6cdc0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:36:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/18198291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc1593c0 0x7f6cdc0cde20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00098f0 tx=0x7f6cd0009c00 comp rx=0 tx=0).stop 2023-03-03T12:36:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 -- 172.21.15.102:0/18198291 shutdown_connections 2023-03-03T12:36:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/18198291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc153290 0x7f6cdc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/18198291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc1593c0 0x7f6cdc0cde20 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 -- 172.21.15.102:0/18198291 >> 172.21.15.102:0/18198291 conn(0x7f6cdc065fb0 msgr2=0x7f6cdc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:36:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 -- 172.21.15.102:0/18198291 shutdown_connections 2023-03-03T12:36:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.445+0000 7f6ce0fff700 1 -- 172.21.15.102:0/18198291 wait complete. 2023-03-03T12:36:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.448+0000 7f6ce0fff700 1 Processor -- start 2023-03-03T12:36:22.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.449+0000 7f6ce0fff700 1 -- start start 2023-03-03T12:36:22.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6ce0fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc0cd8e0 0x7f6cdc0cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:36:22.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6ce0fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cdc153290 0x7f6cdc154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:36:22.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6ce0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc1593c0 0x7f6cdc14fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:36:22.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6ce0fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc153f50 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6ce0fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc1501a0 con 0x7f6cdc153290 2023-03-03T12:36:22.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6ce0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0b2740 con 0x7f6cdc1593c0 2023-03-03T12:36:22.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6cda59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc0cd8e0 0x7f6cdc0cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:36:22.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6cda59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc0cd8e0 0x7f6cdc0cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48136/0 (socket says 172.21.15.102:48136) 2023-03-03T12:36:22.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6cda59c700 1 -- 172.21.15.102:0/2197398322 learned_addr learned my addr 172.21.15.102:0/2197398322 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:36:22.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6cd9d9b700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cdc153290 0x7f6cdc154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:36:22.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.450+0000 7f6cdad9d700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc1593c0 0x7f6cdc14fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:36:22.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6cda59c700 1 -- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc1593c0 msgr2=0x7f6cdc14fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:36:22.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6cda59c700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc1593c0 0x7f6cdc14fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6cda59c700 1 -- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cdc153290 msgr2=0x7f6cdc154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6cda59c700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cdc153290 0x7f6cdc154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6cda59c700 1 -- 172.21.15.102:0/2197398322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd0009550 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6cdad9d700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc1593c0 0x7f6cdc14fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6cda59c700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc0cd8e0 0x7f6cdc0cad00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00e9b0 tx=0x7f6ccc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6ccaffd700 1 -- 172.21.15.102:0/2197398322 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc009710 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6ccaffd700 1 -- 172.21.15.102:0/2197398322 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc00c850 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc0ccde0 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.451+0000 7f6ccaffd700 1 -- 172.21.15.102:0/2197398322 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc00cbc0 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.452+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc15d160 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.453+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc0cd0c0 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.454+0000 7f6ccaffd700 1 -- 172.21.15.102:0/2197398322 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6ccc019030 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.455+0000 7f6ccaffd700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6cbc0682f0 0x7f6cbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:36:22.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.455+0000 7f6ccaffd700 1 -- 172.21.15.102:0/2197398322 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6ccc085d00 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.455+0000 7f6cd9d9b700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6cbc0682f0 0x7f6cbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:36:22.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.456+0000 7f6cd9d9b700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6cbc0682f0 0x7f6cbc06a7a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009130 tx=0x7f6cd0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:36:22.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.459+0000 7f6ccaffd700 1 -- 172.21.15.102:0/2197398322 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ccc052410 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.695+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cdc04e6e0 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.696+0000 7f6ccaffd700 1 -- 172.21.15.102:0/2197398322 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6ccc017090 con 0x7f6cdc0cd8e0 2023-03-03T12:36:22.699 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:36:22.699 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:36:22.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.698+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6cbc0682f0 msgr2=0x7f6cbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:36:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.698+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6cbc0682f0 0x7f6cbc06a7a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009130 tx=0x7f6cd0008030 comp rx=0 tx=0).stop 2023-03-03T12:36:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.698+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc0cd8e0 msgr2=0x7f6cdc0cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:36:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.698+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc0cd8e0 0x7f6cdc0cad00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00e9b0 tx=0x7f6ccc00ecc0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 shutdown_connections 2023-03-03T12:36:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6cbc0682f0 0x7f6cbc06a7a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6cdc1593c0 0x7f6cdc14fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cdc0cd8e0 0x7f6cdc0cad00 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 --2- 172.21.15.102:0/2197398322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cdc153290 0x7f6cdc154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:36:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 >> 172.21.15.102:0/2197398322 conn(0x7f6cdc065fb0 msgr2=0x7f6cdc1e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:36:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 shutdown_connections 2023-03-03T12:36:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:36:22.699+0000 7f6ce0fff700 1 -- 172.21.15.102:0/2197398322 wait complete. 2023-03-03T12:36:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:36:22.728 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-03-03T12:36:30.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:30.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:00.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:00.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:31.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:31.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:43.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:37:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.974+0000 7f6a663ff700 1 Processor -- start 2023-03-03T12:37:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.975+0000 7f6a663ff700 1 -- start start 2023-03-03T12:37:43.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a663ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a60153290 0x7f6a60158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:43.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a663ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a600cd8e0 0x7f6a600ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:43.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a663ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a60153660 con 0x7f6a601593c0 2023-03-03T12:37:43.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a663ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6006c360 con 0x7f6a600cd8e0 2023-03-03T12:37:43.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a663ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6015cea0 con 0x7f6a60153290 2023-03-03T12:37:43.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a5ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a60153290 0x7f6a60158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:43.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a5f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6a601593c0 0x7f6a600cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53038/0 (socket says 172.21.15.102:53038) 2023-03-03T12:37:43.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a5f7fe700 1 -- 172.21.15.102:0/2565535536 learned_addr learned my addr 172.21.15.102:0/2565535536 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:37:43.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.976+0000 7f6a6499c700 1 --2- 172.21.15.102:0/2565535536 >> v2:172.21.15.102:3300/0 conn(0x7f6a600cd8e0 0x7f6a600ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:43.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.977+0000 7f6a6499c700 1 -- 172.21.15.102:0/2565535536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a60153290 msgr2=0x7f6a60158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:37:43.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.977+0000 7f6a6499c700 1 --2- 172.21.15.102:0/2565535536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a60153290 0x7f6a60158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:43.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.977+0000 7f6a6499c700 1 -- 172.21.15.102:0/2565535536 >> v1:172.21.15.102:6789/0 conn(0x7f6a601593c0 legacy=0x7f6a600cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:43.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.977+0000 7f6a5effd700 1 -- 172.21.15.102:0/2565535536 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814632822 0 0) 0x7f6a60153660 con 0x7f6a601593c0 2023-03-03T12:37:43.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.977+0000 7f6a6499c700 1 -- 172.21.15.102:0/2565535536 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6006c7f0 con 0x7f6a600cd8e0 2023-03-03T12:37:43.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.977+0000 7f6a5ffff700 1 --2- 172.21.15.102:0/2565535536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a60153290 0x7f6a60158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:37:43.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.977+0000 7f6a6499c700 1 --2- 172.21.15.102:0/2565535536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a600cd8e0 0x7f6a600ce530 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009d90 tx=0x7f6a5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7416a754262e199a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:43.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a5effd700 1 -- 172.21.15.102:0/2565535536 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800e070 con 0x7f6a600cd8e0 2023-03-03T12:37:43.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a5effd700 1 -- 172.21.15.102:0/2565535536 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5800bb20 con 0x7f6a600cd8e0 2023-03-03T12:37:43.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a5effd700 1 -- 172.21.15.102:0/2565535536 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58012430 con 0x7f6a600cd8e0 2023-03-03T12:37:43.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 -- 172.21.15.102:0/2565535536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a600cd8e0 msgr2=0x7f6a600ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:43.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2565535536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a600cd8e0 0x7f6a600ce530 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009d90 tx=0x7f6a5800b330 comp rx=0 tx=0).stop 2023-03-03T12:37:43.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 -- 172.21.15.102:0/2565535536 shutdown_connections 2023-03-03T12:37:43.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2565535536 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a60153290 0x7f6a60158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:43.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2565535536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a600cd8e0 0x7f6a600ce530 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:43.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 -- 172.21.15.102:0/2565535536 >> 172.21.15.102:0/2565535536 conn(0x7f6a60065fb0 msgr2=0x7f6a600663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:43.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 -- 172.21.15.102:0/2565535536 shutdown_connections 2023-03-03T12:37:43.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.978+0000 7f6a663ff700 1 -- 172.21.15.102:0/2565535536 wait complete. 2023-03-03T12:37:43.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.980+0000 7f6a663ff700 1 Processor -- start 2023-03-03T12:37:43.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.980+0000 7f6a663ff700 1 -- start start 2023-03-03T12:37:43.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.981+0000 7f6a663ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a600cd8e0 0x7f6a600c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.981+0000 7f6a663ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60153290 0x7f6a601ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.981+0000 7f6a663ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a601593c0 0x7f6a6014f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.981+0000 7f6a663ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a600c6b50 con 0x7f6a601593c0 2023-03-03T12:37:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.981+0000 7f6a663ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a60156e90 con 0x7f6a60153290 2023-03-03T12:37:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.981+0000 7f6a663ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a600c6da0 con 0x7f6a600cd8e0 2023-03-03T12:37:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a601593c0 0x7f6a6014f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a601593c0 0x7f6a6014f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47430/0 (socket says 172.21.15.102:47430) 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a5f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60153290 0x7f6a601ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 -- 172.21.15.102:0/2723220059 learned_addr learned my addr 172.21.15.102:0/2723220059 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a5ffff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a600cd8e0 0x7f6a600c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 -- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a600cd8e0 msgr2=0x7f6a600c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a600cd8e0 0x7f6a600c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 -- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60153290 msgr2=0x7f6a601ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:43.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60153290 0x7f6a601ef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.982+0000 7f6a6499c700 1 -- 172.21.15.102:0/2723220059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a500094a0 con 0x7f6a601593c0 2023-03-03T12:37:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.983+0000 7f6a6499c700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a601593c0 0x7f6a6014f5a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6a60074540 tx=0x7f6a58004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.983+0000 7f6a5cff9700 1 -- 172.21.15.102:0/2723220059 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58012430 con 0x7f6a601593c0 2023-03-03T12:37:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.983+0000 7f6a5cff9700 1 -- 172.21.15.102:0/2723220059 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a58020070 con 0x7f6a601593c0 2023-03-03T12:37:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.983+0000 7f6a5cff9700 1 -- 172.21.15.102:0/2723220059 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58004280 con 0x7f6a601593c0 2023-03-03T12:37:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.983+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a58009a70 con 0x7f6a601593c0 2023-03-03T12:37:43.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.983+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a600ccde0 con 0x7f6a601593c0 2023-03-03T12:37:43.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.984+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6015d160 con 0x7f6a601593c0 2023-03-03T12:37:43.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.985+0000 7f6a5cff9700 1 -- 172.21.15.102:0/2723220059 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a5802e070 con 0x7f6a601593c0 2023-03-03T12:37:43.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.987+0000 7f6a5cff9700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a4c068320 0x7f6a4c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:43.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.987+0000 7f6a5cff9700 1 -- 172.21.15.102:0/2723220059 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6a58022070 con 0x7f6a601593c0 2023-03-03T12:37:43.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.987+0000 7f6a5ffff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a4c068320 0x7f6a4c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:43.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.988+0000 7f6a5ffff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a4c068320 0x7f6a4c06a7d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009d70 tx=0x7f6a50009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:43.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:43.992+0000 7f6a5cff9700 1 -- 172.21.15.102:0/2723220059 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a580540f0 con 0x7f6a601593c0 2023-03-03T12:37:44.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.227+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a600cd0c0 con 0x7f6a601593c0 2023-03-03T12:37:44.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.227+0000 7f6a5cff9700 1 -- 172.21.15.102:0/2723220059 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6a58057710 con 0x7f6a601593c0 2023-03-03T12:37:44.229 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:37:44.230 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:37:44.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a4c068320 msgr2=0x7f6a4c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:44.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a4c068320 0x7f6a4c06a7d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009d70 tx=0x7f6a50009210 comp rx=0 tx=0).stop 2023-03-03T12:37:44.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a601593c0 msgr2=0x7f6a6014f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:44.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a601593c0 0x7f6a6014f5a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6a60074540 tx=0x7f6a58004ad0 comp rx=0 tx=0).stop 2023-03-03T12:37:44.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 shutdown_connections 2023-03-03T12:37:44.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a4c068320 0x7f6a4c06a7d0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:44.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a600cd8e0 0x7f6a600c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:44.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a601593c0 0x7f6a6014f5a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:44.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 --2- 172.21.15.102:0/2723220059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60153290 0x7f6a601ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:44.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 >> 172.21.15.102:0/2723220059 conn(0x7f6a60065fb0 msgr2=0x7f6a600cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:44.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 shutdown_connections 2023-03-03T12:37:44.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:37:44.230+0000 7f6a663ff700 1 -- 172.21.15.102:0/2723220059 wait complete. 2023-03-03T12:37:44.234 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:37:44.260 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-03-03T12:38:01.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:01.192 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:31.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:31.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:57.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:38:57.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.513+0000 7f0f859ff700 1 Processor -- start 2023-03-03T12:38:57.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.513+0000 7f0f859ff700 1 -- start start 2023-03-03T12:38:57.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f859ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f80153290 0x7f0f80158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:57.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f859ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0f801593c0 0x7f0f800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:57.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f859ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f80153660 con 0x7f0f800cd8e0 2023-03-03T12:38:57.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f859ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8006c360 con 0x7f0f80153290 2023-03-03T12:38:57.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f859ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8015cea0 con 0x7f0f801593c0 2023-03-03T12:38:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f7effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f80153290 0x7f0f80158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f7effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f80153290 0x7f0f80158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60614/0 (socket says 172.21.15.102:60614) 2023-03-03T12:38:57.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f7effd700 1 -- 172.21.15.102:0/389087726 learned_addr learned my addr 172.21.15.102:0/389087726 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:38:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.515+0000 7f0f7e7fc700 1 --2- 172.21.15.102:0/389087726 >> v2:172.21.15.172:3300/0 conn(0x7f0f801593c0 0x7f0f800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7effd700 1 -- 172.21.15.102:0/389087726 >> v2:172.21.15.172:3300/0 conn(0x7f0f801593c0 msgr2=0x7f0f800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:38:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7effd700 1 --2- 172.21.15.102:0/389087726 >> v2:172.21.15.172:3300/0 conn(0x7f0f801593c0 0x7f0f800cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7effd700 1 -- 172.21.15.102:0/389087726 >> v1:172.21.15.102:6789/0 conn(0x7f0f800cd8e0 legacy=0x7f0f800ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7effd700 1 -- 172.21.15.102:0/389087726 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8006c7f0 con 0x7f0f80153290 2023-03-03T12:38:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7effd700 1 --2- 172.21.15.102:0/389087726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f80153290 0x7f0f80158e80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0f80159fb0 tx=0x7f0f70009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ccd6de972ba461 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:57.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7dffb700 1 -- 172.21.15.102:0/389087726 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70011830 con 0x7f0f80153290 2023-03-03T12:38:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7dffb700 1 -- 172.21.15.102:0/389087726 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70011990 con 0x7f0f80153290 2023-03-03T12:38:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.516+0000 7f0f7dffb700 1 -- 172.21.15.102:0/389087726 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70011d20 con 0x7f0f80153290 2023-03-03T12:38:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 -- 172.21.15.102:0/389087726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f80153290 msgr2=0x7f0f80158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:57.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 --2- 172.21.15.102:0/389087726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f80153290 0x7f0f80158e80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0f80159fb0 tx=0x7f0f70009bd0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 -- 172.21.15.102:0/389087726 shutdown_connections 2023-03-03T12:38:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 --2- 172.21.15.102:0/389087726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f80153290 0x7f0f80158e80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 --2- 172.21.15.102:0/389087726 >> v2:172.21.15.172:3300/0 conn(0x7f0f801593c0 0x7f0f800cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 -- 172.21.15.102:0/389087726 >> 172.21.15.102:0/389087726 conn(0x7f0f80065fb0 msgr2=0x7f0f800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:57.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 -- 172.21.15.102:0/389087726 shutdown_connections 2023-03-03T12:38:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.517+0000 7f0f859ff700 1 -- 172.21.15.102:0/389087726 wait complete. 2023-03-03T12:38:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.519+0000 7f0f859ff700 1 Processor -- start 2023-03-03T12:38:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.519+0000 7f0f859ff700 1 -- start start 2023-03-03T12:38:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.520+0000 7f0f859ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f800cd8e0 0x7f0f8014c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.520+0000 7f0f859ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f80153290 0x7f0f80155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.520+0000 7f0f859ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f801593c0 0x7f0f80156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:57.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.520+0000 7f0f859ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f801ee930 con 0x7f0f800cd8e0 2023-03-03T12:38:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.520+0000 7f0f859ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800b45f0 con 0x7f0f801593c0 2023-03-03T12:38:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.520+0000 7f0f859ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800b1d10 con 0x7f0f80153290 2023-03-03T12:38:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f800cd8e0 0x7f0f8014c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f800cd8e0 0x7f0f8014c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60630/0 (socket says 172.21.15.102:60630) 2023-03-03T12:38:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 -- 172.21.15.102:0/4222015046 learned_addr learned my addr 172.21.15.102:0/4222015046 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:38:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7e7fc700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f80153290 0x7f0f80155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:57.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7f7fe700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f801593c0 0x7f0f80156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 -- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f80153290 msgr2=0x7f0f80155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f80153290 0x7f0f80155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 -- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f801593c0 msgr2=0x7f0f80156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f801593c0 0x7f0f80156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.521+0000 7f0f7effd700 1 -- 172.21.15.102:0/4222015046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70009550 con 0x7f0f800cd8e0 2023-03-03T12:38:57.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.522+0000 7f0f7effd700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f800cd8e0 0x7f0f8014c770 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009c10 tx=0x7f0f70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:57.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.522+0000 7f0f67fff700 1 -- 172.21.15.102:0/4222015046 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70004380 con 0x7f0f800cd8e0 2023-03-03T12:38:57.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.522+0000 7f0f67fff700 1 -- 172.21.15.102:0/4222015046 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7001f070 con 0x7f0f800cd8e0 2023-03-03T12:38:57.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.522+0000 7f0f67fff700 1 -- 172.21.15.102:0/4222015046 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f700036a0 con 0x7f0f800cd8e0 2023-03-03T12:38:57.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.522+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f800ccde0 con 0x7f0f800cd8e0 2023-03-03T12:38:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.522+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8015d160 con 0x7f0f800cd8e0 2023-03-03T12:38:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.524+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f800cd0c0 con 0x7f0f800cd8e0 2023-03-03T12:38:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.524+0000 7f0f67fff700 1 -- 172.21.15.102:0/4222015046 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f7002a020 con 0x7f0f800cd8e0 2023-03-03T12:38:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.526+0000 7f0f67fff700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f6c0682f0 0x7f0f6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.526+0000 7f0f67fff700 1 -- 172.21.15.102:0/4222015046 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0f70087310 con 0x7f0f800cd8e0 2023-03-03T12:38:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.526+0000 7f0f7e7fc700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f6c0682f0 0x7f0f6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.527+0000 7f0f7e7fc700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f6c0682f0 0x7f0f6c06a7a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0f8015a4d0 tx=0x7f0f74009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.530+0000 7f0f67fff700 1 -- 172.21.15.102:0/4222015046 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f700539a0 con 0x7f0f800cd8e0 2023-03-03T12:38:57.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.766+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f8004e6e0 con 0x7f0f800cd8e0 2023-03-03T12:38:57.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.767+0000 7f0f67fff700 1 -- 172.21.15.102:0/4222015046 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0f70056fc0 con 0x7f0f800cd8e0 2023-03-03T12:38:57.769 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:38:57.769 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:38:57.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.769+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f6c0682f0 msgr2=0x7f0f6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.769+0000 7f0f859ff700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f6c0682f0 0x7f0f6c06a7a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0f8015a4d0 tx=0x7f0f74009210 comp rx=0 tx=0).stop 2023-03-03T12:38:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.769+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f800cd8e0 msgr2=0x7f0f8014c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.769+0000 7f0f859ff700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f800cd8e0 0x7f0f8014c770 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009c10 tx=0x7f0f70004750 comp rx=0 tx=0).stop 2023-03-03T12:38:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 shutdown_connections 2023-03-03T12:38:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f801593c0 0x7f0f80156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f800cd8e0 0x7f0f8014c770 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0f6c0682f0 0x7f0f6c06a7a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 --2- 172.21.15.102:0/4222015046 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0f80153290 0x7f0f80155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 >> 172.21.15.102:0/4222015046 conn(0x7f0f80065fb0 msgr2=0x7f0f800cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 shutdown_connections 2023-03-03T12:38:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:38:57.770+0000 7f0f859ff700 1 -- 172.21.15.102:0/4222015046 wait complete. 2023-03-03T12:38:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:38:57.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T12:39:01.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:01.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:31.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:31.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:40.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:39:41.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.044+0000 7fb1f139d700 1 Processor -- start 2023-03-03T12:39:41.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.044+0000 7fb1f139d700 1 -- start start 2023-03-03T12:39:41.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.045+0000 7fb1f139d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb1ec1564b0 0x7fb1ec0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:41.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.045+0000 7fb1f139d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb1ec0cab50 0x7fb1ec0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:41.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1f139d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ec0c9ca0 con 0x7fb1ec0c9610 2023-03-03T12:39:41.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1f139d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ec0b4920 con 0x7fb1ec1564b0 2023-03-03T12:39:41.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1f139d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ec0b4c00 con 0x7fb1ec0cab50 2023-03-03T12:39:41.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1ebfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb1ec1564b0 0x7fb1ec0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:41.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1eb7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb1ec0c9610 0x7fb1ec0ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41766/0 (socket says 172.21.15.102:41766) 2023-03-03T12:39:41.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1f0b9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb1ec0cab50 0x7fb1ec0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:41.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1eb7fe700 1 -- 172.21.15.102:0/2295109091 learned_addr learned my addr 172.21.15.102:0/2295109091 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:39:41.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1ebfff700 1 -- 172.21.15.102:0/2295109091 >> v2:172.21.15.172:3300/0 conn(0x7fb1ec0cab50 msgr2=0x7fb1ec0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1ebfff700 1 --2- 172.21.15.102:0/2295109091 >> v2:172.21.15.172:3300/0 conn(0x7fb1ec0cab50 0x7fb1ec0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1ebfff700 1 -- 172.21.15.102:0/2295109091 >> v1:172.21.15.102:6789/0 conn(0x7fb1ec0c9610 legacy=0x7fb1ec0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1ebfff700 1 -- 172.21.15.102:0/2295109091 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec0b46f0 con 0x7fb1ec1564b0 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.046+0000 7fb1eaffd700 1 -- 172.21.15.102:0/2295109091 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512287485 0 0) 0x7fb1ec0c9ca0 con 0x7fb1ec0c9610 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.047+0000 7fb1ebfff700 1 --2- 172.21.15.102:0/2295109091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 0x7fb1ec0cb660 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec06ca60 tx=0x7fb1dc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7854a8b39ace620f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.047+0000 7fb1eaffd700 1 -- 172.21.15.102:0/2295109091 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc011830 con 0x7fb1ec1564b0 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.047+0000 7fb1eaffd700 1 -- 172.21.15.102:0/2295109091 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc011990 con 0x7fb1ec1564b0 2023-03-03T12:39:41.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.047+0000 7fb1eaffd700 1 -- 172.21.15.102:0/2295109091 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc011d20 con 0x7fb1ec1564b0 2023-03-03T12:39:41.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 -- 172.21.15.102:0/2295109091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 msgr2=0x7fb1ec0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:41.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 --2- 172.21.15.102:0/2295109091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 0x7fb1ec0cb660 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec06ca60 tx=0x7fb1dc009b20 comp rx=0 tx=0).stop 2023-03-03T12:39:41.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 -- 172.21.15.102:0/2295109091 shutdown_connections 2023-03-03T12:39:41.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 --2- 172.21.15.102:0/2295109091 >> v2:172.21.15.172:3300/0 conn(0x7fb1ec0cab50 0x7fb1ec0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 --2- 172.21.15.102:0/2295109091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 0x7fb1ec0cb660 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 -- 172.21.15.102:0/2295109091 >> 172.21.15.102:0/2295109091 conn(0x7fb1ec065fb0 msgr2=0x7fb1ec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:41.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 -- 172.21.15.102:0/2295109091 shutdown_connections 2023-03-03T12:39:41.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.048+0000 7fb1f139d700 1 -- 172.21.15.102:0/2295109091 wait complete. 2023-03-03T12:39:41.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.050+0000 7fb1f139d700 1 Processor -- start 2023-03-03T12:39:41.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.050+0000 7fb1f139d700 1 -- start start 2023-03-03T12:39:41.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1f139d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1ec0c9610 0x7fb1ec15c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1f139d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1ec0cab50 0x7fb1ec156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1f139d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 0x7fb1ec157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1f139d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec1eedb0 con 0x7fb1ec1564b0 2023-03-03T12:39:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1f139d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec0c8180 con 0x7fb1ec0c9610 2023-03-03T12:39:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1f139d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec1ee1a0 con 0x7fb1ec0cab50 2023-03-03T12:39:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1ebfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1ec0c9610 0x7fb1ec15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1f0b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 0x7fb1ec157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:41.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1eb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1ec0cab50 0x7fb1ec156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:41.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1ebfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1ec0c9610 0x7fb1ec15c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:44752/0 (socket says 172.21.15.102:44752) 2023-03-03T12:39:41.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.051+0000 7fb1ebfff700 1 -- 172.21.15.102:0/1956333704 learned_addr learned my addr 172.21.15.102:0/1956333704 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:39:41.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1ebfff700 1 -- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1ec0cab50 msgr2=0x7fb1ec156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:41.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1ebfff700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1ec0cab50 0x7fb1ec156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1ebfff700 1 -- 172.21.15.102:0/1956333704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 msgr2=0x7fb1ec157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:41.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1ebfff700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 0x7fb1ec157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1ebfff700 1 -- 172.21.15.102:0/1956333704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0094a0 con 0x7fb1ec0c9610 2023-03-03T12:39:41.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1ebfff700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1ec0c9610 0x7fb1ec15c080 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc014040 tx=0x7fb1dc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:41.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1e8ff9700 1 -- 172.21.15.102:0/1956333704 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0036a0 con 0x7fb1ec0c9610 2023-03-03T12:39:41.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1e8ff9700 1 -- 172.21.15.102:0/1956333704 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc01e070 con 0x7fb1ec0c9610 2023-03-03T12:39:41.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec0b2210 con 0x7fb1ec0c9610 2023-03-03T12:39:41.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.052+0000 7fb1e8ff9700 1 -- 172.21.15.102:0/1956333704 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc003a20 con 0x7fb1ec0c9610 2023-03-03T12:39:41.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.053+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec0b1f30 con 0x7fb1ec0c9610 2023-03-03T12:39:41.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.054+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec156880 con 0x7fb1ec0c9610 2023-03-03T12:39:41.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.055+0000 7fb1e8ff9700 1 -- 172.21.15.102:0/1956333704 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb1dc00f050 con 0x7fb1ec0c9610 2023-03-03T12:39:41.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.056+0000 7fb1e8ff9700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1d80682f0 0x7fb1d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:41.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.056+0000 7fb1e8ff9700 1 -- 172.21.15.102:0/1956333704 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb1dc086c40 con 0x7fb1ec0c9610 2023-03-03T12:39:41.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.056+0000 7fb1eb7fe700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1d80682f0 0x7fb1d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:41.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.057+0000 7fb1eb7fe700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1d80682f0 0x7fb1d806a7a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec155bb0 tx=0x7fb1e0007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:41.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.060+0000 7fb1e8ff9700 1 -- 172.21.15.102:0/1956333704 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb1dc053350 con 0x7fb1ec0c9610 2023-03-03T12:39:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.293+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1ec0677f0 con 0x7fb1ec0c9610 2023-03-03T12:39:41.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.294+0000 7fb1e8ff9700 1 -- 172.21.15.102:0/1956333704 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb1dc056970 con 0x7fb1ec0c9610 2023-03-03T12:39:41.297 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:39:41.297 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:39:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1d80682f0 msgr2=0x7fb1d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1d80682f0 0x7fb1d806a7a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec155bb0 tx=0x7fb1e0007de0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1ec0c9610 msgr2=0x7fb1ec15c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1ec0c9610 0x7fb1ec15c080 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc014040 tx=0x7fb1dc004750 comp rx=0 tx=0).stop 2023-03-03T12:39:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 shutdown_connections 2023-03-03T12:39:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1ec1564b0 0x7fb1ec157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1ec0c9610 0x7fb1ec15c080 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1d80682f0 0x7fb1d806a7a0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 --2- 172.21.15.102:0/1956333704 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb1ec0cab50 0x7fb1ec156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 >> 172.21.15.102:0/1956333704 conn(0x7fb1ec065fb0 msgr2=0x7fb1ec066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 shutdown_connections 2023-03-03T12:39:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:41.297+0000 7fb1f139d700 1 -- 172.21.15.102:0/1956333704 wait complete. 2023-03-03T12:39:41.302 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:39:41.327 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-03-03T12:39:53.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:39:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.591+0000 7f915bdff700 1 Processor -- start 2023-03-03T12:39:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.592+0000 7f915bdff700 1 -- start start 2023-03-03T12:39:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.593+0000 7f915bdff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f91540cd390 0x7f91540ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.593+0000 7f915bdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9154153660 con 0x7f91540cd8e0 2023-03-03T12:39:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.593+0000 7f915bdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915406c360 con 0x7f9154153290 2023-03-03T12:39:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.593+0000 7f915bdff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915415cea0 con 0x7f91540cd390 2023-03-03T12:39:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.593+0000 7f9159b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9154153290 0x7f9154158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38834/0 (socket says 172.21.15.102:38834) 2023-03-03T12:39:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.593+0000 7f9159b9b700 1 -- 172.21.15.102:0/1416079267 learned_addr learned my addr 172.21.15.102:0/1416079267 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:39:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.593+0000 7f915a39c700 1 --2- 172.21.15.102:0/1416079267 >> v2:172.21.15.172:3300/0 conn(0x7f91540cd390 0x7f91540ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.594+0000 7f9158b99700 1 -- 172.21.15.102:0/1416079267 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169593321 0 0) 0x7f915406c360 con 0x7f9154153290 2023-03-03T12:39:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.594+0000 7f9158b99700 1 -- 172.21.15.102:0/1416079267 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91400035b0 con 0x7f9154153290 2023-03-03T12:39:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.594+0000 7f915a39c700 1 -- 172.21.15.102:0/1416079267 >> v1:172.21.15.172:6789/0 conn(0x7f9154153290 legacy=0x7f9154158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.594+0000 7f9158b99700 1 -- 172.21.15.102:0/1416079267 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143393327 0 0) 0x7f9154153660 con 0x7f91540cd8e0 2023-03-03T12:39:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.594+0000 7f915a39c700 1 -- 172.21.15.102:0/1416079267 >> v1:172.21.15.102:6789/0 conn(0x7f91540cd8e0 legacy=0x7f91540cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.594+0000 7f915a39c700 1 -- 172.21.15.102:0/1416079267 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915406c7f0 con 0x7f91540cd390 2023-03-03T12:39:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.594+0000 7f915a39c700 1 --2- 172.21.15.102:0/1416079267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 0x7f91540ce530 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f914c009d90 tx=0x7f914c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8fca5a981d10e8f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f9158b99700 1 -- 172.21.15.102:0/1416079267 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c00e070 con 0x7f91540cd390 2023-03-03T12:39:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f9158b99700 1 -- 172.21.15.102:0/1416079267 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c00bb20 con 0x7f91540cd390 2023-03-03T12:39:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f9158b99700 1 -- 172.21.15.102:0/1416079267 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c00bed0 con 0x7f91540cd390 2023-03-03T12:39:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f915bdff700 1 -- 172.21.15.102:0/1416079267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 msgr2=0x7f91540ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f915bdff700 1 --2- 172.21.15.102:0/1416079267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 0x7f91540ce530 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f914c009d90 tx=0x7f914c00b330 comp rx=0 tx=0).stop 2023-03-03T12:39:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f915bdff700 1 -- 172.21.15.102:0/1416079267 shutdown_connections 2023-03-03T12:39:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f915bdff700 1 --2- 172.21.15.102:0/1416079267 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 0x7f91540ce530 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f915bdff700 1 -- 172.21.15.102:0/1416079267 >> 172.21.15.102:0/1416079267 conn(0x7f9154065fb0 msgr2=0x7f91540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f915bdff700 1 -- 172.21.15.102:0/1416079267 shutdown_connections 2023-03-03T12:39:53.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.595+0000 7f915bdff700 1 -- 172.21.15.102:0/1416079267 wait complete. 2023-03-03T12:39:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.597+0000 7f915bdff700 1 Processor -- start 2023-03-03T12:39:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.597+0000 7f915bdff700 1 -- start start 2023-03-03T12:39:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.598+0000 7f915bdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 0x7f91540c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:53.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.598+0000 7f915bdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91540cd8e0 0x7f91541eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.598+0000 7f915bdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9154153290 0x7f91541ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.598+0000 7f915bdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91541537f0 con 0x7f91540cd8e0 2023-03-03T12:39:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.598+0000 7f915bdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91540c7340 con 0x7f91540cd390 2023-03-03T12:39:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.598+0000 7f915bdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9154153d20 con 0x7f9154153290 2023-03-03T12:39:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91540cd8e0 0x7f91541eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91540cd8e0 0x7f91541eeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48300/0 (socket says 172.21.15.102:48300) 2023-03-03T12:39:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 -- 172.21.15.102:0/3137820761 learned_addr learned my addr 172.21.15.102:0/3137820761 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:39:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915a39c700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9154153290 0x7f91541ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f9159b9b700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 0x7f91540c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 -- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9154153290 msgr2=0x7f91541ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9154153290 0x7f91541ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 -- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 msgr2=0x7f91540c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 0x7f91540c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.599+0000 7f915939a700 1 -- 172.21.15.102:0/3137820761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c009a70 con 0x7f91540cd8e0 2023-03-03T12:39:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.600+0000 7f915939a700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91540cd8e0 0x7f91541eeb00 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f91440011e0 tx=0x7f914400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.600+0000 7f914a7fc700 1 -- 172.21.15.102:0/3137820761 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91440054e0 con 0x7f91540cd8e0 2023-03-03T12:39:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.600+0000 7f914a7fc700 1 -- 172.21.15.102:0/3137820761 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9144003680 con 0x7f91540cd8e0 2023-03-03T12:39:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.600+0000 7f914a7fc700 1 -- 172.21.15.102:0/3137820761 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9144003a30 con 0x7f91540cd8e0 2023-03-03T12:39:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.600+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91540ccde0 con 0x7f91540cd8e0 2023-03-03T12:39:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.600+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915415d160 con 0x7f91540cd8e0 2023-03-03T12:39:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.601+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91540cd0c0 con 0x7f91540cd8e0 2023-03-03T12:39:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.603+0000 7f914a7fc700 1 -- 172.21.15.102:0/3137820761 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9144019030 con 0x7f91540cd8e0 2023-03-03T12:39:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.604+0000 7f914a7fc700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9138068320 0x7f913806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.604+0000 7f914a7fc700 1 -- 172.21.15.102:0/3137820761 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f91440850a0 con 0x7f91540cd8e0 2023-03-03T12:39:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.604+0000 7f9159b9b700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9138068320 0x7f913806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.605+0000 7f9159b9b700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9138068320 0x7f913806a7d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9150007fd0 tx=0x7f9150007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.608+0000 7f914a7fc700 1 -- 172.21.15.102:0/3137820761 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91440517b0 con 0x7f91540cd8e0 2023-03-03T12:39:53.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.841+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91540c71a0 con 0x7f91540cd8e0 2023-03-03T12:39:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.842+0000 7f914a7fc700 1 -- 172.21.15.102:0/3137820761 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f91540c71a0 con 0x7f91540cd8e0 2023-03-03T12:39:53.844 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:39:53.844 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:39:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9138068320 msgr2=0x7f913806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9138068320 0x7f913806a7d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9150007fd0 tx=0x7f9150007dc0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91540cd8e0 msgr2=0x7f91541eeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91540cd8e0 0x7f91541eeb00 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f91440011e0 tx=0x7f914400ca30 comp rx=0 tx=0).stop 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 shutdown_connections 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9138068320 0x7f913806a7d0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9154153290 0x7f91541ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91540cd390 0x7f91540c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 --2- 172.21.15.102:0/3137820761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91540cd8e0 0x7f91541eeb00 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 >> 172.21.15.102:0/3137820761 conn(0x7f9154065fb0 msgr2=0x7f91541575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 shutdown_connections 2023-03-03T12:39:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:39:53.845+0000 7f915bdff700 1 -- 172.21.15.102:0/3137820761 wait complete. 2023-03-03T12:39:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:39:53.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-03-03T12:40:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T12:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:40:01.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:01.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:32.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:32.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:46.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:40:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.122+0000 7f8ff9d9d700 1 Processor -- start 2023-03-03T12:40:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.123+0000 7f8ff9d9d700 1 -- start start 2023-03-03T12:40:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff9d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40c9610 0x7f8ff40ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff9d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8ff40ca980 0x7f8ff41636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff9d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40c9ca0 con 0x7f8ff41564b0 2023-03-03T12:40:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff9d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40cadc0 con 0x7f8ff40ca980 2023-03-03T12:40:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff9d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b4920 con 0x7f8ff40c9610 2023-03-03T12:40:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff8d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8ff41564b0 0x7f8ff40cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36124/0 (socket says 172.21.15.102:36124) 2023-03-03T12:40:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff8d9b700 1 -- 172.21.15.102:0/1074578716 learned_addr learned my addr 172.21.15.102:0/1074578716 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:40:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff3fff700 1 --2- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40c9610 0x7f8ff40ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.124+0000 7f8ff959c700 1 --2- 172.21.15.102:0/1074578716 >> v2:172.21.15.172:3300/0 conn(0x7f8ff40ca980 0x7f8ff41636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:47.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff37fe700 1 -- 172.21.15.102:0/1074578716 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042568734 0 0) 0x7f8ff40c9ca0 con 0x7f8ff41564b0 2023-03-03T12:40:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff37fe700 1 -- 172.21.15.102:0/1074578716 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe40035b0 con 0x7f8ff41564b0 2023-03-03T12:40:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff959c700 1 -- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40c9610 msgr2=0x7f8ff40ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff959c700 1 --2- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40c9610 0x7f8ff40ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff959c700 1 -- 172.21.15.102:0/1074578716 >> v1:172.21.15.172:6789/0 conn(0x7f8ff41564b0 legacy=0x7f8ff40cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff959c700 1 -- 172.21.15.102:0/1074578716 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40b46f0 con 0x7f8ff40ca980 2023-03-03T12:40:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff3fff700 1 --2- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40c9610 0x7f8ff40ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:40:47.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.125+0000 7f8ff959c700 1 --2- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40ca980 0x7f8ff41636a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009d90 tx=0x7f8fec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf7ac2d7cb35c9b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:47.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff37fe700 1 -- 172.21.15.102:0/1074578716 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00e070 con 0x7f8ff40ca980 2023-03-03T12:40:47.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff37fe700 1 -- 172.21.15.102:0/1074578716 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec00bb20 con 0x7f8ff40ca980 2023-03-03T12:40:47.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff37fe700 1 -- 172.21.15.102:0/1074578716 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec012430 con 0x7f8ff40ca980 2023-03-03T12:40:47.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40ca980 msgr2=0x7f8ff41636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:47.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40ca980 0x7f8ff41636a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009d90 tx=0x7f8fec00b330 comp rx=0 tx=0).stop 2023-03-03T12:40:47.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/1074578716 shutdown_connections 2023-03-03T12:40:47.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40ca980 0x7f8ff41636a0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/1074578716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40c9610 0x7f8ff40ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/1074578716 >> 172.21.15.102:0/1074578716 conn(0x7f8ff4065fb0 msgr2=0x7f8ff40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:47.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/1074578716 shutdown_connections 2023-03-03T12:40:47.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.126+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/1074578716 wait complete. 2023-03-03T12:40:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.129+0000 7f8ff9d9d700 1 Processor -- start 2023-03-03T12:40:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.129+0000 7f8ff9d9d700 1 -- start start 2023-03-03T12:40:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff9d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40c9610 0x7f8ff41f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff9d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40ca980 0x7f8ff41df0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff9d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff41564b0 0x7f8ff4144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff9d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4066c00 con 0x7f8ff41564b0 2023-03-03T12:40:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff9d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4156b90 con 0x7f8ff40c9610 2023-03-03T12:40:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff9d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41f1ab0 con 0x7f8ff40ca980 2023-03-03T12:40:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff8d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40c9610 0x7f8ff41f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff8d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40c9610 0x7f8ff41f1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55598/0 (socket says 172.21.15.102:55598) 2023-03-03T12:40:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff8d9b700 1 -- 172.21.15.102:0/3950728866 learned_addr learned my addr 172.21.15.102:0/3950728866 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:40:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff3fff700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40ca980 0x7f8ff41df0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.130+0000 7f8ff959c700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff41564b0 0x7f8ff4144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff8d9b700 1 -- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40ca980 msgr2=0x7f8ff41df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff8d9b700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40ca980 0x7f8ff41df0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff8d9b700 1 -- 172.21.15.102:0/3950728866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff41564b0 msgr2=0x7f8ff4144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff8d9b700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff41564b0 0x7f8ff4144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff8d9b700 1 -- 172.21.15.102:0/3950728866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00094a0 con 0x7f8ff40c9610 2023-03-03T12:40:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff3fff700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40ca980 0x7f8ff41df0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:40:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff8d9b700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40c9610 0x7f8ff41f1570 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8001eb0 tx=0x7f8fe800c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff17fa700 1 -- 172.21.15.102:0/3950728866 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8005d70 con 0x7f8ff40c9610 2023-03-03T12:40:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff17fa700 1 -- 172.21.15.102:0/3950728866 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe80070d0 con 0x7f8ff40c9610 2023-03-03T12:40:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff17fa700 1 -- 172.21.15.102:0/3950728866 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8007460 con 0x7f8ff40c9610 2023-03-03T12:40:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.131+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec009a70 con 0x7f8ff40c9610 2023-03-03T12:40:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.132+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff40b1f30 con 0x7f8ff40c9610 2023-03-03T12:40:47.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.133+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff40b4c00 con 0x7f8ff40c9610 2023-03-03T12:40:47.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.134+0000 7f8ff17fa700 1 -- 172.21.15.102:0/3950728866 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8fe8019030 con 0x7f8ff40c9610 2023-03-03T12:40:47.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.135+0000 7f8ff17fa700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8fd8068320 0x7f8fd806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:47.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.135+0000 7f8ff17fa700 1 -- 172.21.15.102:0/3950728866 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8fe8085060 con 0x7f8ff40c9610 2023-03-03T12:40:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.136+0000 7f8ff3fff700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8fd8068320 0x7f8fd806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.136+0000 7f8ff3fff700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8fd8068320 0x7f8fd806a7d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40cdfa0 tx=0x7f8fe0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.139+0000 7f8ff17fa700 1 -- 172.21.15.102:0/3950728866 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8fe8051770 con 0x7f8ff40c9610 2023-03-03T12:40:47.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.374+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ff40b2210 con 0x7f8ff40c9610 2023-03-03T12:40:47.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.375+0000 7f8ff17fa700 1 -- 172.21.15.102:0/3950728866 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8fe8017090 con 0x7f8ff40c9610 2023-03-03T12:40:47.377 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:40:47.377 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:40:47.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.377+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8fd8068320 msgr2=0x7f8fd806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:47.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.377+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8fd8068320 0x7f8fd806a7d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40cdfa0 tx=0x7f8fe0009210 comp rx=0 tx=0).stop 2023-03-03T12:40:47.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.377+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40c9610 msgr2=0x7f8ff41f1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:47.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.377+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40c9610 0x7f8ff41f1570 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8001eb0 tx=0x7f8fe800c7b0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 shutdown_connections 2023-03-03T12:40:47.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff41564b0 0x7f8ff4144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ff40c9610 0x7f8ff41f1570 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8fd8068320 0x7f8fd806a7d0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 --2- 172.21.15.102:0/3950728866 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8ff40ca980 0x7f8ff41df0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:47.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 >> 172.21.15.102:0/3950728866 conn(0x7f8ff4065fb0 msgr2=0x7f8ff406b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:47.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 shutdown_connections 2023-03-03T12:40:47.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:40:47.378+0000 7f8ff9d9d700 1 -- 172.21.15.102:0/3950728866 wait complete. 2023-03-03T12:40:47.381 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:40:47.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-03-03T12:41:02.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:02.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:32.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:32.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:02.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:02.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:04.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:42:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.657+0000 7fe0393ff700 1 Processor -- start 2023-03-03T12:42:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.657+0000 7fe0393ff700 1 -- start start 2023-03-03T12:42:04.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.658+0000 7fe0393ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe0340cd8e0 0x7fe0340cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:04.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.658+0000 7fe0393ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034153660 con 0x7fe0340cd390 2023-03-03T12:42:04.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.658+0000 7fe0393ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03406c360 con 0x7fe034153290 2023-03-03T12:42:04.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.658+0000 7fe0393ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03415cea0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe0327fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe0340cd8e0 0x7fe0340cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe032ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe034153290 0x7fe034158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34556/0 (socket says 172.21.15.102:34556) 2023-03-03T12:42:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe032ffd700 1 -- 172.21.15.102:0/1361058295 learned_addr learned my addr 172.21.15.102:0/1361058295 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:42:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe031ffb700 1 -- 172.21.15.102:0/1361058295 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071335977 0 0) 0x7fe03406c360 con 0x7fe034153290 2023-03-03T12:42:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe031ffb700 1 -- 172.21.15.102:0/1361058295 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c0035b0 con 0x7fe034153290 2023-03-03T12:42:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe0327fc700 1 -- 172.21.15.102:0/1361058295 >> v1:172.21.15.172:6789/0 conn(0x7fe034153290 legacy=0x7fe034158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe0327fc700 1 -- 172.21.15.102:0/1361058295 >> v1:172.21.15.102:6789/0 conn(0x7fe0340cd390 legacy=0x7fe0340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe031ffb700 1 -- 172.21.15.102:0/1361058295 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837967231 0 0) 0x7fe034153660 con 0x7fe0340cd390 2023-03-03T12:42:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.659+0000 7fe0327fc700 1 -- 172.21.15.102:0/1361058295 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03406c7f0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.660+0000 7fe0327fc700 1 --2- 172.21.15.102:0/1361058295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd8e0 0x7fe0340cde20 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe028009840 tx=0x7fe028009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91759be83ac96b75 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:04.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.660+0000 7fe031ffb700 1 -- 172.21.15.102:0/1361058295 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028011830 con 0x7fe0340cd8e0 2023-03-03T12:42:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.660+0000 7fe031ffb700 1 -- 172.21.15.102:0/1361058295 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe028011990 con 0x7fe0340cd8e0 2023-03-03T12:42:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.660+0000 7fe031ffb700 1 -- 172.21.15.102:0/1361058295 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028011d20 con 0x7fe0340cd8e0 2023-03-03T12:42:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.661+0000 7fe0393ff700 1 -- 172.21.15.102:0/1361058295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd8e0 msgr2=0x7fe0340cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.661+0000 7fe0393ff700 1 --2- 172.21.15.102:0/1361058295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd8e0 0x7fe0340cde20 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe028009840 tx=0x7fe028009b50 comp rx=0 tx=0).stop 2023-03-03T12:42:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.661+0000 7fe0393ff700 1 -- 172.21.15.102:0/1361058295 shutdown_connections 2023-03-03T12:42:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.661+0000 7fe0393ff700 1 --2- 172.21.15.102:0/1361058295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd8e0 0x7fe0340cde20 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:04.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.661+0000 7fe0393ff700 1 -- 172.21.15.102:0/1361058295 >> 172.21.15.102:0/1361058295 conn(0x7fe034065fb0 msgr2=0x7fe0340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:42:04.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.661+0000 7fe0393ff700 1 -- 172.21.15.102:0/1361058295 shutdown_connections 2023-03-03T12:42:04.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.661+0000 7fe0393ff700 1 -- 172.21.15.102:0/1361058295 wait complete. 2023-03-03T12:42:04.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.663+0000 7fe0393ff700 1 Processor -- start 2023-03-03T12:42:04.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.663+0000 7fe0393ff700 1 -- start start 2023-03-03T12:42:04.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0393ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd390 0x7fe0340b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:04.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0393ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340cd8e0 0x7fe0340c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:04.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0393ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe034153290 0x7fe0340c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0393ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340c9210 con 0x7fe0340cd390 2023-03-03T12:42:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0393ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe034154750 con 0x7fe0340cd8e0 2023-03-03T12:42:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0393ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340c75a0 con 0x7fe034153290 2023-03-03T12:42:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe032ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd390 0x7fe0340b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0327fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340cd8e0 0x7fe0340c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0327fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340cd8e0 0x7fe0340c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47244/0 (socket says 172.21.15.102:47244) 2023-03-03T12:42:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0327fc700 1 -- 172.21.15.102:0/2117559858 learned_addr learned my addr 172.21.15.102:0/2117559858 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:42:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.664+0000 7fe0337fe700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe034153290 0x7fe0340c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe0327fc700 1 -- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe034153290 msgr2=0x7fe0340c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe0327fc700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe034153290 0x7fe0340c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe0327fc700 1 -- 172.21.15.102:0/2117559858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd390 msgr2=0x7fe0340b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe0327fc700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd390 0x7fe0340b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe0327fc700 1 -- 172.21.15.102:0/2117559858 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0280094a0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe0327fc700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340cd8e0 0x7fe0340c7d10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe0280090b0 tx=0x7fe028004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe01b7fe700 1 -- 172.21.15.102:0/2117559858 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02801a7e0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe01b7fe700 1 -- 172.21.15.102:0/2117559858 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02801a940 con 0x7fe0340cd8e0 2023-03-03T12:42:04.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe01b7fe700 1 -- 172.21.15.102:0/2117559858 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02801acd0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.665+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0340ccde0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.666+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03415d160 con 0x7fe0340cd8e0 2023-03-03T12:42:04.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.667+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0340cd0c0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.668+0000 7fe01b7fe700 1 -- 172.21.15.102:0/2117559858 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe02802b030 con 0x7fe0340cd8e0 2023-03-03T12:42:04.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.669+0000 7fe01b7fe700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe020068320 0x7fe02006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:04.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.669+0000 7fe01b7fe700 1 -- 172.21.15.102:0/2117559858 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe028086fe0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.670+0000 7fe032ffd700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe020068320 0x7fe02006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:04.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.670+0000 7fe032ffd700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe020068320 0x7fe02006a7d0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe024007fd0 tx=0x7fe024009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.675+0000 7fe01b7fe700 1 -- 172.21.15.102:0/2117559858 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe0280536a0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.905+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe03404e6e0 con 0x7fe0340cd8e0 2023-03-03T12:42:04.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.906+0000 7fe01b7fe700 1 -- 172.21.15.102:0/2117559858 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe028029070 con 0x7fe0340cd8e0 2023-03-03T12:42:04.908 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:42:04.908 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:42:04.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe020068320 msgr2=0x7fe02006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:04.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe020068320 0x7fe02006a7d0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe024007fd0 tx=0x7fe024009040 comp rx=0 tx=0).stop 2023-03-03T12:42:04.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340cd8e0 msgr2=0x7fe0340c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:04.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340cd8e0 0x7fe0340c7d10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe0280090b0 tx=0x7fe028004750 comp rx=0 tx=0).stop 2023-03-03T12:42:04.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 shutdown_connections 2023-03-03T12:42:04.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe020068320 0x7fe02006a7d0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:04.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe034153290 0x7fe0340c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:04.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0340cd390 0x7fe0340b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:04.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 --2- 172.21.15.102:0/2117559858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0340cd8e0 0x7fe0340c7d10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:04.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 >> 172.21.15.102:0/2117559858 conn(0x7fe034065fb0 msgr2=0x7fe034158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:42:04.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 shutdown_connections 2023-03-03T12:42:04.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:04.909+0000 7fe0393ff700 1 -- 172.21.15.102:0/2117559858 wait complete. 2023-03-03T12:42:04.912 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:42:04.938 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 15 secs before thrashing 2023-03-03T12:42:19.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:42:20.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.195+0000 7ff48bfff700 1 Processor -- start 2023-03-03T12:42:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.195+0000 7ff48bfff700 1 -- start start 2023-03-03T12:42:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff48bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff48bfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff484153660 con 0x7ff484153290 2023-03-03T12:42:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff48bfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48406c360 con 0x7ff4840cd8e0 2023-03-03T12:42:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff48bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48415cea0 con 0x7ff4840cd390 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff489d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff484153290 0x7ff484158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38302/0 (socket says 172.21.15.102:38302) 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff489d9b700 1 -- 172.21.15.102:0/2418640331 learned_addr learned my addr 172.21.15.102:0/2418640331 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff48a59c700 1 --2- 172.21.15.102:0/2418640331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.197+0000 7ff488d99700 1 -- 172.21.15.102:0/2418640331 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223619656 0 0) 0x7ff484153660 con 0x7ff484153290 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff488d99700 1 -- 172.21.15.102:0/2418640331 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4700035b0 con 0x7ff484153290 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff488d99700 1 -- 172.21.15.102:0/2418640331 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120598325 0 0) 0x7ff48406c360 con 0x7ff4840cd8e0 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff488d99700 1 -- 172.21.15.102:0/2418640331 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff484153660 con 0x7ff4840cd8e0 2023-03-03T12:42:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff48a59c700 1 -- 172.21.15.102:0/2418640331 >> v1:172.21.15.172:6789/0 conn(0x7ff4840cd8e0 legacy=0x7ff4840cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff48a59c700 1 -- 172.21.15.102:0/2418640331 >> v1:172.21.15.102:6789/0 conn(0x7ff484153290 legacy=0x7ff484158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff48a59c700 1 -- 172.21.15.102:0/2418640331 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48406c7f0 con 0x7ff4840cd390 2023-03-03T12:42:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff48a59c700 1 --2- 172.21.15.102:0/2418640331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840ce530 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009dc0 tx=0x7ff47c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e134655ee90861f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff488d99700 1 -- 172.21.15.102:0/2418640331 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00e070 con 0x7ff4840cd390 2023-03-03T12:42:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff488d99700 1 -- 172.21.15.102:0/2418640331 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c00bb20 con 0x7ff4840cd390 2023-03-03T12:42:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.198+0000 7ff488d99700 1 -- 172.21.15.102:0/2418640331 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c012430 con 0x7ff4840cd390 2023-03-03T12:42:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.199+0000 7ff48bfff700 1 -- 172.21.15.102:0/2418640331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 msgr2=0x7ff4840ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.199+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2418640331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840ce530 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009dc0 tx=0x7ff47c00b330 comp rx=0 tx=0).stop 2023-03-03T12:42:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.199+0000 7ff48bfff700 1 -- 172.21.15.102:0/2418640331 shutdown_connections 2023-03-03T12:42:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.199+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2418640331 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840ce530 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.199+0000 7ff48bfff700 1 -- 172.21.15.102:0/2418640331 >> 172.21.15.102:0/2418640331 conn(0x7ff484065fb0 msgr2=0x7ff4840663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:42:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.199+0000 7ff48bfff700 1 -- 172.21.15.102:0/2418640331 shutdown_connections 2023-03-03T12:42:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.199+0000 7ff48bfff700 1 -- 172.21.15.102:0/2418640331 wait complete. 2023-03-03T12:42:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.201+0000 7ff48bfff700 1 Processor -- start 2023-03-03T12:42:20.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.201+0000 7ff48bfff700 1 -- start start 2023-03-03T12:42:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.202+0000 7ff48bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.202+0000 7ff48bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4840cd8e0 0x7ff48414e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff484153290 0x7ff4840c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48bfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484154360 con 0x7ff4840cd8e0 2023-03-03T12:42:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48bfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48414ea20 con 0x7ff484153290 2023-03-03T12:42:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48bfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4840c91c0 con 0x7ff4840cd390 2023-03-03T12:42:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff484153290 0x7ff4840c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48959a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4840cd8e0 0x7ff48414e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff489d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff489d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840c7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51910/0 (socket says 172.21.15.102:51910) 2023-03-03T12:42:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff489d9b700 1 -- 172.21.15.102:0/2245247581 learned_addr learned my addr 172.21.15.102:0/2245247581 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:42:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48959a700 1 -- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 msgr2=0x7ff4840c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48959a700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48959a700 1 -- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff484153290 msgr2=0x7ff4840c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48959a700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff484153290 0x7ff4840c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48959a700 1 -- 172.21.15.102:0/2245247581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c009a70 con 0x7ff4840cd8e0 2023-03-03T12:42:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff489d9b700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:42:20.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.203+0000 7ff48a59c700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff484153290 0x7ff4840c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.204+0000 7ff48959a700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4840cd8e0 0x7ff48414e4e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff47400c450 tx=0x7ff47400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.204+0000 7ff47a7fc700 1 -- 172.21.15.102:0/2245247581 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4740052d0 con 0x7ff4840cd8e0 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.204+0000 7ff47a7fc700 1 -- 172.21.15.102:0/2245247581 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff474005430 con 0x7ff4840cd8e0 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.204+0000 7ff47a7fc700 1 -- 172.21.15.102:0/2245247581 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4740038a0 con 0x7ff4840cd8e0 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.204+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4840ccde0 con 0x7ff4840cd8e0 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.204+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48415d160 con 0x7ff4840cd8e0 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.205+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4840cd0c0 con 0x7ff4840cd8e0 2023-03-03T12:42:20.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.206+0000 7ff47a7fc700 1 -- 172.21.15.102:0/2245247581 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff474019030 con 0x7ff4840cd8e0 2023-03-03T12:42:20.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.208+0000 7ff47a7fc700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff4680682f0 0x7ff46806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:20.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.208+0000 7ff47a7fc700 1 -- 172.21.15.102:0/2245247581 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff474085070 con 0x7ff4840cd8e0 2023-03-03T12:42:20.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.209+0000 7ff489d9b700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff4680682f0 0x7ff46806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:20.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.209+0000 7ff489d9b700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff4680682f0 0x7ff46806a7a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff480000f60 tx=0x7ff480009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:20.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.213+0000 7ff47a7fc700 1 -- 172.21.15.102:0/2245247581 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4740516f0 con 0x7ff4840cd8e0 2023-03-03T12:42:20.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.446+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff48404e6e0 con 0x7ff4840cd8e0 2023-03-03T12:42:20.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.447+0000 7ff47a7fc700 1 -- 172.21.15.102:0/2245247581 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff474017020 con 0x7ff4840cd8e0 2023-03-03T12:42:20.449 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:42:20.449 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:42:20.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.449+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff4680682f0 msgr2=0x7ff46806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:20.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.449+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff4680682f0 0x7ff46806a7a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff480000f60 tx=0x7ff480009040 comp rx=0 tx=0).stop 2023-03-03T12:42:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.449+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4840cd8e0 msgr2=0x7ff48414e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.449+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4840cd8e0 0x7ff48414e4e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff47400c450 tx=0x7ff47400c760 comp rx=0 tx=0).stop 2023-03-03T12:42:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.449+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 shutdown_connections 2023-03-03T12:42:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.450+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff4680682f0 0x7ff46806a7a0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.450+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4840cd390 0x7ff4840c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.450+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff484153290 0x7ff4840c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.450+0000 7ff48bfff700 1 --2- 172.21.15.102:0/2245247581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4840cd8e0 0x7ff48414e4e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:20.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.450+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 >> 172.21.15.102:0/2245247581 conn(0x7ff484065fb0 msgr2=0x7ff4841585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:42:20.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.450+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 shutdown_connections 2023-03-03T12:42:20.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:42:20.450+0000 7ff48bfff700 1 -- 172.21.15.102:0/2245247581 wait complete. 2023-03-03T12:42:20.453 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:42:20.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 78 secs before thrashing 2023-03-03T12:42:32.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:32.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:03.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:03.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:33.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:33.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:38.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:43:38.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.746+0000 7fc7873ff700 1 Processor -- start 2023-03-03T12:43:38.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.746+0000 7fc7873ff700 1 -- start start 2023-03-03T12:43:38.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.747+0000 7fc7873ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7800cd8e0 0x7fc7800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:38.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.747+0000 7fc7873ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd390 0x7fc7800ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:38.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.747+0000 7fc7873ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc780153660 con 0x7fc780153290 2023-03-03T12:43:38.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.747+0000 7fc7873ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78006c360 con 0x7fc7800cd8e0 2023-03-03T12:43:38.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.747+0000 7fc7873ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78015cea0 con 0x7fc7800cd390 2023-03-03T12:43:38.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78499a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7800cd8e0 0x7fc7800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:38.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78519b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc780153290 0x7fc780158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41052/0 (socket says 172.21.15.102:41052) 2023-03-03T12:43:38.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78519b700 1 -- 172.21.15.102:0/1074439382 learned_addr learned my addr 172.21.15.102:0/1074439382 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:43:38.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78599c700 1 --2- 172.21.15.102:0/1074439382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd390 0x7fc7800ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:38.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc777fff700 1 -- 172.21.15.102:0/1074439382 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031602458 0 0) 0x7fc780153660 con 0x7fc780153290 2023-03-03T12:43:38.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78499a700 1 -- 172.21.15.102:0/1074439382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd390 msgr2=0x7fc7800ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:43:38.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78499a700 1 --2- 172.21.15.102:0/1074439382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd390 0x7fc7800ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:38.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78499a700 1 -- 172.21.15.102:0/1074439382 >> v1:172.21.15.172:6789/0 conn(0x7fc780153290 legacy=0x7fc780158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:38.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.748+0000 7fc78499a700 1 -- 172.21.15.102:0/1074439382 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78006c7f0 con 0x7fc7800cd8e0 2023-03-03T12:43:38.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.749+0000 7fc78499a700 1 --2- 172.21.15.102:0/1074439382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd8e0 0x7fc7800cde20 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc770009840 tx=0x7fc770009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf1b50443824e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:38.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.749+0000 7fc777fff700 1 -- 172.21.15.102:0/1074439382 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770011830 con 0x7fc7800cd8e0 2023-03-03T12:43:38.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.749+0000 7fc777fff700 1 -- 172.21.15.102:0/1074439382 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc770011990 con 0x7fc7800cd8e0 2023-03-03T12:43:38.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.749+0000 7fc777fff700 1 -- 172.21.15.102:0/1074439382 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770011d20 con 0x7fc7800cd8e0 2023-03-03T12:43:38.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.749+0000 7fc7873ff700 1 -- 172.21.15.102:0/1074439382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd8e0 msgr2=0x7fc7800cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:38.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.749+0000 7fc7873ff700 1 --2- 172.21.15.102:0/1074439382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd8e0 0x7fc7800cde20 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc770009840 tx=0x7fc770009b50 comp rx=0 tx=0).stop 2023-03-03T12:43:38.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.750+0000 7fc7873ff700 1 -- 172.21.15.102:0/1074439382 shutdown_connections 2023-03-03T12:43:38.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.750+0000 7fc7873ff700 1 --2- 172.21.15.102:0/1074439382 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd390 0x7fc7800ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:38.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.750+0000 7fc7873ff700 1 --2- 172.21.15.102:0/1074439382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd8e0 0x7fc7800cde20 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:38.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.750+0000 7fc7873ff700 1 -- 172.21.15.102:0/1074439382 >> 172.21.15.102:0/1074439382 conn(0x7fc780065fb0 msgr2=0x7fc7800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:38.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.750+0000 7fc7873ff700 1 -- 172.21.15.102:0/1074439382 shutdown_connections 2023-03-03T12:43:38.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.750+0000 7fc7873ff700 1 -- 172.21.15.102:0/1074439382 wait complete. 2023-03-03T12:43:38.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.752+0000 7fc7873ff700 1 Processor -- start 2023-03-03T12:43:38.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.752+0000 7fc7873ff700 1 -- start start 2023-03-03T12:43:38.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc7873ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd390 0x7fc7800b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:38.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc7873ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd8e0 0x7fc78015ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:38.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc7873ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc780153290 0x7fc7800c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc7873ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801e3ed0 con 0x7fc7800cd390 2023-03-03T12:43:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc7873ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801ef270 con 0x7fc780153290 2023-03-03T12:43:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc7873ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801ef550 con 0x7fc7800cd8e0 2023-03-03T12:43:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc78499a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd8e0 0x7fc78015ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc78599c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc780153290 0x7fc7800c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc78499a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd8e0 0x7fc78015ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:33764/0 (socket says 172.21.15.102:33764) 2023-03-03T12:43:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc78499a700 1 -- 172.21.15.102:0/3467530012 learned_addr learned my addr 172.21.15.102:0/3467530012 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:43:38.757 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.753+0000 7fc78519b700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd390 0x7fc7800b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:38.757 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc78599c700 1 -- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd8e0 msgr2=0x7fc78015ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:38.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc78599c700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd8e0 0x7fc78015ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:38.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc78599c700 1 -- 172.21.15.102:0/3467530012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd390 msgr2=0x7fc7800b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:38.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc78599c700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd390 0x7fc7800b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:38.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc78599c700 1 -- 172.21.15.102:0/3467530012 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7700094a0 con 0x7fc780153290 2023-03-03T12:43:38.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc78499a700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd8e0 0x7fc78015ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:43:38.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc78599c700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc780153290 0x7fc7800c7930 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc77800eba0 tx=0x7fc77800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:38.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc775ffb700 1 -- 172.21.15.102:0/3467530012 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc778009a80 con 0x7fc780153290 2023-03-03T12:43:38.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc775ffb700 1 -- 172.21.15.102:0/3467530012 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77800f040 con 0x7fc780153290 2023-03-03T12:43:38.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7800ccde0 con 0x7fc780153290 2023-03-03T12:43:38.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.754+0000 7fc775ffb700 1 -- 172.21.15.102:0/3467530012 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77800cc40 con 0x7fc780153290 2023-03-03T12:43:38.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.755+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78015d160 con 0x7fc780153290 2023-03-03T12:43:38.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.756+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7800cd0c0 con 0x7fc780153290 2023-03-03T12:43:38.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.757+0000 7fc775ffb700 1 -- 172.21.15.102:0/3467530012 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc778014480 con 0x7fc780153290 2023-03-03T12:43:38.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.758+0000 7fc775ffb700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:38.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.758+0000 7fc775ffb700 1 -- 172.21.15.102:0/3467530012 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc7780868e0 con 0x7fc780153290 2023-03-03T12:43:38.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.758+0000 7fc78519b700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:38.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.759+0000 7fc78519b700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc77c007fd0 tx=0x7fc77c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:38.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.762+0000 7fc775ffb700 1 -- 172.21.15.102:0/3467530012 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc778052f70 con 0x7fc780153290 2023-03-03T12:43:38.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.996+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc78004e6e0 con 0x7fc780153290 2023-03-03T12:43:38.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:38.997+0000 7fc775ffb700 1 -- 172.21.15.102:0/3467530012 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc778056590 con 0x7fc780153290 2023-03-03T12:43:38.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:43:39.000 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:43:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 msgr2=0x7fc76406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc77c007fd0 tx=0x7fc77c007de0 comp rx=0 tx=0).stop 2023-03-03T12:43:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc780153290 msgr2=0x7fc7800c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc780153290 0x7fc7800c7930 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc77800eba0 tx=0x7fc77800eeb0 comp rx=0 tx=0).stop 2023-03-03T12:43:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 shutdown_connections 2023-03-03T12:43:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc780153290 0x7fc7800c7930 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7800cd390 0x7fc7800b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc764068320 0x7fc76406a7d0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 --2- 172.21.15.102:0/3467530012 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7800cd8e0 0x7fc78015ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.000+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 >> 172.21.15.102:0/3467530012 conn(0x7fc780065fb0 msgr2=0x7fc780158940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.001+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 shutdown_connections 2023-03-03T12:43:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:43:39.001+0000 7fc7873ff700 1 -- 172.21.15.102:0/3467530012 wait complete. 2023-03-03T12:43:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:43:39.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T12:44:03.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:03.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:27.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:44:27.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.292+0000 7fb74dbff700 1 Processor -- start 2023-03-03T12:44:27.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.292+0000 7fb74dbff700 1 -- start start 2023-03-03T12:44:27.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.294+0000 7fb74dbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7480cd390 0x7fb7480ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.294+0000 7fb74dbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb748153660 con 0x7fb7480cd8e0 2023-03-03T12:44:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.294+0000 7fb74dbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74806c360 con 0x7fb748153290 2023-03-03T12:44:27.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.294+0000 7fb74dbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74815cea0 con 0x7fb7480cd390 2023-03-03T12:44:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.294+0000 7fb746ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb7480cd8e0 0x7fb7480cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55074/0 (socket says 172.21.15.102:55074) 2023-03-03T12:44:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.294+0000 7fb746ffd700 1 -- 172.21.15.102:0/2894057738 learned_addr learned my addr 172.21.15.102:0/2894057738 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:44:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.294+0000 7fb747fff700 1 --2- 172.21.15.102:0/2894057738 >> v2:172.21.15.102:3300/0 conn(0x7fb7480cd390 0x7fb7480ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb7467fc700 1 -- 172.21.15.102:0/2894057738 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050135610 0 0) 0x7fb74806c360 con 0x7fb748153290 2023-03-03T12:44:27.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb7467fc700 1 -- 172.21.15.102:0/2894057738 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7340035b0 con 0x7fb748153290 2023-03-03T12:44:27.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb747fff700 1 -- 172.21.15.102:0/2894057738 >> v1:172.21.15.172:6789/0 conn(0x7fb748153290 legacy=0x7fb748158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:27.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb747fff700 1 -- 172.21.15.102:0/2894057738 >> v1:172.21.15.102:6789/0 conn(0x7fb7480cd8e0 legacy=0x7fb7480cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:27.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb7467fc700 1 -- 172.21.15.102:0/2894057738 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632644668 0 0) 0x7fb748153660 con 0x7fb7480cd8e0 2023-03-03T12:44:27.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb747fff700 1 -- 172.21.15.102:0/2894057738 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74806c7f0 con 0x7fb7480cd390 2023-03-03T12:44:27.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb747fff700 1 --2- 172.21.15.102:0/2894057738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7480cd390 0x7fb7480ce530 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb740009d90 tx=0x7fb74000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590bf16d5a003417 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:27.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb7467fc700 1 -- 172.21.15.102:0/2894057738 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74000e070 con 0x7fb7480cd390 2023-03-03T12:44:27.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb7467fc700 1 -- 172.21.15.102:0/2894057738 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74000bb20 con 0x7fb7480cd390 2023-03-03T12:44:27.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.295+0000 7fb7467fc700 1 -- 172.21.15.102:0/2894057738 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74000bed0 con 0x7fb7480cd390 2023-03-03T12:44:27.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.296+0000 7fb74dbff700 1 -- 172.21.15.102:0/2894057738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7480cd390 msgr2=0x7fb7480ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:27.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.296+0000 7fb74dbff700 1 --2- 172.21.15.102:0/2894057738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7480cd390 0x7fb7480ce530 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb740009d90 tx=0x7fb74000b330 comp rx=0 tx=0).stop 2023-03-03T12:44:27.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.296+0000 7fb74dbff700 1 -- 172.21.15.102:0/2894057738 shutdown_connections 2023-03-03T12:44:27.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.296+0000 7fb74dbff700 1 --2- 172.21.15.102:0/2894057738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7480cd390 0x7fb7480ce530 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.296+0000 7fb74dbff700 1 -- 172.21.15.102:0/2894057738 >> 172.21.15.102:0/2894057738 conn(0x7fb748065fb0 msgr2=0x7fb7480663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:44:27.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.296+0000 7fb74dbff700 1 -- 172.21.15.102:0/2894057738 shutdown_connections 2023-03-03T12:44:27.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.296+0000 7fb74dbff700 1 -- 172.21.15.102:0/2894057738 wait complete. 2023-03-03T12:44:27.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.298+0000 7fb74dbff700 1 Processor -- start 2023-03-03T12:44:27.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.298+0000 7fb74dbff700 1 -- start start 2023-03-03T12:44:27.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.299+0000 7fb74dbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7480cd390 0x7fb7480c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:27.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.299+0000 7fb74dbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7480cd8e0 0x7fb7481eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:27.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.299+0000 7fb74dbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb748153290 0x7fb7481ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:27.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.299+0000 7fb74dbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7481537f0 con 0x7fb748153290 2023-03-03T12:44:27.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.299+0000 7fb74dbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7480c7340 con 0x7fb7480cd390 2023-03-03T12:44:27.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.299+0000 7fb74dbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb748153d20 con 0x7fb7480cd8e0 2023-03-03T12:44:27.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb748153290 0x7fb7481ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:27.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb746ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7480cd8e0 0x7fb7481eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:27.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb748153290 0x7fb7481ef040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51548/0 (socket says 172.21.15.102:51548) 2023-03-03T12:44:27.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 -- 172.21.15.102:0/83509462 learned_addr learned my addr 172.21.15.102:0/83509462 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:44:27.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb7477fe700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7480cd390 0x7fb7480c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:27.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 -- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7480cd8e0 msgr2=0x7fb7481eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:27.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7480cd8e0 0x7fb7481eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 -- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7480cd390 msgr2=0x7fb7480c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:27.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7480cd390 0x7fb7480c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb747fff700 1 -- 172.21.15.102:0/83509462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb740009a70 con 0x7fb748153290 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.300+0000 7fb746ffd700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7480cd8e0 0x7fb7481eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.301+0000 7fb747fff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb748153290 0x7fb7481ef040 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb740014040 tx=0x7fb740004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.301+0000 7fb73ffff700 1 -- 172.21.15.102:0/83509462 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74001b810 con 0x7fb748153290 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.301+0000 7fb73ffff700 1 -- 172.21.15.102:0/83509462 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740020070 con 0x7fb748153290 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.301+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7480ccde0 con 0x7fb748153290 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.301+0000 7fb73ffff700 1 -- 172.21.15.102:0/83509462 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74001bbb0 con 0x7fb748153290 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.301+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74815d160 con 0x7fb748153290 2023-03-03T12:44:27.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.302+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7480cd0c0 con 0x7fb748153290 2023-03-03T12:44:27.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.303+0000 7fb73ffff700 1 -- 172.21.15.102:0/83509462 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb74002e070 con 0x7fb748153290 2023-03-03T12:44:27.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.305+0000 7fb73ffff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb728068320 0x7fb72806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:27.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.305+0000 7fb73ffff700 1 -- 172.21.15.102:0/83509462 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb740022070 con 0x7fb748153290 2023-03-03T12:44:27.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.305+0000 7fb7477fe700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb728068320 0x7fb72806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:27.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.306+0000 7fb7477fe700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb728068320 0x7fb72806a7d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb7300011e0 tx=0x7fb730009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:27.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.309+0000 7fb73ffff700 1 -- 172.21.15.102:0/83509462 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7400543f0 con 0x7fb748153290 2023-03-03T12:44:27.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.545+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7480c71a0 con 0x7fb748153290 2023-03-03T12:44:27.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.545+0000 7fb73ffff700 1 -- 172.21.15.102:0/83509462 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb7480c71a0 con 0x7fb748153290 2023-03-03T12:44:27.547 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:44:27.547 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:44:27.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb728068320 msgr2=0x7fb72806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:27.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb728068320 0x7fb72806a7d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb7300011e0 tx=0x7fb730009040 comp rx=0 tx=0).stop 2023-03-03T12:44:27.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb748153290 msgr2=0x7fb7481ef040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:27.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb748153290 0x7fb7481ef040 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb740014040 tx=0x7fb740004ad0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 shutdown_connections 2023-03-03T12:44:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb748153290 0x7fb7481ef040 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7480cd390 0x7fb7480c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb728068320 0x7fb72806a7d0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 --2- 172.21.15.102:0/83509462 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7480cd8e0 0x7fb7481eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 >> 172.21.15.102:0/83509462 conn(0x7fb748065fb0 msgr2=0x7fb7481575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:44:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 shutdown_connections 2023-03-03T12:44:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:27.548+0000 7fb74dbff700 1 -- 172.21.15.102:0/83509462 wait complete. 2023-03-03T12:44:27.552 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:44:27.577 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 22 secs before thrashing 2023-03-03T12:44:33.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:33.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:49.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:44:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.826+0000 7f6d9a79d700 1 Processor -- start 2023-03-03T12:44:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.826+0000 7f6d9a79d700 1 -- start start 2023-03-03T12:44:49.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d9a79d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6d940b3970 0x7f6d940c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:49.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d9a79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940c41b0 0x7f6d940cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:49.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d9a79d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d940c3d20 con 0x7f6d940ced00 2023-03-03T12:44:49.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d9a79d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d940c4930 con 0x7f6d940b3970 2023-03-03T12:44:49.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d9a79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940cf610 con 0x7f6d940c41b0 2023-03-03T12:44:49.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d9979b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6d940ced00 0x7f6d94158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47088/0 (socket says 172.21.15.102:47088) 2023-03-03T12:44:49.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d99f9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940c41b0 0x7f6d940cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:49.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d9979b700 1 -- 172.21.15.102:0/34532773 learned_addr learned my addr 172.21.15.102:0/34532773 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:44:49.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.828+0000 7f6d98f9a700 1 --2- 172.21.15.102:0/34532773 >> v2:172.21.15.172:3300/0 conn(0x7f6d940b3970 0x7f6d940c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d8bfff700 1 -- 172.21.15.102:0/34532773 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243209280 0 0) 0x7f6d940c3d20 con 0x7f6d940ced00 2023-03-03T12:44:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d8bfff700 1 -- 172.21.15.102:0/34532773 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d800035b0 con 0x7f6d940ced00 2023-03-03T12:44:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d99f9c700 1 -- 172.21.15.102:0/34532773 >> v2:172.21.15.172:3300/0 conn(0x7f6d940b3970 msgr2=0x7f6d940c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:44:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d99f9c700 1 --2- 172.21.15.102:0/34532773 >> v2:172.21.15.172:3300/0 conn(0x7f6d940b3970 0x7f6d940c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d99f9c700 1 -- 172.21.15.102:0/34532773 >> v1:172.21.15.172:6789/0 conn(0x7f6d940ced00 legacy=0x7f6d94158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d99f9c700 1 -- 172.21.15.102:0/34532773 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9415cba0 con 0x7f6d940c41b0 2023-03-03T12:44:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d99f9c700 1 --2- 172.21.15.102:0/34532773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940c41b0 0x7f6d940cf0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009d90 tx=0x7f6d8c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1683619fbd2aadd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:49.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d8bfff700 1 -- 172.21.15.102:0/34532773 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c00e070 con 0x7f6d940c41b0 2023-03-03T12:44:49.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d8bfff700 1 -- 172.21.15.102:0/34532773 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c00bb20 con 0x7f6d940c41b0 2023-03-03T12:44:49.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.829+0000 7f6d8bfff700 1 -- 172.21.15.102:0/34532773 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c012430 con 0x7f6d940c41b0 2023-03-03T12:44:49.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 -- 172.21.15.102:0/34532773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940c41b0 msgr2=0x7f6d940cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:49.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/34532773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940c41b0 0x7f6d940cf0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009d90 tx=0x7f6d8c00b330 comp rx=0 tx=0).stop 2023-03-03T12:44:49.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 -- 172.21.15.102:0/34532773 shutdown_connections 2023-03-03T12:44:49.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/34532773 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940c41b0 0x7f6d940cf0d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:49.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/34532773 >> v2:172.21.15.172:3300/0 conn(0x7f6d940b3970 0x7f6d940c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:49.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 -- 172.21.15.102:0/34532773 >> 172.21.15.102:0/34532773 conn(0x7f6d9408b1c0 msgr2=0x7f6d94071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:44:49.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 -- 172.21.15.102:0/34532773 shutdown_connections 2023-03-03T12:44:49.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.830+0000 7f6d9a79d700 1 -- 172.21.15.102:0/34532773 wait complete. 2023-03-03T12:44:49.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.832+0000 7f6d9a79d700 1 Processor -- start 2023-03-03T12:44:49.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.833+0000 7f6d9a79d700 1 -- start start 2023-03-03T12:44:49.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9a79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d940b3970 0x7f6d9414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:49.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9a79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d940c41b0 0x7f6d94159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:49.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9a79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940ced00 0x7f6d941517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:49.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9a79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d94152360 con 0x7f6d940c41b0 2023-03-03T12:44:49.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9a79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9414ef40 con 0x7f6d940b3970 2023-03-03T12:44:49.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9a79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940c8930 con 0x7f6d940ced00 2023-03-03T12:44:49.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d98f9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d940c41b0 0x7f6d94159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:49.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9979b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d940b3970 0x7f6d9414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:49.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d98f9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d940c41b0 0x7f6d94159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49932/0 (socket says 172.21.15.102:49932) 2023-03-03T12:44:49.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d98f9a700 1 -- 172.21.15.102:0/3200778078 learned_addr learned my addr 172.21.15.102:0/3200778078 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:44:49.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d99f9c700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940ced00 0x7f6d941517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:49.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9979b700 1 -- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940ced00 msgr2=0x7f6d941517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:49.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9979b700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940ced00 0x7f6d941517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:49.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9979b700 1 -- 172.21.15.102:0/3200778078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d940c41b0 msgr2=0x7f6d94159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:49.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9979b700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d940c41b0 0x7f6d94159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:49.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d9979b700 1 -- 172.21.15.102:0/3200778078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c009a70 con 0x7f6d940b3970 2023-03-03T12:44:49.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.834+0000 7f6d98f9a700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d940c41b0 0x7f6d94159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:44:49.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.835+0000 7f6d9979b700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d940b3970 0x7f6d9414e900 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6d9000c490 tx=0x7f6d9000c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:49.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.835+0000 7f6d89ffb700 1 -- 172.21.15.102:0/3200778078 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d90005d70 con 0x7f6d940b3970 2023-03-03T12:44:49.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.835+0000 7f6d89ffb700 1 -- 172.21.15.102:0/3200778078 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d900070d0 con 0x7f6d940b3970 2023-03-03T12:44:49.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.835+0000 7f6d89ffb700 1 -- 172.21.15.102:0/3200778078 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d90007440 con 0x7f6d940b3970 2023-03-03T12:44:49.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.835+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9406bf20 con 0x7f6d940b3970 2023-03-03T12:44:49.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.835+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9415ce80 con 0x7f6d940b3970 2023-03-03T12:44:49.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.836+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9406c200 con 0x7f6d940b3970 2023-03-03T12:44:49.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.837+0000 7f6d89ffb700 1 -- 172.21.15.102:0/3200778078 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6d90010070 con 0x7f6d940b3970 2023-03-03T12:44:49.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.839+0000 7f6d89ffb700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d780682f0 0x7f6d7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:49.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.839+0000 7f6d89ffb700 1 -- 172.21.15.102:0/3200778078 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6d90084be0 con 0x7f6d940b3970 2023-03-03T12:44:49.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.839+0000 7f6d98f9a700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d780682f0 0x7f6d7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:49.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.840+0000 7f6d98f9a700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d780682f0 0x7f6d7806a7a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6d940cd7c0 tx=0x7f6d84009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:49.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:49.843+0000 7f6d89ffb700 1 -- 172.21.15.102:0/3200778078 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6d9004a5e0 con 0x7f6d940b3970 2023-03-03T12:44:50.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.077+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d9404e6e0 con 0x7f6d940b3970 2023-03-03T12:44:50.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.078+0000 7f6d89ffb700 1 -- 172.21.15.102:0/3200778078 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6d900513a0 con 0x7f6d940b3970 2023-03-03T12:44:50.080 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:44:50.080 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:44:50.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.080+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d780682f0 msgr2=0x7f6d7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:50.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.080+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d780682f0 0x7f6d7806a7a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6d940cd7c0 tx=0x7f6d84009210 comp rx=0 tx=0).stop 2023-03-03T12:44:50.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.080+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d940b3970 msgr2=0x7f6d9414e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:50.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.080+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d940b3970 0x7f6d9414e900 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6d9000c490 tx=0x7f6d9000c7a0 comp rx=0 tx=0).stop 2023-03-03T12:44:50.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 shutdown_connections 2023-03-03T12:44:50.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d780682f0 0x7f6d7806a7a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:50.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d940ced00 0x7f6d941517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:50.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d940b3970 0x7f6d9414e900 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:50.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 --2- 172.21.15.102:0/3200778078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d940c41b0 0x7f6d94159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:50.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 >> 172.21.15.102:0/3200778078 conn(0x7f6d9408b1c0 msgr2=0x7f6d94071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:44:50.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 shutdown_connections 2023-03-03T12:44:50.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:44:50.081+0000 7f6d9a79d700 1 -- 172.21.15.102:0/3200778078 wait complete. 2023-03-03T12:44:50.084 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:44:50.110 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-03-03T12:45:03.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:03.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:34.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:34.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:04.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:04.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:07.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:46:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.363+0000 7f7ea55ff700 1 Processor -- start 2023-03-03T12:46:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.364+0000 7f7ea55ff700 1 -- start start 2023-03-03T12:46:07.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.365+0000 7f7ea55ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ea0153290 0x7f7ea0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:07.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.365+0000 7f7ea55ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ea00cd8e0 0x7f7ea00ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:07.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.365+0000 7f7ea55ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea0153660 con 0x7f7ea01593c0 2023-03-03T12:46:07.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.365+0000 7f7ea55ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea006c360 con 0x7f7ea00cd8e0 2023-03-03T12:46:07.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.365+0000 7f7ea55ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea015cea0 con 0x7f7ea0153290 2023-03-03T12:46:07.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.365+0000 7f7e9f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ea00cd8e0 0x7f7ea00ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:07.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.365+0000 7f7e9effd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ea0153290 0x7f7ea0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9e7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7ea01593c0 0x7f7ea00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37318/0 (socket says 172.21.15.102:37318) 2023-03-03T12:46:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9e7fc700 1 -- 172.21.15.102:0/4202909891 learned_addr learned my addr 172.21.15.102:0/4202909891 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:46:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9dffb700 1 -- 172.21.15.102:0/4202909891 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879949087 0 0) 0x7f7ea0153660 con 0x7f7ea01593c0 2023-03-03T12:46:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9f7fe700 1 -- 172.21.15.102:0/4202909891 >> v2:172.21.15.172:3300/0 conn(0x7f7ea0153290 msgr2=0x7f7ea0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:46:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9f7fe700 1 --2- 172.21.15.102:0/4202909891 >> v2:172.21.15.172:3300/0 conn(0x7f7ea0153290 0x7f7ea0158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9f7fe700 1 -- 172.21.15.102:0/4202909891 >> v1:172.21.15.172:6789/0 conn(0x7f7ea01593c0 legacy=0x7f7ea00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9f7fe700 1 -- 172.21.15.102:0/4202909891 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea006c7f0 con 0x7f7ea00cd8e0 2023-03-03T12:46:07.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.366+0000 7f7e9f7fe700 1 --2- 172.21.15.102:0/4202909891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea00cd8e0 0x7f7ea00ce530 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7ea008cc70 tx=0x7f7e9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da916f949a26f0c2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:07.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7e9dffb700 1 -- 172.21.15.102:0/4202909891 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800e070 con 0x7f7ea00cd8e0 2023-03-03T12:46:07.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7e9dffb700 1 -- 172.21.15.102:0/4202909891 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9800bb20 con 0x7f7ea00cd8e0 2023-03-03T12:46:07.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7e9dffb700 1 -- 172.21.15.102:0/4202909891 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98012430 con 0x7f7ea00cd8e0 2023-03-03T12:46:07.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 -- 172.21.15.102:0/4202909891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea00cd8e0 msgr2=0x7f7ea00ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:07.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/4202909891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea00cd8e0 0x7f7ea00ce530 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7ea008cc70 tx=0x7f7e9800b330 comp rx=0 tx=0).stop 2023-03-03T12:46:07.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 -- 172.21.15.102:0/4202909891 shutdown_connections 2023-03-03T12:46:07.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/4202909891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea00cd8e0 0x7f7ea00ce530 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/4202909891 >> v2:172.21.15.172:3300/0 conn(0x7f7ea0153290 0x7f7ea0158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 -- 172.21.15.102:0/4202909891 >> 172.21.15.102:0/4202909891 conn(0x7f7ea0065fb0 msgr2=0x7f7ea00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 -- 172.21.15.102:0/4202909891 shutdown_connections 2023-03-03T12:46:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.367+0000 7f7ea55ff700 1 -- 172.21.15.102:0/4202909891 wait complete. 2023-03-03T12:46:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.369+0000 7f7ea55ff700 1 Processor -- start 2023-03-03T12:46:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.369+0000 7f7ea55ff700 1 -- start start 2023-03-03T12:46:07.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.370+0000 7f7ea55ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ea00cd8e0 0x7f7ea00b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:07.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.370+0000 7f7ea55ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea0153290 0x7f7ea014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:07.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.370+0000 7f7ea55ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ea01593c0 0x7f7ea01480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:07.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.370+0000 7f7ea55ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea01e4f70 con 0x7f7ea0153290 2023-03-03T12:46:07.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.370+0000 7f7ea55ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea01e2660 con 0x7f7ea00cd8e0 2023-03-03T12:46:07.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.370+0000 7f7ea55ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea014cb20 con 0x7f7ea01593c0 2023-03-03T12:46:07.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea0153290 0x7f7ea014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea0153290 0x7f7ea014c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42364/0 (socket says 172.21.15.102:42364) 2023-03-03T12:46:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 -- 172.21.15.102:0/169503066 learned_addr learned my addr 172.21.15.102:0/169503066 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:46:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9effd700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ea00cd8e0 0x7f7ea00b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9f7fe700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ea01593c0 0x7f7ea01480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 -- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ea01593c0 msgr2=0x7f7ea01480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ea01593c0 0x7f7ea01480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 -- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ea00cd8e0 msgr2=0x7f7ea00b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:07.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ea00cd8e0 0x7f7ea00b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.371+0000 7f7e9e7fc700 1 -- 172.21.15.102:0/169503066 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e98009a70 con 0x7f7ea0153290 2023-03-03T12:46:07.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.372+0000 7f7e9e7fc700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea0153290 0x7f7ea014c5e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7e940011e0 tx=0x7f7e9400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:07.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.372+0000 7f7e877fe700 1 -- 172.21.15.102:0/169503066 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94004c80 con 0x7f7ea0153290 2023-03-03T12:46:07.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.372+0000 7f7e877fe700 1 -- 172.21.15.102:0/169503066 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e94004e00 con 0x7f7ea0153290 2023-03-03T12:46:07.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.372+0000 7f7e877fe700 1 -- 172.21.15.102:0/169503066 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94007310 con 0x7f7ea0153290 2023-03-03T12:46:07.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.372+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea00ccde0 con 0x7f7ea0153290 2023-03-03T12:46:07.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.372+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea015d160 con 0x7f7ea0153290 2023-03-03T12:46:07.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.373+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea00cd0c0 con 0x7f7ea0153290 2023-03-03T12:46:07.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.374+0000 7f7e877fe700 1 -- 172.21.15.102:0/169503066 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e9401a020 con 0x7f7ea0153290 2023-03-03T12:46:07.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.376+0000 7f7e877fe700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e8c0682f0 0x7f7e8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:07.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.376+0000 7f7e877fe700 1 -- 172.21.15.102:0/169503066 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7e94085970 con 0x7f7ea0153290 2023-03-03T12:46:07.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.376+0000 7f7e9effd700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e8c0682f0 0x7f7e8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:07.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.377+0000 7f7e9effd700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e8c0682f0 0x7f7e8c06a7a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009be0 tx=0x7f7e90009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:07.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.380+0000 7f7e877fe700 1 -- 172.21.15.102:0/169503066 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e94051f90 con 0x7f7ea0153290 2023-03-03T12:46:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.612+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ea01e27d0 con 0x7f7ea0153290 2023-03-03T12:46:07.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.612+0000 7f7e877fe700 1 -- 172.21.15.102:0/169503066 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7ea01e27d0 con 0x7f7ea0153290 2023-03-03T12:46:07.615 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:46:07.615 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:46:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e8c0682f0 msgr2=0x7f7e8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e8c0682f0 0x7f7e8c06a7a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009be0 tx=0x7f7e90009210 comp rx=0 tx=0).stop 2023-03-03T12:46:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea0153290 msgr2=0x7f7ea014c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea0153290 0x7f7ea014c5e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7e940011e0 tx=0x7f7e9400ca30 comp rx=0 tx=0).stop 2023-03-03T12:46:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 shutdown_connections 2023-03-03T12:46:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e8c0682f0 0x7f7e8c06a7a0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ea01593c0 0x7f7ea01480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ea00cd8e0 0x7f7ea00b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 --2- 172.21.15.102:0/169503066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ea0153290 0x7f7ea014c5e0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 >> 172.21.15.102:0/169503066 conn(0x7f7ea0065fb0 msgr2=0x7f7ea015ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 shutdown_connections 2023-03-03T12:46:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:46:07.615+0000 7f7ea55ff700 1 -- 172.21.15.102:0/169503066 wait complete. 2023-03-03T12:46:07.619 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:46:07.644 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-03-03T12:46:34.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:34.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:04.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:04.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:34.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:34.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:55.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:47:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.898+0000 7f25b09fe700 1 Processor -- start 2023-03-03T12:47:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.898+0000 7f25b09fe700 1 -- start start 2023-03-03T12:47:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25b09fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25ac0ced00 0x7f25ac0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25b09fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0b3970 0x7f25ac0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25b09fe700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0c3d20 con 0x7f25ac0cf0d0 2023-03-03T12:47:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25b09fe700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac0c4930 con 0x7f25ac0ced00 2023-03-03T12:47:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25b09fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac15cba0 con 0x7f25ac0b3970 2023-03-03T12:47:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25aa59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0b3970 0x7f25ac0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25aad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25ac0ced00 0x7f25ac0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25ab59e700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f25ac0cf0d0 0x7f25ac158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36154/0 (socket says 172.21.15.102:36154) 2023-03-03T12:47:55.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.900+0000 7f25ab59e700 1 -- 172.21.15.102:0/3448302155 learned_addr learned my addr 172.21.15.102:0/3448302155 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:47:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25aa59c700 1 -- 172.21.15.102:0/3448302155 >> v2:172.21.15.172:3300/0 conn(0x7f25ac0ced00 msgr2=0x7f25ac0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:47:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25aa59c700 1 --2- 172.21.15.102:0/3448302155 >> v2:172.21.15.172:3300/0 conn(0x7f25ac0ced00 0x7f25ac0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25aa59c700 1 -- 172.21.15.102:0/3448302155 >> v1:172.21.15.102:6789/0 conn(0x7f25ac0cf0d0 legacy=0x7f25ac158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25aa59c700 1 -- 172.21.15.102:0/3448302155 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac15ce80 con 0x7f25ac0b3970 2023-03-03T12:47:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25aa59c700 1 --2- 172.21.15.102:0/3448302155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0b3970 0x7f25ac0c41b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009840 tx=0x7f25a0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bde6da0d8652c062 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:47:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25a9d9b700 1 -- 172.21.15.102:0/3448302155 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0011830 con 0x7f25ac0b3970 2023-03-03T12:47:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25a9d9b700 1 -- 172.21.15.102:0/3448302155 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a0011990 con 0x7f25ac0b3970 2023-03-03T12:47:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.901+0000 7f25a9d9b700 1 -- 172.21.15.102:0/3448302155 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0011d20 con 0x7f25ac0b3970 2023-03-03T12:47:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 -- 172.21.15.102:0/3448302155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0b3970 msgr2=0x7f25ac0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 --2- 172.21.15.102:0/3448302155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0b3970 0x7f25ac0c41b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009840 tx=0x7f25a0009b50 comp rx=0 tx=0).stop 2023-03-03T12:47:55.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 -- 172.21.15.102:0/3448302155 shutdown_connections 2023-03-03T12:47:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 --2- 172.21.15.102:0/3448302155 >> v2:172.21.15.172:3300/0 conn(0x7f25ac0ced00 0x7f25ac0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 --2- 172.21.15.102:0/3448302155 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0b3970 0x7f25ac0c41b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 -- 172.21.15.102:0/3448302155 >> 172.21.15.102:0/3448302155 conn(0x7f25ac08b1c0 msgr2=0x7f25ac071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:47:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 -- 172.21.15.102:0/3448302155 shutdown_connections 2023-03-03T12:47:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.902+0000 7f25b09fe700 1 -- 172.21.15.102:0/3448302155 wait complete. 2023-03-03T12:47:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.904+0000 7f25b09fe700 1 Processor -- start 2023-03-03T12:47:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.904+0000 7f25b09fe700 1 -- start start 2023-03-03T12:47:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25b09fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0b3970 0x7f25ac1e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25b09fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25ac0ced00 0x7f25ac1e5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25b09fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0cf0d0 0x7f25ac0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:55.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25b09fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1e5680 con 0x7f25ac0b3970 2023-03-03T12:47:55.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25b09fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1e60a0 con 0x7f25ac0ced00 2023-03-03T12:47:55.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25b09fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac14e340 con 0x7f25ac0cf0d0 2023-03-03T12:47:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25aad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0b3970 0x7f25ac1e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25aa59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25ac0ced00 0x7f25ac1e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25ab59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0cf0d0 0x7f25ac0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25aad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0b3970 0x7f25ac1e5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47238/0 (socket says 172.21.15.102:47238) 2023-03-03T12:47:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.905+0000 7f25aad9d700 1 -- 172.21.15.102:0/767733896 learned_addr learned my addr 172.21.15.102:0/767733896 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:47:55.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.906+0000 7f25aa59c700 1 -- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0cf0d0 msgr2=0x7f25ac0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.906+0000 7f25aa59c700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0cf0d0 0x7f25ac0ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.906+0000 7f25aa59c700 1 -- 172.21.15.102:0/767733896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0b3970 msgr2=0x7f25ac1e5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.906+0000 7f25aa59c700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0b3970 0x7f25ac1e5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.906+0000 7f25aa59c700 1 -- 172.21.15.102:0/767733896 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a00094a0 con 0x7f25ac0ced00 2023-03-03T12:47:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.906+0000 7f25aa59c700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25ac0ced00 0x7f25ac1e5140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f25a0016000 tx=0x7f25a0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:47:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.906+0000 7f25937fe700 1 -- 172.21.15.102:0/767733896 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00036a0 con 0x7f25ac0ced00 2023-03-03T12:47:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.907+0000 7f25937fe700 1 -- 172.21.15.102:0/767733896 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a001e070 con 0x7f25ac0ced00 2023-03-03T12:47:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.907+0000 7f25937fe700 1 -- 172.21.15.102:0/767733896 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0003a40 con 0x7f25ac0ced00 2023-03-03T12:47:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.907+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac06c200 con 0x7f25ac0ced00 2023-03-03T12:47:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.907+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac06bf20 con 0x7f25ac0ced00 2023-03-03T12:47:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.908+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac0cf540 con 0x7f25ac0ced00 2023-03-03T12:47:55.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.909+0000 7f25937fe700 1 -- 172.21.15.102:0/767733896 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f25a000f050 con 0x7f25ac0ced00 2023-03-03T12:47:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.910+0000 7f25937fe700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f25980682f0 0x7f259806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.910+0000 7f25937fe700 1 -- 172.21.15.102:0/767733896 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f25a0086e00 con 0x7f25ac0ced00 2023-03-03T12:47:55.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.911+0000 7f25aad9d700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f25980682f0 0x7f259806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:55.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.911+0000 7f25aad9d700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f25980682f0 0x7f259806a7a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ca0 tx=0x7f259c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:47:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:55.915+0000 7f25937fe700 1 -- 172.21.15.102:0/767733896 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25a0053510 con 0x7f25ac0ced00 2023-03-03T12:47:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.151+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25ac04e6e0 con 0x7f25ac0ced00 2023-03-03T12:47:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.152+0000 7f25937fe700 1 -- 172.21.15.102:0/767733896 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f25a0020090 con 0x7f25ac0ced00 2023-03-03T12:47:56.154 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:47:56.154 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:47:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f25980682f0 msgr2=0x7f259806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f25980682f0 0x7f259806a7a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ca0 tx=0x7f259c009210 comp rx=0 tx=0).stop 2023-03-03T12:47:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25ac0ced00 msgr2=0x7f25ac1e5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25ac0ced00 0x7f25ac1e5140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f25a0016000 tx=0x7f25a0004750 comp rx=0 tx=0).stop 2023-03-03T12:47:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 shutdown_connections 2023-03-03T12:47:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f25980682f0 0x7f259806a7a0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25ac0cf0d0 0x7f25ac0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0b3970 0x7f25ac1e5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 --2- 172.21.15.102:0/767733896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25ac0ced00 0x7f25ac1e5140 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.154+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 >> 172.21.15.102:0/767733896 conn(0x7f25ac08b1c0 msgr2=0x7f25ac151610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:47:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.155+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 shutdown_connections 2023-03-03T12:47:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:47:56.155+0000 7f25b09fe700 1 -- 172.21.15.102:0/767733896 wait complete. 2023-03-03T12:47:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:47:56.183 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T12:48:04.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:04.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:35.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:35.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:35.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:48:35.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.448+0000 7f788519d700 1 Processor -- start 2023-03-03T12:48:35.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.448+0000 7f788519d700 1 -- start start 2023-03-03T12:48:35.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f788519d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800ced00 0x7f78800c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f788519d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78800c41b0 0x7f78800cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f788519d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78800c3d20 con 0x7f78800b3970 2023-03-03T12:48:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f788519d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78800c4930 con 0x7f78800c41b0 2023-03-03T12:48:35.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f788519d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78800cf610 con 0x7f78800ced00 2023-03-03T12:48:35.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f787f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f78800b3970 0x7f7880158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47848/0 (socket says 172.21.15.102:47848) 2023-03-03T12:48:35.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f787f7fe700 1 -- 172.21.15.102:0/3168795847 learned_addr learned my addr 172.21.15.102:0/3168795847 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:48:35.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f788499c700 1 --2- 172.21.15.102:0/3168795847 >> v2:172.21.15.172:3300/0 conn(0x7f78800c41b0 0x7f78800cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:35.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.450+0000 7f787ffff700 1 --2- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800ced00 0x7f78800c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f788499c700 1 -- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800ced00 msgr2=0x7f78800c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:48:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f787effd700 1 -- 172.21.15.102:0/3168795847 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832423376 0 0) 0x7f78800c3d20 con 0x7f78800b3970 2023-03-03T12:48:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f788499c700 1 --2- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800ced00 0x7f78800c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f788499c700 1 -- 172.21.15.102:0/3168795847 >> v1:172.21.15.102:6789/0 conn(0x7f78800b3970 legacy=0x7f7880158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f788499c700 1 -- 172.21.15.102:0/3168795847 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788015cba0 con 0x7f78800c41b0 2023-03-03T12:48:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f788499c700 1 --2- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800c41b0 0x7f78800cf0d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7878009dc0 tx=0x7f787800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5a2a12a197ed1d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:48:35.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f787effd700 1 -- 172.21.15.102:0/3168795847 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787800e070 con 0x7f78800c41b0 2023-03-03T12:48:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f787effd700 1 -- 172.21.15.102:0/3168795847 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787800bae0 con 0x7f78800c41b0 2023-03-03T12:48:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.451+0000 7f787effd700 1 -- 172.21.15.102:0/3168795847 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787800be90 con 0x7f78800c41b0 2023-03-03T12:48:35.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 -- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800c41b0 msgr2=0x7f78800cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 --2- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800c41b0 0x7f78800cf0d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7878009dc0 tx=0x7f787800b330 comp rx=0 tx=0).stop 2023-03-03T12:48:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 -- 172.21.15.102:0/3168795847 shutdown_connections 2023-03-03T12:48:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 --2- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800c41b0 0x7f78800cf0d0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 --2- 172.21.15.102:0/3168795847 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800ced00 0x7f78800c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 -- 172.21.15.102:0/3168795847 >> 172.21.15.102:0/3168795847 conn(0x7f788008b1c0 msgr2=0x7f7880071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:48:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 -- 172.21.15.102:0/3168795847 shutdown_connections 2023-03-03T12:48:35.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.452+0000 7f788519d700 1 -- 172.21.15.102:0/3168795847 wait complete. 2023-03-03T12:48:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.454+0000 7f788519d700 1 Processor -- start 2023-03-03T12:48:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.454+0000 7f788519d700 1 -- start start 2023-03-03T12:48:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.455+0000 7f788519d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800b3970 0x7f7880148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.455+0000 7f788519d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800c41b0 0x7f7880159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.455+0000 7f788519d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ced00 0x7f788015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:35.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f788519d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788014dab0 con 0x7f78800ced00 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f788519d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788014ea40 con 0x7f78800b3970 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f788519d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788014d110 con 0x7f78800c41b0 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800b3970 0x7f7880148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800b3970 0x7f7880148df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43938/0 (socket says 172.21.15.102:43938) 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 -- 172.21.15.102:0/1831874547 learned_addr learned my addr 172.21.15.102:0/1831874547 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787f7fe700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800c41b0 0x7f7880159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f788499c700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ced00 0x7f788015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:35.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 -- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800c41b0 msgr2=0x7f7880159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800c41b0 0x7f7880159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 -- 172.21.15.102:0/1831874547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ced00 msgr2=0x7f788015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ced00 0x7f788015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.456+0000 7f787ffff700 1 -- 172.21.15.102:0/1831874547 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878009a70 con 0x7f78800b3970 2023-03-03T12:48:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.457+0000 7f787ffff700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800b3970 0x7f7880148df0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f787000e9b0 tx=0x7f787000ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:48:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.457+0000 7f787cff9700 1 -- 172.21.15.102:0/1831874547 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870009710 con 0x7f78800b3970 2023-03-03T12:48:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.457+0000 7f787cff9700 1 -- 172.21.15.102:0/1831874547 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787000ee50 con 0x7f78800b3970 2023-03-03T12:48:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.457+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788006bf20 con 0x7f78800b3970 2023-03-03T12:48:35.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.457+0000 7f787cff9700 1 -- 172.21.15.102:0/1831874547 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000ca50 con 0x7f78800b3970 2023-03-03T12:48:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.457+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788015ce80 con 0x7f78800b3970 2023-03-03T12:48:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.458+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788006c200 con 0x7f78800b3970 2023-03-03T12:48:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.459+0000 7f787cff9700 1 -- 172.21.15.102:0/1831874547 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7870019030 con 0x7f78800b3970 2023-03-03T12:48:35.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.461+0000 7f787cff9700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f786c068320 0x7f786c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.461+0000 7f787cff9700 1 -- 172.21.15.102:0/1831874547 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7870085c20 con 0x7f78800b3970 2023-03-03T12:48:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.461+0000 7f787f7fe700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f786c068320 0x7f786c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:35.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.462+0000 7f787f7fe700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f786c068320 0x7f786c06a7d0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f78801e6250 tx=0x7f7874007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:48:35.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.465+0000 7f787cff9700 1 -- 172.21.15.102:0/1831874547 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7870052330 con 0x7f78800b3970 2023-03-03T12:48:35.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.701+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f788004e6e0 con 0x7f78800b3970 2023-03-03T12:48:35.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.702+0000 7f787cff9700 1 -- 172.21.15.102:0/1831874547 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7870055950 con 0x7f78800b3970 2023-03-03T12:48:35.703 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:48:35.704 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:48:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f786c068320 msgr2=0x7f786c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f786c068320 0x7f786c06a7d0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f78801e6250 tx=0x7f7874007e70 comp rx=0 tx=0).stop 2023-03-03T12:48:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800b3970 msgr2=0x7f7880148df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800b3970 0x7f7880148df0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f787000e9b0 tx=0x7f787000ecc0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 shutdown_connections 2023-03-03T12:48:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ced00 0x7f788015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78800b3970 0x7f7880148df0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f786c068320 0x7f786c06a7d0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 --2- 172.21.15.102:0/1831874547 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f78800c41b0 0x7f7880159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 >> 172.21.15.102:0/1831874547 conn(0x7f788008b1c0 msgr2=0x7f7880071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:48:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 shutdown_connections 2023-03-03T12:48:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:48:35.704+0000 7f788519d700 1 -- 172.21.15.102:0/1831874547 wait complete. 2023-03-03T12:48:35.708 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:48:35.732 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T12:49:05.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:05.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:31.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:49:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.996+0000 7f5d321ff700 1 Processor -- start 2023-03-03T12:49:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.997+0000 7f5d321ff700 1 -- start start 2023-03-03T12:49:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.998+0000 7f5d321ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5d2c0ca130 0x7f5d2c15c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.998+0000 7f5d321ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c0c9590 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.998+0000 7f5d321ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c159880 con 0x7f5d2c152e50 2023-03-03T12:49:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.998+0000 7f5d321ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c7ae0 con 0x7f5d2c0ca130 2023-03-03T12:49:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.999+0000 7f5d2affd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5d2c0ca130 0x7f5d2c15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.999+0000 7f5d2b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5d2c152e50 0x7f5d2c0c9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40026/0 (socket says 172.21.15.102:40026) 2023-03-03T12:49:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.999+0000 7f5d2b7fe700 1 -- 172.21.15.102:0/1792017683 learned_addr learned my addr 172.21.15.102:0/1792017683 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:49:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.999+0000 7f5d2a7fc700 1 -- 172.21.15.102:0/1792017683 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464433972 0 0) 0x7f5d2c159880 con 0x7f5d2c152e50 2023-03-03T12:49:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.999+0000 7f5d2affd700 1 -- 172.21.15.102:0/1792017683 >> v1:172.21.15.172:6789/0 conn(0x7f5d2c152e50 legacy=0x7f5d2c0c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.999+0000 7f5d2affd700 1 -- 172.21.15.102:0/1792017683 >> v1:172.21.15.102:6789/0 conn(0x7f5d2c0c8eb0 legacy=0x7f5d2c15c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:31.999+0000 7f5d2affd700 1 -- 172.21.15.102:0/1792017683 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c0c7dc0 con 0x7f5d2c0ca130 2023-03-03T12:49:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d2affd700 1 --2- 172.21.15.102:0/1792017683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0ca130 0x7f5d2c15c250 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009840 tx=0x7f5d20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c9d3554b7de63e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d2a7fc700 1 -- 172.21.15.102:0/1792017683 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d20011830 con 0x7f5d2c0ca130 2023-03-03T12:49:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d2a7fc700 1 -- 172.21.15.102:0/1792017683 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d20011990 con 0x7f5d2c0ca130 2023-03-03T12:49:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d2a7fc700 1 -- 172.21.15.102:0/1792017683 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d20011d20 con 0x7f5d2c0ca130 2023-03-03T12:49:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d321ff700 1 -- 172.21.15.102:0/1792017683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0ca130 msgr2=0x7f5d2c15c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d321ff700 1 --2- 172.21.15.102:0/1792017683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0ca130 0x7f5d2c15c250 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009840 tx=0x7f5d20009b50 comp rx=0 tx=0).stop 2023-03-03T12:49:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d321ff700 1 -- 172.21.15.102:0/1792017683 shutdown_connections 2023-03-03T12:49:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d321ff700 1 --2- 172.21.15.102:0/1792017683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0ca130 0x7f5d2c15c250 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.000+0000 7f5d321ff700 1 -- 172.21.15.102:0/1792017683 >> 172.21.15.102:0/1792017683 conn(0x7f5d2c0704c0 msgr2=0x7f5d2c0708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:49:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.001+0000 7f5d321ff700 1 -- 172.21.15.102:0/1792017683 shutdown_connections 2023-03-03T12:49:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.001+0000 7f5d321ff700 1 -- 172.21.15.102:0/1792017683 wait complete. 2023-03-03T12:49:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.003+0000 7f5d321ff700 1 Processor -- start 2023-03-03T12:49:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.003+0000 7f5d321ff700 1 -- start start 2023-03-03T12:49:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d321ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0c8eb0 0x7f5d2c1eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d321ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d2c0ca130 0x7f5d2c157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d321ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d2c152e50 0x7f5d2c14c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d321ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0cdc10 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d321ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c1ee370 con 0x7f5d2c0ca130 2023-03-03T12:49:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d321ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c1ee8f0 con 0x7f5d2c152e50 2023-03-03T12:49:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0c8eb0 0x7f5d2c1eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d2affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d2c0ca130 0x7f5d2c157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d2b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0c8eb0 0x7f5d2c1eed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44984/0 (socket says 172.21.15.102:44984) 2023-03-03T12:49:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d2b7fe700 1 -- 172.21.15.102:0/3168818187 learned_addr learned my addr 172.21.15.102:0/3168818187 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:49:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.004+0000 7f5d2bfff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d2c152e50 0x7f5d2c14c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:32.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d2b7fe700 1 -- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d2c152e50 msgr2=0x7f5d2c14c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:32.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d2b7fe700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d2c152e50 0x7f5d2c14c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:32.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d2b7fe700 1 -- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d2c0ca130 msgr2=0x7f5d2c157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:32.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d2b7fe700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d2c0ca130 0x7f5d2c157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:32.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d2b7fe700 1 -- 172.21.15.102:0/3168818187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d200094a0 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d2b7fe700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0c8eb0 0x7f5d2c1eed80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00c500 tx=0x7f5d1c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:32.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d13fff700 1 -- 172.21.15.102:0/3168818187 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c004c80 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d13fff700 1 -- 172.21.15.102:0/3168818187 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c004de0 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d13fff700 1 -- 172.21.15.102:0/3168818187 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c007310 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c159b60 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.005+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c0c80a0 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.007+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c0c7690 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.008+0000 7f5d13fff700 1 -- 172.21.15.102:0/3168818187 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d1c01a020 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.009+0000 7f5d13fff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d18068320 0x7f5d1806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.009+0000 7f5d13fff700 1 -- 172.21.15.102:0/3168818187 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5d1c085810 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.009+0000 7f5d2affd700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d18068320 0x7f5d1806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.010+0000 7f5d2affd700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d18068320 0x7f5d1806a7d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009840 tx=0x7f5d2001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:32.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.014+0000 7f5d13fff700 1 -- 172.21.15.102:0/3168818187 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d1c051e90 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.250+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d2c1eebd0 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.251+0000 7f5d13fff700 1 -- 172.21.15.102:0/3168818187 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5d1c0554b0 con 0x7f5d2c0c8eb0 2023-03-03T12:49:32.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:49:32.253 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:49:32.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d18068320 msgr2=0x7f5d1806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:32.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d18068320 0x7f5d1806a7d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009840 tx=0x7f5d2001c040 comp rx=0 tx=0).stop 2023-03-03T12:49:32.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0c8eb0 msgr2=0x7f5d2c1eed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0c8eb0 0x7f5d2c1eed80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00c500 tx=0x7f5d1c00c810 comp rx=0 tx=0).stop 2023-03-03T12:49:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 shutdown_connections 2023-03-03T12:49:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5d18068320 0x7f5d1806a7d0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5d2c152e50 0x7f5d2c14c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d2c0c8eb0 0x7f5d2c1eed80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 --2- 172.21.15.102:0/3168818187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d2c0ca130 0x7f5d2c157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.253+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 >> 172.21.15.102:0/3168818187 conn(0x7f5d2c0704c0 msgr2=0x7f5d2c14db70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:49:32.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.254+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 shutdown_connections 2023-03-03T12:49:32.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:49:32.254+0000 7f5d321ff700 1 -- 172.21.15.102:0/3168818187 wait complete. 2023-03-03T12:49:32.257 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:49:32.282 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2023-03-03T12:49:35.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:35.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T12:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:50:05.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:05.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:35.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:35.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:05.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:05.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:28.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:51:28.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.532+0000 7f848759e700 1 Processor -- start 2023-03-03T12:51:28.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.532+0000 7f848759e700 1 -- start start 2023-03-03T12:51:28.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848759e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84880ced00 0x7f84880c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:28.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848759e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f84880b3970 0x7f84880c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:28.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84880c3d20 con 0x7f84880cf0d0 2023-03-03T12:51:28.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848759e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84880c4930 con 0x7f84880ced00 2023-03-03T12:51:28.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848759e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84880cf570 con 0x7f84880b3970 2023-03-03T12:51:28.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f8485d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f84880b3970 0x7f84880c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:28.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84880ced00 0x7f84880c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:28.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84880ced00 0x7f84880c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57534/0 (socket says 172.21.15.102:57534) 2023-03-03T12:51:28.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f8485d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f84880b3970 0x7f84880c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54088/0 (socket says 172.21.15.102:54088) 2023-03-03T12:51:28.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.534+0000 7f848659c700 1 -- 172.21.15.102:0/381059159 learned_addr learned my addr 172.21.15.102:0/381059159 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:51:28.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848659c700 1 -- 172.21.15.102:0/381059159 >> v2:172.21.15.172:3300/0 conn(0x7f84880b3970 msgr2=0x7f84880c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:51:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848659c700 1 --2- 172.21.15.102:0/381059159 >> v2:172.21.15.172:3300/0 conn(0x7f84880b3970 0x7f84880c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848659c700 1 -- 172.21.15.102:0/381059159 >> v1:172.21.15.102:6789/0 conn(0x7f84880cf0d0 legacy=0x7f8488158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848659c700 1 -- 172.21.15.102:0/381059159 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848815cba0 con 0x7f84880ced00 2023-03-03T12:51:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848559a700 1 -- 172.21.15.102:0/381059159 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276403291 0 0) 0x7f84880c3d20 con 0x7f84880cf0d0 2023-03-03T12:51:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848659c700 1 --2- 172.21.15.102:0/381059159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880ced00 0x7f84880c5a90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f84880c4ba0 tx=0x7f8478009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1d24638a0ee188 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848559a700 1 -- 172.21.15.102:0/381059159 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478011830 con 0x7f84880ced00 2023-03-03T12:51:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848559a700 1 -- 172.21.15.102:0/381059159 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478011990 con 0x7f84880ced00 2023-03-03T12:51:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.535+0000 7f848559a700 1 -- 172.21.15.102:0/381059159 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478011d20 con 0x7f84880ced00 2023-03-03T12:51:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 -- 172.21.15.102:0/381059159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880ced00 msgr2=0x7f84880c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 --2- 172.21.15.102:0/381059159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880ced00 0x7f84880c5a90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f84880c4ba0 tx=0x7f8478009bd0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 -- 172.21.15.102:0/381059159 shutdown_connections 2023-03-03T12:51:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 --2- 172.21.15.102:0/381059159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880ced00 0x7f84880c5a90 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 --2- 172.21.15.102:0/381059159 >> v2:172.21.15.172:3300/0 conn(0x7f84880b3970 0x7f84880c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 -- 172.21.15.102:0/381059159 >> 172.21.15.102:0/381059159 conn(0x7f848808b1c0 msgr2=0x7f8488071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:51:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 -- 172.21.15.102:0/381059159 shutdown_connections 2023-03-03T12:51:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.536+0000 7f848759e700 1 -- 172.21.15.102:0/381059159 wait complete. 2023-03-03T12:51:28.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.538+0000 7f848759e700 1 Processor -- start 2023-03-03T12:51:28.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.539+0000 7f848759e700 1 -- start start 2023-03-03T12:51:28.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.539+0000 7f848759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880b3970 0x7f84881f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.539+0000 7f848759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84880ced00 0x7f848815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.539+0000 7f848759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84880cf0d0 0x7f8488150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.539+0000 7f848759e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84881518e0 con 0x7f84880b3970 2023-03-03T12:51:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.539+0000 7f848759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488151490 con 0x7f84880ced00 2023-03-03T12:51:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.539+0000 7f848759e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848815c3d0 con 0x7f84880cf0d0 2023-03-03T12:51:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f848659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880b3970 0x7f84881f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84880ced00 0x7f848815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84880ced00 0x7f848815a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54090/0 (socket says 172.21.15.102:54090) 2023-03-03T12:51:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8486d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84880cf0d0 0x7f8488150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 -- 172.21.15.102:0/215505906 learned_addr learned my addr 172.21.15.102:0/215505906 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:51:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f848659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880b3970 0x7f84881f18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57550/0 (socket says 172.21.15.102:57550) 2023-03-03T12:51:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 -- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84880cf0d0 msgr2=0x7f8488150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84880cf0d0 0x7f8488150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 -- 172.21.15.102:0/215505906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880b3970 msgr2=0x7f84881f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880b3970 0x7f84881f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8485d9b700 1 -- 172.21.15.102:0/215505906 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8478009550 con 0x7f84880ced00 2023-03-03T12:51:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f848659c700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880b3970 0x7f84881f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:51:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.540+0000 7f8486d9d700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84880cf0d0 0x7f8488150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:51:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.541+0000 7f8485d9b700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84880ced00 0x7f848815a7b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f847c00e840 tx=0x7f847c00eb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:28.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.541+0000 7f8476ffd700 1 -- 172.21.15.102:0/215505906 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c009590 con 0x7f84880ced00 2023-03-03T12:51:28.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.541+0000 7f8476ffd700 1 -- 172.21.15.102:0/215505906 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c013680 con 0x7f84880ced00 2023-03-03T12:51:28.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.541+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848806bf20 con 0x7f84880ced00 2023-03-03T12:51:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.541+0000 7f8476ffd700 1 -- 172.21.15.102:0/215505906 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c013a10 con 0x7f84880ced00 2023-03-03T12:51:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.541+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848815ce80 con 0x7f84880ced00 2023-03-03T12:51:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.542+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848806c200 con 0x7f84880ced00 2023-03-03T12:51:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.544+0000 7f8476ffd700 1 -- 172.21.15.102:0/215505906 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f847c019030 con 0x7f84880ced00 2023-03-03T12:51:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.545+0000 7f8476ffd700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84680682f0 0x7f846806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.545+0000 7f8476ffd700 1 -- 172.21.15.102:0/215505906 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f847c085aa0 con 0x7f84880ced00 2023-03-03T12:51:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.545+0000 7f848659c700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84680682f0 0x7f846806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.546+0000 7f848659c700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84680682f0 0x7f846806a7a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8478014040 tx=0x7f8478008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.549+0000 7f8476ffd700 1 -- 172.21.15.102:0/215505906 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f847c0521b0 con 0x7f84880ced00 2023-03-03T12:51:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.781+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f848804e6e0 con 0x7f84880ced00 2023-03-03T12:51:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.783+0000 7f8476ffd700 1 -- 172.21.15.102:0/215505906 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f847c0557d0 con 0x7f84880ced00 2023-03-03T12:51:28.786 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:51:28.786 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:51:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84680682f0 msgr2=0x7f846806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84680682f0 0x7f846806a7a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8478014040 tx=0x7f8478008030 comp rx=0 tx=0).stop 2023-03-03T12:51:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84880ced00 msgr2=0x7f848815a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84880ced00 0x7f848815a7b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f847c00e840 tx=0x7f847c00eb50 comp rx=0 tx=0).stop 2023-03-03T12:51:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 shutdown_connections 2023-03-03T12:51:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84680682f0 0x7f846806a7a0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f84880cf0d0 0x7f8488150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84880b3970 0x7f84881f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 --2- 172.21.15.102:0/215505906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84880ced00 0x7f848815a7b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 >> 172.21.15.102:0/215505906 conn(0x7f848808b1c0 msgr2=0x7f84880ca0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:51:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 shutdown_connections 2023-03-03T12:51:28.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:51:28.786+0000 7f848759e700 1 -- 172.21.15.102:0/215505906 wait complete. 2023-03-03T12:51:28.791 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:51:28.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T12:51:36.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:36.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:06.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:06.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:35.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:52:36.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.080+0000 7f235f7ff700 1 Processor -- start 2023-03-03T12:52:36.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.080+0000 7f235f7ff700 1 -- start start 2023-03-03T12:52:36.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.081+0000 7f235f7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f2358158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:36.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.081+0000 7f235f7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f23580cd8e0 0x7f23580ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:36.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.081+0000 7f235f7ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2358153660 con 0x7f23581593c0 2023-03-03T12:52:36.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.081+0000 7f235f7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235806c360 con 0x7f23580cd8e0 2023-03-03T12:52:36.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.081+0000 7f235f7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235815cea0 con 0x7f2358153290 2023-03-03T12:52:36.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.081+0000 7f235cd9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f23581593c0 0x7f23580cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33194/0 (socket says 172.21.15.102:33194) 2023-03-03T12:52:36.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235cd9a700 1 -- 172.21.15.102:0/2683752270 learned_addr learned my addr 172.21.15.102:0/2683752270 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:52:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235d59b700 1 --2- 172.21.15.102:0/2683752270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f2358158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235dd9c700 1 --2- 172.21.15.102:0/2683752270 >> v2:172.21.15.102:3300/0 conn(0x7f23580cd8e0 0x7f23580ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235dd9c700 1 -- 172.21.15.102:0/2683752270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 msgr2=0x7f2358158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:52:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235dd9c700 1 --2- 172.21.15.102:0/2683752270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f2358158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f234ffff700 1 -- 172.21.15.102:0/2683752270 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759614109 0 0) 0x7f2358153660 con 0x7f23581593c0 2023-03-03T12:52:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235dd9c700 1 -- 172.21.15.102:0/2683752270 >> v1:172.21.15.102:6789/0 conn(0x7f23581593c0 legacy=0x7f23580cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235dd9c700 1 -- 172.21.15.102:0/2683752270 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235806c7f0 con 0x7f23580cd8e0 2023-03-03T12:52:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235d59b700 1 --2- 172.21.15.102:0/2683752270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f2358158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:52:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.082+0000 7f235dd9c700 1 --2- 172.21.15.102:0/2683752270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580cd8e0 0x7f23580ce530 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2350009d90 tx=0x7f235000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba32b5f645c888f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f234ffff700 1 -- 172.21.15.102:0/2683752270 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000e070 con 0x7f23580cd8e0 2023-03-03T12:52:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f234ffff700 1 -- 172.21.15.102:0/2683752270 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235000bb20 con 0x7f23580cd8e0 2023-03-03T12:52:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f234ffff700 1 -- 172.21.15.102:0/2683752270 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350012430 con 0x7f23580cd8e0 2023-03-03T12:52:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f235f7ff700 1 -- 172.21.15.102:0/2683752270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580cd8e0 msgr2=0x7f23580ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f235f7ff700 1 --2- 172.21.15.102:0/2683752270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580cd8e0 0x7f23580ce530 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2350009d90 tx=0x7f235000b330 comp rx=0 tx=0).stop 2023-03-03T12:52:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f235f7ff700 1 -- 172.21.15.102:0/2683752270 shutdown_connections 2023-03-03T12:52:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f235f7ff700 1 --2- 172.21.15.102:0/2683752270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f2358158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f235f7ff700 1 --2- 172.21.15.102:0/2683752270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580cd8e0 0x7f23580ce530 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f235f7ff700 1 -- 172.21.15.102:0/2683752270 >> 172.21.15.102:0/2683752270 conn(0x7f2358065fb0 msgr2=0x7f23580663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:52:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.083+0000 7f235f7ff700 1 -- 172.21.15.102:0/2683752270 shutdown_connections 2023-03-03T12:52:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.084+0000 7f235f7ff700 1 -- 172.21.15.102:0/2683752270 wait complete. 2023-03-03T12:52:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.085+0000 7f235f7ff700 1 Processor -- start 2023-03-03T12:52:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.086+0000 7f235f7ff700 1 -- start start 2023-03-03T12:52:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235f7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23580cd8e0 0x7f23580c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235f7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f23581e5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235f7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23581593c0 0x7f235814f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235f7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580c8670 con 0x7f23581593c0 2023-03-03T12:52:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235f7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580c7090 con 0x7f23580cd8e0 2023-03-03T12:52:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235f7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2358156ca0 con 0x7f2358153290 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235dd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23581593c0 0x7f235814f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235cd9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f23581e5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235d59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23580cd8e0 0x7f23580c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235cd9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f23581e5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45180/0 (socket says 172.21.15.102:45180) 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235cd9a700 1 -- 172.21.15.102:0/3527414807 learned_addr learned my addr 172.21.15.102:0/3527414807 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.087+0000 7f235d59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23580cd8e0 0x7f23580c6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:41794/0 (socket says 172.21.15.102:41794) 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235cd9a700 1 -- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23580cd8e0 msgr2=0x7f23580c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235cd9a700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23580cd8e0 0x7f23580c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235cd9a700 1 -- 172.21.15.102:0/3527414807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23581593c0 msgr2=0x7f235814f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:36.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235cd9a700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23581593c0 0x7f235814f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235cd9a700 1 -- 172.21.15.102:0/3527414807 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23540094a0 con 0x7f2358153290 2023-03-03T12:52:36.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235d59b700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23580cd8e0 0x7f23580c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:52:36.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235dd9c700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23581593c0 0x7f235814f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:52:36.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235cd9a700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f23581e5560 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f234800c750 tx=0x7f234800ca60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:36.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f234dffb700 1 -- 172.21.15.102:0/3527414807 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348004c80 con 0x7f2358153290 2023-03-03T12:52:36.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f234dffb700 1 -- 172.21.15.102:0/3527414807 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2348004e00 con 0x7f2358153290 2023-03-03T12:52:36.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f234dffb700 1 -- 172.21.15.102:0/3527414807 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348007310 con 0x7f2358153290 2023-03-03T12:52:36.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2350009a70 con 0x7f2358153290 2023-03-03T12:52:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.088+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23580ccde0 con 0x7f2358153290 2023-03-03T12:52:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.089+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235815d160 con 0x7f2358153290 2023-03-03T12:52:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.091+0000 7f234dffb700 1 -- 172.21.15.102:0/3527414807 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f234801a020 con 0x7f2358153290 2023-03-03T12:52:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.092+0000 7f234dffb700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f233c0682f0 0x7f233c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.092+0000 7f234dffb700 1 -- 172.21.15.102:0/3527414807 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f23480858f0 con 0x7f2358153290 2023-03-03T12:52:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.092+0000 7f235d59b700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f233c0682f0 0x7f233c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.093+0000 7f235d59b700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f233c0682f0 0x7f233c06a7a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2354011380 tx=0x7f2354009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:36.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.096+0000 7f234dffb700 1 -- 172.21.15.102:0/3527414807 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2348052050 con 0x7f2358153290 2023-03-03T12:52:36.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.330+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23580cd0c0 con 0x7f2358153290 2023-03-03T12:52:36.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.331+0000 7f234dffb700 1 -- 172.21.15.102:0/3527414807 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2348055670 con 0x7f2358153290 2023-03-03T12:52:36.332 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:52:36.333 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:52:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f233c0682f0 msgr2=0x7f233c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f233c0682f0 0x7f233c06a7a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2354011380 tx=0x7f2354009210 comp rx=0 tx=0).stop 2023-03-03T12:52:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 msgr2=0x7f23581e5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f23581e5560 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f234800c750 tx=0x7f234800ca60 comp rx=0 tx=0).stop 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 shutdown_connections 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23581593c0 0x7f235814f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23580cd8e0 0x7f23580c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f233c0682f0 0x7f233c06a7a0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 --2- 172.21.15.102:0/3527414807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2358153290 0x7f23581e5560 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 >> 172.21.15.102:0/3527414807 conn(0x7f2358065fb0 msgr2=0x7f23580cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.333+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 shutdown_connections 2023-03-03T12:52:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:52:36.334+0000 7f235f7ff700 1 -- 172.21.15.102:0/3527414807 wait complete. 2023-03-03T12:52:36.337 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:52:36.363 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-03-03T12:52:36.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:36.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:06.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:06.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:23.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:53:23.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.618+0000 7f79099ff700 1 Processor -- start 2023-03-03T12:53:23.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.619+0000 7f79099ff700 1 -- start start 2023-03-03T12:53:23.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.620+0000 7f79099ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7904153290 0x7f7904158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:23.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.620+0000 7f79099ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7904153660 con 0x7f79040cd8e0 2023-03-03T12:53:23.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.620+0000 7f79099ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790406c360 con 0x7f79041593c0 2023-03-03T12:53:23.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.620+0000 7f79099ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79040cec40 con 0x7f7904153290 2023-03-03T12:53:23.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.620+0000 7f7902ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7904153290 0x7f7904158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:23.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.620+0000 7f79027fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f79041593c0 0x7f79040cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36972/0 (socket says 172.21.15.102:36972) 2023-03-03T12:53:23.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.620+0000 7f79027fc700 1 -- 172.21.15.102:0/3930562051 learned_addr learned my addr 172.21.15.102:0/3930562051 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:53:23.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7902ffd700 1 -- 172.21.15.102:0/3930562051 >> v1:172.21.15.172:6789/0 conn(0x7f79041593c0 legacy=0x7f79040cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:23.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7902ffd700 1 -- 172.21.15.102:0/3930562051 >> v1:172.21.15.102:6789/0 conn(0x7f79040cd8e0 legacy=0x7f79040ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:23.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7902ffd700 1 -- 172.21.15.102:0/3930562051 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790415cea0 con 0x7f7904153290 2023-03-03T12:53:23.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7901ffb700 1 -- 172.21.15.102:0/3930562051 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927467355 0 0) 0x7f790406c360 con 0x7f79041593c0 2023-03-03T12:53:23.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7902ffd700 1 --2- 172.21.15.102:0/3930562051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7904153290 0x7f7904158e80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009840 tx=0x7f78f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95c765cad4bcb2dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:23.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7901ffb700 1 -- 172.21.15.102:0/3930562051 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4011830 con 0x7f7904153290 2023-03-03T12:53:23.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7901ffb700 1 -- 172.21.15.102:0/3930562051 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f4011990 con 0x7f7904153290 2023-03-03T12:53:23.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.621+0000 7f7901ffb700 1 -- 172.21.15.102:0/3930562051 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4011d20 con 0x7f7904153290 2023-03-03T12:53:23.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.622+0000 7f79099ff700 1 -- 172.21.15.102:0/3930562051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7904153290 msgr2=0x7f7904158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:23.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.622+0000 7f79099ff700 1 --2- 172.21.15.102:0/3930562051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7904153290 0x7f7904158e80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009840 tx=0x7f78f4009b50 comp rx=0 tx=0).stop 2023-03-03T12:53:23.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.622+0000 7f79099ff700 1 -- 172.21.15.102:0/3930562051 shutdown_connections 2023-03-03T12:53:23.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.622+0000 7f79099ff700 1 --2- 172.21.15.102:0/3930562051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7904153290 0x7f7904158e80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:23.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.622+0000 7f79099ff700 1 -- 172.21.15.102:0/3930562051 >> 172.21.15.102:0/3930562051 conn(0x7f7904065fb0 msgr2=0x7f79040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:53:23.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.622+0000 7f79099ff700 1 -- 172.21.15.102:0/3930562051 shutdown_connections 2023-03-03T12:53:23.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.622+0000 7f79099ff700 1 -- 172.21.15.102:0/3930562051 wait complete. 2023-03-03T12:53:23.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.624+0000 7f79099ff700 1 Processor -- start 2023-03-03T12:53:23.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.624+0000 7f79099ff700 1 -- start start 2023-03-03T12:53:23.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.625+0000 7f79099ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd8e0 0x7f79041502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:23.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.625+0000 7f79099ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7904153290 0x7f79041e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:23.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.625+0000 7f79099ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79041593c0 0x7f79040c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:23.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.625+0000 7f79099ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79041e6430 con 0x7f79040cd8e0 2023-03-03T12:53:23.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.625+0000 7f79099ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7904150010 con 0x7f79041593c0 2023-03-03T12:53:23.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.625+0000 7f79099ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790414fa00 con 0x7f7904153290 2023-03-03T12:53:23.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd8e0 0x7f79041502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:23.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd8e0 0x7f79041502f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60114/0 (socket says 172.21.15.102:60114) 2023-03-03T12:53:23.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 -- 172.21.15.102:0/4050501541 learned_addr learned my addr 172.21.15.102:0/4050501541 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:53:23.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f79027fc700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7904153290 0x7f79041e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:23.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f79037fe700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79041593c0 0x7f79040c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:23.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 -- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7904153290 msgr2=0x7f79041e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:23.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7904153290 0x7f79041e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 -- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79041593c0 msgr2=0x7f79040c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79041593c0 0x7f79040c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.626+0000 7f7902ffd700 1 -- 172.21.15.102:0/4050501541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f40094a0 con 0x7f79040cd8e0 2023-03-03T12:53:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.627+0000 7f7902ffd700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd8e0 0x7f79041502f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f78f4014040 tx=0x7f78f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.627+0000 7f78ebfff700 1 -- 172.21.15.102:0/4050501541 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f401a7e0 con 0x7f79040cd8e0 2023-03-03T12:53:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.627+0000 7f78ebfff700 1 -- 172.21.15.102:0/4050501541 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f401a940 con 0x7f79040cd8e0 2023-03-03T12:53:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.627+0000 7f78ebfff700 1 -- 172.21.15.102:0/4050501541 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f401acb0 con 0x7f79040cd8e0 2023-03-03T12:53:23.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.627+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790415d160 con 0x7f79040cd8e0 2023-03-03T12:53:23.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.627+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790406c7f0 con 0x7f79040cd8e0 2023-03-03T12:53:23.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.628+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79040ccde0 con 0x7f79040cd8e0 2023-03-03T12:53:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.629+0000 7f78ebfff700 1 -- 172.21.15.102:0/4050501541 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f78f402b030 con 0x7f79040cd8e0 2023-03-03T12:53:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.630+0000 7f78ebfff700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f00682f0 0x7f78f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.631+0000 7f78ebfff700 1 -- 172.21.15.102:0/4050501541 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f78f4086fc0 con 0x7f79040cd8e0 2023-03-03T12:53:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.631+0000 7f79027fc700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f00682f0 0x7f78f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.631+0000 7f79027fc700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f00682f0 0x7f78f006a7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f78f80011e0 tx=0x7f78f8001130 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:23.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.635+0000 7f78ebfff700 1 -- 172.21.15.102:0/4050501541 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78f4053780 con 0x7f79040cd8e0 2023-03-03T12:53:23.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.871+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79040c67e0 con 0x7f79040cd8e0 2023-03-03T12:53:23.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.872+0000 7f78ebfff700 1 -- 172.21.15.102:0/4050501541 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f78f4029090 con 0x7f79040cd8e0 2023-03-03T12:53:23.874 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:53:23.874 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:53:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f00682f0 msgr2=0x7f78f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f00682f0 0x7f78f006a7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f78f80011e0 tx=0x7f78f8001130 comp rx=0 tx=0).stop 2023-03-03T12:53:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd8e0 msgr2=0x7f79041502f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd8e0 0x7f79041502f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f78f4014040 tx=0x7f78f4004750 comp rx=0 tx=0).stop 2023-03-03T12:53:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 shutdown_connections 2023-03-03T12:53:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79041593c0 0x7f79040c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd8e0 0x7f79041502f0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f78f00682f0 0x7f78f006a7a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 --2- 172.21.15.102:0/4050501541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7904153290 0x7f79041e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 >> 172.21.15.102:0/4050501541 conn(0x7f7904065fb0 msgr2=0x7f7904073680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:53:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 shutdown_connections 2023-03-03T12:53:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:53:23.875+0000 7f79099ff700 1 -- 172.21.15.102:0/4050501541 wait complete. 2023-03-03T12:53:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:53:23.904 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T12:53:36.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:36.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:06.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:06.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:36.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:36.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:07.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:07.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:17.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:55:18.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.151+0000 7f93657ff700 1 Processor -- start 2023-03-03T12:55:18.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.152+0000 7f93657ff700 1 -- start start 2023-03-03T12:55:18.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f93657ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9360153260 0x7f93600ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:18.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f93657ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93600c9cb0 0x7f93600cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:18.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f93657ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93600cac20 0x7f93600cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:18.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f93657ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93600cfe70 con 0x7f93600cac20 2023-03-03T12:55:18.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f93657ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9360158d70 con 0x7f93600c9cb0 2023-03-03T12:55:18.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f93657ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93600cf940 con 0x7f9360153260 2023-03-03T12:55:18.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f935e7fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93600c9cb0 0x7f93600cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f935effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9360153260 0x7f93600ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f935f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93600cac20 0x7f93600cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f935e7fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f93600c9cb0 0x7f93600cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53748/0 (socket says 172.21.15.102:53748) 2023-03-03T12:55:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f935e7fc700 1 -- 172.21.15.102:0/975035241 learned_addr learned my addr 172.21.15.102:0/975035241 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:55:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.153+0000 7f935effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9360153260 0x7f93600ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41252/0 (socket says 172.21.15.102:41252) 2023-03-03T12:55:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935e7fc700 1 -- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9360153260 msgr2=0x7f93600ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:55:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935e7fc700 1 --2- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9360153260 0x7f93600ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935e7fc700 1 -- 172.21.15.102:0/975035241 >> v2:172.21.15.102:3300/0 conn(0x7f93600cac20 msgr2=0x7f93600cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:55:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935e7fc700 1 --2- 172.21.15.102:0/975035241 >> v2:172.21.15.102:3300/0 conn(0x7f93600cac20 0x7f93600cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935e7fc700 1 -- 172.21.15.102:0/975035241 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93600cfb90 con 0x7f93600c9cb0 2023-03-03T12:55:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935effd700 1 --2- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9360153260 0x7f93600ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:55:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935e7fc700 1 --2- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93600c9cb0 0x7f93600cb880 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f936006a590 tx=0x7f9354009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea5c2ba568fe10df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935dffb700 1 -- 172.21.15.102:0/975035241 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354011830 con 0x7f93600c9cb0 2023-03-03T12:55:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935dffb700 1 -- 172.21.15.102:0/975035241 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9354011990 con 0x7f93600c9cb0 2023-03-03T12:55:18.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.154+0000 7f935dffb700 1 -- 172.21.15.102:0/975035241 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354011d20 con 0x7f93600c9cb0 2023-03-03T12:55:18.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 -- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93600c9cb0 msgr2=0x7f93600cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:18.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 --2- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93600c9cb0 0x7f93600cb880 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f936006a590 tx=0x7f9354009bd0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 -- 172.21.15.102:0/975035241 shutdown_connections 2023-03-03T12:55:18.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 --2- 172.21.15.102:0/975035241 >> v2:172.21.15.102:3300/0 conn(0x7f93600cac20 0x7f93600cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 --2- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9360153260 0x7f93600ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 --2- 172.21.15.102:0/975035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93600c9cb0 0x7f93600cb880 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 -- 172.21.15.102:0/975035241 >> 172.21.15.102:0/975035241 conn(0x7f9360066c00 msgr2=0x7f93600ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:55:18.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 -- 172.21.15.102:0/975035241 shutdown_connections 2023-03-03T12:55:18.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.155+0000 7f93657ff700 1 -- 172.21.15.102:0/975035241 wait complete. 2023-03-03T12:55:18.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.157+0000 7f93657ff700 1 Processor -- start 2023-03-03T12:55:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.157+0000 7f93657ff700 1 -- start start 2023-03-03T12:55:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.158+0000 7f93657ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600cac20 0x7f93601ef920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.158+0000 7f93657ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9360153260 0x7f93600ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.158+0000 7f93657ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93601efe60 0x7f93600cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.158+0000 7f93657ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936014f9c0 con 0x7f93600cac20 2023-03-03T12:55:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.158+0000 7f93657ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93601507e0 con 0x7f9360153260 2023-03-03T12:55:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.158+0000 7f93657ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93601f1b70 con 0x7f93601efe60 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935e7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9360153260 0x7f93600ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93601efe60 0x7f93600cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600cac20 0x7f93601ef920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93601efe60 0x7f93600cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41266/0 (socket says 172.21.15.102:41266) 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 -- 172.21.15.102:0/4114806801 learned_addr learned my addr 172.21.15.102:0/4114806801 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935e7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9360153260 0x7f93600ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:53762/0 (socket says 172.21.15.102:53762) 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 -- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9360153260 msgr2=0x7f93600ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9360153260 0x7f93600ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 -- 172.21.15.102:0/4114806801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600cac20 msgr2=0x7f93601ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600cac20 0x7f93601ef920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 -- 172.21.15.102:0/4114806801 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500094a0 con 0x7f93601efe60 2023-03-03T12:55:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935effd700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600cac20 0x7f93601ef920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:55:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.159+0000 7f935f7fe700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93601efe60 0x7f93600cd1c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f935800ef00 tx=0x7f935800b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.160+0000 7f93477fe700 1 -- 172.21.15.102:0/4114806801 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358010040 con 0x7f93601efe60 2023-03-03T12:55:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.160+0000 7f93477fe700 1 -- 172.21.15.102:0/4114806801 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935800bc80 con 0x7f93601efe60 2023-03-03T12:55:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.160+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9354009550 con 0x7f93601efe60 2023-03-03T12:55:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.160+0000 7f93477fe700 1 -- 172.21.15.102:0/4114806801 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358009e40 con 0x7f93601efe60 2023-03-03T12:55:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.160+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9360159260 con 0x7f93601efe60 2023-03-03T12:55:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.161+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9360158f80 con 0x7f93601efe60 2023-03-03T12:55:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.162+0000 7f93477fe700 1 -- 172.21.15.102:0/4114806801 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9358014400 con 0x7f93601efe60 2023-03-03T12:55:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.163+0000 7f93477fe700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f934c0682f0 0x7f934c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.163+0000 7f93477fe700 1 -- 172.21.15.102:0/4114806801 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9358086340 con 0x7f93601efe60 2023-03-03T12:55:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.164+0000 7f935effd700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f934c0682f0 0x7f934c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.164+0000 7f935effd700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f934c0682f0 0x7f934c06a7a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f93500114b0 tx=0x7f9350009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.168+0000 7f93477fe700 1 -- 172.21.15.102:0/4114806801 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9358052a10 con 0x7f93601efe60 2023-03-03T12:55:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.407+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9360159540 con 0x7f93601efe60 2023-03-03T12:55:18.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.408+0000 7f93477fe700 1 -- 172.21.15.102:0/4114806801 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9358018020 con 0x7f93601efe60 2023-03-03T12:55:18.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:55:18.411 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f934c0682f0 msgr2=0x7f934c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f934c0682f0 0x7f934c06a7a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f93500114b0 tx=0x7f9350009210 comp rx=0 tx=0).stop 2023-03-03T12:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93601efe60 msgr2=0x7f93600cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93601efe60 0x7f93600cd1c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f935800ef00 tx=0x7f935800b390 comp rx=0 tx=0).stop 2023-03-03T12:55:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 shutdown_connections 2023-03-03T12:55:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f934c0682f0 0x7f934c06a7a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f93601efe60 0x7f93600cd1c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600cac20 0x7f93601ef920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 --2- 172.21.15.102:0/4114806801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9360153260 0x7f93600ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.410+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 >> 172.21.15.102:0/4114806801 conn(0x7f93600c9cb0 msgr2=0x7f9360151920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.411+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 shutdown_connections 2023-03-03T12:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:18.411+0000 7f93657ff700 1 -- 172.21.15.102:0/4114806801 wait complete. 2023-03-03T12:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:55:18.440 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T12:55:37.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:37.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:50.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:55:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.694+0000 7f4cdfdff700 1 Processor -- start 2023-03-03T12:55:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.694+0000 7f4cdfdff700 1 -- start start 2023-03-03T12:55:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.695+0000 7f4cdfdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4cd8153290 0x7f4cd8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cdfdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cdfdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd8153660 con 0x7f4cd81593c0 2023-03-03T12:55:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cdfdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd806c360 con 0x7f4cd8153290 2023-03-03T12:55:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cdfdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd815cea0 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cddb9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4cd8153290 0x7f4cd8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cddb9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4cd8153290 0x7f4cd8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35714/0 (socket says 172.21.15.102:35714) 2023-03-03T12:55:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cddb9b700 1 -- 172.21.15.102:0/71416972 learned_addr learned my addr 172.21.15.102:0/71416972 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:55:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cde39c700 1 --2- 172.21.15.102:0/71416972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cddb9b700 1 -- 172.21.15.102:0/71416972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 msgr2=0x7f4cd80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:55:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cddb9b700 1 --2- 172.21.15.102:0/71416972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cddb9b700 1 -- 172.21.15.102:0/71416972 >> v1:172.21.15.102:6789/0 conn(0x7f4cd81593c0 legacy=0x7f4cd80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.696+0000 7f4cddb9b700 1 -- 172.21.15.102:0/71416972 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd806c7f0 con 0x7f4cd8153290 2023-03-03T12:55:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.697+0000 7f4cddb9b700 1 --2- 172.21.15.102:0/71416972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd8153290 0x7f4cd8158e80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40098f0 tx=0x7f4cd4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb944b8b1baa1b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.697+0000 7f4cdcb99700 1 -- 172.21.15.102:0/71416972 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4011830 con 0x7f4cd8153290 2023-03-03T12:55:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.697+0000 7f4cdcb99700 1 -- 172.21.15.102:0/71416972 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd4011990 con 0x7f4cd8153290 2023-03-03T12:55:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.697+0000 7f4cdcb99700 1 -- 172.21.15.102:0/71416972 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4011d20 con 0x7f4cd8153290 2023-03-03T12:55:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 -- 172.21.15.102:0/71416972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd8153290 msgr2=0x7f4cd8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/71416972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd8153290 0x7f4cd8158e80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40098f0 tx=0x7f4cd4009c00 comp rx=0 tx=0).stop 2023-03-03T12:55:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 -- 172.21.15.102:0/71416972 shutdown_connections 2023-03-03T12:55:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/71416972 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/71416972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd8153290 0x7f4cd8158e80 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 -- 172.21.15.102:0/71416972 >> 172.21.15.102:0/71416972 conn(0x7f4cd8065fb0 msgr2=0x7f4cd80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:55:50.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 -- 172.21.15.102:0/71416972 shutdown_connections 2023-03-03T12:55:50.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.698+0000 7f4cdfdff700 1 -- 172.21.15.102:0/71416972 wait complete. 2023-03-03T12:55:50.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.700+0000 7f4cdfdff700 1 Processor -- start 2023-03-03T12:55:50.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.700+0000 7f4cdfdff700 1 -- start start 2023-03-03T12:55:50.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cdfdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:50.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cdfdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cd8153290 0x7f4cd8150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:50.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cdfdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd81593c0 0x7f4cd80c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cdfdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c44b0 con 0x7f4cd81593c0 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cdfdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c73f0 con 0x7f4cd8153290 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cdfdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd8150a40 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cddb9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cddb9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46686/0 (socket says 172.21.15.102:46686) 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cddb9b700 1 -- 172.21.15.102:0/1298415291 learned_addr learned my addr 172.21.15.102:0/1298415291 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.701+0000 7f4cdd39a700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cd8153290 0x7f4cd8150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cde39c700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd81593c0 0x7f4cd80c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:50.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cddb9b700 1 -- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cd8153290 msgr2=0x7f4cd8150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cddb9b700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cd8153290 0x7f4cd8150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cddb9b700 1 -- 172.21.15.102:0/1298415291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd81593c0 msgr2=0x7f4cd80c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cddb9b700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd81593c0 0x7f4cd80c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cddb9b700 1 -- 172.21.15.102:0/1298415291 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd4009550 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cddb9b700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80c3f70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4014040 tx=0x7f4cd4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.702+0000 7f4cce7fc700 1 -- 172.21.15.102:0/1298415291 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4004450 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.703+0000 7f4cce7fc700 1 -- 172.21.15.102:0/1298415291 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd40045b0 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.703+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd80ccde0 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.703+0000 7f4cce7fc700 1 -- 172.21.15.102:0/1298415291 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd40038e0 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.703+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd815d160 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.704+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd80cd0c0 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.705+0000 7f4cce7fc700 1 -- 172.21.15.102:0/1298415291 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4cd400f050 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.706+0000 7f4cce7fc700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.706+0000 7f4cce7fc700 1 -- 172.21.15.102:0/1298415291 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4cd4087090 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.707+0000 7f4cdd39a700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:50.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.707+0000 7f4cdd39a700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f4cd80cbb60 tx=0x7f4cc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:50.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.711+0000 7f4cce7fc700 1 -- 172.21.15.102:0/1298415291 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4cd40537a0 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.947+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd815a8e0 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.949+0000 7f4cce7fc700 1 -- 172.21.15.102:0/1298415291 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4cd4020090 con 0x7f4cd80cd8e0 2023-03-03T12:55:50.950 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:55:50.950 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:55:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4cbc0682f0 msgr2=0x7f4cbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f4cd80cbb60 tx=0x7f4cc8009040 comp rx=0 tx=0).stop 2023-03-03T12:55:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 msgr2=0x7f4cd80c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80c3f70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4014040 tx=0x7f4cd4004750 comp rx=0 tx=0).stop 2023-03-03T12:55:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 shutdown_connections 2023-03-03T12:55:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4cd80cd8e0 0x7f4cd80c3f70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cd81593c0 0x7f4cd80c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 --2- 172.21.15.102:0/1298415291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cd8153290 0x7f4cd8150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:50.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 >> 172.21.15.102:0/1298415291 conn(0x7f4cd8065fb0 msgr2=0x7f4cd80b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:55:50.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 shutdown_connections 2023-03-03T12:55:50.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:55:50.951+0000 7f4cdfdff700 1 -- 172.21.15.102:0/1298415291 wait complete. 2023-03-03T12:55:50.955 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:55:50.979 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-03-03T12:56:02.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:56:03.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.227+0000 7f72419ff700 1 Processor -- start 2023-03-03T12:56:03.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.227+0000 7f72419ff700 1 -- start start 2023-03-03T12:56:03.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f72419ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f723c153290 0x7f723c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:03.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f72419ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f723c1593c0 0x7f723c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:03.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f72419ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723c153660 con 0x7f723c0cd8e0 2023-03-03T12:56:03.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f72419ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723c06c360 con 0x7f723c1593c0 2023-03-03T12:56:03.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f72419ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723c15cea0 con 0x7f723c153290 2023-03-03T12:56:03.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f723affd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f723c153290 0x7f723c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:03.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f723b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f723c0cd8e0 0x7f723c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46926/0 (socket says 172.21.15.102:46926) 2023-03-03T12:56:03.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f723b7fe700 1 -- 172.21.15.102:0/1530073744 learned_addr learned my addr 172.21.15.102:0/1530073744 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:56:03.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.229+0000 7f723a7fc700 1 --2- 172.21.15.102:0/1530073744 >> v2:172.21.15.102:3300/0 conn(0x7f723c1593c0 0x7f723c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:03.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f723affd700 1 -- 172.21.15.102:0/1530073744 >> v2:172.21.15.102:3300/0 conn(0x7f723c1593c0 msgr2=0x7f723c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:56:03.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f723affd700 1 --2- 172.21.15.102:0/1530073744 >> v2:172.21.15.102:3300/0 conn(0x7f723c1593c0 0x7f723c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f723affd700 1 -- 172.21.15.102:0/1530073744 >> v1:172.21.15.172:6789/0 conn(0x7f723c0cd8e0 legacy=0x7f723c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:03.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f723affd700 1 -- 172.21.15.102:0/1530073744 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c06c7f0 con 0x7f723c153290 2023-03-03T12:56:03.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f7239ffb700 1 -- 172.21.15.102:0/1530073744 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919232350 0 0) 0x7f723c153660 con 0x7f723c0cd8e0 2023-03-03T12:56:03.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f723affd700 1 --2- 172.21.15.102:0/1530073744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 0x7f723c158e80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f723c08cc70 tx=0x7f722c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bca0866a2b3228f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:03.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f7239ffb700 1 -- 172.21.15.102:0/1530073744 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c011830 con 0x7f723c153290 2023-03-03T12:56:03.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f7239ffb700 1 -- 172.21.15.102:0/1530073744 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c011990 con 0x7f723c153290 2023-03-03T12:56:03.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.230+0000 7f7239ffb700 1 -- 172.21.15.102:0/1530073744 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c011d20 con 0x7f723c153290 2023-03-03T12:56:03.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 -- 172.21.15.102:0/1530073744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 msgr2=0x7f723c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:03.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 --2- 172.21.15.102:0/1530073744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 0x7f723c158e80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f723c08cc70 tx=0x7f722c009b20 comp rx=0 tx=0).stop 2023-03-03T12:56:03.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 -- 172.21.15.102:0/1530073744 shutdown_connections 2023-03-03T12:56:03.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 --2- 172.21.15.102:0/1530073744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 0x7f723c158e80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 --2- 172.21.15.102:0/1530073744 >> v2:172.21.15.102:3300/0 conn(0x7f723c1593c0 0x7f723c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 -- 172.21.15.102:0/1530073744 >> 172.21.15.102:0/1530073744 conn(0x7f723c065fb0 msgr2=0x7f723c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:03.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 -- 172.21.15.102:0/1530073744 shutdown_connections 2023-03-03T12:56:03.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.231+0000 7f72419ff700 1 -- 172.21.15.102:0/1530073744 wait complete. 2023-03-03T12:56:03.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.233+0000 7f72419ff700 1 Processor -- start 2023-03-03T12:56:03.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.233+0000 7f72419ff700 1 -- start start 2023-03-03T12:56:03.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.234+0000 7f72419ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c0cd8e0 0x7f723c0b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:03.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.234+0000 7f72419ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 0x7f723c158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:03.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.234+0000 7f72419ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f723c1593c0 0x7f723c1e2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:03.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.234+0000 7f72419ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c0c7310 con 0x7f723c0cd8e0 2023-03-03T12:56:03.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.234+0000 7f72419ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c0c75f0 con 0x7f723c153290 2023-03-03T12:56:03.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.234+0000 7f72419ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c0c78d0 con 0x7f723c1593c0 2023-03-03T12:56:03.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c0cd8e0 0x7f723c0b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:03.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c0cd8e0 0x7f723c0b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43552/0 (socket says 172.21.15.102:43552) 2023-03-03T12:56:03.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 -- 172.21.15.102:0/2159557330 learned_addr learned my addr 172.21.15.102:0/2159557330 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:56:03.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723a7fc700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 0x7f723c158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723b7fe700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f723c1593c0 0x7f723c1e2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 -- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f723c1593c0 msgr2=0x7f723c1e2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f723c1593c0 0x7f723c1e2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 -- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 msgr2=0x7f723c158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 0x7f723c158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.235+0000 7f723affd700 1 -- 172.21.15.102:0/2159557330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0094a0 con 0x7f723c0cd8e0 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.236+0000 7f723affd700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c0cd8e0 0x7f723c0b27d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f722c0090b0 tx=0x7f722c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:03.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.236+0000 7f7223fff700 1 -- 172.21.15.102:0/2159557330 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c0036a0 con 0x7f723c0cd8e0 2023-03-03T12:56:03.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.236+0000 7f7223fff700 1 -- 172.21.15.102:0/2159557330 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c01f070 con 0x7f723c0cd8e0 2023-03-03T12:56:03.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.236+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723c0ccde0 con 0x7f723c0cd8e0 2023-03-03T12:56:03.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.236+0000 7f7223fff700 1 -- 172.21.15.102:0/2159557330 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c003a20 con 0x7f723c0cd8e0 2023-03-03T12:56:03.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.236+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723c15d160 con 0x7f723c0cd8e0 2023-03-03T12:56:03.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.237+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723c0cd0c0 con 0x7f723c0cd8e0 2023-03-03T12:56:03.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.238+0000 7f7223fff700 1 -- 172.21.15.102:0/2159557330 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f722c00f050 con 0x7f723c0cd8e0 2023-03-03T12:56:03.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.239+0000 7f7223fff700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72280682f0 0x7f722806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:03.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.240+0000 7f7223fff700 1 -- 172.21.15.102:0/2159557330 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f722c021070 con 0x7f723c0cd8e0 2023-03-03T12:56:03.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.240+0000 7f723a7fc700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72280682f0 0x7f722806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:03.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.240+0000 7f723a7fc700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72280682f0 0x7f722806a7a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f723c148d50 tx=0x7f7230009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:03.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.244+0000 7f7223fff700 1 -- 172.21.15.102:0/2159557330 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f722c053ce0 con 0x7f723c0cd8e0 2023-03-03T12:56:03.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.481+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f723c06b120 con 0x7f723c0cd8e0 2023-03-03T12:56:03.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.481+0000 7f7223fff700 1 -- 172.21.15.102:0/2159557330 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f722c057300 con 0x7f723c0cd8e0 2023-03-03T12:56:03.484 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:56:03.484 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:56:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72280682f0 msgr2=0x7f722806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72280682f0 0x7f722806a7a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f723c148d50 tx=0x7f7230009210 comp rx=0 tx=0).stop 2023-03-03T12:56:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c0cd8e0 msgr2=0x7f723c0b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c0cd8e0 0x7f723c0b27d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f722c0090b0 tx=0x7f722c004750 comp rx=0 tx=0).stop 2023-03-03T12:56:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 shutdown_connections 2023-03-03T12:56:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72280682f0 0x7f722806a7a0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f723c1593c0 0x7f723c1e2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c0cd8e0 0x7f723c0b27d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 --2- 172.21.15.102:0/2159557330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f723c153290 0x7f723c158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 >> 172.21.15.102:0/2159557330 conn(0x7f723c065fb0 msgr2=0x7f723c15a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 shutdown_connections 2023-03-03T12:56:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:03.484+0000 7f72419ff700 1 -- 172.21.15.102:0/2159557330 wait complete. 2023-03-03T12:56:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:56:03.513 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2023-03-03T12:56:07.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:07.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:34.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:56:34.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.762+0000 7fc067dff700 1 Processor -- start 2023-03-03T12:56:34.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.762+0000 7fc067dff700 1 -- start start 2023-03-03T12:56:34.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.763+0000 7fc067dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0600cd8e0 0x7fc0600cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:34.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.763+0000 7fc067dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd390 0x7fc0600ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:34.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.763+0000 7fc067dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc060153660 con 0x7fc060153290 2023-03-03T12:56:34.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.763+0000 7fc067dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06006c360 con 0x7fc0600cd8e0 2023-03-03T12:56:34.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.763+0000 7fc067dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06015cea0 con 0x7fc0600cd390 2023-03-03T12:56:34.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06539a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0600cd8e0 0x7fc0600cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:34.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06539a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0600cd8e0 0x7fc0600cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56528/0 (socket says 172.21.15.102:56528) 2023-03-03T12:56:34.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06539a700 1 -- 172.21.15.102:0/2213262958 learned_addr learned my addr 172.21.15.102:0/2213262958 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:56:34.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc065b9b700 1 --1- 172.21.15.102:0/2213262958 >> v1:172.21.15.102:6789/0 conn(0x7fc060153290 0x7fc060158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56530/0 (socket says 172.21.15.102:56530) 2023-03-03T12:56:34.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06639c700 1 --2- 172.21.15.102:0/2213262958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd390 0x7fc0600ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:34.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06539a700 1 -- 172.21.15.102:0/2213262958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd390 msgr2=0x7fc0600ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:56:34.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06539a700 1 --2- 172.21.15.102:0/2213262958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd390 0x7fc0600ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06539a700 1 -- 172.21.15.102:0/2213262958 >> v1:172.21.15.102:6789/0 conn(0x7fc060153290 legacy=0x7fc060158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.764+0000 7fc06539a700 1 -- 172.21.15.102:0/2213262958 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06006c7f0 con 0x7fc0600cd8e0 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.765+0000 7fc06539a700 1 --2- 172.21.15.102:0/2213262958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd8e0 0x7fc0600cde20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc0500098f0 tx=0x7fc050009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930fa7a385518227 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.765+0000 7fc064b99700 1 -- 172.21.15.102:0/2213262958 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc050011830 con 0x7fc0600cd8e0 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.765+0000 7fc064b99700 1 -- 172.21.15.102:0/2213262958 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc050011990 con 0x7fc0600cd8e0 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.765+0000 7fc064b99700 1 -- 172.21.15.102:0/2213262958 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc050011d20 con 0x7fc0600cd8e0 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.765+0000 7fc067dff700 1 -- 172.21.15.102:0/2213262958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd8e0 msgr2=0x7fc0600cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:34.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.765+0000 7fc067dff700 1 --2- 172.21.15.102:0/2213262958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd8e0 0x7fc0600cde20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc0500098f0 tx=0x7fc050009c00 comp rx=0 tx=0).stop 2023-03-03T12:56:34.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.766+0000 7fc067dff700 1 -- 172.21.15.102:0/2213262958 shutdown_connections 2023-03-03T12:56:34.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.766+0000 7fc067dff700 1 --2- 172.21.15.102:0/2213262958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd390 0x7fc0600ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:34.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.766+0000 7fc067dff700 1 --2- 172.21.15.102:0/2213262958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd8e0 0x7fc0600cde20 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:34.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.766+0000 7fc067dff700 1 -- 172.21.15.102:0/2213262958 >> 172.21.15.102:0/2213262958 conn(0x7fc060065fb0 msgr2=0x7fc0600663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:34.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.766+0000 7fc067dff700 1 -- 172.21.15.102:0/2213262958 shutdown_connections 2023-03-03T12:56:34.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.766+0000 7fc067dff700 1 -- 172.21.15.102:0/2213262958 wait complete. 2023-03-03T12:56:34.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.768+0000 7fc067dff700 1 Processor -- start 2023-03-03T12:56:34.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.768+0000 7fc067dff700 1 -- start start 2023-03-03T12:56:34.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc067dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd390 0x7fc0600b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:34.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc067dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd8e0 0x7fc0600c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:34.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc067dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc060153290 0x7fc0601f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:34.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc067dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600b4410 con 0x7fc0600cd390 2023-03-03T12:56:34.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc067dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600b2ac0 con 0x7fc060153290 2023-03-03T12:56:34.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc067dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600b36c0 con 0x7fc0600cd8e0 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc065b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd390 0x7fc0600b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc06639c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc060153290 0x7fc0601f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc06539a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd8e0 0x7fc0600c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc065b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd390 0x7fc0600b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56532/0 (socket says 172.21.15.102:56532) 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc065b9b700 1 -- 172.21.15.102:0/4232071054 learned_addr learned my addr 172.21.15.102:0/4232071054 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc065b9b700 1 -- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd8e0 msgr2=0x7fc0600c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.769+0000 7fc065b9b700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd8e0 0x7fc0600c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:34.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc065b9b700 1 -- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc060153290 msgr2=0x7fc0601f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:34.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc065b9b700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc060153290 0x7fc0601f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:34.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc065b9b700 1 -- 172.21.15.102:0/4232071054 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc050009550 con 0x7fc0600cd390 2023-03-03T12:56:34.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc065b9b700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd390 0x7fc0600b4830 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00c520 tx=0x7fc05c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:34.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc0567fc700 1 -- 172.21.15.102:0/4232071054 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c005cb0 con 0x7fc0600cd390 2023-03-03T12:56:34.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc0567fc700 1 -- 172.21.15.102:0/4232071054 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c005e10 con 0x7fc0600cd390 2023-03-03T12:56:34.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0600ccde0 con 0x7fc0600cd390 2023-03-03T12:56:34.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc0567fc700 1 -- 172.21.15.102:0/4232071054 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0072d0 con 0x7fc0600cd390 2023-03-03T12:56:34.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.770+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06015d160 con 0x7fc0600cd390 2023-03-03T12:56:34.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.772+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0600cd0c0 con 0x7fc0600cd390 2023-03-03T12:56:34.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.773+0000 7fc0567fc700 1 -- 172.21.15.102:0/4232071054 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc05c019030 con 0x7fc0600cd390 2023-03-03T12:56:34.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.774+0000 7fc0567fc700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0440682f0 0x7fc04406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:34.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.774+0000 7fc0567fc700 1 -- 172.21.15.102:0/4232071054 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc05c084f20 con 0x7fc0600cd390 2023-03-03T12:56:34.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.774+0000 7fc06539a700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0440682f0 0x7fc04406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:34.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.775+0000 7fc06539a700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0440682f0 0x7fc04406a7a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc050014040 tx=0x7fc050008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:34.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:34.778+0000 7fc0567fc700 1 -- 172.21.15.102:0/4232071054 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc05c04b5e0 con 0x7fc0600cd390 2023-03-03T12:56:35.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.014+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0600b4680 con 0x7fc0600cd390 2023-03-03T12:56:35.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.015+0000 7fc0567fc700 1 -- 172.21.15.102:0/4232071054 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc05c017020 con 0x7fc0600cd390 2023-03-03T12:56:35.016 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:56:35.017 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:56:35.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0440682f0 msgr2=0x7fc04406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:35.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0440682f0 0x7fc04406a7a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc050014040 tx=0x7fc050008030 comp rx=0 tx=0).stop 2023-03-03T12:56:35.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd390 msgr2=0x7fc0600b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:35.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd390 0x7fc0600b4830 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00c520 tx=0x7fc05c00c830 comp rx=0 tx=0).stop 2023-03-03T12:56:35.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 shutdown_connections 2023-03-03T12:56:35.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc060153290 0x7fc0601f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:35.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0600cd390 0x7fc0600b4830 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:35.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc0440682f0 0x7fc04406a7a0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:35.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 --2- 172.21.15.102:0/4232071054 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0600cd8e0 0x7fc0600c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:35.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 >> 172.21.15.102:0/4232071054 conn(0x7fc060065fb0 msgr2=0x7fc0600cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:35.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 shutdown_connections 2023-03-03T12:56:35.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:56:35.017+0000 7fc067dff700 1 -- 172.21.15.102:0/4232071054 wait complete. 2023-03-03T12:56:35.021 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:56:35.045 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 90 secs before thrashing 2023-03-03T12:56:37.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:37.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:07.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:07.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:37.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:37.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:05.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:58:05.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.294+0000 7f33c01ff700 1 Processor -- start 2023-03-03T12:58:05.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.294+0000 7f33c01ff700 1 -- start start 2023-03-03T12:58:05.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.295+0000 7f33c01ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33b8153290 0x7f33b8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:05.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.295+0000 7f33c01ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b80cd8e0 0x7f33b80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:05.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.295+0000 7f33c01ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b8153660 con 0x7f33b81593c0 2023-03-03T12:58:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33c01ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b806c360 con 0x7f33b8153290 2023-03-03T12:58:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33c01ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b815cea0 con 0x7f33b80cd8e0 2023-03-03T12:58:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bdf9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33b8153290 0x7f33b8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bdf9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33b8153290 0x7f33b8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48966/0 (socket says 172.21.15.102:48966) 2023-03-03T12:58:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bdf9b700 1 -- 172.21.15.102:0/3810955719 learned_addr learned my addr 172.21.15.102:0/3810955719 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:58:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33be79c700 1 --2- 172.21.15.102:0/3810955719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b80cd8e0 0x7f33b80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bd79a700 1 --1- 172.21.15.102:0/3810955719 >> v1:172.21.15.172:6789/0 conn(0x7f33b81593c0 0x7f33b80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41836/0 (socket says 172.21.15.102:41836) 2023-03-03T12:58:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bdf9b700 1 -- 172.21.15.102:0/3810955719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b80cd8e0 msgr2=0x7f33b80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:58:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bdf9b700 1 --2- 172.21.15.102:0/3810955719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b80cd8e0 0x7f33b80ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bdf9b700 1 -- 172.21.15.102:0/3810955719 >> v1:172.21.15.172:6789/0 conn(0x7f33b81593c0 legacy=0x7f33b80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bdf9b700 1 -- 172.21.15.102:0/3810955719 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b806c7f0 con 0x7f33b8153290 2023-03-03T12:58:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.296+0000 7f33bcf99700 1 -- 172.21.15.102:0/3810955719 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908535218 0 0) 0x7f33b8153660 con 0x7f33b81593c0 2023-03-03T12:58:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.297+0000 7f33bdf9b700 1 --2- 172.21.15.102:0/3810955719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b8158e80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f33b40098f0 tx=0x7f33b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae3084252dd89ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:58:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.297+0000 7f33bcf99700 1 -- 172.21.15.102:0/3810955719 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4011830 con 0x7f33b8153290 2023-03-03T12:58:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.297+0000 7f33bcf99700 1 -- 172.21.15.102:0/3810955719 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b4011990 con 0x7f33b8153290 2023-03-03T12:58:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.297+0000 7f33bcf99700 1 -- 172.21.15.102:0/3810955719 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4011d20 con 0x7f33b8153290 2023-03-03T12:58:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.297+0000 7f33c01ff700 1 -- 172.21.15.102:0/3810955719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 msgr2=0x7f33b8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.297+0000 7f33c01ff700 1 --2- 172.21.15.102:0/3810955719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b8158e80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f33b40098f0 tx=0x7f33b4009c00 comp rx=0 tx=0).stop 2023-03-03T12:58:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.298+0000 7f33c01ff700 1 -- 172.21.15.102:0/3810955719 shutdown_connections 2023-03-03T12:58:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.298+0000 7f33c01ff700 1 --2- 172.21.15.102:0/3810955719 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b80cd8e0 0x7f33b80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.298+0000 7f33c01ff700 1 --2- 172.21.15.102:0/3810955719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b8158e80 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.298+0000 7f33c01ff700 1 -- 172.21.15.102:0/3810955719 >> 172.21.15.102:0/3810955719 conn(0x7f33b8065fb0 msgr2=0x7f33b80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:58:05.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.298+0000 7f33c01ff700 1 -- 172.21.15.102:0/3810955719 shutdown_connections 2023-03-03T12:58:05.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.298+0000 7f33c01ff700 1 -- 172.21.15.102:0/3810955719 wait complete. 2023-03-03T12:58:05.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.300+0000 7f33c01ff700 1 Processor -- start 2023-03-03T12:58:05.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.300+0000 7f33c01ff700 1 -- start start 2023-03-03T12:58:05.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33c01ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33b80cd8e0 0x7f33b80c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:05.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33c01ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b814e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:05.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33c01ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b81593c0 0x7f33b814e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:05.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33c01ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80b1f90 con 0x7f33b8153290 2023-03-03T12:58:05.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33c01ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80b2860 con 0x7f33b80cd8e0 2023-03-03T12:58:05.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33c01ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80b2180 con 0x7f33b81593c0 2023-03-03T12:58:05.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33bdf9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33b80cd8e0 0x7f33b80c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:05.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33bd79a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b814e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:05.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33bdf9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33b80cd8e0 0x7f33b80c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35668/0 (socket says 172.21.15.102:35668) 2023-03-03T12:58:05.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33be79c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b81593c0 0x7f33b814e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:05.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.301+0000 7f33bdf9b700 1 -- 172.21.15.102:0/1367436541 learned_addr learned my addr 172.21.15.102:0/1367436541 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:58:05.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.302+0000 7f33bdf9b700 1 -- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b81593c0 msgr2=0x7f33b814e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:05.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.302+0000 7f33bdf9b700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b81593c0 0x7f33b814e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.302+0000 7f33bdf9b700 1 -- 172.21.15.102:0/1367436541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 msgr2=0x7f33b814e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:05.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.302+0000 7f33bdf9b700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b814e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.302+0000 7f33bdf9b700 1 -- 172.21.15.102:0/1367436541 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b4009550 con 0x7f33b80cd8e0 2023-03-03T12:58:05.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.302+0000 7f33bd79a700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b814e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:58:05.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.302+0000 7f33bdf9b700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33b80cd8e0 0x7f33b80c4750 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f33b4009130 tx=0x7f33b4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:58:05.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.303+0000 7f33ae7fc700 1 -- 172.21.15.102:0/1367436541 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4004510 con 0x7f33b80cd8e0 2023-03-03T12:58:05.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.303+0000 7f33ae7fc700 1 -- 172.21.15.102:0/1367436541 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b401e070 con 0x7f33b80cd8e0 2023-03-03T12:58:05.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.303+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b80ccde0 con 0x7f33b80cd8e0 2023-03-03T12:58:05.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.303+0000 7f33ae7fc700 1 -- 172.21.15.102:0/1367436541 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b40038c0 con 0x7f33b80cd8e0 2023-03-03T12:58:05.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.303+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b815d160 con 0x7f33b80cd8e0 2023-03-03T12:58:05.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.304+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b80cd0c0 con 0x7f33b80cd8e0 2023-03-03T12:58:05.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.306+0000 7f33ae7fc700 1 -- 172.21.15.102:0/1367436541 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f33b400f050 con 0x7f33b80cd8e0 2023-03-03T12:58:05.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.307+0000 7f33ae7fc700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f339c068320 0x7f339c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:05.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.307+0000 7f33ae7fc700 1 -- 172.21.15.102:0/1367436541 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f33b40870f0 con 0x7f33b80cd8e0 2023-03-03T12:58:05.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.307+0000 7f33bd79a700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f339c068320 0x7f339c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:05.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.308+0000 7f33bd79a700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f339c068320 0x7f339c06a7d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f33b815bdd0 tx=0x7f33a8007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:58:05.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.311+0000 7f33ae7fc700 1 -- 172.21.15.102:0/1367436541 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f33b4053780 con 0x7f33b80cd8e0 2023-03-03T12:58:05.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.544+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33b804e6e0 con 0x7f33b80cd8e0 2023-03-03T12:58:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.545+0000 7f33ae7fc700 1 -- 172.21.15.102:0/1367436541 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f33b4056da0 con 0x7f33b80cd8e0 2023-03-03T12:58:05.547 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:58:05.547 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:58:05.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f339c068320 msgr2=0x7f339c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:05.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f339c068320 0x7f339c06a7d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f33b815bdd0 tx=0x7f33a8007de0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33b80cd8e0 msgr2=0x7f33b80c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33b80cd8e0 0x7f33b80c4750 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f33b4009130 tx=0x7f33b4004750 comp rx=0 tx=0).stop 2023-03-03T12:58:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 shutdown_connections 2023-03-03T12:58:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f339c068320 0x7f339c06a7d0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f33b81593c0 0x7f33b814e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33b80cd8e0 0x7f33b80c4750 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 --2- 172.21.15.102:0/1367436541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33b8153290 0x7f33b814e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 >> 172.21.15.102:0/1367436541 conn(0x7f33b8065fb0 msgr2=0x7f33b8150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:58:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 shutdown_connections 2023-03-03T12:58:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:58:05.548+0000 7f33c01ff700 1 -- 172.21.15.102:0/1367436541 wait complete. 2023-03-03T12:58:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:58:05.577 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T12:58:08.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:08.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:38.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:38.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:08.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:08.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:38.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:38.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:56.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:59:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.825+0000 7f43b3fff700 1 Processor -- start 2023-03-03T12:59:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.825+0000 7f43b3fff700 1 -- start start 2023-03-03T12:59:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.827+0000 7f43b3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f43b40b3970 0x7f43b40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.827+0000 7f43b3fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b40c3d20 con 0x7f43b40ced00 2023-03-03T12:59:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.827+0000 7f43b3fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b415cba0 con 0x7f43b40c41b0 2023-03-03T12:59:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.827+0000 7f43b3fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40c4930 con 0x7f43b40b3970 2023-03-03T12:59:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.827+0000 7f43b2ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f43b40ced00 0x7f43b4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52510/0 (socket says 172.21.15.102:52510) 2023-03-03T12:59:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.827+0000 7f43b2ffd700 1 -- 172.21.15.102:0/575221669 learned_addr learned my addr 172.21.15.102:0/575221669 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T12:59:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.827+0000 7f43b27fc700 1 --2- 172.21.15.102:0/575221669 >> v2:172.21.15.102:3300/0 conn(0x7f43b40b3970 0x7f43b40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b1ffb700 1 -- 172.21.15.102:0/575221669 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223851544 0 0) 0x7f43b40c3d20 con 0x7f43b40ced00 2023-03-03T12:59:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b1ffb700 1 -- 172.21.15.102:0/575221669 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f439c0035b0 con 0x7f43b40ced00 2023-03-03T12:59:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b1ffb700 1 -- 172.21.15.102:0/575221669 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933327081 0 0) 0x7f43b415cba0 con 0x7f43b40c41b0 2023-03-03T12:59:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b1ffb700 1 -- 172.21.15.102:0/575221669 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b40c3d20 con 0x7f43b40c41b0 2023-03-03T12:59:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b27fc700 1 -- 172.21.15.102:0/575221669 >> v1:172.21.15.172:6789/0 conn(0x7f43b40c41b0 legacy=0x7f43b4163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b27fc700 1 -- 172.21.15.102:0/575221669 >> v1:172.21.15.102:6789/0 conn(0x7f43b40ced00 legacy=0x7f43b4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b27fc700 1 -- 172.21.15.102:0/575221669 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b415ce80 con 0x7f43b40b3970 2023-03-03T12:59:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.828+0000 7f43b27fc700 1 --2- 172.21.15.102:0/575221669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b40c5a90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009840 tx=0x7f43a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2e4d1ac2d77750 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b1ffb700 1 -- 172.21.15.102:0/575221669 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8011830 con 0x7f43b40b3970 2023-03-03T12:59:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b1ffb700 1 -- 172.21.15.102:0/575221669 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8011990 con 0x7f43b40b3970 2023-03-03T12:59:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b1ffb700 1 -- 172.21.15.102:0/575221669 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8011d20 con 0x7f43b40b3970 2023-03-03T12:59:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b3fff700 1 -- 172.21.15.102:0/575221669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 msgr2=0x7f43b40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b3fff700 1 --2- 172.21.15.102:0/575221669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b40c5a90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009840 tx=0x7f43a8009b50 comp rx=0 tx=0).stop 2023-03-03T12:59:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b3fff700 1 -- 172.21.15.102:0/575221669 shutdown_connections 2023-03-03T12:59:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b3fff700 1 --2- 172.21.15.102:0/575221669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b40c5a90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b3fff700 1 -- 172.21.15.102:0/575221669 >> 172.21.15.102:0/575221669 conn(0x7f43b408b1c0 msgr2=0x7f43b4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b3fff700 1 -- 172.21.15.102:0/575221669 shutdown_connections 2023-03-03T12:59:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.829+0000 7f43b3fff700 1 -- 172.21.15.102:0/575221669 wait complete. 2023-03-03T12:59:56.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.831+0000 7f43b3fff700 1 Processor -- start 2023-03-03T12:59:56.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.832+0000 7f43b3fff700 1 -- start start 2023-03-03T12:59:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b414f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43b40c41b0 0x7f43b414b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43b40ced00 0x7f43b4151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41511b0 con 0x7f43b40b3970 2023-03-03T12:59:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b414bd60 con 0x7f43b40ced00 2023-03-03T12:59:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41523e0 con 0x7f43b40c41b0 2023-03-03T12:59:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b414f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53454/0 (socket says 172.21.15.102:53454) 2023-03-03T12:59:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b2ffd700 1 -- 172.21.15.102:0/2730512127 learned_addr learned my addr 172.21.15.102:0/2730512127 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T12:59:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b27fc700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43b40c41b0 0x7f43b414b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b37fe700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43b40ced00 0x7f43b4151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b2ffd700 1 -- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43b40c41b0 msgr2=0x7f43b414b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b2ffd700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43b40c41b0 0x7f43b414b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b2ffd700 1 -- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43b40ced00 msgr2=0x7f43b4151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.833+0000 7f43b2ffd700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43b40ced00 0x7f43b4151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.834+0000 7f43b2ffd700 1 -- 172.21.15.102:0/2730512127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a80094a0 con 0x7f43b40b3970 2023-03-03T12:59:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.834+0000 7f43b2ffd700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b414f680 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f43a400c730 tx=0x7f43a400ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.834+0000 7f439b7fe700 1 -- 172.21.15.102:0/2730512127 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4005d70 con 0x7f43b40b3970 2023-03-03T12:59:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.834+0000 7f439b7fe700 1 -- 172.21.15.102:0/2730512127 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40070d0 con 0x7f43b40b3970 2023-03-03T12:59:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.834+0000 7f439b7fe700 1 -- 172.21.15.102:0/2730512127 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4007480 con 0x7f43b40b3970 2023-03-03T12:59:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.834+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b406c200 con 0x7f43b40b3970 2023-03-03T12:59:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.834+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b406bf20 con 0x7f43b40b3970 2023-03-03T12:59:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.835+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b415c890 con 0x7f43b40b3970 2023-03-03T12:59:56.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.837+0000 7f439b7fe700 1 -- 172.21.15.102:0/2730512127 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f43a4019030 con 0x7f43b40b3970 2023-03-03T12:59:56.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.838+0000 7f439b7fe700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43a0068320 0x7f43a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:56.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.838+0000 7f439b7fe700 1 -- 172.21.15.102:0/2730512127 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f43a4085170 con 0x7f43b40b3970 2023-03-03T12:59:56.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.838+0000 7f43b27fc700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43a0068320 0x7f43a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:56.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.839+0000 7f43b27fc700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43a0068320 0x7f43a006a7d0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009fd0 tx=0x7f43a801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:56.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:56.842+0000 7f439b7fe700 1 -- 172.21.15.102:0/2730512127 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f43a40517f0 con 0x7f43b40b3970 2023-03-03T12:59:57.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.078+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43b40cd970 con 0x7f43b40b3970 2023-03-03T12:59:57.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.078+0000 7f439b7fe700 1 -- 172.21.15.102:0/2730512127 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f43a4017090 con 0x7f43b40b3970 2023-03-03T12:59:57.081 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T12:59:57.081 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T12:59:57.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43a0068320 msgr2=0x7f43a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:57.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43a0068320 0x7f43a006a7d0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009fd0 tx=0x7f43a801c040 comp rx=0 tx=0).stop 2023-03-03T12:59:57.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 msgr2=0x7f43b414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:57.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b414f680 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f43a400c730 tx=0x7f43a400ca40 comp rx=0 tx=0).stop 2023-03-03T12:59:57.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 shutdown_connections 2023-03-03T12:59:57.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43b40ced00 0x7f43b4151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:57.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40b3970 0x7f43b414f680 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:57.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43a0068320 0x7f43a006a7d0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:57.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 --2- 172.21.15.102:0/2730512127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43b40c41b0 0x7f43b414b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:57.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.081+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 >> 172.21.15.102:0/2730512127 conn(0x7f43b408b1c0 msgr2=0x7f43b415bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:57.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.082+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 shutdown_connections 2023-03-03T12:59:57.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T12:59:57.082+0000 7f43b3fff700 1 -- 172.21.15.102:0/2730512127 wait complete. 2023-03-03T12:59:57.086 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T12:59:57.110 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T13:00:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T12:59:59.998+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:00:08.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:08.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:31.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:00:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.374+0000 7f2dcf1ff700 1 Processor -- start 2023-03-03T13:00:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.375+0000 7f2dcf1ff700 1 -- start start 2023-03-03T13:00:31.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dcf1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2dc8153290 0x7f2dc8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:31.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dcf1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2dc81593c0 0x7f2dc80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:31.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dcf1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8153660 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dcf1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc806c360 con 0x7f2dc81593c0 2023-03-03T13:00:31.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dcf1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc815cea0 con 0x7f2dc8153290 2023-03-03T13:00:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dcd79c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2dc80cd8e0 0x7f2dc80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50724/0 (socket says 172.21.15.102:50724) 2023-03-03T13:00:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dcd79c700 1 -- 172.21.15.102:0/672151024 learned_addr learned my addr 172.21.15.102:0/672151024 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:00:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dccf9b700 1 --2- 172.21.15.102:0/672151024 >> v2:172.21.15.172:3300/0 conn(0x7f2dc8153290 0x7f2dc8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.376+0000 7f2dc7fff700 1 --2- 172.21.15.102:0/672151024 >> v2:172.21.15.102:3300/0 conn(0x7f2dc81593c0 0x7f2dc80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.377+0000 7f2dc77fe700 1 -- 172.21.15.102:0/672151024 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781334640 0 0) 0x7f2dc8153660 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.377+0000 7f2dc77fe700 1 -- 172.21.15.102:0/672151024 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db80035b0 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.377+0000 7f2dccf9b700 1 -- 172.21.15.102:0/672151024 >> v2:172.21.15.102:3300/0 conn(0x7f2dc81593c0 msgr2=0x7f2dc80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:00:31.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.377+0000 7f2dccf9b700 1 --2- 172.21.15.102:0/672151024 >> v2:172.21.15.102:3300/0 conn(0x7f2dc81593c0 0x7f2dc80cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.377+0000 7f2dccf9b700 1 -- 172.21.15.102:0/672151024 >> v1:172.21.15.102:6789/0 conn(0x7f2dc80cd8e0 legacy=0x7f2dc80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:31.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.377+0000 7f2dccf9b700 1 -- 172.21.15.102:0/672151024 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc806c7f0 con 0x7f2dc8153290 2023-03-03T13:00:31.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.377+0000 7f2dccf9b700 1 --2- 172.21.15.102:0/672151024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc8153290 0x7f2dc8158e80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2dc808cc70 tx=0x7f2dbc009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb5e61d97574358 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:00:31.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dc77fe700 1 -- 172.21.15.102:0/672151024 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc011830 con 0x7f2dc8153290 2023-03-03T13:00:31.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dc77fe700 1 -- 172.21.15.102:0/672151024 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc011990 con 0x7f2dc8153290 2023-03-03T13:00:31.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dc77fe700 1 -- 172.21.15.102:0/672151024 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc011d20 con 0x7f2dc8153290 2023-03-03T13:00:31.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/672151024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc8153290 msgr2=0x7f2dc8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/672151024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc8153290 0x7f2dc8158e80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2dc808cc70 tx=0x7f2dbc009b20 comp rx=0 tx=0).stop 2023-03-03T13:00:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/672151024 shutdown_connections 2023-03-03T13:00:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/672151024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc8153290 0x7f2dc8158e80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/672151024 >> v2:172.21.15.102:3300/0 conn(0x7f2dc81593c0 0x7f2dc80cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/672151024 >> 172.21.15.102:0/672151024 conn(0x7f2dc8065fb0 msgr2=0x7f2dc80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:00:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/672151024 shutdown_connections 2023-03-03T13:00:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.378+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/672151024 wait complete. 2023-03-03T13:00:31.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.380+0000 7f2dcf1ff700 1 Processor -- start 2023-03-03T13:00:31.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.380+0000 7f2dcf1ff700 1 -- start start 2023-03-03T13:00:31.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.381+0000 7f2dcf1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc80cd8e0 0x7f2dc80b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:31.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.381+0000 7f2dcf1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2dc8153290 0x7f2dc81eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:31.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dcf1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2dc81593c0 0x7f2dc81ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:31.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dcf1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc81541c0 con 0x7f2dc81593c0 2023-03-03T13:00:31.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dcf1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8153970 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dcf1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc80b3490 con 0x7f2dc8153290 2023-03-03T13:00:31.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc80cd8e0 0x7f2dc80b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:31.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dcd79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2dc81593c0 0x7f2dc81ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:31.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dc7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2dc8153290 0x7f2dc81eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:31.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc80cd8e0 0x7f2dc80b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48646/0 (socket says 172.21.15.102:48646) 2023-03-03T13:00:31.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 -- 172.21.15.102:0/1019554384 learned_addr learned my addr 172.21.15.102:0/1019554384 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:00:31.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 -- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2dc8153290 msgr2=0x7f2dc81eed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:31.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2dc8153290 0x7f2dc81eed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 -- 172.21.15.102:0/1019554384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2dc81593c0 msgr2=0x7f2dc81ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:31.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2dc81593c0 0x7f2dc81ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dccf9b700 1 -- 172.21.15.102:0/1019554384 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0094a0 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.382+0000 7f2dcd79c700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2dc81593c0 0x7f2dc81ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:00:31.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.383+0000 7f2dccf9b700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc80cd8e0 0x7f2dc80b44c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009b60 tx=0x7f2dbc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:00:31.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.383+0000 7f2dc57fa700 1 -- 172.21.15.102:0/1019554384 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc004590 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.383+0000 7f2dc57fa700 1 -- 172.21.15.102:0/1019554384 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc01f070 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.383+0000 7f2dc57fa700 1 -- 172.21.15.102:0/1019554384 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0038e0 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.383+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc80ccde0 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.383+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc815d160 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.384+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80cd0c0 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.385+0000 7f2dc57fa700 1 -- 172.21.15.102:0/1019554384 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2dbc00f050 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.387+0000 7f2dc57fa700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dac068320 0x7f2dac06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:31.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.387+0000 7f2dc57fa700 1 -- 172.21.15.102:0/1019554384 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2dbc021070 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.387+0000 7f2dc7fff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dac068320 0x7f2dac06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:31.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.388+0000 7f2dc7fff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dac068320 0x7f2dac06a7d0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80c5720 tx=0x7f2db4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:00:31.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.391+0000 7f2dc57fa700 1 -- 172.21.15.102:0/1019554384 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2dbc053bb0 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.627+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dc806b180 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.628+0000 7f2dc57fa700 1 -- 172.21.15.102:0/1019554384 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2dbc0571d0 con 0x7f2dc80cd8e0 2023-03-03T13:00:31.630 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:00:31.630 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:00:31.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dac068320 msgr2=0x7f2dac06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:31.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dac068320 0x7f2dac06a7d0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80c5720 tx=0x7f2db4009210 comp rx=0 tx=0).stop 2023-03-03T13:00:31.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc80cd8e0 msgr2=0x7f2dc80b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:31.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc80cd8e0 0x7f2dc80b44c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009b60 tx=0x7f2dbc004750 comp rx=0 tx=0).stop 2023-03-03T13:00:31.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 shutdown_connections 2023-03-03T13:00:31.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2dc81593c0 0x7f2dc81ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dc80cd8e0 0x7f2dc80b44c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dac068320 0x7f2dac06a7d0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 --2- 172.21.15.102:0/1019554384 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2dc8153290 0x7f2dc81eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:31.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 >> 172.21.15.102:0/1019554384 conn(0x7f2dc8065fb0 msgr2=0x7f2dc80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:00:31.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 shutdown_connections 2023-03-03T13:00:31.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:00:31.631+0000 7f2dcf1ff700 1 -- 172.21.15.102:0/1019554384 wait complete. 2023-03-03T13:00:31.635 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:00:31.659 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T13:00:38.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:38.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:09.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:09.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:39.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:39.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:50.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:01:50.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.910+0000 7f4ecad9d700 1 Processor -- start 2023-03-03T13:01:50.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.910+0000 7f4ecad9d700 1 -- start start 2023-03-03T13:01:50.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ecad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40ced00 0x7f4ec40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:50.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ecad9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ec40c41b0 0x7f4ec40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:50.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ecad9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40c3d20 con 0x7f4ec40b3970 2023-03-03T13:01:50.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ecad9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec40c4930 con 0x7f4ec40c41b0 2023-03-03T13:01:50.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ecad9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40cf610 con 0x7f4ec40ced00 2023-03-03T13:01:50.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4eca59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ec40c41b0 0x7f4ec40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:50.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ec9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40ced00 0x7f4ec40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:50.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ec959a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4ec40b3970 0x7f4ec4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35168/0 (socket says 172.21.15.102:35168) 2023-03-03T13:01:50.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ec9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40ced00 0x7f4ec40c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54856/0 (socket says 172.21.15.102:54856) 2023-03-03T13:01:50.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4ec959a700 1 -- 172.21.15.102:0/1241858285 learned_addr learned my addr 172.21.15.102:0/1241858285 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:01:50.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.912+0000 7f4eca59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ec40c41b0 0x7f4ec40cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43376/0 (socket says 172.21.15.102:43376) 2023-03-03T13:01:50.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.913+0000 7f4eca59c700 1 -- 172.21.15.102:0/1241858285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40ced00 msgr2=0x7f4ec40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:01:50.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.913+0000 7f4eca59c700 1 --2- 172.21.15.102:0/1241858285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40ced00 0x7f4ec40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:50.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.913+0000 7f4eca59c700 1 -- 172.21.15.102:0/1241858285 >> v1:172.21.15.102:6789/0 conn(0x7f4ec40b3970 legacy=0x7f4ec4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:50.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.913+0000 7f4eca59c700 1 -- 172.21.15.102:0/1241858285 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec415cba0 con 0x7f4ec40c41b0 2023-03-03T13:01:50.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.913+0000 7f4ec8d99700 1 -- 172.21.15.102:0/1241858285 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855044142 0 0) 0x7f4ec40c3d20 con 0x7f4ec40b3970 2023-03-03T13:01:50.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.913+0000 7f4eca59c700 1 --2- 172.21.15.102:0/1241858285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40c41b0 0x7f4ec40cf0d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009dc0 tx=0x7f4ebc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604af9523342b2b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:50.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ec8d99700 1 -- 172.21.15.102:0/1241858285 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00e070 con 0x7f4ec40c41b0 2023-03-03T13:01:50.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ec8d99700 1 -- 172.21.15.102:0/1241858285 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc00bae0 con 0x7f4ec40c41b0 2023-03-03T13:01:50.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ec8d99700 1 -- 172.21.15.102:0/1241858285 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00be90 con 0x7f4ec40c41b0 2023-03-03T13:01:50.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 -- 172.21.15.102:0/1241858285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40c41b0 msgr2=0x7f4ec40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:50.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/1241858285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40c41b0 0x7f4ec40cf0d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009dc0 tx=0x7f4ebc00b330 comp rx=0 tx=0).stop 2023-03-03T13:01:50.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 -- 172.21.15.102:0/1241858285 shutdown_connections 2023-03-03T13:01:50.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/1241858285 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40ced00 0x7f4ec40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:50.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/1241858285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40c41b0 0x7f4ec40cf0d0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:50.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 -- 172.21.15.102:0/1241858285 >> 172.21.15.102:0/1241858285 conn(0x7f4ec408b1c0 msgr2=0x7f4ec4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:01:50.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 -- 172.21.15.102:0/1241858285 shutdown_connections 2023-03-03T13:01:50.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.914+0000 7f4ecad9d700 1 -- 172.21.15.102:0/1241858285 wait complete. 2023-03-03T13:01:50.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.916+0000 7f4ecad9d700 1 Processor -- start 2023-03-03T13:01:50.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.916+0000 7f4ecad9d700 1 -- start start 2023-03-03T13:01:50.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.917+0000 7f4ecad9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec40b3970 0x7f4ec41e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:50.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.917+0000 7f4ecad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40c41b0 0x7f4ec415bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:50.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.917+0000 7f4ecad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40ced00 0x7f4ec4145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:50.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.917+0000 7f4ecad9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec4151390 con 0x7f4ec40ced00 2023-03-03T13:01:50.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.917+0000 7f4ecad9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec415b8c0 con 0x7f4ec40b3970 2023-03-03T13:01:50.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.917+0000 7f4ecad9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec415b3e0 con 0x7f4ec40c41b0 2023-03-03T13:01:50.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40ced00 0x7f4ec4145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:50.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40ced00 0x7f4ec4145470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43382/0 (socket says 172.21.15.102:43382) 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 -- 172.21.15.102:0/625305230 learned_addr learned my addr 172.21.15.102:0/625305230 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4ec9d9b700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec40b3970 0x7f4ec41e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4ec959a700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40c41b0 0x7f4ec415bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 -- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40c41b0 msgr2=0x7f4ec415bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40c41b0 0x7f4ec415bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 -- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec40b3970 msgr2=0x7f4ec41e5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec40b3970 0x7f4ec41e5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:50.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.918+0000 7f4eca59c700 1 -- 172.21.15.102:0/625305230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc009a70 con 0x7f4ec40ced00 2023-03-03T13:01:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.919+0000 7f4eca59c700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40ced00 0x7f4ec4145470 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc014040 tx=0x7f4ebc004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.919+0000 7f4eba7fc700 1 -- 172.21.15.102:0/625305230 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc004610 con 0x7f4ec40ced00 2023-03-03T13:01:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.919+0000 7f4eba7fc700 1 -- 172.21.15.102:0/625305230 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc020070 con 0x7f4ec40ced00 2023-03-03T13:01:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.919+0000 7f4eba7fc700 1 -- 172.21.15.102:0/625305230 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0049b0 con 0x7f4ec40ced00 2023-03-03T13:01:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.919+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec406bf20 con 0x7f4ec40ced00 2023-03-03T13:01:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.919+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec415ce80 con 0x7f4ec40ced00 2023-03-03T13:01:50.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.920+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec406c200 con 0x7f4ec40ced00 2023-03-03T13:01:50.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.921+0000 7f4eba7fc700 1 -- 172.21.15.102:0/625305230 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ebc02e070 con 0x7f4ec40ced00 2023-03-03T13:01:50.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.922+0000 7f4eba7fc700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea80682f0 0x7f4ea806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:50.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.923+0000 7f4eba7fc700 1 -- 172.21.15.102:0/625305230 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4ebc022070 con 0x7f4ec40ced00 2023-03-03T13:01:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.923+0000 7f4ec9d9b700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea80682f0 0x7f4ea806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.924+0000 7f4ec9d9b700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea80682f0 0x7f4ea806a7a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009be0 tx=0x7f4ec0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:50.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:50.927+0000 7f4eba7fc700 1 -- 172.21.15.102:0/625305230 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ebc054100 con 0x7f4ec40ced00 2023-03-03T13:01:51.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.168+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ec415b6e0 con 0x7f4ec40ced00 2023-03-03T13:01:51.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.168+0000 7f4eba7fc700 1 -- 172.21.15.102:0/625305230 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4ebc02c090 con 0x7f4ec40ced00 2023-03-03T13:01:51.170 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:01:51.170 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:01:51.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea80682f0 msgr2=0x7f4ea806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:51.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea80682f0 0x7f4ea806a7a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009be0 tx=0x7f4ec0009210 comp rx=0 tx=0).stop 2023-03-03T13:01:51.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40ced00 msgr2=0x7f4ec4145470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:51.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40ced00 0x7f4ec4145470 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc014040 tx=0x7f4ebc004b20 comp rx=0 tx=0).stop 2023-03-03T13:01:51.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 shutdown_connections 2023-03-03T13:01:51.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec40ced00 0x7f4ec4145470 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:51.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec40b3970 0x7f4ec41e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:51.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea80682f0 0x7f4ea806a7a0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:51.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 --2- 172.21.15.102:0/625305230 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec40c41b0 0x7f4ec415bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:51.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 >> 172.21.15.102:0/625305230 conn(0x7f4ec408b1c0 msgr2=0x7f4ec4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:01:51.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 shutdown_connections 2023-03-03T13:01:51.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:01:51.171+0000 7f4ecad9d700 1 -- 172.21.15.102:0/625305230 wait complete. 2023-03-03T13:01:51.174 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:01:51.199 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2023-03-03T13:02:09.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:09.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:39.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:39.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:10.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:10.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:36.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:03:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.450+0000 7f32c9fff700 1 Processor -- start 2023-03-03T13:03:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.450+0000 7f32c9fff700 1 -- start start 2023-03-03T13:03:36.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.451+0000 7f32c9fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32c40cd8e0 0x7f32c40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.451+0000 7f32c9fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32c40cd390 0x7f32c40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.451+0000 7f32c9fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c4153660 con 0x7f32c4153290 2023-03-03T13:03:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.451+0000 7f32c9fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c406c360 con 0x7f32c40cd8e0 2023-03-03T13:03:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.451+0000 7f32c9fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c415cea0 con 0x7f32c40cd390 2023-03-03T13:03:36.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.451+0000 7f32c2ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32c40cd8e0 0x7f32c40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:36.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c37fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f32c4153290 0x7f32c4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45442/0 (socket says 172.21.15.102:45442) 2023-03-03T13:03:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c37fe700 1 -- 172.21.15.102:0/1494017092 learned_addr learned my addr 172.21.15.102:0/1494017092 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:03:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c2ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f32c40cd8e0 0x7f32c40cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42340/0 (socket says 172.21.15.102:42340) 2023-03-03T13:03:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c3fff700 1 --2- 172.21.15.102:0/1494017092 >> v2:172.21.15.172:3300/0 conn(0x7f32c40cd390 0x7f32c40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c2ffd700 1 -- 172.21.15.102:0/1494017092 >> v2:172.21.15.172:3300/0 conn(0x7f32c40cd390 msgr2=0x7f32c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:03:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c2ffd700 1 --2- 172.21.15.102:0/1494017092 >> v2:172.21.15.172:3300/0 conn(0x7f32c40cd390 0x7f32c40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c2ffd700 1 -- 172.21.15.102:0/1494017092 >> v1:172.21.15.102:6789/0 conn(0x7f32c4153290 legacy=0x7f32c4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:36.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c2ffd700 1 -- 172.21.15.102:0/1494017092 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c406c7f0 con 0x7f32c40cd8e0 2023-03-03T13:03:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.452+0000 7f32c2ffd700 1 --2- 172.21.15.102:0/1494017092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c40cde20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f32c40c9390 tx=0x7f32b8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193fead201a3b241 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c27fc700 1 -- 172.21.15.102:0/1494017092 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8011830 con 0x7f32c40cd8e0 2023-03-03T13:03:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c27fc700 1 -- 172.21.15.102:0/1494017092 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b8011990 con 0x7f32c40cd8e0 2023-03-03T13:03:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c27fc700 1 -- 172.21.15.102:0/1494017092 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8011d20 con 0x7f32c40cd8e0 2023-03-03T13:03:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 -- 172.21.15.102:0/1494017092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 msgr2=0x7f32c40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:36.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 --2- 172.21.15.102:0/1494017092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c40cde20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f32c40c9390 tx=0x7f32b8009b20 comp rx=0 tx=0).stop 2023-03-03T13:03:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 -- 172.21.15.102:0/1494017092 shutdown_connections 2023-03-03T13:03:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 --2- 172.21.15.102:0/1494017092 >> v2:172.21.15.172:3300/0 conn(0x7f32c40cd390 0x7f32c40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 --2- 172.21.15.102:0/1494017092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c40cde20 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 -- 172.21.15.102:0/1494017092 >> 172.21.15.102:0/1494017092 conn(0x7f32c4065fb0 msgr2=0x7f32c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:03:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 -- 172.21.15.102:0/1494017092 shutdown_connections 2023-03-03T13:03:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.453+0000 7f32c9fff700 1 -- 172.21.15.102:0/1494017092 wait complete. 2023-03-03T13:03:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.455+0000 7f32c9fff700 1 Processor -- start 2023-03-03T13:03:36.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.456+0000 7f32c9fff700 1 -- start start 2023-03-03T13:03:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c9fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32c40cd390 0x7f32c415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c9fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c41efb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c9fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32c4153290 0x7f32c41f0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c9fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40b3eb0 con 0x7f32c40cd8e0 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c9fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40b2d40 con 0x7f32c40cd390 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c9fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40c8de0 con 0x7f32c4153290 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c41efb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c41efb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42356/0 (socket says 172.21.15.102:42356) 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 -- 172.21.15.102:0/3456360718 learned_addr learned my addr 172.21.15.102:0/3456360718 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c37fe700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32c40cd390 0x7f32c415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:36.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c3fff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32c4153290 0x7f32c41f0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 -- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32c4153290 msgr2=0x7f32c41f0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32c4153290 0x7f32c41f0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 -- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32c40cd390 msgr2=0x7f32c415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32c40cd390 0x7f32c415ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c2ffd700 1 -- 172.21.15.102:0/3456360718 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80094a0 con 0x7f32c40cd8e0 2023-03-03T13:03:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.457+0000 7f32c37fe700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32c40cd390 0x7f32c415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:03:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.458+0000 7f32c2ffd700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c41efb50 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f32b80090b0 tx=0x7f32b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.458+0000 7f32abfff700 1 -- 172.21.15.102:0/3456360718 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80036a0 con 0x7f32c40cd8e0 2023-03-03T13:03:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.458+0000 7f32abfff700 1 -- 172.21.15.102:0/3456360718 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b801f070 con 0x7f32c40cd8e0 2023-03-03T13:03:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.458+0000 7f32abfff700 1 -- 172.21.15.102:0/3456360718 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8003a40 con 0x7f32c40cd8e0 2023-03-03T13:03:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.458+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c40ccde0 con 0x7f32c40cd8e0 2023-03-03T13:03:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.458+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c415d160 con 0x7f32c40cd8e0 2023-03-03T13:03:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.459+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c40cd0c0 con 0x7f32c40cd8e0 2023-03-03T13:03:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.461+0000 7f32abfff700 1 -- 172.21.15.102:0/3456360718 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f32b800f050 con 0x7f32c40cd8e0 2023-03-03T13:03:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.462+0000 7f32abfff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32b00682f0 0x7f32b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.462+0000 7f32abfff700 1 -- 172.21.15.102:0/3456360718 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f32b8021070 con 0x7f32c40cd8e0 2023-03-03T13:03:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.462+0000 7f32c37fe700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32b00682f0 0x7f32b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.463+0000 7f32c37fe700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32b00682f0 0x7f32b006a7a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f32b4007fd0 tx=0x7f32b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:36.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.466+0000 7f32abfff700 1 -- 172.21.15.102:0/3456360718 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f32b8053d70 con 0x7f32c40cd8e0 2023-03-03T13:03:36.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.700+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32c40b3ff0 con 0x7f32c40cd8e0 2023-03-03T13:03:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.701+0000 7f32abfff700 1 -- 172.21.15.102:0/3456360718 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f32b8057390 con 0x7f32c40cd8e0 2023-03-03T13:03:36.702 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:03:36.702 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:03:36.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.703+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32b00682f0 msgr2=0x7f32b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:36.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.703+0000 7f32c9fff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32b00682f0 0x7f32b006a7a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f32b4007fd0 tx=0x7f32b4009040 comp rx=0 tx=0).stop 2023-03-03T13:03:36.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.703+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 msgr2=0x7f32c41efb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:36.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.703+0000 7f32c9fff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c41efb50 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f32b80090b0 tx=0x7f32b8004750 comp rx=0 tx=0).stop 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.703+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 shutdown_connections 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.703+0000 7f32c9fff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f32b00682f0 0x7f32b006a7a0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.703+0000 7f32c9fff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f32c4153290 0x7f32c41f0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.704+0000 7f32c9fff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32c40cd390 0x7f32c415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.704+0000 7f32c9fff700 1 --2- 172.21.15.102:0/3456360718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32c40cd8e0 0x7f32c41efb50 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.704+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 >> 172.21.15.102:0/3456360718 conn(0x7f32c4065fb0 msgr2=0x7f32c4156c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.704+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 shutdown_connections 2023-03-03T13:03:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:03:36.704+0000 7f32c9fff700 1 -- 172.21.15.102:0/3456360718 wait complete. 2023-03-03T13:03:36.707 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:03:36.732 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-03-03T13:03:40.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:40.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:10.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:10.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:40.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:40.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:10.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:10.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:24.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:05:24.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.980+0000 7fa2c239d700 1 Processor -- start 2023-03-03T13:05:24.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.981+0000 7fa2c239d700 1 -- start start 2023-03-03T13:05:24.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.982+0000 7fa2c239d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2bc1564b0 0x7fa2bc0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:24.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.982+0000 7fa2c239d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0cab50 0x7fa2bc0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:24.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.982+0000 7fa2c239d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0c9ca0 con 0x7fa2bc0c9610 2023-03-03T13:05:24.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.982+0000 7fa2c239d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2bc0b4920 con 0x7fa2bc1564b0 2023-03-03T13:05:24.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.982+0000 7fa2c239d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0b4c00 con 0x7fa2bc0cab50 2023-03-03T13:05:24.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c139b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2bc1564b0 0x7fa2bc0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:24.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c0b9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa2bc0c9610 0x7fa2bc0ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43344/0 (socket says 172.21.15.102:43344) 2023-03-03T13:05:24.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c1b9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0cab50 0x7fa2bc0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:24.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c139b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa2bc1564b0 0x7fa2bc0cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42724/0 (socket says 172.21.15.102:42724) 2023-03-03T13:05:24.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c0b9a700 1 -- 172.21.15.102:0/728585919 learned_addr learned my addr 172.21.15.102:0/728585919 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:05:24.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c1b9c700 1 -- 172.21.15.102:0/728585919 >> v2:172.21.15.102:3300/0 conn(0x7fa2bc1564b0 msgr2=0x7fa2bc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:05:24.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c1b9c700 1 --2- 172.21.15.102:0/728585919 >> v2:172.21.15.102:3300/0 conn(0x7fa2bc1564b0 0x7fa2bc0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:24.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2b3fff700 1 -- 172.21.15.102:0/728585919 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556631016 0 0) 0x7fa2bc0c9ca0 con 0x7fa2bc0c9610 2023-03-03T13:05:24.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c1b9c700 1 -- 172.21.15.102:0/728585919 >> v1:172.21.15.102:6789/0 conn(0x7fa2bc0c9610 legacy=0x7fa2bc0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:24.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c1b9c700 1 -- 172.21.15.102:0/728585919 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0b46f0 con 0x7fa2bc0cab50 2023-03-03T13:05:24.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.983+0000 7fa2c1b9c700 1 --2- 172.21.15.102:0/728585919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0cab50 0x7fa2bc0cbba0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009d90 tx=0x7fa2b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8d9c7cb20345240 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:24.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2b3fff700 1 -- 172.21.15.102:0/728585919 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b400e070 con 0x7fa2bc0cab50 2023-03-03T13:05:24.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2b3fff700 1 -- 172.21.15.102:0/728585919 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b400bb20 con 0x7fa2bc0cab50 2023-03-03T13:05:24.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2b3fff700 1 -- 172.21.15.102:0/728585919 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4012430 con 0x7fa2bc0cab50 2023-03-03T13:05:24.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2c239d700 1 -- 172.21.15.102:0/728585919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0cab50 msgr2=0x7fa2bc0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:24.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2c239d700 1 --2- 172.21.15.102:0/728585919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0cab50 0x7fa2bc0cbba0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009d90 tx=0x7fa2b400b330 comp rx=0 tx=0).stop 2023-03-03T13:05:24.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2c239d700 1 -- 172.21.15.102:0/728585919 shutdown_connections 2023-03-03T13:05:24.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2c239d700 1 --2- 172.21.15.102:0/728585919 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0cab50 0x7fa2bc0cbba0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:24.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2c239d700 1 --2- 172.21.15.102:0/728585919 >> v2:172.21.15.102:3300/0 conn(0x7fa2bc1564b0 0x7fa2bc0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:24.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2c239d700 1 -- 172.21.15.102:0/728585919 >> 172.21.15.102:0/728585919 conn(0x7fa2bc065fb0 msgr2=0x7fa2bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:24.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.984+0000 7fa2c239d700 1 -- 172.21.15.102:0/728585919 shutdown_connections 2023-03-03T13:05:24.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.985+0000 7fa2c239d700 1 -- 172.21.15.102:0/728585919 wait complete. 2023-03-03T13:05:24.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.987+0000 7fa2c239d700 1 Processor -- start 2023-03-03T13:05:24.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.988+0000 7fa2c239d700 1 -- start start 2023-03-03T13:05:24.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c239d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0c9610 0x7fa2bc066890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:24.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c239d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc0cab50 0x7fa2bc15c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:24.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c239d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1564b0 0x7fa2bc0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:24.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c239d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc15b290 con 0x7fa2bc1564b0 2023-03-03T13:05:24.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c239d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc15cd30 con 0x7fa2bc0cab50 2023-03-03T13:05:24.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c239d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc15d010 con 0x7fa2bc0c9610 2023-03-03T13:05:24.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c139b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0c9610 0x7fa2bc066890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:24.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c1b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1564b0 0x7fa2bc0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:24.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c0b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc0cab50 0x7fa2bc15c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:24.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c139b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0c9610 0x7fa2bc066890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:52028/0 (socket says 172.21.15.102:52028) 2023-03-03T13:05:24.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.989+0000 7fa2c139b700 1 -- 172.21.15.102:0/3917644060 learned_addr learned my addr 172.21.15.102:0/3917644060 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:05:24.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2c139b700 1 -- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc0cab50 msgr2=0x7fa2bc15c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:24.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2c139b700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc0cab50 0x7fa2bc15c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2c139b700 1 -- 172.21.15.102:0/3917644060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1564b0 msgr2=0x7fa2bc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2c139b700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1564b0 0x7fa2bc0b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2c139b700 1 -- 172.21.15.102:0/3917644060 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b4009a70 con 0x7fa2bc0c9610 2023-03-03T13:05:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2c139b700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0c9610 0x7fa2bc066890 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800e9b0 tx=0x7fa2b800ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2b1ffb700 1 -- 172.21.15.102:0/3917644060 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8009710 con 0x7fa2bc0c9610 2023-03-03T13:05:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2b1ffb700 1 -- 172.21.15.102:0/3917644060 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b800ee50 con 0x7fa2bc0c9610 2023-03-03T13:05:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2b1ffb700 1 -- 172.21.15.102:0/3917644060 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b800ca50 con 0x7fa2bc0c9610 2023-03-03T13:05:24.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.990+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc0b2210 con 0x7fa2bc0c9610 2023-03-03T13:05:24.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.991+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc0b1f30 con 0x7fa2bc0c9610 2023-03-03T13:05:24.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.992+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc156880 con 0x7fa2bc0c9610 2023-03-03T13:05:24.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.993+0000 7fa2b1ffb700 1 -- 172.21.15.102:0/3917644060 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa2b8019030 con 0x7fa2bc0c9610 2023-03-03T13:05:24.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.994+0000 7fa2b1ffb700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2a00682f0 0x7fa2a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:24.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.994+0000 7fa2b1ffb700 1 -- 172.21.15.102:0/3917644060 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa2b8085c80 con 0x7fa2bc0c9610 2023-03-03T13:05:24.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.995+0000 7fa2c0b9a700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2a00682f0 0x7fa2a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:24.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.995+0000 7fa2c0b9a700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2a00682f0 0x7fa2a006a7a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0cc6a0 tx=0x7fa2ac007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:24.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:24.998+0000 7fa2b1ffb700 1 -- 172.21.15.102:0/3917644060 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa2b8052390 con 0x7fa2bc0c9610 2023-03-03T13:05:25.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.236+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2bc0677f0 con 0x7fa2bc0c9610 2023-03-03T13:05:25.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.237+0000 7fa2b1ffb700 1 -- 172.21.15.102:0/3917644060 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa2b80559b0 con 0x7fa2bc0c9610 2023-03-03T13:05:25.239 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:05:25.239 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:05:25.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2a00682f0 msgr2=0x7fa2a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:25.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2a00682f0 0x7fa2a006a7a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0cc6a0 tx=0x7fa2ac007e70 comp rx=0 tx=0).stop 2023-03-03T13:05:25.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0c9610 msgr2=0x7fa2bc066890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:25.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0c9610 0x7fa2bc066890 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800e9b0 tx=0x7fa2b800ecc0 comp rx=0 tx=0).stop 2023-03-03T13:05:25.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 shutdown_connections 2023-03-03T13:05:25.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa2a00682f0 0x7fa2a006a7a0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:25.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa2bc0c9610 0x7fa2bc066890 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:25.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1564b0 0x7fa2bc0b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:25.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 --2- 172.21.15.102:0/3917644060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2bc0cab50 0x7fa2bc15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:25.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 >> 172.21.15.102:0/3917644060 conn(0x7fa2bc065fb0 msgr2=0x7fa2bc157510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:25.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 shutdown_connections 2023-03-03T13:05:25.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:05:25.240+0000 7fa2c239d700 1 -- 172.21.15.102:0/3917644060 wait complete. 2023-03-03T13:05:25.244 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:05:25.269 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T13:05:41.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:41.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:11.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:11.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:27.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:06:27.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.532+0000 7f6a54bff700 1 Processor -- start 2023-03-03T13:06:27.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.532+0000 7f6a54bff700 1 -- start start 2023-03-03T13:06:27.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a54bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500c9cb0 0x7f6a500cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:27.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a54bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a500ca190 0x7f6a500cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:27.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a54bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a500cf940 con 0x7f6a50153260 2023-03-03T13:06:27.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a54bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a500cfe70 con 0x7f6a500ca190 2023-03-03T13:06:27.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a54bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a50158d70 con 0x7f6a500c9cb0 2023-03-03T13:06:27.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a4e59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6a50153260 0x7f6a500ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42922/0 (socket says 172.21.15.102:42922) 2023-03-03T13:06:27.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a4e59c700 1 -- 172.21.15.102:0/796607767 learned_addr learned my addr 172.21.15.102:0/796607767 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:06:27.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a4ed9d700 1 --2- 172.21.15.102:0/796607767 >> v2:172.21.15.102:3300/0 conn(0x7f6a500ca190 0x7f6a500cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:27.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.534+0000 7f6a4dd9b700 1 --2- 172.21.15.102:0/796607767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500c9cb0 0x7f6a500cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:27.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4ed9d700 1 -- 172.21.15.102:0/796607767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500c9cb0 msgr2=0x7f6a500cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:06:27.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4ed9d700 1 --2- 172.21.15.102:0/796607767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500c9cb0 0x7f6a500cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4ed9d700 1 -- 172.21.15.102:0/796607767 >> v1:172.21.15.102:6789/0 conn(0x7f6a50153260 legacy=0x7f6a500ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:27.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4ed9d700 1 -- 172.21.15.102:0/796607767 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a500cfb90 con 0x7f6a500ca190 2023-03-03T13:06:27.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4d59a700 1 -- 172.21.15.102:0/796607767 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390157612 0 0) 0x7f6a500cf940 con 0x7f6a50153260 2023-03-03T13:06:27.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4ed9d700 1 --2- 172.21.15.102:0/796607767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a500ca190 0x7f6a500cbdc0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009d90 tx=0x7f6a4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3f2a6b5af9b477 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:27.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4d59a700 1 -- 172.21.15.102:0/796607767 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4800e070 con 0x7f6a500ca190 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4d59a700 1 -- 172.21.15.102:0/796607767 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4800bb20 con 0x7f6a500ca190 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.535+0000 7f6a4d59a700 1 -- 172.21.15.102:0/796607767 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48012430 con 0x7f6a500ca190 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 -- 172.21.15.102:0/796607767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a500ca190 msgr2=0x7f6a500cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 --2- 172.21.15.102:0/796607767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a500ca190 0x7f6a500cbdc0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009d90 tx=0x7f6a4800b330 comp rx=0 tx=0).stop 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 -- 172.21.15.102:0/796607767 shutdown_connections 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 --2- 172.21.15.102:0/796607767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a500ca190 0x7f6a500cbdc0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 --2- 172.21.15.102:0/796607767 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500c9cb0 0x7f6a500cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 -- 172.21.15.102:0/796607767 >> 172.21.15.102:0/796607767 conn(0x7f6a50066c00 msgr2=0x7f6a500ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:06:27.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 -- 172.21.15.102:0/796607767 shutdown_connections 2023-03-03T13:06:27.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.536+0000 7f6a54bff700 1 -- 172.21.15.102:0/796607767 wait complete. 2023-03-03T13:06:27.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.538+0000 7f6a54bff700 1 Processor -- start 2023-03-03T13:06:27.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.538+0000 7f6a54bff700 1 -- start start 2023-03-03T13:06:27.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.539+0000 7f6a54bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500ca190 0x7f6a5014da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:27.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.539+0000 7f6a54bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a50153260 0x7f6a5014e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:27.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.539+0000 7f6a54bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a50154730 0x7f6a5014e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:27.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.539+0000 7f6a54bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5014dfc0 con 0x7f6a50153260 2023-03-03T13:06:27.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.539+0000 7f6a54bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5014c650 con 0x7f6a50154730 2023-03-03T13:06:27.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.539+0000 7f6a54bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a500c5b60 con 0x7f6a500ca190 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a50153260 0x7f6a5014e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a50153260 0x7f6a5014e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46756/0 (socket says 172.21.15.102:46756) 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 -- 172.21.15.102:0/1865361104 learned_addr learned my addr 172.21.15.102:0/1865361104 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4e59c700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500ca190 0x7f6a5014da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4ed9d700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a50154730 0x7f6a5014e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 -- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500ca190 msgr2=0x7f6a5014da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500ca190 0x7f6a5014da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 -- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a50154730 msgr2=0x7f6a5014e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:27.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a50154730 0x7f6a5014e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.540+0000 7f6a4dd9b700 1 -- 172.21.15.102:0/1865361104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a48009a70 con 0x7f6a50153260 2023-03-03T13:06:27.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.541+0000 7f6a4dd9b700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a50153260 0x7f6a5014e3d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400e870 tx=0x7f6a4400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:27.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.541+0000 7f6a3effd700 1 -- 172.21.15.102:0/1865361104 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44009590 con 0x7f6a50153260 2023-03-03T13:06:27.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.541+0000 7f6a3effd700 1 -- 172.21.15.102:0/1865361104 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a44013680 con 0x7f6a50153260 2023-03-03T13:06:27.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.541+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a50159260 con 0x7f6a50153260 2023-03-03T13:06:27.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.541+0000 7f6a3effd700 1 -- 172.21.15.102:0/1865361104 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440139f0 con 0x7f6a50153260 2023-03-03T13:06:27.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.541+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a50158f80 con 0x7f6a50153260 2023-03-03T13:06:27.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.542+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a50159540 con 0x7f6a50153260 2023-03-03T13:06:27.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.543+0000 7f6a3effd700 1 -- 172.21.15.102:0/1865361104 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a44019030 con 0x7f6a50153260 2023-03-03T13:06:27.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.545+0000 7f6a3effd700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a30068320 0x7f6a3006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:27.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.545+0000 7f6a3effd700 1 -- 172.21.15.102:0/1865361104 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6a44085a30 con 0x7f6a50153260 2023-03-03T13:06:27.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.545+0000 7f6a4e59c700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a30068320 0x7f6a3006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:27.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.546+0000 7f6a4e59c700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a30068320 0x7f6a3006a7d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6a40007fd0 tx=0x7f6a40007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:27.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.549+0000 7f6a3effd700 1 -- 172.21.15.102:0/1865361104 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a44052140 con 0x7f6a50153260 2023-03-03T13:06:27.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.783+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a50067ce0 con 0x7f6a50153260 2023-03-03T13:06:27.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.784+0000 7f6a3effd700 1 -- 172.21.15.102:0/1865361104 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6a44055760 con 0x7f6a50153260 2023-03-03T13:06:27.787 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:06:27.787 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:06:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a30068320 msgr2=0x7f6a3006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a30068320 0x7f6a3006a7d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6a40007fd0 tx=0x7f6a40007de0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a50153260 msgr2=0x7f6a5014e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a50153260 0x7f6a5014e3d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400e870 tx=0x7f6a4400eb80 comp rx=0 tx=0).stop 2023-03-03T13:06:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 shutdown_connections 2023-03-03T13:06:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6a30068320 0x7f6a3006a7d0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a500ca190 0x7f6a5014da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a50154730 0x7f6a5014e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 --2- 172.21.15.102:0/1865361104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a50153260 0x7f6a5014e3d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 >> 172.21.15.102:0/1865361104 conn(0x7f6a500c9cb0 msgr2=0x7f6a500cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:06:27.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 shutdown_connections 2023-03-03T13:06:27.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:06:27.787+0000 7f6a54bff700 1 -- 172.21.15.102:0/1865361104 wait complete. 2023-03-03T13:06:27.791 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:06:27.816 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 41 secs before thrashing 2023-03-03T13:06:41.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:41.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:08.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:07:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.064+0000 7f92509fe700 1 Processor -- start 2023-03-03T13:07:09.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.065+0000 7f92509fe700 1 -- start start 2023-03-03T13:07:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f92509fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c0c9610 0x7f924c0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f92509fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f924c0ca980 0x7f924c1636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f92509fe700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f924c0c9ca0 con 0x7f924c1564b0 2023-03-03T13:07:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f92509fe700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924c0cadc0 con 0x7f924c0ca980 2023-03-03T13:07:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f92509fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c0b4920 con 0x7f924c0c9610 2023-03-03T13:07:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f924ad9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f924c1564b0 0x7f924c0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38556/0 (socket says 172.21.15.102:38556) 2023-03-03T13:07:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f924ad9d700 1 -- 172.21.15.102:0/1107015839 learned_addr learned my addr 172.21.15.102:0/1107015839 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:07:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f9243fff700 1 --2- 172.21.15.102:0/1107015839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c0c9610 0x7f924c0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:09.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.066+0000 7f924b59e700 1 --2- 172.21.15.102:0/1107015839 >> v2:172.21.15.102:3300/0 conn(0x7f924c0ca980 0x7f924c1636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.067+0000 7f924a59c700 1 -- 172.21.15.102:0/1107015839 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621463947 0 0) 0x7f924c0c9ca0 con 0x7f924c1564b0 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.067+0000 7f924a59c700 1 -- 172.21.15.102:0/1107015839 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92380035b0 con 0x7f924c1564b0 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.067+0000 7f924b59e700 1 -- 172.21.15.102:0/1107015839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c0c9610 msgr2=0x7f924c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.067+0000 7f924b59e700 1 --2- 172.21.15.102:0/1107015839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c0c9610 0x7f924c0ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.067+0000 7f924b59e700 1 -- 172.21.15.102:0/1107015839 >> v1:172.21.15.102:6789/0 conn(0x7f924c1564b0 legacy=0x7f924c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.067+0000 7f924b59e700 1 -- 172.21.15.102:0/1107015839 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c0b46f0 con 0x7f924c0ca980 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.067+0000 7f924b59e700 1 --2- 172.21.15.102:0/1107015839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0ca980 0x7f924c1636a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9244009d90 tx=0x7f924400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73770e734cd10692 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:07:09.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f924a59c700 1 -- 172.21.15.102:0/1107015839 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924400e070 con 0x7f924c0ca980 2023-03-03T13:07:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f924a59c700 1 -- 172.21.15.102:0/1107015839 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924400bb20 con 0x7f924c0ca980 2023-03-03T13:07:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f924a59c700 1 -- 172.21.15.102:0/1107015839 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9244012430 con 0x7f924c0ca980 2023-03-03T13:07:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f92509fe700 1 -- 172.21.15.102:0/1107015839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0ca980 msgr2=0x7f924c1636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f92509fe700 1 --2- 172.21.15.102:0/1107015839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0ca980 0x7f924c1636a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9244009d90 tx=0x7f924400b330 comp rx=0 tx=0).stop 2023-03-03T13:07:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f92509fe700 1 -- 172.21.15.102:0/1107015839 shutdown_connections 2023-03-03T13:07:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f92509fe700 1 --2- 172.21.15.102:0/1107015839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0ca980 0x7f924c1636a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f92509fe700 1 --2- 172.21.15.102:0/1107015839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c0c9610 0x7f924c0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f92509fe700 1 -- 172.21.15.102:0/1107015839 >> 172.21.15.102:0/1107015839 conn(0x7f924c065fb0 msgr2=0x7f924c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:07:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.068+0000 7f92509fe700 1 -- 172.21.15.102:0/1107015839 shutdown_connections 2023-03-03T13:07:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.069+0000 7f92509fe700 1 -- 172.21.15.102:0/1107015839 wait complete. 2023-03-03T13:07:09.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.070+0000 7f92509fe700 1 Processor -- start 2023-03-03T13:07:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.071+0000 7f92509fe700 1 -- start start 2023-03-03T13:07:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f92509fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0c9610 0x7f924c1f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f92509fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f924c0ca980 0x7f924c1df0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f92509fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c1564b0 0x7f924c144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f92509fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c066c00 con 0x7f924c0c9610 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f92509fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c156b90 con 0x7f924c0ca980 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f92509fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c1f1ab0 con 0x7f924c1564b0 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0c9610 0x7f924c1f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c1564b0 0x7f924c144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0c9610 0x7f924c1f1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42084/0 (socket says 172.21.15.102:42084) 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f9243fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f924c0ca980 0x7f924c1df0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 -- 172.21.15.102:0/887689808 learned_addr learned my addr 172.21.15.102:0/887689808 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:07:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c1564b0 0x7f924c144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58578/0 (socket says 172.21.15.102:58578) 2023-03-03T13:07:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 -- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c1564b0 msgr2=0x7f924c144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c1564b0 0x7f924c144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 -- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f924c0ca980 msgr2=0x7f924c1df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f924c0ca980 0x7f924c1df0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924ad9d700 1 -- 172.21.15.102:0/887689808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9244009a70 con 0x7f924c0c9610 2023-03-03T13:07:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.072+0000 7f924b59e700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c1564b0 0x7f924c144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:07:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.073+0000 7f924ad9d700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0c9610 0x7f924c1f1570 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f923c001eb0 tx=0x7f923c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:07:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.073+0000 7f92437fe700 1 -- 172.21.15.102:0/887689808 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c005d70 con 0x7f924c0c9610 2023-03-03T13:07:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.073+0000 7f92437fe700 1 -- 172.21.15.102:0/887689808 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c0070d0 con 0x7f924c0c9610 2023-03-03T13:07:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.073+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924c0b1f30 con 0x7f924c0c9610 2023-03-03T13:07:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.073+0000 7f92437fe700 1 -- 172.21.15.102:0/887689808 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c007460 con 0x7f924c0c9610 2023-03-03T13:07:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.073+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c0b4c00 con 0x7f924c0c9610 2023-03-03T13:07:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.074+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924c0b2210 con 0x7f924c0c9610 2023-03-03T13:07:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.075+0000 7f92437fe700 1 -- 172.21.15.102:0/887689808 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f923c019030 con 0x7f924c0c9610 2023-03-03T13:07:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.076+0000 7f92437fe700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f922c068320 0x7f922c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.077+0000 7f92437fe700 1 -- 172.21.15.102:0/887689808 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f923c085060 con 0x7f924c0c9610 2023-03-03T13:07:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.077+0000 7f9243fff700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f922c068320 0x7f922c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.078+0000 7f9243fff700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f922c068320 0x7f922c06a7d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f924c0cdfa0 tx=0x7f9234009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:07:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.081+0000 7f92437fe700 1 -- 172.21.15.102:0/887689808 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f923c051770 con 0x7f924c0c9610 2023-03-03T13:07:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.320+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f924c0677f0 con 0x7f924c0c9610 2023-03-03T13:07:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.320+0000 7f92437fe700 1 -- 172.21.15.102:0/887689808 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f923c017090 con 0x7f924c0c9610 2023-03-03T13:07:09.323 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:07:09.323 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:07:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f922c068320 msgr2=0x7f922c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f922c068320 0x7f922c06a7d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f924c0cdfa0 tx=0x7f9234009210 comp rx=0 tx=0).stop 2023-03-03T13:07:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0c9610 msgr2=0x7f924c1f1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0c9610 0x7f924c1f1570 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f923c001eb0 tx=0x7f923c00c7b0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 shutdown_connections 2023-03-03T13:07:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f922c068320 0x7f922c06a7d0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f924c1564b0 0x7f924c144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924c0c9610 0x7f924c1f1570 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 --2- 172.21.15.102:0/887689808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f924c0ca980 0x7f924c1df0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:09.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 >> 172.21.15.102:0/887689808 conn(0x7f924c065fb0 msgr2=0x7f924c06b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:07:09.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 shutdown_connections 2023-03-03T13:07:09.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:07:09.323+0000 7f92509fe700 1 -- 172.21.15.102:0/887689808 wait complete. 2023-03-03T13:07:09.327 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:07:09.352 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-03-03T13:07:11.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:11.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:41.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:41.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:12.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:12.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:42.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:42.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:56.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:08:56.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.605+0000 7febbd39d700 1 Processor -- start 2023-03-03T13:08:56.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.605+0000 7febbd39d700 1 -- start start 2023-03-03T13:08:56.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febbd39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80ced00 0x7febb80c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:56.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febbd39d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7febb80c41b0 0x7febb80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:56.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febbd39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80c3d20 con 0x7febb80b3970 2023-03-03T13:08:56.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febbd39d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb80c4930 con 0x7febb80c41b0 2023-03-03T13:08:56.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febbd39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80cf610 con 0x7febb80ced00 2023-03-03T13:08:56.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febb77fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7febb80b3970 0x7febb8158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47420/0 (socket says 172.21.15.102:47420) 2023-03-03T13:08:56.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febb77fe700 1 -- 172.21.15.102:0/2723201904 learned_addr learned my addr 172.21.15.102:0/2723201904 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:08:56.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febbcb9c700 1 --2- 172.21.15.102:0/2723201904 >> v2:172.21.15.172:3300/0 conn(0x7febb80c41b0 0x7febb80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:56.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.606+0000 7febb7fff700 1 --2- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80ced00 0x7febb80c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.607+0000 7febb6ffd700 1 -- 172.21.15.102:0/2723201904 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631790917 0 0) 0x7febb80c3d20 con 0x7febb80b3970 2023-03-03T13:08:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.607+0000 7febbcb9c700 1 -- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80ced00 msgr2=0x7febb80c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.607+0000 7febbcb9c700 1 --2- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80ced00 0x7febb80c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.607+0000 7febbcb9c700 1 -- 172.21.15.102:0/2723201904 >> v1:172.21.15.102:6789/0 conn(0x7febb80b3970 legacy=0x7febb8158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.607+0000 7febbcb9c700 1 -- 172.21.15.102:0/2723201904 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb815cba0 con 0x7febb80c41b0 2023-03-03T13:08:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.607+0000 7febbcb9c700 1 --2- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80c41b0 0x7febb80cf0d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7febb0009dc0 tx=0x7febb000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eecfad794de2059 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.607+0000 7febb6ffd700 1 -- 172.21.15.102:0/2723201904 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb000e070 con 0x7febb80c41b0 2023-03-03T13:08:56.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febb6ffd700 1 -- 172.21.15.102:0/2723201904 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb000bae0 con 0x7febb80c41b0 2023-03-03T13:08:56.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febb6ffd700 1 -- 172.21.15.102:0/2723201904 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb000be90 con 0x7febb80c41b0 2023-03-03T13:08:56.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 -- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80c41b0 msgr2=0x7febb80cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:56.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 --2- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80c41b0 0x7febb80cf0d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7febb0009dc0 tx=0x7febb000b330 comp rx=0 tx=0).stop 2023-03-03T13:08:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 -- 172.21.15.102:0/2723201904 shutdown_connections 2023-03-03T13:08:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 --2- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80c41b0 0x7febb80cf0d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 --2- 172.21.15.102:0/2723201904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80ced00 0x7febb80c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 -- 172.21.15.102:0/2723201904 >> 172.21.15.102:0/2723201904 conn(0x7febb808b1c0 msgr2=0x7febb8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 -- 172.21.15.102:0/2723201904 shutdown_connections 2023-03-03T13:08:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.608+0000 7febbd39d700 1 -- 172.21.15.102:0/2723201904 wait complete. 2023-03-03T13:08:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.611+0000 7febbd39d700 1 Processor -- start 2023-03-03T13:08:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.611+0000 7febbd39d700 1 -- start start 2023-03-03T13:08:56.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febbd39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80b3970 0x7febb8159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:56.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febbd39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febb80c41b0 0x7febb815aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:56.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febbd39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80ced00 0x7febb815b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:56.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febbd39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb8159a30 con 0x7febb80c41b0 2023-03-03T13:08:56.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febbd39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb81480c0 con 0x7febb80ced00 2023-03-03T13:08:56.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febbd39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb8148360 con 0x7febb80b3970 2023-03-03T13:08:56.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febb77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febb80c41b0 0x7febb815aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:56.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febb7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80b3970 0x7febb8159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:56.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febb7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80b3970 0x7febb8159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48794/0 (socket says 172.21.15.102:48794) 2023-03-03T13:08:56.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febb7fff700 1 -- 172.21.15.102:0/1112385204 learned_addr learned my addr 172.21.15.102:0/1112385204 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:08:56.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.613+0000 7febbcb9c700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80ced00 0x7febb815b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:56.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb7fff700 1 -- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80ced00 msgr2=0x7febb815b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:56.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb7fff700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80ced00 0x7febb815b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb7fff700 1 -- 172.21.15.102:0/1112385204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febb80c41b0 msgr2=0x7febb815aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:56.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb7fff700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febb80c41b0 0x7febb815aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb7fff700 1 -- 172.21.15.102:0/1112385204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb0009a70 con 0x7febb80b3970 2023-03-03T13:08:56.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb7fff700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80b3970 0x7febb8159f30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feba800e9b0 tx=0x7feba800ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:56.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb4ff9700 1 -- 172.21.15.102:0/1112385204 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8009710 con 0x7febb80b3970 2023-03-03T13:08:56.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb4ff9700 1 -- 172.21.15.102:0/1112385204 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba800ee50 con 0x7febb80b3970 2023-03-03T13:08:56.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febb4ff9700 1 -- 172.21.15.102:0/1112385204 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba800ca50 con 0x7febb80b3970 2023-03-03T13:08:56.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb806bf20 con 0x7febb80b3970 2023-03-03T13:08:56.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.614+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb815ce80 con 0x7febb80b3970 2023-03-03T13:08:56.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.616+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb806c200 con 0x7febb80b3970 2023-03-03T13:08:56.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.617+0000 7febb4ff9700 1 -- 172.21.15.102:0/1112385204 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feba8023070 con 0x7febb80b3970 2023-03-03T13:08:56.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.618+0000 7febb4ff9700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feba4068320 0x7feba406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:56.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.618+0000 7febb4ff9700 1 -- 172.21.15.102:0/1112385204 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7feba8085f80 con 0x7febb80b3970 2023-03-03T13:08:56.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.618+0000 7febb77fe700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feba4068320 0x7feba406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:56.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.619+0000 7febb77fe700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feba4068320 0x7feba406a7d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7febb8159c50 tx=0x7febac007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:56.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.623+0000 7febb4ff9700 1 -- 172.21.15.102:0/1112385204 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feba8052630 con 0x7febb80b3970 2023-03-03T13:08:56.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.862+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febb804e6e0 con 0x7febb80b3970 2023-03-03T13:08:56.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.863+0000 7febb4ff9700 1 -- 172.21.15.102:0/1112385204 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7feba8055c50 con 0x7febb80b3970 2023-03-03T13:08:56.866 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:08:56.866 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:08:56.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feba4068320 msgr2=0x7feba406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:56.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feba4068320 0x7feba406a7d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7febb8159c50 tx=0x7febac007e70 comp rx=0 tx=0).stop 2023-03-03T13:08:56.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80b3970 msgr2=0x7febb8159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:56.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80b3970 0x7febb8159f30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feba800e9b0 tx=0x7feba800ecc0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 shutdown_connections 2023-03-03T13:08:56.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feba4068320 0x7feba406a7d0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febb80b3970 0x7febb8159f30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febb80ced00 0x7febb815b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 --2- 172.21.15.102:0/1112385204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febb80c41b0 0x7febb815aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:56.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 >> 172.21.15.102:0/1112385204 conn(0x7febb808b1c0 msgr2=0x7febb80ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:56.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 shutdown_connections 2023-03-03T13:08:56.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:08:56.866+0000 7febbd39d700 1 -- 172.21.15.102:0/1112385204 wait complete. 2023-03-03T13:08:56.870 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:08:56.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 20 secs before thrashing 2023-03-03T13:09:12.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:12.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:16.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:09:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.166+0000 7f538179d700 1 Processor -- start 2023-03-03T13:09:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.167+0000 7f538179d700 1 -- start start 2023-03-03T13:09:17.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.168+0000 7f538179d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f537c0ced00 0x7f537c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.168+0000 7f538179d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f537c0b3970 0x7f537c0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.168+0000 7f538179d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c0c3d20 con 0x7f537c0cf0d0 2023-03-03T13:09:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.168+0000 7f538179d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f537c0c4930 con 0x7f537c0ced00 2023-03-03T13:09:17.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.168+0000 7f538179d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f537c0cf570 con 0x7f537c0b3970 2023-03-03T13:09:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.168+0000 7f537bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f537c0ced00 0x7f537c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.168+0000 7f5373fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f537c0b3970 0x7f537c0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f5373fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f537c0b3970 0x7f537c0c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47824/0 (socket says 172.21.15.102:47824) 2023-03-03T13:09:17.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f537bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f537c0ced00 0x7f537c0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59446/0 (socket says 172.21.15.102:59446) 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f5380f9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f537c0cf0d0 0x7f537c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60828/0 (socket says 172.21.15.102:60828) 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f5373fff700 1 -- 172.21.15.102:0/4168251004 learned_addr learned my addr 172.21.15.102:0/4168251004 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f537bfff700 1 -- 172.21.15.102:0/4168251004 >> v2:172.21.15.172:3300/0 conn(0x7f537c0b3970 msgr2=0x7f537c0c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f537bfff700 1 --2- 172.21.15.102:0/4168251004 >> v2:172.21.15.172:3300/0 conn(0x7f537c0b3970 0x7f537c0c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f537bfff700 1 -- 172.21.15.102:0/4168251004 >> v1:172.21.15.102:6789/0 conn(0x7f537c0cf0d0 legacy=0x7f537c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f537bfff700 1 -- 172.21.15.102:0/4168251004 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c15cba0 con 0x7f537c0ced00 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f5373fff700 1 --2- 172.21.15.102:0/4168251004 >> v2:172.21.15.172:3300/0 conn(0x7f537c0b3970 0x7f537c0c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:09:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.169+0000 7f537bfff700 1 --2- 172.21.15.102:0/4168251004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c0c5a90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f537c0c4ba0 tx=0x7f536c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d2d9e42274f797 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:17.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f537b7fe700 1 -- 172.21.15.102:0/4168251004 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c011830 con 0x7f537c0ced00 2023-03-03T13:09:17.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f537b7fe700 1 -- 172.21.15.102:0/4168251004 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c011990 con 0x7f537c0ced00 2023-03-03T13:09:17.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f537b7fe700 1 -- 172.21.15.102:0/4168251004 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c011d20 con 0x7f537c0ced00 2023-03-03T13:09:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f538179d700 1 -- 172.21.15.102:0/4168251004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 msgr2=0x7f537c0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f538179d700 1 --2- 172.21.15.102:0/4168251004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c0c5a90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f537c0c4ba0 tx=0x7f536c009b20 comp rx=0 tx=0).stop 2023-03-03T13:09:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f538179d700 1 -- 172.21.15.102:0/4168251004 shutdown_connections 2023-03-03T13:09:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f538179d700 1 --2- 172.21.15.102:0/4168251004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c0c5a90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f538179d700 1 --2- 172.21.15.102:0/4168251004 >> v2:172.21.15.172:3300/0 conn(0x7f537c0b3970 0x7f537c0c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f538179d700 1 -- 172.21.15.102:0/4168251004 >> 172.21.15.102:0/4168251004 conn(0x7f537c08b1c0 msgr2=0x7f537c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.170+0000 7f538179d700 1 -- 172.21.15.102:0/4168251004 shutdown_connections 2023-03-03T13:09:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.171+0000 7f538179d700 1 -- 172.21.15.102:0/4168251004 wait complete. 2023-03-03T13:09:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.172+0000 7f538179d700 1 Processor -- start 2023-03-03T13:09:17.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.173+0000 7f538179d700 1 -- start start 2023-03-03T13:09:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f538179d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f537c0b3970 0x7f537c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f538179d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c15b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:17.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f538179d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f537c0cf0d0 0x7f537c0c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f538179d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c1512c0 con 0x7f537c0ced00 2023-03-03T13:09:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f538179d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c071c50 con 0x7f537c0b3970 2023-03-03T13:09:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f538179d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537c151d90 con 0x7f537c0cf0d0 2023-03-03T13:09:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f5373fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c15b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f537bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f537c0b3970 0x7f537c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f5380f9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f537c0cf0d0 0x7f537c0c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:17.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f537bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f537c0b3970 0x7f537c14dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47840/0 (socket says 172.21.15.102:47840) 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.174+0000 7f537bfff700 1 -- 172.21.15.102:0/1241145652 learned_addr learned my addr 172.21.15.102:0/1241145652 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f537bfff700 1 -- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f537c0cf0d0 msgr2=0x7f537c0c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f537bfff700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f537c0cf0d0 0x7f537c0c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f537bfff700 1 -- 172.21.15.102:0/1241145652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 msgr2=0x7f537c15b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f537bfff700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c15b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f537bfff700 1 -- 172.21.15.102:0/1241145652 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364009550 con 0x7f537c0b3970 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f5380f9c700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f537c0cf0d0 0x7f537c0c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:09:17.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f5373fff700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c15b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:09:17.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f537bfff700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f537c0b3970 0x7f537c14dfe0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f536c0090b0 tx=0x7f536c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:17.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f53797fa700 1 -- 172.21.15.102:0/1241145652 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c0036a0 con 0x7f537c0b3970 2023-03-03T13:09:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f53797fa700 1 -- 172.21.15.102:0/1241145652 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c01e070 con 0x7f537c0b3970 2023-03-03T13:09:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f53797fa700 1 -- 172.21.15.102:0/1241145652 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c003a60 con 0x7f537c0b3970 2023-03-03T13:09:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.175+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c0094a0 con 0x7f537c0b3970 2023-03-03T13:09:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.176+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537c06bf20 con 0x7f537c0b3970 2023-03-03T13:09:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.177+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537c15ce80 con 0x7f537c0b3970 2023-03-03T13:09:17.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.178+0000 7f53797fa700 1 -- 172.21.15.102:0/1241145652 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f536c00f050 con 0x7f537c0b3970 2023-03-03T13:09:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.179+0000 7f53797fa700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f535c0682f0 0x7f535c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.179+0000 7f53797fa700 1 -- 172.21.15.102:0/1241145652 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f536c086c00 con 0x7f537c0b3970 2023-03-03T13:09:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.180+0000 7f5373fff700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f535c0682f0 0x7f535c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:17.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.180+0000 7f5373fff700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f535c0682f0 0x7f535c06a7a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f537c0b3e50 tx=0x7f53640092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:17.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.184+0000 7f53797fa700 1 -- 172.21.15.102:0/1241145652 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f536c053310 con 0x7f537c0b3970 2023-03-03T13:09:17.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.422+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f537c06c200 con 0x7f537c0b3970 2023-03-03T13:09:17.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.423+0000 7f53797fa700 1 -- 172.21.15.102:0/1241145652 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f536c020090 con 0x7f537c0b3970 2023-03-03T13:09:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:09:17.425 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:09:17.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f535c0682f0 msgr2=0x7f535c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:17.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f535c0682f0 0x7f535c06a7a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f537c0b3e50 tx=0x7f53640092c0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f537c0b3970 msgr2=0x7f537c14dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:17.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f537c0b3970 0x7f537c14dfe0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f536c0090b0 tx=0x7f536c004750 comp rx=0 tx=0).stop 2023-03-03T13:09:17.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 shutdown_connections 2023-03-03T13:09:17.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f535c0682f0 0x7f535c06a7a0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f537c0cf0d0 0x7f537c0c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f537c0b3970 0x7f537c14dfe0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.425+0000 7f538179d700 1 --2- 172.21.15.102:0/1241145652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f537c0ced00 0x7f537c15b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:17.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.426+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 >> 172.21.15.102:0/1241145652 conn(0x7f537c08b1c0 msgr2=0x7f537c0cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:17.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.426+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 shutdown_connections 2023-03-03T13:09:17.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:09:17.426+0000 7f538179d700 1 -- 172.21.15.102:0/1241145652 wait complete. 2023-03-03T13:09:17.429 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:09:17.454 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 54 secs before thrashing 2023-03-03T13:09:42.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:42.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T13:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:10:11.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:10:11.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.716+0000 7f90163ff700 1 Processor -- start 2023-03-03T13:10:11.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.716+0000 7f90163ff700 1 -- start start 2023-03-03T13:10:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.717+0000 7f90163ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f90100cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f90163ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90100cb9f0 con 0x7f90100ced80 2023-03-03T13:10:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f90163ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9010159c80 con 0x7f90100cf9a0 2023-03-03T13:10:11.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f90163ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90100ca750 con 0x7f90101597f0 2023-03-03T13:10:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f900ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f90100ced80 0x7f90100cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54666/0 (socket says 172.21.15.102:54666) 2023-03-03T13:10:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f900ffff700 1 -- 172.21.15.102:0/2525077339 learned_addr learned my addr 172.21.15.102:0/2525077339 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:10:11.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f900f7fe700 1 --2- 172.21.15.102:0/2525077339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f90100cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f900effd700 1 -- 172.21.15.102:0/2525077339 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578800712 0 0) 0x7f90100cb9f0 con 0x7f90100ced80 2023-03-03T13:10:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f900effd700 1 -- 172.21.15.102:0/2525077339 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff80035b0 con 0x7f90100ced80 2023-03-03T13:10:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f900effd700 1 -- 172.21.15.102:0/2525077339 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701702342 0 0) 0x7f9010159c80 con 0x7f90100cf9a0 2023-03-03T13:10:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.718+0000 7f900effd700 1 -- 172.21.15.102:0/2525077339 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90100cb9f0 con 0x7f90100cf9a0 2023-03-03T13:10:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.719+0000 7f900f7fe700 1 -- 172.21.15.102:0/2525077339 >> v1:172.21.15.172:6789/0 conn(0x7f90100cf9a0 legacy=0x7f9010163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.719+0000 7f900f7fe700 1 -- 172.21.15.102:0/2525077339 >> v1:172.21.15.102:6789/0 conn(0x7f90100ced80 legacy=0x7f90100cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:11.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.719+0000 7f900f7fe700 1 -- 172.21.15.102:0/2525077339 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90100cade0 con 0x7f90101597f0 2023-03-03T13:10:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.719+0000 7f900f7fe700 1 --2- 172.21.15.102:0/2525077339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f90100cf150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9004009870 tx=0x7f9004009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8344eb6d366726b3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.719+0000 7f900effd700 1 -- 172.21.15.102:0/2525077339 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004004230 con 0x7f90101597f0 2023-03-03T13:10:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.719+0000 7f900effd700 1 -- 172.21.15.102:0/2525077339 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9004004390 con 0x7f90101597f0 2023-03-03T13:10:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.719+0000 7f900effd700 1 -- 172.21.15.102:0/2525077339 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90040036a0 con 0x7f90101597f0 2023-03-03T13:10:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.720+0000 7f90163ff700 1 -- 172.21.15.102:0/2525077339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 msgr2=0x7f90100cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:11.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.720+0000 7f90163ff700 1 --2- 172.21.15.102:0/2525077339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f90100cf150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9004009870 tx=0x7f9004009b80 comp rx=0 tx=0).stop 2023-03-03T13:10:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.720+0000 7f90163ff700 1 -- 172.21.15.102:0/2525077339 shutdown_connections 2023-03-03T13:10:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.720+0000 7f90163ff700 1 --2- 172.21.15.102:0/2525077339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f90100cf150 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.720+0000 7f90163ff700 1 -- 172.21.15.102:0/2525077339 >> 172.21.15.102:0/2525077339 conn(0x7f9010065fb0 msgr2=0x7f90100663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.720+0000 7f90163ff700 1 -- 172.21.15.102:0/2525077339 shutdown_connections 2023-03-03T13:10:11.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.720+0000 7f90163ff700 1 -- 172.21.15.102:0/2525077339 wait complete. 2023-03-03T13:10:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.722+0000 7f90163ff700 1 Processor -- start 2023-03-03T13:10:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.722+0000 7f90163ff700 1 -- start start 2023-03-03T13:10:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f90163ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90100ced80 0x7f90100c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f90163ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90100cf9a0 0x7f90101f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:11.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f90163ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f9010151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f90163ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9010157e50 con 0x7f90100cf9a0 2023-03-03T13:10:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f90163ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90100c5ca0 con 0x7f90100ced80 2023-03-03T13:10:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f90163ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90101f2980 con 0x7f90101597f0 2023-03-03T13:10:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f900ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90100ced80 0x7f90100c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f900ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90100ced80 0x7f90100c4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50856/0 (socket says 172.21.15.102:50856) 2023-03-03T13:10:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f900ffff700 1 -- 172.21.15.102:0/3798555434 learned_addr learned my addr 172.21.15.102:0/3798555434 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:10:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.723+0000 7f901499c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f9010151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900f7fe700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90100cf9a0 0x7f90101f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900ffff700 1 -- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 msgr2=0x7f9010151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900ffff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f9010151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900ffff700 1 -- 172.21.15.102:0/3798555434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90100cf9a0 msgr2=0x7f90101f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900ffff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90100cf9a0 0x7f90101f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900ffff700 1 -- 172.21.15.102:0/3798555434 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040094a0 con 0x7f90100ced80 2023-03-03T13:10:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f901499c700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f9010151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:10:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900ffff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90100ced80 0x7f90100c4380 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9000001eb0 tx=0x7f900000c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.724+0000 7f900cff9700 1 -- 172.21.15.102:0/3798555434 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000005d70 con 0x7f90100ced80 2023-03-03T13:10:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.725+0000 7f900cff9700 1 -- 172.21.15.102:0/3798555434 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90000070d0 con 0x7f90100ced80 2023-03-03T13:10:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.725+0000 7f900cff9700 1 -- 172.21.15.102:0/3798555434 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000007460 con 0x7f90100ced80 2023-03-03T13:10:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.725+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90100cb3a0 con 0x7f90100ced80 2023-03-03T13:10:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.725+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90100cb0c0 con 0x7f90100ced80 2023-03-03T13:10:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.726+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90100cb680 con 0x7f90100ced80 2023-03-03T13:10:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.727+0000 7f900cff9700 1 -- 172.21.15.102:0/3798555434 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9000019030 con 0x7f90100ced80 2023-03-03T13:10:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.728+0000 7f900cff9700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ffc0682f0 0x7f8ffc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.728+0000 7f900cff9700 1 -- 172.21.15.102:0/3798555434 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9000085070 con 0x7f90100ced80 2023-03-03T13:10:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.729+0000 7f900f7fe700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ffc0682f0 0x7f8ffc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.729+0000 7f900f7fe700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ffc0682f0 0x7f8ffc06a7a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f90040090b0 tx=0x7f9004008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.733+0000 7f900cff9700 1 -- 172.21.15.102:0/3798555434 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9000051780 con 0x7f90100ced80 2023-03-03T13:10:11.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.963+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90100677f0 con 0x7f90100ced80 2023-03-03T13:10:11.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.965+0000 7f900cff9700 1 -- 172.21.15.102:0/3798555434 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9000017090 con 0x7f90100ced80 2023-03-03T13:10:11.966 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:10:11.966 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:10:11.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ffc0682f0 msgr2=0x7f8ffc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:11.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ffc0682f0 0x7f8ffc06a7a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f90040090b0 tx=0x7f9004008030 comp rx=0 tx=0).stop 2023-03-03T13:10:11.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90100ced80 msgr2=0x7f90100c4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:11.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90100ced80 0x7f90100c4380 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9000001eb0 tx=0x7f900000c7b0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 shutdown_connections 2023-03-03T13:10:11.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8ffc0682f0 0x7f8ffc06a7a0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90101597f0 0x7f9010151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90100ced80 0x7f90100c4380 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 --2- 172.21.15.102:0/3798555434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90100cf9a0 0x7f90101f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:11.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 >> 172.21.15.102:0/3798555434 conn(0x7f9010065fb0 msgr2=0x7f90100cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:11.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 shutdown_connections 2023-03-03T13:10:11.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:11.967+0000 7f90163ff700 1 -- 172.21.15.102:0/3798555434 wait complete. 2023-03-03T13:10:11.970 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:10:11.996 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2023-03-03T13:10:12.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:12.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:24.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:10:25.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.246+0000 7f2deffff700 1 Processor -- start 2023-03-03T13:10:25.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.247+0000 7f2deffff700 1 -- start start 2023-03-03T13:10:25.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.248+0000 7f2deffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de8150c80 0x7f2de806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:25.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.248+0000 7f2deffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2de80c2020 0x7f2de80c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:25.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.248+0000 7f2deffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80c5390 con 0x7f2de80c5d60 2023-03-03T13:10:25.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.248+0000 7f2deffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de8157ad0 con 0x7f2de80c2020 2023-03-03T13:10:25.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.248+0000 7f2deffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80c61e0 con 0x7f2de8150c80 2023-03-03T13:10:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dedd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de8150c80 0x7f2de806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dee59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2de80c5d60 0x7f2de80c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45870/0 (socket says 172.21.15.102:45870) 2023-03-03T13:10:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dee59c700 1 -- 172.21.15.102:0/425397018 learned_addr learned my addr 172.21.15.102:0/425397018 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:10:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2ded59a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2de80c2020 0x7f2de80c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2decd99700 1 -- 172.21.15.102:0/425397018 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757588191 0 0) 0x7f2de80c5390 con 0x7f2de80c5d60 2023-03-03T13:10:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2decd99700 1 -- 172.21.15.102:0/425397018 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd40035b0 con 0x7f2de80c5d60 2023-03-03T13:10:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dedd9b700 1 -- 172.21.15.102:0/425397018 >> v2:172.21.15.172:3300/0 conn(0x7f2de80c2020 msgr2=0x7f2de80c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:10:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dedd9b700 1 --2- 172.21.15.102:0/425397018 >> v2:172.21.15.172:3300/0 conn(0x7f2de80c2020 0x7f2de80c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dedd9b700 1 -- 172.21.15.102:0/425397018 >> v1:172.21.15.172:6789/0 conn(0x7f2de80c5d60 legacy=0x7f2de80c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dedd9b700 1 -- 172.21.15.102:0/425397018 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de80c4a20 con 0x7f2de8150c80 2023-03-03T13:10:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2ded59a700 1 --2- 172.21.15.102:0/425397018 >> v2:172.21.15.172:3300/0 conn(0x7f2de80c2020 0x7f2de80c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:10:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.249+0000 7f2dedd9b700 1 --2- 172.21.15.102:0/425397018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de8150c80 0x7f2de806d100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009840 tx=0x7f2de4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23150c6e69728914 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2decd99700 1 -- 172.21.15.102:0/425397018 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4011830 con 0x7f2de8150c80 2023-03-03T13:10:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2decd99700 1 -- 172.21.15.102:0/425397018 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de4011990 con 0x7f2de8150c80 2023-03-03T13:10:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2decd99700 1 -- 172.21.15.102:0/425397018 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4011d20 con 0x7f2de8150c80 2023-03-03T13:10:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 -- 172.21.15.102:0/425397018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de8150c80 msgr2=0x7f2de806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 --2- 172.21.15.102:0/425397018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de8150c80 0x7f2de806d100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009840 tx=0x7f2de4009b50 comp rx=0 tx=0).stop 2023-03-03T13:10:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 -- 172.21.15.102:0/425397018 shutdown_connections 2023-03-03T13:10:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 --2- 172.21.15.102:0/425397018 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de8150c80 0x7f2de806d100 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 --2- 172.21.15.102:0/425397018 >> v2:172.21.15.172:3300/0 conn(0x7f2de80c2020 0x7f2de80c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 -- 172.21.15.102:0/425397018 >> 172.21.15.102:0/425397018 conn(0x7f2de806af90 msgr2=0x7f2de806c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:25.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 -- 172.21.15.102:0/425397018 shutdown_connections 2023-03-03T13:10:25.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.250+0000 7f2deffff700 1 -- 172.21.15.102:0/425397018 wait complete. 2023-03-03T13:10:25.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.252+0000 7f2deffff700 1 Processor -- start 2023-03-03T13:10:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.253+0000 7f2deffff700 1 -- start start 2023-03-03T13:10:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2deffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c2020 0x7f2de80c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2deffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de80c5d60 0x7f2de80706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2deffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2de8150c80 0x7f2de81474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2deffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de806cd20 con 0x7f2de80c2020 2023-03-03T13:10:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2deffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de8159760 con 0x7f2de8150c80 2023-03-03T13:10:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2deffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de806ca60 con 0x7f2de80c5d60 2023-03-03T13:10:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c2020 0x7f2de80c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c2020 0x7f2de80c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54468/0 (socket says 172.21.15.102:54468) 2023-03-03T13:10:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 -- 172.21.15.102:0/576002023 learned_addr learned my addr 172.21.15.102:0/576002023 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:10:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dee59c700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2de8150c80 0x7f2de81474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2ded59a700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de80c5d60 0x7f2de80706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 -- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de80c5d60 msgr2=0x7f2de80706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:25.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de80c5d60 0x7f2de80706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 -- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2de8150c80 msgr2=0x7f2de81474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:25.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2de8150c80 0x7f2de81474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.254+0000 7f2dedd9b700 1 -- 172.21.15.102:0/576002023 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd80094a0 con 0x7f2de80c2020 2023-03-03T13:10:25.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.255+0000 7f2dedd9b700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c2020 0x7f2de80c9b80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2de4014040 tx=0x7f2de4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:25.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.255+0000 7f2dde7fc700 1 -- 172.21.15.102:0/576002023 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4004590 con 0x7f2de80c2020 2023-03-03T13:10:25.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.255+0000 7f2dde7fc700 1 -- 172.21.15.102:0/576002023 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de401f070 con 0x7f2de80c2020 2023-03-03T13:10:25.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.255+0000 7f2dde7fc700 1 -- 172.21.15.102:0/576002023 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de40038a0 con 0x7f2de80c2020 2023-03-03T13:10:25.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.255+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de40094a0 con 0x7f2de80c2020 2023-03-03T13:10:25.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.255+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de8157510 con 0x7f2de80c2020 2023-03-03T13:10:25.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.256+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de806f010 con 0x7f2de80c2020 2023-03-03T13:10:25.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.257+0000 7f2dde7fc700 1 -- 172.21.15.102:0/576002023 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2de400f050 con 0x7f2de80c2020 2023-03-03T13:10:25.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.259+0000 7f2dde7fc700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dcc0682f0 0x7f2dcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:25.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.259+0000 7f2dde7fc700 1 -- 172.21.15.102:0/576002023 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2de4021070 con 0x7f2de80c2020 2023-03-03T13:10:25.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.259+0000 7f2ded59a700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dcc0682f0 0x7f2dcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:25.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.260+0000 7f2ded59a700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dcc0682f0 0x7f2dcc06a7a0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f2de8070330 tx=0x7f2dd8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:25.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.263+0000 7f2dde7fc700 1 -- 172.21.15.102:0/576002023 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2de4053ba0 con 0x7f2de80c2020 2023-03-03T13:10:25.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.494+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2de81577f0 con 0x7f2de80c2020 2023-03-03T13:10:25.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.495+0000 7f2dde7fc700 1 -- 172.21.15.102:0/576002023 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2de40571c0 con 0x7f2de80c2020 2023-03-03T13:10:25.498 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:10:25.498 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:10:25.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dcc0682f0 msgr2=0x7f2dcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:25.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dcc0682f0 0x7f2dcc06a7a0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f2de8070330 tx=0x7f2dd8009210 comp rx=0 tx=0).stop 2023-03-03T13:10:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c2020 msgr2=0x7f2de80c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c2020 0x7f2de80c9b80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2de4014040 tx=0x7f2de4004750 comp rx=0 tx=0).stop 2023-03-03T13:10:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 shutdown_connections 2023-03-03T13:10:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2de8150c80 0x7f2de81474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c2020 0x7f2de80c9b80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dcc0682f0 0x7f2dcc06a7a0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 --2- 172.21.15.102:0/576002023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2de80c5d60 0x7f2de80706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 >> 172.21.15.102:0/576002023 conn(0x7f2de806af90 msgr2=0x7f2de80caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 shutdown_connections 2023-03-03T13:10:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:25.498+0000 7f2deffff700 1 -- 172.21.15.102:0/576002023 wait complete. 2023-03-03T13:10:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:10:25.527 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T13:10:42.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:42.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:53.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:10:53.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.775+0000 7f491d9ff700 1 Processor -- start 2023-03-03T13:10:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.775+0000 7f491d9ff700 1 -- start start 2023-03-03T13:10:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f491d9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49180cd8e0 0x7f49180cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f491d9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49180ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f491d9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918153660 con 0x7f4918153290 2023-03-03T13:10:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f491d9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491806c360 con 0x7f49180cd8e0 2023-03-03T13:10:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f491d9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491815cea0 con 0x7f49180cd390 2023-03-03T13:10:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f49167fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49180cd8e0 0x7f49180cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f4916ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4918153290 0x7f4918158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40820/0 (socket says 172.21.15.102:40820) 2023-03-03T13:10:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f4916ffd700 1 -- 172.21.15.102:0/1091338410 learned_addr learned my addr 172.21.15.102:0/1091338410 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:10:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f49177fe700 1 --2- 172.21.15.102:0/1091338410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49180ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.777+0000 7f4915ffb700 1 -- 172.21.15.102:0/1091338410 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202050728 0 0) 0x7f4918153660 con 0x7f4918153290 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f4915ffb700 1 -- 172.21.15.102:0/1091338410 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49000035b0 con 0x7f4918153290 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f49177fe700 1 -- 172.21.15.102:0/1091338410 >> v2:172.21.15.102:3300/0 conn(0x7f49180cd8e0 msgr2=0x7f49180cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f49177fe700 1 --2- 172.21.15.102:0/1091338410 >> v2:172.21.15.102:3300/0 conn(0x7f49180cd8e0 0x7f49180cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f49177fe700 1 -- 172.21.15.102:0/1091338410 >> v1:172.21.15.172:6789/0 conn(0x7f4918153290 legacy=0x7f4918158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f49177fe700 1 -- 172.21.15.102:0/1091338410 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491806c7f0 con 0x7f49180cd390 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f49167fc700 1 --2- 172.21.15.102:0/1091338410 >> v2:172.21.15.102:3300/0 conn(0x7f49180cd8e0 0x7f49180cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f49177fe700 1 --2- 172.21.15.102:0/1091338410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49180ce360 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4910009d90 tx=0x7f491000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a68cabbda08e554e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f4915ffb700 1 -- 172.21.15.102:0/1091338410 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491000e070 con 0x7f49180cd390 2023-03-03T13:10:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f4915ffb700 1 -- 172.21.15.102:0/1091338410 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000bb20 con 0x7f49180cd390 2023-03-03T13:10:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.778+0000 7f4915ffb700 1 -- 172.21.15.102:0/1091338410 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910012430 con 0x7f49180cd390 2023-03-03T13:10:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 -- 172.21.15.102:0/1091338410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 msgr2=0x7f49180ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 --2- 172.21.15.102:0/1091338410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49180ce360 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4910009d90 tx=0x7f491000b330 comp rx=0 tx=0).stop 2023-03-03T13:10:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 -- 172.21.15.102:0/1091338410 shutdown_connections 2023-03-03T13:10:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 --2- 172.21.15.102:0/1091338410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49180ce360 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 --2- 172.21.15.102:0/1091338410 >> v2:172.21.15.102:3300/0 conn(0x7f49180cd8e0 0x7f49180cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 -- 172.21.15.102:0/1091338410 >> 172.21.15.102:0/1091338410 conn(0x7f4918065fb0 msgr2=0x7f49180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 -- 172.21.15.102:0/1091338410 shutdown_connections 2023-03-03T13:10:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.779+0000 7f491d9ff700 1 -- 172.21.15.102:0/1091338410 wait complete. 2023-03-03T13:10:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.781+0000 7f491d9ff700 1 Processor -- start 2023-03-03T13:10:53.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.781+0000 7f491d9ff700 1 -- start start 2023-03-03T13:10:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.782+0000 7f491d9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49181f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.782+0000 7f491d9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49180cd8e0 0x7f4918154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.782+0000 7f491d9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4918153290 0x7f491814d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.782+0000 7f491d9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49181f25e0 con 0x7f49180cd8e0 2023-03-03T13:10:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.782+0000 7f491d9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49181f2370 con 0x7f4918153290 2023-03-03T13:10:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.782+0000 7f491d9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4918158b50 con 0x7f49180cd390 2023-03-03T13:10:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f4916ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49181f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f4916ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49181f1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36956/0 (socket says 172.21.15.102:36956) 2023-03-03T13:10:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f4916ffd700 1 -- 172.21.15.102:0/2338601283 learned_addr learned my addr 172.21.15.102:0/2338601283 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:10:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49177fe700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4918153290 0x7f491814d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49167fc700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49180cd8e0 0x7f4918154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49177fe700 1 -- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 msgr2=0x7f49181f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49177fe700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49181f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49177fe700 1 -- 172.21.15.102:0/2338601283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49180cd8e0 msgr2=0x7f4918154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49177fe700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49180cd8e0 0x7f4918154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49177fe700 1 -- 172.21.15.102:0/2338601283 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c0094a0 con 0x7f4918153290 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f4916ffd700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49181f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.783+0000 7f49177fe700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4918153290 0x7f491814d360 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4910016000 tx=0x7f4910004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:53.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.784+0000 7f48fffff700 1 -- 172.21.15.102:0/2338601283 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910012430 con 0x7f4918153290 2023-03-03T13:10:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.784+0000 7f48fffff700 1 -- 172.21.15.102:0/2338601283 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4910020070 con 0x7f4918153290 2023-03-03T13:10:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.784+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4910009a70 con 0x7f4918153290 2023-03-03T13:10:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.784+0000 7f48fffff700 1 -- 172.21.15.102:0/2338601283 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49100098f0 con 0x7f4918153290 2023-03-03T13:10:53.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.784+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49180ccde0 con 0x7f4918153290 2023-03-03T13:10:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.785+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491815d160 con 0x7f4918153290 2023-03-03T13:10:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.786+0000 7f48fffff700 1 -- 172.21.15.102:0/2338601283 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f491002e070 con 0x7f4918153290 2023-03-03T13:10:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.788+0000 7f48fffff700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49040682f0 0x7f490406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.788+0000 7f48fffff700 1 -- 172.21.15.102:0/2338601283 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4910022070 con 0x7f4918153290 2023-03-03T13:10:53.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.788+0000 7f4916ffd700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49040682f0 0x7f490406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:53.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.788+0000 7f4916ffd700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49040682f0 0x7f490406a7a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f49080011e0 tx=0x7f4908009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:53.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:53.792+0000 7f48fffff700 1 -- 172.21.15.102:0/2338601283 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4910053fb0 con 0x7f4918153290 2023-03-03T13:10:54.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.023+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49180cd0c0 con 0x7f4918153290 2023-03-03T13:10:54.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.023+0000 7f48fffff700 1 -- 172.21.15.102:0/2338601283 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f49100575d0 con 0x7f4918153290 2023-03-03T13:10:54.025 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:10:54.025 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:10:54.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49040682f0 msgr2=0x7f490406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49040682f0 0x7f490406a7a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f49080011e0 tx=0x7f4908009040 comp rx=0 tx=0).stop 2023-03-03T13:10:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4918153290 msgr2=0x7f491814d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4918153290 0x7f491814d360 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4910016000 tx=0x7f4910004ad0 comp rx=0 tx=0).stop 2023-03-03T13:10:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 shutdown_connections 2023-03-03T13:10:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49040682f0 0x7f490406a7a0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49180cd390 0x7f49181f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:54.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4918153290 0x7f491814d360 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 --2- 172.21.15.102:0/2338601283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49180cd8e0 0x7f4918154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 >> 172.21.15.102:0/2338601283 conn(0x7f4918065fb0 msgr2=0x7f4918157560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 shutdown_connections 2023-03-03T13:10:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:10:54.026+0000 7f491d9ff700 1 -- 172.21.15.102:0/2338601283 wait complete. 2023-03-03T13:10:54.029 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:10:54.054 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-03-03T13:11:01.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:11:01.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.303+0000 7fb8a35ff700 1 Processor -- start 2023-03-03T13:11:01.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.303+0000 7fb8a35ff700 1 -- start start 2023-03-03T13:11:01.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.304+0000 7fb8a35ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c153290 0x7fb89c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:01.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.304+0000 7fb8a35ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c153660 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.304+0000 7fb8a35ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c06c360 con 0x7fb89c1593c0 2023-03-03T13:11:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.304+0000 7fb8a35ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0cec40 con 0x7fb89c153290 2023-03-03T13:11:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a1b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb89c0cd8e0 0x7fb89c0ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55786/0 (socket says 172.21.15.102:55786) 2023-03-03T13:11:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a1b9c700 1 -- 172.21.15.102:0/3557259877 learned_addr learned my addr 172.21.15.102:0/3557259877 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:11:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a0b9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb89c1593c0 0x7fb89c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36466/0 (socket says 172.21.15.102:36466) 2023-03-03T13:11:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a139b700 1 --2- 172.21.15.102:0/3557259877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c153290 0x7fb89c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb893fff700 1 -- 172.21.15.102:0/3557259877 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057261805 0 0) 0x7fb89c06c360 con 0x7fb89c1593c0 2023-03-03T13:11:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb893fff700 1 -- 172.21.15.102:0/3557259877 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8880035b0 con 0x7fb89c1593c0 2023-03-03T13:11:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a139b700 1 -- 172.21.15.102:0/3557259877 >> v1:172.21.15.172:6789/0 conn(0x7fb89c1593c0 legacy=0x7fb89c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a139b700 1 -- 172.21.15.102:0/3557259877 >> v1:172.21.15.102:6789/0 conn(0x7fb89c0cd8e0 legacy=0x7fb89c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb893fff700 1 -- 172.21.15.102:0/3557259877 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679746370 0 0) 0x7fb89c153660 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a139b700 1 -- 172.21.15.102:0/3557259877 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c15cea0 con 0x7fb89c153290 2023-03-03T13:11:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.305+0000 7fb8a139b700 1 --2- 172.21.15.102:0/3557259877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c153290 0x7fb89c158e80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb898009870 tx=0x7fb898009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2739b91c606981a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:01.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb893fff700 1 -- 172.21.15.102:0/3557259877 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898011830 con 0x7fb89c153290 2023-03-03T13:11:01.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb893fff700 1 -- 172.21.15.102:0/3557259877 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb898011990 con 0x7fb89c153290 2023-03-03T13:11:01.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb893fff700 1 -- 172.21.15.102:0/3557259877 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898011d20 con 0x7fb89c153290 2023-03-03T13:11:01.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb8a35ff700 1 -- 172.21.15.102:0/3557259877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c153290 msgr2=0x7fb89c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/3557259877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c153290 0x7fb89c158e80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb898009870 tx=0x7fb898009b80 comp rx=0 tx=0).stop 2023-03-03T13:11:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb8a35ff700 1 -- 172.21.15.102:0/3557259877 shutdown_connections 2023-03-03T13:11:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/3557259877 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c153290 0x7fb89c158e80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb8a35ff700 1 -- 172.21.15.102:0/3557259877 >> 172.21.15.102:0/3557259877 conn(0x7fb89c065fb0 msgr2=0x7fb89c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.306+0000 7fb8a35ff700 1 -- 172.21.15.102:0/3557259877 shutdown_connections 2023-03-03T13:11:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.307+0000 7fb8a35ff700 1 -- 172.21.15.102:0/3557259877 wait complete. 2023-03-03T13:11:01.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.309+0000 7fb8a35ff700 1 Processor -- start 2023-03-03T13:11:01.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.310+0000 7fb8a35ff700 1 -- start start 2023-03-03T13:11:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a35ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb89c0cd8e0 0x7fb89c154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a35ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb89c153290 0x7fb89c1550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a35ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c1593c0 0x7fb89c0c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a35ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0b1c70 con 0x7fb89c153290 2023-03-03T13:11:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a35ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c155af0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a35ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c155f80 con 0x7fb89c1593c0 2023-03-03T13:11:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a139b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb89c0cd8e0 0x7fb89c154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a0b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb89c153290 0x7fb89c1550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a139b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb89c0cd8e0 0x7fb89c154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34358/0 (socket says 172.21.15.102:34358) 2023-03-03T13:11:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.311+0000 7fb8a139b700 1 -- 172.21.15.102:0/2641060122 learned_addr learned my addr 172.21.15.102:0/2641060122 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:11:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb8a1b9c700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c1593c0 0x7fb89c0c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb8a139b700 1 -- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c1593c0 msgr2=0x7fb89c0c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb8a139b700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c1593c0 0x7fb89c0c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb8a139b700 1 -- 172.21.15.102:0/2641060122 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb89c153290 msgr2=0x7fb89c1550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb8a139b700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb89c153290 0x7fb89c1550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb8a139b700 1 -- 172.21.15.102:0/2641060122 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980094a0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb8a139b700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb89c0cd8e0 0x7fb89c154120 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb898014040 tx=0x7fb898004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.312+0000 7fb891ffb700 1 -- 172.21.15.102:0/2641060122 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89801a7e0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.313+0000 7fb891ffb700 1 -- 172.21.15.102:0/2641060122 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89801a940 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.313+0000 7fb891ffb700 1 -- 172.21.15.102:0/2641060122 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89801acd0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.313+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89c15d160 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.313+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89c06c7f0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.314+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89c0ccde0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.315+0000 7fb891ffb700 1 -- 172.21.15.102:0/2641060122 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb89800f050 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.316+0000 7fb891ffb700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8800682f0 0x7fb88006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.316+0000 7fb891ffb700 1 -- 172.21.15.102:0/2641060122 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb898086bb0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.316+0000 7fb8a0b9a700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8800682f0 0x7fb88006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.317+0000 7fb8a0b9a700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8800682f0 0x7fb88006a7a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0b4d90 tx=0x7fb88c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.321+0000 7fb891ffb700 1 -- 172.21.15.102:0/2641060122 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb8980532c0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.554+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb89c04e6e0 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.554+0000 7fb891ffb700 1 -- 172.21.15.102:0/2641060122 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb898029090 con 0x7fb89c0cd8e0 2023-03-03T13:11:01.558 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:11:01.558 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:11:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8800682f0 msgr2=0x7fb88006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8800682f0 0x7fb88006a7a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0b4d90 tx=0x7fb88c007d40 comp rx=0 tx=0).stop 2023-03-03T13:11:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb89c0cd8e0 msgr2=0x7fb89c154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb89c0cd8e0 0x7fb89c154120 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb898014040 tx=0x7fb898004750 comp rx=0 tx=0).stop 2023-03-03T13:11:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 shutdown_connections 2023-03-03T13:11:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8800682f0 0x7fb88006a7a0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb89c1593c0 0x7fb89c0c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb89c0cd8e0 0x7fb89c154120 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 --2- 172.21.15.102:0/2641060122 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb89c153290 0x7fb89c1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 >> 172.21.15.102:0/2641060122 conn(0x7fb89c065fb0 msgr2=0x7fb89c15bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 shutdown_connections 2023-03-03T13:11:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:01.557+0000 7fb8a35ff700 1 -- 172.21.15.102:0/2641060122 wait complete. 2023-03-03T13:11:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:11:01.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 24 secs before thrashing 2023-03-03T13:11:12.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:12.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:25.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:11:25.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.832+0000 7f2ddb59e700 1 Processor -- start 2023-03-03T13:11:25.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.832+0000 7f2ddb59e700 1 -- start start 2023-03-03T13:11:25.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.834+0000 7f2ddb59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc0c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:25.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.834+0000 7f2ddb59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc0c1bc0 con 0x7f2ddc0c4780 2023-03-03T13:11:25.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.834+0000 7f2ddb59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc0c2740 con 0x7f2ddc150c30 2023-03-03T13:11:25.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.834+0000 7f2ddb59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0c36a0 con 0x7f2ddc0c6c40 2023-03-03T13:11:25.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.834+0000 7f2dd9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc0c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:25.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.834+0000 7f2dda59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2ddc150c30 0x7f2ddc0c5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37166/0 (socket says 172.21.15.102:37166) 2023-03-03T13:11:25.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.834+0000 7f2dda59c700 1 -- 172.21.15.102:0/1264343821 learned_addr learned my addr 172.21.15.102:0/1264343821 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:11:25.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd959a700 1 -- 172.21.15.102:0/1264343821 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060626459 0 0) 0x7f2ddc0c2740 con 0x7f2ddc150c30 2023-03-03T13:11:25.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd9d9b700 1 -- 172.21.15.102:0/1264343821 >> v1:172.21.15.172:6789/0 conn(0x7f2ddc150c30 legacy=0x7f2ddc0c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:25.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd9d9b700 1 -- 172.21.15.102:0/1264343821 >> v1:172.21.15.102:6789/0 conn(0x7f2ddc0c4780 legacy=0x7f2ddc161310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd9d9b700 1 -- 172.21.15.102:0/1264343821 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc0c5730 con 0x7f2ddc0c6c40 2023-03-03T13:11:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd9d9b700 1 --2- 172.21.15.102:0/1264343821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc0c4240 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009870 tx=0x7f2dd0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1dc36c4f9b1572a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd959a700 1 -- 172.21.15.102:0/1264343821 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0011830 con 0x7f2ddc0c6c40 2023-03-03T13:11:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd959a700 1 -- 172.21.15.102:0/1264343821 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd0011990 con 0x7f2ddc0c6c40 2023-03-03T13:11:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.835+0000 7f2dd959a700 1 -- 172.21.15.102:0/1264343821 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0011d20 con 0x7f2ddc0c6c40 2023-03-03T13:11:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.836+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1264343821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 msgr2=0x7f2ddc0c4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.836+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1264343821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc0c4240 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009870 tx=0x7f2dd0009b80 comp rx=0 tx=0).stop 2023-03-03T13:11:25.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.836+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1264343821 shutdown_connections 2023-03-03T13:11:25.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.836+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1264343821 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc0c4240 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:25.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.836+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1264343821 >> 172.21.15.102:0/1264343821 conn(0x7f2ddc0ca7c0 msgr2=0x7f2ddc0cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:25.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.836+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1264343821 shutdown_connections 2023-03-03T13:11:25.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.836+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1264343821 wait complete. 2023-03-03T13:11:25.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.838+0000 7f2ddb59e700 1 Processor -- start 2023-03-03T13:11:25.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.838+0000 7f2ddb59e700 1 -- start start 2023-03-03T13:11:25.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.839+0000 7f2ddb59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ddc0c4780 0x7f2ddc1ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:25.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.839+0000 7f2ddb59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:25.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.839+0000 7f2ddb59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ddc150c30 0x7f2ddc1e3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:25.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.839+0000 7f2ddb59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc1e2290 con 0x7f2ddc150c30 2023-03-03T13:11:25.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.839+0000 7f2ddb59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc070e40 con 0x7f2ddc0c4780 2023-03-03T13:11:25.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.839+0000 7f2ddb59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc155950 con 0x7f2ddc0c6c40 2023-03-03T13:11:25.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ddc0c4780 0x7f2ddc1ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:25.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dd9d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:25.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2ddad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ddc150c30 0x7f2ddc1e3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ddc0c4780 0x7f2ddc1ee6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46968/0 (socket says 172.21.15.102:46968) 2023-03-03T13:11:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 -- 172.21.15.102:0/1547892856 learned_addr learned my addr 172.21.15.102:0/1547892856 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:11:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 -- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 msgr2=0x7f2ddc142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc142590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 -- 172.21.15.102:0/1547892856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ddc150c30 msgr2=0x7f2ddc1e3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ddc150c30 0x7f2ddc1e3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 -- 172.21.15.102:0/1547892856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00094a0 con 0x7f2ddc0c4780 2023-03-03T13:11:25.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dd9d9b700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:11:25.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2ddad9d700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ddc150c30 0x7f2ddc1e3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:11:25.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.840+0000 7f2dda59c700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ddc0c4780 0x7f2ddc1ee6d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00c550 tx=0x7f2dcc00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:25.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.841+0000 7f2dcaffd700 1 -- 172.21.15.102:0/1547892856 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc004c80 con 0x7f2ddc0c4780 2023-03-03T13:11:25.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.841+0000 7f2dcaffd700 1 -- 172.21.15.102:0/1547892856 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc004de0 con 0x7f2ddc0c4780 2023-03-03T13:11:25.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.841+0000 7f2dcaffd700 1 -- 172.21.15.102:0/1547892856 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc007310 con 0x7f2ddc0c4780 2023-03-03T13:11:25.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.841+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc0c1e10 con 0x7f2ddc0c4780 2023-03-03T13:11:25.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.841+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc156ba0 con 0x7f2ddc0c4780 2023-03-03T13:11:25.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.842+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc0c20f0 con 0x7f2ddc0c4780 2023-03-03T13:11:25.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.843+0000 7f2dcaffd700 1 -- 172.21.15.102:0/1547892856 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2dcc01a020 con 0x7f2ddc0c4780 2023-03-03T13:11:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.844+0000 7f2dcaffd700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dbc0682f0 0x7f2dbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.844+0000 7f2dcaffd700 1 -- 172.21.15.102:0/1547892856 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2dcc0857e0 con 0x7f2ddc0c4780 2023-03-03T13:11:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.845+0000 7f2dd9d9b700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dbc0682f0 0x7f2dbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.845+0000 7f2dd9d9b700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dbc0682f0 0x7f2dbc06a7a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0000c00 tx=0x7f2dd0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:25.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:25.848+0000 7f2dcaffd700 1 -- 172.21.15.102:0/1547892856 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2dcc051ef0 con 0x7f2ddc0c4780 2023-03-03T13:11:26.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.082+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ddc0c45a0 con 0x7f2ddc0c4780 2023-03-03T13:11:26.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.083+0000 7f2dcaffd700 1 -- 172.21.15.102:0/1547892856 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2dcc055510 con 0x7f2ddc0c4780 2023-03-03T13:11:26.085 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:11:26.085 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:11:26.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dbc0682f0 msgr2=0x7f2dbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:26.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dbc0682f0 0x7f2dbc06a7a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0000c00 tx=0x7f2dd0008030 comp rx=0 tx=0).stop 2023-03-03T13:11:26.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ddc0c4780 msgr2=0x7f2ddc1ee6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ddc0c4780 0x7f2ddc1ee6d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00c550 tx=0x7f2dcc00c860 comp rx=0 tx=0).stop 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 shutdown_connections 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ddc150c30 0x7f2ddc1e3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ddc0c4780 0x7f2ddc1ee6d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2dbc0682f0 0x7f2dbc06a7a0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 --2- 172.21.15.102:0/1547892856 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2ddc0c6c40 0x7f2ddc142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 >> 172.21.15.102:0/1547892856 conn(0x7f2ddc0ca7c0 msgr2=0x7f2ddc0c4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 shutdown_connections 2023-03-03T13:11:26.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:11:26.086+0000 7f2ddb59e700 1 -- 172.21.15.102:0/1547892856 wait complete. 2023-03-03T13:11:26.089 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:11:26.114 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T13:11:43.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:43.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:13.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:13.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:26.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:12:26.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.367+0000 7f8b54fff700 1 Processor -- start 2023-03-03T13:12:26.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.367+0000 7f8b54fff700 1 -- start start 2023-03-03T13:12:26.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.368+0000 7f8b54fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b500cd8e0 0x7f8b500cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:26.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.368+0000 7f8b54fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b500cd390 0x7f8b500ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:26.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.368+0000 7f8b54fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b50153660 con 0x7f8b50153290 2023-03-03T13:12:26.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b54fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5006c360 con 0x7f8b500cd390 2023-03-03T13:12:26.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b54fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5015cea0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4ed9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8b500cd390 0x7f8b500ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:26.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4dd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b500cd8e0 0x7f8b500cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:26.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4e59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8b50153290 0x7f8b50158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60734/0 (socket says 172.21.15.102:60734) 2023-03-03T13:12:26.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4e59c700 1 -- 172.21.15.102:0/1803241327 learned_addr learned my addr 172.21.15.102:0/1803241327 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:12:26.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4dd9b700 1 -- 172.21.15.102:0/1803241327 >> v2:172.21.15.172:3300/0 conn(0x7f8b500cd390 msgr2=0x7f8b500ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:12:26.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4dd9b700 1 --2- 172.21.15.102:0/1803241327 >> v2:172.21.15.172:3300/0 conn(0x7f8b500cd390 0x7f8b500ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4dd9b700 1 -- 172.21.15.102:0/1803241327 >> v1:172.21.15.172:6789/0 conn(0x7f8b50153290 legacy=0x7f8b50158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:26.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.369+0000 7f8b4dd9b700 1 -- 172.21.15.102:0/1803241327 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5006c7f0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.370+0000 7f8b4dd9b700 1 --2- 172.21.15.102:0/1803241327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b500cd8e0 0x7f8b500cde20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009840 tx=0x7f8b44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=848354638227a2bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:26.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.370+0000 7f8b4d59a700 1 -- 172.21.15.102:0/1803241327 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44011830 con 0x7f8b500cd8e0 2023-03-03T13:12:26.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.370+0000 7f8b4d59a700 1 -- 172.21.15.102:0/1803241327 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44011990 con 0x7f8b500cd8e0 2023-03-03T13:12:26.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.370+0000 7f8b4d59a700 1 -- 172.21.15.102:0/1803241327 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44011d20 con 0x7f8b500cd8e0 2023-03-03T13:12:26.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.370+0000 7f8b54fff700 1 -- 172.21.15.102:0/1803241327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b500cd8e0 msgr2=0x7f8b500cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:26.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.370+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1803241327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b500cd8e0 0x7f8b500cde20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009840 tx=0x7f8b44009b50 comp rx=0 tx=0).stop 2023-03-03T13:12:26.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.371+0000 7f8b54fff700 1 -- 172.21.15.102:0/1803241327 shutdown_connections 2023-03-03T13:12:26.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.371+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1803241327 >> v2:172.21.15.172:3300/0 conn(0x7f8b500cd390 0x7f8b500ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.371+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1803241327 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b500cd8e0 0x7f8b500cde20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.371+0000 7f8b54fff700 1 -- 172.21.15.102:0/1803241327 >> 172.21.15.102:0/1803241327 conn(0x7f8b50065fb0 msgr2=0x7f8b500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:12:26.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.371+0000 7f8b54fff700 1 -- 172.21.15.102:0/1803241327 shutdown_connections 2023-03-03T13:12:26.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.371+0000 7f8b54fff700 1 -- 172.21.15.102:0/1803241327 wait complete. 2023-03-03T13:12:26.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.373+0000 7f8b54fff700 1 Processor -- start 2023-03-03T13:12:26.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.373+0000 7f8b54fff700 1 -- start start 2023-03-03T13:12:26.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b54fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b500cd390 0x7f8b5014ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:26.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b54fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b500cd8e0 0x7f8b500c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:26.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b54fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b50153290 0x7f8b5014f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:26.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b54fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b50153a60 con 0x7f8b500cd390 2023-03-03T13:12:26.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b54fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5014fce0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b54fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500c8b60 con 0x7f8b50153290 2023-03-03T13:12:26.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b4e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b500cd390 0x7f8b5014ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:26.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b4dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b500cd8e0 0x7f8b500c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:26.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b4ed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b50153290 0x7f8b5014f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:26.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b4dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b500cd8e0 0x7f8b500c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37880/0 (socket says 172.21.15.102:37880) 2023-03-03T13:12:26.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b4e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b500cd390 0x7f8b5014ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56192/0 (socket says 172.21.15.102:56192) 2023-03-03T13:12:26.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.374+0000 7f8b4dd9b700 1 -- 172.21.15.102:0/1323748232 learned_addr learned my addr 172.21.15.102:0/1323748232 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:12:26.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.375+0000 7f8b4dd9b700 1 -- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b50153290 msgr2=0x7f8b5014f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:26.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.375+0000 7f8b4dd9b700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b50153290 0x7f8b5014f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.375+0000 7f8b4dd9b700 1 -- 172.21.15.102:0/1323748232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b500cd390 msgr2=0x7f8b5014ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:26.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.375+0000 7f8b4dd9b700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b500cd390 0x7f8b5014ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.375+0000 7f8b4dd9b700 1 -- 172.21.15.102:0/1323748232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440094a0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.375+0000 7f8b4dd9b700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b500cd8e0 0x7f8b500c8520 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009fd0 tx=0x7f8b44004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.376+0000 7f8b3effd700 1 -- 172.21.15.102:0/1323748232 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b440036a0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.376+0000 7f8b3effd700 1 -- 172.21.15.102:0/1323748232 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4401f070 con 0x7f8b500cd8e0 2023-03-03T13:12:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.376+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b500ccde0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.376+0000 7f8b3effd700 1 -- 172.21.15.102:0/1323748232 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44003a60 con 0x7f8b500cd8e0 2023-03-03T13:12:26.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.376+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5015d160 con 0x7f8b500cd8e0 2023-03-03T13:12:26.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.377+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b500cd0c0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.378+0000 7f8b3effd700 1 -- 172.21.15.102:0/1323748232 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b4400f050 con 0x7f8b500cd8e0 2023-03-03T13:12:26.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.379+0000 7f8b3effd700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b300682f0 0x7f8b3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:26.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.380+0000 7f8b3effd700 1 -- 172.21.15.102:0/1323748232 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8b44021070 con 0x7f8b500cd8e0 2023-03-03T13:12:26.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.380+0000 7f8b4e59c700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b300682f0 0x7f8b3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:26.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.380+0000 7f8b4e59c700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b300682f0 0x7f8b3006a7a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005910 tx=0x7f8b40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:26.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.384+0000 7f8b3effd700 1 -- 172.21.15.102:0/1323748232 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b44053ce0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.616+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b5015bbf0 con 0x7f8b500cd8e0 2023-03-03T13:12:26.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.617+0000 7f8b3effd700 1 -- 172.21.15.102:0/1323748232 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8b44015070 con 0x7f8b500cd8e0 2023-03-03T13:12:26.619 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:12:26.619 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:12:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b300682f0 msgr2=0x7f8b3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b300682f0 0x7f8b3006a7a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005910 tx=0x7f8b40009040 comp rx=0 tx=0).stop 2023-03-03T13:12:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b500cd8e0 msgr2=0x7f8b500c8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b500cd8e0 0x7f8b500c8520 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009fd0 tx=0x7f8b44004750 comp rx=0 tx=0).stop 2023-03-03T13:12:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 shutdown_connections 2023-03-03T13:12:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8b300682f0 0x7f8b3006a7a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8b50153290 0x7f8b5014f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b500cd390 0x7f8b5014ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 --2- 172.21.15.102:0/1323748232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b500cd8e0 0x7f8b500c8520 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:26.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 >> 172.21.15.102:0/1323748232 conn(0x7f8b50065fb0 msgr2=0x7f8b500cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:12:26.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 shutdown_connections 2023-03-03T13:12:26.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:26.620+0000 7f8b54fff700 1 -- 172.21.15.102:0/1323748232 wait complete. 2023-03-03T13:12:26.623 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:12:26.649 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-03-03T13:12:38.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:12:38.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.896+0000 7fbf7679d700 1 Processor -- start 2023-03-03T13:12:38.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.896+0000 7fbf7679d700 1 -- start start 2023-03-03T13:12:38.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf7679d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700ced00 0x7fbf700c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:38.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf7679d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf700c41b0 0x7fbf700cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf7679d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700c3d20 con 0x7fbf700b3970 2023-03-03T13:12:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf7679d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf700c4930 con 0x7fbf700c41b0 2023-03-03T13:12:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf7679d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700cf610 con 0x7fbf700ced00 2023-03-03T13:12:38.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf75f9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbf700c41b0 0x7fbf700cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:38.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf7579b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700ced00 0x7fbf700c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:38.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.898+0000 7fbf74f9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbf700b3970 0x7fbf70158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43050/0 (socket says 172.21.15.102:43050) 2023-03-03T13:12:38.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.899+0000 7fbf74f9a700 1 -- 172.21.15.102:0/169794 learned_addr learned my addr 172.21.15.102:0/169794 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:12:38.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.899+0000 7fbf75f9c700 1 -- 172.21.15.102:0/169794 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700ced00 msgr2=0x7fbf700c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:12:38.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.899+0000 7fbf75f9c700 1 --2- 172.21.15.102:0/169794 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700ced00 0x7fbf700c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:38.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.899+0000 7fbf75f9c700 1 -- 172.21.15.102:0/169794 >> v1:172.21.15.102:6789/0 conn(0x7fbf700b3970 legacy=0x7fbf70158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:38.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.899+0000 7fbf75f9c700 1 -- 172.21.15.102:0/169794 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7015cba0 con 0x7fbf700c41b0 2023-03-03T13:12:38.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.899+0000 7fbf75f9c700 1 --2- 172.21.15.102:0/169794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 0x7fbf700cf0d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009dc0 tx=0x7fbf6800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8ae5fe5bd7d872d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:38.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf67fff700 1 -- 172.21.15.102:0/169794 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800e070 con 0x7fbf700c41b0 2023-03-03T13:12:38.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf67fff700 1 -- 172.21.15.102:0/169794 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6800bae0 con 0x7fbf700c41b0 2023-03-03T13:12:38.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf67fff700 1 -- 172.21.15.102:0/169794 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800be90 con 0x7fbf700c41b0 2023-03-03T13:12:38.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 -- 172.21.15.102:0/169794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 msgr2=0x7fbf700cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:38.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 --2- 172.21.15.102:0/169794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 0x7fbf700cf0d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009dc0 tx=0x7fbf6800b330 comp rx=0 tx=0).stop 2023-03-03T13:12:38.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 -- 172.21.15.102:0/169794 shutdown_connections 2023-03-03T13:12:38.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 --2- 172.21.15.102:0/169794 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 0x7fbf700cf0d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:38.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 --2- 172.21.15.102:0/169794 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700ced00 0x7fbf700c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:38.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 -- 172.21.15.102:0/169794 >> 172.21.15.102:0/169794 conn(0x7fbf7008b1c0 msgr2=0x7fbf70071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:12:38.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 -- 172.21.15.102:0/169794 shutdown_connections 2023-03-03T13:12:38.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.900+0000 7fbf7679d700 1 -- 172.21.15.102:0/169794 wait complete. 2023-03-03T13:12:38.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.902+0000 7fbf7679d700 1 Processor -- start 2023-03-03T13:12:38.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.903+0000 7fbf7679d700 1 -- start start 2023-03-03T13:12:38.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7679d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700b3970 0x7fbf7015c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:38.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7679d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 0x7fbf70150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:38.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7679d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf700ced00 0x7fbf7014eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:38.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7679d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7014e020 con 0x7fbf700ced00 2023-03-03T13:12:38.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7679d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf701e5d00 con 0x7fbf700c41b0 2023-03-03T13:12:38.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7679d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf701e50a0 con 0x7fbf700b3970 2023-03-03T13:12:38.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf75f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf700ced00 0x7fbf7014eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:38.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7579b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700b3970 0x7fbf7015c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:38.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf74f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 0x7fbf70150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:38.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7579b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700b3970 0x7fbf7015c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:49690/0 (socket says 172.21.15.102:49690) 2023-03-03T13:12:38.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.904+0000 7fbf7579b700 1 -- 172.21.15.102:0/3113054334 learned_addr learned my addr 172.21.15.102:0/3113054334 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:12:38.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7579b700 1 -- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 msgr2=0x7fbf70150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:38.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7579b700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 0x7fbf70150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:38.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7579b700 1 -- 172.21.15.102:0/3113054334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf700ced00 msgr2=0x7fbf7014eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:38.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7579b700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf700ced00 0x7fbf7014eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:38.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7579b700 1 -- 172.21.15.102:0/3113054334 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf68009a70 con 0x7fbf700b3970 2023-03-03T13:12:38.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7579b700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700b3970 0x7fbf7015c060 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e9e0 tx=0x7fbf6c00ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf65ffb700 1 -- 172.21.15.102:0/3113054334 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c009710 con 0x7fbf700b3970 2023-03-03T13:12:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf65ffb700 1 -- 172.21.15.102:0/3113054334 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c00ee80 con 0x7fbf700b3970 2023-03-03T13:12:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7006bf20 con 0x7fbf700b3970 2023-03-03T13:12:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf65ffb700 1 -- 172.21.15.102:0/3113054334 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00ca50 con 0x7fbf700b3970 2023-03-03T13:12:38.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.905+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7015ce80 con 0x7fbf700b3970 2023-03-03T13:12:38.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.906+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7006c200 con 0x7fbf700b3970 2023-03-03T13:12:38.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.908+0000 7fbf65ffb700 1 -- 172.21.15.102:0/3113054334 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf6c019030 con 0x7fbf700b3970 2023-03-03T13:12:38.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.909+0000 7fbf65ffb700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf540682f0 0x7fbf5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:38.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.909+0000 7fbf65ffb700 1 -- 172.21.15.102:0/3113054334 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbf6c085d60 con 0x7fbf700b3970 2023-03-03T13:12:38.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.909+0000 7fbf74f9a700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf540682f0 0x7fbf5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:38.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.910+0000 7fbf74f9a700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf540682f0 0x7fbf5406a7a0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fbf700cd2d0 tx=0x7fbf60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:38.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:38.913+0000 7fbf65ffb700 1 -- 172.21.15.102:0/3113054334 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf6c052380 con 0x7fbf700b3970 2023-03-03T13:12:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.144+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf701e6020 con 0x7fbf700b3970 2023-03-03T13:12:39.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.145+0000 7fbf65ffb700 1 -- 172.21.15.102:0/3113054334 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbf6c021090 con 0x7fbf700b3970 2023-03-03T13:12:39.148 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:12:39.148 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:12:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.147+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf540682f0 msgr2=0x7fbf5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.147+0000 7fbf7679d700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf540682f0 0x7fbf5406a7a0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fbf700cd2d0 tx=0x7fbf60009040 comp rx=0 tx=0).stop 2023-03-03T13:12:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.147+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700b3970 msgr2=0x7fbf7015c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:39.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.147+0000 7fbf7679d700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700b3970 0x7fbf7015c060 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e9e0 tx=0x7fbf6c00ecf0 comp rx=0 tx=0).stop 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 shutdown_connections 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbf540682f0 0x7fbf5406a7a0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbf700b3970 0x7fbf7015c060 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf700ced00 0x7fbf7014eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 --2- 172.21.15.102:0/3113054334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf700c41b0 0x7fbf70150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 >> 172.21.15.102:0/3113054334 conn(0x7fbf7008b1c0 msgr2=0x7fbf700cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 shutdown_connections 2023-03-03T13:12:39.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:12:39.148+0000 7fbf7679d700 1 -- 172.21.15.102:0/3113054334 wait complete. 2023-03-03T13:12:39.152 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:12:39.177 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T13:12:43.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:43.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:13.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:13.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:44.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:44.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:58.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:13:58.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.428+0000 7feeb259d700 1 Processor -- start 2023-03-03T13:13:58.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.429+0000 7feeb259d700 1 -- start start 2023-03-03T13:13:58.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb259d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feeac1564b0 0x7feeac0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:58.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb259d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0cab50 0x7feeac0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:58.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb259d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0c9ca0 con 0x7feeac0c9610 2023-03-03T13:13:58.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb259d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeac0b4920 con 0x7feeac1564b0 2023-03-03T13:13:58.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb259d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0b4c00 con 0x7feeac0cab50 2023-03-03T13:13:58.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb1d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0cab50 0x7feeac0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:58.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb159b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feeac1564b0 0x7feeac0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:58.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.430+0000 7feeb0d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7feeac0c9610 0x7feeac0ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37076/0 (socket says 172.21.15.102:37076) 2023-03-03T13:13:58.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feeb0d9a700 1 -- 172.21.15.102:0/1209853733 learned_addr learned my addr 172.21.15.102:0/1209853733 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:13:58.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feea3fff700 1 -- 172.21.15.102:0/1209853733 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489985301 0 0) 0x7feeac0c9ca0 con 0x7feeac0c9610 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feea3fff700 1 -- 172.21.15.102:0/1209853733 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee980035b0 con 0x7feeac0c9610 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feeb1d9c700 1 -- 172.21.15.102:0/1209853733 >> v2:172.21.15.172:3300/0 conn(0x7feeac1564b0 msgr2=0x7feeac0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feeb1d9c700 1 --2- 172.21.15.102:0/1209853733 >> v2:172.21.15.172:3300/0 conn(0x7feeac1564b0 0x7feeac0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feeb1d9c700 1 -- 172.21.15.102:0/1209853733 >> v1:172.21.15.172:6789/0 conn(0x7feeac0c9610 legacy=0x7feeac0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feeb1d9c700 1 -- 172.21.15.102:0/1209853733 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0b46f0 con 0x7feeac0cab50 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.431+0000 7feeb1d9c700 1 --2- 172.21.15.102:0/1209853733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0cab50 0x7feeac0cbba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feea4009d90 tx=0x7feea400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74d59bf794907c21 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feea3fff700 1 -- 172.21.15.102:0/1209853733 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea400e070 con 0x7feeac0cab50 2023-03-03T13:13:58.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feea3fff700 1 -- 172.21.15.102:0/1209853733 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea400bb20 con 0x7feeac0cab50 2023-03-03T13:13:58.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feea3fff700 1 -- 172.21.15.102:0/1209853733 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4012430 con 0x7feeac0cab50 2023-03-03T13:13:58.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 -- 172.21.15.102:0/1209853733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0cab50 msgr2=0x7feeac0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 --2- 172.21.15.102:0/1209853733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0cab50 0x7feeac0cbba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feea4009d90 tx=0x7feea400b330 comp rx=0 tx=0).stop 2023-03-03T13:13:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 -- 172.21.15.102:0/1209853733 shutdown_connections 2023-03-03T13:13:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 --2- 172.21.15.102:0/1209853733 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0cab50 0x7feeac0cbba0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 --2- 172.21.15.102:0/1209853733 >> v2:172.21.15.172:3300/0 conn(0x7feeac1564b0 0x7feeac0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 -- 172.21.15.102:0/1209853733 >> 172.21.15.102:0/1209853733 conn(0x7feeac065fb0 msgr2=0x7feeac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:13:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 -- 172.21.15.102:0/1209853733 shutdown_connections 2023-03-03T13:13:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.432+0000 7feeb259d700 1 -- 172.21.15.102:0/1209853733 wait complete. 2023-03-03T13:13:58.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.435+0000 7feeb259d700 1 Processor -- start 2023-03-03T13:13:58.437 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.435+0000 7feeb259d700 1 -- start start 2023-03-03T13:13:58.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb259d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0c9610 0x7feeac152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:58.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb259d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac0cab50 0x7feeac150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:58.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb259d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeac1564b0 0x7feeac151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:58.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb259d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac15b610 con 0x7feeac0cab50 2023-03-03T13:13:58.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb259d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac1edac0 con 0x7feeac1564b0 2023-03-03T13:13:58.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb259d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac1edcd0 con 0x7feeac0c9610 2023-03-03T13:13:58.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb0d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac0cab50 0x7feeac150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:58.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb1d9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeac1564b0 0x7feeac151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:58.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb159b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0c9610 0x7feeac152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:58.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb0d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac0cab50 0x7feeac150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35134/0 (socket says 172.21.15.102:35134) 2023-03-03T13:13:58.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb0d9a700 1 -- 172.21.15.102:0/426698075 learned_addr learned my addr 172.21.15.102:0/426698075 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:13:58.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.436+0000 7feeb159b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0c9610 0x7feeac152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37656/0 (socket says 172.21.15.102:37656) 2023-03-03T13:13:58.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb1d9c700 1 -- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0c9610 msgr2=0x7feeac152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:58.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb1d9c700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0c9610 0x7feeac152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb1d9c700 1 -- 172.21.15.102:0/426698075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac0cab50 msgr2=0x7feeac150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:58.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb1d9c700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac0cab50 0x7feeac150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb1d9c700 1 -- 172.21.15.102:0/426698075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4009a70 con 0x7feeac1564b0 2023-03-03T13:13:58.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb0d9a700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac0cab50 0x7feeac150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:13:58.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb1d9c700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeac1564b0 0x7feeac151530 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feea4016000 tx=0x7feea4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:13:58.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feea1ffb700 1 -- 172.21.15.102:0/426698075 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4012430 con 0x7feeac1564b0 2023-03-03T13:13:58.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feea1ffb700 1 -- 172.21.15.102:0/426698075 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea4020070 con 0x7feeac1564b0 2023-03-03T13:13:58.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feea1ffb700 1 -- 172.21.15.102:0/426698075 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea40098f0 con 0x7feeac1564b0 2023-03-03T13:13:58.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.437+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac0b2210 con 0x7feeac1564b0 2023-03-03T13:13:58.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.438+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac0b1f30 con 0x7feeac1564b0 2023-03-03T13:13:58.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.439+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac156880 con 0x7feeac1564b0 2023-03-03T13:13:58.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.440+0000 7feea1ffb700 1 -- 172.21.15.102:0/426698075 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feea402e070 con 0x7feeac1564b0 2023-03-03T13:13:58.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.441+0000 7feea1ffb700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee90068320 0x7fee9006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:58.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.441+0000 7feea1ffb700 1 -- 172.21.15.102:0/426698075 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7feea4022070 con 0x7feeac1564b0 2023-03-03T13:13:58.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.442+0000 7feeb159b700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee90068320 0x7fee9006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:58.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.442+0000 7feeb159b700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee90068320 0x7fee9006a7d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7feea8009be0 tx=0x7feea8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:13:58.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.445+0000 7feea1ffb700 1 -- 172.21.15.102:0/426698075 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feea4053fb0 con 0x7feeac1564b0 2023-03-03T13:13:58.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.678+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feeac15b750 con 0x7feeac1564b0 2023-03-03T13:13:58.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.679+0000 7feea1ffb700 1 -- 172.21.15.102:0/426698075 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7feeac15b750 con 0x7feeac1564b0 2023-03-03T13:13:58.680 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:13:58.680 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:13:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.681+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee90068320 msgr2=0x7fee9006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.681+0000 7feeb259d700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee90068320 0x7fee9006a7d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7feea8009be0 tx=0x7feea8009210 comp rx=0 tx=0).stop 2023-03-03T13:13:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.681+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeac1564b0 msgr2=0x7feeac151530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.681+0000 7feeb259d700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeac1564b0 0x7feeac151530 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feea4016000 tx=0x7feea4004ad0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 shutdown_connections 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee90068320 0x7fee9006a7d0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeac0c9610 0x7feeac152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeac1564b0 0x7feeac151530 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 --2- 172.21.15.102:0/426698075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac0cab50 0x7feeac150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 >> 172.21.15.102:0/426698075 conn(0x7feeac065fb0 msgr2=0x7feeac156c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 shutdown_connections 2023-03-03T13:13:58.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:13:58.682+0000 7feeb259d700 1 -- 172.21.15.102:0/426698075 wait complete. 2023-03-03T13:13:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:13:58.710 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2023-03-03T13:14:14.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:14.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:15.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:14:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.957+0000 7fdcf0dff700 1 Processor -- start 2023-03-03T13:14:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.958+0000 7fdcf0dff700 1 -- start start 2023-03-03T13:14:15.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.959+0000 7fdcf0dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd8e0 0x7fdcec0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:15.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.959+0000 7fdcf0dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcec0cd390 0x7fdcec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.959+0000 7fdcf0dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec153660 con 0x7fdcec153290 2023-03-03T13:14:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.959+0000 7fdcf0dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcec06c360 con 0x7fdcec0cd390 2023-03-03T13:14:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.959+0000 7fdcf0dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec15cea0 con 0x7fdcec0cd8e0 2023-03-03T13:14:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdcead9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcec0cd390 0x7fdcec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdcea59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdcec153290 0x7fdcec158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:48198/0 (socket says 172.21.15.102:48198) 2023-03-03T13:14:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdcea59c700 1 -- 172.21.15.102:0/1550352069 learned_addr learned my addr 172.21.15.102:0/1550352069 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:14:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdce9d9b700 1 --2- 172.21.15.102:0/1550352069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd8e0 0x7fdcec0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdce959a700 1 -- 172.21.15.102:0/1550352069 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402737529 0 0) 0x7fdcec153660 con 0x7fdcec153290 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdce959a700 1 -- 172.21.15.102:0/1550352069 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd40035b0 con 0x7fdcec153290 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdce9d9b700 1 -- 172.21.15.102:0/1550352069 >> v2:172.21.15.172:3300/0 conn(0x7fdcec0cd390 msgr2=0x7fdcec0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdce9d9b700 1 --2- 172.21.15.102:0/1550352069 >> v2:172.21.15.172:3300/0 conn(0x7fdcec0cd390 0x7fdcec0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdce9d9b700 1 -- 172.21.15.102:0/1550352069 >> v1:172.21.15.172:6789/0 conn(0x7fdcec153290 legacy=0x7fdcec158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdce9d9b700 1 -- 172.21.15.102:0/1550352069 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec06c7f0 con 0x7fdcec0cd8e0 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.960+0000 7fdcead9d700 1 --2- 172.21.15.102:0/1550352069 >> v2:172.21.15.172:3300/0 conn(0x7fdcec0cd390 0x7fdcec0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:14:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdce9d9b700 1 --2- 172.21.15.102:0/1550352069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd8e0 0x7fdcec0cde20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009840 tx=0x7fdce0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc67522dd4ee19be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:15.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdce959a700 1 -- 172.21.15.102:0/1550352069 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0011830 con 0x7fdcec0cd8e0 2023-03-03T13:14:15.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdce959a700 1 -- 172.21.15.102:0/1550352069 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce0011990 con 0x7fdcec0cd8e0 2023-03-03T13:14:15.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdce959a700 1 -- 172.21.15.102:0/1550352069 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0011d20 con 0x7fdcec0cd8e0 2023-03-03T13:14:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdcf0dff700 1 -- 172.21.15.102:0/1550352069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd8e0 msgr2=0x7fdcec0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/1550352069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd8e0 0x7fdcec0cde20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009840 tx=0x7fdce0009b50 comp rx=0 tx=0).stop 2023-03-03T13:14:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdcf0dff700 1 -- 172.21.15.102:0/1550352069 shutdown_connections 2023-03-03T13:14:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/1550352069 >> v2:172.21.15.172:3300/0 conn(0x7fdcec0cd390 0x7fdcec0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/1550352069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd8e0 0x7fdcec0cde20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.961+0000 7fdcf0dff700 1 -- 172.21.15.102:0/1550352069 >> 172.21.15.102:0/1550352069 conn(0x7fdcec065fb0 msgr2=0x7fdcec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:14:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.962+0000 7fdcf0dff700 1 -- 172.21.15.102:0/1550352069 shutdown_connections 2023-03-03T13:14:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.962+0000 7fdcf0dff700 1 -- 172.21.15.102:0/1550352069 wait complete. 2023-03-03T13:14:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.963+0000 7fdcf0dff700 1 Processor -- start 2023-03-03T13:14:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.964+0000 7fdcf0dff700 1 -- start start 2023-03-03T13:14:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcf0dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd390 0x7fdcec1efed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcf0dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcec0cd8e0 0x7fdcec0b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcf0dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcec153290 0x7fdcec155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcf0dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0b44b0 con 0x7fdcec153290 2023-03-03T13:14:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcf0dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0b2cc0 con 0x7fdcec0cd8e0 2023-03-03T13:14:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcf0dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0b2ed0 con 0x7fdcec0cd390 2023-03-03T13:14:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcead9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcec153290 0x7fdcec155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcea59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd390 0x7fdcec1efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdce9d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcec0cd8e0 0x7fdcec0b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcead9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcec153290 0x7fdcec155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55358/0 (socket says 172.21.15.102:55358) 2023-03-03T13:14:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcead9d700 1 -- 172.21.15.102:0/2268662211 learned_addr learned my addr 172.21.15.102:0/2268662211 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:14:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcead9d700 1 -- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd390 msgr2=0x7fdcec1efed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcead9d700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd390 0x7fdcec1efed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcead9d700 1 -- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcec0cd8e0 msgr2=0x7fdcec0b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.965+0000 7fdcead9d700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcec0cd8e0 0x7fdcec0b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdcead9d700 1 -- 172.21.15.102:0/2268662211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009a70 con 0x7fdcec153290 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdce9d9b700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcec0cd8e0 0x7fdcec0b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdcead9d700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcec153290 0x7fdcec155800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdce400efc0 tx=0x7fdce400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdcdaffd700 1 -- 172.21.15.102:0/2268662211 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400ca90 con 0x7fdcec153290 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdcdaffd700 1 -- 172.21.15.102:0/2268662211 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce400f040 con 0x7fdcec153290 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce00094a0 con 0x7fdcec153290 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdcdaffd700 1 -- 172.21.15.102:0/2268662211 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400ce30 con 0x7fdcec153290 2023-03-03T13:14:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.966+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec0ccde0 con 0x7fdcec153290 2023-03-03T13:14:15.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.967+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec15d160 con 0x7fdcec153290 2023-03-03T13:14:15.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.969+0000 7fdcdaffd700 1 -- 172.21.15.102:0/2268662211 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdce4014480 con 0x7fdcec153290 2023-03-03T13:14:15.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.970+0000 7fdcdaffd700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdccc0682f0 0x7fdccc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:15.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.970+0000 7fdcdaffd700 1 -- 172.21.15.102:0/2268662211 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdce40866a0 con 0x7fdcec153290 2023-03-03T13:14:15.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.970+0000 7fdcea59c700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdccc0682f0 0x7fdccc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:15.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.971+0000 7fdcea59c700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdccc0682f0 0x7fdccc06a7a0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc007fd0 tx=0x7fdcdc007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:15.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:15.974+0000 7fdcdaffd700 1 -- 172.21.15.102:0/2268662211 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdce4052cc0 con 0x7fdcec153290 2023-03-03T13:14:16.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.209+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcec0cd0c0 con 0x7fdcec153290 2023-03-03T13:14:16.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.210+0000 7fdcdaffd700 1 -- 172.21.15.102:0/2268662211 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdce4009cd0 con 0x7fdcec153290 2023-03-03T13:14:16.212 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:14:16.213 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:14:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdccc0682f0 msgr2=0x7fdccc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdccc0682f0 0x7fdccc06a7a0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc007fd0 tx=0x7fdcdc007dc0 comp rx=0 tx=0).stop 2023-03-03T13:14:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcec153290 msgr2=0x7fdcec155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcec153290 0x7fdcec155800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdce400efc0 tx=0x7fdce400c3a0 comp rx=0 tx=0).stop 2023-03-03T13:14:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 shutdown_connections 2023-03-03T13:14:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdccc0682f0 0x7fdccc06a7a0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcec0cd390 0x7fdcec1efed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcec153290 0x7fdcec155800 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 --2- 172.21.15.102:0/2268662211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcec0cd8e0 0x7fdcec0b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 >> 172.21.15.102:0/2268662211 conn(0x7fdcec065fb0 msgr2=0x7fdcec0cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:14:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 shutdown_connections 2023-03-03T13:14:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:16.213+0000 7fdcf0dff700 1 -- 172.21.15.102:0/2268662211 wait complete. 2023-03-03T13:14:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:14:16.242 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-03-03T13:14:24.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:14:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.507+0000 7ff7721ff700 1 Processor -- start 2023-03-03T13:14:24.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.507+0000 7ff7721ff700 1 -- start start 2023-03-03T13:14:24.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff7721ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff76c153290 0x7ff76c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:24.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff7721ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c153660 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff7721ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c06c360 con 0x7ff76c1593c0 2023-03-03T13:14:24.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff7721ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76c0cec40 con 0x7ff76c153290 2023-03-03T13:14:24.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff76bfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff76c0cd8e0 0x7ff76c0ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40164/0 (socket says 172.21.15.102:40164) 2023-03-03T13:14:24.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff76bfff700 1 -- 172.21.15.102:0/1660644744 learned_addr learned my addr 172.21.15.102:0/1660644744 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff76b7fe700 1 --2- 172.21.15.102:0/1660644744 >> v2:172.21.15.172:3300/0 conn(0x7ff76c153290 0x7ff76c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.509+0000 7ff76a7fc700 1 -- 172.21.15.102:0/1660644744 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104130612 0 0) 0x7ff76c153660 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76a7fc700 1 -- 172.21.15.102:0/1660644744 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7540035b0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76a7fc700 1 -- 172.21.15.102:0/1660644744 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581439604 0 0) 0x7ff76c06c360 con 0x7ff76c1593c0 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76b7fe700 1 -- 172.21.15.102:0/1660644744 >> v1:172.21.15.172:6789/0 conn(0x7ff76c1593c0 legacy=0x7ff76c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76b7fe700 1 -- 172.21.15.102:0/1660644744 >> v1:172.21.15.102:6789/0 conn(0x7ff76c0cd8e0 legacy=0x7ff76c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76b7fe700 1 -- 172.21.15.102:0/1660644744 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c15cea0 con 0x7ff76c153290 2023-03-03T13:14:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76b7fe700 1 --2- 172.21.15.102:0/1660644744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c158e80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009840 tx=0x7ff75c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee9b6bc30115b641 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76a7fc700 1 -- 172.21.15.102:0/1660644744 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c011830 con 0x7ff76c153290 2023-03-03T13:14:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76a7fc700 1 -- 172.21.15.102:0/1660644744 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c011990 con 0x7ff76c153290 2023-03-03T13:14:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.510+0000 7ff76a7fc700 1 -- 172.21.15.102:0/1660644744 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c011d20 con 0x7ff76c153290 2023-03-03T13:14:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.511+0000 7ff7721ff700 1 -- 172.21.15.102:0/1660644744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 msgr2=0x7ff76c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.511+0000 7ff7721ff700 1 --2- 172.21.15.102:0/1660644744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c158e80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009840 tx=0x7ff75c009b50 comp rx=0 tx=0).stop 2023-03-03T13:14:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.511+0000 7ff7721ff700 1 -- 172.21.15.102:0/1660644744 shutdown_connections 2023-03-03T13:14:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.511+0000 7ff7721ff700 1 --2- 172.21.15.102:0/1660644744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c158e80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.511+0000 7ff7721ff700 1 -- 172.21.15.102:0/1660644744 >> 172.21.15.102:0/1660644744 conn(0x7ff76c065fb0 msgr2=0x7ff76c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:14:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.511+0000 7ff7721ff700 1 -- 172.21.15.102:0/1660644744 shutdown_connections 2023-03-03T13:14:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.511+0000 7ff7721ff700 1 -- 172.21.15.102:0/1660644744 wait complete. 2023-03-03T13:14:24.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.542+0000 7ff7721ff700 1 Processor -- start 2023-03-03T13:14:24.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.542+0000 7ff7721ff700 1 -- start start 2023-03-03T13:14:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.543+0000 7ff7721ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd8e0 0x7ff76c13f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.543+0000 7ff7721ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.543+0000 7ff7721ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c1593c0 0x7ff76c0c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.543+0000 7ff7721ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0c8f40 con 0x7ff76c1593c0 2023-03-03T13:14:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.543+0000 7ff7721ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c1426b0 con 0x7ff76c153290 2023-03-03T13:14:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.543+0000 7ff7721ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0b35d0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c1593c0 0x7ff76c0c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd8e0 0x7ff76c13f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c1593c0 0x7ff76c0c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42076/0 (socket says 172.21.15.102:42076) 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76bfff700 1 -- 172.21.15.102:0/2712559539 learned_addr learned my addr 172.21.15.102:0/2712559539 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76b7fe700 1 -- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 msgr2=0x7ff76c0c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76b7fe700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76b7fe700 1 -- 172.21.15.102:0/2712559539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c1593c0 msgr2=0x7ff76c0c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:24.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76b7fe700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c1593c0 0x7ff76c0c7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:24.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76b7fe700 1 -- 172.21.15.102:0/2712559539 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c0094a0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76bfff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c1593c0 0x7ff76c0c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:14:24.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.544+0000 7ff76b7fe700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd8e0 0x7ff76c13f840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff75c016000 tx=0x7ff75c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:24.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.545+0000 7ff753fff700 1 -- 172.21.15.102:0/2712559539 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c01a7e0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.545+0000 7ff753fff700 1 -- 172.21.15.102:0/2712559539 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c01a940 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.545+0000 7ff753fff700 1 -- 172.21.15.102:0/2712559539 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c01acf0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.545+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c15d160 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.545+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c06c7f0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.546+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c0ccde0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.547+0000 7ff753fff700 1 -- 172.21.15.102:0/2712559539 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff75c00f050 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.548+0000 7ff753fff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7580682f0 0x7ff75806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:24.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.548+0000 7ff753fff700 1 -- 172.21.15.102:0/2712559539 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff75c086b80 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.549+0000 7ff76affd700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7580682f0 0x7ff75806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:24.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.549+0000 7ff76affd700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7580682f0 0x7ff75806a7a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0c23e0 tx=0x7ff760009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:24.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.552+0000 7ff753fff700 1 -- 172.21.15.102:0/2712559539 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff75c053290 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.787+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff76c04e6e0 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.788+0000 7ff753fff700 1 -- 172.21.15.102:0/2712559539 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff75c015090 con 0x7ff76c0cd8e0 2023-03-03T13:14:24.791 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:14:24.791 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:14:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7580682f0 msgr2=0x7ff75806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7580682f0 0x7ff75806a7a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0c23e0 tx=0x7ff760009040 comp rx=0 tx=0).stop 2023-03-03T13:14:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd8e0 msgr2=0x7ff76c13f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd8e0 0x7ff76c13f840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff75c016000 tx=0x7ff75c004750 comp rx=0 tx=0).stop 2023-03-03T13:14:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 shutdown_connections 2023-03-03T13:14:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7580682f0 0x7ff75806a7a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff76c0cd8e0 0x7ff76c13f840 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff76c1593c0 0x7ff76c0c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 --2- 172.21.15.102:0/2712559539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff76c153290 0x7ff76c0c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 >> 172.21.15.102:0/2712559539 conn(0x7ff76c065fb0 msgr2=0x7ff76c0cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:14:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 shutdown_connections 2023-03-03T13:14:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:14:24.791+0000 7ff7721ff700 1 -- 172.21.15.102:0/2712559539 wait complete. 2023-03-03T13:14:24.796 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:14:24.820 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T13:14:44.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:44.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:12.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:15:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.076+0000 7f6518bff700 1 Processor -- start 2023-03-03T13:15:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.077+0000 7f6518bff700 1 -- start start 2023-03-03T13:15:13.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6518bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65140cd8e0 0x7f65140cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:13.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6518bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65140cd390 0x7f65140ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6518bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6514153660 con 0x7f6514153290 2023-03-03T13:15:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6518bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651406c360 con 0x7f65140cd8e0 2023-03-03T13:15:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6518bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651415cea0 con 0x7f65140cd390 2023-03-03T13:15:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6511d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65140cd8e0 0x7f65140cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6512d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65140cd390 0x7f65140ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6511d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65140cd8e0 0x7f65140cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46966/0 (socket says 172.21.15.102:46966) 2023-03-03T13:15:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6511d9b700 1 -- 172.21.15.102:0/1892196337 learned_addr learned my addr 172.21.15.102:0/1892196337 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:15:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.078+0000 7f6512d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65140cd390 0x7f65140ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:51112/0 (socket says 172.21.15.102:51112) 2023-03-03T13:15:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.079+0000 7f6511d9b700 1 -- 172.21.15.102:0/1892196337 >> v2:172.21.15.172:3300/0 conn(0x7f65140cd390 msgr2=0x7f65140ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:15:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.079+0000 7f6511d9b700 1 --2- 172.21.15.102:0/1892196337 >> v2:172.21.15.172:3300/0 conn(0x7f65140cd390 0x7f65140ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.079+0000 7f6511d9b700 1 -- 172.21.15.102:0/1892196337 >> v1:172.21.15.102:6789/0 conn(0x7f6514153290 legacy=0x7f6514158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.079+0000 7f6511d9b700 1 -- 172.21.15.102:0/1892196337 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651406c7f0 con 0x7f65140cd8e0 2023-03-03T13:15:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.079+0000 7f6511d9b700 1 --2- 172.21.15.102:0/1892196337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65140cde20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f65140c9390 tx=0x7f6508009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40c5542bb34622f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:15:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.079+0000 7f651159a700 1 -- 172.21.15.102:0/1892196337 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508011830 con 0x7f65140cd8e0 2023-03-03T13:15:13.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f651159a700 1 -- 172.21.15.102:0/1892196337 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6508011990 con 0x7f65140cd8e0 2023-03-03T13:15:13.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f651159a700 1 -- 172.21.15.102:0/1892196337 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508011d20 con 0x7f65140cd8e0 2023-03-03T13:15:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 -- 172.21.15.102:0/1892196337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 msgr2=0x7f65140cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 --2- 172.21.15.102:0/1892196337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65140cde20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f65140c9390 tx=0x7f6508009bd0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 -- 172.21.15.102:0/1892196337 shutdown_connections 2023-03-03T13:15:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 --2- 172.21.15.102:0/1892196337 >> v2:172.21.15.172:3300/0 conn(0x7f65140cd390 0x7f65140ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 --2- 172.21.15.102:0/1892196337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65140cde20 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 -- 172.21.15.102:0/1892196337 >> 172.21.15.102:0/1892196337 conn(0x7f6514065fb0 msgr2=0x7f65140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:15:13.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 -- 172.21.15.102:0/1892196337 shutdown_connections 2023-03-03T13:15:13.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.080+0000 7f6518bff700 1 -- 172.21.15.102:0/1892196337 wait complete. 2023-03-03T13:15:13.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.082+0000 7f6518bff700 1 Processor -- start 2023-03-03T13:15:13.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.083+0000 7f6518bff700 1 -- start start 2023-03-03T13:15:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.083+0000 7f6518bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65140cd390 0x7f65141516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6518bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65141effa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6518bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6514153290 0x7f65141f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6518bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65141e57b0 con 0x7f65140cd8e0 2023-03-03T13:15:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6518bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65141e5a90 con 0x7f6514153290 2023-03-03T13:15:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6518bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65141e5d70 con 0x7f65140cd390 2023-03-03T13:15:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65141effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f651259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65140cd390 0x7f65141516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65141effa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46974/0 (socket says 172.21.15.102:46974) 2023-03-03T13:15:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 -- 172.21.15.102:0/3357565190 learned_addr learned my addr 172.21.15.102:0/3357565190 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:15:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6512d9d700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6514153290 0x7f65141f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 -- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65140cd390 msgr2=0x7f65141516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65140cd390 0x7f65141516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 -- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6514153290 msgr2=0x7f65141f04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6514153290 0x7f65141f04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.084+0000 7f6511d9b700 1 -- 172.21.15.102:0/3357565190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6508009550 con 0x7f65140cd8e0 2023-03-03T13:15:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.085+0000 7f6511d9b700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65141effa0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6508000c00 tx=0x7f6508004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:15:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.085+0000 7f6502ffd700 1 -- 172.21.15.102:0/3357565190 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508004380 con 0x7f65140cd8e0 2023-03-03T13:15:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.085+0000 7f6502ffd700 1 -- 172.21.15.102:0/3357565190 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650801f070 con 0x7f65140cd8e0 2023-03-03T13:15:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.085+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65140ccde0 con 0x7f65140cd8e0 2023-03-03T13:15:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.085+0000 7f6502ffd700 1 -- 172.21.15.102:0/3357565190 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65080036a0 con 0x7f65140cd8e0 2023-03-03T13:15:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.085+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651415d160 con 0x7f65140cd8e0 2023-03-03T13:15:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.086+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65140cd0c0 con 0x7f65140cd8e0 2023-03-03T13:15:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.088+0000 7f6502ffd700 1 -- 172.21.15.102:0/3357565190 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f650802a020 con 0x7f65140cd8e0 2023-03-03T13:15:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.089+0000 7f6502ffd700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64f40682f0 0x7f64f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.089+0000 7f6502ffd700 1 -- 172.21.15.102:0/3357565190 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6508087390 con 0x7f65140cd8e0 2023-03-03T13:15:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.090+0000 7f651259c700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64f40682f0 0x7f64f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:13.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.090+0000 7f651259c700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64f40682f0 0x7f64f406a7a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f6504007fd0 tx=0x7f6504009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:15:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.094+0000 7f6502ffd700 1 -- 172.21.15.102:0/3357565190 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6508053a20 con 0x7f65140cd8e0 2023-03-03T13:15:13.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.325+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65141efcc0 con 0x7f65140cd8e0 2023-03-03T13:15:13.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.326+0000 7f6502ffd700 1 -- 172.21.15.102:0/3357565190 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f65141efcc0 con 0x7f65140cd8e0 2023-03-03T13:15:13.328 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:15:13.329 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:15:13.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.328+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64f40682f0 msgr2=0x7f64f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:13.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.328+0000 7f6518bff700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64f40682f0 0x7f64f406a7a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f6504007fd0 tx=0x7f6504009040 comp rx=0 tx=0).stop 2023-03-03T13:15:13.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.328+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 msgr2=0x7f65141effa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:13.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.328+0000 7f6518bff700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65141effa0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6508000c00 tx=0x7f6508004750 comp rx=0 tx=0).stop 2023-03-03T13:15:13.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 shutdown_connections 2023-03-03T13:15:13.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64f40682f0 0x7f64f406a7a0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65140cd390 0x7f65141516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6514153290 0x7f65141f04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 --2- 172.21.15.102:0/3357565190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65140cd8e0 0x7f65141effa0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:13.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 >> 172.21.15.102:0/3357565190 conn(0x7f6514065fb0 msgr2=0x7f6514158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:15:13.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 shutdown_connections 2023-03-03T13:15:13.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:15:13.329+0000 7f6518bff700 1 -- 172.21.15.102:0/3357565190 wait complete. 2023-03-03T13:15:13.334 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:15:13.359 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 61 secs before thrashing 2023-03-03T13:15:14.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:14.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:45.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:45.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:14.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:16:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.610+0000 7f702c3ff700 1 Processor -- start 2023-03-03T13:16:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.611+0000 7f702c3ff700 1 -- start start 2023-03-03T13:16:14.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702c3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70240cd8e0 0x7f70240cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:14.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702c3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70240cd390 0x7f70240ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:14.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702c3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7024153660 con 0x7f7024153290 2023-03-03T13:16:14.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702c3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702406c360 con 0x7f70240cd390 2023-03-03T13:16:14.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702c3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702415cea0 con 0x7f70240cd8e0 2023-03-03T13:16:14.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702a99c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70240cd390 0x7f70240ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:14.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702a99c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70240cd390 0x7f70240ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58684/0 (socket says 172.21.15.102:58684) 2023-03-03T13:16:14.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.612+0000 7f702a99c700 1 -- 172.21.15.102:0/2038075392 learned_addr learned my addr 172.21.15.102:0/2038075392 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:16:14.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.613+0000 7f702999a700 1 --2- 172.21.15.102:0/2038075392 >> v2:172.21.15.172:3300/0 conn(0x7f70240cd8e0 0x7f70240cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.613+0000 7f702a99c700 1 -- 172.21.15.102:0/2038075392 >> v2:172.21.15.172:3300/0 conn(0x7f70240cd8e0 msgr2=0x7f70240cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.613+0000 7f702a99c700 1 --2- 172.21.15.102:0/2038075392 >> v2:172.21.15.172:3300/0 conn(0x7f70240cd8e0 0x7f70240cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.613+0000 7f702a99c700 1 -- 172.21.15.102:0/2038075392 >> v1:172.21.15.172:6789/0 conn(0x7f7024153290 legacy=0x7f7024158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.613+0000 7f702a99c700 1 -- 172.21.15.102:0/2038075392 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702406c7f0 con 0x7f70240cd390 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.613+0000 7f702a99c700 1 --2- 172.21.15.102:0/2038075392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70240cd390 0x7f70240ce360 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7024159fb0 tx=0x7f701c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b6d05fdb7cbb26 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.613+0000 7f7029199700 1 -- 172.21.15.102:0/2038075392 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c00e070 con 0x7f70240cd390 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f7029199700 1 -- 172.21.15.102:0/2038075392 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c00b910 con 0x7f70240cd390 2023-03-03T13:16:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f7029199700 1 -- 172.21.15.102:0/2038075392 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c00bcc0 con 0x7f70240cd390 2023-03-03T13:16:14.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 -- 172.21.15.102:0/2038075392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70240cd390 msgr2=0x7f70240ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:14.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 --2- 172.21.15.102:0/2038075392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70240cd390 0x7f70240ce360 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7024159fb0 tx=0x7f701c00b330 comp rx=0 tx=0).stop 2023-03-03T13:16:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 -- 172.21.15.102:0/2038075392 shutdown_connections 2023-03-03T13:16:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 --2- 172.21.15.102:0/2038075392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70240cd390 0x7f70240ce360 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 --2- 172.21.15.102:0/2038075392 >> v2:172.21.15.172:3300/0 conn(0x7f70240cd8e0 0x7f70240cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 -- 172.21.15.102:0/2038075392 >> 172.21.15.102:0/2038075392 conn(0x7f7024065fb0 msgr2=0x7f70240663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:16:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 -- 172.21.15.102:0/2038075392 shutdown_connections 2023-03-03T13:16:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.614+0000 7f702c3ff700 1 -- 172.21.15.102:0/2038075392 wait complete. 2023-03-03T13:16:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.616+0000 7f702c3ff700 1 Processor -- start 2023-03-03T13:16:14.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.616+0000 7f702c3ff700 1 -- start start 2023-03-03T13:16:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.617+0000 7f702c3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70240cd390 0x7f70241e4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.617+0000 7f702c3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cd8e0 0x7f7024158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.617+0000 7f702c3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7024153290 0x7f70241f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.617+0000 7f702c3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70240c8310 con 0x7f7024153290 2023-03-03T13:16:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.617+0000 7f702c3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7024153b60 con 0x7f70240cd8e0 2023-03-03T13:16:14.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.617+0000 7f702c3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70241f2aa0 con 0x7f70240cd390 2023-03-03T13:16:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cd8e0 0x7f7024158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70240cd390 0x7f70241e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cd8e0 0x7f7024158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60766/0 (socket says 172.21.15.102:60766) 2023-03-03T13:16:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702999a700 1 -- 172.21.15.102:0/4119453676 learned_addr learned my addr 172.21.15.102:0/4119453676 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:16:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702a99c700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7024153290 0x7f70241f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702a19b700 1 -- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cd8e0 msgr2=0x7f7024158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:14.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702a19b700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cd8e0 0x7f7024158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702a19b700 1 -- 172.21.15.102:0/4119453676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7024153290 msgr2=0x7f70241f0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702a19b700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7024153290 0x7f70241f0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702a19b700 1 -- 172.21.15.102:0/4119453676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c009b30 con 0x7f70240cd390 2023-03-03T13:16:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.618+0000 7f702999a700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cd8e0 0x7f7024158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:16:14.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.619+0000 7f702a19b700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70240cd390 0x7f70241e4320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7020000f60 tx=0x7f702000c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:16:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.619+0000 7f701affd700 1 -- 172.21.15.102:0/4119453676 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70200052d0 con 0x7f70240cd390 2023-03-03T13:16:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.619+0000 7f701affd700 1 -- 172.21.15.102:0/4119453676 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7020005430 con 0x7f70240cd390 2023-03-03T13:16:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.619+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70240ccde0 con 0x7f70240cd390 2023-03-03T13:16:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.619+0000 7f701affd700 1 -- 172.21.15.102:0/4119453676 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70200038c0 con 0x7f70240cd390 2023-03-03T13:16:14.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.619+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702415d160 con 0x7f70240cd390 2023-03-03T13:16:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.620+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70240cd0c0 con 0x7f70240cd390 2023-03-03T13:16:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.621+0000 7f701affd700 1 -- 172.21.15.102:0/4119453676 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7020019030 con 0x7f70240cd390 2023-03-03T13:16:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.623+0000 7f701affd700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70080682f0 0x7f700806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:14.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.623+0000 7f701affd700 1 -- 172.21.15.102:0/4119453676 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7020084d20 con 0x7f70240cd390 2023-03-03T13:16:14.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.623+0000 7f702999a700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70080682f0 0x7f700806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:14.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.624+0000 7f702999a700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70080682f0 0x7f700806a7a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f70241e38a0 tx=0x7f7014009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:16:14.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.627+0000 7f701affd700 1 -- 172.21.15.102:0/4119453676 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7020051430 con 0x7f70240cd390 2023-03-03T13:16:14.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.863+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f702404e6e0 con 0x7f70240cd390 2023-03-03T13:16:14.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.864+0000 7f701affd700 1 -- 172.21.15.102:0/4119453676 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7020054a50 con 0x7f70240cd390 2023-03-03T13:16:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:16:14.865 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:16:14.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70080682f0 msgr2=0x7f700806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70080682f0 0x7f700806a7a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f70241e38a0 tx=0x7f7014009210 comp rx=0 tx=0).stop 2023-03-03T13:16:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70240cd390 msgr2=0x7f70241e4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70240cd390 0x7f70241e4320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7020000f60 tx=0x7f702000c6e0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 shutdown_connections 2023-03-03T13:16:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70080682f0 0x7f700806a7a0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70240cd390 0x7f70241e4320 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7024153290 0x7f70241f0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 --2- 172.21.15.102:0/4119453676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70240cd8e0 0x7f7024158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 >> 172.21.15.102:0/4119453676 conn(0x7f7024065fb0 msgr2=0x7f7024159270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:16:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 shutdown_connections 2023-03-03T13:16:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:16:14.866+0000 7f702c3ff700 1 -- 172.21.15.102:0/4119453676 wait complete. 2023-03-03T13:16:14.869 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:16:14.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 51 secs before thrashing 2023-03-03T13:16:15.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:15.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:45.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:45.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:05.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:17:06.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.150+0000 7ff134b9d700 1 Processor -- start 2023-03-03T13:17:06.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.150+0000 7ff134b9d700 1 -- start start 2023-03-03T13:17:06.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.152+0000 7ff134b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300c5a90 0x7ff1300c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.152+0000 7ff134b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1300c3d20 con 0x7ff1300b3970 2023-03-03T13:17:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.152+0000 7ff134b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13015cba0 con 0x7ff1300ced00 2023-03-03T13:17:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.152+0000 7ff134b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300c4930 con 0x7ff1300c5a90 2023-03-03T13:17:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.152+0000 7ff12effd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff1300b3970 0x7ff130161470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44008/0 (socket says 172.21.15.102:44008) 2023-03-03T13:17:06.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.152+0000 7ff12effd700 1 -- 172.21.15.102:0/2957688445 learned_addr learned my addr 172.21.15.102:0/2957688445 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:17:06.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.152+0000 7ff12ffff700 1 --2- 172.21.15.102:0/2957688445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300c5a90 0x7ff1300c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:06.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530518008 0 0) 0x7ff1300c3d20 con 0x7ff1300b3970 2023-03-03T13:17:06.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1180035b0 con 0x7ff1300b3970 2023-03-03T13:17:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372306667 0 0) 0x7ff13015cba0 con 0x7ff1300ced00 2023-03-03T13:17:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1300c3d20 con 0x7ff1300ced00 2023-03-03T13:17:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974791586 0 0) 0x7ff1180035b0 con 0x7ff1300b3970 2023-03-03T13:17:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12ffff700 1 -- 172.21.15.102:0/2957688445 >> v1:172.21.15.172:6789/0 conn(0x7ff1300ced00 legacy=0x7ff130158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12ffff700 1 -- 172.21.15.102:0/2957688445 >> v1:172.21.15.102:6789/0 conn(0x7ff1300b3970 legacy=0x7ff130161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12ffff700 1 -- 172.21.15.102:0/2957688445 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13015ce80 con 0x7ff1300c5a90 2023-03-03T13:17:06.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.153+0000 7ff12ffff700 1 --2- 172.21.15.102:0/2957688445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300c5a90 0x7ff1300c41b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff128009dc0 tx=0x7ff12800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2174fb1e887e352 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12800e070 con 0x7ff1300c5a90 2023-03-03T13:17:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff128003a70 con 0x7ff1300c5a90 2023-03-03T13:17:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff12e7fc700 1 -- 172.21.15.102:0/2957688445 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124001670 con 0x7ff1300c5a90 2023-03-03T13:17:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff134b9d700 1 -- 172.21.15.102:0/2957688445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300c5a90 msgr2=0x7ff1300c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:06.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2957688445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300c5a90 0x7ff1300c41b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff128009dc0 tx=0x7ff12800b330 comp rx=0 tx=0).stop 2023-03-03T13:17:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff134b9d700 1 -- 172.21.15.102:0/2957688445 shutdown_connections 2023-03-03T13:17:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2957688445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300c5a90 0x7ff1300c41b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff134b9d700 1 -- 172.21.15.102:0/2957688445 >> 172.21.15.102:0/2957688445 conn(0x7ff13008b1c0 msgr2=0x7ff130071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff134b9d700 1 -- 172.21.15.102:0/2957688445 shutdown_connections 2023-03-03T13:17:06.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.154+0000 7ff134b9d700 1 -- 172.21.15.102:0/2957688445 wait complete. 2023-03-03T13:17:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.156+0000 7ff134b9d700 1 Processor -- start 2023-03-03T13:17:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.157+0000 7ff134b9d700 1 -- start start 2023-03-03T13:17:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff134b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1300b3970 0x7ff1300c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff134b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1300c5a90 0x7ff13014d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff134b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300ced00 0x7ff13014dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff134b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13014e230 con 0x7ff1300c5a90 2023-03-03T13:17:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff134b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1301e1f00 con 0x7ff1300b3970 2023-03-03T13:17:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff134b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1301e27a0 con 0x7ff1300ced00 2023-03-03T13:17:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff12effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1300c5a90 0x7ff13014d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff12ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300ced00 0x7ff13014dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff12f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1300b3970 0x7ff1300c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff12ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300ced00 0x7ff13014dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41408/0 (socket says 172.21.15.102:41408) 2023-03-03T13:17:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.158+0000 7ff12ffff700 1 -- 172.21.15.102:0/2464268753 learned_addr learned my addr 172.21.15.102:0/2464268753 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:17:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12ffff700 1 -- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1300b3970 msgr2=0x7ff1300c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12ffff700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1300b3970 0x7ff1300c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12ffff700 1 -- 172.21.15.102:0/2464268753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1300c5a90 msgr2=0x7ff13014d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12ffff700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1300c5a90 0x7ff13014d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12ffff700 1 -- 172.21.15.102:0/2464268753 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128009a70 con 0x7ff1300ced00 2023-03-03T13:17:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12f7fe700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1300b3970 0x7ff1300c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:17:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12effd700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1300c5a90 0x7ff13014d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:17:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff12ffff700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300ced00 0x7ff13014dcf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff128004000 tx=0x7ff128004b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff117fff700 1 -- 172.21.15.102:0/2464268753 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280044c0 con 0x7ff1300ced00 2023-03-03T13:17:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff117fff700 1 -- 172.21.15.102:0/2464268753 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff128004620 con 0x7ff1300ced00 2023-03-03T13:17:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff117fff700 1 -- 172.21.15.102:0/2464268753 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12801d470 con 0x7ff1300ced00 2023-03-03T13:17:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13006c200 con 0x7ff1300ced00 2023-03-03T13:17:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.159+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13006bf20 con 0x7ff1300ced00 2023-03-03T13:17:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.160+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13015c890 con 0x7ff1300ced00 2023-03-03T13:17:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.162+0000 7ff117fff700 1 -- 172.21.15.102:0/2464268753 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff128018030 con 0x7ff1300ced00 2023-03-03T13:17:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.163+0000 7ff117fff700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff11c0682f0 0x7ff11c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:06.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.163+0000 7ff117fff700 1 -- 172.21.15.102:0/2464268753 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff128087190 con 0x7ff1300ced00 2023-03-03T13:17:06.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.163+0000 7ff12f7fe700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff11c0682f0 0x7ff11c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:06.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.164+0000 7ff12f7fe700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff11c0682f0 0x7ff11c06a7a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff120001220 tx=0x7ff120009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:06.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.167+0000 7ff117fff700 1 -- 172.21.15.102:0/2464268753 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff1280538a0 con 0x7ff1300ced00 2023-03-03T13:17:06.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.405+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1300c9860 con 0x7ff1300ced00 2023-03-03T13:17:06.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.407+0000 7ff117fff700 1 -- 172.21.15.102:0/2464268753 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff128056ec0 con 0x7ff1300ced00 2023-03-03T13:17:06.408 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:17:06.408 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:17:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff11c0682f0 msgr2=0x7ff11c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff11c0682f0 0x7ff11c06a7a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff120001220 tx=0x7ff120009040 comp rx=0 tx=0).stop 2023-03-03T13:17:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300ced00 msgr2=0x7ff13014dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300ced00 0x7ff13014dcf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff128004000 tx=0x7ff128004b20 comp rx=0 tx=0).stop 2023-03-03T13:17:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 shutdown_connections 2023-03-03T13:17:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff11c0682f0 0x7ff11c06a7a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1300ced00 0x7ff13014dcf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1300b3970 0x7ff1300c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 --2- 172.21.15.102:0/2464268753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1300c5a90 0x7ff13014d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 >> 172.21.15.102:0/2464268753 conn(0x7ff13008b1c0 msgr2=0x7ff130065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:06.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.409+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 shutdown_connections 2023-03-03T13:17:06.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:17:06.410+0000 7ff134b9d700 1 -- 172.21.15.102:0/2464268753 wait complete. 2023-03-03T13:17:06.413 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:17:06.438 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-03-03T13:17:15.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:15.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:45.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:45.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:16.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:16.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:46.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:46.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:56.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:18:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.685+0000 7f423bdff700 1 Processor -- start 2023-03-03T13:18:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.686+0000 7f423bdff700 1 -- start start 2023-03-03T13:18:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f423bdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42340cd8e0 0x7f42340cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f423bdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234153660 con 0x7f42340cd390 2023-03-03T13:18:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f423bdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423406c360 con 0x7f4234153290 2023-03-03T13:18:56.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f423bdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423415cea0 con 0x7f42340cd8e0 2023-03-03T13:18:56.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f4239b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4234153290 0x7f4234158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38470/0 (socket says 172.21.15.102:38470) 2023-03-03T13:18:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f423a39c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f42340cd390 0x7f42340ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51406/0 (socket says 172.21.15.102:51406) 2023-03-03T13:18:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f4239b9b700 1 -- 172.21.15.102:0/3607498937 learned_addr learned my addr 172.21.15.102:0/3607498937 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:18:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.687+0000 7f423939a700 1 --2- 172.21.15.102:0/3607498937 >> v2:172.21.15.102:3300/0 conn(0x7f42340cd8e0 0x7f42340cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f4238b99700 1 -- 172.21.15.102:0/3607498937 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389535796 0 0) 0x7f423406c360 con 0x7f4234153290 2023-03-03T13:18:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f4238b99700 1 -- 172.21.15.102:0/3607498937 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42200035b0 con 0x7f4234153290 2023-03-03T13:18:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f423939a700 1 -- 172.21.15.102:0/3607498937 >> v1:172.21.15.172:6789/0 conn(0x7f4234153290 legacy=0x7f4234158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:56.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f423939a700 1 -- 172.21.15.102:0/3607498937 >> v1:172.21.15.102:6789/0 conn(0x7f42340cd390 legacy=0x7f42340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f4238b99700 1 -- 172.21.15.102:0/3607498937 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045224863 0 0) 0x7f4234153660 con 0x7f42340cd390 2023-03-03T13:18:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f423939a700 1 -- 172.21.15.102:0/3607498937 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423406c7f0 con 0x7f42340cd8e0 2023-03-03T13:18:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f423939a700 1 --2- 172.21.15.102:0/3607498937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340cde20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4224009840 tx=0x7f4224009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496113b3e3261fe6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f4238b99700 1 -- 172.21.15.102:0/3607498937 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224011830 con 0x7f42340cd8e0 2023-03-03T13:18:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f4238b99700 1 -- 172.21.15.102:0/3607498937 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4224011990 con 0x7f42340cd8e0 2023-03-03T13:18:56.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.688+0000 7f4238b99700 1 -- 172.21.15.102:0/3607498937 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224011d20 con 0x7f42340cd8e0 2023-03-03T13:18:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.689+0000 7f423bdff700 1 -- 172.21.15.102:0/3607498937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 msgr2=0x7f42340cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.689+0000 7f423bdff700 1 --2- 172.21.15.102:0/3607498937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340cde20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4224009840 tx=0x7f4224009b50 comp rx=0 tx=0).stop 2023-03-03T13:18:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.689+0000 7f423bdff700 1 -- 172.21.15.102:0/3607498937 shutdown_connections 2023-03-03T13:18:56.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.689+0000 7f423bdff700 1 --2- 172.21.15.102:0/3607498937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340cde20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.689+0000 7f423bdff700 1 -- 172.21.15.102:0/3607498937 >> 172.21.15.102:0/3607498937 conn(0x7f4234065fb0 msgr2=0x7f42340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.689+0000 7f423bdff700 1 -- 172.21.15.102:0/3607498937 shutdown_connections 2023-03-03T13:18:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.689+0000 7f423bdff700 1 -- 172.21.15.102:0/3607498937 wait complete. 2023-03-03T13:18:56.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.692+0000 7f423bdff700 1 Processor -- start 2023-03-03T13:18:56.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.692+0000 7f423bdff700 1 -- start start 2023-03-03T13:18:56.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.693+0000 7f423bdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42340cd390 0x7f42340b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:56.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423bdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:56.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423bdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4234153290 0x7f42340c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423bdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42340c9210 con 0x7f42340cd8e0 2023-03-03T13:18:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423bdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4234154750 con 0x7f42340cd390 2023-03-03T13:18:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423bdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42340c75a0 con 0x7f4234153290 2023-03-03T13:18:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36096/0 (socket says 172.21.15.102:36096) 2023-03-03T13:18:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 -- 172.21.15.102:0/3556274484 learned_addr learned my addr 172.21.15.102:0/3556274484 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:18:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423a39c700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4234153290 0x7f42340c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:56.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f4239b9b700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42340cd390 0x7f42340b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:56.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 -- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4234153290 msgr2=0x7f42340c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4234153290 0x7f42340c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 -- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42340cd390 msgr2=0x7f42340b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42340cd390 0x7f42340b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.694+0000 7f423939a700 1 -- 172.21.15.102:0/3556274484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42240094a0 con 0x7f42340cd8e0 2023-03-03T13:18:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.695+0000 7f423939a700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340c7d10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4224009fd0 tx=0x7f4224004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.695+0000 7f422a7fc700 1 -- 172.21.15.102:0/3556274484 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422401a9f0 con 0x7f42340cd8e0 2023-03-03T13:18:56.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.695+0000 7f422a7fc700 1 -- 172.21.15.102:0/3556274484 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422401ab50 con 0x7f42340cd8e0 2023-03-03T13:18:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.695+0000 7f422a7fc700 1 -- 172.21.15.102:0/3556274484 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42240243f0 con 0x7f42340cd8e0 2023-03-03T13:18:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.695+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42340ccde0 con 0x7f42340cd8e0 2023-03-03T13:18:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.695+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423415d160 con 0x7f42340cd8e0 2023-03-03T13:18:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.696+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42340cd0c0 con 0x7f42340cd8e0 2023-03-03T13:18:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.697+0000 7f422a7fc700 1 -- 172.21.15.102:0/3556274484 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f422400f050 con 0x7f42340cd8e0 2023-03-03T13:18:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.699+0000 7f422a7fc700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4218068320 0x7f421806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.699+0000 7f422a7fc700 1 -- 172.21.15.102:0/3556274484 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f42240870d0 con 0x7f42340cd8e0 2023-03-03T13:18:56.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.699+0000 7f4239b9b700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4218068320 0x7f421806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:56.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.700+0000 7f4239b9b700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4218068320 0x7f421806a7d0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f4230007fd0 tx=0x7f4230009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:56.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.703+0000 7f422a7fc700 1 -- 172.21.15.102:0/3556274484 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4224053750 con 0x7f42340cd8e0 2023-03-03T13:18:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.939+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f423404e6e0 con 0x7f42340cd8e0 2023-03-03T13:18:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.939+0000 7f422a7fc700 1 -- 172.21.15.102:0/3556274484 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4224015030 con 0x7f42340cd8e0 2023-03-03T13:18:56.942 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:18:56.942 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:18:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4218068320 msgr2=0x7f421806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4218068320 0x7f421806a7d0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f4230007fd0 tx=0x7f4230009040 comp rx=0 tx=0).stop 2023-03-03T13:18:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 msgr2=0x7f42340c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340c7d10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4224009fd0 tx=0x7f4224004750 comp rx=0 tx=0).stop 2023-03-03T13:18:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 shutdown_connections 2023-03-03T13:18:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4218068320 0x7f421806a7d0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:56.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4234153290 0x7f42340c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42340cd390 0x7f42340b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 --2- 172.21.15.102:0/3556274484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42340cd8e0 0x7f42340c7d10 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 >> 172.21.15.102:0/3556274484 conn(0x7f4234065fb0 msgr2=0x7f4234158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 shutdown_connections 2023-03-03T13:18:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:18:56.942+0000 7f423bdff700 1 -- 172.21.15.102:0/3556274484 wait complete. 2023-03-03T13:18:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:18:56.972 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2023-03-03T13:19:16.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:16.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:46.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:46.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T13:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:20:16.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:16.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:24.973 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:20:25.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.221+0000 7f42e3fff700 1 Processor -- start 2023-03-03T13:20:25.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.222+0000 7f42e3fff700 1 -- start start 2023-03-03T13:20:25.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.223+0000 7f42e3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f42e4153030 0x7f42e40c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:25.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.223+0000 7f42e3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e40b3f30 0x7f42e40cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:25.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.223+0000 7f42e3fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e406bee0 con 0x7f42e40b1e90 2023-03-03T13:20:25.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.223+0000 7f42e3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e406c780 con 0x7f42e4153030 2023-03-03T13:20:25.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.223+0000 7f42e3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e415c620 con 0x7f42e40b3f30 2023-03-03T13:20:25.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.223+0000 7f42dbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f42e40b1e90 0x7f42e415ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44646/0 (socket says 172.21.15.102:44646) 2023-03-03T13:20:25.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42dbfff700 1 -- 172.21.15.102:0/3762682204 learned_addr learned my addr 172.21.15.102:0/3762682204 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:20:25.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42e37fe700 1 --2- 172.21.15.102:0/3762682204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e40b3f30 0x7f42e40cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:25.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.223+0000 7f42e2ffd700 1 --2- 172.21.15.102:0/3762682204 >> v2:172.21.15.172:3300/0 conn(0x7f42e4153030 0x7f42e40c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:25.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42e37fe700 1 -- 172.21.15.102:0/3762682204 >> v2:172.21.15.172:3300/0 conn(0x7f42e4153030 msgr2=0x7f42e40c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:20:25.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42e37fe700 1 --2- 172.21.15.102:0/3762682204 >> v2:172.21.15.172:3300/0 conn(0x7f42e4153030 0x7f42e40c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42e37fe700 1 -- 172.21.15.102:0/3762682204 >> v1:172.21.15.102:6789/0 conn(0x7f42e40b1e90 legacy=0x7f42e415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:25.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42e37fe700 1 -- 172.21.15.102:0/3762682204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e406c1c0 con 0x7f42e40b3f30 2023-03-03T13:20:25.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42e27fc700 1 -- 172.21.15.102:0/3762682204 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237033080 0 0) 0x7f42e406bee0 con 0x7f42e40b1e90 2023-03-03T13:20:25.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.224+0000 7f42e37fe700 1 --2- 172.21.15.102:0/3762682204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e40b3f30 0x7f42e40cece0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009d90 tx=0x7f42dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b03e04ecae9ae63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:25.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e27fc700 1 -- 172.21.15.102:0/3762682204 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc00e070 con 0x7f42e40b3f30 2023-03-03T13:20:25.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e27fc700 1 -- 172.21.15.102:0/3762682204 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc00bb20 con 0x7f42e40b3f30 2023-03-03T13:20:25.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e27fc700 1 -- 172.21.15.102:0/3762682204 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc012430 con 0x7f42e40b3f30 2023-03-03T13:20:25.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 -- 172.21.15.102:0/3762682204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e40b3f30 msgr2=0x7f42e40cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:25.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 --2- 172.21.15.102:0/3762682204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e40b3f30 0x7f42e40cece0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009d90 tx=0x7f42dc00b330 comp rx=0 tx=0).stop 2023-03-03T13:20:25.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 -- 172.21.15.102:0/3762682204 shutdown_connections 2023-03-03T13:20:25.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 --2- 172.21.15.102:0/3762682204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e40b3f30 0x7f42e40cece0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 --2- 172.21.15.102:0/3762682204 >> v2:172.21.15.172:3300/0 conn(0x7f42e4153030 0x7f42e40c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 -- 172.21.15.102:0/3762682204 >> 172.21.15.102:0/3762682204 conn(0x7f42e408b1c0 msgr2=0x7f42e4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:25.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 -- 172.21.15.102:0/3762682204 shutdown_connections 2023-03-03T13:20:25.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.225+0000 7f42e3fff700 1 -- 172.21.15.102:0/3762682204 wait complete. 2023-03-03T13:20:25.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.227+0000 7f42e3fff700 1 Processor -- start 2023-03-03T13:20:25.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.228+0000 7f42e3fff700 1 -- start start 2023-03-03T13:20:25.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e40b1e90 0x7f42e40cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:25.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e40b3f30 0x7f42e41e3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:25.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e4153030 0x7f42e41e3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:25.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41e3100 con 0x7f42e40b1e90 2023-03-03T13:20:25.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41ef330 con 0x7f42e40b3f30 2023-03-03T13:20:25.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41ee4a0 con 0x7f42e4153030 2023-03-03T13:20:25.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e40b1e90 0x7f42e40cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:25.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e4153030 0x7f42e41e3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:25.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42dbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e40b3f30 0x7f42e41e3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:25.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e40b1e90 0x7f42e40cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40360/0 (socket says 172.21.15.102:40360) 2023-03-03T13:20:25.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e2ffd700 1 -- 172.21.15.102:0/4055326841 learned_addr learned my addr 172.21.15.102:0/4055326841 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:20:25.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42dbfff700 1 -- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e4153030 msgr2=0x7f42e41e3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:25.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42dbfff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e4153030 0x7f42e41e3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42dbfff700 1 -- 172.21.15.102:0/4055326841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e40b1e90 msgr2=0x7f42e40cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:25.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42dbfff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e40b1e90 0x7f42e40cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42dbfff700 1 -- 172.21.15.102:0/4055326841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc009a70 con 0x7f42e40b3f30 2023-03-03T13:20:25.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.229+0000 7f42e2ffd700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e40b1e90 0x7f42e40cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:20:25.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.230+0000 7f42dbfff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e40b3f30 0x7f42e41e3590 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f42e406a8d0 tx=0x7f42cc00c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:25.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.230+0000 7f42db7fe700 1 -- 172.21.15.102:0/4055326841 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc004c80 con 0x7f42e40b3f30 2023-03-03T13:20:25.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.230+0000 7f42db7fe700 1 -- 172.21.15.102:0/4055326841 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc004e00 con 0x7f42e40b3f30 2023-03-03T13:20:25.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.230+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e415c240 con 0x7f42e40b3f30 2023-03-03T13:20:25.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.230+0000 7f42db7fe700 1 -- 172.21.15.102:0/4055326841 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc007310 con 0x7f42e40b3f30 2023-03-03T13:20:25.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.230+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e406c4a0 con 0x7f42e40b3f30 2023-03-03T13:20:25.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.231+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e40b39e0 con 0x7f42e40b3f30 2023-03-03T13:20:25.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.233+0000 7f42db7fe700 1 -- 172.21.15.102:0/4055326841 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f42cc01a020 con 0x7f42e40b3f30 2023-03-03T13:20:25.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.234+0000 7f42db7fe700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c40682f0 0x7f42c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:25.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.234+0000 7f42db7fe700 1 -- 172.21.15.102:0/4055326841 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f42cc0858a0 con 0x7f42e40b3f30 2023-03-03T13:20:25.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.234+0000 7f42e2ffd700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c40682f0 0x7f42c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:25.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.235+0000 7f42e2ffd700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c40682f0 0x7f42c406a7a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f42d40112c0 tx=0x7f42d4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:25.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.238+0000 7f42db7fe700 1 -- 172.21.15.102:0/4055326841 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42cc051fe0 con 0x7f42e40b3f30 2023-03-03T13:20:25.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.477+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42e41e3270 con 0x7f42e40b3f30 2023-03-03T13:20:25.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.478+0000 7f42db7fe700 1 -- 172.21.15.102:0/4055326841 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f42cc00ce60 con 0x7f42e40b3f30 2023-03-03T13:20:25.481 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:20:25.481 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:20:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c40682f0 msgr2=0x7f42c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c40682f0 0x7f42c406a7a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f42d40112c0 tx=0x7f42d4009210 comp rx=0 tx=0).stop 2023-03-03T13:20:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e40b3f30 msgr2=0x7f42e41e3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e40b3f30 0x7f42e41e3590 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f42e406a8d0 tx=0x7f42cc00c790 comp rx=0 tx=0).stop 2023-03-03T13:20:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 shutdown_connections 2023-03-03T13:20:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42c40682f0 0x7f42c406a7a0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42e4153030 0x7f42e41e3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42e40b1e90 0x7f42e40cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 --2- 172.21.15.102:0/4055326841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42e40b3f30 0x7f42e41e3590 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:25.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 >> 172.21.15.102:0/4055326841 conn(0x7f42e408b1c0 msgr2=0x7f42e415ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:25.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 shutdown_connections 2023-03-03T13:20:25.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:20:25.481+0000 7f42e3fff700 1 -- 172.21.15.102:0/4055326841 wait complete. 2023-03-03T13:20:25.486 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:20:25.511 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T13:20:46.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:47.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:17.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:17.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:21.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:21:21.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.760+0000 7f1f2c3ff700 1 Processor -- start 2023-03-03T13:21:21.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.760+0000 7f1f2c3ff700 1 -- start start 2023-03-03T13:21:21.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.761+0000 7f1f2c3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f240cd8e0 0x7f1f240cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:21.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.761+0000 7f1f2c3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f240cd390 0x7f1f240ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:21.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.761+0000 7f1f2c3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f24153660 con 0x7f1f24153290 2023-03-03T13:21:21.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.761+0000 7f1f2c3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2406c360 con 0x7f1f240cd390 2023-03-03T13:21:21.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.761+0000 7f1f2c3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2415cea0 con 0x7f1f240cd8e0 2023-03-03T13:21:21.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2a99c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f240cd390 0x7f1f240ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:21.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2a19b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1f24153290 0x7f1f24158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38096/0 (socket says 172.21.15.102:38096) 2023-03-03T13:21:21.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2999a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f240cd8e0 0x7f1f240cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2a99c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f240cd390 0x7f1f240ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39216/0 (socket says 172.21.15.102:39216) 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2a19b700 1 -- 172.21.15.102:0/4203372186 learned_addr learned my addr 172.21.15.102:0/4203372186 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2999a700 1 -- 172.21.15.102:0/4203372186 >> v2:172.21.15.102:3300/0 conn(0x7f1f240cd390 msgr2=0x7f1f240ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2999a700 1 --2- 172.21.15.102:0/4203372186 >> v2:172.21.15.102:3300/0 conn(0x7f1f240cd390 0x7f1f240ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2999a700 1 -- 172.21.15.102:0/4203372186 >> v1:172.21.15.102:6789/0 conn(0x7f1f24153290 legacy=0x7f1f24158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2999a700 1 -- 172.21.15.102:0/4203372186 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2406c7f0 con 0x7f1f240cd8e0 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.762+0000 7f1f2a99c700 1 --2- 172.21.15.102:0/4203372186 >> v2:172.21.15.102:3300/0 conn(0x7f1f240cd390 0x7f1f240ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:21:21.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f2999a700 1 --2- 172.21.15.102:0/4203372186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd8e0 0x7f1f240cde20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1f24159fb0 tx=0x7f1f14009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac1c3d72339dd6af server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:21.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f29199700 1 -- 172.21.15.102:0/4203372186 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14011830 con 0x7f1f240cd8e0 2023-03-03T13:21:21.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f29199700 1 -- 172.21.15.102:0/4203372186 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f14011990 con 0x7f1f240cd8e0 2023-03-03T13:21:21.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f29199700 1 -- 172.21.15.102:0/4203372186 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14011d20 con 0x7f1f240cd8e0 2023-03-03T13:21:21.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/4203372186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd8e0 msgr2=0x7f1f240cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:21.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/4203372186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd8e0 0x7f1f240cde20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1f24159fb0 tx=0x7f1f14009b20 comp rx=0 tx=0).stop 2023-03-03T13:21:21.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/4203372186 shutdown_connections 2023-03-03T13:21:21.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/4203372186 >> v2:172.21.15.102:3300/0 conn(0x7f1f240cd390 0x7f1f240ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:21.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.763+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/4203372186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd8e0 0x7f1f240cde20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:21.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.764+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/4203372186 >> 172.21.15.102:0/4203372186 conn(0x7f1f24065fb0 msgr2=0x7f1f240663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:21.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.764+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/4203372186 shutdown_connections 2023-03-03T13:21:21.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.764+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/4203372186 wait complete. 2023-03-03T13:21:21.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.766+0000 7f1f2c3ff700 1 Processor -- start 2023-03-03T13:21:21.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.767+0000 7f1f2c3ff700 1 -- start start 2023-03-03T13:21:21.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2c3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd390 0x7f1f241f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:21.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2c3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f240cd8e0 0x7f1f240c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:21.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2c3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f24153290 0x7f1f240c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:21.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2c3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240c3850 con 0x7f1f240cd8e0 2023-03-03T13:21:21.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2c3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f241f28b0 con 0x7f1f240cd390 2023-03-03T13:21:21.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2c3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240c4500 con 0x7f1f24153290 2023-03-03T13:21:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2a19b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd390 0x7f1f241f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.768+0000 7f1f2999a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f240cd8e0 0x7f1f240c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2a19b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd390 0x7f1f241f2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57024/0 (socket says 172.21.15.102:57024) 2023-03-03T13:21:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2a99c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f24153290 0x7f1f240c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2a19b700 1 -- 172.21.15.102:0/3921508992 learned_addr learned my addr 172.21.15.102:0/3921508992 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:21:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2999a700 1 -- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f24153290 msgr2=0x7f1f240c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2999a700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f24153290 0x7f1f240c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2999a700 1 -- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd390 msgr2=0x7f1f241f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2999a700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd390 0x7f1f241f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2999a700 1 -- 172.21.15.102:0/3921508992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c009a70 con 0x7f1f240cd8e0 2023-03-03T13:21:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.769+0000 7f1f2999a700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f240cd8e0 0x7f1f240c4990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1f14016000 tx=0x7f1f14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.770+0000 7f1f1affd700 1 -- 172.21.15.102:0/3921508992 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140045d0 con 0x7f1f240cd8e0 2023-03-03T13:21:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.770+0000 7f1f1affd700 1 -- 172.21.15.102:0/3921508992 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1401f070 con 0x7f1f240cd8e0 2023-03-03T13:21:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.770+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f140094a0 con 0x7f1f240cd8e0 2023-03-03T13:21:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.770+0000 7f1f1affd700 1 -- 172.21.15.102:0/3921508992 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140038e0 con 0x7f1f240cd8e0 2023-03-03T13:21:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.770+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f240ccde0 con 0x7f1f240cd8e0 2023-03-03T13:21:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.771+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2415d160 con 0x7f1f240cd8e0 2023-03-03T13:21:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.772+0000 7f1f1affd700 1 -- 172.21.15.102:0/3921508992 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f1400f050 con 0x7f1f240cd8e0 2023-03-03T13:21:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.773+0000 7f1f1affd700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f080682f0 0x7f1f0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.773+0000 7f1f1affd700 1 -- 172.21.15.102:0/3921508992 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1f14021070 con 0x7f1f240cd8e0 2023-03-03T13:21:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.774+0000 7f1f2a19b700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f080682f0 0x7f1f0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.774+0000 7f1f2a19b700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f080682f0 0x7f1f0806a7a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1f20005910 tx=0x7f1f20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:21.778+0000 7f1f1affd700 1 -- 172.21.15.102:0/3921508992 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f14053c90 con 0x7f1f240cd8e0 2023-03-03T13:21:22.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.009+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f240cd0c0 con 0x7f1f240cd8e0 2023-03-03T13:21:22.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.009+0000 7f1f1affd700 1 -- 172.21.15.102:0/3921508992 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1f14015090 con 0x7f1f240cd8e0 2023-03-03T13:21:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:21:22.012 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:21:22.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f080682f0 msgr2=0x7f1f0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:22.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f080682f0 0x7f1f0806a7a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1f20005910 tx=0x7f1f20009040 comp rx=0 tx=0).stop 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f240cd8e0 msgr2=0x7f1f240c4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f240cd8e0 0x7f1f240c4990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1f14016000 tx=0x7f1f14004750 comp rx=0 tx=0).stop 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 shutdown_connections 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f080682f0 0x7f1f0806a7a0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f24153290 0x7f1f240c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f240cd390 0x7f1f241f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 --2- 172.21.15.102:0/3921508992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f240cd8e0 0x7f1f240c4990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 >> 172.21.15.102:0/3921508992 conn(0x7f1f24065fb0 msgr2=0x7f1f2415a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:22.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 shutdown_connections 2023-03-03T13:21:22.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:22.012+0000 7f1f2c3ff700 1 -- 172.21.15.102:0/3921508992 wait complete. 2023-03-03T13:21:22.016 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:21:22.041 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-03-03T13:21:45.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:21:45.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.296+0000 7f1fb3fff700 1 Processor -- start 2023-03-03T13:21:45.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.296+0000 7f1fb3fff700 1 -- start start 2023-03-03T13:21:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.297+0000 7f1fb3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1fb40b3970 0x7f1fb40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.297+0000 7f1fb3fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb40c3d20 con 0x7f1fb40ced00 2023-03-03T13:21:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.297+0000 7f1fb3fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb415cba0 con 0x7f1fb40c41b0 2023-03-03T13:21:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.297+0000 7f1fb3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb40c4930 con 0x7f1fb40b3970 2023-03-03T13:21:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb2ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1fb40ced00 0x7f1fb4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45282/0 (socket says 172.21.15.102:45282) 2023-03-03T13:21:45.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb2ffd700 1 -- 172.21.15.102:0/902443653 learned_addr learned my addr 172.21.15.102:0/902443653 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:21:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb27fc700 1 --2- 172.21.15.102:0/902443653 >> v2:172.21.15.172:3300/0 conn(0x7f1fb40b3970 0x7f1fb40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb1ffb700 1 -- 172.21.15.102:0/902443653 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930849407 0 0) 0x7f1fb40c3d20 con 0x7f1fb40ced00 2023-03-03T13:21:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb1ffb700 1 -- 172.21.15.102:0/902443653 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f9c0035b0 con 0x7f1fb40ced00 2023-03-03T13:21:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb1ffb700 1 -- 172.21.15.102:0/902443653 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23910397 0 0) 0x7f1fb415cba0 con 0x7f1fb40c41b0 2023-03-03T13:21:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb1ffb700 1 -- 172.21.15.102:0/902443653 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb40c3d20 con 0x7f1fb40c41b0 2023-03-03T13:21:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb27fc700 1 -- 172.21.15.102:0/902443653 >> v1:172.21.15.172:6789/0 conn(0x7f1fb40c41b0 legacy=0x7f1fb4163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb27fc700 1 -- 172.21.15.102:0/902443653 >> v1:172.21.15.102:6789/0 conn(0x7f1fb40ced00 legacy=0x7f1fb4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.298+0000 7f1fb27fc700 1 -- 172.21.15.102:0/902443653 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb415ce80 con 0x7f1fb40b3970 2023-03-03T13:21:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.299+0000 7f1fb27fc700 1 --2- 172.21.15.102:0/902443653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40b3970 0x7f1fb40c5a90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009840 tx=0x7f1fa8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30068e65bcc3ce66 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.299+0000 7f1fb1ffb700 1 -- 172.21.15.102:0/902443653 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8011830 con 0x7f1fb40b3970 2023-03-03T13:21:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.299+0000 7f1fb1ffb700 1 -- 172.21.15.102:0/902443653 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa8011990 con 0x7f1fb40b3970 2023-03-03T13:21:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.299+0000 7f1fb1ffb700 1 -- 172.21.15.102:0/902443653 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8011d20 con 0x7f1fb40b3970 2023-03-03T13:21:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.299+0000 7f1fb3fff700 1 -- 172.21.15.102:0/902443653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40b3970 msgr2=0x7f1fb40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:45.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.299+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/902443653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40b3970 0x7f1fb40c5a90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009840 tx=0x7f1fa8009b50 comp rx=0 tx=0).stop 2023-03-03T13:21:45.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.300+0000 7f1fb3fff700 1 -- 172.21.15.102:0/902443653 shutdown_connections 2023-03-03T13:21:45.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.300+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/902443653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40b3970 0x7f1fb40c5a90 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:45.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.300+0000 7f1fb3fff700 1 -- 172.21.15.102:0/902443653 >> 172.21.15.102:0/902443653 conn(0x7f1fb408b1c0 msgr2=0x7f1fb4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:45.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.300+0000 7f1fb3fff700 1 -- 172.21.15.102:0/902443653 shutdown_connections 2023-03-03T13:21:45.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.300+0000 7f1fb3fff700 1 -- 172.21.15.102:0/902443653 wait complete. 2023-03-03T13:21:45.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.302+0000 7f1fb3fff700 1 Processor -- start 2023-03-03T13:21:45.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.302+0000 7f1fb3fff700 1 -- start start 2023-03-03T13:21:45.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fb40b3970 0x7f1fb414f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:45.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40c41b0 0x7f1fb414b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:45.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fb40ced00 0x7f1fb4151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:45.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb41511b0 con 0x7f1fb40b3970 2023-03-03T13:21:45.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb414bd60 con 0x7f1fb40c41b0 2023-03-03T13:21:45.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb41523e0 con 0x7f1fb40ced00 2023-03-03T13:21:45.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fb40b3970 0x7f1fb414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:45.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fb40b3970 0x7f1fb414f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48144/0 (socket says 172.21.15.102:48144) 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb2ffd700 1 -- 172.21.15.102:0/3456274552 learned_addr learned my addr 172.21.15.102:0/3456274552 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb27fc700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40c41b0 0x7f1fb414b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.303+0000 7f1fb37fe700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fb40ced00 0x7f1fb4151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb2ffd700 1 -- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fb40ced00 msgr2=0x7f1fb4151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb2ffd700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fb40ced00 0x7f1fb4151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb2ffd700 1 -- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40c41b0 msgr2=0x7f1fb414b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb2ffd700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40c41b0 0x7f1fb414b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:45.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb2ffd700 1 -- 172.21.15.102:0/3456274552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa80094a0 con 0x7f1fb40b3970 2023-03-03T13:21:45.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb27fc700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40c41b0 0x7f1fb414b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:21:45.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb2ffd700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fb40b3970 0x7f1fb414f680 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c730 tx=0x7f1fa400ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:45.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1f9bfff700 1 -- 172.21.15.102:0/3456274552 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4005d70 con 0x7f1fb40b3970 2023-03-03T13:21:45.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1f9bfff700 1 -- 172.21.15.102:0/3456274552 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa40070d0 con 0x7f1fb40b3970 2023-03-03T13:21:45.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1f9bfff700 1 -- 172.21.15.102:0/3456274552 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4007440 con 0x7f1fb40b3970 2023-03-03T13:21:45.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.304+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb406c200 con 0x7f1fb40b3970 2023-03-03T13:21:45.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.305+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb406bf20 con 0x7f1fb40b3970 2023-03-03T13:21:45.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.306+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb415c890 con 0x7f1fb40b3970 2023-03-03T13:21:45.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.307+0000 7f1f9bfff700 1 -- 172.21.15.102:0/3456274552 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1fa4019030 con 0x7f1fb40b3970 2023-03-03T13:21:45.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.309+0000 7f1f9bfff700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1fa0068320 0x7f1fa006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:45.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.309+0000 7f1f9bfff700 1 -- 172.21.15.102:0/3456274552 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1fa4084fe0 con 0x7f1fb40b3970 2023-03-03T13:21:45.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.309+0000 7f1fb27fc700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1fa0068320 0x7f1fa006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:45.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.310+0000 7f1fb27fc700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1fa0068320 0x7f1fa006a7d0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009fd0 tx=0x7f1fa801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:45.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.313+0000 7f1f9bfff700 1 -- 172.21.15.102:0/3456274552 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1fa404b5e0 con 0x7f1fb40b3970 2023-03-03T13:21:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.550+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fb40cd970 con 0x7f1fb40b3970 2023-03-03T13:21:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.550+0000 7f1f9bfff700 1 -- 172.21.15.102:0/3456274552 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1fa4017020 con 0x7f1fb40b3970 2023-03-03T13:21:45.552 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:21:45.552 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:21:45.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1fa0068320 msgr2=0x7f1fa006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1fa0068320 0x7f1fa006a7d0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009fd0 tx=0x7f1fa801c040 comp rx=0 tx=0).stop 2023-03-03T13:21:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fb40b3970 msgr2=0x7f1fb414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fb40b3970 0x7f1fb414f680 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c730 tx=0x7f1fa400ca40 comp rx=0 tx=0).stop 2023-03-03T13:21:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 shutdown_connections 2023-03-03T13:21:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1fa0068320 0x7f1fa006a7d0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:45.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fb40ced00 0x7f1fb4151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fb40b3970 0x7f1fb414f680 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 --2- 172.21.15.102:0/3456274552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fb40c41b0 0x7f1fb414b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 >> 172.21.15.102:0/3456274552 conn(0x7f1fb408b1c0 msgr2=0x7f1fb415bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 shutdown_connections 2023-03-03T13:21:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:21:45.553+0000 7f1fb3fff700 1 -- 172.21.15.102:0/3456274552 wait complete. 2023-03-03T13:21:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:21:45.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T13:21:47.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:47.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:17.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:17.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:47.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:47.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:55.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:22:55.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.836+0000 7f45029ff700 1 Processor -- start 2023-03-03T13:22:55.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.837+0000 7f45029ff700 1 -- start start 2023-03-03T13:22:55.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.838+0000 7f45029ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f44fc153290 0x7f44fc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:55.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.838+0000 7f45029ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc1593c0 0x7f44fc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:55.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.838+0000 7f45029ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc153660 con 0x7f44fc0cd8e0 2023-03-03T13:22:55.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.838+0000 7f45029ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc06c360 con 0x7f44fc153290 2023-03-03T13:22:55.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.838+0000 7f45029ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc15cea0 con 0x7f44fc1593c0 2023-03-03T13:22:55.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.838+0000 7f44fb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc1593c0 0x7f44fc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:55.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.838+0000 7f44fbfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f44fc153290 0x7f44fc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:55.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f4500f9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f44fc0cd8e0 0x7f44fc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50966/0 (socket says 172.21.15.102:50966) 2023-03-03T13:22:55.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f4500f9c700 1 -- 172.21.15.102:0/499105695 learned_addr learned my addr 172.21.15.102:0/499105695 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:22:55.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44faffd700 1 -- 172.21.15.102:0/499105695 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806000662 0 0) 0x7f44fc153660 con 0x7f44fc0cd8e0 2023-03-03T13:22:55.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44faffd700 1 -- 172.21.15.102:0/499105695 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e40035b0 con 0x7f44fc0cd8e0 2023-03-03T13:22:55.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44fbfff700 1 -- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc1593c0 msgr2=0x7f44fc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:22:55.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44fbfff700 1 --2- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc1593c0 0x7f44fc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:55.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44fbfff700 1 -- 172.21.15.102:0/499105695 >> v1:172.21.15.172:6789/0 conn(0x7f44fc0cd8e0 legacy=0x7f44fc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:55.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44fbfff700 1 -- 172.21.15.102:0/499105695 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc06c7f0 con 0x7f44fc153290 2023-03-03T13:22:55.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44fb7fe700 1 --2- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc1593c0 0x7f44fc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:22:55.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.839+0000 7f44fbfff700 1 --2- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc158e80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009840 tx=0x7f44ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5374a964d29ac24 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:22:55.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.840+0000 7f44faffd700 1 -- 172.21.15.102:0/499105695 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec011830 con 0x7f44fc153290 2023-03-03T13:22:55.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.840+0000 7f44faffd700 1 -- 172.21.15.102:0/499105695 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec011990 con 0x7f44fc153290 2023-03-03T13:22:55.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.840+0000 7f44faffd700 1 -- 172.21.15.102:0/499105695 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec011d20 con 0x7f44fc153290 2023-03-03T13:22:55.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 -- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 msgr2=0x7f44fc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:55.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 --2- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc158e80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009840 tx=0x7f44ec009b50 comp rx=0 tx=0).stop 2023-03-03T13:22:55.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 -- 172.21.15.102:0/499105695 shutdown_connections 2023-03-03T13:22:55.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 --2- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc158e80 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:55.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 --2- 172.21.15.102:0/499105695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc1593c0 0x7f44fc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:55.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 -- 172.21.15.102:0/499105695 >> 172.21.15.102:0/499105695 conn(0x7f44fc065fb0 msgr2=0x7f44fc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:22:55.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 -- 172.21.15.102:0/499105695 shutdown_connections 2023-03-03T13:22:55.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.841+0000 7f45029ff700 1 -- 172.21.15.102:0/499105695 wait complete. 2023-03-03T13:22:55.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.843+0000 7f45029ff700 1 Processor -- start 2023-03-03T13:22:55.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.843+0000 7f45029ff700 1 -- start start 2023-03-03T13:22:55.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f45029ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc0cd8e0 0x7f44fc149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:55.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f45029ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc1f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:55.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f45029ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc1593c0 0x7f44fc1f1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:55.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f45029ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc14c8c0 con 0x7f44fc1593c0 2023-03-03T13:22:55.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f45029ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc14bca0 con 0x7f44fc153290 2023-03-03T13:22:55.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f45029ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc149d60 con 0x7f44fc0cd8e0 2023-03-03T13:22:55.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f4500f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc1593c0 0x7f44fc1f1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:55.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f44fb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc1f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:55.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f44fbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc0cd8e0 0x7f44fc149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:55.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f44fb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc1f16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43604/0 (socket says 172.21.15.102:43604) 2023-03-03T13:22:55.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.844+0000 7f44fb7fe700 1 -- 172.21.15.102:0/4083174892 learned_addr learned my addr 172.21.15.102:0/4083174892 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:22:55.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44fb7fe700 1 -- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc0cd8e0 msgr2=0x7f44fc149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:55.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44fb7fe700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc0cd8e0 0x7f44fc149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44fb7fe700 1 -- 172.21.15.102:0/4083174892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc1593c0 msgr2=0x7f44fc1f1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44fb7fe700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc1593c0 0x7f44fc1f1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44fb7fe700 1 -- 172.21.15.102:0/4083174892 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00094a0 con 0x7f44fc153290 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44fb7fe700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc1f16a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f44f000ec10 tx=0x7f44f000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44f8ff9700 1 -- 172.21.15.102:0/4083174892 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f000c850 con 0x7f44fc153290 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.845+0000 7f44f8ff9700 1 -- 172.21.15.102:0/4083174892 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f000c9b0 con 0x7f44fc153290 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.846+0000 7f44f8ff9700 1 -- 172.21.15.102:0/4083174892 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f000cd20 con 0x7f44fc153290 2023-03-03T13:22:55.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.846+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec0094a0 con 0x7f44fc153290 2023-03-03T13:22:55.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.846+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc0ccde0 con 0x7f44fc153290 2023-03-03T13:22:55.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.847+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc15d160 con 0x7f44fc153290 2023-03-03T13:22:55.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.848+0000 7f44f8ff9700 1 -- 172.21.15.102:0/4083174892 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f44f0023070 con 0x7f44fc153290 2023-03-03T13:22:55.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.849+0000 7f44f8ff9700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f44e8068320 0x7f44e806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:55.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.849+0000 7f44f8ff9700 1 -- 172.21.15.102:0/4083174892 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f44f0085ff0 con 0x7f44fc153290 2023-03-03T13:22:55.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.850+0000 7f44fbfff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f44e8068320 0x7f44e806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:55.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.850+0000 7f44fbfff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f44e8068320 0x7f44e806a7d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000c00 tx=0x7f44ec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:22:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:55.853+0000 7f44f8ff9700 1 -- 172.21.15.102:0/4083174892 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f44f0052700 con 0x7f44fc153290 2023-03-03T13:22:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.086+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44fc0cd0c0 con 0x7f44fc153290 2023-03-03T13:22:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.087+0000 7f44f8ff9700 1 -- 172.21.15.102:0/4083174892 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f44f0055d20 con 0x7f44fc153290 2023-03-03T13:22:56.090 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:22:56.090 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:22:56.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f44e8068320 msgr2=0x7f44e806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f44e8068320 0x7f44e806a7d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000c00 tx=0x7f44ec008030 comp rx=0 tx=0).stop 2023-03-03T13:22:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 msgr2=0x7f44fc1f16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc1f16a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f44f000ec10 tx=0x7f44f000ef20 comp rx=0 tx=0).stop 2023-03-03T13:22:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 shutdown_connections 2023-03-03T13:22:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f44e8068320 0x7f44e806a7d0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44fc0cd8e0 0x7f44fc149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:56.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc1593c0 0x7f44fc1f1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:56.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 --2- 172.21.15.102:0/4083174892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44fc153290 0x7f44fc1f16a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:56.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 >> 172.21.15.102:0/4083174892 conn(0x7f44fc065fb0 msgr2=0x7f44fc15ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:22:56.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 shutdown_connections 2023-03-03T13:22:56.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:22:56.090+0000 7f45029ff700 1 -- 172.21.15.102:0/4083174892 wait complete. 2023-03-03T13:22:56.094 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:22:56.119 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T13:23:17.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:17.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:48.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:48.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:15.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:24:15.368 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.367+0000 7f42a3fff700 1 Processor -- start 2023-03-03T13:24:15.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.367+0000 7f42a3fff700 1 -- start start 2023-03-03T13:24:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f42a40ced00 0x7f42a40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42a40c41b0 0x7f42a40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:15.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a3fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a40c3d20 con 0x7f42a40b3970 2023-03-03T13:24:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a3fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a40c4930 con 0x7f42a40c41b0 2023-03-03T13:24:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a40cf610 con 0x7f42a40ced00 2023-03-03T13:24:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a27fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f42a40b3970 0x7f42a4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48054/0 (socket says 172.21.15.102:48054) 2023-03-03T13:24:15.372 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a27fc700 1 -- 172.21.15.102:0/4061373300 learned_addr learned my addr 172.21.15.102:0/4061373300 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:24:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a37fe700 1 --2- 172.21.15.102:0/4061373300 >> v2:172.21.15.102:3300/0 conn(0x7f42a40c41b0 0x7f42a40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.369+0000 7f42a2ffd700 1 --2- 172.21.15.102:0/4061373300 >> v2:172.21.15.172:3300/0 conn(0x7f42a40ced00 0x7f42a40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a37fe700 1 -- 172.21.15.102:0/4061373300 >> v2:172.21.15.172:3300/0 conn(0x7f42a40ced00 msgr2=0x7f42a40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:24:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a37fe700 1 --2- 172.21.15.102:0/4061373300 >> v2:172.21.15.172:3300/0 conn(0x7f42a40ced00 0x7f42a40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a37fe700 1 -- 172.21.15.102:0/4061373300 >> v1:172.21.15.102:6789/0 conn(0x7f42a40b3970 legacy=0x7f42a4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:15.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a37fe700 1 -- 172.21.15.102:0/4061373300 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a415cba0 con 0x7f42a40c41b0 2023-03-03T13:24:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a1ffb700 1 -- 172.21.15.102:0/4061373300 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619296777 0 0) 0x7f42a40c3d20 con 0x7f42a40b3970 2023-03-03T13:24:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a37fe700 1 --2- 172.21.15.102:0/4061373300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a40cf0d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f42a40b1b40 tx=0x7f429c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b889dc80e3b28ac5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:24:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a1ffb700 1 -- 172.21.15.102:0/4061373300 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c00e070 con 0x7f42a40c41b0 2023-03-03T13:24:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a1ffb700 1 -- 172.21.15.102:0/4061373300 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c00bae0 con 0x7f42a40c41b0 2023-03-03T13:24:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.370+0000 7f42a1ffb700 1 -- 172.21.15.102:0/4061373300 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c00be90 con 0x7f42a40c41b0 2023-03-03T13:24:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 -- 172.21.15.102:0/4061373300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 msgr2=0x7f42a40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 --2- 172.21.15.102:0/4061373300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a40cf0d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f42a40b1b40 tx=0x7f429c00b330 comp rx=0 tx=0).stop 2023-03-03T13:24:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 -- 172.21.15.102:0/4061373300 shutdown_connections 2023-03-03T13:24:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 --2- 172.21.15.102:0/4061373300 >> v2:172.21.15.172:3300/0 conn(0x7f42a40ced00 0x7f42a40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 --2- 172.21.15.102:0/4061373300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a40cf0d0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 -- 172.21.15.102:0/4061373300 >> 172.21.15.102:0/4061373300 conn(0x7f42a408b1c0 msgr2=0x7f42a4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:24:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 -- 172.21.15.102:0/4061373300 shutdown_connections 2023-03-03T13:24:15.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.371+0000 7f42a3fff700 1 -- 172.21.15.102:0/4061373300 wait complete. 2023-03-03T13:24:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.373+0000 7f42a3fff700 1 Processor -- start 2023-03-03T13:24:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.373+0000 7f42a3fff700 1 -- start start 2023-03-03T13:24:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.374+0000 7f42a3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a40b3970 0x7f42a41e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.374+0000 7f42a3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a41ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.374+0000 7f42a3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a40ced00 0x7f42a41ee960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.374+0000 7f42a3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a41f25b0 con 0x7f42a40c41b0 2023-03-03T13:24:15.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.374+0000 7f42a3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4151be0 con 0x7f42a40b3970 2023-03-03T13:24:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.374+0000 7f42a3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40ccef0 con 0x7f42a40ced00 2023-03-03T13:24:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a41ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a2ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a40b3970 0x7f42a41e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a40ced00 0x7f42a41ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a41ee420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34642/0 (socket says 172.21.15.102:34642) 2023-03-03T13:24:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 -- 172.21.15.102:0/1164458955 learned_addr learned my addr 172.21.15.102:0/1164458955 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:24:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 -- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a40ced00 msgr2=0x7f42a41ee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a40ced00 0x7f42a41ee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 -- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a40b3970 msgr2=0x7f42a41e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a40b3970 0x7f42a41e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a27fc700 1 -- 172.21.15.102:0/1164458955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c009a70 con 0x7f42a40c41b0 2023-03-03T13:24:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.375+0000 7f42a37fe700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a40ced00 0x7f42a41ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:24:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.376+0000 7f42a27fc700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a41ee420 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f429800c520 tx=0x7f429800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:24:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.376+0000 7f428bfff700 1 -- 172.21.15.102:0/1164458955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4298004c80 con 0x7f42a40c41b0 2023-03-03T13:24:15.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.376+0000 7f428bfff700 1 -- 172.21.15.102:0/1164458955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4298004de0 con 0x7f42a40c41b0 2023-03-03T13:24:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.376+0000 7f428bfff700 1 -- 172.21.15.102:0/1164458955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42980072d0 con 0x7f42a40c41b0 2023-03-03T13:24:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.376+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a406bf20 con 0x7f42a40c41b0 2023-03-03T13:24:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.376+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a415ce80 con 0x7f42a40c41b0 2023-03-03T13:24:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.377+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a406c200 con 0x7f42a40c41b0 2023-03-03T13:24:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.378+0000 7f428bfff700 1 -- 172.21.15.102:0/1164458955 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f429801a020 con 0x7f42a40c41b0 2023-03-03T13:24:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.379+0000 7f428bfff700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42900682f0 0x7f429006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.379+0000 7f428bfff700 1 -- 172.21.15.102:0/1164458955 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4298085850 con 0x7f42a40c41b0 2023-03-03T13:24:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.380+0000 7f42a2ffd700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42900682f0 0x7f429006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.381+0000 7f42a2ffd700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42900682f0 0x7f429006a7a0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4294009be0 tx=0x7f4294009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:24:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.384+0000 7f428bfff700 1 -- 172.21.15.102:0/1164458955 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4298051f60 con 0x7f42a40c41b0 2023-03-03T13:24:15.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.617+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42a404e6e0 con 0x7f42a40c41b0 2023-03-03T13:24:15.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.618+0000 7f428bfff700 1 -- 172.21.15.102:0/1164458955 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4298055580 con 0x7f42a40c41b0 2023-03-03T13:24:15.620 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:24:15.620 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:24:15.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42900682f0 msgr2=0x7f429006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:15.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42900682f0 0x7f429006a7a0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4294009be0 tx=0x7f4294009210 comp rx=0 tx=0).stop 2023-03-03T13:24:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 msgr2=0x7f42a41ee420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a41ee420 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f429800c520 tx=0x7f429800c830 comp rx=0 tx=0).stop 2023-03-03T13:24:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 shutdown_connections 2023-03-03T13:24:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42900682f0 0x7f429006a7a0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42a40ced00 0x7f42a41ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42a40b3970 0x7f42a41e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 --2- 172.21.15.102:0/1164458955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a40c41b0 0x7f42a41ee420 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:15.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 >> 172.21.15.102:0/1164458955 conn(0x7f42a408b1c0 msgr2=0x7f42a4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:24:15.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 shutdown_connections 2023-03-03T13:24:15.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:24:15.621+0000 7f42a3fff700 1 -- 172.21.15.102:0/1164458955 wait complete. 2023-03-03T13:24:15.624 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:24:15.650 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 65 secs before thrashing 2023-03-03T13:24:18.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:18.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:48.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:48.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:18.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:18.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:20.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:25:20.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.908+0000 7fd8749fe700 1 Processor -- start 2023-03-03T13:25:20.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.908+0000 7fd8749fe700 1 -- start start 2023-03-03T13:25:20.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd8749fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8701564b0 0x7fd8700cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:20.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd8749fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700cab50 0x7fd8700cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:20.912 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd8749fe700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8700c9ca0 con 0x7fd8700c9610 2023-03-03T13:25:20.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd8749fe700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8700b4920 con 0x7fd8701564b0 2023-03-03T13:25:20.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd8749fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700b4c00 con 0x7fd8700cab50 2023-03-03T13:25:20.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd86e59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd8700c9610 0x7fd8700ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51132/0 (socket says 172.21.15.102:51132) 2023-03-03T13:25:20.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd86e59c700 1 -- 172.21.15.102:0/4175070939 learned_addr learned my addr 172.21.15.102:0/4175070939 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:25:20.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd86ed9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8701564b0 0x7fd8700cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:20.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.910+0000 7fd86f59e700 1 --2- 172.21.15.102:0/4175070939 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700cab50 0x7fd8700cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:20.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86dd9b700 1 -- 172.21.15.102:0/4175070939 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071499955 0 0) 0x7fd8700c9ca0 con 0x7fd8700c9610 2023-03-03T13:25:20.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86dd9b700 1 -- 172.21.15.102:0/4175070939 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8580035b0 con 0x7fd8700c9610 2023-03-03T13:25:20.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86ed9d700 1 -- 172.21.15.102:0/4175070939 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700cab50 msgr2=0x7fd8700cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:25:20.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86ed9d700 1 --2- 172.21.15.102:0/4175070939 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700cab50 0x7fd8700cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:20.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86ed9d700 1 -- 172.21.15.102:0/4175070939 >> v1:172.21.15.172:6789/0 conn(0x7fd8700c9610 legacy=0x7fd8700ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:20.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86ed9d700 1 -- 172.21.15.102:0/4175070939 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8700b46f0 con 0x7fd8701564b0 2023-03-03T13:25:20.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86ed9d700 1 --2- 172.21.15.102:0/4175070939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cb660 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd860009840 tx=0x7fd860009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79b4c4beaf2b1ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:20.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86dd9b700 1 -- 172.21.15.102:0/4175070939 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860011830 con 0x7fd8701564b0 2023-03-03T13:25:20.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86dd9b700 1 -- 172.21.15.102:0/4175070939 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860011990 con 0x7fd8701564b0 2023-03-03T13:25:20.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.911+0000 7fd86dd9b700 1 -- 172.21.15.102:0/4175070939 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860011d20 con 0x7fd8701564b0 2023-03-03T13:25:20.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 -- 172.21.15.102:0/4175070939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 msgr2=0x7fd8700cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:20.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 --2- 172.21.15.102:0/4175070939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cb660 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd860009840 tx=0x7fd860009b50 comp rx=0 tx=0).stop 2023-03-03T13:25:20.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 -- 172.21.15.102:0/4175070939 shutdown_connections 2023-03-03T13:25:20.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 --2- 172.21.15.102:0/4175070939 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700cab50 0x7fd8700cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:20.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 --2- 172.21.15.102:0/4175070939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cb660 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:20.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 -- 172.21.15.102:0/4175070939 >> 172.21.15.102:0/4175070939 conn(0x7fd870065fb0 msgr2=0x7fd8700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:25:20.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 -- 172.21.15.102:0/4175070939 shutdown_connections 2023-03-03T13:25:20.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.912+0000 7fd8749fe700 1 -- 172.21.15.102:0/4175070939 wait complete. 2023-03-03T13:25:20.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.914+0000 7fd8749fe700 1 Processor -- start 2023-03-03T13:25:20.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.914+0000 7fd8749fe700 1 -- start start 2023-03-03T13:25:20.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.915+0000 7fd8749fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700c9610 0x7fd870157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:20.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.915+0000 7fd8749fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8700cab50 0x7fd8700c2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:20.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.915+0000 7fd8749fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:20.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.915+0000 7fd8749fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd870150b70 con 0x7fd8701564b0 2023-03-03T13:25:20.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.915+0000 7fd8749fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd870158470 con 0x7fd8700cab50 2023-03-03T13:25:20.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.915+0000 7fd8749fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701f15e0 con 0x7fd8700c9610 2023-03-03T13:25:20.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86f59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:20.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8700cab50 0x7fd8700c2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:20.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86f59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39068/0 (socket says 172.21.15.102:39068) 2023-03-03T13:25:20.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86f59e700 1 -- 172.21.15.102:0/666625465 learned_addr learned my addr 172.21.15.102:0/666625465 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:25:20.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86ed9d700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700c9610 0x7fd870157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:20.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86e59c700 1 -- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700c9610 msgr2=0x7fd870157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:20.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86e59c700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700c9610 0x7fd870157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:20.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86e59c700 1 -- 172.21.15.102:0/666625465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 msgr2=0x7fd8700cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86e59c700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86e59c700 1 -- 172.21.15.102:0/666625465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600094a0 con 0x7fd8700cab50 2023-03-03T13:25:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86f59e700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:25:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.916+0000 7fd86e59c700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8700cab50 0x7fd8700c2f40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd86400c400 tx=0x7fd86400c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.917+0000 7fd8577fe700 1 -- 172.21.15.102:0/666625465 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864005d70 con 0x7fd8700cab50 2023-03-03T13:25:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.917+0000 7fd8577fe700 1 -- 172.21.15.102:0/666625465 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86400ce90 con 0x7fd8700cab50 2023-03-03T13:25:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.917+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8700b2210 con 0x7fd8700cab50 2023-03-03T13:25:20.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.917+0000 7fd8577fe700 1 -- 172.21.15.102:0/666625465 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640072f0 con 0x7fd8700cab50 2023-03-03T13:25:20.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.917+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8700b1f30 con 0x7fd8700cab50 2023-03-03T13:25:20.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.918+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd870156880 con 0x7fd8700cab50 2023-03-03T13:25:20.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.919+0000 7fd8577fe700 1 -- 172.21.15.102:0/666625465 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd864019030 con 0x7fd8700cab50 2023-03-03T13:25:20.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.920+0000 7fd8577fe700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c068340 0x7fd85c06a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:20.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.920+0000 7fd8577fe700 1 -- 172.21.15.102:0/666625465 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd864084fc0 con 0x7fd8700cab50 2023-03-03T13:25:20.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.921+0000 7fd86ed9d700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c068340 0x7fd85c06a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:20.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.921+0000 7fd86ed9d700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c068340 0x7fd85c06a7f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fd860000c00 tx=0x7fd860008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:20.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:20.925+0000 7fd8577fe700 1 -- 172.21.15.102:0/666625465 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8640516c0 con 0x7fd8700cab50 2023-03-03T13:25:21.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.155+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8700677f0 con 0x7fd8700cab50 2023-03-03T13:25:21.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.156+0000 7fd8577fe700 1 -- 172.21.15.102:0/666625465 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd864017020 con 0x7fd8700cab50 2023-03-03T13:25:21.158 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:25:21.158 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:25:21.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c068340 msgr2=0x7fd85c06a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:21.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c068340 0x7fd85c06a7f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fd860000c00 tx=0x7fd860008030 comp rx=0 tx=0).stop 2023-03-03T13:25:21.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8700cab50 msgr2=0x7fd8700c2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:21.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8700cab50 0x7fd8700c2f40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd86400c400 tx=0x7fd86400c710 comp rx=0 tx=0).stop 2023-03-03T13:25:21.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 shutdown_connections 2023-03-03T13:25:21.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd85c068340 0x7fd85c06a7f0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:21.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8700c9610 0x7fd870157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:21.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8701564b0 0x7fd8700cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:21.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 --2- 172.21.15.102:0/666625465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8700cab50 0x7fd8700c2f40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:21.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 >> 172.21.15.102:0/666625465 conn(0x7fd870065fb0 msgr2=0x7fd87006b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:25:21.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 shutdown_connections 2023-03-03T13:25:21.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:25:21.158+0000 7fd8749fe700 1 -- 172.21.15.102:0/666625465 wait complete. 2023-03-03T13:25:21.163 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:25:21.188 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-03-03T13:25:48.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:48.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:08.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:26:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.437+0000 7fb6bf59e700 1 Processor -- start 2023-03-03T13:26:08.440 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.437+0000 7fb6bf59e700 1 -- start start 2023-03-03T13:26:08.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bf59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb6c00ced00 0x7fb6c00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:08.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bf59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00b3970 0x7fb6c00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:08.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bf59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c00c3d20 con 0x7fb6c00cf0d0 2023-03-03T13:26:08.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bf59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c00c4930 con 0x7fb6c00ced00 2023-03-03T13:26:08.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bf59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c015cba0 con 0x7fb6c00b3970 2023-03-03T13:26:08.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bdd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00b3970 0x7fb6c00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:08.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6be59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb6c00ced00 0x7fb6c00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:08.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bdd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00b3970 0x7fb6c00c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45110/0 (socket says 172.21.15.102:45110) 2023-03-03T13:26:08.443 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bdd9b700 1 -- 172.21.15.102:0/5399196 learned_addr learned my addr 172.21.15.102:0/5399196 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:26:08.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.439+0000 7fb6bed9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb6c00cf0d0 0x7fb6c0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36414/0 (socket says 172.21.15.102:36414) 2023-03-03T13:26:08.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bd59a700 1 -- 172.21.15.102:0/5399196 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382251952 0 0) 0x7fb6c00c3d20 con 0x7fb6c00cf0d0 2023-03-03T13:26:08.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bd59a700 1 -- 172.21.15.102:0/5399196 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a80035b0 con 0x7fb6c00cf0d0 2023-03-03T13:26:08.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bdd9b700 1 -- 172.21.15.102:0/5399196 >> v2:172.21.15.172:3300/0 conn(0x7fb6c00ced00 msgr2=0x7fb6c00c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:26:08.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bdd9b700 1 --2- 172.21.15.102:0/5399196 >> v2:172.21.15.172:3300/0 conn(0x7fb6c00ced00 0x7fb6c00c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bdd9b700 1 -- 172.21.15.102:0/5399196 >> v1:172.21.15.172:6789/0 conn(0x7fb6c00cf0d0 legacy=0x7fb6c0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:08.444 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bdd9b700 1 -- 172.21.15.102:0/5399196 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c015ce80 con 0x7fb6c00b3970 2023-03-03T13:26:08.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bdd9b700 1 --2- 172.21.15.102:0/5399196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00b3970 0x7fb6c00c41b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098f0 tx=0x7fb6b4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1abfa10ecacf986d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:08.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bd59a700 1 -- 172.21.15.102:0/5399196 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4011830 con 0x7fb6c00b3970 2023-03-03T13:26:08.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bd59a700 1 -- 172.21.15.102:0/5399196 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b4011990 con 0x7fb6c00b3970 2023-03-03T13:26:08.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.440+0000 7fb6bd59a700 1 -- 172.21.15.102:0/5399196 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4011d20 con 0x7fb6c00b3970 2023-03-03T13:26:08.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 -- 172.21.15.102:0/5399196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00b3970 msgr2=0x7fb6c00c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:08.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/5399196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00b3970 0x7fb6c00c41b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098f0 tx=0x7fb6b4009c00 comp rx=0 tx=0).stop 2023-03-03T13:26:08.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 -- 172.21.15.102:0/5399196 shutdown_connections 2023-03-03T13:26:08.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/5399196 >> v2:172.21.15.172:3300/0 conn(0x7fb6c00ced00 0x7fb6c00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/5399196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00b3970 0x7fb6c00c41b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 -- 172.21.15.102:0/5399196 >> 172.21.15.102:0/5399196 conn(0x7fb6c008b1c0 msgr2=0x7fb6c0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:26:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 -- 172.21.15.102:0/5399196 shutdown_connections 2023-03-03T13:26:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.441+0000 7fb6bf59e700 1 -- 172.21.15.102:0/5399196 wait complete. 2023-03-03T13:26:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.443+0000 7fb6bf59e700 1 Processor -- start 2023-03-03T13:26:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.443+0000 7fb6bf59e700 1 -- start start 2023-03-03T13:26:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.444+0000 7fb6bf59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6c00b3970 0x7fb6c01f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.444+0000 7fb6bf59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6c00ced00 0x7fb6c01f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.444+0000 7fb6bf59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00cf0d0 0x7fb6c01f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.444+0000 7fb6bf59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c0155fb0 con 0x7fb6c00b3970 2023-03-03T13:26:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.444+0000 7fb6bf59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c01f0460 con 0x7fb6c00ced00 2023-03-03T13:26:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.444+0000 7fb6bf59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c0153b80 con 0x7fb6c00cf0d0 2023-03-03T13:26:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6c00b3970 0x7fb6c01f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6bed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00cf0d0 0x7fb6c01f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6c00b3970 0x7fb6c01f19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38734/0 (socket says 172.21.15.102:38734) 2023-03-03T13:26:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6bdd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6c00ced00 0x7fb6c01f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 -- 172.21.15.102:0/3553440387 learned_addr learned my addr 172.21.15.102:0/3553440387 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:26:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6bed9d700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00cf0d0 0x7fb6c01f0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45116/0 (socket says 172.21.15.102:45116) 2023-03-03T13:26:08.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 -- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00cf0d0 msgr2=0x7fb6c01f0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:08.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00cf0d0 0x7fb6c01f0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 -- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6c00ced00 msgr2=0x7fb6c01f08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:08.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6c00ced00 0x7fb6c01f08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6be59c700 1 -- 172.21.15.102:0/3553440387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b4009550 con 0x7fb6c00b3970 2023-03-03T13:26:08.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.445+0000 7fb6bdd9b700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6c00ced00 0x7fb6c01f08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:26:08.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.446+0000 7fb6be59c700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6c00b3970 0x7fb6c01f19b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000ea10 tx=0x7fb6b000ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:08.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.446+0000 7fb6aeffd700 1 -- 172.21.15.102:0/3553440387 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0009710 con 0x7fb6c00b3970 2023-03-03T13:26:08.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.446+0000 7fb6aeffd700 1 -- 172.21.15.102:0/3553440387 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b000eeb0 con 0x7fb6c00b3970 2023-03-03T13:26:08.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.446+0000 7fb6aeffd700 1 -- 172.21.15.102:0/3553440387 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000ca50 con 0x7fb6c00b3970 2023-03-03T13:26:08.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.446+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c006c200 con 0x7fb6c00b3970 2023-03-03T13:26:08.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.446+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c006bf20 con 0x7fb6c00b3970 2023-03-03T13:26:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.447+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c00cf540 con 0x7fb6c00b3970 2023-03-03T13:26:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.448+0000 7fb6aeffd700 1 -- 172.21.15.102:0/3553440387 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb6b0019030 con 0x7fb6c00b3970 2023-03-03T13:26:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.450+0000 7fb6aeffd700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6a00682f0 0x7fb6a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.450+0000 7fb6aeffd700 1 -- 172.21.15.102:0/3553440387 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb6b0085d50 con 0x7fb6c00b3970 2023-03-03T13:26:08.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.450+0000 7fb6bdd9b700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6a00682f0 0x7fb6a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:08.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.451+0000 7fb6bdd9b700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6a00682f0 0x7fb6a006a7a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098f0 tx=0x7fb6b4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:08.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.454+0000 7fb6aeffd700 1 -- 172.21.15.102:0/3553440387 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb6b0052370 con 0x7fb6c00b3970 2023-03-03T13:26:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.689+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6c01f1370 con 0x7fb6c00b3970 2023-03-03T13:26:08.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.690+0000 7fb6aeffd700 1 -- 172.21.15.102:0/3553440387 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb6b0021090 con 0x7fb6c00b3970 2023-03-03T13:26:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:26:08.693 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:26:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6a00682f0 msgr2=0x7fb6a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6a00682f0 0x7fb6a006a7a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098f0 tx=0x7fb6b4008030 comp rx=0 tx=0).stop 2023-03-03T13:26:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6c00b3970 msgr2=0x7fb6c01f19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6c00b3970 0x7fb6c01f19b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000ea10 tx=0x7fb6b000ed20 comp rx=0 tx=0).stop 2023-03-03T13:26:08.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 shutdown_connections 2023-03-03T13:26:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6a00682f0 0x7fb6a006a7a0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6c00cf0d0 0x7fb6c01f0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6c00b3970 0x7fb6c01f19b0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 --2- 172.21.15.102:0/3553440387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6c00ced00 0x7fb6c01f08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.692+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 >> 172.21.15.102:0/3553440387 conn(0x7fb6c008b1c0 msgr2=0x7fb6c00c40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:26:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.693+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 shutdown_connections 2023-03-03T13:26:08.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:08.693+0000 7fb6bf59e700 1 -- 172.21.15.102:0/3553440387 wait complete. 2023-03-03T13:26:08.697 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:26:08.722 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T13:26:19.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:19.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:48.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:26:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.979+0000 7f0aa359e700 1 Processor -- start 2023-03-03T13:26:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.979+0000 7f0aa359e700 1 -- start start 2023-03-03T13:26:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40b3970 0x7f0aa40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0aa40c41b0 0x7f0aa40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa359e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa40c3d20 con 0x7f0aa40ced00 2023-03-03T13:26:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa40c4930 con 0x7f0aa40c41b0 2023-03-03T13:26:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40cf610 con 0x7f0aa40b3970 2023-03-03T13:26:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa1d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40b3970 0x7f0aa40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa1d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40b3970 0x7f0aa40c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:49056/0 (socket says 172.21.15.102:49056) 2023-03-03T13:26:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa1d9b700 1 -- 172.21.15.102:0/2551706664 learned_addr learned my addr 172.21.15.102:0/2551706664 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:26:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa2d9d700 1 --2- 172.21.15.102:0/2551706664 >> v2:172.21.15.172:3300/0 conn(0x7f0aa40c41b0 0x7f0aa40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.981+0000 7f0aa259c700 1 --1- 172.21.15.102:0/2551706664 >> v1:172.21.15.172:6789/0 conn(0x7f0aa40ced00 0x7f0aa4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:48714/0 (socket says 172.21.15.102:48714) 2023-03-03T13:26:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.982+0000 7f0aa1d9b700 1 -- 172.21.15.102:0/2551706664 >> v2:172.21.15.172:3300/0 conn(0x7f0aa40c41b0 msgr2=0x7f0aa40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:26:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.982+0000 7f0aa1d9b700 1 --2- 172.21.15.102:0/2551706664 >> v2:172.21.15.172:3300/0 conn(0x7f0aa40c41b0 0x7f0aa40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.982+0000 7f0aa1d9b700 1 -- 172.21.15.102:0/2551706664 >> v1:172.21.15.172:6789/0 conn(0x7f0aa40ced00 legacy=0x7f0aa4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.982+0000 7f0aa1d9b700 1 -- 172.21.15.102:0/2551706664 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa415cba0 con 0x7f0aa40b3970 2023-03-03T13:26:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.982+0000 7f0aa1d9b700 1 --2- 172.21.15.102:0/2551706664 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40b3970 0x7f0aa40c5a90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0a980098f0 tx=0x7f0a98009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9906b7bcabf93c25 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.982+0000 7f0aa159a700 1 -- 172.21.15.102:0/2551706664 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98011830 con 0x7f0aa40b3970 2023-03-03T13:26:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.982+0000 7f0aa159a700 1 -- 172.21.15.102:0/2551706664 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a98011990 con 0x7f0aa40b3970 2023-03-03T13:26:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa159a700 1 -- 172.21.15.102:0/2551706664 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98011d20 con 0x7f0aa40b3970 2023-03-03T13:26:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 -- 172.21.15.102:0/2551706664 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40b3970 msgr2=0x7f0aa40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 --2- 172.21.15.102:0/2551706664 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40b3970 0x7f0aa40c5a90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0a980098f0 tx=0x7f0a98009c00 comp rx=0 tx=0).stop 2023-03-03T13:26:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 -- 172.21.15.102:0/2551706664 shutdown_connections 2023-03-03T13:26:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 --2- 172.21.15.102:0/2551706664 >> v2:172.21.15.172:3300/0 conn(0x7f0aa40c41b0 0x7f0aa40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 --2- 172.21.15.102:0/2551706664 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40b3970 0x7f0aa40c5a90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 -- 172.21.15.102:0/2551706664 >> 172.21.15.102:0/2551706664 conn(0x7f0aa408b1c0 msgr2=0x7f0aa4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:26:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 -- 172.21.15.102:0/2551706664 shutdown_connections 2023-03-03T13:26:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.983+0000 7f0aa359e700 1 -- 172.21.15.102:0/2551706664 wait complete. 2023-03-03T13:26:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.985+0000 7f0aa359e700 1 Processor -- start 2023-03-03T13:26:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.986+0000 7f0aa359e700 1 -- start start 2023-03-03T13:26:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa40b3970 0x7f0aa414f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40c41b0 0x7f0aa41f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0aa40ced00 0x7f0aa4148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa41560b0 con 0x7f0aa40b3970 2023-03-03T13:26:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa415c110 con 0x7f0aa40ced00 2023-03-03T13:26:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa414d970 con 0x7f0aa40c41b0 2023-03-03T13:26:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa40b3970 0x7f0aa414f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa40b3970 0x7f0aa414f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46046/0 (socket says 172.21.15.102:46046) 2023-03-03T13:26:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa259c700 1 -- 172.21.15.102:0/3623395159 learned_addr learned my addr 172.21.15.102:0/3623395159 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:26:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa1d9b700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40c41b0 0x7f0aa41f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa2d9d700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0aa40ced00 0x7f0aa4148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa2d9d700 1 -- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40c41b0 msgr2=0x7f0aa41f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa2d9d700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40c41b0 0x7f0aa41f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa2d9d700 1 -- 172.21.15.102:0/3623395159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa40b3970 msgr2=0x7f0aa414f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.987+0000 7f0aa2d9d700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa40b3970 0x7f0aa414f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0aa2d9d700 1 -- 172.21.15.102:0/3623395159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98009550 con 0x7f0aa40ced00 2023-03-03T13:26:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0aa259c700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa40b3970 0x7f0aa414f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:26:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0aa2d9d700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0aa40ced00 0x7f0aa4148960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00ef90 tx=0x7f0a9c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0a92ffd700 1 -- 172.21.15.102:0/3623395159 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c00cc90 con 0x7f0aa40ced00 2023-03-03T13:26:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0a92ffd700 1 -- 172.21.15.102:0/3623395159 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c00f040 con 0x7f0aa40ced00 2023-03-03T13:26:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0a92ffd700 1 -- 172.21.15.102:0/3623395159 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0146c0 con 0x7f0aa40ced00 2023-03-03T13:26:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa406bf20 con 0x7f0aa40ced00 2023-03-03T13:26:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.988+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa415ce80 con 0x7f0aa40ced00 2023-03-03T13:26:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.990+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa406c200 con 0x7f0aa40ced00 2023-03-03T13:26:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.991+0000 7f0a92ffd700 1 -- 172.21.15.102:0/3623395159 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0a9c024070 con 0x7f0aa40ced00 2023-03-03T13:26:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.992+0000 7f0a92ffd700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0a84068320 0x7f0a8406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.992+0000 7f0a92ffd700 1 -- 172.21.15.102:0/3623395159 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0a9c086840 con 0x7f0aa40ced00 2023-03-03T13:26:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.992+0000 7f0aa259c700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0a84068320 0x7f0a8406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.993+0000 7f0aa259c700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0a84068320 0x7f0a8406a7d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f0a94000f60 tx=0x7f0a94007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:48.996+0000 7f0a92ffd700 1 -- 172.21.15.102:0/3623395159 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a9c052f50 con 0x7f0aa40ced00 2023-03-03T13:26:49.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.228+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aa404e6e0 con 0x7f0aa40ced00 2023-03-03T13:26:49.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.229+0000 7f0a92ffd700 1 -- 172.21.15.102:0/3623395159 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0a9c009e60 con 0x7f0aa40ced00 2023-03-03T13:26:49.232 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:26:49.232 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:26:49.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0a84068320 msgr2=0x7f0a8406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0a84068320 0x7f0a8406a7d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f0a94000f60 tx=0x7f0a94007f90 comp rx=0 tx=0).stop 2023-03-03T13:26:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0aa40ced00 msgr2=0x7f0aa4148960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0aa40ced00 0x7f0aa4148960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00ef90 tx=0x7f0a9c00c3a0 comp rx=0 tx=0).stop 2023-03-03T13:26:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 shutdown_connections 2023-03-03T13:26:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0aa40ced00 0x7f0aa4148960 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa40b3970 0x7f0aa414f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0a84068320 0x7f0a8406a7d0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 --2- 172.21.15.102:0/3623395159 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0aa40c41b0 0x7f0aa41f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 >> 172.21.15.102:0/3623395159 conn(0x7f0aa408b1c0 msgr2=0x7f0aa4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:26:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 shutdown_connections 2023-03-03T13:26:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:26:49.232+0000 7f0aa359e700 1 -- 172.21.15.102:0/3623395159 wait complete. 2023-03-03T13:26:49.237 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:26:49.261 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2023-03-03T13:26:49.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:49.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:16.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:27:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.511+0000 7fe2a359e700 1 Processor -- start 2023-03-03T13:27:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.511+0000 7fe2a359e700 1 -- start start 2023-03-03T13:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2a40ced00 0x7fe2a40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe2a40c41b0 0x7fe2a40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:16.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40c3d20 con 0x7fe2a40b3970 2023-03-03T13:27:16.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a359e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a40c4930 con 0x7fe2a40ced00 2023-03-03T13:27:16.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a40cf610 con 0x7fe2a40c41b0 2023-03-03T13:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a259c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2a40ced00 0x7fe2a40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a259c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe2a40ced00 0x7fe2a40c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56668/0 (socket says 172.21.15.102:56668) 2023-03-03T13:27:16.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a259c700 1 -- 172.21.15.102:0/1240681871 learned_addr learned my addr 172.21.15.102:0/1240681871 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.513+0000 7fe2a2d9d700 1 --2- 172.21.15.102:0/1240681871 >> v2:172.21.15.172:3300/0 conn(0x7fe2a40c41b0 0x7fe2a40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.514+0000 7fe2a259c700 1 -- 172.21.15.102:0/1240681871 >> v2:172.21.15.172:3300/0 conn(0x7fe2a40c41b0 msgr2=0x7fe2a40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.514+0000 7fe2a259c700 1 --2- 172.21.15.102:0/1240681871 >> v2:172.21.15.172:3300/0 conn(0x7fe2a40c41b0 0x7fe2a40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.514+0000 7fe2a259c700 1 -- 172.21.15.102:0/1240681871 >> v1:172.21.15.102:6789/0 conn(0x7fe2a40b3970 legacy=0x7fe2a4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.514+0000 7fe2a259c700 1 -- 172.21.15.102:0/1240681871 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a415cba0 con 0x7fe2a40ced00 2023-03-03T13:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.514+0000 7fe2a259c700 1 --2- 172.21.15.102:0/1240681871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40ced00 0x7fe2a40c5a90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40ca080 tx=0x7fe294009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4f749bb5a6fea5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:16.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.514+0000 7fe2a159a700 1 -- 172.21.15.102:0/1240681871 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294011830 con 0x7fe2a40ced00 2023-03-03T13:27:16.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a159a700 1 -- 172.21.15.102:0/1240681871 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe294011990 con 0x7fe2a40ced00 2023-03-03T13:27:16.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a159a700 1 -- 172.21.15.102:0/1240681871 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294011d20 con 0x7fe2a40ced00 2023-03-03T13:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 -- 172.21.15.102:0/1240681871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40ced00 msgr2=0x7fe2a40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 --2- 172.21.15.102:0/1240681871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40ced00 0x7fe2a40c5a90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40ca080 tx=0x7fe294009c00 comp rx=0 tx=0).stop 2023-03-03T13:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 -- 172.21.15.102:0/1240681871 shutdown_connections 2023-03-03T13:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 --2- 172.21.15.102:0/1240681871 >> v2:172.21.15.172:3300/0 conn(0x7fe2a40c41b0 0x7fe2a40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 --2- 172.21.15.102:0/1240681871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40ced00 0x7fe2a40c5a90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 -- 172.21.15.102:0/1240681871 >> 172.21.15.102:0/1240681871 conn(0x7fe2a408b1c0 msgr2=0x7fe2a4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:16.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 -- 172.21.15.102:0/1240681871 shutdown_connections 2023-03-03T13:27:16.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.515+0000 7fe2a359e700 1 -- 172.21.15.102:0/1240681871 wait complete. 2023-03-03T13:27:16.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.517+0000 7fe2a359e700 1 Processor -- start 2023-03-03T13:27:16.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.517+0000 7fe2a359e700 1 -- start start 2023-03-03T13:27:16.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.518+0000 7fe2a359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2a40b3970 0x7fe2a40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:16.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.518+0000 7fe2a359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40c41b0 0x7fe2a41ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:16.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.518+0000 7fe2a359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2a40ced00 0x7fe2a41ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:16.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.518+0000 7fe2a359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a415a020 con 0x7fe2a40c41b0 2023-03-03T13:27:16.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.518+0000 7fe2a359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a41f22c0 con 0x7fe2a40b3970 2023-03-03T13:27:16.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.518+0000 7fe2a359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a40cf1b0 con 0x7fe2a40ced00 2023-03-03T13:27:16.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40c41b0 0x7fe2a41ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40c41b0 0x7fe2a41ee1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56682/0 (socket says 172.21.15.102:56682) 2023-03-03T13:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 -- 172.21.15.102:0/2411380036 learned_addr learned my addr 172.21.15.102:0/2411380036 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a259c700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2a40b3970 0x7fe2a40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a2d9d700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2a40ced00 0x7fe2a41ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 -- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2a40ced00 msgr2=0x7fe2a41ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2a40ced00 0x7fe2a41ee6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 -- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2a40b3970 msgr2=0x7fe2a40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:16.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2a40b3970 0x7fe2a40b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 -- 172.21.15.102:0/2411380036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe294009550 con 0x7fe2a40c41b0 2023-03-03T13:27:16.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a2d9d700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2a40ced00 0x7fe2a41ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:27:16.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a259c700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2a40b3970 0x7fe2a40b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:27:16.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.519+0000 7fe2a1d9b700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40c41b0 0x7fe2a41ee1a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe29800c420 tx=0x7fe29800c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:16.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.520+0000 7fe292ffd700 1 -- 172.21.15.102:0/2411380036 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2980052d0 con 0x7fe2a40c41b0 2023-03-03T13:27:16.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.520+0000 7fe292ffd700 1 -- 172.21.15.102:0/2411380036 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe298005430 con 0x7fe2a40c41b0 2023-03-03T13:27:16.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.520+0000 7fe292ffd700 1 -- 172.21.15.102:0/2411380036 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2980038c0 con 0x7fe2a40c41b0 2023-03-03T13:27:16.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.520+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a406bf20 con 0x7fe2a40c41b0 2023-03-03T13:27:16.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.520+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a415ce80 con 0x7fe2a40c41b0 2023-03-03T13:27:16.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.521+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a406c200 con 0x7fe2a40c41b0 2023-03-03T13:27:16.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.522+0000 7fe292ffd700 1 -- 172.21.15.102:0/2411380036 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe298019030 con 0x7fe2a40c41b0 2023-03-03T13:27:16.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.523+0000 7fe292ffd700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2840682f0 0x7fe28406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:16.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.523+0000 7fe292ffd700 1 -- 172.21.15.102:0/2411380036 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe298084fe0 con 0x7fe2a40c41b0 2023-03-03T13:27:16.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.524+0000 7fe2a259c700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2840682f0 0x7fe28406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:16.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.525+0000 7fe2a259c700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2840682f0 0x7fe28406a7a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe294009870 tx=0x7fe294008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:16.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.528+0000 7fe292ffd700 1 -- 172.21.15.102:0/2411380036 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2980516f0 con 0x7fe2a40c41b0 2023-03-03T13:27:16.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.759+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2a41f2430 con 0x7fe2a40c41b0 2023-03-03T13:27:16.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.760+0000 7fe292ffd700 1 -- 172.21.15.102:0/2411380036 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe298017090 con 0x7fe2a40c41b0 2023-03-03T13:27:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:27:16.762 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.762+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2840682f0 msgr2=0x7fe28406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.762+0000 7fe2a359e700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2840682f0 0x7fe28406a7a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe294009870 tx=0x7fe294008030 comp rx=0 tx=0).stop 2023-03-03T13:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40c41b0 msgr2=0x7fe2a41ee1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40c41b0 0x7fe2a41ee1a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe29800c420 tx=0x7fe29800c730 comp rx=0 tx=0).stop 2023-03-03T13:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 shutdown_connections 2023-03-03T13:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe2840682f0 0x7fe28406a7a0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe2a40ced00 0x7fe2a41ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2a40b3970 0x7fe2a40b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 --2- 172.21.15.102:0/2411380036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2a40c41b0 0x7fe2a41ee1a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:16.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 >> 172.21.15.102:0/2411380036 conn(0x7fe2a408b1c0 msgr2=0x7fe2a40669c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:16.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 shutdown_connections 2023-03-03T13:27:16.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:27:16.763+0000 7fe2a359e700 1 -- 172.21.15.102:0/2411380036 wait complete. 2023-03-03T13:27:16.766 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:27:16.791 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2023-03-03T13:27:19.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:19.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:49.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:49.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:05.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:28:06.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.042+0000 7f30cf5ff700 1 Processor -- start 2023-03-03T13:28:06.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.042+0000 7f30cf5ff700 1 -- start start 2023-03-03T13:28:06.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.043+0000 7f30cf5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f30c80cd8e0 0x7f30c80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:06.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.043+0000 7f30cf5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cd390 0x7f30c80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30cf5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c8153660 con 0x7f30c8153290 2023-03-03T13:28:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30cf5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c806c360 con 0x7f30c80cd8e0 2023-03-03T13:28:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30cf5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c815cea0 con 0x7f30c80cd390 2023-03-03T13:28:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30ccb9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f30c80cd8e0 0x7f30c80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30cd39b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f30c8153290 0x7f30c8158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38180/0 (socket says 172.21.15.102:38180) 2023-03-03T13:28:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30cdb9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cd390 0x7f30c80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30cd39b700 1 -- 172.21.15.102:0/2986573696 learned_addr learned my addr 172.21.15.102:0/2986573696 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:28:06.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30ccb9a700 1 -- 172.21.15.102:0/2986573696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cd390 msgr2=0x7f30c80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:06.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30ccb9a700 1 --2- 172.21.15.102:0/2986573696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cd390 0x7f30c80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30ccb9a700 1 -- 172.21.15.102:0/2986573696 >> v1:172.21.15.172:6789/0 conn(0x7f30c8153290 legacy=0x7f30c8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:06.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30bffff700 1 -- 172.21.15.102:0/2986573696 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133524913 0 0) 0x7f30c8153660 con 0x7f30c8153290 2023-03-03T13:28:06.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.044+0000 7f30ccb9a700 1 -- 172.21.15.102:0/2986573696 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c806c7f0 con 0x7f30c80cd8e0 2023-03-03T13:28:06.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.045+0000 7f30ccb9a700 1 --2- 172.21.15.102:0/2986573696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd8e0 0x7f30c80cde20 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009840 tx=0x7f30b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cca7768ada20102 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:06.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.045+0000 7f30bffff700 1 -- 172.21.15.102:0/2986573696 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8011830 con 0x7f30c80cd8e0 2023-03-03T13:28:06.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.045+0000 7f30bffff700 1 -- 172.21.15.102:0/2986573696 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b8011990 con 0x7f30c80cd8e0 2023-03-03T13:28:06.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.045+0000 7f30bffff700 1 -- 172.21.15.102:0/2986573696 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8011d20 con 0x7f30c80cd8e0 2023-03-03T13:28:06.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.045+0000 7f30cf5ff700 1 -- 172.21.15.102:0/2986573696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd8e0 msgr2=0x7f30c80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:06.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.045+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/2986573696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd8e0 0x7f30c80cde20 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009840 tx=0x7f30b8009b50 comp rx=0 tx=0).stop 2023-03-03T13:28:06.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.046+0000 7f30cf5ff700 1 -- 172.21.15.102:0/2986573696 shutdown_connections 2023-03-03T13:28:06.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.046+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/2986573696 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c80cd390 0x7f30c80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.046+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/2986573696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd8e0 0x7f30c80cde20 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.046+0000 7f30cf5ff700 1 -- 172.21.15.102:0/2986573696 >> 172.21.15.102:0/2986573696 conn(0x7f30c8065fb0 msgr2=0x7f30c80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:06.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.046+0000 7f30cf5ff700 1 -- 172.21.15.102:0/2986573696 shutdown_connections 2023-03-03T13:28:06.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.046+0000 7f30cf5ff700 1 -- 172.21.15.102:0/2986573696 wait complete. 2023-03-03T13:28:06.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.048+0000 7f30cf5ff700 1 Processor -- start 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.048+0000 7f30cf5ff700 1 -- start start 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cf5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd390 0x7f30c80b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cf5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80cd8e0 0x7f30c815ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cf5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c8153290 0x7f30c80c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cf5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81e3ed0 con 0x7f30c80cd390 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cf5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81ef270 con 0x7f30c80cd8e0 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cf5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81ef550 con 0x7f30c8153290 2023-03-03T13:28:06.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cd39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd390 0x7f30c80b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:06.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cd39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd390 0x7f30c80b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56666/0 (socket says 172.21.15.102:56666) 2023-03-03T13:28:06.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cd39b700 1 -- 172.21.15.102:0/1151167121 learned_addr learned my addr 172.21.15.102:0/1151167121 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:28:06.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30ccb9a700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80cd8e0 0x7f30c815ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:06.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.049+0000 7f30cdb9c700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c8153290 0x7f30c80c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:06.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30cd39b700 1 -- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c8153290 msgr2=0x7f30c80c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:06.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30cd39b700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c8153290 0x7f30c80c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30cd39b700 1 -- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80cd8e0 msgr2=0x7f30c815ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:06.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30cd39b700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80cd8e0 0x7f30c815ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30cd39b700 1 -- 172.21.15.102:0/1151167121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b80094a0 con 0x7f30c80cd390 2023-03-03T13:28:06.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30cd39b700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd390 0x7f30c80b1e80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f30c40011e0 tx=0x7f30c400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:06.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30bdffb700 1 -- 172.21.15.102:0/1151167121 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4004c80 con 0x7f30c80cd390 2023-03-03T13:28:06.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30bdffb700 1 -- 172.21.15.102:0/1151167121 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4004e00 con 0x7f30c80cd390 2023-03-03T13:28:06.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c80ccde0 con 0x7f30c80cd390 2023-03-03T13:28:06.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.050+0000 7f30bdffb700 1 -- 172.21.15.102:0/1151167121 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4007310 con 0x7f30c80cd390 2023-03-03T13:28:06.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.051+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c815d160 con 0x7f30c80cd390 2023-03-03T13:28:06.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.052+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c80cd0c0 con 0x7f30c80cd390 2023-03-03T13:28:06.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.053+0000 7f30bdffb700 1 -- 172.21.15.102:0/1151167121 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30c401a020 con 0x7f30c80cd390 2023-03-03T13:28:06.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.054+0000 7f30bdffb700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac068320 0x7f30ac06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:06.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.054+0000 7f30bdffb700 1 -- 172.21.15.102:0/1151167121 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f30c4085930 con 0x7f30c80cd390 2023-03-03T13:28:06.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.054+0000 7f30ccb9a700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac068320 0x7f30ac06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:06.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.055+0000 7f30ccb9a700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac068320 0x7f30ac06a7d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f30b80090b0 tx=0x7f30b801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:06.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.058+0000 7f30bdffb700 1 -- 172.21.15.102:0/1151167121 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30c40520f0 con 0x7f30c80cd390 2023-03-03T13:28:06.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.291+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c804e6e0 con 0x7f30c80cd390 2023-03-03T13:28:06.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.292+0000 7f30bdffb700 1 -- 172.21.15.102:0/1151167121 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f30c4055710 con 0x7f30c80cd390 2023-03-03T13:28:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:28:06.294 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:28:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac068320 msgr2=0x7f30ac06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac068320 0x7f30ac06a7d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f30b80090b0 tx=0x7f30b801c040 comp rx=0 tx=0).stop 2023-03-03T13:28:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd390 msgr2=0x7f30c80b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd390 0x7f30c80b1e80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f30c40011e0 tx=0x7f30c400ca30 comp rx=0 tx=0).stop 2023-03-03T13:28:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 shutdown_connections 2023-03-03T13:28:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ac068320 0x7f30ac06a7d0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30c8153290 0x7f30c80c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30c80cd390 0x7f30c80b1e80 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 --2- 172.21.15.102:0/1151167121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30c80cd8e0 0x7f30c815ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:06.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 >> 172.21.15.102:0/1151167121 conn(0x7f30c8065fb0 msgr2=0x7f30c8158940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:06.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 shutdown_connections 2023-03-03T13:28:06.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:28:06.295+0000 7f30cf5ff700 1 -- 172.21.15.102:0/1151167121 wait complete. 2023-03-03T13:28:06.298 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:28:06.324 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 119 secs before thrashing 2023-03-03T13:28:19.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:19.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:49.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:49.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:19.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:19.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:50.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:50.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T13:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:30:05.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:30:05.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.577+0000 7f0ce09ff700 1 Processor -- start 2023-03-03T13:30:05.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.577+0000 7f0ce09ff700 1 -- start start 2023-03-03T13:30:05.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.578+0000 7f0ce09ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0cdc153290 0x7f0cdc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:05.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.578+0000 7f0ce09ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0cdc0cd8e0 0x7f0cdc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:05.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.578+0000 7f0ce09ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cdc153660 con 0x7f0cdc1593c0 2023-03-03T13:30:05.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.578+0000 7f0ce09ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc06c360 con 0x7f0cdc153290 2023-03-03T13:30:05.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.578+0000 7f0ce09ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc15cea0 con 0x7f0cdc0cd8e0 2023-03-03T13:30:05.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.578+0000 7f0cda59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0cdc153290 0x7f0cdc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:05.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cd9d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0cdc1593c0 0x7f0cdc0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33984/0 (socket says 172.21.15.102:33984) 2023-03-03T13:30:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cd9d9b700 1 -- 172.21.15.102:0/4282939433 learned_addr learned my addr 172.21.15.102:0/4282939433 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:30:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cdad9d700 1 --2- 172.21.15.102:0/4282939433 >> v2:172.21.15.172:3300/0 conn(0x7f0cdc0cd8e0 0x7f0cdc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cda59c700 1 -- 172.21.15.102:0/4282939433 >> v2:172.21.15.172:3300/0 conn(0x7f0cdc0cd8e0 msgr2=0x7f0cdc0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:30:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cda59c700 1 --2- 172.21.15.102:0/4282939433 >> v2:172.21.15.172:3300/0 conn(0x7f0cdc0cd8e0 0x7f0cdc0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cda59c700 1 -- 172.21.15.102:0/4282939433 >> v1:172.21.15.102:6789/0 conn(0x7f0cdc1593c0 legacy=0x7f0cdc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cda59c700 1 -- 172.21.15.102:0/4282939433 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc06c7f0 con 0x7f0cdc153290 2023-03-03T13:30:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cd959a700 1 -- 172.21.15.102:0/4282939433 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781755303 0 0) 0x7f0cdc153660 con 0x7f0cdc1593c0 2023-03-03T13:30:05.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.579+0000 7f0cda59c700 1 --2- 172.21.15.102:0/4282939433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc153290 0x7f0cdc158e80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0c9390 tx=0x7f0ccc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bc9c7dca9a21dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:05.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0cd959a700 1 -- 172.21.15.102:0/4282939433 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc011830 con 0x7f0cdc153290 2023-03-03T13:30:05.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0cd959a700 1 -- 172.21.15.102:0/4282939433 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc011990 con 0x7f0cdc153290 2023-03-03T13:30:05.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0cd959a700 1 -- 172.21.15.102:0/4282939433 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc011d20 con 0x7f0cdc153290 2023-03-03T13:30:05.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0ce09ff700 1 -- 172.21.15.102:0/4282939433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc153290 msgr2=0x7f0cdc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:05.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/4282939433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc153290 0x7f0cdc158e80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0c9390 tx=0x7f0ccc009b20 comp rx=0 tx=0).stop 2023-03-03T13:30:05.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0ce09ff700 1 -- 172.21.15.102:0/4282939433 shutdown_connections 2023-03-03T13:30:05.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/4282939433 >> v2:172.21.15.172:3300/0 conn(0x7f0cdc0cd8e0 0x7f0cdc0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/4282939433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc153290 0x7f0cdc158e80 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.580+0000 7f0ce09ff700 1 -- 172.21.15.102:0/4282939433 >> 172.21.15.102:0/4282939433 conn(0x7f0cdc065fb0 msgr2=0x7f0cdc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:30:05.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.581+0000 7f0ce09ff700 1 -- 172.21.15.102:0/4282939433 shutdown_connections 2023-03-03T13:30:05.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.581+0000 7f0ce09ff700 1 -- 172.21.15.102:0/4282939433 wait complete. 2023-03-03T13:30:05.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.583+0000 7f0ce09ff700 1 Processor -- start 2023-03-03T13:30:05.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.583+0000 7f0ce09ff700 1 -- start start 2023-03-03T13:30:05.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0ce09ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cdc0cd8e0 0x7f0cdc0b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:05.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0ce09ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cdc153290 0x7f0cdc0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:05.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0ce09ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1593c0 0x7f0cdc14dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0ce09ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1eef20 con 0x7f0cdc1593c0 2023-03-03T13:30:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0ce09ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1ee650 con 0x7f0cdc153290 2023-03-03T13:30:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0ce09ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc156130 con 0x7f0cdc0cd8e0 2023-03-03T13:30:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0cd9d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cdc153290 0x7f0cdc0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0cda59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cdc0cd8e0 0x7f0cdc0b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0cd9d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cdc153290 0x7f0cdc0c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37304/0 (socket says 172.21.15.102:37304) 2023-03-03T13:30:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0cd9d9b700 1 -- 172.21.15.102:0/2033585324 learned_addr learned my addr 172.21.15.102:0/2033585324 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:30:05.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0cda59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cdc0cd8e0 0x7f0cdc0b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54746/0 (socket says 172.21.15.102:54746) 2023-03-03T13:30:05.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.584+0000 7f0cdad9d700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1593c0 0x7f0cdc14dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:05.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cd9d9b700 1 -- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cdc0cd8e0 msgr2=0x7f0cdc0b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:05.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cd9d9b700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cdc0cd8e0 0x7f0cdc0b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cd9d9b700 1 -- 172.21.15.102:0/2033585324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1593c0 msgr2=0x7f0cdc14dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:05.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cd9d9b700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1593c0 0x7f0cdc14dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cd9d9b700 1 -- 172.21.15.102:0/2033585324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0094a0 con 0x7f0cdc153290 2023-03-03T13:30:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cdad9d700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1593c0 0x7f0cdc14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:30:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cda59c700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cdc0cd8e0 0x7f0cdc0b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:30:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0cd9d9b700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cdc153290 0x7f0cdc0c63b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00011e0 tx=0x7f0cd000ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0ccaffd700 1 -- 172.21.15.102:0/2033585324 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0004c80 con 0x7f0cdc153290 2023-03-03T13:30:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0ccaffd700 1 -- 172.21.15.102:0/2033585324 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd0004e00 con 0x7f0cdc153290 2023-03-03T13:30:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0ccaffd700 1 -- 172.21.15.102:0/2033585324 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0007310 con 0x7f0cdc153290 2023-03-03T13:30:05.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc0ccde0 con 0x7f0cdc153290 2023-03-03T13:30:05.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.585+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc15d160 con 0x7f0cdc153290 2023-03-03T13:30:05.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.587+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cdc0cd0c0 con 0x7f0cdc153290 2023-03-03T13:30:05.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.588+0000 7f0ccaffd700 1 -- 172.21.15.102:0/2033585324 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0cd001a020 con 0x7f0cdc153290 2023-03-03T13:30:05.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.589+0000 7f0ccaffd700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cbc068320 0x7f0cbc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:05.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.589+0000 7f0ccaffd700 1 -- 172.21.15.102:0/2033585324 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0cd0085840 con 0x7f0cdc153290 2023-03-03T13:30:05.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.589+0000 7f0cda59c700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cbc068320 0x7f0cbc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:05.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.590+0000 7f0cda59c700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cbc068320 0x7f0cbc06a7d0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0090b0 tx=0x7f0ccc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:05.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.593+0000 7f0ccaffd700 1 -- 172.21.15.102:0/2033585324 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0cd0052000 con 0x7f0cdc153290 2023-03-03T13:30:05.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.825+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cdc157370 con 0x7f0cdc153290 2023-03-03T13:30:05.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.826+0000 7f0ccaffd700 1 -- 172.21.15.102:0/2033585324 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0cd0055620 con 0x7f0cdc153290 2023-03-03T13:30:05.828 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:30:05.828 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:30:05.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.828+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cbc068320 msgr2=0x7f0cbc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:05.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.828+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cbc068320 0x7f0cbc06a7d0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0090b0 tx=0x7f0ccc008030 comp rx=0 tx=0).stop 2023-03-03T13:30:05.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.828+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cdc153290 msgr2=0x7f0cdc0c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:05.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.828+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cdc153290 0x7f0cdc0c63b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00011e0 tx=0x7f0cd000ca30 comp rx=0 tx=0).stop 2023-03-03T13:30:05.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 shutdown_connections 2023-03-03T13:30:05.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cbc068320 0x7f0cbc06a7d0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cdc0cd8e0 0x7f0cdc0b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1593c0 0x7f0cdc14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 --2- 172.21.15.102:0/2033585324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cdc153290 0x7f0cdc0c63b0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:05.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 >> 172.21.15.102:0/2033585324 conn(0x7f0cdc065fb0 msgr2=0x7f0cdc150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:30:05.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 shutdown_connections 2023-03-03T13:30:05.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:05.829+0000 7f0ce09ff700 1 -- 172.21.15.102:0/2033585324 wait complete. 2023-03-03T13:30:05.832 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:30:05.857 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T13:30:20.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:20.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:50.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:50.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:57.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:30:58.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.105+0000 7fda9ffff700 1 Processor -- start 2023-03-03T13:30:58.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.105+0000 7fda9ffff700 1 -- start start 2023-03-03T13:30:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdaa00b3970 0x7fdaa00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdaa00c41b0 0x7fdaa00cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa00c3d20 con 0x7fdaa00ced00 2023-03-03T13:30:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9ffff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa00c4930 con 0x7fdaa00c41b0 2023-03-03T13:30:58.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9ffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa00cf610 con 0x7fdaa00b3970 2023-03-03T13:30:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9e7fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdaa00b3970 0x7fdaa00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdaa00c41b0 0x7fdaa00cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9effd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdaa00ced00 0x7fdaa0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36132/0 (socket says 172.21.15.102:36132) 2023-03-03T13:30:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9effd700 1 -- 172.21.15.102:0/2833964620 learned_addr learned my addr 172.21.15.102:0/2833964620 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:30:58.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.107+0000 7fda9dffb700 1 -- 172.21.15.102:0/2833964620 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394594614 0 0) 0x7fdaa00c3d20 con 0x7fdaa00ced00 2023-03-03T13:30:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9dffb700 1 -- 172.21.15.102:0/2833964620 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda880035b0 con 0x7fdaa00ced00 2023-03-03T13:30:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9f7fe700 1 -- 172.21.15.102:0/2833964620 >> v2:172.21.15.172:3300/0 conn(0x7fdaa00b3970 msgr2=0x7fdaa00c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:30:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9f7fe700 1 --2- 172.21.15.102:0/2833964620 >> v2:172.21.15.172:3300/0 conn(0x7fdaa00b3970 0x7fdaa00c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9f7fe700 1 -- 172.21.15.102:0/2833964620 >> v1:172.21.15.172:6789/0 conn(0x7fdaa00ced00 legacy=0x7fdaa0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9f7fe700 1 -- 172.21.15.102:0/2833964620 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa015cba0 con 0x7fdaa00c41b0 2023-03-03T13:30:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9e7fc700 1 --2- 172.21.15.102:0/2833964620 >> v2:172.21.15.172:3300/0 conn(0x7fdaa00b3970 0x7fdaa00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:30:58.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9f7fe700 1 --2- 172.21.15.102:0/2833964620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00c41b0 0x7fdaa00cf0d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00c4ba0 tx=0x7fda9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506e0fe991f66f18 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9dffb700 1 -- 172.21.15.102:0/2833964620 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9800e070 con 0x7fdaa00c41b0 2023-03-03T13:30:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9dffb700 1 -- 172.21.15.102:0/2833964620 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9800bae0 con 0x7fdaa00c41b0 2023-03-03T13:30:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.108+0000 7fda9dffb700 1 -- 172.21.15.102:0/2833964620 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9800be90 con 0x7fdaa00c41b0 2023-03-03T13:30:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 -- 172.21.15.102:0/2833964620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00c41b0 msgr2=0x7fdaa00cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 --2- 172.21.15.102:0/2833964620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00c41b0 0x7fdaa00cf0d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00c4ba0 tx=0x7fda9800b330 comp rx=0 tx=0).stop 2023-03-03T13:30:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 -- 172.21.15.102:0/2833964620 shutdown_connections 2023-03-03T13:30:58.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 --2- 172.21.15.102:0/2833964620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00c41b0 0x7fdaa00cf0d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 --2- 172.21.15.102:0/2833964620 >> v2:172.21.15.172:3300/0 conn(0x7fdaa00b3970 0x7fdaa00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 -- 172.21.15.102:0/2833964620 >> 172.21.15.102:0/2833964620 conn(0x7fdaa008b1c0 msgr2=0x7fdaa0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:30:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 -- 172.21.15.102:0/2833964620 shutdown_connections 2023-03-03T13:30:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.109+0000 7fda9ffff700 1 -- 172.21.15.102:0/2833964620 wait complete. 2023-03-03T13:30:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.111+0000 7fda9ffff700 1 Processor -- start 2023-03-03T13:30:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.111+0000 7fda9ffff700 1 -- start start 2023-03-03T13:30:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.112+0000 7fda9ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaa00b3970 0x7fdaa00c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.112+0000 7fda9ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa00c41b0 0x7fdaa014e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.112+0000 7fda9ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00ced00 0x7fdaa014ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.112+0000 7fda9ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa01e5ed0 con 0x7fdaa00ced00 2023-03-03T13:30:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.112+0000 7fda9ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa01e52e0 con 0x7fdaa00c41b0 2023-03-03T13:30:58.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.112+0000 7fda9ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa01ee300 con 0x7fdaa00b3970 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00ced00 0x7fdaa014ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00ced00 0x7fdaa014ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56730/0 (socket says 172.21.15.102:56730) 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 -- 172.21.15.102:0/490424471 learned_addr learned my addr 172.21.15.102:0/490424471 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9effd700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaa00b3970 0x7fdaa00c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9e7fc700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa00c41b0 0x7fdaa014e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 -- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaa00b3970 msgr2=0x7fdaa00c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaa00b3970 0x7fdaa00c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 -- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa00c41b0 msgr2=0x7fdaa014e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa00c41b0 0x7fdaa014e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 -- 172.21.15.102:0/490424471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940094a0 con 0x7fdaa00ced00 2023-03-03T13:30:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.113+0000 7fda9f7fe700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00ced00 0x7fdaa014ed00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fda98009f00 tx=0x7fda98004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.114+0000 7fda877fe700 1 -- 172.21.15.102:0/490424471 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98004550 con 0x7fdaa00ced00 2023-03-03T13:30:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.114+0000 7fda877fe700 1 -- 172.21.15.102:0/490424471 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda98020070 con 0x7fdaa00ced00 2023-03-03T13:30:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.114+0000 7fda877fe700 1 -- 172.21.15.102:0/490424471 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98004910 con 0x7fdaa00ced00 2023-03-03T13:30:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.114+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda98009a70 con 0x7fdaa00ced00 2023-03-03T13:30:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.114+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa006bf20 con 0x7fdaa00ced00 2023-03-03T13:30:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.115+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa015ce80 con 0x7fdaa00ced00 2023-03-03T13:30:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.116+0000 7fda877fe700 1 -- 172.21.15.102:0/490424471 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda9802e070 con 0x7fdaa00ced00 2023-03-03T13:30:58.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.117+0000 7fda877fe700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda8c0682f0 0x7fda8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:58.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.117+0000 7fda877fe700 1 -- 172.21.15.102:0/490424471 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fda98022070 con 0x7fdaa00ced00 2023-03-03T13:30:58.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.118+0000 7fda9effd700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda8c0682f0 0x7fda8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:58.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.118+0000 7fda9effd700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda8c0682f0 0x7fda8c06a7a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fda90007fd0 tx=0x7fda90007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:58.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.122+0000 7fda877fe700 1 -- 172.21.15.102:0/490424471 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda980540a0 con 0x7fdaa00ced00 2023-03-03T13:30:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.356+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa006c200 con 0x7fdaa00ced00 2023-03-03T13:30:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.357+0000 7fda877fe700 1 -- 172.21.15.102:0/490424471 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fda9802c090 con 0x7fdaa00ced00 2023-03-03T13:30:58.359 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:30:58.359 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:30:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda8c0682f0 msgr2=0x7fda8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda8c0682f0 0x7fda8c06a7a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fda90007fd0 tx=0x7fda90007dc0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00ced00 msgr2=0x7fdaa014ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00ced00 0x7fdaa014ed00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fda98009f00 tx=0x7fda98004b20 comp rx=0 tx=0).stop 2023-03-03T13:30:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 shutdown_connections 2023-03-03T13:30:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fda8c0682f0 0x7fda8c06a7a0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdaa00b3970 0x7fdaa00c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdaa00ced00 0x7fdaa014ed00 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 --2- 172.21.15.102:0/490424471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa00c41b0 0x7fdaa014e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 >> 172.21.15.102:0/490424471 conn(0x7fdaa008b1c0 msgr2=0x7fdaa0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:30:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 shutdown_connections 2023-03-03T13:30:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:30:58.360+0000 7fda9ffff700 1 -- 172.21.15.102:0/490424471 wait complete. 2023-03-03T13:30:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:30:58.389 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T13:31:20.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:20.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:50.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:31:50.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.637+0000 7ff0547ff700 1 Processor -- start 2023-03-03T13:31:50.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.637+0000 7ff0547ff700 1 -- start start 2023-03-03T13:31:50.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.638+0000 7ff0547ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:50.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.638+0000 7ff0547ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c153660 con 0x7ff04c0cd390 2023-03-03T13:31:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff0547ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c06c360 con 0x7ff04c153290 2023-03-03T13:31:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff0547ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c15cea0 con 0x7ff04c0cd8e0 2023-03-03T13:31:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff05259b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff04c153290 0x7ff04c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52884/0 (socket says 172.21.15.102:52884) 2023-03-03T13:31:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff05259b700 1 -- 172.21.15.102:0/1968424652 learned_addr learned my addr 172.21.15.102:0/1968424652 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:31:50.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051d9a700 1 --2- 172.21.15.102:0/1968424652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:50.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051599700 1 -- 172.21.15.102:0/1968424652 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51396829 0 0) 0x7ff04c06c360 con 0x7ff04c153290 2023-03-03T13:31:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051599700 1 -- 172.21.15.102:0/1968424652 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0380035b0 con 0x7ff04c153290 2023-03-03T13:31:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051599700 1 -- 172.21.15.102:0/1968424652 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257549672 0 0) 0x7ff04c153660 con 0x7ff04c0cd390 2023-03-03T13:31:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051599700 1 -- 172.21.15.102:0/1968424652 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c06c360 con 0x7ff04c0cd390 2023-03-03T13:31:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051d9a700 1 -- 172.21.15.102:0/1968424652 >> v1:172.21.15.172:6789/0 conn(0x7ff04c153290 legacy=0x7ff04c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051d9a700 1 -- 172.21.15.102:0/1968424652 >> v1:172.21.15.102:6789/0 conn(0x7ff04c0cd390 legacy=0x7ff04c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:50.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.639+0000 7ff051d9a700 1 -- 172.21.15.102:0/1968424652 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c06c7f0 con 0x7ff04c0cd8e0 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.640+0000 7ff051d9a700 1 --2- 172.21.15.102:0/1968424652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c0cde20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009870 tx=0x7ff03c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a852b17278b79c72 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.640+0000 7ff051599700 1 -- 172.21.15.102:0/1968424652 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c011830 con 0x7ff04c0cd8e0 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.640+0000 7ff051599700 1 -- 172.21.15.102:0/1968424652 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c011990 con 0x7ff04c0cd8e0 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.640+0000 7ff051599700 1 -- 172.21.15.102:0/1968424652 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c011d20 con 0x7ff04c0cd8e0 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.640+0000 7ff0547ff700 1 -- 172.21.15.102:0/1968424652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 msgr2=0x7ff04c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.640+0000 7ff0547ff700 1 --2- 172.21.15.102:0/1968424652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c0cde20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009870 tx=0x7ff03c009b80 comp rx=0 tx=0).stop 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.641+0000 7ff0547ff700 1 -- 172.21.15.102:0/1968424652 shutdown_connections 2023-03-03T13:31:50.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.641+0000 7ff0547ff700 1 --2- 172.21.15.102:0/1968424652 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c0cde20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.641+0000 7ff0547ff700 1 -- 172.21.15.102:0/1968424652 >> 172.21.15.102:0/1968424652 conn(0x7ff04c065fb0 msgr2=0x7ff04c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.641+0000 7ff0547ff700 1 -- 172.21.15.102:0/1968424652 shutdown_connections 2023-03-03T13:31:50.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.641+0000 7ff0547ff700 1 -- 172.21.15.102:0/1968424652 wait complete. 2023-03-03T13:31:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.643+0000 7ff0547ff700 1 Processor -- start 2023-03-03T13:31:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.643+0000 7ff0547ff700 1 -- start start 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff0547ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04c0cd390 0x7ff04c1e2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff0547ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c1e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff0547ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04c153290 0x7ff04c1f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff0547ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c1e3d90 con 0x7ff04c0cd390 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff0547ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c1e5070 con 0x7ff04c153290 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff0547ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c0c6a30 con 0x7ff04c0cd8e0 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff05259b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04c0cd390 0x7ff04c1e2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:50.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff05259b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04c0cd390 0x7ff04c1e2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50606/0 (socket says 172.21.15.102:50606) 2023-03-03T13:31:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff05259b700 1 -- 172.21.15.102:0/3940746180 learned_addr learned my addr 172.21.15.102:0/3940746180 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:31:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff051d9a700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c1e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.644+0000 7ff052d9c700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04c153290 0x7ff04c1f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff05259b700 1 -- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 msgr2=0x7ff04c1e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff05259b700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c1e28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff05259b700 1 -- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04c153290 msgr2=0x7ff04c1f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:50.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff05259b700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04c153290 0x7ff04c1f20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff05259b700 1 -- 172.21.15.102:0/3940746180 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c0094a0 con 0x7ff04c0cd390 2023-03-03T13:31:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff05259b700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04c0cd390 0x7ff04c1e2370 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff048005910 tx=0x7ff04800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff042ffd700 1 -- 172.21.15.102:0/3940746180 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048005d70 con 0x7ff04c0cd390 2023-03-03T13:31:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff042ffd700 1 -- 172.21.15.102:0/3940746180 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0480070d0 con 0x7ff04c0cd390 2023-03-03T13:31:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff042ffd700 1 -- 172.21.15.102:0/3940746180 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048007440 con 0x7ff04c0cd390 2023-03-03T13:31:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.645+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c0ccde0 con 0x7ff04c0cd390 2023-03-03T13:31:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.646+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c15d160 con 0x7ff04c0cd390 2023-03-03T13:31:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.647+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c0cd0c0 con 0x7ff04c0cd390 2023-03-03T13:31:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.648+0000 7ff042ffd700 1 -- 172.21.15.102:0/3940746180 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff048019030 con 0x7ff04c0cd390 2023-03-03T13:31:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.649+0000 7ff042ffd700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0300682f0 0x7ff03006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.649+0000 7ff042ffd700 1 -- 172.21.15.102:0/3940746180 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff048084fe0 con 0x7ff04c0cd390 2023-03-03T13:31:50.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.649+0000 7ff051d9a700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0300682f0 0x7ff03006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:50.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.650+0000 7ff051d9a700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0300682f0 0x7ff03006a7a0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0097c0 tx=0x7ff03c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:50.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.653+0000 7ff042ffd700 1 -- 172.21.15.102:0/3940746180 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff04804b5e0 con 0x7ff04c0cd390 2023-03-03T13:31:50.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:50.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:50.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.895+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff04c04e6e0 con 0x7ff04c0cd390 2023-03-03T13:31:50.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.896+0000 7ff042ffd700 1 -- 172.21.15.102:0/3940746180 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff048017090 con 0x7ff04c0cd390 2023-03-03T13:31:50.898 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:31:50.898 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:31:50.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0300682f0 msgr2=0x7ff03006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:50.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0300682f0 0x7ff03006a7a0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0097c0 tx=0x7ff03c01c040 comp rx=0 tx=0).stop 2023-03-03T13:31:50.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04c0cd390 msgr2=0x7ff04c1e2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:50.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04c0cd390 0x7ff04c1e2370 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff048005910 tx=0x7ff04800c9e0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 shutdown_connections 2023-03-03T13:31:50.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff04c153290 0x7ff04c1f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04c0cd390 0x7ff04c1e2370 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff0300682f0 0x7ff03006a7a0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 --2- 172.21.15.102:0/3940746180 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff04c0cd8e0 0x7ff04c1e28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:50.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 >> 172.21.15.102:0/3940746180 conn(0x7ff04c065fb0 msgr2=0x7ff04c0cb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:50.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 shutdown_connections 2023-03-03T13:31:50.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:31:50.899+0000 7ff0547ff700 1 -- 172.21.15.102:0/3940746180 wait complete. 2023-03-03T13:31:50.902 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:31:50.928 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-03-03T13:32:20.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:20.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:51.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:51.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:21.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:21.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:25.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:33:26.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.176+0000 7f6f5abff700 1 Processor -- start 2023-03-03T13:33:26.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.176+0000 7f6f5abff700 1 -- start start 2023-03-03T13:33:26.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.177+0000 7f6f5abff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f54153290 0x7f6f54158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:26.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.177+0000 7f6f5abff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6f540cd8e0 0x7f6f540ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:26.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5abff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f54153660 con 0x7f6f541593c0 2023-03-03T13:33:26.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5abff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5406c360 con 0x7f6f540cd8e0 2023-03-03T13:33:26.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5abff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5415cea0 con 0x7f6f54153290 2023-03-03T13:33:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f53fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6f541593c0 0x7f6f540cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49818/0 (socket says 172.21.15.102:49818) 2023-03-03T13:33:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f53fff700 1 -- 172.21.15.102:0/1417580839 learned_addr learned my addr 172.21.15.102:0/1417580839 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:33:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5919c700 1 --2- 172.21.15.102:0/1417580839 >> v2:172.21.15.172:3300/0 conn(0x7f6f540cd8e0 0x7f6f540ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5899b700 1 --2- 172.21.15.102:0/1417580839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f54153290 0x7f6f54158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5899b700 1 -- 172.21.15.102:0/1417580839 >> v2:172.21.15.172:3300/0 conn(0x7f6f540cd8e0 msgr2=0x7f6f540ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:33:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5899b700 1 --2- 172.21.15.102:0/1417580839 >> v2:172.21.15.172:3300/0 conn(0x7f6f540cd8e0 0x7f6f540ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f537fe700 1 -- 172.21.15.102:0/1417580839 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156892411 0 0) 0x7f6f54153660 con 0x7f6f541593c0 2023-03-03T13:33:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5899b700 1 -- 172.21.15.102:0/1417580839 >> v1:172.21.15.102:6789/0 conn(0x7f6f541593c0 legacy=0x7f6f540cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5899b700 1 -- 172.21.15.102:0/1417580839 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5406c7f0 con 0x7f6f54153290 2023-03-03T13:33:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.178+0000 7f6f5919c700 1 --2- 172.21.15.102:0/1417580839 >> v2:172.21.15.172:3300/0 conn(0x7f6f540cd8e0 0x7f6f540ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:33:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.179+0000 7f6f5899b700 1 --2- 172.21.15.102:0/1417580839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f54153290 0x7f6f54158e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009840 tx=0x7f6f48009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f053393a0e81371 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.179+0000 7f6f537fe700 1 -- 172.21.15.102:0/1417580839 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48011830 con 0x7f6f54153290 2023-03-03T13:33:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.179+0000 7f6f537fe700 1 -- 172.21.15.102:0/1417580839 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48011990 con 0x7f6f54153290 2023-03-03T13:33:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.179+0000 7f6f537fe700 1 -- 172.21.15.102:0/1417580839 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48011d20 con 0x7f6f54153290 2023-03-03T13:33:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 -- 172.21.15.102:0/1417580839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f54153290 msgr2=0x7f6f54158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 --2- 172.21.15.102:0/1417580839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f54153290 0x7f6f54158e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009840 tx=0x7f6f48009b50 comp rx=0 tx=0).stop 2023-03-03T13:33:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 -- 172.21.15.102:0/1417580839 shutdown_connections 2023-03-03T13:33:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 --2- 172.21.15.102:0/1417580839 >> v2:172.21.15.172:3300/0 conn(0x7f6f540cd8e0 0x7f6f540ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 --2- 172.21.15.102:0/1417580839 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f54153290 0x7f6f54158e80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 -- 172.21.15.102:0/1417580839 >> 172.21.15.102:0/1417580839 conn(0x7f6f54065fb0 msgr2=0x7f6f540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:33:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 -- 172.21.15.102:0/1417580839 shutdown_connections 2023-03-03T13:33:26.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.180+0000 7f6f5abff700 1 -- 172.21.15.102:0/1417580839 wait complete. 2023-03-03T13:33:26.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.182+0000 7f6f5abff700 1 Processor -- start 2023-03-03T13:33:26.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.182+0000 7f6f5abff700 1 -- start start 2023-03-03T13:33:26.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5abff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f540cd8e0 0x7f6f540c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:26.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5abff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f54153290 0x7f6f540c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5abff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f541593c0 0x7f6f540c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5abff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540c67f0 con 0x7f6f541593c0 2023-03-03T13:33:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5abff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540c61e0 con 0x7f6f54153290 2023-03-03T13:33:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5abff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f541ef1e0 con 0x7f6f540cd8e0 2023-03-03T13:33:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5899b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f540cd8e0 0x7f6f540c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5899b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f540cd8e0 0x7f6f540c81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38862/0 (socket says 172.21.15.102:38862) 2023-03-03T13:33:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5899b700 1 -- 172.21.15.102:0/3452187124 learned_addr learned my addr 172.21.15.102:0/3452187124 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:33:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f53fff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f54153290 0x7f6f540c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.183+0000 7f6f5919c700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f541593c0 0x7f6f540c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5899b700 1 -- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f54153290 msgr2=0x7f6f540c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5899b700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f54153290 0x7f6f540c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5899b700 1 -- 172.21.15.102:0/3452187124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f541593c0 msgr2=0x7f6f540c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5899b700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f541593c0 0x7f6f540c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5899b700 1 -- 172.21.15.102:0/3452187124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c009a70 con 0x7f6f540cd8e0 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5919c700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f541593c0 0x7f6f540c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f53fff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f54153290 0x7f6f540c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5899b700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f540cd8e0 0x7f6f540c81c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6f48014040 tx=0x7f6f48004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f517fa700 1 -- 172.21.15.102:0/3452187124 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f480038a0 con 0x7f6f540cd8e0 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f517fa700 1 -- 172.21.15.102:0/3452187124 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4801e070 con 0x7f6f540cd8e0 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f480094a0 con 0x7f6f540cd8e0 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f517fa700 1 -- 172.21.15.102:0/3452187124 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48003c20 con 0x7f6f540cd8e0 2023-03-03T13:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.184+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f540ccde0 con 0x7f6f540cd8e0 2023-03-03T13:33:26.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.185+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5415d160 con 0x7f6f540cd8e0 2023-03-03T13:33:26.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.187+0000 7f6f517fa700 1 -- 172.21.15.102:0/3452187124 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f4800f050 con 0x7f6f540cd8e0 2023-03-03T13:33:26.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.188+0000 7f6f517fa700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f380682f0 0x7f6f3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:26.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.188+0000 7f6f517fa700 1 -- 172.21.15.102:0/3452187124 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6f48087780 con 0x7f6f540cd8e0 2023-03-03T13:33:26.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.189+0000 7f6f53fff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f380682f0 0x7f6f3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:26.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.189+0000 7f6f53fff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f380682f0 0x7f6f3806a7a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f6f540b1e70 tx=0x7f6f40007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:26.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.191+0000 7f6f517fa700 1 -- 172.21.15.102:0/3452187124 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f48053f10 con 0x7f6f540cd8e0 2023-03-03T13:33:26.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.424+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f540cd0c0 con 0x7f6f540cd8e0 2023-03-03T13:33:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.426+0000 7f6f517fa700 1 -- 172.21.15.102:0/3452187124 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6f48057530 con 0x7f6f540cd8e0 2023-03-03T13:33:26.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:33:26.428 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:33:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f380682f0 msgr2=0x7f6f3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f380682f0 0x7f6f3806a7a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f6f540b1e70 tx=0x7f6f40007ca0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f540cd8e0 msgr2=0x7f6f540c81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f540cd8e0 0x7f6f540c81c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6f48014040 tx=0x7f6f48004750 comp rx=0 tx=0).stop 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 shutdown_connections 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6f380682f0 0x7f6f3806a7a0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f540cd8e0 0x7f6f540c81c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f541593c0 0x7f6f540c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 --2- 172.21.15.102:0/3452187124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f54153290 0x7f6f540c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 >> 172.21.15.102:0/3452187124 conn(0x7f6f54065fb0 msgr2=0x7f6f54157bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 shutdown_connections 2023-03-03T13:33:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:33:26.428+0000 7f6f5abff700 1 -- 172.21.15.102:0/3452187124 wait complete. 2023-03-03T13:33:26.432 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:33:26.457 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-03-03T13:33:51.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:51.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:21.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:21.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:49.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:34:49.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.708+0000 7fc1af19d700 1 Processor -- start 2023-03-03T13:34:49.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.708+0000 7fc1af19d700 1 -- start start 2023-03-03T13:34:49.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1af19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc1a8153030 0x7fc1a80c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:49.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1af19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:49.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1af19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a806bee0 con 0x7fc1a80b3f30 2023-03-03T13:34:49.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1af19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a806c780 con 0x7fc1a8153030 2023-03-03T13:34:49.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1af19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a815c620 con 0x7fc1a80b1e90 2023-03-03T13:34:49.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1ad99a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:49.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1ae19b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc1a8153030 0x7fc1a80c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:49.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1ae99c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc1a80b3f30 0x7fc1a80cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36620/0 (socket says 172.21.15.102:36620) 2023-03-03T13:34:49.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.710+0000 7fc1ae99c700 1 -- 172.21.15.102:0/1703891600 learned_addr learned my addr 172.21.15.102:0/1703891600 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:34:49.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ad99a700 1 -- 172.21.15.102:0/1703891600 >> v2:172.21.15.172:3300/0 conn(0x7fc1a8153030 msgr2=0x7fc1a80c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:34:49.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ad99a700 1 --2- 172.21.15.102:0/1703891600 >> v2:172.21.15.172:3300/0 conn(0x7fc1a8153030 0x7fc1a80c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ad99a700 1 -- 172.21.15.102:0/1703891600 >> v1:172.21.15.172:6789/0 conn(0x7fc1a80b3f30 legacy=0x7fc1a80cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:49.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ad99a700 1 -- 172.21.15.102:0/1703891600 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a806c1c0 con 0x7fc1a80b1e90 2023-03-03T13:34:49.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ae19b700 1 --2- 172.21.15.102:0/1703891600 >> v2:172.21.15.172:3300/0 conn(0x7fc1a8153030 0x7fc1a80c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:34:49.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ad199700 1 -- 172.21.15.102:0/1703891600 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724626542 0 0) 0x7fc1a806bee0 con 0x7fc1a80b3f30 2023-03-03T13:34:49.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ad99a700 1 --2- 172.21.15.102:0/1703891600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a815ca40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc198009840 tx=0x7fc198009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e7d3986c4a4d214 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:49.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.711+0000 7fc1ad199700 1 -- 172.21.15.102:0/1703891600 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198011830 con 0x7fc1a80b1e90 2023-03-03T13:34:49.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1ad199700 1 -- 172.21.15.102:0/1703891600 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc198011990 con 0x7fc1a80b1e90 2023-03-03T13:34:49.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1ad199700 1 -- 172.21.15.102:0/1703891600 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198011d20 con 0x7fc1a80b1e90 2023-03-03T13:34:49.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 -- 172.21.15.102:0/1703891600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 msgr2=0x7fc1a815ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:49.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 --2- 172.21.15.102:0/1703891600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a815ca40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc198009840 tx=0x7fc198009b50 comp rx=0 tx=0).stop 2023-03-03T13:34:49.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 -- 172.21.15.102:0/1703891600 shutdown_connections 2023-03-03T13:34:49.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 --2- 172.21.15.102:0/1703891600 >> v2:172.21.15.172:3300/0 conn(0x7fc1a8153030 0x7fc1a80c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 --2- 172.21.15.102:0/1703891600 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a815ca40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 -- 172.21.15.102:0/1703891600 >> 172.21.15.102:0/1703891600 conn(0x7fc1a808b1c0 msgr2=0x7fc1a8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:34:49.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 -- 172.21.15.102:0/1703891600 shutdown_connections 2023-03-03T13:34:49.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.712+0000 7fc1af19d700 1 -- 172.21.15.102:0/1703891600 wait complete. 2023-03-03T13:34:49.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.714+0000 7fc1af19d700 1 Processor -- start 2023-03-03T13:34:49.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.714+0000 7fc1af19d700 1 -- start start 2023-03-03T13:34:49.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.715+0000 7fc1af19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a81ef640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:49.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.715+0000 7fc1af19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1a80b3f30 0x7fc1a814e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:49.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.715+0000 7fc1af19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1a8153030 0x7fc1a814eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:49.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.715+0000 7fc1af19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80c6ac0 con 0x7fc1a8153030 2023-03-03T13:34:49.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.715+0000 7fc1af19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a81f1870 con 0x7fc1a80b3f30 2023-03-03T13:34:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.715+0000 7fc1af19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80c6d10 con 0x7fc1a80b1e90 2023-03-03T13:34:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a81ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ad99a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1a80b3f30 0x7fc1a814e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a81ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:47696/0 (socket says 172.21.15.102:47696) 2023-03-03T13:34:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 -- 172.21.15.102:0/2440818559 learned_addr learned my addr 172.21.15.102:0/2440818559 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:34:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae99c700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1a8153030 0x7fc1a814eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 -- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1a80b3f30 msgr2=0x7fc1a814e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1a80b3f30 0x7fc1a814e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 -- 172.21.15.102:0/2440818559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1a8153030 msgr2=0x7fc1a814eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1a8153030 0x7fc1a814eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 -- 172.21.15.102:0/2440818559 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a40094a0 con 0x7fc1a80b1e90 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae99c700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1a8153030 0x7fc1a814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.716+0000 7fc1ae19b700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a81ef640 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400ebe0 tx=0x7fc1a400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.717+0000 7fc19effd700 1 -- 172.21.15.102:0/2440818559 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a400c850 con 0x7fc1a80b1e90 2023-03-03T13:34:49.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.717+0000 7fc19effd700 1 -- 172.21.15.102:0/2440818559 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a400c9b0 con 0x7fc1a80b1e90 2023-03-03T13:34:49.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.717+0000 7fc19effd700 1 -- 172.21.15.102:0/2440818559 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a400cd40 con 0x7fc1a80b1e90 2023-03-03T13:34:49.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.717+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1980094a0 con 0x7fc1a80b1e90 2023-03-03T13:34:49.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.717+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a815c240 con 0x7fc1a80b1e90 2023-03-03T13:34:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.718+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a806c4a0 con 0x7fc1a80b1e90 2023-03-03T13:34:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.719+0000 7fc19effd700 1 -- 172.21.15.102:0/2440818559 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1a4019030 con 0x7fc1a80b1e90 2023-03-03T13:34:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.720+0000 7fc19effd700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc18c068320 0x7fc18c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.720+0000 7fc19effd700 1 -- 172.21.15.102:0/2440818559 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc1a4085d30 con 0x7fc1a80b1e90 2023-03-03T13:34:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.721+0000 7fc1ad99a700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc18c068320 0x7fc18c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:49.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.721+0000 7fc1ad99a700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc18c068320 0x7fc18c06a7d0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc1980090b0 tx=0x7fc198008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:49.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.725+0000 7fc19effd700 1 -- 172.21.15.102:0/2440818559 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1a4052440 con 0x7fc1a80b1e90 2023-03-03T13:34:49.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.961+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1a815cf80 con 0x7fc1a80b1e90 2023-03-03T13:34:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.961+0000 7fc19effd700 1 -- 172.21.15.102:0/2440818559 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc1a400cea0 con 0x7fc1a80b1e90 2023-03-03T13:34:49.964 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:34:49.964 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:34:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc18c068320 msgr2=0x7fc18c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc18c068320 0x7fc18c06a7d0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc1980090b0 tx=0x7fc198008030 comp rx=0 tx=0).stop 2023-03-03T13:34:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 msgr2=0x7fc1a81ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a81ef640 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400ebe0 tx=0x7fc1a400eef0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 shutdown_connections 2023-03-03T13:34:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc18c068320 0x7fc18c06a7d0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc1a80b1e90 0x7fc1a81ef640 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1a8153030 0x7fc1a814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 --2- 172.21.15.102:0/2440818559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1a80b3f30 0x7fc1a814e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 >> 172.21.15.102:0/2440818559 conn(0x7fc1a808b1c0 msgr2=0x7fc1a81507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:34:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 shutdown_connections 2023-03-03T13:34:49.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:34:49.964+0000 7fc1af19d700 1 -- 172.21.15.102:0/2440818559 wait complete. 2023-03-03T13:34:49.968 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:34:49.993 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2023-03-03T13:34:51.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:51.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:21.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:21.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:44.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:35:45.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.242+0000 7f3ba6b9d700 1 Processor -- start 2023-03-03T13:35:45.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.242+0000 7f3ba6b9d700 1 -- start start 2023-03-03T13:35:45.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.244+0000 7f3ba6b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ba00ced00 0x7f3ba00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:45.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.244+0000 7f3ba6b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00c3d20 con 0x7f3ba00c41b0 2023-03-03T13:35:45.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.244+0000 7f3ba6b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba015cba0 con 0x7f3ba00b3970 2023-03-03T13:35:45.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.244+0000 7f3ba6b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba00c4930 con 0x7f3ba00ced00 2023-03-03T13:35:45.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.244+0000 7f3ba5b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3ba00ced00 0x7f3ba00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:45.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.244+0000 7f3ba539a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3ba00b3970 0x7f3ba0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45008/0 (socket says 172.21.15.102:45008) 2023-03-03T13:35:45.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.244+0000 7f3ba539a700 1 -- 172.21.15.102:0/3384221233 learned_addr learned my addr 172.21.15.102:0/3384221233 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:35:45.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba5b9b700 1 -- 172.21.15.102:0/3384221233 >> v1:172.21.15.172:6789/0 conn(0x7f3ba00b3970 legacy=0x7f3ba0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:45.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba5b9b700 1 -- 172.21.15.102:0/3384221233 >> v1:172.21.15.102:6789/0 conn(0x7f3ba00c41b0 legacy=0x7f3ba0163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:45.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba5b9b700 1 -- 172.21.15.102:0/3384221233 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba015ce80 con 0x7f3ba00ced00 2023-03-03T13:35:45.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba4b99700 1 -- 172.21.15.102:0/3384221233 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540797968 0 0) 0x7f3ba015cba0 con 0x7f3ba00b3970 2023-03-03T13:35:45.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba5b9b700 1 --2- 172.21.15.102:0/3384221233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00ced00 0x7f3ba00c5a90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009840 tx=0x7f3b9c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=434eb314ba65521 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:35:45.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba4b99700 1 -- 172.21.15.102:0/3384221233 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c004230 con 0x7f3ba00ced00 2023-03-03T13:35:45.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba4b99700 1 -- 172.21.15.102:0/3384221233 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c004390 con 0x7f3ba00ced00 2023-03-03T13:35:45.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.245+0000 7f3ba4b99700 1 -- 172.21.15.102:0/3384221233 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0036a0 con 0x7f3ba00ced00 2023-03-03T13:35:45.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.246+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3384221233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00ced00 msgr2=0x7f3ba00c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:45.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.246+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3384221233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00ced00 0x7f3ba00c5a90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009840 tx=0x7f3b9c009b50 comp rx=0 tx=0).stop 2023-03-03T13:35:45.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.246+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3384221233 shutdown_connections 2023-03-03T13:35:45.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.246+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3384221233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00ced00 0x7f3ba00c5a90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:45.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.246+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3384221233 >> 172.21.15.102:0/3384221233 conn(0x7f3ba008b1c0 msgr2=0x7f3ba0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:35:45.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.246+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3384221233 shutdown_connections 2023-03-03T13:35:45.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.246+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3384221233 wait complete. 2023-03-03T13:35:45.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.248+0000 7f3ba6b9d700 1 Processor -- start 2023-03-03T13:35:45.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.248+0000 7f3ba6b9d700 1 -- start start 2023-03-03T13:35:45.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba6b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00b3970 0x7f3ba00c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:45.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba6b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ba00c41b0 0x7f3ba01e3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:45.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba6b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00ced00 0x7f3ba014dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:45.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba6b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00ceb20 con 0x7f3ba00ced00 2023-03-03T13:35:45.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba6b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00ce840 con 0x7f3ba00b3970 2023-03-03T13:35:45.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba6b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00c6210 con 0x7f3ba00c41b0 2023-03-03T13:35:45.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba639c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00ced00 0x7f3ba014dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:45.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba5b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00b3970 0x7f3ba00c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:45.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba539a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ba00c41b0 0x7f3ba01e3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:45.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba639c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00ced00 0x7f3ba014dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34372/0 (socket says 172.21.15.102:34372) 2023-03-03T13:35:45.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba639c700 1 -- 172.21.15.102:0/3878694312 learned_addr learned my addr 172.21.15.102:0/3878694312 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.249+0000 7f3ba5b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00b3970 0x7f3ba00c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46424/0 (socket says 172.21.15.102:46424) 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.250+0000 7f3ba639c700 1 -- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ba00c41b0 msgr2=0x7f3ba01e3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.250+0000 7f3ba639c700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ba00c41b0 0x7f3ba01e3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.250+0000 7f3ba639c700 1 -- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00b3970 msgr2=0x7f3ba00c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.250+0000 7f3ba639c700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00b3970 0x7f3ba00c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.250+0000 7f3ba639c700 1 -- 172.21.15.102:0/3878694312 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c0094a0 con 0x7f3ba00ced00 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.250+0000 7f3ba5b9b700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00b3970 0x7f3ba00c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:35:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.250+0000 7f3ba639c700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00ced00 0x7f3ba014dad0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800cca0 tx=0x7f3b9800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:35:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.251+0000 7f3b967fc700 1 -- 172.21.15.102:0/3878694312 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b98003e80 con 0x7f3ba00ced00 2023-03-03T13:35:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.251+0000 7f3b967fc700 1 -- 172.21.15.102:0/3878694312 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9800a700 con 0x7f3ba00ced00 2023-03-03T13:35:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.251+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba006c200 con 0x7f3ba00ced00 2023-03-03T13:35:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.251+0000 7f3b967fc700 1 -- 172.21.15.102:0/3878694312 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9800aa90 con 0x7f3ba00ced00 2023-03-03T13:35:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.251+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba006bf20 con 0x7f3ba00ced00 2023-03-03T13:35:45.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.252+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba015c890 con 0x7f3ba00ced00 2023-03-03T13:35:45.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.253+0000 7f3b967fc700 1 -- 172.21.15.102:0/3878694312 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3b98010070 con 0x7f3ba00ced00 2023-03-03T13:35:45.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.254+0000 7f3b967fc700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3b840682f0 0x7f3b8406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:45.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.255+0000 7f3b967fc700 1 -- 172.21.15.102:0/3878694312 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3b98085c90 con 0x7f3ba00ced00 2023-03-03T13:35:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.255+0000 7f3ba5b9b700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3b840682f0 0x7f3b8406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.255+0000 7f3ba5b9b700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3b840682f0 0x7f3b8406a7a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0090b0 tx=0x7f3b9c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:35:45.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.259+0000 7f3b967fc700 1 -- 172.21.15.102:0/3878694312 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b980523a0 con 0x7f3ba00ced00 2023-03-03T13:35:45.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.493+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ba004e6e0 con 0x7f3ba00ced00 2023-03-03T13:35:45.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.494+0000 7f3b967fc700 1 -- 172.21.15.102:0/3878694312 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3b980559c0 con 0x7f3ba00ced00 2023-03-03T13:35:45.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:35:45.496 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:35:45.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.496+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3b840682f0 msgr2=0x7f3b8406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:45.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.496+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3b840682f0 0x7f3b8406a7a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0090b0 tx=0x7f3b9c008030 comp rx=0 tx=0).stop 2023-03-03T13:35:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.496+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00ced00 msgr2=0x7f3ba014dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.496+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00ced00 0x7f3ba014dad0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800cca0 tx=0x7f3b9800a060 comp rx=0 tx=0).stop 2023-03-03T13:35:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 shutdown_connections 2023-03-03T13:35:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00ced00 0x7f3ba014dad0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ba00b3970 0x7f3ba00c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3b840682f0 0x7f3b8406a7a0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:45.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 --2- 172.21.15.102:0/3878694312 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3ba00c41b0 0x7f3ba01e3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 >> 172.21.15.102:0/3878694312 conn(0x7f3ba008b1c0 msgr2=0x7f3ba0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:35:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 shutdown_connections 2023-03-03T13:35:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:45.497+0000 7f3ba6b9d700 1 -- 172.21.15.102:0/3878694312 wait complete. 2023-03-03T13:35:45.500 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:35:45.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 11 secs before thrashing 2023-03-03T13:35:52.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:52.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:56.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:35:56.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.775+0000 7fb3a61ff700 1 Processor -- start 2023-03-03T13:35:56.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.776+0000 7fb3a61ff700 1 -- start start 2023-03-03T13:35:56.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.777+0000 7fb3a61ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3a014ffb0 0x7fb3a00c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:56.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.777+0000 7fb3a61ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a00c3c80 con 0x7fb3a014fbe0 2023-03-03T13:35:56.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.777+0000 7fb3a61ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a00c41a0 con 0x7fb3a00716c0 2023-03-03T13:35:56.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.777+0000 7fb3a61ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a0071cd0 con 0x7fb3a014ffb0 2023-03-03T13:35:56.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb397fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3a014ffb0 0x7fb3a00c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:56.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb39ffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb3a00716c0 0x7fb3a00706f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54676/0 (socket says 172.21.15.102:54676) 2023-03-03T13:35:56.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb39ffff700 1 -- 172.21.15.102:0/533104728 learned_addr learned my addr 172.21.15.102:0/533104728 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb39f7fe700 1 --1- 172.21.15.102:0/533104728 >> v1:172.21.15.102:6789/0 conn(0x7fb3a014fbe0 0x7fb3a00c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52824/0 (socket says 172.21.15.102:52824) 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb397fff700 1 -- 172.21.15.102:0/533104728 >> v1:172.21.15.172:6789/0 conn(0x7fb3a00716c0 legacy=0x7fb3a00706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb397fff700 1 -- 172.21.15.102:0/533104728 >> v1:172.21.15.102:6789/0 conn(0x7fb3a014fbe0 legacy=0x7fb3a00c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb39effd700 1 -- 172.21.15.102:0/533104728 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563276047 0 0) 0x7fb3a00c41a0 con 0x7fb3a00716c0 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb397fff700 1 -- 172.21.15.102:0/533104728 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00c3ec0 con 0x7fb3a014ffb0 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.778+0000 7fb397fff700 1 --2- 172.21.15.102:0/533104728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 0x7fb3a00c3740 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb388009840 tx=0x7fb388009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc6743594d597c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb39effd700 1 -- 172.21.15.102:0/533104728 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388011830 con 0x7fb3a014ffb0 2023-03-03T13:35:56.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb39effd700 1 -- 172.21.15.102:0/533104728 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb388011990 con 0x7fb3a014ffb0 2023-03-03T13:35:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb39effd700 1 -- 172.21.15.102:0/533104728 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388011d20 con 0x7fb3a014ffb0 2023-03-03T13:35:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb3a61ff700 1 -- 172.21.15.102:0/533104728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 msgr2=0x7fb3a00c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:56.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/533104728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 0x7fb3a00c3740 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb388009840 tx=0x7fb388009b50 comp rx=0 tx=0).stop 2023-03-03T13:35:56.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb3a61ff700 1 -- 172.21.15.102:0/533104728 shutdown_connections 2023-03-03T13:35:56.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/533104728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 0x7fb3a00c3740 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:56.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb3a61ff700 1 -- 172.21.15.102:0/533104728 >> 172.21.15.102:0/533104728 conn(0x7fb3a006af90 msgr2=0x7fb3a006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:35:56.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb3a61ff700 1 -- 172.21.15.102:0/533104728 shutdown_connections 2023-03-03T13:35:56.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.779+0000 7fb3a61ff700 1 -- 172.21.15.102:0/533104728 wait complete. 2023-03-03T13:35:56.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.781+0000 7fb3a61ff700 1 Processor -- start 2023-03-03T13:35:56.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.782+0000 7fb3a61ff700 1 -- start start 2023-03-03T13:35:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb3a61ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a00716c0 0x7fb3a0144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb3a61ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3a014fbe0 0x7fb3a01ec030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb3a61ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 0x7fb3a01ec570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:56.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb3a61ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a01f0170 con 0x7fb3a014ffb0 2023-03-03T13:35:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb3a61ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a01ef250 con 0x7fb3a00716c0 2023-03-03T13:35:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb3a61ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a01ef4f0 con 0x7fb3a014fbe0 2023-03-03T13:35:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a00716c0 0x7fb3a0144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a00716c0 0x7fb3a0144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39456/0 (socket says 172.21.15.102:39456) 2023-03-03T13:35:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb397fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3a014fbe0 0x7fb3a01ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 -- 172.21.15.102:0/2117652834 learned_addr learned my addr 172.21.15.102:0/2117652834 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:35:56.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39ffff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 0x7fb3a01ec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 -- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3a014fbe0 msgr2=0x7fb3a01ec030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3a014fbe0 0x7fb3a01ec030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 -- 172.21.15.102:0/2117652834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 msgr2=0x7fb3a01ec570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 0x7fb3a01ec570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.783+0000 7fb39f7fe700 1 -- 172.21.15.102:0/2117652834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880094a0 con 0x7fb3a00716c0 2023-03-03T13:35:56.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.784+0000 7fb39f7fe700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a00716c0 0x7fb3a0144ca0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb39000c520 tx=0x7fb39000c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:35:56.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.784+0000 7fb39cff9700 1 -- 172.21.15.102:0/2117652834 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390005cb0 con 0x7fb3a00716c0 2023-03-03T13:35:56.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.784+0000 7fb39cff9700 1 -- 172.21.15.102:0/2117652834 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb390005e10 con 0x7fb3a00716c0 2023-03-03T13:35:56.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.784+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a0072250 con 0x7fb3a00716c0 2023-03-03T13:35:56.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.784+0000 7fb39cff9700 1 -- 172.21.15.102:0/2117652834 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3900072d0 con 0x7fb3a00716c0 2023-03-03T13:35:56.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.784+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a0071f70 con 0x7fb3a00716c0 2023-03-03T13:35:56.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.785+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a0072530 con 0x7fb3a00716c0 2023-03-03T13:35:56.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.787+0000 7fb39cff9700 1 -- 172.21.15.102:0/2117652834 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb390019030 con 0x7fb3a00716c0 2023-03-03T13:35:56.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.788+0000 7fb39cff9700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3800682f0 0x7fb38006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:35:56.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.788+0000 7fb39cff9700 1 -- 172.21.15.102:0/2117652834 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb390084d70 con 0x7fb3a00716c0 2023-03-03T13:35:56.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.788+0000 7fb397fff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3800682f0 0x7fb38006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:35:56.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.789+0000 7fb397fff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3800682f0 0x7fb38006a7a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb3880090b0 tx=0x7fb38801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:35:56.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:56.792+0000 7fb39cff9700 1 -- 172.21.15.102:0/2117652834 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb39004b5e0 con 0x7fb3a00716c0 2023-03-03T13:35:57.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.024+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3a00c3c80 con 0x7fb3a00716c0 2023-03-03T13:35:57.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.025+0000 7fb39cff9700 1 -- 172.21.15.102:0/2117652834 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb3a00c3c80 con 0x7fb3a00716c0 2023-03-03T13:35:57.027 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:35:57.027 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:35:57.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.027+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3800682f0 msgr2=0x7fb38006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:57.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.027+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3800682f0 0x7fb38006a7a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb3880090b0 tx=0x7fb38801c040 comp rx=0 tx=0).stop 2023-03-03T13:35:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a00716c0 msgr2=0x7fb3a0144ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:35:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a00716c0 0x7fb3a0144ca0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb39000c520 tx=0x7fb39000c830 comp rx=0 tx=0).stop 2023-03-03T13:35:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 shutdown_connections 2023-03-03T13:35:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3a014ffb0 0x7fb3a01ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a00716c0 0x7fb3a0144ca0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3800682f0 0x7fb38006a7a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 --2- 172.21.15.102:0/2117652834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3a014fbe0 0x7fb3a01ec030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:35:57.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 >> 172.21.15.102:0/2117652834 conn(0x7fb3a006af90 msgr2=0x7fb3a0155b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:35:57.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 shutdown_connections 2023-03-03T13:35:57.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:35:57.028+0000 7fb3a61ff700 1 -- 172.21.15.102:0/2117652834 wait complete. 2023-03-03T13:35:57.031 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:35:57.056 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-03-03T13:36:22.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:22.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:52.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:52.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:22.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:22.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:45.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:37:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.324+0000 7fe48a3ff700 1 Processor -- start 2023-03-03T13:37:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.324+0000 7fe48a3ff700 1 -- start start 2023-03-03T13:37:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.325+0000 7fe48a3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe484153290 0x7fe484158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.325+0000 7fe48a3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe4841593c0 0x7fe4840cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe48a3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe484153660 con 0x7fe4840cd8e0 2023-03-03T13:37:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe48a3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48406c360 con 0x7fe484153290 2023-03-03T13:37:45.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe48a3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48415cea0 con 0x7fe4841593c0 2023-03-03T13:37:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe483fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe484153290 0x7fe484158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe4837fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe4841593c0 0x7fe4840cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe483fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe484153290 0x7fe484158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59440/0 (socket says 172.21.15.102:59440) 2023-03-03T13:37:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe483fff700 1 -- 172.21.15.102:0/3430851448 learned_addr learned my addr 172.21.15.102:0/3430851448 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:37:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe483fff700 1 -- 172.21.15.102:0/3430851448 >> v2:172.21.15.172:3300/0 conn(0x7fe4841593c0 msgr2=0x7fe4840cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:37:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe483fff700 1 --2- 172.21.15.102:0/3430851448 >> v2:172.21.15.172:3300/0 conn(0x7fe4841593c0 0x7fe4840cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe483fff700 1 -- 172.21.15.102:0/3430851448 >> v1:172.21.15.102:6789/0 conn(0x7fe4840cd8e0 legacy=0x7fe4840ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe483fff700 1 -- 172.21.15.102:0/3430851448 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48406c7f0 con 0x7fe484153290 2023-03-03T13:37:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.326+0000 7fe482ffd700 1 -- 172.21.15.102:0/3430851448 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108430161 0 0) 0x7fe484153660 con 0x7fe4840cd8e0 2023-03-03T13:37:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.327+0000 7fe483fff700 1 --2- 172.21.15.102:0/3430851448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe484153290 0x7fe484158e80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe484159fb0 tx=0x7fe474009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f2e34240817670 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.327+0000 7fe482ffd700 1 -- 172.21.15.102:0/3430851448 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474011830 con 0x7fe484153290 2023-03-03T13:37:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.327+0000 7fe482ffd700 1 -- 172.21.15.102:0/3430851448 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe474011990 con 0x7fe484153290 2023-03-03T13:37:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.327+0000 7fe482ffd700 1 -- 172.21.15.102:0/3430851448 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474011d20 con 0x7fe484153290 2023-03-03T13:37:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 -- 172.21.15.102:0/3430851448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe484153290 msgr2=0x7fe484158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/3430851448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe484153290 0x7fe484158e80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe484159fb0 tx=0x7fe474009bd0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 -- 172.21.15.102:0/3430851448 shutdown_connections 2023-03-03T13:37:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/3430851448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe484153290 0x7fe484158e80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/3430851448 >> v2:172.21.15.172:3300/0 conn(0x7fe4841593c0 0x7fe4840cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 -- 172.21.15.102:0/3430851448 >> 172.21.15.102:0/3430851448 conn(0x7fe484065fb0 msgr2=0x7fe4840663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:37:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 -- 172.21.15.102:0/3430851448 shutdown_connections 2023-03-03T13:37:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.328+0000 7fe48a3ff700 1 -- 172.21.15.102:0/3430851448 wait complete. 2023-03-03T13:37:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.330+0000 7fe48a3ff700 1 Processor -- start 2023-03-03T13:37:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.330+0000 7fe48a3ff700 1 -- start start 2023-03-03T13:37:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe48a3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4840cd8e0 0x7fe48414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe48a3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe484153290 0x7fe4841e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe48a3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4841593c0 0x7fe4841e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe48a3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48414ea90 con 0x7fe4840cd8e0 2023-03-03T13:37:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe48a3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4841f02b0 con 0x7fe4841593c0 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe48a3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4840b3e20 con 0x7fe484153290 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe483fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4840cd8e0 0x7fe48414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe483fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4840cd8e0 0x7fe48414e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59454/0 (socket says 172.21.15.102:59454) 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.331+0000 7fe483fff700 1 -- 172.21.15.102:0/414743268 learned_addr learned my addr 172.21.15.102:0/414743268 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe4837fe700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe484153290 0x7fe4841e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe48899c700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4841593c0 0x7fe4841e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe483fff700 1 -- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe484153290 msgr2=0x7fe4841e5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe483fff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe484153290 0x7fe4841e5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe483fff700 1 -- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4841593c0 msgr2=0x7fe4841e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe483fff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4841593c0 0x7fe4841e5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe483fff700 1 -- 172.21.15.102:0/414743268 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe474009550 con 0x7fe4840cd8e0 2023-03-03T13:37:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.332+0000 7fe483fff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4840cd8e0 0x7fe48414e550 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe474009c10 tx=0x7fe474004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.333+0000 7fe480ff9700 1 -- 172.21.15.102:0/414743268 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474004510 con 0x7fe4840cd8e0 2023-03-03T13:37:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.333+0000 7fe480ff9700 1 -- 172.21.15.102:0/414743268 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4740036a0 con 0x7fe4840cd8e0 2023-03-03T13:37:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.333+0000 7fe480ff9700 1 -- 172.21.15.102:0/414743268 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474003a10 con 0x7fe4840cd8e0 2023-03-03T13:37:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.333+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4840ccde0 con 0x7fe4840cd8e0 2023-03-03T13:37:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.333+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48415d160 con 0x7fe4840cd8e0 2023-03-03T13:37:45.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.334+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4840cd0c0 con 0x7fe4840cd8e0 2023-03-03T13:37:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.335+0000 7fe480ff9700 1 -- 172.21.15.102:0/414743268 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe47402d070 con 0x7fe4840cd8e0 2023-03-03T13:37:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.336+0000 7fe480ff9700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.336+0000 7fe480ff9700 1 -- 172.21.15.102:0/414743268 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe474021030 con 0x7fe4840cd8e0 2023-03-03T13:37:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.337+0000 7fe4837fe700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:45.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.337+0000 7fe4837fe700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe4840cf820 tx=0x7fe478009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.341+0000 7fe480ff9700 1 -- 172.21.15.102:0/414743268 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe474053b10 con 0x7fe4840cd8e0 2023-03-03T13:37:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.577+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe48404e6e0 con 0x7fe4840cd8e0 2023-03-03T13:37:45.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.578+0000 7fe480ff9700 1 -- 172.21.15.102:0/414743268 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe474015090 con 0x7fe4840cd8e0 2023-03-03T13:37:45.581 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:37:45.581 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:37:45.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 msgr2=0x7fe47006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe4840cf820 tx=0x7fe478009210 comp rx=0 tx=0).stop 2023-03-03T13:37:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4840cd8e0 msgr2=0x7fe48414e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4840cd8e0 0x7fe48414e550 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe474009c10 tx=0x7fe474004750 comp rx=0 tx=0).stop 2023-03-03T13:37:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 shutdown_connections 2023-03-03T13:37:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4841593c0 0x7fe4841e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4840cd8e0 0x7fe48414e550 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 --2- 172.21.15.102:0/414743268 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe484153290 0x7fe4841e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 >> 172.21.15.102:0/414743268 conn(0x7fe484065fb0 msgr2=0x7fe48415ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:37:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 shutdown_connections 2023-03-03T13:37:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:37:45.581+0000 7fe48a3ff700 1 -- 172.21.15.102:0/414743268 wait complete. 2023-03-03T13:37:45.585 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:37:45.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T13:37:52.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:53.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:23.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:23.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:53.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:53.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:54.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:38:54.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.864+0000 7f04cb59e700 1 Processor -- start 2023-03-03T13:38:54.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.865+0000 7f04cb59e700 1 -- start start 2023-03-03T13:38:54.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.866+0000 7f04cb59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0b3970 0x7f04cc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:38:54.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.866+0000 7f04cb59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0c3d20 con 0x7f04cc0c41b0 2023-03-03T13:38:54.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.866+0000 7f04cb59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc15cba0 con 0x7f04cc0ced00 2023-03-03T13:38:54.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.866+0000 7f04cb59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0c4930 con 0x7f04cc0b3970 2023-03-03T13:38:54.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.866+0000 7f04c3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0b3970 0x7f04cc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:38:54.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04ca59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f04cc0ced00 0x7f04cc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:58254/0 (socket says 172.21.15.102:58254) 2023-03-03T13:38:54.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04ca59c700 1 -- 172.21.15.102:0/1426883240 learned_addr learned my addr 172.21.15.102:0/1426883240 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:38:54.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04c9d9b700 1 -- 172.21.15.102:0/1426883240 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032866959 0 0) 0x7f04cc15cba0 con 0x7f04cc0ced00 2023-03-03T13:38:54.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04c9d9b700 1 -- 172.21.15.102:0/1426883240 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b80035b0 con 0x7f04cc0ced00 2023-03-03T13:38:54.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04c3fff700 1 -- 172.21.15.102:0/1426883240 >> v1:172.21.15.172:6789/0 conn(0x7f04cc0ced00 legacy=0x7f04cc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:38:54.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04c3fff700 1 -- 172.21.15.102:0/1426883240 >> v1:172.21.15.102:6789/0 conn(0x7f04cc0c41b0 legacy=0x7f04cc163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:38:54.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04c3fff700 1 -- 172.21.15.102:0/1426883240 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc15ce80 con 0x7f04cc0b3970 2023-03-03T13:38:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.867+0000 7f04c3fff700 1 --2- 172.21.15.102:0/1426883240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0b3970 0x7f04cc0c5a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009870 tx=0x7f04b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed248a543f67414 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:38:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04c9d9b700 1 -- 172.21.15.102:0/1426883240 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4011830 con 0x7f04cc0b3970 2023-03-03T13:38:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04c9d9b700 1 -- 172.21.15.102:0/1426883240 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b4011990 con 0x7f04cc0b3970 2023-03-03T13:38:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04c9d9b700 1 -- 172.21.15.102:0/1426883240 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4011d00 con 0x7f04cc0b3970 2023-03-03T13:38:54.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04cb59e700 1 -- 172.21.15.102:0/1426883240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0b3970 msgr2=0x7f04cc0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:38:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04cb59e700 1 --2- 172.21.15.102:0/1426883240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0b3970 0x7f04cc0c5a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009870 tx=0x7f04b4009b80 comp rx=0 tx=0).stop 2023-03-03T13:38:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04cb59e700 1 -- 172.21.15.102:0/1426883240 shutdown_connections 2023-03-03T13:38:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04cb59e700 1 --2- 172.21.15.102:0/1426883240 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0b3970 0x7f04cc0c5a90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:38:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.868+0000 7f04cb59e700 1 -- 172.21.15.102:0/1426883240 >> 172.21.15.102:0/1426883240 conn(0x7f04cc08b1c0 msgr2=0x7f04cc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:38:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.869+0000 7f04cb59e700 1 -- 172.21.15.102:0/1426883240 shutdown_connections 2023-03-03T13:38:54.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.869+0000 7f04cb59e700 1 -- 172.21.15.102:0/1426883240 wait complete. 2023-03-03T13:38:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.871+0000 7f04cb59e700 1 Processor -- start 2023-03-03T13:38:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.871+0000 7f04cb59e700 1 -- start start 2023-03-03T13:38:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04cb59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04cc0b3970 0x7f04cc14c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:38:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04cb59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0c41b0 0x7f04cc151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:38:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04cb59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc0ced00 0x7f04cc14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:38:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04cb59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc14d8a0 con 0x7f04cc0ced00 2023-03-03T13:38:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04cb59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc1f1920 con 0x7f04cc0b3970 2023-03-03T13:38:54.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04cb59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc1f21c0 con 0x7f04cc0c41b0 2023-03-03T13:38:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04ca59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04cc0b3970 0x7f04cc14c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:38:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04c3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0c41b0 0x7f04cc151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:38:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04cad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc0ced00 0x7f04cc14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:38:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04c3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0c41b0 0x7f04cc151b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:49804/0 (socket says 172.21.15.102:49804) 2023-03-03T13:38:54.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.872+0000 7f04c3fff700 1 -- 172.21.15.102:0/2083811646 learned_addr learned my addr 172.21.15.102:0/2083811646 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:38:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c3fff700 1 -- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04cc0b3970 msgr2=0x7f04cc14c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:38:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c3fff700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04cc0b3970 0x7f04cc14c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:38:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c3fff700 1 -- 172.21.15.102:0/2083811646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc0ced00 msgr2=0x7f04cc14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:38:54.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c3fff700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc0ced00 0x7f04cc14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:38:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c3fff700 1 -- 172.21.15.102:0/2083811646 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40094a0 con 0x7f04cc0c41b0 2023-03-03T13:38:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c3fff700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0c41b0 0x7f04cc151b10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04b40097c0 tx=0x7f04b4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:38:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c2ffd700 1 -- 172.21.15.102:0/2083811646 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4004550 con 0x7f04cc0c41b0 2023-03-03T13:38:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c2ffd700 1 -- 172.21.15.102:0/2083811646 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b401e070 con 0x7f04cc0c41b0 2023-03-03T13:38:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.873+0000 7f04c2ffd700 1 -- 172.21.15.102:0/2083811646 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40038e0 con 0x7f04cc0c41b0 2023-03-03T13:38:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.874+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc06c200 con 0x7f04cc0c41b0 2023-03-03T13:38:54.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.874+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc06bf20 con 0x7f04cc0c41b0 2023-03-03T13:38:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.875+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc15c890 con 0x7f04cc0c41b0 2023-03-03T13:38:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.876+0000 7f04c2ffd700 1 -- 172.21.15.102:0/2083811646 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f04b400f050 con 0x7f04cc0c41b0 2023-03-03T13:38:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.877+0000 7f04c2ffd700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04ac068370 0x7f04ac06a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:38:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.877+0000 7f04c2ffd700 1 -- 172.21.15.102:0/2083811646 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f04b4086bf0 con 0x7f04cc0c41b0 2023-03-03T13:38:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.878+0000 7f04ca59c700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04ac068370 0x7f04ac06a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:38:54.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.878+0000 7f04ca59c700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04ac068370 0x7f04ac06a820 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f04bc007fd0 tx=0x7f04bc007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:38:54.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:54.882+0000 7f04c2ffd700 1 -- 172.21.15.102:0/2083811646 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04b4053380 con 0x7f04cc0c41b0 2023-03-03T13:38:55.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.116+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04cc04e6e0 con 0x7f04cc0c41b0 2023-03-03T13:38:55.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.118+0000 7f04c2ffd700 1 -- 172.21.15.102:0/2083811646 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f04b4020090 con 0x7f04cc0c41b0 2023-03-03T13:38:55.119 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:38:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:38:55.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04ac068370 msgr2=0x7f04ac06a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:38:55.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04ac068370 0x7f04ac06a820 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f04bc007fd0 tx=0x7f04bc007d80 comp rx=0 tx=0).stop 2023-03-03T13:38:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0c41b0 msgr2=0x7f04cc151b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:38:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0c41b0 0x7f04cc151b10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04b40097c0 tx=0x7f04b4004750 comp rx=0 tx=0).stop 2023-03-03T13:38:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 shutdown_connections 2023-03-03T13:38:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc0ced00 0x7f04cc14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:38:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04cc0b3970 0x7f04cc14c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:38:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04ac068370 0x7f04ac06a820 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:38:55.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 --2- 172.21.15.102:0/2083811646 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04cc0c41b0 0x7f04cc151b10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:38:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 >> 172.21.15.102:0/2083811646 conn(0x7f04cc08b1c0 msgr2=0x7f04cc06b190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:38:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 shutdown_connections 2023-03-03T13:38:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:38:55.121+0000 7f04cb59e700 1 -- 172.21.15.102:0/2083811646 wait complete. 2023-03-03T13:38:55.124 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:38:55.149 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 41 secs before thrashing 2023-03-03T13:39:23.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:23.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:36.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:39:36.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.415+0000 7f57c119d700 1 Processor -- start 2023-03-03T13:39:36.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.416+0000 7f57c119d700 1 -- start start 2023-03-03T13:39:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.417+0000 7f57c119d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f57bc0cab50 0x7f57bc1646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.417+0000 7f57c119d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc0c9ca0 con 0x7f57bc0c9610 2023-03-03T13:39:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.417+0000 7f57c119d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc0b46f0 con 0x7f57bc1564b0 2023-03-03T13:39:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.417+0000 7f57c119d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc0b4920 con 0x7f57bc0cab50 2023-03-03T13:39:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.417+0000 7f57bb7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f57bc0c9610 0x7f57bc0ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42892/0 (socket says 172.21.15.102:42892) 2023-03-03T13:39:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.417+0000 7f57bb7fe700 1 -- 172.21.15.102:0/707066780 learned_addr learned my addr 172.21.15.102:0/707066780 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:39:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.417+0000 7f57c099c700 1 --2- 172.21.15.102:0/707066780 >> v2:172.21.15.172:3300/0 conn(0x7f57bc0cab50 0x7f57bc1646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.418+0000 7f57baffd700 1 -- 172.21.15.102:0/707066780 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228905607 0 0) 0x7f57bc0c9ca0 con 0x7f57bc0c9610 2023-03-03T13:39:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.418+0000 7f57baffd700 1 -- 172.21.15.102:0/707066780 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a40035b0 con 0x7f57bc0c9610 2023-03-03T13:39:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.418+0000 7f57c099c700 1 -- 172.21.15.102:0/707066780 >> v1:172.21.15.172:6789/0 conn(0x7f57bc1564b0 legacy=0x7f57bc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.418+0000 7f57baffd700 1 -- 172.21.15.102:0/707066780 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141710731 0 0) 0x7f57bc0b46f0 con 0x7f57bc1564b0 2023-03-03T13:39:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.418+0000 7f57c099c700 1 -- 172.21.15.102:0/707066780 >> v1:172.21.15.102:6789/0 conn(0x7f57bc0c9610 legacy=0x7f57bc0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.418+0000 7f57c099c700 1 -- 172.21.15.102:0/707066780 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc0b4c00 con 0x7f57bc0cab50 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.418+0000 7f57c099c700 1 --2- 172.21.15.102:0/707066780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0cab50 0x7f57bc1646b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009d90 tx=0x7f57b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9c5ce7fcbdcb148 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57baffd700 1 -- 172.21.15.102:0/707066780 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b400e070 con 0x7f57bc0cab50 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57baffd700 1 -- 172.21.15.102:0/707066780 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b400bb20 con 0x7f57bc0cab50 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57baffd700 1 -- 172.21.15.102:0/707066780 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b400bed0 con 0x7f57bc0cab50 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57c119d700 1 -- 172.21.15.102:0/707066780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0cab50 msgr2=0x7f57bc1646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57c119d700 1 --2- 172.21.15.102:0/707066780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0cab50 0x7f57bc1646b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009d90 tx=0x7f57b400b330 comp rx=0 tx=0).stop 2023-03-03T13:39:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57c119d700 1 -- 172.21.15.102:0/707066780 shutdown_connections 2023-03-03T13:39:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57c119d700 1 --2- 172.21.15.102:0/707066780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0cab50 0x7f57bc1646b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57c119d700 1 -- 172.21.15.102:0/707066780 >> 172.21.15.102:0/707066780 conn(0x7f57bc065fb0 msgr2=0x7f57bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:39:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57c119d700 1 -- 172.21.15.102:0/707066780 shutdown_connections 2023-03-03T13:39:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.419+0000 7f57c119d700 1 -- 172.21.15.102:0/707066780 wait complete. 2023-03-03T13:39:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.421+0000 7f57c119d700 1 Processor -- start 2023-03-03T13:39:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.422+0000 7f57c119d700 1 -- start start 2023-03-03T13:39:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57c119d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0c9610 0x7f57bc15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57c119d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57bc0cab50 0x7f57bc15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57c119d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57bc1564b0 0x7f57bc15ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57c119d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc15b240 con 0x7f57bc0cab50 2023-03-03T13:39:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57c119d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc15aef0 con 0x7f57bc0c9610 2023-03-03T13:39:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57c119d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc15a000 con 0x7f57bc1564b0 2023-03-03T13:39:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57bb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57bc0cab50 0x7f57bc15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0c9610 0x7f57bc15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57bbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0c9610 0x7f57bc15bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38928/0 (socket says 172.21.15.102:38928) 2023-03-03T13:39:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57bbfff700 1 -- 172.21.15.102:0/3348467006 learned_addr learned my addr 172.21.15.102:0/3348467006 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:39:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57bb7fe700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57bc0cab50 0x7f57bc15c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37082/0 (socket says 172.21.15.102:37082) 2023-03-03T13:39:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.423+0000 7f57c099c700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57bc1564b0 0x7f57bc15ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57bbfff700 1 -- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57bc1564b0 msgr2=0x7f57bc15ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57bbfff700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57bc1564b0 0x7f57bc15ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57bbfff700 1 -- 172.21.15.102:0/3348467006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57bc0cab50 msgr2=0x7f57bc15c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57bbfff700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57bc0cab50 0x7f57bc15c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57bbfff700 1 -- 172.21.15.102:0/3348467006 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b4009a70 con 0x7f57bc0c9610 2023-03-03T13:39:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57bb7fe700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57bc0cab50 0x7f57bc15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:39:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57bbfff700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0c9610 0x7f57bc15bf70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0011e0 tx=0x7f57ac00ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57b8ff9700 1 -- 172.21.15.102:0/3348467006 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0054e0 con 0x7f57bc0c9610 2023-03-03T13:39:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57b8ff9700 1 -- 172.21.15.102:0/3348467006 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac003680 con 0x7f57bc0c9610 2023-03-03T13:39:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57b8ff9700 1 -- 172.21.15.102:0/3348467006 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac003a30 con 0x7f57bc0c9610 2023-03-03T13:39:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.424+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc0b2210 con 0x7f57bc0c9610 2023-03-03T13:39:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.425+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc0b1f30 con 0x7f57bc0c9610 2023-03-03T13:39:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.425+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc156880 con 0x7f57bc0c9610 2023-03-03T13:39:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.427+0000 7f57b8ff9700 1 -- 172.21.15.102:0/3348467006 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57ac019030 con 0x7f57bc0c9610 2023-03-03T13:39:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.428+0000 7f57b8ff9700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a8068320 0x7f57a806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.428+0000 7f57b8ff9700 1 -- 172.21.15.102:0/3348467006 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f57ac085130 con 0x7f57bc0c9610 2023-03-03T13:39:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.428+0000 7f57bb7fe700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a8068320 0x7f57a806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.429+0000 7f57bb7fe700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a8068320 0x7f57a806a7d0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0c5cf0 tx=0x7f57b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.432+0000 7f57b8ff9700 1 -- 172.21.15.102:0/3348467006 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57ac0517b0 con 0x7f57bc0c9610 2023-03-03T13:39:36.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.671+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57bc0677f0 con 0x7f57bc0c9610 2023-03-03T13:39:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.672+0000 7f57b8ff9700 1 -- 172.21.15.102:0/3348467006 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f57ac017020 con 0x7f57bc0c9610 2023-03-03T13:39:36.674 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:39:36.674 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:39:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a8068320 msgr2=0x7f57a806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a8068320 0x7f57a806a7d0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0c5cf0 tx=0x7f57b0009040 comp rx=0 tx=0).stop 2023-03-03T13:39:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0c9610 msgr2=0x7f57bc15bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0c9610 0x7f57bc15bf70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0011e0 tx=0x7f57ac00ca30 comp rx=0 tx=0).stop 2023-03-03T13:39:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 shutdown_connections 2023-03-03T13:39:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f57a8068320 0x7f57a806a7d0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f57bc1564b0 0x7f57bc15ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:36.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f57bc0c9610 0x7f57bc15bf70 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 --2- 172.21.15.102:0/3348467006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57bc0cab50 0x7f57bc15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 >> 172.21.15.102:0/3348467006 conn(0x7f57bc065fb0 msgr2=0x7f57bc066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:39:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 shutdown_connections 2023-03-03T13:39:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:39:36.675+0000 7f57c119d700 1 -- 172.21.15.102:0/3348467006 wait complete. 2023-03-03T13:39:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:39:36.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-03-03T13:39:53.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:53.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T13:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:40:23.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:23.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:49.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:40:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.953+0000 7f509d59d700 1 Processor -- start 2023-03-03T13:40:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.953+0000 7f509d59d700 1 -- start start 2023-03-03T13:40:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f509d59d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f50980b3970 0x7f50980c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f509d59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980c41b0 0x7f50980cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f509d59d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50980c3d20 con 0x7f50980ced00 2023-03-03T13:40:49.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f509d59d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50980c4930 con 0x7f50980b3970 2023-03-03T13:40:49.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f509d59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50980cf610 con 0x7f50980c41b0 2023-03-03T13:40:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f5097fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f50980ced00 0x7f5098158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37384/0 (socket says 172.21.15.102:37384) 2023-03-03T13:40:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f5097fff700 1 -- 172.21.15.102:0/2064903855 learned_addr learned my addr 172.21.15.102:0/2064903855 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:40:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f50977fe700 1 --2- 172.21.15.102:0/2064903855 >> v2:172.21.15.172:3300/0 conn(0x7f50980b3970 0x7f50980c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f509cd9c700 1 --2- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980c41b0 0x7f50980cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.955+0000 7f5096ffd700 1 -- 172.21.15.102:0/2064903855 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823980091 0 0) 0x7f50980c3d20 con 0x7f50980ced00 2023-03-03T13:40:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f50977fe700 1 -- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980c41b0 msgr2=0x7f50980cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:40:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f50977fe700 1 --2- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980c41b0 0x7f50980cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f50977fe700 1 -- 172.21.15.102:0/2064903855 >> v1:172.21.15.102:6789/0 conn(0x7f50980ced00 legacy=0x7f5098158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f50977fe700 1 -- 172.21.15.102:0/2064903855 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509815cba0 con 0x7f50980b3970 2023-03-03T13:40:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f50977fe700 1 --2- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980b3970 0x7f50980c5a90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f508c009870 tx=0x7f508c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c7ed50a83e316a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f5096ffd700 1 -- 172.21.15.102:0/2064903855 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c011830 con 0x7f50980b3970 2023-03-03T13:40:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f5096ffd700 1 -- 172.21.15.102:0/2064903855 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c011990 con 0x7f50980b3970 2023-03-03T13:40:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.956+0000 7f5096ffd700 1 -- 172.21.15.102:0/2064903855 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c011d20 con 0x7f50980b3970 2023-03-03T13:40:49.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 -- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980b3970 msgr2=0x7f50980c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:49.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 --2- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980b3970 0x7f50980c5a90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f508c009870 tx=0x7f508c009b80 comp rx=0 tx=0).stop 2023-03-03T13:40:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 -- 172.21.15.102:0/2064903855 shutdown_connections 2023-03-03T13:40:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 --2- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980c41b0 0x7f50980cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 --2- 172.21.15.102:0/2064903855 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980b3970 0x7f50980c5a90 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 -- 172.21.15.102:0/2064903855 >> 172.21.15.102:0/2064903855 conn(0x7f509808b1c0 msgr2=0x7f5098071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 -- 172.21.15.102:0/2064903855 shutdown_connections 2023-03-03T13:40:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.957+0000 7f509d59d700 1 -- 172.21.15.102:0/2064903855 wait complete. 2023-03-03T13:40:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.959+0000 7f509d59d700 1 Processor -- start 2023-03-03T13:40:49.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.959+0000 7f509d59d700 1 -- start start 2023-03-03T13:40:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f509d59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980b3970 0x7f50981500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f509d59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980c41b0 0x7f5098150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f509d59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50980ced00 0x7f5098159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f509d59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509814c8c0 con 0x7f50980ced00 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f509d59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50981e5c60 con 0x7f50980c41b0 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f509d59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50981511e0 con 0x7f50980b3970 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f50977fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980c41b0 0x7f5098150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f5097fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980b3970 0x7f50981500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f50977fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980c41b0 0x7f5098150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42092/0 (socket says 172.21.15.102:42092) 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f50977fe700 1 -- 172.21.15.102:0/842297226 learned_addr learned my addr 172.21.15.102:0/842297226 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.960+0000 7f509cd9c700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50980ced00 0x7f5098159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.961+0000 7f50977fe700 1 -- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980b3970 msgr2=0x7f50981500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:49.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.961+0000 7f50977fe700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980b3970 0x7f50981500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:49.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.961+0000 7f50977fe700 1 -- 172.21.15.102:0/842297226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50980ced00 msgr2=0x7f5098159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:49.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.961+0000 7f50977fe700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50980ced00 0x7f5098159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:49.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.961+0000 7f50977fe700 1 -- 172.21.15.102:0/842297226 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c0094a0 con 0x7f50980c41b0 2023-03-03T13:40:49.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.961+0000 7f50977fe700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980c41b0 0x7f5098150620 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f508c014040 tx=0x7f508c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:49.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.961+0000 7f5094ff9700 1 -- 172.21.15.102:0/842297226 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c0036a0 con 0x7f50980c41b0 2023-03-03T13:40:49.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.962+0000 7f5094ff9700 1 -- 172.21.15.102:0/842297226 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c01e070 con 0x7f50980c41b0 2023-03-03T13:40:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.962+0000 7f5094ff9700 1 -- 172.21.15.102:0/842297226 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c003a00 con 0x7f50980c41b0 2023-03-03T13:40:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.962+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509806bf20 con 0x7f50980c41b0 2023-03-03T13:40:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.962+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509815ce80 con 0x7f50980c41b0 2023-03-03T13:40:49.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.963+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509806c200 con 0x7f50980c41b0 2023-03-03T13:40:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.964+0000 7f5094ff9700 1 -- 172.21.15.102:0/842297226 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f508c00f050 con 0x7f50980c41b0 2023-03-03T13:40:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.965+0000 7f5094ff9700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50840682f0 0x7f508406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.965+0000 7f5094ff9700 1 -- 172.21.15.102:0/842297226 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f508c086c90 con 0x7f50980c41b0 2023-03-03T13:40:49.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.966+0000 7f5097fff700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50840682f0 0x7f508406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:49.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.966+0000 7f5097fff700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50840682f0 0x7f508406a7a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5088007fd0 tx=0x7f5088007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:49.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:49.970+0000 7f5094ff9700 1 -- 172.21.15.102:0/842297226 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f508c0533a0 con 0x7f50980c41b0 2023-03-03T13:40:50.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.207+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f509814fe00 con 0x7f50980c41b0 2023-03-03T13:40:50.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.208+0000 7f5094ff9700 1 -- 172.21.15.102:0/842297226 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f509814fe00 con 0x7f50980c41b0 2023-03-03T13:40:50.209 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:40:50.210 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:40:50.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50840682f0 msgr2=0x7f508406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:50.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50840682f0 0x7f508406a7a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5088007fd0 tx=0x7f5088007de0 comp rx=0 tx=0).stop 2023-03-03T13:40:50.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980c41b0 msgr2=0x7f5098150620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:50.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980c41b0 0x7f5098150620 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f508c014040 tx=0x7f508c004750 comp rx=0 tx=0).stop 2023-03-03T13:40:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 shutdown_connections 2023-03-03T13:40:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50840682f0 0x7f508406a7a0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50980b3970 0x7f50981500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50980ced00 0x7f5098159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 --2- 172.21.15.102:0/842297226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50980c41b0 0x7f5098150620 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.210+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 >> 172.21.15.102:0/842297226 conn(0x7f509808b1c0 msgr2=0x7f50981e1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.211+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 shutdown_connections 2023-03-03T13:40:50.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:50.211+0000 7f509d59d700 1 -- 172.21.15.102:0/842297226 wait complete. 2023-03-03T13:40:50.214 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:40:50.240 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-03-03T13:40:54.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:54.246 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:55.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:40:55.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.487+0000 7f6ed79ff700 1 Processor -- start 2023-03-03T13:40:55.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.487+0000 7f6ed79ff700 1 -- start start 2023-03-03T13:40:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.488+0000 7f6ed79ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.488+0000 7f6ed79ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ed01593c0 0x7f6ed00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed79ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6ed00cd8e0 0x7f6ed00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:55.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed79ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed0153660 con 0x7f6ed01593c0 2023-03-03T13:40:55.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed79ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed006c360 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed79ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed015cea0 con 0x7f6ed0153290 2023-03-03T13:40:55.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed4f9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ed01593c0 0x7f6ed00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed5f9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6ed00cd8e0 0x7f6ed00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed4f9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ed01593c0 0x7f6ed00cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52904/0 (socket says 172.21.15.102:52904) 2023-03-03T13:40:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed4f9a700 1 -- 172.21.15.102:0/1494340542 learned_addr learned my addr 172.21.15.102:0/1494340542 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:40:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed579b700 1 --2- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed5f9c700 1 -- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 msgr2=0x7f6ed0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:40:55.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed5f9c700 1 --2- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed0158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed5f9c700 1 -- 172.21.15.102:0/1494340542 >> v2:172.21.15.102:3300/0 conn(0x7f6ed01593c0 msgr2=0x7f6ed00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:40:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed5f9c700 1 --2- 172.21.15.102:0/1494340542 >> v2:172.21.15.102:3300/0 conn(0x7f6ed01593c0 0x7f6ed00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed5f9c700 1 -- 172.21.15.102:0/1494340542 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed006c7f0 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.489+0000 7f6ed4f9a700 1 --2- 172.21.15.102:0/1494340542 >> v2:172.21.15.102:3300/0 conn(0x7f6ed01593c0 0x7f6ed00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:40:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.490+0000 7f6ed5f9c700 1 --2- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00ce360 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6ed006d0e0 tx=0x7f6ec800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1df162372d092ae9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.490+0000 7f6ec7fff700 1 -- 172.21.15.102:0/1494340542 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec800e070 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.490+0000 7f6ec7fff700 1 -- 172.21.15.102:0/1494340542 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec800bb20 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.490+0000 7f6ec7fff700 1 -- 172.21.15.102:0/1494340542 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8012430 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 -- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 msgr2=0x7f6ed00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00ce360 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6ed006d0e0 tx=0x7f6ec800b330 comp rx=0 tx=0).stop 2023-03-03T13:40:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 -- 172.21.15.102:0/1494340542 shutdown_connections 2023-03-03T13:40:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00ce360 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/1494340542 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed0158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/1494340542 >> v2:172.21.15.102:3300/0 conn(0x7f6ed01593c0 0x7f6ed00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 -- 172.21.15.102:0/1494340542 >> 172.21.15.102:0/1494340542 conn(0x7f6ed0065fb0 msgr2=0x7f6ed00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 -- 172.21.15.102:0/1494340542 shutdown_connections 2023-03-03T13:40:55.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.491+0000 7f6ed79ff700 1 -- 172.21.15.102:0/1494340542 wait complete. 2023-03-03T13:40:55.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.493+0000 7f6ed79ff700 1 Processor -- start 2023-03-03T13:40:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.493+0000 7f6ed79ff700 1 -- start start 2023-03-03T13:40:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.494+0000 7f6ed79ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.494+0000 7f6ed79ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed01541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.494+0000 7f6ed79ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed01593c0 0x7f6ed00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.494+0000 7f6ed79ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00c7630 con 0x7f6ed01593c0 2023-03-03T13:40:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.494+0000 7f6ed79ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed01ef4b0 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.494+0000 7f6ed79ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00b1df0 con 0x7f6ed0153290 2023-03-03T13:40:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed5f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed01593c0 0x7f6ed00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed4f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed01541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42120/0 (socket says 172.21.15.102:42120) 2023-03-03T13:40:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 -- 172.21.15.102:0/276029118 learned_addr learned my addr 172.21.15.102:0/276029118 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:40:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 -- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 msgr2=0x7f6ed01541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed01541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 -- 172.21.15.102:0/276029118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed01593c0 msgr2=0x7f6ed00cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed01593c0 0x7f6ed00cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 -- 172.21.15.102:0/276029118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec0009550 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.495+0000 7f6ed579b700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00c7ac0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00e8a0 tx=0x7f6ecc00ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.496+0000 7f6ec5ffb700 1 -- 172.21.15.102:0/276029118 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc009590 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.496+0000 7f6ec5ffb700 1 -- 172.21.15.102:0/276029118 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc013680 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.496+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec8009a70 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.496+0000 7f6ec5ffb700 1 -- 172.21.15.102:0/276029118 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc0139f0 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.496+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed00ccde0 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.497+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed015d160 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.498+0000 7f6ec5ffb700 1 -- 172.21.15.102:0/276029118 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6ecc023070 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.499+0000 7f6ec5ffb700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eb40682f0 0x7f6eb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.499+0000 7f6ec5ffb700 1 -- 172.21.15.102:0/276029118 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6ecc085df0 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.500+0000 7f6ed4f9a700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eb40682f0 0x7f6eb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.500+0000 7f6ed4f9a700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eb40682f0 0x7f6eb406a7a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006d30 tx=0x7f6ec00092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.502+0000 7f6ec5ffb700 1 -- 172.21.15.102:0/276029118 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ecc052520 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.737+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ed00cd0c0 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.739+0000 7f6ec5ffb700 1 -- 172.21.15.102:0/276029118 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6ecc017090 con 0x7f6ed00cd8e0 2023-03-03T13:40:55.741 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:40:55.741 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:40:55.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eb40682f0 msgr2=0x7f6eb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:55.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eb40682f0 0x7f6eb406a7a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006d30 tx=0x7f6ec00092c0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 msgr2=0x7f6ed00c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:55.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00c7ac0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00e8a0 tx=0x7f6ecc00ebb0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 shutdown_connections 2023-03-03T13:40:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed01593c0 0x7f6ed00cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed00cd8e0 0x7f6ed00c7ac0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eb40682f0 0x7f6eb406a7a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 --2- 172.21.15.102:0/276029118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6ed0153290 0x7f6ed01541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 >> 172.21.15.102:0/276029118 conn(0x7f6ed0065fb0 msgr2=0x7f6ed015b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.741+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 shutdown_connections 2023-03-03T13:40:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:40:55.742+0000 7f6ed79ff700 1 -- 172.21.15.102:0/276029118 wait complete. 2023-03-03T13:40:55.746 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:40:55.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T13:41:24.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:24.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:54.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:54.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:22.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:42:23.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.037+0000 7f5b77fff700 1 Processor -- start 2023-03-03T13:42:23.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.037+0000 7f5b77fff700 1 -- start start 2023-03-03T13:42:23.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b77fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b70153290 0x7f5b70158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:23.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b77fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5b700cd8e0 0x7f5b700ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:23.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b77fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b70153660 con 0x7f5b701593c0 2023-03-03T13:42:23.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b77fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7006c360 con 0x7f5b700cd8e0 2023-03-03T13:42:23.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b77fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7015cea0 con 0x7f5b70153290 2023-03-03T13:42:23.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b75d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b70153290 0x7f5b70158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:23.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b75d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b70153290 0x7f5b70158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:32964/0 (socket says 172.21.15.102:32964) 2023-03-03T13:42:23.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b75d9b700 1 -- 172.21.15.102:0/2750519617 learned_addr learned my addr 172.21.15.102:0/2750519617 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:42:23.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b7559a700 1 --1- 172.21.15.102:0/2750519617 >> v1:172.21.15.102:6789/0 conn(0x7f5b701593c0 0x7f5b700cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45810/0 (socket says 172.21.15.102:45810) 2023-03-03T13:42:23.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.039+0000 7f5b7659c700 1 --2- 172.21.15.102:0/2750519617 >> v2:172.21.15.102:3300/0 conn(0x7f5b700cd8e0 0x7f5b700ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:23.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.040+0000 7f5b75d9b700 1 -- 172.21.15.102:0/2750519617 >> v2:172.21.15.102:3300/0 conn(0x7f5b700cd8e0 msgr2=0x7f5b700ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:42:23.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.040+0000 7f5b75d9b700 1 --2- 172.21.15.102:0/2750519617 >> v2:172.21.15.102:3300/0 conn(0x7f5b700cd8e0 0x7f5b700ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.040+0000 7f5b75d9b700 1 -- 172.21.15.102:0/2750519617 >> v1:172.21.15.102:6789/0 conn(0x7f5b701593c0 legacy=0x7f5b700cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:23.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.040+0000 7f5b75d9b700 1 -- 172.21.15.102:0/2750519617 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7006c7f0 con 0x7f5b70153290 2023-03-03T13:42:23.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.040+0000 7f5b74d99700 1 -- 172.21.15.102:0/2750519617 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662933921 0 0) 0x7f5b70153660 con 0x7f5b701593c0 2023-03-03T13:42:23.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.040+0000 7f5b7659c700 1 --2- 172.21.15.102:0/2750519617 >> v2:172.21.15.102:3300/0 conn(0x7f5b700cd8e0 0x7f5b700ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:42:23.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.040+0000 7f5b75d9b700 1 --2- 172.21.15.102:0/2750519617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b70153290 0x7f5b70158e80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5b7008cc70 tx=0x7f5b6c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9013c444432828a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:42:23.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b74d99700 1 -- 172.21.15.102:0/2750519617 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c011830 con 0x7f5b70153290 2023-03-03T13:42:23.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b74d99700 1 -- 172.21.15.102:0/2750519617 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c011990 con 0x7f5b70153290 2023-03-03T13:42:23.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b74d99700 1 -- 172.21.15.102:0/2750519617 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c011d20 con 0x7f5b70153290 2023-03-03T13:42:23.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 -- 172.21.15.102:0/2750519617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b70153290 msgr2=0x7f5b70158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:23.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 --2- 172.21.15.102:0/2750519617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b70153290 0x7f5b70158e80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5b7008cc70 tx=0x7f5b6c009bd0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 -- 172.21.15.102:0/2750519617 shutdown_connections 2023-03-03T13:42:23.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 --2- 172.21.15.102:0/2750519617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b70153290 0x7f5b70158e80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 --2- 172.21.15.102:0/2750519617 >> v2:172.21.15.102:3300/0 conn(0x7f5b700cd8e0 0x7f5b700ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 -- 172.21.15.102:0/2750519617 >> 172.21.15.102:0/2750519617 conn(0x7f5b70065fb0 msgr2=0x7f5b700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:42:23.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 -- 172.21.15.102:0/2750519617 shutdown_connections 2023-03-03T13:42:23.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.041+0000 7f5b77fff700 1 -- 172.21.15.102:0/2750519617 wait complete. 2023-03-03T13:42:23.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.043+0000 7f5b77fff700 1 Processor -- start 2023-03-03T13:42:23.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.043+0000 7f5b77fff700 1 -- start start 2023-03-03T13:42:23.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.044+0000 7f5b77fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b700cd8e0 0x7f5b700cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.044+0000 7f5b77fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b70153290 0x7f5b7014ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b77fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b701593c0 0x7f5b7014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b77fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7014f9b0 con 0x7f5b70153290 2023-03-03T13:42:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b77fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7014f610 con 0x7f5b701593c0 2023-03-03T13:42:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b77fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700c1c50 con 0x7f5b700cd8e0 2023-03-03T13:42:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b70153290 0x7f5b7014ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b75d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b700cd8e0 0x7f5b700cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b701593c0 0x7f5b7014d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b70153290 0x7f5b7014ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53766/0 (socket says 172.21.15.102:53766) 2023-03-03T13:42:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 -- 172.21.15.102:0/1923788329 learned_addr learned my addr 172.21.15.102:0/1923788329 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:42:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 -- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b700cd8e0 msgr2=0x7f5b700cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b700cd8e0 0x7f5b700cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 -- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b701593c0 msgr2=0x7f5b7014d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b701593c0 0x7f5b7014d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.045+0000 7f5b7559a700 1 -- 172.21.15.102:0/1923788329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b68009a70 con 0x7f5b70153290 2023-03-03T13:42:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b7659c700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b701593c0 0x7f5b7014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:42:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b75d9b700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b700cd8e0 0x7f5b700cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:42:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b7559a700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b70153290 0x7f5b7014ea30 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5b600011e0 tx=0x7f5b6000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:42:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b667fc700 1 -- 172.21.15.102:0/1923788329 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60004c80 con 0x7f5b70153290 2023-03-03T13:42:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b667fc700 1 -- 172.21.15.102:0/1923788329 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b60004de0 con 0x7f5b70153290 2023-03-03T13:42:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b667fc700 1 -- 172.21.15.102:0/1923788329 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60007310 con 0x7f5b70153290 2023-03-03T13:42:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c009550 con 0x7f5b70153290 2023-03-03T13:42:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.046+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b700ccde0 con 0x7f5b70153290 2023-03-03T13:42:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.048+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7015d160 con 0x7f5b70153290 2023-03-03T13:42:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.049+0000 7f5b667fc700 1 -- 172.21.15.102:0/1923788329 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b6001a020 con 0x7f5b70153290 2023-03-03T13:42:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.050+0000 7f5b667fc700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5b540682f0 0x7f5b5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.050+0000 7f5b667fc700 1 -- 172.21.15.102:0/1923788329 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5b600858b0 con 0x7f5b70153290 2023-03-03T13:42:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.050+0000 7f5b75d9b700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5b540682f0 0x7f5b5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.051+0000 7f5b75d9b700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5b540682f0 0x7f5b5406a7a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009c10 tx=0x7f5b6c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:42:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.054+0000 7f5b667fc700 1 -- 172.21.15.102:0/1923788329 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b60051fc0 con 0x7f5b70153290 2023-03-03T13:42:23.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.285+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b700cd0c0 con 0x7f5b70153290 2023-03-03T13:42:23.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.286+0000 7f5b667fc700 1 -- 172.21.15.102:0/1923788329 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5b600555e0 con 0x7f5b70153290 2023-03-03T13:42:23.287 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:42:23.288 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:42:23.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5b540682f0 msgr2=0x7f5b5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:23.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5b540682f0 0x7f5b5406a7a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009c10 tx=0x7f5b6c008030 comp rx=0 tx=0).stop 2023-03-03T13:42:23.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b70153290 msgr2=0x7f5b7014ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:23.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b70153290 0x7f5b7014ea30 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5b600011e0 tx=0x7f5b6000ca30 comp rx=0 tx=0).stop 2023-03-03T13:42:23.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 shutdown_connections 2023-03-03T13:42:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5b540682f0 0x7f5b5406a7a0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5b700cd8e0 0x7f5b700cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b701593c0 0x7f5b7014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 --2- 172.21.15.102:0/1923788329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b70153290 0x7f5b7014ea30 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.288+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 >> 172.21.15.102:0/1923788329 conn(0x7f5b70065fb0 msgr2=0x7f5b700cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:42:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.289+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 shutdown_connections 2023-03-03T13:42:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:42:23.289+0000 7f5b77fff700 1 -- 172.21.15.102:0/1923788329 wait complete. 2023-03-03T13:42:23.292 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:42:23.317 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2023-03-03T13:42:24.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:24.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:55.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:55.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:25.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:25.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:43.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:43:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.566+0000 7f13fb19d700 1 Processor -- start 2023-03-03T13:43:43.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.566+0000 7f13fb19d700 1 -- start start 2023-03-03T13:43:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fb19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13f40b3970 0x7f13f40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:43.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fb19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13f40c41b0 0x7f13f40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fb19d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f40c3d20 con 0x7f13f40ced00 2023-03-03T13:43:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fb19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f40c4930 con 0x7f13f40b3970 2023-03-03T13:43:43.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fb19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f40cf610 con 0x7f13f40c41b0 2023-03-03T13:43:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13f999a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13f40b3970 0x7f13f40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fa19b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f13f40ced00 0x7f13f4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33878/0 (socket says 172.21.15.102:33878) 2023-03-03T13:43:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fa19b700 1 -- 172.21.15.102:0/4196215662 learned_addr learned my addr 172.21.15.102:0/4196215662 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:43:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.568+0000 7f13fa99c700 1 --2- 172.21.15.102:0/4196215662 >> v2:172.21.15.172:3300/0 conn(0x7f13f40c41b0 0x7f13f40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:43.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f999a700 1 -- 172.21.15.102:0/4196215662 >> v2:172.21.15.172:3300/0 conn(0x7f13f40c41b0 msgr2=0x7f13f40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:43:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f999a700 1 --2- 172.21.15.102:0/4196215662 >> v2:172.21.15.172:3300/0 conn(0x7f13f40c41b0 0x7f13f40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f999a700 1 -- 172.21.15.102:0/4196215662 >> v1:172.21.15.102:6789/0 conn(0x7f13f40ced00 legacy=0x7f13f4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f999a700 1 -- 172.21.15.102:0/4196215662 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f415cba0 con 0x7f13f40b3970 2023-03-03T13:43:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f999a700 1 --2- 172.21.15.102:0/4196215662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f40c5a90 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f13f408ce30 tx=0x7f13e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79cf6633d741ec39 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f9199700 1 -- 172.21.15.102:0/4196215662 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4011830 con 0x7f13f40b3970 2023-03-03T13:43:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f9199700 1 -- 172.21.15.102:0/4196215662 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e4011990 con 0x7f13f40b3970 2023-03-03T13:43:43.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.569+0000 7f13f9199700 1 -- 172.21.15.102:0/4196215662 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4011d20 con 0x7f13f40b3970 2023-03-03T13:43:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 -- 172.21.15.102:0/4196215662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 msgr2=0x7f13f40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:43.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 --2- 172.21.15.102:0/4196215662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f40c5a90 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f13f408ce30 tx=0x7f13e4009b50 comp rx=0 tx=0).stop 2023-03-03T13:43:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 -- 172.21.15.102:0/4196215662 shutdown_connections 2023-03-03T13:43:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 --2- 172.21.15.102:0/4196215662 >> v2:172.21.15.172:3300/0 conn(0x7f13f40c41b0 0x7f13f40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 --2- 172.21.15.102:0/4196215662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f40c5a90 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 -- 172.21.15.102:0/4196215662 >> 172.21.15.102:0/4196215662 conn(0x7f13f408b1c0 msgr2=0x7f13f4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 -- 172.21.15.102:0/4196215662 shutdown_connections 2023-03-03T13:43:43.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.570+0000 7f13fb19d700 1 -- 172.21.15.102:0/4196215662 wait complete. 2023-03-03T13:43:43.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.572+0000 7f13fb19d700 1 Processor -- start 2023-03-03T13:43:43.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.572+0000 7f13fb19d700 1 -- start start 2023-03-03T13:43:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.573+0000 7f13fb19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f41f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.573+0000 7f13fb19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13f40c41b0 0x7f13f414cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.573+0000 7f13fb19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13f40ced00 0x7f13f414d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fb19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f415bce0 con 0x7f13f40b3970 2023-03-03T13:43:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fb19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f41f1ba0 con 0x7f13f40ced00 2023-03-03T13:43:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fb19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f414c9d0 con 0x7f13f40c41b0 2023-03-03T13:43:43.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f41f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f41f2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50186/0 (socket says 172.21.15.102:50186) 2023-03-03T13:43:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 -- 172.21.15.102:0/1799394313 learned_addr learned my addr 172.21.15.102:0/1799394313 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:43:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa99c700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13f40ced00 0x7f13f414d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:43.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13f999a700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13f40c41b0 0x7f13f414cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 -- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13f40c41b0 msgr2=0x7f13f414cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13f40c41b0 0x7f13f414cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 -- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13f40ced00 msgr2=0x7f13f414d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:43.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13f40ced00 0x7f13f414d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.574+0000 7f13fa19b700 1 -- 172.21.15.102:0/1799394313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e40094a0 con 0x7f13f40b3970 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.575+0000 7f13fa19b700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f41f2030 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c520 tx=0x7f13f000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.575+0000 7f13eaffd700 1 -- 172.21.15.102:0/1799394313 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0005d70 con 0x7f13f40b3970 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.575+0000 7f13eaffd700 1 -- 172.21.15.102:0/1799394313 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f00070d0 con 0x7f13f40b3970 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.575+0000 7f13eaffd700 1 -- 172.21.15.102:0/1799394313 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0007440 con 0x7f13f40b3970 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.575+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f406bf20 con 0x7f13f40b3970 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.575+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f415ce80 con 0x7f13f40b3970 2023-03-03T13:43:43.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.576+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f406c200 con 0x7f13f40b3970 2023-03-03T13:43:43.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.577+0000 7f13eaffd700 1 -- 172.21.15.102:0/1799394313 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13f0019030 con 0x7f13f40b3970 2023-03-03T13:43:43.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.579+0000 7f13eaffd700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f13d80682f0 0x7f13d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.579+0000 7f13eaffd700 1 -- 172.21.15.102:0/1799394313 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f13f0084f50 con 0x7f13f40b3970 2023-03-03T13:43:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.579+0000 7f13f999a700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f13d80682f0 0x7f13d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:43.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.580+0000 7f13f999a700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f13d80682f0 0x7f13d806a7a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000c00 tx=0x7f13e401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.583+0000 7f13eaffd700 1 -- 172.21.15.102:0/1799394313 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13f004b5e0 con 0x7f13f40b3970 2023-03-03T13:43:43.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.817+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13f415a6a0 con 0x7f13f40b3970 2023-03-03T13:43:43.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.818+0000 7f13eaffd700 1 -- 172.21.15.102:0/1799394313 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f13f0017090 con 0x7f13f40b3970 2023-03-03T13:43:43.820 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:43:43.820 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:43:43.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.820+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f13d80682f0 msgr2=0x7f13d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:43.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.820+0000 7f13fb19d700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f13d80682f0 0x7f13d806a7a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000c00 tx=0x7f13e401c040 comp rx=0 tx=0).stop 2023-03-03T13:43:43.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.820+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 msgr2=0x7f13f41f2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:43.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.820+0000 7f13fb19d700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f41f2030 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c520 tx=0x7f13f000c830 comp rx=0 tx=0).stop 2023-03-03T13:43:43.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.820+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 shutdown_connections 2023-03-03T13:43:43.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.820+0000 7f13fb19d700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13f40ced00 0x7f13f414d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.821+0000 7f13fb19d700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13f40b3970 0x7f13f41f2030 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.821+0000 7f13fb19d700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f13d80682f0 0x7f13d806a7a0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.821+0000 7f13fb19d700 1 --2- 172.21.15.102:0/1799394313 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f13f40c41b0 0x7f13f414cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:43.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.821+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 >> 172.21.15.102:0/1799394313 conn(0x7f13f408b1c0 msgr2=0x7f13f4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:43.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.821+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 shutdown_connections 2023-03-03T13:43:43.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:43.821+0000 7f13fb19d700 1 -- 172.21.15.102:0/1799394313 wait complete. 2023-03-03T13:43:43.824 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:43:43.850 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2023-03-03T13:43:49.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:43:50.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.118+0000 7f6f113ff700 1 Processor -- start 2023-03-03T13:43:50.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.118+0000 7f6f113ff700 1 -- start start 2023-03-03T13:43:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.119+0000 7f6f113ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c153290 0x7f6f0c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.119+0000 7f6f113ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f0c1593c0 0x7f6f0c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.119+0000 7f6f113ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c153660 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.119+0000 7f6f113ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0c06c360 con 0x7f6f0c1593c0 2023-03-03T13:43:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.119+0000 7f6f113ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c15cea0 con 0x7f6f0c153290 2023-03-03T13:43:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f03fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f0c1593c0 0x7f6f0c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f0b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6f0c0cd8e0 0x7f6f0c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56462/0 (socket says 172.21.15.102:56462) 2023-03-03T13:43:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f0b7fe700 1 -- 172.21.15.102:0/2820361184 learned_addr learned my addr 172.21.15.102:0/2820361184 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:43:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f0affd700 1 --2- 172.21.15.102:0/2820361184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c153290 0x7f6f0c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f03fff700 1 -- 172.21.15.102:0/2820361184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c153290 msgr2=0x7f6f0c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:43:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f03fff700 1 --2- 172.21.15.102:0/2820361184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c153290 0x7f6f0c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f03fff700 1 -- 172.21.15.102:0/2820361184 >> v1:172.21.15.102:6789/0 conn(0x7f6f0c0cd8e0 legacy=0x7f6f0c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.120+0000 7f6f03fff700 1 -- 172.21.15.102:0/2820361184 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c06c7f0 con 0x7f6f0c1593c0 2023-03-03T13:43:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.121+0000 7f6f03fff700 1 --2- 172.21.15.102:0/2820361184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c1593c0 0x7f6f0c0cde20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009840 tx=0x7f6ef4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea238bf15ee6ca9d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.121+0000 7f6f0a7fc700 1 -- 172.21.15.102:0/2820361184 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4011830 con 0x7f6f0c1593c0 2023-03-03T13:43:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.121+0000 7f6f0a7fc700 1 -- 172.21.15.102:0/2820361184 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef4011990 con 0x7f6f0c1593c0 2023-03-03T13:43:50.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.121+0000 7f6f0a7fc700 1 -- 172.21.15.102:0/2820361184 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4011d20 con 0x7f6f0c1593c0 2023-03-03T13:43:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 -- 172.21.15.102:0/2820361184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c1593c0 msgr2=0x7f6f0c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 --2- 172.21.15.102:0/2820361184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c1593c0 0x7f6f0c0cde20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009840 tx=0x7f6ef4009b50 comp rx=0 tx=0).stop 2023-03-03T13:43:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 -- 172.21.15.102:0/2820361184 shutdown_connections 2023-03-03T13:43:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 --2- 172.21.15.102:0/2820361184 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c153290 0x7f6f0c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 --2- 172.21.15.102:0/2820361184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c1593c0 0x7f6f0c0cde20 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 -- 172.21.15.102:0/2820361184 >> 172.21.15.102:0/2820361184 conn(0x7f6f0c065fb0 msgr2=0x7f6f0c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:50.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 -- 172.21.15.102:0/2820361184 shutdown_connections 2023-03-03T13:43:50.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.122+0000 7f6f113ff700 1 -- 172.21.15.102:0/2820361184 wait complete. 2023-03-03T13:43:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.124+0000 7f6f113ff700 1 Processor -- start 2023-03-03T13:43:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.124+0000 7f6f113ff700 1 -- start start 2023-03-03T13:43:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f113ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c0cd8e0 0x7f6f0c1e5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f113ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0c153290 0x7f6f0c14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f113ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c1593c0 0x7f6f0c14e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f113ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c1e4b20 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f113ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c0c6ac0 con 0x7f6f0c153290 2023-03-03T13:43:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f113ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c1e5050 con 0x7f6f0c1593c0 2023-03-03T13:43:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f0affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c0cd8e0 0x7f6f0c1e5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f0affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c0cd8e0 0x7f6f0c1e5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53786/0 (socket says 172.21.15.102:53786) 2023-03-03T13:43:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f0affd700 1 -- 172.21.15.102:0/97610339 learned_addr learned my addr 172.21.15.102:0/97610339 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:43:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f03fff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0c153290 0x7f6f0c14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.125+0000 7f6f0b7fe700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c1593c0 0x7f6f0c14e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f0affd700 1 -- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c1593c0 msgr2=0x7f6f0c14e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f0affd700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c1593c0 0x7f6f0c14e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f0affd700 1 -- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0c153290 msgr2=0x7f6f0c14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f0affd700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0c153290 0x7f6f0c14e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f0affd700 1 -- 172.21.15.102:0/97610339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef40094a0 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f0b7fe700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c1593c0 0x7f6f0c14e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:43:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f03fff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0c153290 0x7f6f0c14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:43:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f0affd700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c0cd8e0 0x7f6f0c1e5a70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00e870 tx=0x7f6efc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:50.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f037fe700 1 -- 172.21.15.102:0/97610339 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc009590 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f037fe700 1 -- 172.21.15.102:0/97610339 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc013680 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.126+0000 7f6f037fe700 1 -- 172.21.15.102:0/97610339 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc0139f0 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.127+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c0ccde0 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.127+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c15d160 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.128+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c0cd0c0 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.129+0000 7f6f037fe700 1 -- 172.21.15.102:0/97610339 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6efc019030 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.130+0000 7f6f037fe700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eec0682f0 0x7f6eec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.130+0000 7f6f037fe700 1 -- 172.21.15.102:0/97610339 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6efc085b00 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.131+0000 7f6f03fff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eec0682f0 0x7f6eec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.131+0000 7f6f03fff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eec0682f0 0x7f6eec06a7a0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4000c00 tx=0x7f6ef4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:50.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.134+0000 7f6f037fe700 1 -- 172.21.15.102:0/97610339 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6efc087e60 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.369 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.368+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f0c04e6e0 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.368+0000 7f6f037fe700 1 -- 172.21.15.102:0/97610339 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6efc0521b0 con 0x7f6f0c0cd8e0 2023-03-03T13:43:50.370 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:43:50.371 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:43:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.371+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eec0682f0 msgr2=0x7f6eec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.371+0000 7f6f113ff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eec0682f0 0x7f6eec06a7a0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4000c00 tx=0x7f6ef4008030 comp rx=0 tx=0).stop 2023-03-03T13:43:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.371+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c0cd8e0 msgr2=0x7f6f0c1e5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.371+0000 7f6f113ff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c0cd8e0 0x7f6f0c1e5a70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00e870 tx=0x7f6efc00eb80 comp rx=0 tx=0).stop 2023-03-03T13:43:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.371+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 shutdown_connections 2023-03-03T13:43:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.371+0000 7f6f113ff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6eec0682f0 0x7f6eec06a7a0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.371+0000 7f6f113ff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6f0c1593c0 0x7f6f0c14e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.372+0000 7f6f113ff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0c0cd8e0 0x7f6f0c1e5a70 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.372+0000 7f6f113ff700 1 --2- 172.21.15.102:0/97610339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f0c153290 0x7f6f0c14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.372+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 >> 172.21.15.102:0/97610339 conn(0x7f6f0c065fb0 msgr2=0x7f6f0c0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:50.374 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.372+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 shutdown_connections 2023-03-03T13:43:50.375 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:43:50.372+0000 7f6f113ff700 1 -- 172.21.15.102:0/97610339 wait complete. 2023-03-03T13:43:50.375 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:43:50.400 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T13:43:55.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:55.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:25.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:25.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:55.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:55.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:22.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:45:22.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.649+0000 7fc2735ff700 1 Processor -- start 2023-03-03T13:45:22.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.650+0000 7fc2735ff700 1 -- start start 2023-03-03T13:45:22.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc2735ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c153290 0x7fc26c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:22.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc2735ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc26c1593c0 0x7fc26c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc2735ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c153660 con 0x7fc26c0cd8e0 2023-03-03T13:45:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc2735ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc26c06c360 con 0x7fc26c1593c0 2023-03-03T13:45:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc2735ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c15cea0 con 0x7fc26c153290 2023-03-03T13:45:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc271b9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc26c0cd8e0 0x7fc26c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33948/0 (socket says 172.21.15.102:33948) 2023-03-03T13:45:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc271b9c700 1 -- 172.21.15.102:0/806352706 learned_addr learned my addr 172.21.15.102:0/806352706 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:45:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc27139b700 1 --2- 172.21.15.102:0/806352706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c153290 0x7fc26c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:22.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.651+0000 7fc270b9a700 1 --2- 172.21.15.102:0/806352706 >> v2:172.21.15.102:3300/0 conn(0x7fc26c1593c0 0x7fc26c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:22.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.652+0000 7fc263fff700 1 -- 172.21.15.102:0/806352706 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980303785 0 0) 0x7fc26c153660 con 0x7fc26c0cd8e0 2023-03-03T13:45:22.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.652+0000 7fc263fff700 1 -- 172.21.15.102:0/806352706 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2580035b0 con 0x7fc26c0cd8e0 2023-03-03T13:45:22.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.652+0000 7fc270b9a700 1 -- 172.21.15.102:0/806352706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c153290 msgr2=0x7fc26c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:45:22.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.652+0000 7fc270b9a700 1 --2- 172.21.15.102:0/806352706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c153290 0x7fc26c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.652+0000 7fc270b9a700 1 -- 172.21.15.102:0/806352706 >> v1:172.21.15.172:6789/0 conn(0x7fc26c0cd8e0 legacy=0x7fc26c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:22.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.652+0000 7fc270b9a700 1 -- 172.21.15.102:0/806352706 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c06c7f0 con 0x7fc26c1593c0 2023-03-03T13:45:22.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.652+0000 7fc270b9a700 1 --2- 172.21.15.102:0/806352706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c1593c0 0x7fc26c0cde20 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009840 tx=0x7fc25c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cdee195a5ddb919 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:22.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc263fff700 1 -- 172.21.15.102:0/806352706 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c011830 con 0x7fc26c1593c0 2023-03-03T13:45:22.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc263fff700 1 -- 172.21.15.102:0/806352706 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c011990 con 0x7fc26c1593c0 2023-03-03T13:45:22.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc263fff700 1 -- 172.21.15.102:0/806352706 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c011d00 con 0x7fc26c1593c0 2023-03-03T13:45:22.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 -- 172.21.15.102:0/806352706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c1593c0 msgr2=0x7fc26c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:22.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 --2- 172.21.15.102:0/806352706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c1593c0 0x7fc26c0cde20 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009840 tx=0x7fc25c009b50 comp rx=0 tx=0).stop 2023-03-03T13:45:22.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 -- 172.21.15.102:0/806352706 shutdown_connections 2023-03-03T13:45:22.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 --2- 172.21.15.102:0/806352706 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c153290 0x7fc26c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 --2- 172.21.15.102:0/806352706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c1593c0 0x7fc26c0cde20 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 -- 172.21.15.102:0/806352706 >> 172.21.15.102:0/806352706 conn(0x7fc26c065fb0 msgr2=0x7fc26c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:22.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 -- 172.21.15.102:0/806352706 shutdown_connections 2023-03-03T13:45:22.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.653+0000 7fc2735ff700 1 -- 172.21.15.102:0/806352706 wait complete. 2023-03-03T13:45:22.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.655+0000 7fc2735ff700 1 Processor -- start 2023-03-03T13:45:22.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.655+0000 7fc2735ff700 1 -- start start 2023-03-03T13:45:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.656+0000 7fc2735ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc26c0cd8e0 0x7fc26c14cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.656+0000 7fc2735ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c153290 0x7fc26c0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.656+0000 7fc2735ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c1593c0 0x7fc26c0b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc2735ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c0b4190 con 0x7fc26c153290 2023-03-03T13:45:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc2735ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c0b21f0 con 0x7fc26c0cd8e0 2023-03-03T13:45:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc2735ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c14d500 con 0x7fc26c1593c0 2023-03-03T13:45:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c153290 0x7fc26c0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c153290 0x7fc26c0cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42920/0 (socket says 172.21.15.102:42920) 2023-03-03T13:45:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 -- 172.21.15.102:0/4049911653 learned_addr learned my addr 172.21.15.102:0/4049911653 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:45:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc27139b700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc26c0cd8e0 0x7fc26c14cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc271b9c700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c1593c0 0x7fc26c0b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 -- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c1593c0 msgr2=0x7fc26c0b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c1593c0 0x7fc26c0b2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 -- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc26c0cd8e0 msgr2=0x7fc26c14cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc26c0cd8e0 0x7fc26c14cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc270b9a700 1 -- 172.21.15.102:0/4049911653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c0094a0 con 0x7fc26c153290 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.657+0000 7fc271b9c700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c1593c0 0x7fc26c0b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.658+0000 7fc270b9a700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c153290 0x7fc26c0cf020 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0090b0 tx=0x7fc25c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.658+0000 7fc261ffb700 1 -- 172.21.15.102:0/4049911653 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0036a0 con 0x7fc26c153290 2023-03-03T13:45:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.658+0000 7fc261ffb700 1 -- 172.21.15.102:0/4049911653 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c01f070 con 0x7fc26c153290 2023-03-03T13:45:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.658+0000 7fc261ffb700 1 -- 172.21.15.102:0/4049911653 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c003a00 con 0x7fc26c153290 2023-03-03T13:45:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.658+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c0ccde0 con 0x7fc26c153290 2023-03-03T13:45:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.658+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c15d160 con 0x7fc26c153290 2023-03-03T13:45:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.659+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26c0cd0c0 con 0x7fc26c153290 2023-03-03T13:45:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.660+0000 7fc261ffb700 1 -- 172.21.15.102:0/4049911653 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc25c00f050 con 0x7fc26c153290 2023-03-03T13:45:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.662+0000 7fc261ffb700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2500682f0 0x7fc25006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.662+0000 7fc261ffb700 1 -- 172.21.15.102:0/4049911653 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc25c021070 con 0x7fc26c153290 2023-03-03T13:45:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.662+0000 7fc27139b700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2500682f0 0x7fc25006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.663+0000 7fc27139b700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2500682f0 0x7fc25006a7a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fc268009be0 tx=0x7fc268009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.666+0000 7fc261ffb700 1 -- 172.21.15.102:0/4049911653 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc25c053e00 con 0x7fc26c153290 2023-03-03T13:45:22.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.902+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc26c0b2330 con 0x7fc26c153290 2023-03-03T13:45:22.904 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.903+0000 7fc261ffb700 1 -- 172.21.15.102:0/4049911653 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc26c0b2330 con 0x7fc26c153290 2023-03-03T13:45:22.904 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:45:22.905 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:45:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2500682f0 msgr2=0x7fc25006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2500682f0 0x7fc25006a7a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fc268009be0 tx=0x7fc268009210 comp rx=0 tx=0).stop 2023-03-03T13:45:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c153290 msgr2=0x7fc26c0cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:22.907 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c153290 0x7fc26c0cf020 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0090b0 tx=0x7fc25c004750 comp rx=0 tx=0).stop 2023-03-03T13:45:22.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 shutdown_connections 2023-03-03T13:45:22.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2500682f0 0x7fc25006a7a0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc26c1593c0 0x7fc26c0b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc26c0cd8e0 0x7fc26c14cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 --2- 172.21.15.102:0/4049911653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c153290 0x7fc26c0cf020 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:22.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.905+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 >> 172.21.15.102:0/4049911653 conn(0x7fc26c065fb0 msgr2=0x7fc26c157560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:22.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.906+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 shutdown_connections 2023-03-03T13:45:22.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:22.906+0000 7fc2735ff700 1 -- 172.21.15.102:0/4049911653 wait complete. 2023-03-03T13:45:22.909 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:45:22.934 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 15 secs before thrashing 2023-03-03T13:45:26.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:26.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:37.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:45:38.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.185+0000 7feefaf9d700 1 Processor -- start 2023-03-03T13:45:38.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.185+0000 7feefaf9d700 1 -- start start 2023-03-03T13:45:38.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.186+0000 7feefaf9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feef41564b0 0x7feef40cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:38.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feefaf9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:38.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feefaf9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef40c9ca0 con 0x7feef40c9610 2023-03-03T13:45:38.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feefaf9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef40b4920 con 0x7feef41564b0 2023-03-03T13:45:38.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feefaf9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef40b4c00 con 0x7feef40cab50 2023-03-03T13:45:38.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feef979a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7feef40c9610 0x7feef40ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42162/0 (socket says 172.21.15.102:42162) 2023-03-03T13:45:38.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feef979a700 1 -- 172.21.15.102:0/2500251693 learned_addr learned my addr 172.21.15.102:0/2500251693 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:45:38.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feef9f9b700 1 --2- 172.21.15.102:0/2500251693 >> v2:172.21.15.172:3300/0 conn(0x7feef41564b0 0x7feef40cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:38.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feefa79c700 1 --2- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:38.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.187+0000 7feef8f99700 1 -- 172.21.15.102:0/2500251693 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438667029 0 0) 0x7feef40c9ca0 con 0x7feef40c9610 2023-03-03T13:45:38.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef8f99700 1 -- 172.21.15.102:0/2500251693 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee00035b0 con 0x7feef40c9610 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef9f9b700 1 -- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 msgr2=0x7feef40cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef9f9b700 1 --2- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef9f9b700 1 -- 172.21.15.102:0/2500251693 >> v1:172.21.15.102:6789/0 conn(0x7feef40c9610 legacy=0x7feef40ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef9f9b700 1 -- 172.21.15.102:0/2500251693 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40b46f0 con 0x7feef41564b0 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feefa79c700 1 --2- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef9f9b700 1 --2- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef40cb660 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7feef0009840 tx=0x7feef0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aded0bf413a61020 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef8f99700 1 -- 172.21.15.102:0/2500251693 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0011830 con 0x7feef41564b0 2023-03-03T13:45:38.192 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef8f99700 1 -- 172.21.15.102:0/2500251693 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef0011990 con 0x7feef41564b0 2023-03-03T13:45:38.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.188+0000 7feef8f99700 1 -- 172.21.15.102:0/2500251693 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0011d20 con 0x7feef41564b0 2023-03-03T13:45:38.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 -- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 msgr2=0x7feef40cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:38.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 --2- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef40cb660 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7feef0009840 tx=0x7feef0009b50 comp rx=0 tx=0).stop 2023-03-03T13:45:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 -- 172.21.15.102:0/2500251693 shutdown_connections 2023-03-03T13:45:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 --2- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 --2- 172.21.15.102:0/2500251693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef40cb660 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 -- 172.21.15.102:0/2500251693 >> 172.21.15.102:0/2500251693 conn(0x7feef4065fb0 msgr2=0x7feef40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 -- 172.21.15.102:0/2500251693 shutdown_connections 2023-03-03T13:45:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.189+0000 7feefaf9d700 1 -- 172.21.15.102:0/2500251693 wait complete. 2023-03-03T13:45:38.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.192+0000 7feefaf9d700 1 Processor -- start 2023-03-03T13:45:38.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.192+0000 7feefaf9d700 1 -- start start 2023-03-03T13:45:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.193+0000 7feefaf9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feef40c9610 0x7feef415c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.193+0000 7feefaf9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40c2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.193+0000 7feefaf9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef41f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.193+0000 7feefaf9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef40c43b0 con 0x7feef40c9610 2023-03-03T13:45:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.193+0000 7feefaf9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef41f2690 con 0x7feef41564b0 2023-03-03T13:45:38.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.193+0000 7feefaf9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef40c47e0 con 0x7feef40cab50 2023-03-03T13:45:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feef40c9610 0x7feef415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef979a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feef40c9610 0x7feef415c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36346/0 (socket says 172.21.15.102:36346) 2023-03-03T13:45:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feefa79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef41f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef979a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:49072/0 (socket says 172.21.15.102:49072) 2023-03-03T13:45:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 -- 172.21.15.102:0/1125947957 learned_addr learned my addr 172.21.15.102:0/1125947957 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:45:38.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 -- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 msgr2=0x7feef40c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:38.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 -- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 msgr2=0x7feef41f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:38.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef41f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feef9f9b700 1 -- 172.21.15.102:0/1125947957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec009a70 con 0x7feef40c9610 2023-03-03T13:45:38.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.194+0000 7feefa79c700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef41f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:45:38.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.195+0000 7feef9f9b700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feef40c9610 0x7feef415c180 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7feef0000c00 tx=0x7feef0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:38.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.195+0000 7feeea7fc700 1 -- 172.21.15.102:0/1125947957 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0004550 con 0x7feef40c9610 2023-03-03T13:45:38.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.195+0000 7feeea7fc700 1 -- 172.21.15.102:0/1125947957 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef001f070 con 0x7feef40c9610 2023-03-03T13:45:38.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.195+0000 7feeea7fc700 1 -- 172.21.15.102:0/1125947957 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00038e0 con 0x7feef40c9610 2023-03-03T13:45:38.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.195+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef00094a0 con 0x7feef40c9610 2023-03-03T13:45:38.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.195+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef40b2210 con 0x7feef40c9610 2023-03-03T13:45:38.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.196+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef40b1f30 con 0x7feef40c9610 2023-03-03T13:45:38.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.197+0000 7feeea7fc700 1 -- 172.21.15.102:0/1125947957 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feef000f050 con 0x7feef40c9610 2023-03-03T13:45:38.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.198+0000 7feeea7fc700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feed8068320 0x7feed806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:38.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.199+0000 7feeea7fc700 1 -- 172.21.15.102:0/1125947957 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7feef0021070 con 0x7feef40c9610 2023-03-03T13:45:38.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.199+0000 7feef979a700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feed8068320 0x7feed806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:38.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.200+0000 7feef979a700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feed8068320 0x7feed806a7d0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7feef415d3e0 tx=0x7feee4007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:38.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.203+0000 7feeea7fc700 1 -- 172.21.15.102:0/1125947957 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feef0053c40 con 0x7feef40c9610 2023-03-03T13:45:38.441 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.439+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feef4156880 con 0x7feef40c9610 2023-03-03T13:45:38.442 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.439+0000 7feeea7fc700 1 -- 172.21.15.102:0/1125947957 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7feef0057260 con 0x7feef40c9610 2023-03-03T13:45:38.442 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:45:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:45:38.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feed8068320 msgr2=0x7feed806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:38.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feed8068320 0x7feed806a7d0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7feef415d3e0 tx=0x7feee4007dc0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feef40c9610 msgr2=0x7feef415c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:38.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feef40c9610 0x7feef415c180 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7feef0000c00 tx=0x7feef0004750 comp rx=0 tx=0).stop 2023-03-03T13:45:38.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 shutdown_connections 2023-03-03T13:45:38.445 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef41564b0 0x7feef41f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feef40c9610 0x7feef415c180 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7feed8068320 0x7feed806a7d0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 --2- 172.21.15.102:0/1125947957 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feef40cab50 0x7feef40c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:38.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 >> 172.21.15.102:0/1125947957 conn(0x7feef4065fb0 msgr2=0x7feef4157f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:38.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 shutdown_connections 2023-03-03T13:45:38.446 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:45:38.442+0000 7feefaf9d700 1 -- 172.21.15.102:0/1125947957 wait complete. 2023-03-03T13:45:38.447 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:45:38.471 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T13:45:56.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:56.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:10.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:46:10.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.724+0000 7f17a3fff700 1 Processor -- start 2023-03-03T13:46:10.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.725+0000 7f17a3fff700 1 -- start start 2023-03-03T13:46:10.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:10.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f17a40c41b0 0x7f17a40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:10.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a3fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a40c3d20 con 0x7f17a40b3970 2023-03-03T13:46:10.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a40c4930 con 0x7f17a40c41b0 2023-03-03T13:46:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a40cf610 con 0x7f17a40ced00 2023-03-03T13:46:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a27fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f17a40b3970 0x7f17a4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38232/0 (socket says 172.21.15.102:38232) 2023-03-03T13:46:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a40c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48866/0 (socket says 172.21.15.102:48866) 2023-03-03T13:46:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a2ffd700 1 -- 172.21.15.102:0/835134823 learned_addr learned my addr 172.21.15.102:0/835134823 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:46:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.727+0000 7f17a37fe700 1 --2- 172.21.15.102:0/835134823 >> v2:172.21.15.172:3300/0 conn(0x7f17a40c41b0 0x7f17a40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:10.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a2ffd700 1 -- 172.21.15.102:0/835134823 >> v2:172.21.15.172:3300/0 conn(0x7f17a40c41b0 msgr2=0x7f17a40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:46:10.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a2ffd700 1 --2- 172.21.15.102:0/835134823 >> v2:172.21.15.172:3300/0 conn(0x7f17a40c41b0 0x7f17a40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a2ffd700 1 -- 172.21.15.102:0/835134823 >> v1:172.21.15.172:6789/0 conn(0x7f17a40b3970 legacy=0x7f17a4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:10.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a2ffd700 1 -- 172.21.15.102:0/835134823 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a415cba0 con 0x7f17a40ced00 2023-03-03T13:46:10.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a2ffd700 1 --2- 172.21.15.102:0/835134823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a40c5a90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f17940098f0 tx=0x7f1794009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87c910632383bc71 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a1ffb700 1 -- 172.21.15.102:0/835134823 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794011830 con 0x7f17a40ced00 2023-03-03T13:46:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a1ffb700 1 -- 172.21.15.102:0/835134823 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1794011990 con 0x7f17a40ced00 2023-03-03T13:46:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.728+0000 7f17a1ffb700 1 -- 172.21.15.102:0/835134823 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794011d20 con 0x7f17a40ced00 2023-03-03T13:46:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 -- 172.21.15.102:0/835134823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 msgr2=0x7f17a40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 --2- 172.21.15.102:0/835134823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a40c5a90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f17940098f0 tx=0x7f1794009c00 comp rx=0 tx=0).stop 2023-03-03T13:46:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 -- 172.21.15.102:0/835134823 shutdown_connections 2023-03-03T13:46:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 --2- 172.21.15.102:0/835134823 >> v2:172.21.15.172:3300/0 conn(0x7f17a40c41b0 0x7f17a40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 --2- 172.21.15.102:0/835134823 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a40c5a90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 -- 172.21.15.102:0/835134823 >> 172.21.15.102:0/835134823 conn(0x7f17a408b1c0 msgr2=0x7f17a4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:46:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 -- 172.21.15.102:0/835134823 shutdown_connections 2023-03-03T13:46:10.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.729+0000 7f17a3fff700 1 -- 172.21.15.102:0/835134823 wait complete. 2023-03-03T13:46:10.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.732+0000 7f17a3fff700 1 Processor -- start 2023-03-03T13:46:10.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.732+0000 7f17a3fff700 1 -- start start 2023-03-03T13:46:10.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.733+0000 7f17a3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17a40b3970 0x7f17a4150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:10.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a40c41b0 0x7f17a41eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:10.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a41ef0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:10.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a415c490 con 0x7f17a40b3970 2023-03-03T13:46:10.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a414e160 con 0x7f17a40c41b0 2023-03-03T13:46:10.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a41f2670 con 0x7f17a40ced00 2023-03-03T13:46:10.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17a40b3970 0x7f17a4150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:10.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17a40b3970 0x7f17a4150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60662/0 (socket says 172.21.15.102:60662) 2023-03-03T13:46:10.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 -- 172.21.15.102:0/544605555 learned_addr learned my addr 172.21.15.102:0/544605555 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:46:10.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a37fe700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a41ef0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:10.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a27fc700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a40c41b0 0x7f17a41eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:10.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 -- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 msgr2=0x7f17a41ef0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:10.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a41ef0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 -- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a40c41b0 msgr2=0x7f17a41eeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:10.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a40c41b0 0x7f17a41eeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.734+0000 7f17a2ffd700 1 -- 172.21.15.102:0/544605555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1794009550 con 0x7f17a40b3970 2023-03-03T13:46:10.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.735+0000 7f17a2ffd700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17a40b3970 0x7f17a4150b70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1794009c40 tx=0x7f1794004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:10.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.735+0000 7f178b7fe700 1 -- 172.21.15.102:0/544605555 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794004300 con 0x7f17a40b3970 2023-03-03T13:46:10.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.735+0000 7f178b7fe700 1 -- 172.21.15.102:0/544605555 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179401f070 con 0x7f17a40b3970 2023-03-03T13:46:10.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.735+0000 7f178b7fe700 1 -- 172.21.15.102:0/544605555 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17940036a0 con 0x7f17a40b3970 2023-03-03T13:46:10.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.735+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a406bf20 con 0x7f17a40b3970 2023-03-03T13:46:10.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.735+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a415ce80 con 0x7f17a40b3970 2023-03-03T13:46:10.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.736+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a406c200 con 0x7f17a40b3970 2023-03-03T13:46:10.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.738+0000 7f178b7fe700 1 -- 172.21.15.102:0/544605555 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f179402a020 con 0x7f17a40b3970 2023-03-03T13:46:10.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.740+0000 7f178b7fe700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17900682f0 0x7f179006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:10.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.740+0000 7f178b7fe700 1 -- 172.21.15.102:0/544605555 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1794087310 con 0x7f17a40b3970 2023-03-03T13:46:10.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.740+0000 7f17a27fc700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17900682f0 0x7f179006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:10.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.741+0000 7f17a27fc700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17900682f0 0x7f179006a7a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f17a415bf20 tx=0x7f1798007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:10.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.744+0000 7f178b7fe700 1 -- 172.21.15.102:0/544605555 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1794053940 con 0x7f17a40b3970 2023-03-03T13:46:10.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.978+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a40b1a90 con 0x7f17a40b3970 2023-03-03T13:46:10.980 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.979+0000 7f178b7fe700 1 -- 172.21.15.102:0/544605555 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1794056f60 con 0x7f17a40b3970 2023-03-03T13:46:10.980 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:46:10.980 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:46:10.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17900682f0 msgr2=0x7f179006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:10.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17900682f0 0x7f179006a7a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f17a415bf20 tx=0x7f1798007ed0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17a40b3970 msgr2=0x7f17a4150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:10.983 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17a40b3970 0x7f17a4150b70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1794009c40 tx=0x7f1794004750 comp rx=0 tx=0).stop 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 shutdown_connections 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f17900682f0 0x7f179006a7a0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f17a40ced00 0x7f17a41ef0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17a40b3970 0x7f17a4150b70 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 --2- 172.21.15.102:0/544605555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17a40c41b0 0x7f17a41eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 >> 172.21.15.102:0/544605555 conn(0x7f17a408b1c0 msgr2=0x7f17a40cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.981+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 shutdown_connections 2023-03-03T13:46:10.984 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:10.982+0000 7f17a3fff700 1 -- 172.21.15.102:0/544605555 wait complete. 2023-03-03T13:46:10.985 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:46:11.010 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T13:46:26.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:26.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:45.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:46:45.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.259+0000 7fc6661ff700 1 Processor -- start 2023-03-03T13:46:45.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.259+0000 7fc6661ff700 1 -- start start 2023-03-03T13:46:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.260+0000 7fc6661ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc6600cd8e0 0x7fc6600cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.260+0000 7fc6661ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd390 0x7fc6600ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.260+0000 7fc6661ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc660153660 con 0x7fc660153290 2023-03-03T13:46:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.260+0000 7fc6661ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66006c360 con 0x7fc6600cd8e0 2023-03-03T13:46:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.260+0000 7fc6661ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015cea0 con 0x7fc6600cd390 2023-03-03T13:46:45.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc660153290 0x7fc660158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35660/0 (socket says 172.21.15.102:35660) 2023-03-03T13:46:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65f7fe700 1 -- 172.21.15.102:0/1343476444 learned_addr learned my addr 172.21.15.102:0/1343476444 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:46:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65ffff700 1 --2- 172.21.15.102:0/1343476444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd390 0x7fc6600ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65effd700 1 --2- 172.21.15.102:0/1343476444 >> v2:172.21.15.172:3300/0 conn(0x7fc6600cd8e0 0x7fc6600cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65e7fc700 1 -- 172.21.15.102:0/1343476444 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737609268 0 0) 0x7fc660153660 con 0x7fc660153290 2023-03-03T13:46:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65ffff700 1 -- 172.21.15.102:0/1343476444 >> v2:172.21.15.172:3300/0 conn(0x7fc6600cd8e0 msgr2=0x7fc6600cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:46:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65ffff700 1 --2- 172.21.15.102:0/1343476444 >> v2:172.21.15.172:3300/0 conn(0x7fc6600cd8e0 0x7fc6600cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65ffff700 1 -- 172.21.15.102:0/1343476444 >> v1:172.21.15.102:6789/0 conn(0x7fc660153290 legacy=0x7fc660158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65ffff700 1 -- 172.21.15.102:0/1343476444 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66006c7f0 con 0x7fc6600cd390 2023-03-03T13:46:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.261+0000 7fc65effd700 1 --2- 172.21.15.102:0/1343476444 >> v2:172.21.15.172:3300/0 conn(0x7fc6600cd8e0 0x7fc6600cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:46:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.262+0000 7fc65ffff700 1 --2- 172.21.15.102:0/1343476444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd390 0x7fc6600ce360 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc658009840 tx=0x7fc658009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6766282faa199e89 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.262+0000 7fc65e7fc700 1 -- 172.21.15.102:0/1343476444 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658011830 con 0x7fc6600cd390 2023-03-03T13:46:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.262+0000 7fc65e7fc700 1 -- 172.21.15.102:0/1343476444 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc658011990 con 0x7fc6600cd390 2023-03-03T13:46:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.262+0000 7fc65e7fc700 1 -- 172.21.15.102:0/1343476444 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658011d20 con 0x7fc6600cd390 2023-03-03T13:46:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 -- 172.21.15.102:0/1343476444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd390 msgr2=0x7fc6600ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 --2- 172.21.15.102:0/1343476444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd390 0x7fc6600ce360 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc658009840 tx=0x7fc658009b50 comp rx=0 tx=0).stop 2023-03-03T13:46:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 -- 172.21.15.102:0/1343476444 shutdown_connections 2023-03-03T13:46:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 --2- 172.21.15.102:0/1343476444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd390 0x7fc6600ce360 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 --2- 172.21.15.102:0/1343476444 >> v2:172.21.15.172:3300/0 conn(0x7fc6600cd8e0 0x7fc6600cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 -- 172.21.15.102:0/1343476444 >> 172.21.15.102:0/1343476444 conn(0x7fc660065fb0 msgr2=0x7fc6600663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:46:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 -- 172.21.15.102:0/1343476444 shutdown_connections 2023-03-03T13:46:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.263+0000 7fc6661ff700 1 -- 172.21.15.102:0/1343476444 wait complete. 2023-03-03T13:46:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.265+0000 7fc6661ff700 1 Processor -- start 2023-03-03T13:46:45.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.265+0000 7fc6661ff700 1 -- start start 2023-03-03T13:46:45.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc6661ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6600cd390 0x7fc6601f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc6661ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd8e0 0x7fc6601ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc6661ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc660153290 0x7fc6600cf650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc6661ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6601eee70 con 0x7fc660153290 2023-03-03T13:46:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc6661ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c6780 con 0x7fc6600cd390 2023-03-03T13:46:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc6661ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc660153fc0 con 0x7fc6600cd8e0 2023-03-03T13:46:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc65f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6600cd390 0x7fc6601f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:45.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc65effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd8e0 0x7fc6601ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc65f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6600cd390 0x7fc6601f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45404/0 (socket says 172.21.15.102:45404) 2023-03-03T13:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc65f7fe700 1 -- 172.21.15.102:0/3591088270 learned_addr learned my addr 172.21.15.102:0/3591088270 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.266+0000 7fc65ffff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc660153290 0x7fc6600cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc65effd700 1 -- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6600cd390 msgr2=0x7fc6601f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc65effd700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6600cd390 0x7fc6601f2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc65effd700 1 -- 172.21.15.102:0/3591088270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc660153290 msgr2=0x7fc6600cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc65effd700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc660153290 0x7fc6600cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc65effd700 1 -- 172.21.15.102:0/3591088270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc654009a70 con 0x7fc6600cd8e0 2023-03-03T13:46:45.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc65effd700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd8e0 0x7fc6601ef290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc654009dc0 tx=0x7fc65400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:45.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc647fff700 1 -- 172.21.15.102:0/3591088270 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65400cd10 con 0x7fc6600cd8e0 2023-03-03T13:46:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc647fff700 1 -- 172.21.15.102:0/3591088270 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65400f040 con 0x7fc6600cd8e0 2023-03-03T13:46:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc647fff700 1 -- 172.21.15.102:0/3591088270 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6540146c0 con 0x7fc6600cd8e0 2023-03-03T13:46:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.267+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6580094a0 con 0x7fc6600cd8e0 2023-03-03T13:46:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.268+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6600ccde0 con 0x7fc6600cd8e0 2023-03-03T13:46:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.269+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66015d160 con 0x7fc6600cd8e0 2023-03-03T13:46:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.270+0000 7fc647fff700 1 -- 172.21.15.102:0/3591088270 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc654014820 con 0x7fc6600cd8e0 2023-03-03T13:46:45.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.271+0000 7fc647fff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc64c0682f0 0x7fc64c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:45.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.271+0000 7fc647fff700 1 -- 172.21.15.102:0/3591088270 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc654086a50 con 0x7fc6600cd8e0 2023-03-03T13:46:45.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.271+0000 7fc65f7fe700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc64c0682f0 0x7fc64c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:45.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.272+0000 7fc65f7fe700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc64c0682f0 0x7fc64c06a7a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fc650000f60 tx=0x7fc650007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:45.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.275+0000 7fc647fff700 1 -- 172.21.15.102:0/3591088270 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc654053100 con 0x7fc6600cd8e0 2023-03-03T13:46:45.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.509+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6600cd0c0 con 0x7fc6600cd8e0 2023-03-03T13:46:45.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.510+0000 7fc647fff700 1 -- 172.21.15.102:0/3591088270 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc654056720 con 0x7fc6600cd8e0 2023-03-03T13:46:45.512 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:46:45.512 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:46:45.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc64c0682f0 msgr2=0x7fc64c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:45.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc64c0682f0 0x7fc64c06a7a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fc650000f60 tx=0x7fc650007ea0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd8e0 msgr2=0x7fc6601ef290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:45.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd8e0 0x7fc6601ef290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc654009dc0 tx=0x7fc65400c3a0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 shutdown_connections 2023-03-03T13:46:45.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc660153290 0x7fc6600cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6600cd390 0x7fc6601f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc64c0682f0 0x7fc64c06a7a0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 --2- 172.21.15.102:0/3591088270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc6600cd8e0 0x7fc6601ef290 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:45.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 >> 172.21.15.102:0/3591088270 conn(0x7fc660065fb0 msgr2=0x7fc660157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:46:45.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 shutdown_connections 2023-03-03T13:46:45.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:46:45.513+0000 7fc6661ff700 1 -- 172.21.15.102:0/3591088270 wait complete. 2023-03-03T13:46:45.516 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:46:45.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T13:46:56.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:56.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:26.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:26.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:57.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:57.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:22.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:48:22.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.791+0000 7f499599d700 1 Processor -- start 2023-03-03T13:48:22.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.791+0000 7f499599d700 1 -- start start 2023-03-03T13:48:22.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f499599d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49900c5a90 0x7f49900c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:22.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f499599d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49900c3d20 con 0x7f49900b3970 2023-03-03T13:48:22.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f499599d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499015cba0 con 0x7f49900ced00 2023-03-03T13:48:22.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f499599d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49900c4930 con 0x7f49900c5a90 2023-03-03T13:48:22.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f498ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f49900b3970 0x7f4990161470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40782/0 (socket says 172.21.15.102:40782) 2023-03-03T13:48:22.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f499499b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f49900ced00 0x7f4990158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40142/0 (socket says 172.21.15.102:40142) 2023-03-03T13:48:22.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f498ffff700 1 -- 172.21.15.102:0/2259248619 learned_addr learned my addr 172.21.15.102:0/2259248619 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:48:22.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.793+0000 7f499519c700 1 --2- 172.21.15.102:0/2259248619 >> v2:172.21.15.102:3300/0 conn(0x7f49900c5a90 0x7f49900c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:22.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f498f7fe700 1 -- 172.21.15.102:0/2259248619 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295143917 0 0) 0x7f499015cba0 con 0x7f49900ced00 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f498f7fe700 1 -- 172.21.15.102:0/2259248619 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49800035b0 con 0x7f49900ced00 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f498f7fe700 1 -- 172.21.15.102:0/2259248619 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416903764 0 0) 0x7f49900c3d20 con 0x7f49900b3970 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f498f7fe700 1 -- 172.21.15.102:0/2259248619 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f499015cba0 con 0x7f49900b3970 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f499519c700 1 -- 172.21.15.102:0/2259248619 >> v1:172.21.15.172:6789/0 conn(0x7f49900ced00 legacy=0x7f4990158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f499519c700 1 -- 172.21.15.102:0/2259248619 >> v1:172.21.15.102:6789/0 conn(0x7f49900b3970 legacy=0x7f4990161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f499519c700 1 -- 172.21.15.102:0/2259248619 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499015ce80 con 0x7f49900c5a90 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f499519c700 1 --2- 172.21.15.102:0/2259248619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f49900c41b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4988009d90 tx=0x7f498800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a8dbd0734be9cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:22.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f498f7fe700 1 -- 172.21.15.102:0/2259248619 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498800e070 con 0x7f49900c5a90 2023-03-03T13:48:22.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f498f7fe700 1 -- 172.21.15.102:0/2259248619 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498800bb20 con 0x7f49900c5a90 2023-03-03T13:48:22.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.794+0000 7f498f7fe700 1 -- 172.21.15.102:0/2259248619 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988012430 con 0x7f49900c5a90 2023-03-03T13:48:22.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.795+0000 7f499599d700 1 -- 172.21.15.102:0/2259248619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 msgr2=0x7f49900c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.795+0000 7f499599d700 1 --2- 172.21.15.102:0/2259248619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f49900c41b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4988009d90 tx=0x7f498800b330 comp rx=0 tx=0).stop 2023-03-03T13:48:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.795+0000 7f499599d700 1 -- 172.21.15.102:0/2259248619 shutdown_connections 2023-03-03T13:48:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.795+0000 7f499599d700 1 --2- 172.21.15.102:0/2259248619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f49900c41b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.795+0000 7f499599d700 1 -- 172.21.15.102:0/2259248619 >> 172.21.15.102:0/2259248619 conn(0x7f499008b1c0 msgr2=0x7f4990071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.795+0000 7f499599d700 1 -- 172.21.15.102:0/2259248619 shutdown_connections 2023-03-03T13:48:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.795+0000 7f499599d700 1 -- 172.21.15.102:0/2259248619 wait complete. 2023-03-03T13:48:22.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.798+0000 7f499599d700 1 Processor -- start 2023-03-03T13:48:22.800 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.798+0000 7f499599d700 1 -- start start 2023-03-03T13:48:22.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499599d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49900b3970 0x7f499015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:22.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499599d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f499014cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:22.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499599d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49900ced00 0x7f499014d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:22.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499599d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49901e4840 con 0x7f49900c5a90 2023-03-03T13:48:22.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499599d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499015b200 con 0x7f49900ced00 2023-03-03T13:48:22.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499599d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49901e35c0 con 0x7f49900b3970 2023-03-03T13:48:22.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499499b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49900b3970 0x7f499015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:22.802 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f498ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f499014cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:22.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499519c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49900ced00 0x7f499014d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:22.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499499b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49900b3970 0x7f499015a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:52380/0 (socket says 172.21.15.102:52380) 2023-03-03T13:48:22.803 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499499b700 1 -- 172.21.15.102:0/3030878761 learned_addr learned my addr 172.21.15.102:0/3030878761 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:48:22.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f498ffff700 1 -- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49900b3970 msgr2=0x7f499015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:22.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f498ffff700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49900b3970 0x7f499015a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:22.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f498ffff700 1 -- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49900ced00 msgr2=0x7f499014d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:22.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f498ffff700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49900ced00 0x7f499014d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:22.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f498ffff700 1 -- 172.21.15.102:0/3030878761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988009a70 con 0x7f49900c5a90 2023-03-03T13:48:22.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.800+0000 7f499499b700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49900b3970 0x7f499015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:48:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.801+0000 7f498ffff700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f499014cf80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f497c00c430 tx=0x7f497c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.801+0000 7f498d7fa700 1 -- 172.21.15.102:0/3030878761 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c005d70 con 0x7f49900c5a90 2023-03-03T13:48:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.801+0000 7f498d7fa700 1 -- 172.21.15.102:0/3030878761 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c00ce80 con 0x7f49900c5a90 2023-03-03T13:48:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.801+0000 7f498d7fa700 1 -- 172.21.15.102:0/3030878761 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c0072f0 con 0x7f49900c5a90 2023-03-03T13:48:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.801+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499006c200 con 0x7f49900c5a90 2023-03-03T13:48:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.801+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499006bf20 con 0x7f49900c5a90 2023-03-03T13:48:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.802+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499015c890 con 0x7f49900c5a90 2023-03-03T13:48:22.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.804+0000 7f498d7fa700 1 -- 172.21.15.102:0/3030878761 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f497c010070 con 0x7f49900c5a90 2023-03-03T13:48:22.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.805+0000 7f498d7fa700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49740682f0 0x7f497406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:22.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.805+0000 7f498d7fa700 1 -- 172.21.15.102:0/3030878761 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f497c084c20 con 0x7f49900c5a90 2023-03-03T13:48:22.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.805+0000 7f499499b700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49740682f0 0x7f497406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:22.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.806+0000 7f499499b700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49740682f0 0x7f497406a7a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f4984000f60 tx=0x7f4984007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:22.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:22.809+0000 7f498d7fa700 1 -- 172.21.15.102:0/3030878761 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f497c051330 con 0x7f49900c5a90 2023-03-03T13:48:23.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.046+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f499004e6e0 con 0x7f49900c5a90 2023-03-03T13:48:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.046+0000 7f498d7fa700 1 -- 172.21.15.102:0/3030878761 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f497c054950 con 0x7f49900c5a90 2023-03-03T13:48:23.048 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:48:23.048 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:48:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49740682f0 msgr2=0x7f497406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49740682f0 0x7f497406a7a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f4984000f60 tx=0x7f4984007ea0 comp rx=0 tx=0).stop 2023-03-03T13:48:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 msgr2=0x7f499014cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f499014cf80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f497c00c430 tx=0x7f497c00c740 comp rx=0 tx=0).stop 2023-03-03T13:48:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 shutdown_connections 2023-03-03T13:48:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49740682f0 0x7f497406a7a0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49900b3970 0x7f499015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49900ced00 0x7f499014d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 --2- 172.21.15.102:0/3030878761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49900c5a90 0x7f499014cf80 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 >> 172.21.15.102:0/3030878761 conn(0x7f499008b1c0 msgr2=0x7f4990071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 shutdown_connections 2023-03-03T13:48:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:23.049+0000 7f499599d700 1 -- 172.21.15.102:0/3030878761 wait complete. 2023-03-03T13:48:23.053 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:48:23.077 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before thrashing 2023-03-03T13:48:27.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:48:27.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:27.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:27.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.332+0000 7fa5319ff700 1 Processor -- start 2023-03-03T13:48:27.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.332+0000 7fa5319ff700 1 -- start start 2023-03-03T13:48:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.333+0000 7fa5319ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.333+0000 7fa5319ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c153660 con 0x7fa52c153290 2023-03-03T13:48:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa5319ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c06c360 con 0x7fa52c0cd390 2023-03-03T13:48:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa5319ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c15cea0 con 0x7fa52c0cd8e0 2023-03-03T13:48:27.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa52affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa52c153290 0x7fa52c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35228/0 (socket says 172.21.15.102:35228) 2023-03-03T13:48:27.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa52affd700 1 -- 172.21.15.102:0/3067923344 learned_addr learned my addr 172.21.15.102:0/3067923344 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:48:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa52a7fc700 1 --2- 172.21.15.102:0/3067923344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa52a7fc700 1 -- 172.21.15.102:0/3067923344 >> v1:172.21.15.172:6789/0 conn(0x7fa52c0cd390 legacy=0x7fa52c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa52a7fc700 1 -- 172.21.15.102:0/3067923344 >> v1:172.21.15.102:6789/0 conn(0x7fa52c153290 legacy=0x7fa52c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa529ffb700 1 -- 172.21.15.102:0/3067923344 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968929667 0 0) 0x7fa52c06c360 con 0x7fa52c0cd390 2023-03-03T13:48:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.334+0000 7fa52a7fc700 1 -- 172.21.15.102:0/3067923344 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c06c7f0 con 0x7fa52c0cd8e0 2023-03-03T13:48:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.335+0000 7fa52a7fc700 1 --2- 172.21.15.102:0/3067923344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c0cde20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa520009870 tx=0x7fa520009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1780a1319e4003 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.335+0000 7fa529ffb700 1 -- 172.21.15.102:0/3067923344 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520004230 con 0x7fa52c0cd8e0 2023-03-03T13:48:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.335+0000 7fa529ffb700 1 -- 172.21.15.102:0/3067923344 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa520004390 con 0x7fa52c0cd8e0 2023-03-03T13:48:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.335+0000 7fa529ffb700 1 -- 172.21.15.102:0/3067923344 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5200036a0 con 0x7fa52c0cd8e0 2023-03-03T13:48:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.335+0000 7fa5319ff700 1 -- 172.21.15.102:0/3067923344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 msgr2=0x7fa52c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.336+0000 7fa5319ff700 1 --2- 172.21.15.102:0/3067923344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c0cde20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa520009870 tx=0x7fa520009b80 comp rx=0 tx=0).stop 2023-03-03T13:48:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.336+0000 7fa5319ff700 1 -- 172.21.15.102:0/3067923344 shutdown_connections 2023-03-03T13:48:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.336+0000 7fa5319ff700 1 --2- 172.21.15.102:0/3067923344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c0cde20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.336+0000 7fa5319ff700 1 -- 172.21.15.102:0/3067923344 >> 172.21.15.102:0/3067923344 conn(0x7fa52c065fb0 msgr2=0x7fa52c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.336+0000 7fa5319ff700 1 -- 172.21.15.102:0/3067923344 shutdown_connections 2023-03-03T13:48:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.336+0000 7fa5319ff700 1 -- 172.21.15.102:0/3067923344 wait complete. 2023-03-03T13:48:27.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.339+0000 7fa5319ff700 1 Processor -- start 2023-03-03T13:48:27.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.339+0000 7fa5319ff700 1 -- start start 2023-03-03T13:48:27.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.340+0000 7fa5319ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa52c0cd390 0x7fa52c0caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:27.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.340+0000 7fa5319ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c1eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:27.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.340+0000 7fa5319ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa52c153290 0x7fa52c1ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.340+0000 7fa5319ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c1eeb70 con 0x7fa52c153290 2023-03-03T13:48:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.340+0000 7fa5319ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0cbd00 con 0x7fa52c0cd390 2023-03-03T13:48:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.340+0000 7fa5319ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c1f1ed0 con 0x7fa52c0cd8e0 2023-03-03T13:48:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa52c0cd390 0x7fa52c0caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa52c0cd390 0x7fa52c0caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56168/0 (socket says 172.21.15.102:56168) 2023-03-03T13:48:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 -- 172.21.15.102:0/953730031 learned_addr learned my addr 172.21.15.102:0/953730031 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:48:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52b7fe700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa52c153290 0x7fa52c1ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:27.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52a7fc700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c1eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:27.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 -- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 msgr2=0x7fa52c1eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:27.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c1eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 -- 172.21.15.102:0/953730031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa52c153290 msgr2=0x7fa52c1ef4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:27.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa52c153290 0x7fa52c1ef4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52affd700 1 -- 172.21.15.102:0/953730031 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5200094a0 con 0x7fa52c0cd390 2023-03-03T13:48:27.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.341+0000 7fa52a7fc700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c1eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:48:27.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.342+0000 7fa52affd700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa52c0cd390 0x7fa52c0caf70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00c7b0 tx=0x7fa51c00cac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.342+0000 7fa513fff700 1 -- 172.21.15.102:0/953730031 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c004c80 con 0x7fa52c0cd390 2023-03-03T13:48:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.342+0000 7fa513fff700 1 -- 172.21.15.102:0/953730031 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c004de0 con 0x7fa52c0cd390 2023-03-03T13:48:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.342+0000 7fa513fff700 1 -- 172.21.15.102:0/953730031 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c007310 con 0x7fa52c0cd390 2023-03-03T13:48:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.342+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c0ccde0 con 0x7fa52c0cd390 2023-03-03T13:48:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.342+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c15d160 con 0x7fa52c0cd390 2023-03-03T13:48:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.344+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c0cd0c0 con 0x7fa52c0cd390 2023-03-03T13:48:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.346+0000 7fa513fff700 1 -- 172.21.15.102:0/953730031 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa51c01a020 con 0x7fa52c0cd390 2023-03-03T13:48:27.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.347+0000 7fa513fff700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5180682f0 0x7fa51806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:27.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.347+0000 7fa513fff700 1 -- 172.21.15.102:0/953730031 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa51c0859e0 con 0x7fa52c0cd390 2023-03-03T13:48:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.347+0000 7fa52a7fc700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5180682f0 0x7fa51806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.348+0000 7fa52a7fc700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5180682f0 0x7fa51806a7a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fa5200097c0 tx=0x7fa520008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.351+0000 7fa513fff700 1 -- 172.21.15.102:0/953730031 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa51c0520f0 con 0x7fa52c0cd390 2023-03-03T13:48:27.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.587+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa52c04e6e0 con 0x7fa52c0cd390 2023-03-03T13:48:27.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.588+0000 7fa513fff700 1 -- 172.21.15.102:0/953730031 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa51c055710 con 0x7fa52c0cd390 2023-03-03T13:48:27.590 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:48:27.590 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:48:27.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5180682f0 msgr2=0x7fa51806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:27.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5180682f0 0x7fa51806a7a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fa5200097c0 tx=0x7fa520008030 comp rx=0 tx=0).stop 2023-03-03T13:48:27.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa52c0cd390 msgr2=0x7fa52c0caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:27.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa52c0cd390 0x7fa52c0caf70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00c7b0 tx=0x7fa51c00cac0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 shutdown_connections 2023-03-03T13:48:27.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa52c153290 0x7fa52c1ef4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa52c0cd390 0x7fa52c0caf70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa5180682f0 0x7fa51806a7a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 --2- 172.21.15.102:0/953730031 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa52c0cd8e0 0x7fa52c1eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:27.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 >> 172.21.15.102:0/953730031 conn(0x7fa52c065fb0 msgr2=0x7fa52c158670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:27.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 shutdown_connections 2023-03-03T13:48:27.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:27.591+0000 7fa5319ff700 1 -- 172.21.15.102:0/953730031 wait complete. 2023-03-03T13:48:27.594 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:48:27.619 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-03-03T13:48:55.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:48:55.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.869+0000 7f39637ff700 1 Processor -- start 2023-03-03T13:48:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.869+0000 7f39637ff700 1 -- start start 2023-03-03T13:48:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.870+0000 7f39637ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f395c153290 0x7f395c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f39637ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c153660 con 0x7f395c1593c0 2023-03-03T13:48:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f39637ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c06c360 con 0x7f395c0cd8e0 2023-03-03T13:48:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f39637ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395c0cec40 con 0x7f395c153290 2023-03-03T13:48:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f396159b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f395c153290 0x7f395c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f3960d9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f395c1593c0 0x7f395c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39868/0 (socket says 172.21.15.102:39868) 2023-03-03T13:48:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f3960d9a700 1 -- 172.21.15.102:0/105006072 learned_addr learned my addr 172.21.15.102:0/105006072 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:48:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f396159b700 1 -- 172.21.15.102:0/105006072 >> v1:172.21.15.172:6789/0 conn(0x7f395c0cd8e0 legacy=0x7f395c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f396159b700 1 -- 172.21.15.102:0/105006072 >> v1:172.21.15.102:6789/0 conn(0x7f395c1593c0 legacy=0x7f395c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.871+0000 7f396159b700 1 -- 172.21.15.102:0/105006072 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c15cea0 con 0x7f395c153290 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.872+0000 7f396159b700 1 --2- 172.21.15.102:0/105006072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c153290 0x7f395c158e80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f3958009840 tx=0x7f3958009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fd5f16fae8f500 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.872+0000 7f3953fff700 1 -- 172.21.15.102:0/105006072 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958011830 con 0x7f395c153290 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.872+0000 7f3953fff700 1 -- 172.21.15.102:0/105006072 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958011990 con 0x7f395c153290 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.872+0000 7f3953fff700 1 -- 172.21.15.102:0/105006072 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958011d20 con 0x7f395c153290 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.873+0000 7f39637ff700 1 -- 172.21.15.102:0/105006072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c153290 msgr2=0x7f395c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.873+0000 7f39637ff700 1 --2- 172.21.15.102:0/105006072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c153290 0x7f395c158e80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f3958009840 tx=0x7f3958009b50 comp rx=0 tx=0).stop 2023-03-03T13:48:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.873+0000 7f39637ff700 1 -- 172.21.15.102:0/105006072 shutdown_connections 2023-03-03T13:48:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.873+0000 7f39637ff700 1 --2- 172.21.15.102:0/105006072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c153290 0x7f395c158e80 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.873+0000 7f39637ff700 1 -- 172.21.15.102:0/105006072 >> 172.21.15.102:0/105006072 conn(0x7f395c065fb0 msgr2=0x7f395c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.873+0000 7f39637ff700 1 -- 172.21.15.102:0/105006072 shutdown_connections 2023-03-03T13:48:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.873+0000 7f39637ff700 1 -- 172.21.15.102:0/105006072 wait complete. 2023-03-03T13:48:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.875+0000 7f39637ff700 1 Processor -- start 2023-03-03T13:48:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.875+0000 7f39637ff700 1 -- start start 2023-03-03T13:48:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f39637ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f395c0cd8e0 0x7f395c0b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f39637ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395c153290 0x7f395c150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f39637ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c1593c0 0x7f395c1506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f39637ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c0cd200 con 0x7f395c1593c0 2023-03-03T13:48:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f39637ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c1eea10 con 0x7f395c153290 2023-03-03T13:48:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f39637ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395c1f2120 con 0x7f395c0cd8e0 2023-03-03T13:48:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f3961d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c1593c0 0x7f395c1506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f3960d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395c153290 0x7f395c150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f396159b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f395c0cd8e0 0x7f395c0b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f396159b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f395c0cd8e0 0x7f395c0b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:40364/0 (socket says 172.21.15.102:40364) 2023-03-03T13:48:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f396159b700 1 -- 172.21.15.102:0/2861227607 learned_addr learned my addr 172.21.15.102:0/2861227607 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:48:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.876+0000 7f3960d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395c153290 0x7f395c150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:44500/0 (socket says 172.21.15.102:44500) 2023-03-03T13:48:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3961d9c700 1 -- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f395c0cd8e0 msgr2=0x7f395c0b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3961d9c700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f395c0cd8e0 0x7f395c0b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3961d9c700 1 -- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395c153290 msgr2=0x7f395c150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3961d9c700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395c153290 0x7f395c150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3961d9c700 1 -- 172.21.15.102:0/2861227607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39580094a0 con 0x7f395c1593c0 2023-03-03T13:48:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3961d9c700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c1593c0 0x7f395c1506d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f395400cae0 tx=0x7f395400cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3951ffb700 1 -- 172.21.15.102:0/2861227607 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954003c00 con 0x7f395c1593c0 2023-03-03T13:48:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3951ffb700 1 -- 172.21.15.102:0/2861227607 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3954003d60 con 0x7f395c1593c0 2023-03-03T13:48:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c15d160 con 0x7f395c1593c0 2023-03-03T13:48:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.877+0000 7f3951ffb700 1 -- 172.21.15.102:0/2861227607 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400aa20 con 0x7f395c1593c0 2023-03-03T13:48:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.878+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c06c7f0 con 0x7f395c1593c0 2023-03-03T13:48:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.879+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c0ccde0 con 0x7f395c1593c0 2023-03-03T13:48:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.880+0000 7f3951ffb700 1 -- 172.21.15.102:0/2861227607 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3954010050 con 0x7f395c1593c0 2023-03-03T13:48:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.881+0000 7f3951ffb700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39400682f0 0x7f394006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.881+0000 7f3951ffb700 1 -- 172.21.15.102:0/2861227607 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3954085b10 con 0x7f395c1593c0 2023-03-03T13:48:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.881+0000 7f396159b700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39400682f0 0x7f394006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.882+0000 7f396159b700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39400682f0 0x7f394006a7a0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f3958009b90 tx=0x7f395801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:55.885+0000 7f3951ffb700 1 -- 172.21.15.102:0/2861227607 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39540521c0 con 0x7f395c1593c0 2023-03-03T13:48:56.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.117+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f395c04e6e0 con 0x7f395c1593c0 2023-03-03T13:48:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.117+0000 7f3951ffb700 1 -- 172.21.15.102:0/2861227607 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f39540557e0 con 0x7f395c1593c0 2023-03-03T13:48:56.120 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:48:56.120 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:48:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39400682f0 msgr2=0x7f394006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39400682f0 0x7f394006a7a0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f3958009b90 tx=0x7f395801c040 comp rx=0 tx=0).stop 2023-03-03T13:48:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c1593c0 msgr2=0x7f395c1506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c1593c0 0x7f395c1506d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f395400cae0 tx=0x7f395400cdf0 comp rx=0 tx=0).stop 2023-03-03T13:48:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 shutdown_connections 2023-03-03T13:48:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f39400682f0 0x7f394006a7a0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f395c0cd8e0 0x7f395c0b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f395c1593c0 0x7f395c1506d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 --2- 172.21.15.102:0/2861227607 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395c153290 0x7f395c150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 >> 172.21.15.102:0/2861227607 conn(0x7f395c065fb0 msgr2=0x7f395c157ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 shutdown_connections 2023-03-03T13:48:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:48:56.120+0000 7f39637ff700 1 -- 172.21.15.102:0/2861227607 wait complete. 2023-03-03T13:48:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:48:56.149 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T13:48:57.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:57.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:27.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:27.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:57.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:57.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T13:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:50:28.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:28.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:34.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:50:34.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.398+0000 7f7601bff700 1 Processor -- start 2023-03-03T13:50:34.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.398+0000 7f7601bff700 1 -- start start 2023-03-03T13:50:34.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.399+0000 7f7601bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f75fc152e50 0x7f75fc0c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:50:34.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f7601bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc0c9590 con 0x7f75fc0c8eb0 2023-03-03T13:50:34.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f7601bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc0ca400 con 0x7f75fc0c9f60 2023-03-03T13:50:34.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f7601bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75fc0cda80 con 0x7f75fc152e50 2023-03-03T13:50:34.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75fb7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f75fc152e50 0x7f75fc0c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:50:34.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75f3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f75fc0c8eb0 0x7f75fc0ccc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49262/0 (socket says 172.21.15.102:49262) 2023-03-03T13:50:34.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75f3fff700 1 -- 172.21.15.102:0/431610942 learned_addr learned my addr 172.21.15.102:0/431610942 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:50:34.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75fbfff700 1 --1- 172.21.15.102:0/431610942 >> v1:172.21.15.172:6789/0 conn(0x7f75fc0c9f60 0x7f75fc0cd370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:32986/0 (socket says 172.21.15.102:32986) 2023-03-03T13:50:34.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75fb7fe700 1 -- 172.21.15.102:0/431610942 >> v1:172.21.15.172:6789/0 conn(0x7f75fc0c9f60 legacy=0x7f75fc0cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:50:34.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75fb7fe700 1 -- 172.21.15.102:0/431610942 >> v1:172.21.15.102:6789/0 conn(0x7f75fc0c8eb0 legacy=0x7f75fc0ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:50:34.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75fb7fe700 1 -- 172.21.15.102:0/431610942 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc159880 con 0x7f75fc152e50 2023-03-03T13:50:34.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.400+0000 7f75faffd700 1 -- 172.21.15.102:0/431610942 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612723731 0 0) 0x7f75fc0ca400 con 0x7f75fc0c9f60 2023-03-03T13:50:34.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.401+0000 7f75fb7fe700 1 --2- 172.21.15.102:0/431610942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc0c9a20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009840 tx=0x7f75ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb9880f14460d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:50:34.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.401+0000 7f75faffd700 1 -- 172.21.15.102:0/431610942 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec011830 con 0x7f75fc152e50 2023-03-03T13:50:34.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.401+0000 7f75faffd700 1 -- 172.21.15.102:0/431610942 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec011990 con 0x7f75fc152e50 2023-03-03T13:50:34.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.401+0000 7f75faffd700 1 -- 172.21.15.102:0/431610942 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec011d20 con 0x7f75fc152e50 2023-03-03T13:50:34.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.402+0000 7f7601bff700 1 -- 172.21.15.102:0/431610942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 msgr2=0x7f75fc0c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:50:34.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.402+0000 7f7601bff700 1 --2- 172.21.15.102:0/431610942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc0c9a20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009840 tx=0x7f75ec009b50 comp rx=0 tx=0).stop 2023-03-03T13:50:34.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.402+0000 7f7601bff700 1 -- 172.21.15.102:0/431610942 shutdown_connections 2023-03-03T13:50:34.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.402+0000 7f7601bff700 1 --2- 172.21.15.102:0/431610942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc0c9a20 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.402+0000 7f7601bff700 1 -- 172.21.15.102:0/431610942 >> 172.21.15.102:0/431610942 conn(0x7f75fc0704c0 msgr2=0x7f75fc0708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:50:34.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.402+0000 7f7601bff700 1 -- 172.21.15.102:0/431610942 shutdown_connections 2023-03-03T13:50:34.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.402+0000 7f7601bff700 1 -- 172.21.15.102:0/431610942 wait complete. 2023-03-03T13:50:34.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.404+0000 7f7601bff700 1 Processor -- start 2023-03-03T13:50:34.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.404+0000 7f7601bff700 1 -- start start 2023-03-03T13:50:34.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f7601bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75fc0c8eb0 0x7f75fc1f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:50:34.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f7601bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75fc0c9f60 0x7f75fc0b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:50:34.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f7601bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:50:34.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f7601bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0b23d0 con 0x7f75fc152e50 2023-03-03T13:50:34.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f7601bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0b20a0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f7601bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0b00c0 con 0x7f75fc0c8eb0 2023-03-03T13:50:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f75fbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:50:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f75f3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75fc0c9f60 0x7f75fc0b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:50:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f75fb7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75fc0c8eb0 0x7f75fc1f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:50:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f75f3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75fc0c9f60 0x7f75fc0b2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35782/0 (socket says 172.21.15.102:35782) 2023-03-03T13:50:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f75f3fff700 1 -- 172.21.15.102:0/157449669 learned_addr learned my addr 172.21.15.102:0/157449669 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:50:34.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.405+0000 7f75fbfff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc158600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46606/0 (socket says 172.21.15.102:46606) 2023-03-03T13:50:34.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f3fff700 1 -- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75fc0c8eb0 msgr2=0x7f75fc1f1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:50:34.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f3fff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75fc0c8eb0 0x7f75fc1f1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f3fff700 1 -- 172.21.15.102:0/157449669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 msgr2=0x7f75fc158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:50:34.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f3fff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f3fff700 1 -- 172.21.15.102:0/157449669 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec0094a0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f3fff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75fc0c9f60 0x7f75fc0b2860 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f75e400c490 tx=0x7f75e400c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:50:34.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f8ff9700 1 -- 172.21.15.102:0/157449669 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e400cea0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.406+0000 7f75f8ff9700 1 -- 172.21.15.102:0/157449669 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4005d70 con 0x7f75fc0c9f60 2023-03-03T13:50:34.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.407+0000 7f75f8ff9700 1 -- 172.21.15.102:0/157449669 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e40072f0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.407+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc0c7dc0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.407+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc0c7ae0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.408+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75fc0c80a0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.409+0000 7f75f8ff9700 1 -- 172.21.15.102:0/157449669 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f75e4019030 con 0x7f75fc0c9f60 2023-03-03T13:50:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.410+0000 7f75f8ff9700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75dc0682f0 0x7f75dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:50:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.410+0000 7f75f8ff9700 1 -- 172.21.15.102:0/157449669 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f75e4084fd0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.411+0000 7f75fb7fe700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75dc0682f0 0x7f75dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:50:34.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.411+0000 7f75fb7fe700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75dc0682f0 0x7f75dc06a7a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0090b0 tx=0x7f75ec01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:50:34.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.415+0000 7f75f8ff9700 1 -- 172.21.15.102:0/157449669 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f75e40516e0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.650+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75fc04e6e0 con 0x7f75fc0c9f60 2023-03-03T13:50:34.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.652+0000 7f75f8ff9700 1 -- 172.21.15.102:0/157449669 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f75e4017020 con 0x7f75fc0c9f60 2023-03-03T13:50:34.654 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:50:34.654 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:50:34.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.653+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75dc0682f0 msgr2=0x7f75dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:50:34.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.653+0000 7f7601bff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75dc0682f0 0x7f75dc06a7a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0090b0 tx=0x7f75ec01c040 comp rx=0 tx=0).stop 2023-03-03T13:50:34.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.653+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75fc0c9f60 msgr2=0x7f75fc0b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:50:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.653+0000 7f7601bff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75fc0c9f60 0x7f75fc0b2860 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f75e400c490 tx=0x7f75e400c7a0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 shutdown_connections 2023-03-03T13:50:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f75dc0682f0 0x7f75dc06a7a0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f75fc0c8eb0 0x7f75fc1f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75fc152e50 0x7f75fc158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 --2- 172.21.15.102:0/157449669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75fc0c9f60 0x7f75fc0b2860 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:50:34.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 >> 172.21.15.102:0/157449669 conn(0x7f75fc0704c0 msgr2=0x7f75fc065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:50:34.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 shutdown_connections 2023-03-03T13:50:34.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:50:34.654+0000 7f7601bff700 1 -- 172.21.15.102:0/157449669 wait complete. 2023-03-03T13:50:34.658 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:50:34.683 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-03-03T13:50:58.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:58.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:28.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:28.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:47.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:51:47.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.943+0000 7fcfb1d9d700 1 Processor -- start 2023-03-03T13:51:47.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.943+0000 7fcfb1d9d700 1 -- start start 2023-03-03T13:51:47.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.944+0000 7fcfb1d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfac0c9610 0x7fcfac0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:47.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.944+0000 7fcfb1d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac0c9ca0 con 0x7fcfac1564b0 2023-03-03T13:51:47.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.944+0000 7fcfb1d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac0b46f0 con 0x7fcfac0ca980 2023-03-03T13:51:47.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.944+0000 7fcfb1d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac0b4920 con 0x7fcfac0c9610 2023-03-03T13:51:47.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.944+0000 7fcfabfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcfac0c9610 0x7fcfac0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:47.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.944+0000 7fcfb0d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcfac1564b0 0x7fcfac0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47176/0 (socket says 172.21.15.102:47176) 2023-03-03T13:51:47.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.944+0000 7fcfb0d9b700 1 -- 172.21.15.102:0/2056592495 learned_addr learned my addr 172.21.15.102:0/2056592495 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:51:47.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.945+0000 7fcfabfff700 1 -- 172.21.15.102:0/2056592495 >> v1:172.21.15.172:6789/0 conn(0x7fcfac0ca980 legacy=0x7fcfac1636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:47.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.945+0000 7fcfabfff700 1 -- 172.21.15.102:0/2056592495 >> v1:172.21.15.102:6789/0 conn(0x7fcfac1564b0 legacy=0x7fcfac0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:47.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.945+0000 7fcfabfff700 1 -- 172.21.15.102:0/2056592495 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac0b4c00 con 0x7fcfac0c9610 2023-03-03T13:51:47.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.945+0000 7fcfabfff700 1 --2- 172.21.15.102:0/2056592495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0c9610 0x7fcfac0ca440 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009840 tx=0x7fcf98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615ba94c8c22c1b9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:47.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.945+0000 7fcfab7fe700 1 -- 172.21.15.102:0/2056592495 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011830 con 0x7fcfac0c9610 2023-03-03T13:51:47.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfab7fe700 1 -- 172.21.15.102:0/2056592495 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf98011990 con 0x7fcfac0c9610 2023-03-03T13:51:47.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfab7fe700 1 -- 172.21.15.102:0/2056592495 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011d20 con 0x7fcfac0c9610 2023-03-03T13:51:47.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/2056592495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0c9610 msgr2=0x7fcfac0ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:47.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/2056592495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0c9610 0x7fcfac0ca440 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009840 tx=0x7fcf98009b50 comp rx=0 tx=0).stop 2023-03-03T13:51:47.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/2056592495 shutdown_connections 2023-03-03T13:51:47.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/2056592495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0c9610 0x7fcfac0ca440 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:47.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/2056592495 >> 172.21.15.102:0/2056592495 conn(0x7fcfac065fb0 msgr2=0x7fcfac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:51:47.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/2056592495 shutdown_connections 2023-03-03T13:51:47.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.946+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/2056592495 wait complete. 2023-03-03T13:51:47.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.949+0000 7fcfb1d9d700 1 Processor -- start 2023-03-03T13:51:47.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.949+0000 7fcfb1d9d700 1 -- start start 2023-03-03T13:51:47.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb1d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac0c9610 0x7fcfac0c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:47.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb1d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0ca980 0x7fcfac15b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:47.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb1d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac1564b0 0x7fcfac1f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:47.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb1d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0c7780 con 0x7fcfac0ca980 2023-03-03T13:51:47.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb1d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac1543b0 con 0x7fcfac1564b0 2023-03-03T13:51:47.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb1d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac154110 con 0x7fcfac0c9610 2023-03-03T13:51:47.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb0d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac0c9610 0x7fcfac0c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:47.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0ca980 0x7fcfac15b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:47.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb159c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac1564b0 0x7fcfac1f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:47.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0ca980 0x7fcfac15b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57824/0 (socket says 172.21.15.102:57824) 2023-03-03T13:51:47.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb159c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac1564b0 0x7fcfac1f1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46140/0 (socket says 172.21.15.102:46140) 2023-03-03T13:51:47.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfb0d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac0c9610 0x7fcfac0c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59770/0 (socket says 172.21.15.102:59770) 2023-03-03T13:51:47.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.951+0000 7fcfabfff700 1 -- 172.21.15.102:0/541829549 learned_addr learned my addr 172.21.15.102:0/541829549 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:51:47.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb159c700 1 -- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac0c9610 msgr2=0x7fcfac0c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:47.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb159c700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac0c9610 0x7fcfac0c61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:47.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb159c700 1 -- 172.21.15.102:0/541829549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0ca980 msgr2=0x7fcfac15b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:47.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb159c700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0ca980 0x7fcfac15b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb159c700 1 -- 172.21.15.102:0/541829549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980094a0 con 0x7fcfac1564b0 2023-03-03T13:51:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfabfff700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0ca980 0x7fcfac15b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:51:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb159c700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac1564b0 0x7fcfac1f1a50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400cb90 tx=0x7fcfa400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfa97fa700 1 -- 172.21.15.102:0/541829549 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4003c00 con 0x7fcfac1564b0 2023-03-03T13:51:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfa97fa700 1 -- 172.21.15.102:0/541829549 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa4003d60 con 0x7fcfac1564b0 2023-03-03T13:51:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfa97fa700 1 -- 172.21.15.102:0/541829549 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa400aa80 con 0x7fcfac1564b0 2023-03-03T13:51:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac0b2210 con 0x7fcfac1564b0 2023-03-03T13:51:47.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.952+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac0b1f30 con 0x7fcfac1564b0 2023-03-03T13:51:47.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.953+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac156880 con 0x7fcfac1564b0 2023-03-03T13:51:47.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.955+0000 7fcfa97fa700 1 -- 172.21.15.102:0/541829549 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcfa4010050 con 0x7fcfac1564b0 2023-03-03T13:51:47.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.956+0000 7fcfa97fa700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:47.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.956+0000 7fcfa97fa700 1 -- 172.21.15.102:0/541829549 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcfa4085b60 con 0x7fcfac1564b0 2023-03-03T13:51:47.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.956+0000 7fcfb0d9b700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:47.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.957+0000 7fcfb0d9b700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0005910 tx=0x7fcfa0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:47.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:47.960+0000 7fcfa97fa700 1 -- 172.21.15.102:0/541829549 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcfa4052270 con 0x7fcfac1564b0 2023-03-03T13:51:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.192+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfac0cc680 con 0x7fcfac1564b0 2023-03-03T13:51:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.193+0000 7fcfa97fa700 1 -- 172.21.15.102:0/541829549 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcfa4055890 con 0x7fcfac1564b0 2023-03-03T13:51:48.195 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:51:48.196 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:51:48.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 msgr2=0x7fcf9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:48.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0005910 tx=0x7fcfa0009040 comp rx=0 tx=0).stop 2023-03-03T13:51:48.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac1564b0 msgr2=0x7fcfac1f1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:48.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac1564b0 0x7fcfac1f1a50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400cb90 tx=0x7fcfa400cea0 comp rx=0 tx=0).stop 2023-03-03T13:51:48.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 shutdown_connections 2023-03-03T13:51:48.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcf900682f0 0x7fcf9006a7a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:48.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcfac0c9610 0x7fcfac0c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:48.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcfac1564b0 0x7fcfac1f1a50 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:48.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 --2- 172.21.15.102:0/541829549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfac0ca980 0x7fcfac15b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:48.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 >> 172.21.15.102:0/541829549 conn(0x7fcfac065fb0 msgr2=0x7fcfac066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:51:48.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 shutdown_connections 2023-03-03T13:51:48.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:51:48.196+0000 7fcfb1d9d700 1 -- 172.21.15.102:0/541829549 wait complete. 2023-03-03T13:51:48.200 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:51:48.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T13:51:58.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:58.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:28.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:28.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:32.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:52:32.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.486+0000 7febf4b9d700 1 Processor -- start 2023-03-03T13:52:32.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.486+0000 7febf4b9d700 1 -- start start 2023-03-03T13:52:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.487+0000 7febf4b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7febf00ced00 0x7febf00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.487+0000 7febf4b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf00c3d20 con 0x7febf00b3970 2023-03-03T13:52:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.487+0000 7febf4b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf015cba0 con 0x7febf00c41b0 2023-03-03T13:52:32.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.487+0000 7febf4b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf00c4930 con 0x7febf00ced00 2023-03-03T13:52:32.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.488+0000 7febef7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7febf00ced00 0x7febf00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:32.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.488+0000 7febeffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7febf00c41b0 0x7febf0163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42294/0 (socket says 172.21.15.102:42294) 2023-03-03T13:52:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.488+0000 7febeffff700 1 -- 172.21.15.102:0/1181712298 learned_addr learned my addr 172.21.15.102:0/1181712298 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:52:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.488+0000 7febef7fe700 1 -- 172.21.15.102:0/1181712298 >> v1:172.21.15.172:6789/0 conn(0x7febf00c41b0 legacy=0x7febf0163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.488+0000 7febef7fe700 1 -- 172.21.15.102:0/1181712298 >> v1:172.21.15.102:6789/0 conn(0x7febf00b3970 legacy=0x7febf0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.488+0000 7febef7fe700 1 -- 172.21.15.102:0/1181712298 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf015ce80 con 0x7febf00ced00 2023-03-03T13:52:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.488+0000 7febef7fe700 1 --2- 172.21.15.102:0/1181712298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00ced00 0x7febf00c5a90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7febe0009840 tx=0x7febe0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbbd68b0b156aa86 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febee7fc700 1 -- 172.21.15.102:0/1181712298 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0011830 con 0x7febf00ced00 2023-03-03T13:52:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febee7fc700 1 -- 172.21.15.102:0/1181712298 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe0011990 con 0x7febf00ced00 2023-03-03T13:52:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febee7fc700 1 -- 172.21.15.102:0/1181712298 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0011d20 con 0x7febf00ced00 2023-03-03T13:52:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febf4b9d700 1 -- 172.21.15.102:0/1181712298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00ced00 msgr2=0x7febf00c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febf4b9d700 1 --2- 172.21.15.102:0/1181712298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00ced00 0x7febf00c5a90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7febe0009840 tx=0x7febe0009b50 comp rx=0 tx=0).stop 2023-03-03T13:52:32.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febf4b9d700 1 -- 172.21.15.102:0/1181712298 shutdown_connections 2023-03-03T13:52:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febf4b9d700 1 --2- 172.21.15.102:0/1181712298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00ced00 0x7febf00c5a90 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febf4b9d700 1 -- 172.21.15.102:0/1181712298 >> 172.21.15.102:0/1181712298 conn(0x7febf008b1c0 msgr2=0x7febf0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:52:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.489+0000 7febf4b9d700 1 -- 172.21.15.102:0/1181712298 shutdown_connections 2023-03-03T13:52:32.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.490+0000 7febf4b9d700 1 -- 172.21.15.102:0/1181712298 wait complete. 2023-03-03T13:52:32.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.491+0000 7febf4b9d700 1 Processor -- start 2023-03-03T13:52:32.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.492+0000 7febf4b9d700 1 -- start start 2023-03-03T13:52:32.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febf4b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00b3970 0x7febf01ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:32.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febf4b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00c41b0 0x7febf0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:32.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febf4b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf00ced00 0x7febf015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:32.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febf4b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf0152790 con 0x7febf00c41b0 2023-03-03T13:52:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febf4b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf015a9b0 con 0x7febf00b3970 2023-03-03T13:52:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febf4b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf01521d0 con 0x7febf00ced00 2023-03-03T13:52:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febeeffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00c41b0 0x7febf0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febeeffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00c41b0 0x7febf0159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45610/0 (socket says 172.21.15.102:45610) 2023-03-03T13:52:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febeeffd700 1 -- 172.21.15.102:0/249258221 learned_addr learned my addr 172.21.15.102:0/249258221 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:52:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febef7fe700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00b3970 0x7febf01ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:32.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.493+0000 7febeffff700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf00ced00 0x7febf015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febeeffd700 1 -- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf00ced00 msgr2=0x7febf015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febeeffd700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf00ced00 0x7febf015a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febeeffd700 1 -- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00b3970 msgr2=0x7febf01ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febeeffd700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00b3970 0x7febf01ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febeeffd700 1 -- 172.21.15.102:0/249258221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe00094a0 con 0x7febf00c41b0 2023-03-03T13:52:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febef7fe700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00b3970 0x7febf01ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:52:32.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febeeffd700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00c41b0 0x7febf0159f30 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7febe400c450 tx=0x7febe400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:32.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febd7fff700 1 -- 172.21.15.102:0/249258221 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe40052d0 con 0x7febf00c41b0 2023-03-03T13:52:32.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febd7fff700 1 -- 172.21.15.102:0/249258221 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe4005430 con 0x7febf00c41b0 2023-03-03T13:52:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febd7fff700 1 -- 172.21.15.102:0/249258221 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe40038c0 con 0x7febf00c41b0 2023-03-03T13:52:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.494+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf006c200 con 0x7febf00c41b0 2023-03-03T13:52:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.495+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf006bf20 con 0x7febf00c41b0 2023-03-03T13:52:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.496+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf015c890 con 0x7febf00c41b0 2023-03-03T13:52:32.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.497+0000 7febd7fff700 1 -- 172.21.15.102:0/249258221 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7febe4019030 con 0x7febf00c41b0 2023-03-03T13:52:32.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.499+0000 7febd7fff700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7febdc068320 0x7febdc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:32.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.499+0000 7febd7fff700 1 -- 172.21.15.102:0/249258221 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7febe4084f70 con 0x7febf00c41b0 2023-03-03T13:52:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.499+0000 7febef7fe700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7febdc068320 0x7febdc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.500+0000 7febef7fe700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7febdc068320 0x7febdc06a7d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7febe00090b0 tx=0x7febe0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:32.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.503+0000 7febd7fff700 1 -- 172.21.15.102:0/249258221 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7febe4051680 con 0x7febf00c41b0 2023-03-03T13:52:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.739+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febf004e6e0 con 0x7febf00c41b0 2023-03-03T13:52:32.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.740+0000 7febd7fff700 1 -- 172.21.15.102:0/249258221 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7febe4017090 con 0x7febf00c41b0 2023-03-03T13:52:32.742 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:52:32.742 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:52:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7febdc068320 msgr2=0x7febdc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7febdc068320 0x7febdc06a7d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7febe00090b0 tx=0x7febe0008030 comp rx=0 tx=0).stop 2023-03-03T13:52:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00c41b0 msgr2=0x7febf0159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00c41b0 0x7febf0159f30 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7febe400c450 tx=0x7febe400c760 comp rx=0 tx=0).stop 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 shutdown_connections 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7febdc068320 0x7febdc06a7d0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7febf00ced00 0x7febf015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.742+0000 7febf4b9d700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febf00b3970 0x7febf01ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.743+0000 7febf4b9d700 1 --2- 172.21.15.102:0/249258221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00c41b0 0x7febf0159f30 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.743+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 >> 172.21.15.102:0/249258221 conn(0x7febf008b1c0 msgr2=0x7febf0066930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:52:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.743+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 shutdown_connections 2023-03-03T13:52:32.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:32.743+0000 7febf4b9d700 1 -- 172.21.15.102:0/249258221 wait complete. 2023-03-03T13:52:32.746 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:52:32.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-03-03T13:52:42.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:52:43.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.022+0000 7f5e849ff700 1 Processor -- start 2023-03-03T13:52:43.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.022+0000 7f5e849ff700 1 -- start start 2023-03-03T13:52:43.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.024+0000 7f5e849ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5e80153290 0x7f5e80158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:43.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.024+0000 7f5e849ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e80153660 con 0x7f5e800cd8e0 2023-03-03T13:52:43.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.024+0000 7f5e849ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8006c360 con 0x7f5e801593c0 2023-03-03T13:52:43.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.024+0000 7f5e849ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e800cec40 con 0x7f5e80153290 2023-03-03T13:52:43.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.024+0000 7f5e7ed9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5e800cd8e0 0x7f5e800ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44004/0 (socket says 172.21.15.102:44004) 2023-03-03T13:52:43.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.024+0000 7f5e7ed9d700 1 -- 172.21.15.102:0/3164875887 learned_addr learned my addr 172.21.15.102:0/3164875887 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:52:43.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.024+0000 7f5e7e59c700 1 --2- 172.21.15.102:0/3164875887 >> v2:172.21.15.172:3300/0 conn(0x7f5e80153290 0x7f5e80158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:43.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.025+0000 7f5e7d59a700 1 -- 172.21.15.102:0/3164875887 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797717465 0 0) 0x7f5e80153660 con 0x7f5e800cd8e0 2023-03-03T13:52:43.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.025+0000 7f5e7d59a700 1 -- 172.21.15.102:0/3164875887 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e680035b0 con 0x7f5e800cd8e0 2023-03-03T13:52:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.025+0000 7f5e7e59c700 1 -- 172.21.15.102:0/3164875887 >> v1:172.21.15.172:6789/0 conn(0x7f5e801593c0 legacy=0x7f5e800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.025+0000 7f5e7e59c700 1 -- 172.21.15.102:0/3164875887 >> v1:172.21.15.102:6789/0 conn(0x7f5e800cd8e0 legacy=0x7f5e800ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.025+0000 7f5e7e59c700 1 -- 172.21.15.102:0/3164875887 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8015cea0 con 0x7f5e80153290 2023-03-03T13:52:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.025+0000 7f5e7e59c700 1 --2- 172.21.15.102:0/3164875887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e80153290 0x7f5e80158e80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009840 tx=0x7f5e70009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcacc86b3484646c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.025+0000 7f5e7d59a700 1 -- 172.21.15.102:0/3164875887 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70011830 con 0x7f5e80153290 2023-03-03T13:52:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e7d59a700 1 -- 172.21.15.102:0/3164875887 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e70011990 con 0x7f5e80153290 2023-03-03T13:52:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e7d59a700 1 -- 172.21.15.102:0/3164875887 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70011d20 con 0x7f5e80153290 2023-03-03T13:52:43.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e849ff700 1 -- 172.21.15.102:0/3164875887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e80153290 msgr2=0x7f5e80158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:43.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e849ff700 1 --2- 172.21.15.102:0/3164875887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e80153290 0x7f5e80158e80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009840 tx=0x7f5e70009b50 comp rx=0 tx=0).stop 2023-03-03T13:52:43.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e849ff700 1 -- 172.21.15.102:0/3164875887 shutdown_connections 2023-03-03T13:52:43.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e849ff700 1 --2- 172.21.15.102:0/3164875887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e80153290 0x7f5e80158e80 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:43.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e849ff700 1 -- 172.21.15.102:0/3164875887 >> 172.21.15.102:0/3164875887 conn(0x7f5e80065fb0 msgr2=0x7f5e800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:52:43.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e849ff700 1 -- 172.21.15.102:0/3164875887 shutdown_connections 2023-03-03T13:52:43.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.026+0000 7f5e849ff700 1 -- 172.21.15.102:0/3164875887 wait complete. 2023-03-03T13:52:43.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.028+0000 7f5e849ff700 1 Processor -- start 2023-03-03T13:52:43.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.028+0000 7f5e849ff700 1 -- start start 2023-03-03T13:52:43.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.029+0000 7f5e849ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e800cd8e0 0x7f5e801e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:43.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.029+0000 7f5e849ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e80153290 0x7f5e800b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:43.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.029+0000 7f5e849ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e801593c0 0x7f5e800b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:43.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.029+0000 7f5e849ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800b3de0 con 0x7f5e80153290 2023-03-03T13:52:43.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.029+0000 7f5e849ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801e5060 con 0x7f5e800cd8e0 2023-03-03T13:52:43.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.029+0000 7f5e849ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800cbc90 con 0x7f5e801593c0 2023-03-03T13:52:43.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7dd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e80153290 0x7f5e800b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:43.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e800cd8e0 0x7f5e801e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:43.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7ed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e801593c0 0x7f5e800b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:43.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e800cd8e0 0x7f5e801e2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34240/0 (socket says 172.21.15.102:34240) 2023-03-03T13:52:43.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 -- 172.21.15.102:0/302975693 learned_addr learned my addr 172.21.15.102:0/302975693 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:52:43.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 -- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e801593c0 msgr2=0x7f5e800b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:43.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e801593c0 0x7f5e800b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:43.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 -- 172.21.15.102:0/302975693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e80153290 msgr2=0x7f5e800b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:43.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e80153290 0x7f5e800b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:43.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.030+0000 7f5e7e59c700 1 -- 172.21.15.102:0/302975693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e700094a0 con 0x7f5e800cd8e0 2023-03-03T13:52:43.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.031+0000 7f5e7e59c700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e800cd8e0 0x7f5e801e2300 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009b90 tx=0x7f5e70004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:43.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.031+0000 7f5e6effd700 1 -- 172.21.15.102:0/302975693 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e700036a0 con 0x7f5e800cd8e0 2023-03-03T13:52:43.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.031+0000 7f5e6effd700 1 -- 172.21.15.102:0/302975693 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7001e070 con 0x7f5e800cd8e0 2023-03-03T13:52:43.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.031+0000 7f5e6effd700 1 -- 172.21.15.102:0/302975693 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70003a60 con 0x7f5e800cd8e0 2023-03-03T13:52:43.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.031+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8015d160 con 0x7f5e800cd8e0 2023-03-03T13:52:43.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.031+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8006c7f0 con 0x7f5e800cd8e0 2023-03-03T13:52:43.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.032+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e800ccde0 con 0x7f5e800cd8e0 2023-03-03T13:52:43.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.033+0000 7f5e6effd700 1 -- 172.21.15.102:0/302975693 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5e7000f050 con 0x7f5e800cd8e0 2023-03-03T13:52:43.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.034+0000 7f5e6effd700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e600682f0 0x7f5e6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:43.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.035+0000 7f5e6effd700 1 -- 172.21.15.102:0/302975693 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5e70086c90 con 0x7f5e800cd8e0 2023-03-03T13:52:43.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.035+0000 7f5e7dd9b700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e600682f0 0x7f5e6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:43.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.035+0000 7f5e7dd9b700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e600682f0 0x7f5e6006a7a0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f5e800cd370 tx=0x7f5e74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:43.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.039+0000 7f5e6effd700 1 -- 172.21.15.102:0/302975693 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e700533a0 con 0x7f5e800cd8e0 2023-03-03T13:52:43.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.274+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e8004e6e0 con 0x7f5e800cd8e0 2023-03-03T13:52:43.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.275+0000 7f5e6effd700 1 -- 172.21.15.102:0/302975693 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5e70020090 con 0x7f5e800cd8e0 2023-03-03T13:52:43.278 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:52:43.278 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:52:43.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.277+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e600682f0 msgr2=0x7f5e6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:43.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.277+0000 7f5e849ff700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e600682f0 0x7f5e6006a7a0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f5e800cd370 tx=0x7f5e74009040 comp rx=0 tx=0).stop 2023-03-03T13:52:43.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.277+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e800cd8e0 msgr2=0x7f5e801e2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:43.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.277+0000 7f5e849ff700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e800cd8e0 0x7f5e801e2300 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009b90 tx=0x7f5e70004750 comp rx=0 tx=0).stop 2023-03-03T13:52:43.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 shutdown_connections 2023-03-03T13:52:43.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e600682f0 0x7f5e6006a7a0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:43.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e801593c0 0x7f5e800b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:43.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e800cd8e0 0x7f5e801e2300 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:43.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 --2- 172.21.15.102:0/302975693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e80153290 0x7f5e800b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:43.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 >> 172.21.15.102:0/302975693 conn(0x7f5e80065fb0 msgr2=0x7f5e8015bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:52:43.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 shutdown_connections 2023-03-03T13:52:43.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:52:43.278+0000 7f5e849ff700 1 -- 172.21.15.102:0/302975693 wait complete. 2023-03-03T13:52:43.282 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:52:43.307 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T13:52:58.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:58.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:18.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:53:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.556+0000 7f97185ff700 1 Processor -- start 2023-03-03T13:53:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.556+0000 7f97185ff700 1 -- start start 2023-03-03T13:53:18.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f97185ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9710153290 0x7f9710158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f97185ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f97100cd8e0 0x7f97100ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f97185ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9710153660 con 0x7f97101593c0 2023-03-03T13:53:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f97185ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971006c360 con 0x7f9710153290 2023-03-03T13:53:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f97185ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971015cea0 con 0x7f97100cd8e0 2023-03-03T13:53:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f971639b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9710153290 0x7f9710158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f971639b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9710153290 0x7f9710158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51938/0 (socket says 172.21.15.102:51938) 2023-03-03T13:53:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f971639b700 1 -- 172.21.15.102:0/1942171327 learned_addr learned my addr 172.21.15.102:0/1942171327 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:53:18.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f9716b9c700 1 --2- 172.21.15.102:0/1942171327 >> v2:172.21.15.172:3300/0 conn(0x7f97100cd8e0 0x7f97100ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f971639b700 1 -- 172.21.15.102:0/1942171327 >> v2:172.21.15.172:3300/0 conn(0x7f97100cd8e0 msgr2=0x7f97100ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:53:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f971639b700 1 --2- 172.21.15.102:0/1942171327 >> v2:172.21.15.172:3300/0 conn(0x7f97100cd8e0 0x7f97100ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f971639b700 1 -- 172.21.15.102:0/1942171327 >> v1:172.21.15.172:6789/0 conn(0x7f97101593c0 legacy=0x7f97100cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.558+0000 7f9715399700 1 -- 172.21.15.102:0/1942171327 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474922785 0 0) 0x7f9710153660 con 0x7f97101593c0 2023-03-03T13:53:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.559+0000 7f971639b700 1 -- 172.21.15.102:0/1942171327 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971006c7f0 con 0x7f9710153290 2023-03-03T13:53:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.559+0000 7f971639b700 1 --2- 172.21.15.102:0/1942171327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710153290 0x7f9710158e80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f97100c9390 tx=0x7f970c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dedfae82e314a20 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.559+0000 7f9715399700 1 -- 172.21.15.102:0/1942171327 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c011830 con 0x7f9710153290 2023-03-03T13:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.559+0000 7f9715399700 1 -- 172.21.15.102:0/1942171327 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970c011990 con 0x7f9710153290 2023-03-03T13:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.559+0000 7f9715399700 1 -- 172.21.15.102:0/1942171327 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c011d20 con 0x7f9710153290 2023-03-03T13:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 -- 172.21.15.102:0/1942171327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710153290 msgr2=0x7f9710158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 --2- 172.21.15.102:0/1942171327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710153290 0x7f9710158e80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f97100c9390 tx=0x7f970c009bd0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 -- 172.21.15.102:0/1942171327 shutdown_connections 2023-03-03T13:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 --2- 172.21.15.102:0/1942171327 >> v2:172.21.15.172:3300/0 conn(0x7f97100cd8e0 0x7f97100ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 --2- 172.21.15.102:0/1942171327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710153290 0x7f9710158e80 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 -- 172.21.15.102:0/1942171327 >> 172.21.15.102:0/1942171327 conn(0x7f9710065fb0 msgr2=0x7f97100663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:53:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 -- 172.21.15.102:0/1942171327 shutdown_connections 2023-03-03T13:53:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.560+0000 7f97185ff700 1 -- 172.21.15.102:0/1942171327 wait complete. 2023-03-03T13:53:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.562+0000 7f97185ff700 1 Processor -- start 2023-03-03T13:53:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.562+0000 7f97185ff700 1 -- start start 2023-03-03T13:53:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f97185ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97100cd8e0 0x7f971014f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f97185ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9710153290 0x7f97101e5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f97185ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97101593c0 0x7f97100c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f97185ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971015b690 con 0x7f97101593c0 2023-03-03T13:53:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f97185ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100c7e60 con 0x7f9710153290 2023-03-03T13:53:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f97185ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971015aa10 con 0x7f97100cd8e0 2023-03-03T13:53:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f9716b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97101593c0 0x7f97100c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f971639b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97100cd8e0 0x7f971014f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f9715b9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9710153290 0x7f97101e5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f971639b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97100cd8e0 0x7f971014f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45896/0 (socket says 172.21.15.102:45896) 2023-03-03T13:53:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.563+0000 7f971639b700 1 -- 172.21.15.102:0/1588573145 learned_addr learned my addr 172.21.15.102:0/1588573145 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:53:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.564+0000 7f971639b700 1 -- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9710153290 msgr2=0x7f97101e5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.564+0000 7f971639b700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9710153290 0x7f97101e5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.564+0000 7f971639b700 1 -- 172.21.15.102:0/1588573145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97101593c0 msgr2=0x7f97100c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.564+0000 7f971639b700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97101593c0 0x7f97100c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.564+0000 7f971639b700 1 -- 172.21.15.102:0/1588573145 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c009550 con 0x7f97100cd8e0 2023-03-03T13:53:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.564+0000 7f971639b700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97100cd8e0 0x7f971014f5d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f970c009fd0 tx=0x7f970c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:53:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.564+0000 7f9706ffd700 1 -- 172.21.15.102:0/1588573145 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c004550 con 0x7f97100cd8e0 2023-03-03T13:53:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.565+0000 7f9706ffd700 1 -- 172.21.15.102:0/1588573145 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970c01f070 con 0x7f97100cd8e0 2023-03-03T13:53:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.565+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97100ccde0 con 0x7f97100cd8e0 2023-03-03T13:53:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.565+0000 7f9706ffd700 1 -- 172.21.15.102:0/1588573145 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c0038e0 con 0x7f97100cd8e0 2023-03-03T13:53:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.565+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971015d160 con 0x7f97100cd8e0 2023-03-03T13:53:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.566+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97100cd0c0 con 0x7f97100cd8e0 2023-03-03T13:53:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.567+0000 7f9706ffd700 1 -- 172.21.15.102:0/1588573145 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f970c00f050 con 0x7f97100cd8e0 2023-03-03T13:53:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.568+0000 7f9706ffd700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96f40682f0 0x7f96f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.568+0000 7f9706ffd700 1 -- 172.21.15.102:0/1588573145 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f970c021070 con 0x7f97100cd8e0 2023-03-03T13:53:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.568+0000 7f9715b9a700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96f40682f0 0x7f96f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.569+0000 7f9715b9a700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96f40682f0 0x7f96f406a7a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f97100c9240 tx=0x7f9700009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:53:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.572+0000 7f9706ffd700 1 -- 172.21.15.102:0/1588573145 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f970c053cc0 con 0x7f97100cd8e0 2023-03-03T13:53:18.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.807+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97100c8f90 con 0x7f97100cd8e0 2023-03-03T13:53:18.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.808+0000 7f9706ffd700 1 -- 172.21.15.102:0/1588573145 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f970c015090 con 0x7f97100cd8e0 2023-03-03T13:53:18.810 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:53:18.810 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:53:18.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.810+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96f40682f0 msgr2=0x7f96f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:18.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.810+0000 7f97185ff700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96f40682f0 0x7f96f406a7a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f97100c9240 tx=0x7f9700009040 comp rx=0 tx=0).stop 2023-03-03T13:53:18.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.810+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97100cd8e0 msgr2=0x7f971014f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:18.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.810+0000 7f97185ff700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97100cd8e0 0x7f971014f5d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f970c009fd0 tx=0x7f970c004750 comp rx=0 tx=0).stop 2023-03-03T13:53:18.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 shutdown_connections 2023-03-03T13:53:18.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96f40682f0 0x7f96f406a7a0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97100cd8e0 0x7f971014f5d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97101593c0 0x7f97100c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 --2- 172.21.15.102:0/1588573145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9710153290 0x7f97101e5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:18.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 >> 172.21.15.102:0/1588573145 conn(0x7f9710065fb0 msgr2=0x7f97100cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:53:18.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 shutdown_connections 2023-03-03T13:53:18.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:53:18.811+0000 7f97185ff700 1 -- 172.21.15.102:0/1588573145 wait complete. 2023-03-03T13:53:18.814 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:53:18.839 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T13:53:29.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:29.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:59.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:59.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:16.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:54:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.107+0000 7fc7381ff700 1 Processor -- start 2023-03-03T13:54:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.107+0000 7fc7381ff700 1 -- start start 2023-03-03T13:54:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc7381ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7300ced80 0x7fc7300cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc7381ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7300cb9f0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc7381ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc730159c80 con 0x7fc7301597f0 2023-03-03T13:54:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc7381ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300ca750 con 0x7fc7300ced80 2023-03-03T13:54:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc735f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7300ced80 0x7fc7300cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc73679c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc7300cf9a0 0x7fc730163660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42464/0 (socket says 172.21.15.102:42464) 2023-03-03T13:54:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc73579a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc7301597f0 0x7fc7300cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45990/0 (socket says 172.21.15.102:45990) 2023-03-03T13:54:17.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.109+0000 7fc73679c700 1 -- 172.21.15.102:0/2727604475 learned_addr learned my addr 172.21.15.102:0/2727604475 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:54:17.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc734f99700 1 -- 172.21.15.102:0/2727604475 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560883037 0 0) 0x7fc730159c80 con 0x7fc7301597f0 2023-03-03T13:54:17.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc734f99700 1 -- 172.21.15.102:0/2727604475 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc71c0035b0 con 0x7fc7301597f0 2023-03-03T13:54:17.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc735f9b700 1 -- 172.21.15.102:0/2727604475 >> v1:172.21.15.172:6789/0 conn(0x7fc7301597f0 legacy=0x7fc7300cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:17.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc734f99700 1 -- 172.21.15.102:0/2727604475 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096692189 0 0) 0x7fc7300cb9f0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc735f9b700 1 -- 172.21.15.102:0/2727604475 >> v1:172.21.15.102:6789/0 conn(0x7fc7300cf9a0 legacy=0x7fc730163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc735f9b700 1 -- 172.21.15.102:0/2727604475 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300cade0 con 0x7fc7300ced80 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc735f9b700 1 --2- 172.21.15.102:0/2727604475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7300ced80 0x7fc7300cf150 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009870 tx=0x7fc72c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f466ce61c1579b29 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc734f99700 1 -- 172.21.15.102:0/2727604475 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c011830 con 0x7fc7300ced80 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc734f99700 1 -- 172.21.15.102:0/2727604475 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c011990 con 0x7fc7300ced80 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.110+0000 7fc734f99700 1 -- 172.21.15.102:0/2727604475 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c011d20 con 0x7fc7300ced80 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.111+0000 7fc7381ff700 1 -- 172.21.15.102:0/2727604475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7300ced80 msgr2=0x7fc7300cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.111+0000 7fc7381ff700 1 --2- 172.21.15.102:0/2727604475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7300ced80 0x7fc7300cf150 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009870 tx=0x7fc72c009b80 comp rx=0 tx=0).stop 2023-03-03T13:54:17.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.111+0000 7fc7381ff700 1 -- 172.21.15.102:0/2727604475 shutdown_connections 2023-03-03T13:54:17.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.111+0000 7fc7381ff700 1 --2- 172.21.15.102:0/2727604475 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7300ced80 0x7fc7300cf150 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:17.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.111+0000 7fc7381ff700 1 -- 172.21.15.102:0/2727604475 >> 172.21.15.102:0/2727604475 conn(0x7fc730065fb0 msgr2=0x7fc7300663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:54:17.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.111+0000 7fc7381ff700 1 -- 172.21.15.102:0/2727604475 shutdown_connections 2023-03-03T13:54:17.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.111+0000 7fc7381ff700 1 -- 172.21.15.102:0/2727604475 wait complete. 2023-03-03T13:54:17.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.113+0000 7fc7381ff700 1 Processor -- start 2023-03-03T13:54:17.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.113+0000 7fc7381ff700 1 -- start start 2023-03-03T13:54:17.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.114+0000 7fc7381ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7300ced80 0x7fc730154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:17.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.114+0000 7fc7381ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7300cf9a0 0x7fc73015b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:17.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.114+0000 7fc7381ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7301597f0 0x7fc7300b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:17.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.114+0000 7fc7381ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300b4810 con 0x7fc7300cf9a0 2023-03-03T13:54:17.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.114+0000 7fc7381ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300b3c90 con 0x7fc7300ced80 2023-03-03T13:54:17.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.114+0000 7fc7381ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc730154840 con 0x7fc7301597f0 2023-03-03T13:54:17.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7300cf9a0 0x7fc73015b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:17.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc735f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7300ced80 0x7fc730154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:17.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73679c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7301597f0 0x7fc7300b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:17.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7300cf9a0 0x7fc73015b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35168/0 (socket says 172.21.15.102:35168) 2023-03-03T13:54:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 -- 172.21.15.102:0/232243661 learned_addr learned my addr 172.21.15.102:0/232243661 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:54:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 -- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7301597f0 msgr2=0x7fc7300b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7301597f0 0x7fc7300b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 -- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7300ced80 msgr2=0x7fc730154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7300ced80 0x7fc730154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc73579a700 1 -- 172.21.15.102:0/232243661 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c0094a0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.115+0000 7fc735f9b700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7300ced80 0x7fc730154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:54:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.116+0000 7fc73579a700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7300cf9a0 0x7fc73015b010 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc72000c460 tx=0x7fc72000c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:54:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.116+0000 7fc7267fc700 1 -- 172.21.15.102:0/232243661 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720005cb0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.116+0000 7fc7267fc700 1 -- 172.21.15.102:0/232243661 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc720005e10 con 0x7fc7300cf9a0 2023-03-03T13:54:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.116+0000 7fc7267fc700 1 -- 172.21.15.102:0/232243661 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720007310 con 0x7fc7300cf9a0 2023-03-03T13:54:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.116+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7300cb3a0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.116+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7300cb0c0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.117+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7300cb680 con 0x7fc7300cf9a0 2023-03-03T13:54:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.119+0000 7fc7267fc700 1 -- 172.21.15.102:0/232243661 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc720019030 con 0x7fc7300cf9a0 2023-03-03T13:54:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.120+0000 7fc7267fc700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7140682f0 0x7fc71406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:17.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.120+0000 7fc7267fc700 1 -- 172.21.15.102:0/232243661 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc720084ff0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.120+0000 7fc735f9b700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7140682f0 0x7fc71406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:17.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.121+0000 7fc735f9b700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7140682f0 0x7fc71406a7a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0097c0 tx=0x7fc72c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:54:17.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.124+0000 7fc7267fc700 1 -- 172.21.15.102:0/232243661 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc720051720 con 0x7fc7300cf9a0 2023-03-03T13:54:17.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.358+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7300677f0 con 0x7fc7300cf9a0 2023-03-03T13:54:17.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.358+0000 7fc7267fc700 1 -- 172.21.15.102:0/232243661 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc720017020 con 0x7fc7300cf9a0 2023-03-03T13:54:17.361 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:54:17.361 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:54:17.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.361+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7140682f0 msgr2=0x7fc71406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:17.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.361+0000 7fc7381ff700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7140682f0 0x7fc71406a7a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0097c0 tx=0x7fc72c01c040 comp rx=0 tx=0).stop 2023-03-03T13:54:17.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.361+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7300cf9a0 msgr2=0x7fc73015b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:17.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.361+0000 7fc7381ff700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7300cf9a0 0x7fc73015b010 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc72000c460 tx=0x7fc72000c770 comp rx=0 tx=0).stop 2023-03-03T13:54:17.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 shutdown_connections 2023-03-03T13:54:17.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc7140682f0 0x7fc71406a7a0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:17.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7301597f0 0x7fc7300b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:17.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7300ced80 0x7fc730154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:17.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 --2- 172.21.15.102:0/232243661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7300cf9a0 0x7fc73015b010 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:17.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 >> 172.21.15.102:0/232243661 conn(0x7fc730065fb0 msgr2=0x7fc730151020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:54:17.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 shutdown_connections 2023-03-03T13:54:17.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:54:17.362+0000 7fc7381ff700 1 -- 172.21.15.102:0/232243661 wait complete. 2023-03-03T13:54:17.366 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:54:17.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 76 secs before thrashing 2023-03-03T13:54:29.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:29.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:59.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:59.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:29.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:29.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:33.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:55:33.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.640+0000 7f7b941ff700 1 Processor -- start 2023-03-03T13:55:33.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.640+0000 7f7b941ff700 1 -- start start 2023-03-03T13:55:33.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b941ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c153290 0x7f7b8c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:33.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b941ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b8c1593c0 0x7f7b8c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:33.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b941ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b8c0cd8e0 0x7f7b8c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:33.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b941ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c153660 con 0x7f7b8c0cd8e0 2023-03-03T13:55:33.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b941ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c06c360 con 0x7f7b8c1593c0 2023-03-03T13:55:33.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b941ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c15cea0 con 0x7f7b8c153290 2023-03-03T13:55:33.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b9279c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b8c0cd8e0 0x7f7b8c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:33.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b91f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c153290 0x7f7b8c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:33.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b9179a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b8c1593c0 0x7f7b8c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b9279c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b8c0cd8e0 0x7f7b8c0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48000/0 (socket says 172.21.15.102:48000) 2023-03-03T13:55:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b9279c700 1 -- 172.21.15.102:0/545686491 learned_addr learned my addr 172.21.15.102:0/545686491 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:55:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b9279c700 1 -- 172.21.15.102:0/545686491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c153290 msgr2=0x7f7b8c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b9279c700 1 --2- 172.21.15.102:0/545686491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c153290 0x7f7b8c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.642+0000 7f7b9279c700 1 -- 172.21.15.102:0/545686491 >> v2:172.21.15.172:3300/0 conn(0x7f7b8c1593c0 msgr2=0x7f7b8c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.643+0000 7f7b9279c700 1 --2- 172.21.15.102:0/545686491 >> v2:172.21.15.172:3300/0 conn(0x7f7b8c1593c0 0x7f7b8c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.643+0000 7f7b9279c700 1 -- 172.21.15.102:0/545686491 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c06c7f0 con 0x7f7b8c0cd8e0 2023-03-03T13:55:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.643+0000 7f7b9279c700 1 --2- 172.21.15.102:0/545686491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0ce360 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0b1970 tx=0x7f7b8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d8c1ce49d71150 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.643+0000 7f7b90f99700 1 -- 172.21.15.102:0/545686491 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8400e070 con 0x7f7b8c0cd8e0 2023-03-03T13:55:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.643+0000 7f7b90f99700 1 -- 172.21.15.102:0/545686491 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8400b910 con 0x7f7b8c0cd8e0 2023-03-03T13:55:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.643+0000 7f7b90f99700 1 -- 172.21.15.102:0/545686491 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8400bcc0 con 0x7f7b8c0cd8e0 2023-03-03T13:55:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 -- 172.21.15.102:0/545686491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c0cd8e0 msgr2=0x7f7b8c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 --2- 172.21.15.102:0/545686491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0ce360 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0b1970 tx=0x7f7b8400b330 comp rx=0 tx=0).stop 2023-03-03T13:55:33.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 -- 172.21.15.102:0/545686491 shutdown_connections 2023-03-03T13:55:33.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 --2- 172.21.15.102:0/545686491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0ce360 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 --2- 172.21.15.102:0/545686491 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c153290 0x7f7b8c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 --2- 172.21.15.102:0/545686491 >> v2:172.21.15.172:3300/0 conn(0x7f7b8c1593c0 0x7f7b8c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 -- 172.21.15.102:0/545686491 >> 172.21.15.102:0/545686491 conn(0x7f7b8c065fb0 msgr2=0x7f7b8c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:55:33.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 -- 172.21.15.102:0/545686491 shutdown_connections 2023-03-03T13:55:33.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.644+0000 7f7b941ff700 1 -- 172.21.15.102:0/545686491 wait complete. 2023-03-03T13:55:33.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.646+0000 7f7b941ff700 1 Processor -- start 2023-03-03T13:55:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.646+0000 7f7b941ff700 1 -- start start 2023-03-03T13:55:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.647+0000 7f7b941ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.647+0000 7f7b941ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b8c153290 0x7f7b8c1e5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.647+0000 7f7b941ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c1593c0 0x7f7b8c0c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.647+0000 7f7b941ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c0c85b0 con 0x7f7b8c1593c0 2023-03-03T13:55:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.647+0000 7f7b941ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c0c87e0 con 0x7f7b8c153290 2023-03-03T13:55:33.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.647+0000 7f7b941ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c0c8ac0 con 0x7f7b8c0cd8e0 2023-03-03T13:55:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b91f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9179a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b8c153290 0x7f7b8c1e5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9279c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c1593c0 0x7f7b8c0c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b91f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51996/0 (socket says 172.21.15.102:51996) 2023-03-03T13:55:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b91f9b700 1 -- 172.21.15.102:0/3098326993 learned_addr learned my addr 172.21.15.102:0/3098326993 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:55:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9279c700 1 -- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c0cd8e0 msgr2=0x7f7b8c0c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:33.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9279c700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9279c700 1 -- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b8c153290 msgr2=0x7f7b8c1e5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9279c700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b8c153290 0x7f7b8c1e5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9279c700 1 -- 172.21.15.102:0/3098326993 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84009b30 con 0x7f7b8c1593c0 2023-03-03T13:55:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.648+0000 7f7b9279c700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c1593c0 0x7f7b8c0c6940 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7b84015000 tx=0x7f7b84004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:33.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.649+0000 7f7b827fc700 1 -- 172.21.15.102:0/3098326993 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840099b0 con 0x7f7b8c1593c0 2023-03-03T13:55:33.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.649+0000 7f7b827fc700 1 -- 172.21.15.102:0/3098326993 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8400ce80 con 0x7f7b8c1593c0 2023-03-03T13:55:33.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.649+0000 7f7b827fc700 1 -- 172.21.15.102:0/3098326993 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84004490 con 0x7f7b8c1593c0 2023-03-03T13:55:33.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.649+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c0ccde0 con 0x7f7b8c1593c0 2023-03-03T13:55:33.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.649+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c15d160 con 0x7f7b8c1593c0 2023-03-03T13:55:33.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.650+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c0cd0c0 con 0x7f7b8c1593c0 2023-03-03T13:55:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.651+0000 7f7b827fc700 1 -- 172.21.15.102:0/3098326993 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7b840045f0 con 0x7f7b8c1593c0 2023-03-03T13:55:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.652+0000 7f7b827fc700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b700682f0 0x7f7b7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.652+0000 7f7b827fc700 1 -- 172.21.15.102:0/3098326993 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7b84086e80 con 0x7f7b8c1593c0 2023-03-03T13:55:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.653+0000 7f7b91f9b700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b700682f0 0x7f7b7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:33.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.653+0000 7f7b91f9b700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b700682f0 0x7f7b7006a7a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009c00 tx=0x7f7b88009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:33.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.657+0000 7f7b827fc700 1 -- 172.21.15.102:0/3098326993 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7b84053590 con 0x7f7b8c1593c0 2023-03-03T13:55:33.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.890+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b8c04e6e0 con 0x7f7b8c1593c0 2023-03-03T13:55:33.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.891+0000 7f7b827fc700 1 -- 172.21.15.102:0/3098326993 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7b8401f020 con 0x7f7b8c1593c0 2023-03-03T13:55:33.893 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:55:33.893 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:55:33.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b700682f0 msgr2=0x7f7b7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:33.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b700682f0 0x7f7b7006a7a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009c00 tx=0x7f7b88009210 comp rx=0 tx=0).stop 2023-03-03T13:55:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c1593c0 msgr2=0x7f7b8c0c6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c1593c0 0x7f7b8c0c6940 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7b84015000 tx=0x7f7b84004ad0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 shutdown_connections 2023-03-03T13:55:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7b700682f0 0x7f7b7006a7a0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7b8c0cd8e0 0x7f7b8c0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b8c1593c0 0x7f7b8c0c6940 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 --2- 172.21.15.102:0/3098326993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b8c153290 0x7f7b8c1e5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 >> 172.21.15.102:0/3098326993 conn(0x7f7b8c065fb0 msgr2=0x7f7b8c156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:55:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 shutdown_connections 2023-03-03T13:55:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:55:33.894+0000 7f7b941ff700 1 -- 172.21.15.102:0/3098326993 wait complete. 2023-03-03T13:55:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:55:33.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T13:55:59.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:59.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:25.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:56:26.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.172+0000 7f90bd1ff700 1 Processor -- start 2023-03-03T13:56:26.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.172+0000 7f90bd1ff700 1 -- start start 2023-03-03T13:56:26.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.173+0000 7f90bd1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90b80cd8e0 0x7f90b80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:26.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.173+0000 7f90bd1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b8153660 con 0x7f90b8153290 2023-03-03T13:56:26.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90bd1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b806c360 con 0x7f90b80cd390 2023-03-03T13:56:26.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90bd1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b815cea0 con 0x7f90b80cd8e0 2023-03-03T13:56:26.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90b659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90b80cd8e0 0x7f90b80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:26.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90b659c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90b80cd8e0 0x7f90b80cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43288/0 (socket says 172.21.15.102:43288) 2023-03-03T13:56:26.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90b659c700 1 -- 172.21.15.102:0/2068803037 learned_addr learned my addr 172.21.15.102:0/2068803037 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:56:26.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90b659c700 1 -- 172.21.15.102:0/2068803037 >> v1:172.21.15.172:6789/0 conn(0x7f90b80cd390 legacy=0x7f90b80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:26.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90b659c700 1 -- 172.21.15.102:0/2068803037 >> v1:172.21.15.102:6789/0 conn(0x7f90b8153290 legacy=0x7f90b8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:26.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.174+0000 7f90b659c700 1 -- 172.21.15.102:0/2068803037 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b806c7f0 con 0x7f90b80cd8e0 2023-03-03T13:56:26.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.175+0000 7f90b659c700 1 --2- 172.21.15.102:0/2068803037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b80cd8e0 0x7f90b80cde20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0098f0 tx=0x7f90ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2695277d7a534da7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:56:26.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.175+0000 7f90b5d9b700 1 -- 172.21.15.102:0/2068803037 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00b8a0 con 0x7f90b80cd8e0 2023-03-03T13:56:26.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.175+0000 7f90b5d9b700 1 -- 172.21.15.102:0/2068803037 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac00ba00 con 0x7f90b80cd8e0 2023-03-03T13:56:26.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.175+0000 7f90b5d9b700 1 -- 172.21.15.102:0/2068803037 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00bd90 con 0x7f90b80cd8e0 2023-03-03T13:56:26.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.175+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2068803037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b80cd8e0 msgr2=0x7f90b80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:26.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.175+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2068803037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b80cd8e0 0x7f90b80cde20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0098f0 tx=0x7f90ac009c00 comp rx=0 tx=0).stop 2023-03-03T13:56:26.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.176+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2068803037 shutdown_connections 2023-03-03T13:56:26.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.176+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2068803037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b80cd8e0 0x7f90b80cde20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.176+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2068803037 >> 172.21.15.102:0/2068803037 conn(0x7f90b8065fb0 msgr2=0x7f90b80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:56:26.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.176+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2068803037 shutdown_connections 2023-03-03T13:56:26.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.176+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2068803037 wait complete. 2023-03-03T13:56:26.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.178+0000 7f90bd1ff700 1 Processor -- start 2023-03-03T13:56:26.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.178+0000 7f90bd1ff700 1 -- start start 2023-03-03T13:56:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90bd1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90b80cd390 0x7f90b80c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90bd1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80cd8e0 0x7f90b80b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90bd1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b8153290 0x7f90b80b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90bd1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b81544e0 con 0x7f90b8153290 2023-03-03T13:56:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90bd1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b81585d0 con 0x7f90b80cd8e0 2023-03-03T13:56:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90bd1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b80b32c0 con 0x7f90b80cd390 2023-03-03T13:56:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90b6d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90b80cd390 0x7f90b80c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90b759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b8153290 0x7f90b80b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90b659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80cd8e0 0x7f90b80b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90b6d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90b80cd390 0x7f90b80c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:49772/0 (socket says 172.21.15.102:49772) 2023-03-03T13:56:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90b6d9d700 1 -- 172.21.15.102:0/2139684878 learned_addr learned my addr 172.21.15.102:0/2139684878 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:56:26.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.179+0000 7f90b659c700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80cd8e0 0x7f90b80b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54516/0 (socket says 172.21.15.102:54516) 2023-03-03T13:56:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f90b759e700 1 -- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90b80cd390 msgr2=0x7f90b80c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f90b759e700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90b80cd390 0x7f90b80c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f90b759e700 1 -- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80cd8e0 msgr2=0x7f90b80b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f90b759e700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80cd8e0 0x7f90b80b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f90b759e700 1 -- 172.21.15.102:0/2139684878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac009550 con 0x7f90b8153290 2023-03-03T13:56:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f90b759e700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b8153290 0x7f90b80b3c90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f90b000cbc0 tx=0x7f90b000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:56:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f909f7fe700 1 -- 172.21.15.102:0/2139684878 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b0003c00 con 0x7f90b8153290 2023-03-03T13:56:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f909f7fe700 1 -- 172.21.15.102:0/2139684878 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b0003d60 con 0x7f90b8153290 2023-03-03T13:56:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f909f7fe700 1 -- 172.21.15.102:0/2139684878 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b000aa40 con 0x7f90b8153290 2023-03-03T13:56:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.180+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b80ccde0 con 0x7f90b8153290 2023-03-03T13:56:26.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.181+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b815d160 con 0x7f90b8153290 2023-03-03T13:56:26.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.182+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b80cd0c0 con 0x7f90b8153290 2023-03-03T13:56:26.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.183+0000 7f909f7fe700 1 -- 172.21.15.102:0/2139684878 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90b0010050 con 0x7f90b8153290 2023-03-03T13:56:26.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.185+0000 7f909f7fe700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90a40682f0 0x7f90a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:26.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.185+0000 7f909f7fe700 1 -- 172.21.15.102:0/2139684878 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f90b0085c70 con 0x7f90b8153290 2023-03-03T13:56:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.185+0000 7f90b6d9d700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90a40682f0 0x7f90a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.186+0000 7f90b6d9d700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90a40682f0 0x7f90a406a7a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f90a8001880 tx=0x7f90a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:56:26.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.189+0000 7f909f7fe700 1 -- 172.21.15.102:0/2139684878 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90b00522f0 con 0x7f90b8153290 2023-03-03T13:56:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.429+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90b804e6e0 con 0x7f90b8153290 2023-03-03T13:56:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.430+0000 7f909f7fe700 1 -- 172.21.15.102:0/2139684878 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f90b0055910 con 0x7f90b8153290 2023-03-03T13:56:26.432 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:56:26.432 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:56:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.432+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90a40682f0 msgr2=0x7f90a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.432+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90a40682f0 0x7f90a406a7a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f90a8001880 tx=0x7f90a8009040 comp rx=0 tx=0).stop 2023-03-03T13:56:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.432+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b8153290 msgr2=0x7f90b80b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:26.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.432+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b8153290 0x7f90b80b3c90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f90b000cbc0 tx=0x7f90b000ced0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.432+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 shutdown_connections 2023-03-03T13:56:26.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.433+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90a40682f0 0x7f90a406a7a0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.433+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90b80cd390 0x7f90b80c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.433+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90b8153290 0x7f90b80b3c90 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.433+0000 7f90bd1ff700 1 --2- 172.21.15.102:0/2139684878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80cd8e0 0x7f90b80b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:26.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.433+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 >> 172.21.15.102:0/2139684878 conn(0x7f90b8065fb0 msgr2=0x7f90b8157560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:56:26.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.433+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 shutdown_connections 2023-03-03T13:56:26.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:56:26.433+0000 7f90bd1ff700 1 -- 172.21.15.102:0/2139684878 wait complete. 2023-03-03T13:56:26.436 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:56:26.461 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-03-03T13:56:30.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:30.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:00.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:00.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:08.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:57:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.711+0000 7fb9087ff700 1 Processor -- start 2023-03-03T13:57:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.711+0000 7fb9087ff700 1 -- start start 2023-03-03T13:57:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.712+0000 7fb9087ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb900153290 0x7fb900158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.712+0000 7fb9087ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb9001593c0 0x7fb9000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb9087ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb900153660 con 0x7fb9000cd8e0 2023-03-03T13:57:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb9087ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90006c360 con 0x7fb9001593c0 2023-03-03T13:57:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb9087ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90015cea0 con 0x7fb900153290 2023-03-03T13:57:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb905d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb9001593c0 0x7fb9000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb90659b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb900153290 0x7fb900158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb90659b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb900153290 0x7fb900158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46422/0 (socket says 172.21.15.102:46422) 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb90659b700 1 -- 172.21.15.102:0/1416817560 learned_addr learned my addr 172.21.15.102:0/1416817560 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb906d9c700 1 --1- 172.21.15.102:0/1416817560 >> v1:172.21.15.172:6789/0 conn(0x7fb9000cd8e0 0x7fb9000ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56066/0 (socket says 172.21.15.102:56066) 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb90659b700 1 -- 172.21.15.102:0/1416817560 >> v2:172.21.15.102:3300/0 conn(0x7fb9001593c0 msgr2=0x7fb9000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb90659b700 1 --2- 172.21.15.102:0/1416817560 >> v2:172.21.15.102:3300/0 conn(0x7fb9001593c0 0x7fb9000cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb90659b700 1 -- 172.21.15.102:0/1416817560 >> v1:172.21.15.172:6789/0 conn(0x7fb9000cd8e0 legacy=0x7fb9000ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb90659b700 1 -- 172.21.15.102:0/1416817560 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90006c7f0 con 0x7fb900153290 2023-03-03T13:57:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.713+0000 7fb905d9a700 1 --2- 172.21.15.102:0/1416817560 >> v2:172.21.15.102:3300/0 conn(0x7fb9001593c0 0x7fb9000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:57:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.714+0000 7fb90659b700 1 --2- 172.21.15.102:0/1416817560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb900153290 0x7fb900158e80 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb90008cc70 tx=0x7fb8fc009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84cb8d5c0c2f445a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.714+0000 7fb905599700 1 -- 172.21.15.102:0/1416817560 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc011830 con 0x7fb900153290 2023-03-03T13:57:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.714+0000 7fb905599700 1 -- 172.21.15.102:0/1416817560 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc011990 con 0x7fb900153290 2023-03-03T13:57:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.714+0000 7fb905599700 1 -- 172.21.15.102:0/1416817560 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc011d20 con 0x7fb900153290 2023-03-03T13:57:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.714+0000 7fb9087ff700 1 -- 172.21.15.102:0/1416817560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb900153290 msgr2=0x7fb900158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.714+0000 7fb9087ff700 1 --2- 172.21.15.102:0/1416817560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb900153290 0x7fb900158e80 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb90008cc70 tx=0x7fb8fc009bd0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.715+0000 7fb9087ff700 1 -- 172.21.15.102:0/1416817560 shutdown_connections 2023-03-03T13:57:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.715+0000 7fb9087ff700 1 --2- 172.21.15.102:0/1416817560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb900153290 0x7fb900158e80 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.715+0000 7fb9087ff700 1 --2- 172.21.15.102:0/1416817560 >> v2:172.21.15.102:3300/0 conn(0x7fb9001593c0 0x7fb9000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.715+0000 7fb9087ff700 1 -- 172.21.15.102:0/1416817560 >> 172.21.15.102:0/1416817560 conn(0x7fb900065fb0 msgr2=0x7fb9000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.715+0000 7fb9087ff700 1 -- 172.21.15.102:0/1416817560 shutdown_connections 2023-03-03T13:57:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.715+0000 7fb9087ff700 1 -- 172.21.15.102:0/1416817560 wait complete. 2023-03-03T13:57:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.717+0000 7fb9087ff700 1 Processor -- start 2023-03-03T13:57:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.717+0000 7fb9087ff700 1 -- start start 2023-03-03T13:57:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb9087ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9000cd8e0 0x7fb9000c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb9087ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb900153290 0x7fb9001e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb9087ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9001593c0 0x7fb9000c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb9087ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000cfa20 con 0x7fb900153290 2023-03-03T13:57:08.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb9087ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000cf740 con 0x7fb9000cd8e0 2023-03-03T13:57:08.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb9087ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9001e3450 con 0x7fb9001593c0 2023-03-03T13:57:08.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb905d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb900153290 0x7fb9001e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:08.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb90659b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9000cd8e0 0x7fb9000c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:08.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb905d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb900153290 0x7fb9001e4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33102/0 (socket says 172.21.15.102:33102) 2023-03-03T13:57:08.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb905d9a700 1 -- 172.21.15.102:0/771650154 learned_addr learned my addr 172.21.15.102:0/771650154 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:57:08.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb90659b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9000cd8e0 0x7fb9000c6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46432/0 (socket says 172.21.15.102:46432) 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.718+0000 7fb906d9c700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9001593c0 0x7fb9000c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb90659b700 1 -- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9001593c0 msgr2=0x7fb9000c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb90659b700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9001593c0 0x7fb9000c84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb90659b700 1 -- 172.21.15.102:0/771650154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb900153290 msgr2=0x7fb9001e4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb90659b700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb900153290 0x7fb9001e4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb90659b700 1 -- 172.21.15.102:0/771650154 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f00094a0 con 0x7fb9000cd8e0 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb905d9a700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb900153290 0x7fb9001e4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:57:08.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb90659b700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9000cd8e0 0x7fb9000c6140 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc016000 tx=0x7fb8fc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:08.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb8f6ffd700 1 -- 172.21.15.102:0/771650154 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc004380 con 0x7fb9000cd8e0 2023-03-03T13:57:08.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb8f6ffd700 1 -- 172.21.15.102:0/771650154 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc01f070 con 0x7fb9000cd8e0 2023-03-03T13:57:08.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb8f6ffd700 1 -- 172.21.15.102:0/771650154 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0036a0 con 0x7fb9000cd8e0 2023-03-03T13:57:08.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.719+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc009550 con 0x7fb9000cd8e0 2023-03-03T13:57:08.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.720+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9000ccde0 con 0x7fb9000cd8e0 2023-03-03T13:57:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.721+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90015d160 con 0x7fb9000cd8e0 2023-03-03T13:57:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.722+0000 7fb8f6ffd700 1 -- 172.21.15.102:0/771650154 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb8fc02a020 con 0x7fb9000cd8e0 2023-03-03T13:57:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.723+0000 7fb8f6ffd700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8e40682f0 0x7fb8e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.723+0000 7fb8f6ffd700 1 -- 172.21.15.102:0/771650154 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb8fc087340 con 0x7fb9000cd8e0 2023-03-03T13:57:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.723+0000 7fb905d9a700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8e40682f0 0x7fb8e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:08.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.724+0000 7fb905d9a700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8e40682f0 0x7fb8e406a7a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb90014e340 tx=0x7fb8f0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:08.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.728+0000 7fb8f6ffd700 1 -- 172.21.15.102:0/771650154 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb8fc0539d0 con 0x7fb9000cd8e0 2023-03-03T13:57:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.963+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9000cd0c0 con 0x7fb9000cd8e0 2023-03-03T13:57:08.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.963+0000 7fb8f6ffd700 1 -- 172.21.15.102:0/771650154 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb8fc056ff0 con 0x7fb9000cd8e0 2023-03-03T13:57:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:57:08.965 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:57:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8e40682f0 msgr2=0x7fb8e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8e40682f0 0x7fb8e406a7a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb90014e340 tx=0x7fb8f0009210 comp rx=0 tx=0).stop 2023-03-03T13:57:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9000cd8e0 msgr2=0x7fb9000c6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9000cd8e0 0x7fb9000c6140 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc016000 tx=0x7fb8fc004750 comp rx=0 tx=0).stop 2023-03-03T13:57:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 shutdown_connections 2023-03-03T13:57:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb8e40682f0 0x7fb8e406a7a0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb9001593c0 0x7fb9000c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9000cd8e0 0x7fb9000c6140 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 --2- 172.21.15.102:0/771650154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb900153290 0x7fb9001e4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:08.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 >> 172.21.15.102:0/771650154 conn(0x7fb900065fb0 msgr2=0x7fb90015ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:08.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 shutdown_connections 2023-03-03T13:57:08.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:57:08.966+0000 7fb9087ff700 1 -- 172.21.15.102:0/771650154 wait complete. 2023-03-03T13:57:08.969 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:57:08.994 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-03-03T13:57:30.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:30.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:00.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:00.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:01.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:58:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.244+0000 7fccb199d700 1 Processor -- start 2023-03-03T13:58:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.244+0000 7fccb199d700 1 -- start start 2023-03-03T13:58:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.245+0000 7fccb199d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fccac0ced00 0x7fccac0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:58:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb199d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0c3d20 con 0x7fccac0c41b0 2023-03-03T13:58:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb199d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac15cba0 con 0x7fccac0b3970 2023-03-03T13:58:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb199d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccac0c4930 con 0x7fccac0ced00 2023-03-03T13:58:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb099b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fccac0ced00 0x7fccac0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:58:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb099b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fccac0ced00 0x7fccac0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44166/0 (socket says 172.21.15.102:44166) 2023-03-03T13:58:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb099b700 1 -- 172.21.15.102:0/1490291726 learned_addr learned my addr 172.21.15.102:0/1490291726 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:58:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccabfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fccac0b3970 0x7fccac158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49898/0 (socket says 172.21.15.102:49898) 2023-03-03T13:58:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb099b700 1 -- 172.21.15.102:0/1490291726 >> v1:172.21.15.172:6789/0 conn(0x7fccac0b3970 legacy=0x7fccac158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:58:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb099b700 1 -- 172.21.15.102:0/1490291726 >> v1:172.21.15.102:6789/0 conn(0x7fccac0c41b0 legacy=0x7fccac163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:58:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.246+0000 7fccb099b700 1 -- 172.21.15.102:0/1490291726 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac15ce80 con 0x7fccac0ced00 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.247+0000 7fccb099b700 1 --2- 172.21.15.102:0/1490291726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0ced00 0x7fccac0c5a90 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fcca00098f0 tx=0x7fcca0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f59b3b696e50a1f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.247+0000 7fccab7fe700 1 -- 172.21.15.102:0/1490291726 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca000b8a0 con 0x7fccac0ced00 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.247+0000 7fccab7fe700 1 -- 172.21.15.102:0/1490291726 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca000ba00 con 0x7fccac0ced00 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.247+0000 7fccab7fe700 1 -- 172.21.15.102:0/1490291726 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca000bd90 con 0x7fccac0ced00 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.248+0000 7fccb199d700 1 -- 172.21.15.102:0/1490291726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0ced00 msgr2=0x7fccac0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.248+0000 7fccb199d700 1 --2- 172.21.15.102:0/1490291726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0ced00 0x7fccac0c5a90 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fcca00098f0 tx=0x7fcca0009c00 comp rx=0 tx=0).stop 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.248+0000 7fccb199d700 1 -- 172.21.15.102:0/1490291726 shutdown_connections 2023-03-03T13:58:02.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.248+0000 7fccb199d700 1 --2- 172.21.15.102:0/1490291726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0ced00 0x7fccac0c5a90 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.248+0000 7fccb199d700 1 -- 172.21.15.102:0/1490291726 >> 172.21.15.102:0/1490291726 conn(0x7fccac08b1c0 msgr2=0x7fccac071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:58:02.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.248+0000 7fccb199d700 1 -- 172.21.15.102:0/1490291726 shutdown_connections 2023-03-03T13:58:02.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.248+0000 7fccb199d700 1 -- 172.21.15.102:0/1490291726 wait complete. 2023-03-03T13:58:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.250+0000 7fccb199d700 1 Processor -- start 2023-03-03T13:58:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.250+0000 7fccb199d700 1 -- start start 2023-03-03T13:58:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb199d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccac0b3970 0x7fccac1f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:58:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb199d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0c41b0 0x7fccac1eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:58:02.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb199d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccac0ced00 0x7fccac150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb199d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac1ee680 con 0x7fccac0c41b0 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb199d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac1ee000 con 0x7fccac0ced00 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb199d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac14ebd0 con 0x7fccac0b3970 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccac0b3970 0x7fccac1f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccabfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0c41b0 0x7fccac1eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccac0b3970 0x7fccac1f14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51858/0 (socket says 172.21.15.102:51858) 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.251+0000 7fccb099b700 1 -- 172.21.15.102:0/499867533 learned_addr learned my addr 172.21.15.102:0/499867533 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:58:02.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.252+0000 7fccb119c700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccac0ced00 0x7fccac150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:58:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.252+0000 7fccb099b700 1 -- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccac0ced00 msgr2=0x7fccac150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.252+0000 7fccb099b700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccac0ced00 0x7fccac150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.252+0000 7fccb099b700 1 -- 172.21.15.102:0/499867533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0c41b0 msgr2=0x7fccac1eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.252+0000 7fccb099b700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0c41b0 0x7fccac1eeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.252+0000 7fccb099b700 1 -- 172.21.15.102:0/499867533 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca0009550 con 0x7fccac0b3970 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.252+0000 7fccb099b700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccac0b3970 0x7fccac1f14e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcca0000c00 tx=0x7fcca0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.253+0000 7fcca97fa700 1 -- 172.21.15.102:0/499867533 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0004380 con 0x7fccac0b3970 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.253+0000 7fcca97fa700 1 -- 172.21.15.102:0/499867533 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca001f070 con 0x7fccac0b3970 2023-03-03T13:58:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.253+0000 7fcca97fa700 1 -- 172.21.15.102:0/499867533 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca00036a0 con 0x7fccac0b3970 2023-03-03T13:58:02.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.253+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac06c200 con 0x7fccac0b3970 2023-03-03T13:58:02.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.253+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac06bf20 con 0x7fccac0b3970 2023-03-03T13:58:02.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.254+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac15c890 con 0x7fccac0b3970 2023-03-03T13:58:02.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.255+0000 7fcca97fa700 1 -- 172.21.15.102:0/499867533 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcca002a020 con 0x7fccac0b3970 2023-03-03T13:58:02.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.256+0000 7fcca97fa700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc900682f0 0x7fcc9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:58:02.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.256+0000 7fcca97fa700 1 -- 172.21.15.102:0/499867533 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcca0087350 con 0x7fccac0b3970 2023-03-03T13:58:02.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.257+0000 7fccabfff700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc900682f0 0x7fcc9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:58:02.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.257+0000 7fccabfff700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc900682f0 0x7fcc9006a7a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fccac0ce9f0 tx=0x7fcc98007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:58:02.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.261+0000 7fcca97fa700 1 -- 172.21.15.102:0/499867533 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcca00539e0 con 0x7fccac0b3970 2023-03-03T13:58:02.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.493+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccac1ee8f0 con 0x7fccac0b3970 2023-03-03T13:58:02.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.494+0000 7fcca97fa700 1 -- 172.21.15.102:0/499867533 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcca0057000 con 0x7fccac0b3970 2023-03-03T13:58:02.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:58:02.496 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:58:02.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc900682f0 msgr2=0x7fcc9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:58:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc900682f0 0x7fcc9006a7a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fccac0ce9f0 tx=0x7fcc98007ed0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccac0b3970 msgr2=0x7fccac1f14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:58:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccac0b3970 0x7fccac1f14e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcca0000c00 tx=0x7fcca0004750 comp rx=0 tx=0).stop 2023-03-03T13:58:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 shutdown_connections 2023-03-03T13:58:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc900682f0 0x7fcc9006a7a0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fccac0b3970 0x7fccac1f14e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fccac0ced00 0x7fccac150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 --2- 172.21.15.102:0/499867533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fccac0c41b0 0x7fccac1eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:58:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 >> 172.21.15.102:0/499867533 conn(0x7fccac08b1c0 msgr2=0x7fccac0cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:58:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 shutdown_connections 2023-03-03T13:58:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:58:02.497+0000 7fccb199d700 1 -- 172.21.15.102:0/499867533 wait complete. 2023-03-03T13:58:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:58:02.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T13:58:30.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:30.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:00.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:00.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:02.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:59:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.776+0000 7f1a0359e700 1 Processor -- start 2023-03-03T13:59:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.777+0000 7f1a0359e700 1 -- start start 2023-03-03T13:59:02.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.778+0000 7f1a0359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1a040c9610 0x7f1a040ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:02.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.778+0000 7f1a0359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a040c9ca0 con 0x7f1a041564b0 2023-03-03T13:59:02.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.778+0000 7f1a0359e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a040b46f0 con 0x7f1a040ca980 2023-03-03T13:59:02.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.778+0000 7f1a0359e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a040b4920 con 0x7f1a040c9610 2023-03-03T13:59:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.778+0000 7f1a01d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1a040c9610 0x7f1a040ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.778+0000 7f1a0259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1a041564b0 0x7f1a040cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38312/0 (socket says 172.21.15.102:38312) 2023-03-03T13:59:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.778+0000 7f1a0259c700 1 -- 172.21.15.102:0/1535437076 learned_addr learned my addr 172.21.15.102:0/1535437076 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T13:59:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.779+0000 7f1a01d9b700 1 -- 172.21.15.102:0/1535437076 >> v1:172.21.15.172:6789/0 conn(0x7f1a040ca980 legacy=0x7f1a041636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.779+0000 7f1a01d9b700 1 -- 172.21.15.102:0/1535437076 >> v1:172.21.15.102:6789/0 conn(0x7f1a041564b0 legacy=0x7f1a040cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.779+0000 7f1a01d9b700 1 -- 172.21.15.102:0/1535437076 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a040b4c00 con 0x7f1a040c9610 2023-03-03T13:59:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.779+0000 7f1a01d9b700 1 --2- 172.21.15.102:0/1535437076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a040ca440 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009840 tx=0x7f19f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6b51e257eefdf0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.779+0000 7f1a0159a700 1 -- 172.21.15.102:0/1535437076 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8011830 con 0x7f1a040c9610 2023-03-03T13:59:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0159a700 1 -- 172.21.15.102:0/1535437076 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f8011990 con 0x7f1a040c9610 2023-03-03T13:59:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0159a700 1 -- 172.21.15.102:0/1535437076 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8011d20 con 0x7f1a040c9610 2023-03-03T13:59:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0359e700 1 -- 172.21.15.102:0/1535437076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 msgr2=0x7f1a040ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0359e700 1 --2- 172.21.15.102:0/1535437076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a040ca440 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009840 tx=0x7f19f8009b50 comp rx=0 tx=0).stop 2023-03-03T13:59:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0359e700 1 -- 172.21.15.102:0/1535437076 shutdown_connections 2023-03-03T13:59:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0359e700 1 --2- 172.21.15.102:0/1535437076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a040ca440 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0359e700 1 -- 172.21.15.102:0/1535437076 >> 172.21.15.102:0/1535437076 conn(0x7f1a04065fb0 msgr2=0x7f1a040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:59:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0359e700 1 -- 172.21.15.102:0/1535437076 shutdown_connections 2023-03-03T13:59:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.780+0000 7f1a0359e700 1 -- 172.21.15.102:0/1535437076 wait complete. 2023-03-03T13:59:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.782+0000 7f1a0359e700 1 Processor -- start 2023-03-03T13:59:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.782+0000 7f1a0359e700 1 -- start start 2023-03-03T13:59:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a041f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a040ca980 0x7f1a040c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a041564b0 0x7f1a040c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a041f6c40 con 0x7f1a040c9610 2023-03-03T13:59:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a04151810 con 0x7f1a040ca980 2023-03-03T13:59:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a041514c0 con 0x7f1a041564b0 2023-03-03T13:59:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a041f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a041f5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36404/0 (socket says 172.21.15.102:36404) 2023-03-03T13:59:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 -- 172.21.15.102:0/3695243456 learned_addr learned my addr 172.21.15.102:0/3695243456 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T13:59:02.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a01d9b700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a040ca980 0x7f1a040c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a02d9d700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a041564b0 0x7f1a040c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 -- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a041564b0 msgr2=0x7f1a040c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a041564b0 0x7f1a040c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 -- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a040ca980 msgr2=0x7f1a040c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a040ca980 0x7f1a040c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.784+0000 7f1a0259c700 1 -- 172.21.15.102:0/3695243456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f80094a0 con 0x7f1a040c9610 2023-03-03T13:59:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.785+0000 7f1a0259c700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a041f5fc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f19f400c480 tx=0x7f19f400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.785+0000 7f19f2ffd700 1 -- 172.21.15.102:0/3695243456 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f4005d70 con 0x7f1a040c9610 2023-03-03T13:59:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.785+0000 7f19f2ffd700 1 -- 172.21.15.102:0/3695243456 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f400ce90 con 0x7f1a040c9610 2023-03-03T13:59:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.785+0000 7f19f2ffd700 1 -- 172.21.15.102:0/3695243456 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f40072d0 con 0x7f1a040c9610 2023-03-03T13:59:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.785+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a040b2210 con 0x7f1a040c9610 2023-03-03T13:59:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.785+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a040b1f30 con 0x7f1a040c9610 2023-03-03T13:59:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.787+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a04156880 con 0x7f1a040c9610 2023-03-03T13:59:02.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.788+0000 7f19f2ffd700 1 -- 172.21.15.102:0/3695243456 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f19f4019030 con 0x7f1a040c9610 2023-03-03T13:59:02.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.789+0000 7f19f2ffd700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19e40682f0 0x7f19e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:02.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.789+0000 7f19f2ffd700 1 -- 172.21.15.102:0/3695243456 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f19f4084e70 con 0x7f1a040c9610 2023-03-03T13:59:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.789+0000 7f1a01d9b700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19e40682f0 0x7f19e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.790+0000 7f1a01d9b700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19e40682f0 0x7f19e406a7a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f1a040c9a30 tx=0x7f19f8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:02.793+0000 7f19f2ffd700 1 -- 172.21.15.102:0/3695243456 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f19f404b5e0 con 0x7f1a040c9610 2023-03-03T13:59:03.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.026+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a04151630 con 0x7f1a040c9610 2023-03-03T13:59:03.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.027+0000 7f19f2ffd700 1 -- 172.21.15.102:0/3695243456 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f19f4017090 con 0x7f1a040c9610 2023-03-03T13:59:03.029 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T13:59:03.030 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T13:59:03.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19e40682f0 msgr2=0x7f19e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:03.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19e40682f0 0x7f19e406a7a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f1a040c9a30 tx=0x7f19f8008030 comp rx=0 tx=0).stop 2023-03-03T13:59:03.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 msgr2=0x7f1a041f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:03.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a041f5fc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f19f400c480 tx=0x7f19f400c790 comp rx=0 tx=0).stop 2023-03-03T13:59:03.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 shutdown_connections 2023-03-03T13:59:03.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19e40682f0 0x7f19e406a7a0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:03.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1a041564b0 0x7f1a040c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:03.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040c9610 0x7f1a041f5fc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:03.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 --2- 172.21.15.102:0/3695243456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a040ca980 0x7f1a040c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:03.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 >> 172.21.15.102:0/3695243456 conn(0x7f1a04065fb0 msgr2=0x7f1a04066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:59:03.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 shutdown_connections 2023-03-03T13:59:03.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T13:59:03.029+0000 7f1a0359e700 1 -- 172.21.15.102:0/3695243456 wait complete. 2023-03-03T13:59:03.034 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T13:59:03.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-03-03T13:59:31.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:31.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T13:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:00:01.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:01.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:03.060 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:00:03.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.313+0000 7f3726b9d700 1 Processor -- start 2023-03-03T14:00:03.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.313+0000 7f3726b9d700 1 -- start start 2023-03-03T14:00:03.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.315+0000 7f3726b9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37200ced00 0x7f37200c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:03.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.315+0000 7f3726b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200b3970 0x7f37200c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:03.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.315+0000 7f3726b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37200c3d20 con 0x7f37200cf0d0 2023-03-03T14:00:03.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.315+0000 7f3726b9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37200c4930 con 0x7f37200ced00 2023-03-03T14:00:03.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.315+0000 7f3726b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372015cba0 con 0x7f37200b3970 2023-03-03T14:00:03.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.315+0000 7f3725b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37200ced00 0x7f37200c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:03.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f372639c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f37200cf0d0 0x7f3720158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43970/0 (socket says 172.21.15.102:43970) 2023-03-03T14:00:03.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f3725b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37200ced00 0x7f37200c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40482/0 (socket says 172.21.15.102:40482) 2023-03-03T14:00:03.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f372639c700 1 -- 172.21.15.102:0/3820233009 learned_addr learned my addr 172.21.15.102:0/3820233009 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:00:03.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f372539a700 1 --2- 172.21.15.102:0/3820233009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200b3970 0x7f37200c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:03.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f3725b9b700 1 -- 172.21.15.102:0/3820233009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200b3970 msgr2=0x7f37200c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:00:03.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f3725b9b700 1 --2- 172.21.15.102:0/3820233009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200b3970 0x7f37200c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f3725b9b700 1 -- 172.21.15.102:0/3820233009 >> v1:172.21.15.102:6789/0 conn(0x7f37200cf0d0 legacy=0x7f3720158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:03.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f3725b9b700 1 -- 172.21.15.102:0/3820233009 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372015ce80 con 0x7f37200ced00 2023-03-03T14:00:03.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.316+0000 7f3725b9b700 1 --2- 172.21.15.102:0/3820233009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200ced00 0x7f37200c5a90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f371c009840 tx=0x7f371c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6b1f15b3dffd95 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:03.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3724b99700 1 -- 172.21.15.102:0/3820233009 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c011830 con 0x7f37200ced00 2023-03-03T14:00:03.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3724b99700 1 -- 172.21.15.102:0/3820233009 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c011990 con 0x7f37200ced00 2023-03-03T14:00:03.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3724b99700 1 -- 172.21.15.102:0/3820233009 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c011d20 con 0x7f37200ced00 2023-03-03T14:00:03.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 -- 172.21.15.102:0/3820233009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200ced00 msgr2=0x7f37200c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:03.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 --2- 172.21.15.102:0/3820233009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200ced00 0x7f37200c5a90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f371c009840 tx=0x7f371c009b50 comp rx=0 tx=0).stop 2023-03-03T14:00:03.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 -- 172.21.15.102:0/3820233009 shutdown_connections 2023-03-03T14:00:03.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 --2- 172.21.15.102:0/3820233009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200ced00 0x7f37200c5a90 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 --2- 172.21.15.102:0/3820233009 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200b3970 0x7f37200c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 -- 172.21.15.102:0/3820233009 >> 172.21.15.102:0/3820233009 conn(0x7f372008b1c0 msgr2=0x7f3720071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:03.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 -- 172.21.15.102:0/3820233009 shutdown_connections 2023-03-03T14:00:03.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.317+0000 7f3726b9d700 1 -- 172.21.15.102:0/3820233009 wait complete. 2023-03-03T14:00:03.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.319+0000 7f3726b9d700 1 Processor -- start 2023-03-03T14:00:03.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.319+0000 7f3726b9d700 1 -- start start 2023-03-03T14:00:03.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.320+0000 7f3726b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200b3970 0x7f37200cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:03.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.320+0000 7f3726b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37200ced00 0x7f3720066060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:03.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.320+0000 7f3726b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200cf0d0 0x7f372015be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:03.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3726b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372015a4d0 con 0x7f37200b3970 2023-03-03T14:00:03.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3726b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3720066810 con 0x7f37200ced00 2023-03-03T14:00:03.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3726b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3720159cc0 con 0x7f37200cf0d0 2023-03-03T14:00:03.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200b3970 0x7f37200cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:03.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200b3970 0x7f37200cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40484/0 (socket says 172.21.15.102:40484) 2023-03-03T14:00:03.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 -- 172.21.15.102:0/4031700587 learned_addr learned my addr 172.21.15.102:0/4031700587 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:00:03.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f372539a700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37200ced00 0x7f3720066060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f372639c700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200cf0d0 0x7f372015be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 -- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200cf0d0 msgr2=0x7f372015be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200cf0d0 0x7f372015be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 -- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37200ced00 msgr2=0x7f3720066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37200ced00 0x7f3720066060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.321+0000 7f3725b9b700 1 -- 172.21.15.102:0/4031700587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c0094a0 con 0x7f37200b3970 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.322+0000 7f3725b9b700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200b3970 0x7f37200cd6b0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f371c009fd0 tx=0x7f371c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.322+0000 7f37167fc700 1 -- 172.21.15.102:0/4031700587 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c0036a0 con 0x7f37200b3970 2023-03-03T14:00:03.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.322+0000 7f37167fc700 1 -- 172.21.15.102:0/4031700587 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c01f070 con 0x7f37200b3970 2023-03-03T14:00:03.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.322+0000 7f37167fc700 1 -- 172.21.15.102:0/4031700587 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c003a40 con 0x7f37200b3970 2023-03-03T14:00:03.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.322+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372006c200 con 0x7f37200b3970 2023-03-03T14:00:03.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.322+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372006bf20 con 0x7f37200b3970 2023-03-03T14:00:03.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.323+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37200cf540 con 0x7f37200b3970 2023-03-03T14:00:03.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.324+0000 7f37167fc700 1 -- 172.21.15.102:0/4031700587 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f371c00f050 con 0x7f37200b3970 2023-03-03T14:00:03.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.325+0000 7f37167fc700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37040682f0 0x7f370406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:03.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.326+0000 7f37167fc700 1 -- 172.21.15.102:0/4031700587 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f371c021070 con 0x7f37200b3970 2023-03-03T14:00:03.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.326+0000 7f372539a700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37040682f0 0x7f370406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:03.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.327+0000 7f372539a700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37040682f0 0x7f370406a7a0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f372015c5f0 tx=0x7f3710009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:03.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.330+0000 7f37167fc700 1 -- 172.21.15.102:0/4031700587 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f371c053cc0 con 0x7f37200b3970 2023-03-03T14:00:03.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.564+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f372004e6e0 con 0x7f37200b3970 2023-03-03T14:00:03.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.565+0000 7f37167fc700 1 -- 172.21.15.102:0/4031700587 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f371c0572e0 con 0x7f37200b3970 2023-03-03T14:00:03.567 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:00:03.567 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:00:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37040682f0 msgr2=0x7f370406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37040682f0 0x7f370406a7a0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f372015c5f0 tx=0x7f3710009210 comp rx=0 tx=0).stop 2023-03-03T14:00:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200b3970 msgr2=0x7f37200cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:03.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200b3970 0x7f37200cd6b0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f371c009fd0 tx=0x7f371c004750 comp rx=0 tx=0).stop 2023-03-03T14:00:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 shutdown_connections 2023-03-03T14:00:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f37040682f0 0x7f370406a7a0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f37200cf0d0 0x7f372015be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37200b3970 0x7f37200cd6b0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 --2- 172.21.15.102:0/4031700587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37200ced00 0x7f3720066060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.567+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 >> 172.21.15.102:0/4031700587 conn(0x7f372008b1c0 msgr2=0x7f3720071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:03.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.568+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 shutdown_connections 2023-03-03T14:00:03.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:03.568+0000 7f3726b9d700 1 -- 172.21.15.102:0/4031700587 wait complete. 2023-03-03T14:00:03.572 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:00:03.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T14:00:31.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:31.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:43.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:00:43.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.844+0000 7fa91a7ff700 1 Processor -- start 2023-03-03T14:00:43.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.844+0000 7fa91a7ff700 1 -- start start 2023-03-03T14:00:43.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.845+0000 7fa91a7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa914153290 0x7fa914158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:43.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.845+0000 7fa91a7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:43.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.845+0000 7fa91a7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa914153660 con 0x7fa9141593c0 2023-03-03T14:00:43.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.845+0000 7fa91a7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91406c360 con 0x7fa914153290 2023-03-03T14:00:43.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.845+0000 7fa91a7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91415cea0 con 0x7fa9140cd8e0 2023-03-03T14:00:43.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa913fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa914153290 0x7fa914158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:43.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa918d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:43.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa9137fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa9141593c0 0x7fa9140cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50302/0 (socket says 172.21.15.102:50302) 2023-03-03T14:00:43.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa9137fe700 1 -- 172.21.15.102:0/1315820451 learned_addr learned my addr 172.21.15.102:0/1315820451 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:00:43.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa912ffd700 1 -- 172.21.15.102:0/1315820451 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409323318 0 0) 0x7fa914153660 con 0x7fa9141593c0 2023-03-03T14:00:43.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa912ffd700 1 -- 172.21.15.102:0/1315820451 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8fc0035b0 con 0x7fa9141593c0 2023-03-03T14:00:43.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa913fff700 1 -- 172.21.15.102:0/1315820451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 msgr2=0x7fa9140ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:00:43.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa913fff700 1 --2- 172.21.15.102:0/1315820451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:43.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa913fff700 1 -- 172.21.15.102:0/1315820451 >> v1:172.21.15.172:6789/0 conn(0x7fa9141593c0 legacy=0x7fa9140cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:43.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa913fff700 1 -- 172.21.15.102:0/1315820451 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91406c7f0 con 0x7fa914153290 2023-03-03T14:00:43.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa918d9c700 1 --2- 172.21.15.102:0/1315820451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:00:43.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.846+0000 7fa913fff700 1 --2- 172.21.15.102:0/1315820451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa914153290 0x7fa914158e80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fa904009840 tx=0x7fa904009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1652a0c0beb02a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:43.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.847+0000 7fa912ffd700 1 -- 172.21.15.102:0/1315820451 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904011830 con 0x7fa914153290 2023-03-03T14:00:43.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.847+0000 7fa912ffd700 1 -- 172.21.15.102:0/1315820451 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa904011990 con 0x7fa914153290 2023-03-03T14:00:43.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.847+0000 7fa912ffd700 1 -- 172.21.15.102:0/1315820451 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904011d20 con 0x7fa914153290 2023-03-03T14:00:43.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.847+0000 7fa91a7ff700 1 -- 172.21.15.102:0/1315820451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa914153290 msgr2=0x7fa914158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:43.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.847+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/1315820451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa914153290 0x7fa914158e80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fa904009840 tx=0x7fa904009b50 comp rx=0 tx=0).stop 2023-03-03T14:00:43.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.847+0000 7fa91a7ff700 1 -- 172.21.15.102:0/1315820451 shutdown_connections 2023-03-03T14:00:43.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.847+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/1315820451 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.848+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/1315820451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa914153290 0x7fa914158e80 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.848+0000 7fa91a7ff700 1 -- 172.21.15.102:0/1315820451 >> 172.21.15.102:0/1315820451 conn(0x7fa914065fb0 msgr2=0x7fa9140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.848+0000 7fa91a7ff700 1 -- 172.21.15.102:0/1315820451 shutdown_connections 2023-03-03T14:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.848+0000 7fa91a7ff700 1 -- 172.21.15.102:0/1315820451 wait complete. 2023-03-03T14:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.850+0000 7fa91a7ff700 1 Processor -- start 2023-03-03T14:00:43.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.850+0000 7fa91a7ff700 1 -- start start 2023-03-03T14:00:43.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa91a7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa91a7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa914153290 0x7fa9140caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa91a7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9141593c0 0x7fa9140b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa91a7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140cc310 con 0x7fa9141593c0 2023-03-03T14:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa91a7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141f3070 con 0x7fa914153290 2023-03-03T14:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa91a7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141ef070 con 0x7fa9140cd8e0 2023-03-03T14:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa913fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:43.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa918d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9141593c0 0x7fa9140b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa9137fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa914153290 0x7fa9140caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa913fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41904/0 (socket says 172.21.15.102:41904) 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.851+0000 7fa913fff700 1 -- 172.21.15.102:0/108753568 learned_addr learned my addr 172.21.15.102:0/108753568 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa918d9c700 1 -- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 msgr2=0x7fa9140b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa918d9c700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa918d9c700 1 -- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa914153290 msgr2=0x7fa9140caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa918d9c700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa914153290 0x7fa9140caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa918d9c700 1 -- 172.21.15.102:0/108753568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c009a70 con 0x7fa9141593c0 2023-03-03T14:00:43.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa918d9c700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9141593c0 0x7fa9140b3180 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00efc0 tx=0x7fa90c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:43.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa910ff9700 1 -- 172.21.15.102:0/108753568 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c009e80 con 0x7fa9141593c0 2023-03-03T14:00:43.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa910ff9700 1 -- 172.21.15.102:0/108753568 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c00f040 con 0x7fa9141593c0 2023-03-03T14:00:43.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa910ff9700 1 -- 172.21.15.102:0/108753568 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c00ccd0 con 0x7fa9141593c0 2023-03-03T14:00:43.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.852+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9040094a0 con 0x7fa9141593c0 2023-03-03T14:00:43.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.853+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9140ccde0 con 0x7fa9141593c0 2023-03-03T14:00:43.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.854+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91415d160 con 0x7fa9141593c0 2023-03-03T14:00:43.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.855+0000 7fa910ff9700 1 -- 172.21.15.102:0/108753568 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa90c014480 con 0x7fa9141593c0 2023-03-03T14:00:43.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.857+0000 7fa910ff9700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9000682f0 0x7fa90006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:43.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.857+0000 7fa910ff9700 1 -- 172.21.15.102:0/108753568 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa90c086430 con 0x7fa9141593c0 2023-03-03T14:00:43.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.857+0000 7fa913fff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9000682f0 0x7fa90006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:43.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.858+0000 7fa913fff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9000682f0 0x7fa90006a7a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fa904009b90 tx=0x7fa90401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:43.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:43.861+0000 7fa910ff9700 1 -- 172.21.15.102:0/108753568 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa90c052ae0 con 0x7fa9141593c0 2023-03-03T14:00:44.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.099+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9140cd0c0 con 0x7fa9141593c0 2023-03-03T14:00:44.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.099+0000 7fa910ff9700 1 -- 172.21.15.102:0/108753568 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa90c056100 con 0x7fa9141593c0 2023-03-03T14:00:44.102 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:00:44.102 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:00:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9000682f0 msgr2=0x7fa90006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9000682f0 0x7fa90006a7a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fa904009b90 tx=0x7fa90401c040 comp rx=0 tx=0).stop 2023-03-03T14:00:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9141593c0 msgr2=0x7fa9140b3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9141593c0 0x7fa9140b3180 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00efc0 tx=0x7fa90c00c3a0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 shutdown_connections 2023-03-03T14:00:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa9000682f0 0x7fa90006a7a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa9140cd8e0 0x7fa9140b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9141593c0 0x7fa9140b3180 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 --2- 172.21.15.102:0/108753568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa914153290 0x7fa9140caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 >> 172.21.15.102:0/108753568 conn(0x7fa914065fb0 msgr2=0x7fa914073890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 shutdown_connections 2023-03-03T14:00:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:00:44.102+0000 7fa91a7ff700 1 -- 172.21.15.102:0/108753568 wait complete. 2023-03-03T14:00:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:00:44.131 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 16 secs before thrashing 2023-03-03T14:01:00.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:01:00.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.394+0000 7fe9f17ff700 1 Processor -- start 2023-03-03T14:01:00.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.394+0000 7fe9f17ff700 1 -- start start 2023-03-03T14:01:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.395+0000 7fe9f17ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec153290 0x7fe9ec158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9f17ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe9ec0cd8e0 0x7fe9ec0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9f17ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec153660 con 0x7fe9ec1593c0 2023-03-03T14:01:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9f17ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ec06c360 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9f17ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec15cea0 con 0x7fe9ec153290 2023-03-03T14:01:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe9ec0cd8e0 0x7fe9ec0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec153290 0x7fe9ec158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe9ec0cd8e0 0x7fe9ec0ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48346/0 (socket says 172.21.15.102:48346) 2023-03-03T14:01:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eb7fe700 1 -- 172.21.15.102:0/2129075574 learned_addr learned my addr 172.21.15.102:0/2129075574 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:01:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9ea7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe9ec1593c0 0x7fe9ec0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37172/0 (socket says 172.21.15.102:37172) 2023-03-03T14:01:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eb7fe700 1 -- 172.21.15.102:0/2129075574 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec153290 msgr2=0x7fe9ec158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:01:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eb7fe700 1 --2- 172.21.15.102:0/2129075574 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec153290 0x7fe9ec158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eb7fe700 1 -- 172.21.15.102:0/2129075574 >> v1:172.21.15.172:6789/0 conn(0x7fe9ec1593c0 legacy=0x7fe9ec0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.396+0000 7fe9eb7fe700 1 -- 172.21.15.102:0/2129075574 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec06c7f0 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.397+0000 7fe9eb7fe700 1 --2- 172.21.15.102:0/2129075574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0ce530 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009e50 tx=0x7fe9e400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=728214a9aa30807 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:01:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.397+0000 7fe9e9ffb700 1 -- 172.21.15.102:0/2129075574 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e400e070 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.397+0000 7fe9e9ffb700 1 -- 172.21.15.102:0/2129075574 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e400b910 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.397+0000 7fe9e9ffb700 1 -- 172.21.15.102:0/2129075574 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e400bcc0 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 -- 172.21.15.102:0/2129075574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec0cd8e0 msgr2=0x7fe9ec0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/2129075574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0ce530 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009e50 tx=0x7fe9e400b330 comp rx=0 tx=0).stop 2023-03-03T14:01:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 -- 172.21.15.102:0/2129075574 shutdown_connections 2023-03-03T14:01:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/2129075574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0ce530 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/2129075574 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec153290 0x7fe9ec158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 -- 172.21.15.102:0/2129075574 >> 172.21.15.102:0/2129075574 conn(0x7fe9ec065fb0 msgr2=0x7fe9ec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:01:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 -- 172.21.15.102:0/2129075574 shutdown_connections 2023-03-03T14:01:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.398+0000 7fe9f17ff700 1 -- 172.21.15.102:0/2129075574 wait complete. 2023-03-03T14:01:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.400+0000 7fe9f17ff700 1 Processor -- start 2023-03-03T14:01:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.400+0000 7fe9f17ff700 1 -- start start 2023-03-03T14:01:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9f17ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9f17ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9ec153290 0x7fe9ec0cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9f17ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec1593c0 0x7fe9ec1f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9f17ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec14edb0 con 0x7fe9ec153290 2023-03-03T14:01:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9f17ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec0c4f00 con 0x7fe9ec1593c0 2023-03-03T14:01:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9f17ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec0c2c50 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec1593c0 0x7fe9ec1f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec1593c0 0x7fe9ec1f2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48354/0 (socket says 172.21.15.102:48354) 2023-03-03T14:01:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9eb7fe700 1 -- 172.21.15.102:0/3250843761 learned_addr learned my addr 172.21.15.102:0/3250843761 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:01:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.401+0000 7fe9ea7fc700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9ec153290 0x7fe9ec0cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:00.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9eaffd700 1 -- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec1593c0 msgr2=0x7fe9ec1f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9eaffd700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec1593c0 0x7fe9ec1f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9eaffd700 1 -- 172.21.15.102:0/3250843761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9ec153290 msgr2=0x7fe9ec0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9eaffd700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9ec153290 0x7fe9ec0cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9eaffd700 1 -- 172.21.15.102:0/3250843761 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e4009b30 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9ea7fc700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9ec153290 0x7fe9ec0cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9eb7fe700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec1593c0 0x7fe9ec1f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9eaffd700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0c49c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00e900 tx=0x7fe9dc00ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:01:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9d37fe700 1 -- 172.21.15.102:0/3250843761 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00ee50 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9d37fe700 1 -- 172.21.15.102:0/3250843761 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc00c850 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec0ccde0 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9d37fe700 1 -- 172.21.15.102:0/3250843761 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00cbc0 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.402+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec15d160 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.404+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec0cd0c0 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.405+0000 7fe9d37fe700 1 -- 172.21.15.102:0/3250843761 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe9dc023070 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.406+0000 7fe9d37fe700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe9d80682f0 0x7fe9d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.406+0000 7fe9d37fe700 1 -- 172.21.15.102:0/3250843761 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe9dc085f70 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.406+0000 7fe9ea7fc700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe9d80682f0 0x7fe9d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.407+0000 7fe9ea7fc700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe9d80682f0 0x7fe9d806a7a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0cc780 tx=0x7fe9e0007dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:01:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.410+0000 7fe9d37fe700 1 -- 172.21.15.102:0/3250843761 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe9dc052680 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.648+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9ec1f1ea0 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.649+0000 7fe9d37fe700 1 -- 172.21.15.102:0/3250843761 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe9dc055ca0 con 0x7fe9ec0cd8e0 2023-03-03T14:01:00.650 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:01:00.650 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:01:00.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.651+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe9d80682f0 msgr2=0x7fe9d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:00.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.651+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe9d80682f0 0x7fe9d806a7a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0cc780 tx=0x7fe9e0007dd0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.651+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec0cd8e0 msgr2=0x7fe9ec0c49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:00.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.651+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0c49c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00e900 tx=0x7fe9dc00ec10 comp rx=0 tx=0).stop 2023-03-03T14:01:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 shutdown_connections 2023-03-03T14:01:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe9d80682f0 0x7fe9d806a7a0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe9ec0cd8e0 0x7fe9ec0c49c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9ec1593c0 0x7fe9ec1f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 --2- 172.21.15.102:0/3250843761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9ec153290 0x7fe9ec0cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 >> 172.21.15.102:0/3250843761 conn(0x7fe9ec065fb0 msgr2=0x7fe9ec073920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:01:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 shutdown_connections 2023-03-03T14:01:00.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:01:00.652+0000 7fe9f17ff700 1 -- 172.21.15.102:0/3250843761 wait complete. 2023-03-03T14:01:00.655 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:01:00.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T14:01:01.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:01.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:31.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:31.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:02.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:02.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:24.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:02:24.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.929+0000 7f749fdff700 1 Processor -- start 2023-03-03T14:02:24.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.929+0000 7f749fdff700 1 -- start start 2023-03-03T14:02:24.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.930+0000 7f749fdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749814fbe0 0x7f74980c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:24.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.930+0000 7f749fdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f74980716c0 0x7f74980706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:24.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.930+0000 7f749fdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74980c3c80 con 0x7f749814ffb0 2023-03-03T14:02:24.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.930+0000 7f749fdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74980c41a0 con 0x7f74980716c0 2023-03-03T14:02:24.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.930+0000 7f749fdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7498071cd0 con 0x7f749814fbe0 2023-03-03T14:02:24.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749d39a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f749814ffb0 0x7f74980c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40126/0 (socket says 172.21.15.102:40126) 2023-03-03T14:02:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749db9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749814fbe0 0x7f74980c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749d39a700 1 -- 172.21.15.102:0/1027526695 learned_addr learned my addr 172.21.15.102:0/1027526695 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:02:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749e39c700 1 --2- 172.21.15.102:0/1027526695 >> v2:172.21.15.102:3300/0 conn(0x7f74980716c0 0x7f74980706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749cb99700 1 -- 172.21.15.102:0/1027526695 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827004189 0 0) 0x7f7488001110 con 0x7f749814ffb0 2023-03-03T14:02:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749cb99700 1 -- 172.21.15.102:0/1027526695 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74840035b0 con 0x7f749814ffb0 2023-03-03T14:02:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749db9b700 1 -- 172.21.15.102:0/1027526695 >> v2:172.21.15.102:3300/0 conn(0x7f74980716c0 msgr2=0x7f74980706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:02:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749db9b700 1 --2- 172.21.15.102:0/1027526695 >> v2:172.21.15.102:3300/0 conn(0x7f74980716c0 0x7f74980706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749db9b700 1 -- 172.21.15.102:0/1027526695 >> v1:172.21.15.172:6789/0 conn(0x7f749814ffb0 legacy=0x7f74980c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.931+0000 7f749db9b700 1 -- 172.21.15.102:0/1027526695 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74980c3ec0 con 0x7f749814fbe0 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.932+0000 7f749db9b700 1 --2- 172.21.15.102:0/1027526695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749814fbe0 0x7f74980c3740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f749400bae0 tx=0x7f749400bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53069cd2e22aa9c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.932+0000 7f749cb99700 1 -- 172.21.15.102:0/1027526695 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749400dc70 con 0x7f749814fbe0 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.932+0000 7f749cb99700 1 -- 172.21.15.102:0/1027526695 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749400ddd0 con 0x7f749814fbe0 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.932+0000 7f749cb99700 1 -- 172.21.15.102:0/1027526695 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494010690 con 0x7f749814fbe0 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.932+0000 7f749fdff700 1 -- 172.21.15.102:0/1027526695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749814fbe0 msgr2=0x7f74980c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.932+0000 7f749fdff700 1 --2- 172.21.15.102:0/1027526695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749814fbe0 0x7f74980c3740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f749400bae0 tx=0x7f749400bdf0 comp rx=0 tx=0).stop 2023-03-03T14:02:24.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.933+0000 7f749fdff700 1 -- 172.21.15.102:0/1027526695 shutdown_connections 2023-03-03T14:02:24.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.933+0000 7f749fdff700 1 --2- 172.21.15.102:0/1027526695 >> v2:172.21.15.102:3300/0 conn(0x7f74980716c0 0x7f74980706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.933+0000 7f749fdff700 1 --2- 172.21.15.102:0/1027526695 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f749814fbe0 0x7f74980c3740 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.933+0000 7f749fdff700 1 -- 172.21.15.102:0/1027526695 >> 172.21.15.102:0/1027526695 conn(0x7f749806af90 msgr2=0x7f749806c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:02:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.933+0000 7f749fdff700 1 -- 172.21.15.102:0/1027526695 shutdown_connections 2023-03-03T14:02:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.933+0000 7f749fdff700 1 -- 172.21.15.102:0/1027526695 wait complete. 2023-03-03T14:02:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.935+0000 7f749fdff700 1 Processor -- start 2023-03-03T14:02:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.935+0000 7f749fdff700 1 -- start start 2023-03-03T14:02:24.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749fdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74980716c0 0x7f74981eda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:24.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749fdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749814fbe0 0x7f74980c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:24.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749fdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749814ffb0 0x7f74980c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:24.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749fdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749814c2e0 con 0x7f749814fbe0 2023-03-03T14:02:24.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749fdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749814bad0 con 0x7f749814ffb0 2023-03-03T14:02:24.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749fdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74981ee3f0 con 0x7f74980716c0 2023-03-03T14:02:24.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749d39a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749814fbe0 0x7f74980c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:24.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749d39a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749814fbe0 0x7f74980c98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41282/0 (socket says 172.21.15.102:41282) 2023-03-03T14:02:24.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749d39a700 1 -- 172.21.15.102:0/2003679409 learned_addr learned my addr 172.21.15.102:0/2003679409 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:02:24.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749db9b700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74980716c0 0x7f74981eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:24.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.936+0000 7f749e39c700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749814ffb0 0x7f74980c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:24.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f749d39a700 1 -- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74980716c0 msgr2=0x7f74981eda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:24.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f749d39a700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74980716c0 0x7f74981eda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:24.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f749d39a700 1 -- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749814ffb0 msgr2=0x7f74980c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:24.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f749d39a700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749814ffb0 0x7f74980c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:24.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f749d39a700 1 -- 172.21.15.102:0/2003679409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749400b740 con 0x7f749814fbe0 2023-03-03T14:02:24.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f749d39a700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749814fbe0 0x7f74980c98c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7488007fd0 tx=0x7f748800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:24.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f748e7fc700 1 -- 172.21.15.102:0/2003679409 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488005d70 con 0x7f749814fbe0 2023-03-03T14:02:24.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f748e7fc700 1 -- 172.21.15.102:0/2003679409 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74880070d0 con 0x7f749814fbe0 2023-03-03T14:02:24.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7498072250 con 0x7f749814fbe0 2023-03-03T14:02:24.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.937+0000 7f748e7fc700 1 -- 172.21.15.102:0/2003679409 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488007460 con 0x7f749814fbe0 2023-03-03T14:02:24.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.938+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7498071f70 con 0x7f749814fbe0 2023-03-03T14:02:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.939+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7498072530 con 0x7f749814fbe0 2023-03-03T14:02:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.940+0000 7f748e7fc700 1 -- 172.21.15.102:0/2003679409 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7488019030 con 0x7f749814fbe0 2023-03-03T14:02:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.941+0000 7f748e7fc700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f747c0682f0 0x7f747c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.941+0000 7f748e7fc700 1 -- 172.21.15.102:0/2003679409 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7488084ee0 con 0x7f749814fbe0 2023-03-03T14:02:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.941+0000 7f749db9b700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f747c0682f0 0x7f747c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:24.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.942+0000 7f749db9b700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f747c0682f0 0x7f747c06a7a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f749400be30 tx=0x7f7494013000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:24.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:24.945+0000 7f748e7fc700 1 -- 172.21.15.102:0/2003679409 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f74880515f0 con 0x7f749814fbe0 2023-03-03T14:02:25.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.183+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74980c3c80 con 0x7f749814fbe0 2023-03-03T14:02:25.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.184+0000 7f748e7fc700 1 -- 172.21.15.102:0/2003679409 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f74980c3c80 con 0x7f749814fbe0 2023-03-03T14:02:25.187 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:02:25.187 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:02:25.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f747c0682f0 msgr2=0x7f747c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:25.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f747c0682f0 0x7f747c06a7a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f749400be30 tx=0x7f7494013000 comp rx=0 tx=0).stop 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749814fbe0 msgr2=0x7f74980c98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749814fbe0 0x7f74980c98c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7488007fd0 tx=0x7f748800c7b0 comp rx=0 tx=0).stop 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 shutdown_connections 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f747c0682f0 0x7f747c06a7a0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74980716c0 0x7f74981eda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f749814ffb0 0x7f74980c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 --2- 172.21.15.102:0/2003679409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f749814fbe0 0x7f74980c98c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:25.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 >> 172.21.15.102:0/2003679409 conn(0x7f749806af90 msgr2=0x7f74980c7490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:02:25.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 shutdown_connections 2023-03-03T14:02:25.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:02:25.187+0000 7f749fdff700 1 -- 172.21.15.102:0/2003679409 wait complete. 2023-03-03T14:02:25.191 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:02:25.216 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T14:02:32.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:32.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:02.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:02.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:32.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:32.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:57.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:03:57.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.465+0000 7f3d255ff700 1 Processor -- start 2023-03-03T14:03:57.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.465+0000 7f3d255ff700 1 -- start start 2023-03-03T14:03:57.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.467+0000 7f3d255ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d20153260 0x7f3d200ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:57.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.467+0000 7f3d255ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d200cf940 con 0x7f3d200c9cb0 2023-03-03T14:03:57.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.467+0000 7f3d255ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d200cfb90 con 0x7f3d200cac20 2023-03-03T14:03:57.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.467+0000 7f3d255ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d200cfe70 con 0x7f3d20153260 2023-03-03T14:03:57.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.467+0000 7f3d1e7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3d200c9cb0 0x7f3d200cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50254/0 (socket says 172.21.15.102:50254) 2023-03-03T14:03:57.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.467+0000 7f3d1e7fc700 1 -- 172.21.15.102:0/253831873 learned_addr learned my addr 172.21.15.102:0/253831873 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:03:57.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.467+0000 7f3d1effd700 1 --2- 172.21.15.102:0/253831873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d20153260 0x7f3d200ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:57.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1dffb700 1 -- 172.21.15.102:0/253831873 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970719746 0 0) 0x7f3d200cfb90 con 0x7f3d200cac20 2023-03-03T14:03:57.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1dffb700 1 -- 172.21.15.102:0/253831873 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d080035b0 con 0x7f3d200cac20 2023-03-03T14:03:57.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1dffb700 1 -- 172.21.15.102:0/253831873 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922532951 0 0) 0x7f3d200cf940 con 0x7f3d200c9cb0 2023-03-03T14:03:57.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1effd700 1 -- 172.21.15.102:0/253831873 >> v1:172.21.15.172:6789/0 conn(0x7f3d200cac20 legacy=0x7f3d20163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:57.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1effd700 1 -- 172.21.15.102:0/253831873 >> v1:172.21.15.102:6789/0 conn(0x7f3d200c9cb0 legacy=0x7f3d200cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:57.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1effd700 1 -- 172.21.15.102:0/253831873 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20158f80 con 0x7f3d20153260 2023-03-03T14:03:57.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1effd700 1 --2- 172.21.15.102:0/253831873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d20153260 0x7f3d200ca6e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009870 tx=0x7f3d10009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a0e2e9ba9f5649a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:57.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1dffb700 1 -- 172.21.15.102:0/253831873 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10011830 con 0x7f3d20153260 2023-03-03T14:03:57.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1dffb700 1 -- 172.21.15.102:0/253831873 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d10011990 con 0x7f3d20153260 2023-03-03T14:03:57.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.468+0000 7f3d1dffb700 1 -- 172.21.15.102:0/253831873 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10011d20 con 0x7f3d20153260 2023-03-03T14:03:57.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.469+0000 7f3d255ff700 1 -- 172.21.15.102:0/253831873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d20153260 msgr2=0x7f3d200ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:57.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.469+0000 7f3d255ff700 1 --2- 172.21.15.102:0/253831873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d20153260 0x7f3d200ca6e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009870 tx=0x7f3d10009b80 comp rx=0 tx=0).stop 2023-03-03T14:03:57.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.469+0000 7f3d255ff700 1 -- 172.21.15.102:0/253831873 shutdown_connections 2023-03-03T14:03:57.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.469+0000 7f3d255ff700 1 --2- 172.21.15.102:0/253831873 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d20153260 0x7f3d200ca6e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:57.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.469+0000 7f3d255ff700 1 -- 172.21.15.102:0/253831873 >> 172.21.15.102:0/253831873 conn(0x7f3d20066c00 msgr2=0x7f3d200ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:03:57.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.469+0000 7f3d255ff700 1 -- 172.21.15.102:0/253831873 shutdown_connections 2023-03-03T14:03:57.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.469+0000 7f3d255ff700 1 -- 172.21.15.102:0/253831873 wait complete. 2023-03-03T14:03:57.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.471+0000 7f3d255ff700 1 Processor -- start 2023-03-03T14:03:57.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.471+0000 7f3d255ff700 1 -- start start 2023-03-03T14:03:57.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.472+0000 7f3d255ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d200cac20 0x7f3d201e5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:57.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.472+0000 7f3d255ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d20153260 0x7f3d2014e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:57.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.472+0000 7f3d255ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d201536e0 0x7f3d2014e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:57.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.472+0000 7f3d255ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20153b10 con 0x7f3d201536e0 2023-03-03T14:03:57.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.472+0000 7f3d255ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d201f1590 con 0x7f3d20153260 2023-03-03T14:03:57.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.472+0000 7f3d255ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20153d60 con 0x7f3d200cac20 2023-03-03T14:03:57.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d200cac20 0x7f3d201e5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:57.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1e7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d20153260 0x7f3d2014e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:57.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d200cac20 0x7f3d201e5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57944/0 (socket says 172.21.15.102:57944) 2023-03-03T14:03:57.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 -- 172.21.15.102:0/770226306 learned_addr learned my addr 172.21.15.102:0/770226306 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:03:57.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1f7fe700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d201536e0 0x7f3d2014e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:57.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 -- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d20153260 msgr2=0x7f3d2014e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:57.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d20153260 0x7f3d2014e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:57.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 -- 172.21.15.102:0/770226306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d201536e0 msgr2=0x7f3d2014e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:57.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d201536e0 0x7f3d2014e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:57.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 -- 172.21.15.102:0/770226306 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d100094a0 con 0x7f3d200cac20 2023-03-03T14:03:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.473+0000 7f3d1effd700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d200cac20 0x7f3d201e5a00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3d100090b0 tx=0x7f3d10004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.474+0000 7f3d077fe700 1 -- 172.21.15.102:0/770226306 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1001a7e0 con 0x7f3d200cac20 2023-03-03T14:03:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.474+0000 7f3d077fe700 1 -- 172.21.15.102:0/770226306 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1001a940 con 0x7f3d200cac20 2023-03-03T14:03:57.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.474+0000 7f3d077fe700 1 -- 172.21.15.102:0/770226306 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1001acf0 con 0x7f3d200cac20 2023-03-03T14:03:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.474+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d20159540 con 0x7f3d200cac20 2023-03-03T14:03:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.474+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d20159260 con 0x7f3d200cac20 2023-03-03T14:03:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.475+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d20159820 con 0x7f3d200cac20 2023-03-03T14:03:57.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.476+0000 7f3d077fe700 1 -- 172.21.15.102:0/770226306 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d1000f050 con 0x7f3d200cac20 2023-03-03T14:03:57.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.477+0000 7f3d077fe700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3d0c0682f0 0x7f3d0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:57.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.477+0000 7f3d077fe700 1 -- 172.21.15.102:0/770226306 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3d10087370 con 0x7f3d200cac20 2023-03-03T14:03:57.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.478+0000 7f3d1e7fc700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3d0c0682f0 0x7f3d0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:57.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.479+0000 7f3d1e7fc700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3d0c0682f0 0x7f3d0c06a7a0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f3d14001110 tx=0x7f3d14001080 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:57.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.481+0000 7f3d077fe700 1 -- 172.21.15.102:0/770226306 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d10053b00 con 0x7f3d200cac20 2023-03-03T14:03:57.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.715+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d2004e6e0 con 0x7f3d200cac20 2023-03-03T14:03:57.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.716+0000 7f3d077fe700 1 -- 172.21.15.102:0/770226306 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3d10057120 con 0x7f3d200cac20 2023-03-03T14:03:57.718 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:03:57.719 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:03:57.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3d0c0682f0 msgr2=0x7f3d0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:57.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3d0c0682f0 0x7f3d0c06a7a0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f3d14001110 tx=0x7f3d14001080 comp rx=0 tx=0).stop 2023-03-03T14:03:57.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d200cac20 msgr2=0x7f3d201e5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:57.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d200cac20 0x7f3d201e5a00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3d100090b0 tx=0x7f3d10004750 comp rx=0 tx=0).stop 2023-03-03T14:03:57.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 shutdown_connections 2023-03-03T14:03:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3d0c0682f0 0x7f3d0c06a7a0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d200cac20 0x7f3d201e5a00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d201536e0 0x7f3d2014e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 --2- 172.21.15.102:0/770226306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d20153260 0x7f3d2014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.718+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 >> 172.21.15.102:0/770226306 conn(0x7f3d200c9cb0 msgr2=0x7f3d200cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:03:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.719+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 shutdown_connections 2023-03-03T14:03:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:03:57.719+0000 7f3d255ff700 1 -- 172.21.15.102:0/770226306 wait complete. 2023-03-03T14:03:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:03:57.749 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 103 secs before thrashing 2023-03-03T14:04:02.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:02.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:33.012 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:33.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:03.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:03.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:33.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:33.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:40.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:05:41.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.017+0000 7f5ea77ff700 1 Processor -- start 2023-03-03T14:05:41.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.017+0000 7f5ea77ff700 1 -- start start 2023-03-03T14:05:41.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.018+0000 7f5ea77ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ea0150c80 0x7f5ea006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:41.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.018+0000 7f5ea77ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea00c5390 con 0x7f5ea00c5f30 2023-03-03T14:05:41.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.018+0000 7f5ea77ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea00c4a20 con 0x7f5ea00c2020 2023-03-03T14:05:41.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.018+0000 7f5ea77ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea0157ad0 con 0x7f5ea0150c80 2023-03-03T14:05:41.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5ea559b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ea0150c80 0x7f5ea006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:41.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5ea5d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5ea00c5f30 0x7f5ea00c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57122/0 (socket says 172.21.15.102:57122) 2023-03-03T14:05:41.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5ea4d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5ea00c2020 0x7f5ea00c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41004/0 (socket says 172.21.15.102:41004) 2023-03-03T14:05:41.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5ea5d9c700 1 -- 172.21.15.102:0/3015624174 learned_addr learned my addr 172.21.15.102:0/3015624174 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:05:41.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5e97fff700 1 -- 172.21.15.102:0/3015624174 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847781077 0 0) 0x7f5ea00c4a20 con 0x7f5ea00c2020 2023-03-03T14:05:41.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5ea559b700 1 -- 172.21.15.102:0/3015624174 >> v1:172.21.15.172:6789/0 conn(0x7f5ea00c2020 legacy=0x7f5ea00c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:41.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5ea559b700 1 -- 172.21.15.102:0/3015624174 >> v1:172.21.15.102:6789/0 conn(0x7f5ea00c5f30 legacy=0x7f5ea00c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:41.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.019+0000 7f5ea559b700 1 -- 172.21.15.102:0/3015624174 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea006f010 con 0x7f5ea0150c80 2023-03-03T14:05:41.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.020+0000 7f5ea559b700 1 --2- 172.21.15.102:0/3015624174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea006d100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009840 tx=0x7f5e9c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3001878b28ce03b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:41.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.020+0000 7f5e97fff700 1 -- 172.21.15.102:0/3015624174 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c004230 con 0x7f5ea0150c80 2023-03-03T14:05:41.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.020+0000 7f5e97fff700 1 -- 172.21.15.102:0/3015624174 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c004390 con 0x7f5ea0150c80 2023-03-03T14:05:41.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.020+0000 7f5e97fff700 1 -- 172.21.15.102:0/3015624174 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0036a0 con 0x7f5ea0150c80 2023-03-03T14:05:41.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.020+0000 7f5ea77ff700 1 -- 172.21.15.102:0/3015624174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 msgr2=0x7f5ea006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:41.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.020+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/3015624174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea006d100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009840 tx=0x7f5e9c009b50 comp rx=0 tx=0).stop 2023-03-03T14:05:41.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.021+0000 7f5ea77ff700 1 -- 172.21.15.102:0/3015624174 shutdown_connections 2023-03-03T14:05:41.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.021+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/3015624174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea006d100 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:41.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.021+0000 7f5ea77ff700 1 -- 172.21.15.102:0/3015624174 >> 172.21.15.102:0/3015624174 conn(0x7f5ea006af90 msgr2=0x7f5ea006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:41.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.021+0000 7f5ea77ff700 1 -- 172.21.15.102:0/3015624174 shutdown_connections 2023-03-03T14:05:41.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.021+0000 7f5ea77ff700 1 -- 172.21.15.102:0/3015624174 wait complete. 2023-03-03T14:05:41.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.023+0000 7f5ea77ff700 1 Processor -- start 2023-03-03T14:05:41.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.023+0000 7f5ea77ff700 1 -- start start 2023-03-03T14:05:41.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea77ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea00c2020 0x7f5ea0144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:41.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea77ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ea00c5f30 0x7f5ea00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:41.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea77ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea0149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:41.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea77ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01ef570 con 0x7f5ea00c5f30 2023-03-03T14:05:41.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea77ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea00ca780 con 0x7f5ea0150c80 2023-03-03T14:05:41.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea77ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea0149770 con 0x7f5ea00c2020 2023-03-03T14:05:41.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea559b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea00c2020 0x7f5ea0144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:41.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea5d9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea0149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:41.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea559b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea00c2020 0x7f5ea0144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:43014/0 (socket says 172.21.15.102:43014) 2023-03-03T14:05:41.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea559b700 1 -- 172.21.15.102:0/126124476 learned_addr learned my addr 172.21.15.102:0/126124476 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:05:41.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.024+0000 7f5ea4d9a700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ea00c5f30 0x7f5ea00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:41.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5ea5d9c700 1 -- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea00c2020 msgr2=0x7f5ea0144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:41.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5ea5d9c700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea00c2020 0x7f5ea0144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:41.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5ea5d9c700 1 -- 172.21.15.102:0/126124476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ea00c5f30 msgr2=0x7f5ea00717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:41.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5ea5d9c700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ea00c5f30 0x7f5ea00717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:41.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5ea5d9c700 1 -- 172.21.15.102:0/126124476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0094a0 con 0x7f5ea0150c80 2023-03-03T14:05:41.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5ea559b700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea00c2020 0x7f5ea0144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:05:41.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5ea5d9c700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea0149f30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800cc80 tx=0x7f5e9800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:41.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5e95ffb700 1 -- 172.21.15.102:0/126124476 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9800aa80 con 0x7f5ea0150c80 2023-03-03T14:05:41.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5e95ffb700 1 -- 172.21.15.102:0/126124476 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9800abe0 con 0x7f5ea0150c80 2023-03-03T14:05:41.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.025+0000 7f5e95ffb700 1 -- 172.21.15.102:0/126124476 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e980070d0 con 0x7f5ea0150c80 2023-03-03T14:05:41.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.026+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea01577f0 con 0x7f5ea0150c80 2023-03-03T14:05:41.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.026+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea0157510 con 0x7f5ea0150c80 2023-03-03T14:05:41.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.027+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea00c4cc0 con 0x7f5ea0150c80 2023-03-03T14:05:41.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.028+0000 7f5e95ffb700 1 -- 172.21.15.102:0/126124476 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5e98010070 con 0x7f5ea0150c80 2023-03-03T14:05:41.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.029+0000 7f5e95ffb700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e84068320 0x7f5e8406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:41.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.029+0000 7f5e95ffb700 1 -- 172.21.15.102:0/126124476 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5e98085730 con 0x7f5ea0150c80 2023-03-03T14:05:41.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.030+0000 7f5ea559b700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e84068320 0x7f5e8406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:41.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.030+0000 7f5ea559b700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e84068320 0x7f5e8406a7d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c000c00 tx=0x7f5e9c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:41.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.034+0000 7f5e95ffb700 1 -- 172.21.15.102:0/126124476 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e98051d50 con 0x7f5ea0150c80 2023-03-03T14:05:41.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.269+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ea01eeff0 con 0x7f5ea0150c80 2023-03-03T14:05:41.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.270+0000 7f5e95ffb700 1 -- 172.21.15.102:0/126124476 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5e98018020 con 0x7f5ea0150c80 2023-03-03T14:05:41.273 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:05:41.273 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:05:41.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e84068320 msgr2=0x7f5e8406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:41.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e84068320 0x7f5e8406a7d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c000c00 tx=0x7f5e9c008030 comp rx=0 tx=0).stop 2023-03-03T14:05:41.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 msgr2=0x7f5ea0149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea0149f30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800cc80 tx=0x7f5e9800cf90 comp rx=0 tx=0).stop 2023-03-03T14:05:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 shutdown_connections 2023-03-03T14:05:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e84068320 0x7f5e8406a7d0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5ea00c2020 0x7f5ea0144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea0150c80 0x7f5ea0149f30 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 --2- 172.21.15.102:0/126124476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ea00c5f30 0x7f5ea00717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:41.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 >> 172.21.15.102:0/126124476 conn(0x7f5ea006af90 msgr2=0x7f5ea00c63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.273+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 shutdown_connections 2023-03-03T14:05:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:05:41.274+0000 7f5ea77ff700 1 -- 172.21.15.102:0/126124476 wait complete. 2023-03-03T14:05:41.277 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:05:41.302 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-03-03T14:06:03.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:03.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:10.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:06:10.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.568+0000 7f80217ff700 1 Processor -- start 2023-03-03T14:06:10.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.568+0000 7f80217ff700 1 -- start start 2023-03-03T14:06:10.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f80217ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c153290 0x7f801c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:10.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f80217ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f801c0cd8e0 0x7f801c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:10.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f80217ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c153660 con 0x7f801c1593c0 2023-03-03T14:06:10.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f80217ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c06c360 con 0x7f801c0cd8e0 2023-03-03T14:06:10.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f80217ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c15cea0 con 0x7f801c153290 2023-03-03T14:06:10.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f801affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c153290 0x7f801c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:10.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f801a7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f801c1593c0 0x7f801c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38620/0 (socket says 172.21.15.102:38620) 2023-03-03T14:06:10.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f801a7fc700 1 -- 172.21.15.102:0/3660584316 learned_addr learned my addr 172.21.15.102:0/3660584316 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:06:10.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.570+0000 7f801b7fe700 1 --2- 172.21.15.102:0/3660584316 >> v2:172.21.15.102:3300/0 conn(0x7f801c0cd8e0 0x7f801c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:10.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f801affd700 1 -- 172.21.15.102:0/3660584316 >> v2:172.21.15.102:3300/0 conn(0x7f801c0cd8e0 msgr2=0x7f801c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:06:10.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f801affd700 1 --2- 172.21.15.102:0/3660584316 >> v2:172.21.15.102:3300/0 conn(0x7f801c0cd8e0 0x7f801c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f8019ffb700 1 -- 172.21.15.102:0/3660584316 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231676363 0 0) 0x7f801c153660 con 0x7f801c1593c0 2023-03-03T14:06:10.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f801affd700 1 -- 172.21.15.102:0/3660584316 >> v1:172.21.15.172:6789/0 conn(0x7f801c1593c0 legacy=0x7f801c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:10.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f801affd700 1 -- 172.21.15.102:0/3660584316 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c06c7f0 con 0x7f801c153290 2023-03-03T14:06:10.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f801affd700 1 --2- 172.21.15.102:0/3660584316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c153290 0x7f801c158e80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f800c009840 tx=0x7f800c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e6c20053fbc5cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:10.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f8019ffb700 1 -- 172.21.15.102:0/3660584316 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c011830 con 0x7f801c153290 2023-03-03T14:06:10.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f8019ffb700 1 -- 172.21.15.102:0/3660584316 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800c011990 con 0x7f801c153290 2023-03-03T14:06:10.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.571+0000 7f8019ffb700 1 -- 172.21.15.102:0/3660584316 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c011d20 con 0x7f801c153290 2023-03-03T14:06:10.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 -- 172.21.15.102:0/3660584316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c153290 msgr2=0x7f801c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:10.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 --2- 172.21.15.102:0/3660584316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c153290 0x7f801c158e80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f800c009840 tx=0x7f800c009b50 comp rx=0 tx=0).stop 2023-03-03T14:06:10.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 -- 172.21.15.102:0/3660584316 shutdown_connections 2023-03-03T14:06:10.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 --2- 172.21.15.102:0/3660584316 >> v2:172.21.15.102:3300/0 conn(0x7f801c0cd8e0 0x7f801c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 --2- 172.21.15.102:0/3660584316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c153290 0x7f801c158e80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 -- 172.21.15.102:0/3660584316 >> 172.21.15.102:0/3660584316 conn(0x7f801c065fb0 msgr2=0x7f801c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:06:10.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 -- 172.21.15.102:0/3660584316 shutdown_connections 2023-03-03T14:06:10.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.572+0000 7f80217ff700 1 -- 172.21.15.102:0/3660584316 wait complete. 2023-03-03T14:06:10.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.574+0000 7f80217ff700 1 Processor -- start 2023-03-03T14:06:10.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.574+0000 7f80217ff700 1 -- start start 2023-03-03T14:06:10.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.575+0000 7f80217ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f801c0cd8e0 0x7f801c0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:10.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.575+0000 7f80217ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f801c153290 0x7f801c0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:10.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.575+0000 7f80217ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c1593c0 0x7f801c0c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.575+0000 7f80217ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c150d20 con 0x7f801c153290 2023-03-03T14:06:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.575+0000 7f80217ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0c8220 con 0x7f801c0cd8e0 2023-03-03T14:06:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.575+0000 7f80217ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c1f3320 con 0x7f801c1593c0 2023-03-03T14:06:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f801c0cd8e0 0x7f801c0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801a7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f801c153290 0x7f801c0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f801c0cd8e0 0x7f801c0c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:51910/0 (socket says 172.21.15.102:51910) 2023-03-03T14:06:10.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801affd700 1 -- 172.21.15.102:0/2007448001 learned_addr learned my addr 172.21.15.102:0/2007448001 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:06:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801b7fe700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c1593c0 0x7f801c0c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801a7fc700 1 -- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c1593c0 msgr2=0x7f801c0c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801a7fc700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c1593c0 0x7f801c0c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801a7fc700 1 -- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f801c0cd8e0 msgr2=0x7f801c0c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801a7fc700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f801c0cd8e0 0x7f801c0c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801a7fc700 1 -- 172.21.15.102:0/2007448001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c0094a0 con 0x7f801c153290 2023-03-03T14:06:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.576+0000 7f801affd700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f801c0cd8e0 0x7f801c0c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:06:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.577+0000 7f801a7fc700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f801c153290 0x7f801c0c8b60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f801000c4e0 tx=0x7f801000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.577+0000 7f80037fe700 1 -- 172.21.15.102:0/2007448001 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010004c80 con 0x7f801c153290 2023-03-03T14:06:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.577+0000 7f80037fe700 1 -- 172.21.15.102:0/2007448001 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010004e00 con 0x7f801c153290 2023-03-03T14:06:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.577+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c0ccde0 con 0x7f801c153290 2023-03-03T14:06:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.577+0000 7f80037fe700 1 -- 172.21.15.102:0/2007448001 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010007310 con 0x7f801c153290 2023-03-03T14:06:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.577+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c15d160 con 0x7f801c153290 2023-03-03T14:06:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.578+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c0cd0c0 con 0x7f801c153290 2023-03-03T14:06:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.579+0000 7f80037fe700 1 -- 172.21.15.102:0/2007448001 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f801001a020 con 0x7f801c153290 2023-03-03T14:06:10.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.581+0000 7f80037fe700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80080682f0 0x7f800806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.581+0000 7f80037fe700 1 -- 172.21.15.102:0/2007448001 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8010085890 con 0x7f801c153290 2023-03-03T14:06:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.581+0000 7f801affd700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80080682f0 0x7f800806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.582+0000 7f801affd700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80080682f0 0x7f800806a7a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f800c000c00 tx=0x7f800c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.585+0000 7f80037fe700 1 -- 172.21.15.102:0/2007448001 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8010051fe0 con 0x7f801c153290 2023-03-03T14:06:10.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.819+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f801c04e6e0 con 0x7f801c153290 2023-03-03T14:06:10.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.820+0000 7f80037fe700 1 -- 172.21.15.102:0/2007448001 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8010055600 con 0x7f801c153290 2023-03-03T14:06:10.822 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:06:10.823 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:06:10.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.822+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80080682f0 msgr2=0x7f800806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:10.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.822+0000 7f80217ff700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80080682f0 0x7f800806a7a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f800c000c00 tx=0x7f800c008030 comp rx=0 tx=0).stop 2023-03-03T14:06:10.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.822+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f801c153290 msgr2=0x7f801c0c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:10.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.822+0000 7f80217ff700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f801c153290 0x7f801c0c8b60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f801000c4e0 tx=0x7f801000c7f0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.822+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 shutdown_connections 2023-03-03T14:06:10.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.822+0000 7f80217ff700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80080682f0 0x7f800806a7a0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.823+0000 7f80217ff700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f801c1593c0 0x7f801c0c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.823+0000 7f80217ff700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f801c0cd8e0 0x7f801c0c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.823+0000 7f80217ff700 1 --2- 172.21.15.102:0/2007448001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f801c153290 0x7f801c0c8b60 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.823+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 >> 172.21.15.102:0/2007448001 conn(0x7f801c065fb0 msgr2=0x7f801c156b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:06:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.823+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 shutdown_connections 2023-03-03T14:06:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:06:10.823+0000 7f80217ff700 1 -- 172.21.15.102:0/2007448001 wait complete. 2023-03-03T14:06:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:06:10.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-03-03T14:06:33.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:33.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:03.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:03.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:33.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:07:34.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:34.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:34.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.126+0000 7fbb35dff700 1 Processor -- start 2023-03-03T14:07:34.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.126+0000 7fbb35dff700 1 -- start start 2023-03-03T14:07:34.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb35dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb30153290 0x7fbb30158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:34.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb35dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:34.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb35dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb30153660 con 0x7fbb301593c0 2023-03-03T14:07:34.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb35dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3006c360 con 0x7fbb30153290 2023-03-03T14:07:34.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb35dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3015cea0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb2f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb30153290 0x7fbb30158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:34.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb2effd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbb301593c0 0x7fbb300cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32924/0 (socket says 172.21.15.102:32924) 2023-03-03T14:07:34.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb2f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb30153290 0x7fbb30158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33312/0 (socket says 172.21.15.102:33312) 2023-03-03T14:07:34.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb2effd700 1 -- 172.21.15.102:0/1497873564 learned_addr learned my addr 172.21.15.102:0/1497873564 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:07:34.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.128+0000 7fbb2ffff700 1 --2- 172.21.15.102:0/1497873564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:34.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.129+0000 7fbb2f7fe700 1 -- 172.21.15.102:0/1497873564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 msgr2=0x7fbb300ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:07:34.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.129+0000 7fbb2f7fe700 1 --2- 172.21.15.102:0/1497873564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.129+0000 7fbb2f7fe700 1 -- 172.21.15.102:0/1497873564 >> v1:172.21.15.102:6789/0 conn(0x7fbb301593c0 legacy=0x7fbb300cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:34.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.129+0000 7fbb2e7fc700 1 -- 172.21.15.102:0/1497873564 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468284818 0 0) 0x7fbb30153660 con 0x7fbb301593c0 2023-03-03T14:07:34.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.129+0000 7fbb2f7fe700 1 -- 172.21.15.102:0/1497873564 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3006c7f0 con 0x7fbb30153290 2023-03-03T14:07:34.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.129+0000 7fbb2f7fe700 1 --2- 172.21.15.102:0/1497873564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 0x7fbb30158e80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009840 tx=0x7fbb20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6dcb0170a1426e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:34.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb2e7fc700 1 -- 172.21.15.102:0/1497873564 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20011830 con 0x7fbb30153290 2023-03-03T14:07:34.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb2e7fc700 1 -- 172.21.15.102:0/1497873564 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb20011990 con 0x7fbb30153290 2023-03-03T14:07:34.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb2e7fc700 1 -- 172.21.15.102:0/1497873564 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20011d20 con 0x7fbb30153290 2023-03-03T14:07:34.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 -- 172.21.15.102:0/1497873564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 msgr2=0x7fbb30158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:34.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 --2- 172.21.15.102:0/1497873564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 0x7fbb30158e80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009840 tx=0x7fbb20009b50 comp rx=0 tx=0).stop 2023-03-03T14:07:34.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 -- 172.21.15.102:0/1497873564 shutdown_connections 2023-03-03T14:07:34.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 --2- 172.21.15.102:0/1497873564 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 --2- 172.21.15.102:0/1497873564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 0x7fbb30158e80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 -- 172.21.15.102:0/1497873564 >> 172.21.15.102:0/1497873564 conn(0x7fbb30065fb0 msgr2=0x7fbb300663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:07:34.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 -- 172.21.15.102:0/1497873564 shutdown_connections 2023-03-03T14:07:34.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.130+0000 7fbb35dff700 1 -- 172.21.15.102:0/1497873564 wait complete. 2023-03-03T14:07:34.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.133+0000 7fbb35dff700 1 Processor -- start 2023-03-03T14:07:34.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.134+0000 7fbb35dff700 1 -- start start 2023-03-03T14:07:34.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb35dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:34.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb35dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 0x7fbb3014e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:34.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb35dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb301593c0 0x7fbb3014e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:34.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb35dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b1f90 con 0x7fbb30153290 2023-03-03T14:07:34.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb35dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b2860 con 0x7fbb301593c0 2023-03-03T14:07:34.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb35dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b2180 con 0x7fbb300cd8e0 2023-03-03T14:07:34.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb2f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:34.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb2effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 0x7fbb3014e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:34.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb2ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb301593c0 0x7fbb3014e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:34.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb2f7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:47046/0 (socket says 172.21.15.102:47046) 2023-03-03T14:07:34.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.135+0000 7fbb2f7fe700 1 -- 172.21.15.102:0/2910201587 learned_addr learned my addr 172.21.15.102:0/2910201587 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:07:34.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb2f7fe700 1 -- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb301593c0 msgr2=0x7fbb3014e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:34.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb2f7fe700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb301593c0 0x7fbb3014e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb2f7fe700 1 -- 172.21.15.102:0/2910201587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 msgr2=0x7fbb3014e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:34.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb2f7fe700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 0x7fbb3014e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb2f7fe700 1 -- 172.21.15.102:0/2910201587 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200094a0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb2f7fe700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300c4750 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbb200090b0 tx=0x7fbb20004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:34.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb17fff700 1 -- 172.21.15.102:0/2910201587 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb200036a0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb17fff700 1 -- 172.21.15.102:0/2910201587 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2001e070 con 0x7fbb300cd8e0 2023-03-03T14:07:34.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.136+0000 7fbb17fff700 1 -- 172.21.15.102:0/2910201587 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20003a60 con 0x7fbb300cd8e0 2023-03-03T14:07:34.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.137+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb300ccde0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.137+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3015d160 con 0x7fbb300cd8e0 2023-03-03T14:07:34.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.138+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb300cd0c0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.139+0000 7fbb17fff700 1 -- 172.21.15.102:0/2910201587 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb2000f050 con 0x7fbb300cd8e0 2023-03-03T14:07:34.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.140+0000 7fbb17fff700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb1c068320 0x7fbb1c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:34.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.140+0000 7fbb17fff700 1 -- 172.21.15.102:0/2910201587 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbb20086ca0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.140+0000 7fbb2effd700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb1c068320 0x7fbb1c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:34.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.141+0000 7fbb2effd700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb1c068320 0x7fbb1c06a7d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fbb3015bdd0 tx=0x7fbb24007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:34.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.144+0000 7fbb17fff700 1 -- 172.21.15.102:0/2910201587 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb200533b0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.386+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb3004e6e0 con 0x7fbb300cd8e0 2023-03-03T14:07:34.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.387+0000 7fbb17fff700 1 -- 172.21.15.102:0/2910201587 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbb20020090 con 0x7fbb300cd8e0 2023-03-03T14:07:34.389 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:07:34.389 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:07:34.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.389+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb1c068320 msgr2=0x7fbb1c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:34.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.389+0000 7fbb35dff700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb1c068320 0x7fbb1c06a7d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fbb3015bdd0 tx=0x7fbb24007de0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.389+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 msgr2=0x7fbb300c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:34.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.389+0000 7fbb35dff700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300c4750 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbb200090b0 tx=0x7fbb20004750 comp rx=0 tx=0).stop 2023-03-03T14:07:34.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 shutdown_connections 2023-03-03T14:07:34.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb1c068320 0x7fbb1c06a7d0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb300cd8e0 0x7fbb300c4750 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb301593c0 0x7fbb3014e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 --2- 172.21.15.102:0/2910201587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb30153290 0x7fbb3014e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:34.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 >> 172.21.15.102:0/2910201587 conn(0x7fbb30065fb0 msgr2=0x7fbb30150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:07:34.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 shutdown_connections 2023-03-03T14:07:34.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:34.390+0000 7fbb35dff700 1 -- 172.21.15.102:0/2910201587 wait complete. 2023-03-03T14:07:34.394 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:07:34.422 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before thrashing 2023-03-03T14:07:38.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:07:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.672+0000 7fb77c7ff700 1 Processor -- start 2023-03-03T14:07:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.673+0000 7fb77c7ff700 1 -- start start 2023-03-03T14:07:38.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.674+0000 7fb77c7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb774153290 0x7fb774158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:38.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.674+0000 7fb77c7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb7740cd8e0 0x7fb7740ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:38.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.674+0000 7fb77c7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb774153660 con 0x7fb7741593c0 2023-03-03T14:07:38.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.674+0000 7fb77c7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77406c360 con 0x7fb7740cd8e0 2023-03-03T14:07:38.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.674+0000 7fb77c7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77415cea0 con 0x7fb774153290 2023-03-03T14:07:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.674+0000 7fb77a59b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb774153290 0x7fb774158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb77ad9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb7740cd8e0 0x7fb7740ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb779d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb7741593c0 0x7fb7740cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46688/0 (socket says 172.21.15.102:46688) 2023-03-03T14:07:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb779d9a700 1 -- 172.21.15.102:0/3759212822 learned_addr learned my addr 172.21.15.102:0/3759212822 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:07:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb779599700 1 -- 172.21.15.102:0/3759212822 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282650736 0 0) 0x7fb774153660 con 0x7fb7741593c0 2023-03-03T14:07:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb779599700 1 -- 172.21.15.102:0/3759212822 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7600035b0 con 0x7fb7741593c0 2023-03-03T14:07:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb77a59b700 1 -- 172.21.15.102:0/3759212822 >> v2:172.21.15.172:3300/0 conn(0x7fb7740cd8e0 msgr2=0x7fb7740ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:07:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb77a59b700 1 --2- 172.21.15.102:0/3759212822 >> v2:172.21.15.172:3300/0 conn(0x7fb7740cd8e0 0x7fb7740ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb77a59b700 1 -- 172.21.15.102:0/3759212822 >> v1:172.21.15.172:6789/0 conn(0x7fb7741593c0 legacy=0x7fb7740cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb77a59b700 1 -- 172.21.15.102:0/3759212822 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77406c7f0 con 0x7fb774153290 2023-03-03T14:07:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb77ad9c700 1 --2- 172.21.15.102:0/3759212822 >> v2:172.21.15.172:3300/0 conn(0x7fb7740cd8e0 0x7fb7740ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:07:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.675+0000 7fb77a59b700 1 --2- 172.21.15.102:0/3759212822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb774153290 0x7fb774158e80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb770009840 tx=0x7fb770009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5189926f77b4bcf7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb779599700 1 -- 172.21.15.102:0/3759212822 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770011830 con 0x7fb774153290 2023-03-03T14:07:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb779599700 1 -- 172.21.15.102:0/3759212822 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb770011990 con 0x7fb774153290 2023-03-03T14:07:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb779599700 1 -- 172.21.15.102:0/3759212822 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770011d20 con 0x7fb774153290 2023-03-03T14:07:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb77c7ff700 1 -- 172.21.15.102:0/3759212822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb774153290 msgr2=0x7fb774158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/3759212822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb774153290 0x7fb774158e80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb770009840 tx=0x7fb770009b50 comp rx=0 tx=0).stop 2023-03-03T14:07:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb77c7ff700 1 -- 172.21.15.102:0/3759212822 shutdown_connections 2023-03-03T14:07:38.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/3759212822 >> v2:172.21.15.172:3300/0 conn(0x7fb7740cd8e0 0x7fb7740ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/3759212822 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb774153290 0x7fb774158e80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.676+0000 7fb77c7ff700 1 -- 172.21.15.102:0/3759212822 >> 172.21.15.102:0/3759212822 conn(0x7fb774065fb0 msgr2=0x7fb7740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:07:38.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.677+0000 7fb77c7ff700 1 -- 172.21.15.102:0/3759212822 shutdown_connections 2023-03-03T14:07:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.677+0000 7fb77c7ff700 1 -- 172.21.15.102:0/3759212822 wait complete. 2023-03-03T14:07:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.678+0000 7fb77c7ff700 1 Processor -- start 2023-03-03T14:07:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.679+0000 7fb77c7ff700 1 -- start start 2023-03-03T14:07:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77c7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7740cd8e0 0x7fb7741544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77c7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb774153290 0x7fb7741f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77c7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7741593c0 0x7fb7740b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77c7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741f2560 con 0x7fb7740cd8e0 2023-03-03T14:07:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77c7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741f22e0 con 0x7fb774153290 2023-03-03T14:07:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77c7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740c62e0 con 0x7fb7741593c0 2023-03-03T14:07:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77a59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7740cd8e0 0x7fb7741544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77a59b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7740cd8e0 0x7fb7741544f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37990/0 (socket says 172.21.15.102:37990) 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77a59b700 1 -- 172.21.15.102:0/644661991 learned_addr learned my addr 172.21.15.102:0/644661991 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb77ad9c700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7741593c0 0x7fb7740b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.680+0000 7fb779d9a700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb774153290 0x7fb7741f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77ad9c700 1 -- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb774153290 msgr2=0x7fb7741f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77ad9c700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb774153290 0x7fb7741f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77ad9c700 1 -- 172.21.15.102:0/644661991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7740cd8e0 msgr2=0x7fb7741544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77ad9c700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7740cd8e0 0x7fb7741544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77ad9c700 1 -- 172.21.15.102:0/644661991 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c009a70 con 0x7fb7741593c0 2023-03-03T14:07:38.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77a59b700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7740cd8e0 0x7fb7741544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:07:38.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77ad9c700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7741593c0 0x7fb7740b2cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009db0 tx=0x7fb76c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:38.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb76affd700 1 -- 172.21.15.102:0/644661991 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c009c20 con 0x7fb7741593c0 2023-03-03T14:07:38.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb76affd700 1 -- 172.21.15.102:0/644661991 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c00f040 con 0x7fb7741593c0 2023-03-03T14:07:38.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb76affd700 1 -- 172.21.15.102:0/644661991 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c00ccb0 con 0x7fb7741593c0 2023-03-03T14:07:38.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7700094a0 con 0x7fb7741593c0 2023-03-03T14:07:38.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.681+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7740ccde0 con 0x7fb7741593c0 2023-03-03T14:07:38.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.683+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77415d160 con 0x7fb7741593c0 2023-03-03T14:07:38.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.684+0000 7fb76affd700 1 -- 172.21.15.102:0/644661991 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb76c014480 con 0x7fb7741593c0 2023-03-03T14:07:38.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.685+0000 7fb76affd700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb758068320 0x7fb75806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.685+0000 7fb76affd700 1 -- 172.21.15.102:0/644661991 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb76c086410 con 0x7fb7741593c0 2023-03-03T14:07:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.685+0000 7fb77a59b700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb758068320 0x7fb75806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.686+0000 7fb77a59b700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb758068320 0x7fb75806a7d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fb770009b90 tx=0x7fb77001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.689+0000 7fb76affd700 1 -- 172.21.15.102:0/644661991 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb76c052ac0 con 0x7fb7741593c0 2023-03-03T14:07:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.922+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7740cd0c0 con 0x7fb7741593c0 2023-03-03T14:07:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.923+0000 7fb76affd700 1 -- 172.21.15.102:0/644661991 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb76c0560e0 con 0x7fb7741593c0 2023-03-03T14:07:38.927 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:07:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:07:38.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb758068320 msgr2=0x7fb75806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb758068320 0x7fb75806a7d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fb770009b90 tx=0x7fb77001c040 comp rx=0 tx=0).stop 2023-03-03T14:07:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7741593c0 msgr2=0x7fb7740b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7741593c0 0x7fb7740b2cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009db0 tx=0x7fb76c00c3a0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 shutdown_connections 2023-03-03T14:07:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb758068320 0x7fb75806a7d0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb7741593c0 0x7fb7740b2cc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7740cd8e0 0x7fb7741544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 --2- 172.21.15.102:0/644661991 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb774153290 0x7fb7741f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 >> 172.21.15.102:0/644661991 conn(0x7fb774065fb0 msgr2=0x7fb77415bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:07:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 shutdown_connections 2023-03-03T14:07:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:07:38.926+0000 7fb77c7ff700 1 -- 172.21.15.102:0/644661991 wait complete. 2023-03-03T14:07:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:07:38.956 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-03-03T14:08:04.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:04.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:34.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:34.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:01.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:09:02.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.222+0000 7fa75519d700 1 Processor -- start 2023-03-03T14:09:02.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.222+0000 7fa75519d700 1 -- start start 2023-03-03T14:09:02.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.223+0000 7fa75519d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa7500cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:02.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.223+0000 7fa75519d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa7500c9610 0x7fa7500cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:02.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.223+0000 7fa75519d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7500c9ca0 con 0x7fa7500ca440 2023-03-03T14:09:02.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.223+0000 7fa75519d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7500b4920 con 0x7fa7500c9610 2023-03-03T14:09:02.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.223+0000 7fa75519d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7500b4c00 con 0x7fa7501564b0 2023-03-03T14:09:02.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.223+0000 7fa74ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa7500cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:02.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa747fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa7500c9610 0x7fa7500cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:02.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa74ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa7500cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:46530/0 (socket says 172.21.15.102:46530) 2023-03-03T14:09:02.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa74ffff700 1 -- 172.21.15.102:0/2825533376 learned_addr learned my addr 172.21.15.102:0/2825533376 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:09:02.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa74ffff700 1 -- 172.21.15.102:0/2825533376 >> v2:172.21.15.102:3300/0 conn(0x7fa7500c9610 msgr2=0x7fa7500cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:09:02.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa74ffff700 1 --2- 172.21.15.102:0/2825533376 >> v2:172.21.15.102:3300/0 conn(0x7fa7500c9610 0x7fa7500cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa74ffff700 1 -- 172.21.15.102:0/2825533376 >> v1:172.21.15.102:6789/0 conn(0x7fa7500ca440 legacy=0x7fa7500ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa74ffff700 1 -- 172.21.15.102:0/2825533376 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7500b46f0 con 0x7fa7501564b0 2023-03-03T14:09:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa747fff700 1 --2- 172.21.15.102:0/2825533376 >> v2:172.21.15.102:3300/0 conn(0x7fa7500c9610 0x7fa7500cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:09:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.224+0000 7fa74ffff700 1 --2- 172.21.15.102:0/2825533376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa7500cb660 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa7400098f0 tx=0x7fa740009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a6863e34d4b0979 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa74f7fe700 1 -- 172.21.15.102:0/2825533376 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740011830 con 0x7fa7501564b0 2023-03-03T14:09:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa74f7fe700 1 -- 172.21.15.102:0/2825533376 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa740011990 con 0x7fa7501564b0 2023-03-03T14:09:02.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa74f7fe700 1 -- 172.21.15.102:0/2825533376 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740011d20 con 0x7fa7501564b0 2023-03-03T14:09:02.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa75519d700 1 -- 172.21.15.102:0/2825533376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 msgr2=0x7fa7500cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:02.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa75519d700 1 --2- 172.21.15.102:0/2825533376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa7500cb660 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa7400098f0 tx=0x7fa740009c00 comp rx=0 tx=0).stop 2023-03-03T14:09:02.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa75519d700 1 -- 172.21.15.102:0/2825533376 shutdown_connections 2023-03-03T14:09:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa75519d700 1 --2- 172.21.15.102:0/2825533376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa7500cb660 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa75519d700 1 --2- 172.21.15.102:0/2825533376 >> v2:172.21.15.102:3300/0 conn(0x7fa7500c9610 0x7fa7500cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa75519d700 1 -- 172.21.15.102:0/2825533376 >> 172.21.15.102:0/2825533376 conn(0x7fa750065fb0 msgr2=0x7fa7500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.225+0000 7fa75519d700 1 -- 172.21.15.102:0/2825533376 shutdown_connections 2023-03-03T14:09:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.226+0000 7fa75519d700 1 -- 172.21.15.102:0/2825533376 wait complete. 2023-03-03T14:09:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.227+0000 7fa75519d700 1 Processor -- start 2023-03-03T14:09:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.228+0000 7fa75519d700 1 -- start start 2023-03-03T14:09:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa75519d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7500c9610 0x7fa750158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa75519d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7500ca440 0x7fa7501f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa75519d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa75014e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:02.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa75519d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7500c3300 con 0x7fa7500c9610 2023-03-03T14:09:02.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa75519d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa750158a90 con 0x7fa7500ca440 2023-03-03T14:09:02.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa75519d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7501e5eb0 con 0x7fa7501564b0 2023-03-03T14:09:02.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa74ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7500c9610 0x7fa750158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:02.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa74ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7500c9610 0x7fa750158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34998/0 (socket says 172.21.15.102:34998) 2023-03-03T14:09:02.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa74ffff700 1 -- 172.21.15.102:0/884050372 learned_addr learned my addr 172.21.15.102:0/884050372 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:09:02.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa75499c700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa75014e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.229+0000 7fa747fff700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7500ca440 0x7fa7501f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74ffff700 1 -- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 msgr2=0x7fa75014e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74ffff700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa75014e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74ffff700 1 -- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7500ca440 msgr2=0x7fa7501f1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74ffff700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7500ca440 0x7fa7501f1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74ffff700 1 -- 172.21.15.102:0/884050372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7380094a0 con 0x7fa7500c9610 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74ffff700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7500c9610 0x7fa750158f20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa740014040 tx=0x7fa740004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74d7fa700 1 -- 172.21.15.102:0/884050372 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740004400 con 0x7fa7500c9610 2023-03-03T14:09:02.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74d7fa700 1 -- 172.21.15.102:0/884050372 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74001f070 con 0x7fa7500c9610 2023-03-03T14:09:02.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa74d7fa700 1 -- 172.21.15.102:0/884050372 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7400038e0 con 0x7fa7500c9610 2023-03-03T14:09:02.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.230+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa740009550 con 0x7fa7500c9610 2023-03-03T14:09:02.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.231+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7500b2210 con 0x7fa7500c9610 2023-03-03T14:09:02.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.232+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7500b1f30 con 0x7fa7500c9610 2023-03-03T14:09:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.234+0000 7fa74d7fa700 1 -- 172.21.15.102:0/884050372 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa74002c070 con 0x7fa7500c9610 2023-03-03T14:09:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.236+0000 7fa74d7fa700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa730068320 0x7fa73006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.236+0000 7fa74d7fa700 1 -- 172.21.15.102:0/884050372 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa7400871c0 con 0x7fa7500c9610 2023-03-03T14:09:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.236+0000 7fa747fff700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa730068320 0x7fa73006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.237+0000 7fa747fff700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa730068320 0x7fa73006a7d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fa7500c2410 tx=0x7fa738009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:02.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.238+0000 7fa74d7fa700 1 -- 172.21.15.102:0/884050372 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7400538c0 con 0x7fa7500c9610 2023-03-03T14:09:02.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.472+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa750156880 con 0x7fa7500c9610 2023-03-03T14:09:02.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.473+0000 7fa74d7fa700 1 -- 172.21.15.102:0/884050372 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa74002a090 con 0x7fa7500c9610 2023-03-03T14:09:02.474 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:09:02.474 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:09:02.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa730068320 msgr2=0x7fa73006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:02.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa730068320 0x7fa73006a7d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fa7500c2410 tx=0x7fa738009210 comp rx=0 tx=0).stop 2023-03-03T14:09:02.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7500c9610 msgr2=0x7fa750158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:02.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7500c9610 0x7fa750158f20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa740014040 tx=0x7fa740004750 comp rx=0 tx=0).stop 2023-03-03T14:09:02.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 shutdown_connections 2023-03-03T14:09:02.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa730068320 0x7fa73006a7d0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa7501564b0 0x7fa75014e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7500c9610 0x7fa750158f20 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 --2- 172.21.15.102:0/884050372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7500ca440 0x7fa7501f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:02.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.475+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 >> 172.21.15.102:0/884050372 conn(0x7fa750065fb0 msgr2=0x7fa7500cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:02.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.476+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 shutdown_connections 2023-03-03T14:09:02.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:02.476+0000 7fa75519d700 1 -- 172.21.15.102:0/884050372 wait complete. 2023-03-03T14:09:02.479 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:09:02.503 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-03-03T14:09:04.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:04.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:21.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:09:21.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.767+0000 7fc005dff700 1 Processor -- start 2023-03-03T14:09:21.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.768+0000 7fc005dff700 1 -- start start 2023-03-03T14:09:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.769+0000 7fc005dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc0000cd8e0 0x7fc0000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.769+0000 7fc005dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0000cd390 0x7fc0000ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.769+0000 7fc005dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc000153660 con 0x7fc000153290 2023-03-03T14:09:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.769+0000 7fc005dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00006c360 con 0x7fc0000cd8e0 2023-03-03T14:09:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.769+0000 7fc005dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00015cea0 con 0x7fc0000cd390 2023-03-03T14:09:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbfff7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc000153290 0x7fc000158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34528/0 (socket says 172.21.15.102:34528) 2023-03-03T14:09:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffeffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc0000cd8e0 0x7fc0000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbfff7fe700 1 -- 172.21.15.102:0/2098804153 learned_addr learned my addr 172.21.15.102:0/2098804153 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:09:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffffff700 1 --2- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0000cd390 0x7fc0000ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffeffd700 1 -- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0000cd390 msgr2=0x7fc0000ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:09:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffeffd700 1 --2- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0000cd390 0x7fc0000ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffeffd700 1 -- 172.21.15.102:0/2098804153 >> v1:172.21.15.102:6789/0 conn(0x7fc000153290 legacy=0x7fc000158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffeffd700 1 -- 172.21.15.102:0/2098804153 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00006c7f0 con 0x7fc0000cd8e0 2023-03-03T14:09:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffe7fc700 1 -- 172.21.15.102:0/2098804153 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992362660 0 0) 0x7fc000153660 con 0x7fc000153290 2023-03-03T14:09:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.770+0000 7fbffeffd700 1 --2- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd8e0 0x7fc0000cde20 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009840 tx=0x7fbff4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e70dbfe2532ddc77 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fbffe7fc700 1 -- 172.21.15.102:0/2098804153 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4011830 con 0x7fc0000cd8e0 2023-03-03T14:09:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fbffe7fc700 1 -- 172.21.15.102:0/2098804153 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff4011990 con 0x7fc0000cd8e0 2023-03-03T14:09:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fbffe7fc700 1 -- 172.21.15.102:0/2098804153 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4011d20 con 0x7fc0000cd8e0 2023-03-03T14:09:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fc005dff700 1 -- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd8e0 msgr2=0x7fc0000cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fc005dff700 1 --2- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd8e0 0x7fc0000cde20 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009840 tx=0x7fbff4009b50 comp rx=0 tx=0).stop 2023-03-03T14:09:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fc005dff700 1 -- 172.21.15.102:0/2098804153 shutdown_connections 2023-03-03T14:09:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fc005dff700 1 --2- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc0000cd390 0x7fc0000ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fc005dff700 1 --2- 172.21.15.102:0/2098804153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd8e0 0x7fc0000cde20 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:21.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.771+0000 7fc005dff700 1 -- 172.21.15.102:0/2098804153 >> 172.21.15.102:0/2098804153 conn(0x7fc000065fb0 msgr2=0x7fc0000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:21.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.772+0000 7fc005dff700 1 -- 172.21.15.102:0/2098804153 shutdown_connections 2023-03-03T14:09:21.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.772+0000 7fc005dff700 1 -- 172.21.15.102:0/2098804153 wait complete. 2023-03-03T14:09:21.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.773+0000 7fc005dff700 1 Processor -- start 2023-03-03T14:09:21.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.774+0000 7fc005dff700 1 -- start start 2023-03-03T14:09:21.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fc005dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd390 0x7fc0001f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:21.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fc005dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0000cd8e0 0x7fc000158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:21.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fc005dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc000153290 0x7fc00014e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:21.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fc005dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000c6880 con 0x7fc0000cd8e0 2023-03-03T14:09:21.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fc005dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000c62c0 con 0x7fc0000cd390 2023-03-03T14:09:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fc005dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000c65a0 con 0x7fc000153290 2023-03-03T14:09:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0000cd8e0 0x7fc000158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0000cd8e0 0x7fc000158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57244/0 (socket says 172.21.15.102:57244) 2023-03-03T14:09:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 -- 172.21.15.102:0/3159998944 learned_addr learned my addr 172.21.15.102:0/3159998944 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:09:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbfff7fe700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd390 0x7fc0001f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffffff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc000153290 0x7fc00014e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:21.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 -- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc000153290 msgr2=0x7fc00014e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:21.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc000153290 0x7fc00014e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:21.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 -- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd390 msgr2=0x7fc0001f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:21.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd390 0x7fc0001f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:21.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.775+0000 7fbffeffd700 1 -- 172.21.15.102:0/3159998944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff40094a0 con 0x7fc0000cd8e0 2023-03-03T14:09:21.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.776+0000 7fbfff7fe700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd390 0x7fc0001f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:09:21.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.776+0000 7fbffeffd700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0000cd8e0 0x7fc000158f90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009b90 tx=0x7fbff4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:21.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.776+0000 7fbfe7fff700 1 -- 172.21.15.102:0/3159998944 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff40036a0 con 0x7fc0000cd8e0 2023-03-03T14:09:21.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.776+0000 7fbfe7fff700 1 -- 172.21.15.102:0/3159998944 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff401e070 con 0x7fc0000cd8e0 2023-03-03T14:09:21.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.776+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0000ccde0 con 0x7fc0000cd8e0 2023-03-03T14:09:21.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.776+0000 7fbfe7fff700 1 -- 172.21.15.102:0/3159998944 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4003a60 con 0x7fc0000cd8e0 2023-03-03T14:09:21.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.776+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00015d160 con 0x7fc0000cd8e0 2023-03-03T14:09:21.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.778+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0000cd0c0 con 0x7fc0000cd8e0 2023-03-03T14:09:21.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.779+0000 7fbfe7fff700 1 -- 172.21.15.102:0/3159998944 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbff400f050 con 0x7fc0000cd8e0 2023-03-03T14:09:21.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.780+0000 7fbfe7fff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfec0682f0 0x7fbfec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:21.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.780+0000 7fbfe7fff700 1 -- 172.21.15.102:0/3159998944 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbff4086c70 con 0x7fc0000cd8e0 2023-03-03T14:09:21.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.780+0000 7fbfff7fe700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfec0682f0 0x7fbfec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:21.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.781+0000 7fbfff7fe700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfec0682f0 0x7fbfec06a7a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbff0007fd0 tx=0x7fbff0007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:21.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:21.784+0000 7fbfe7fff700 1 -- 172.21.15.102:0/3159998944 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbff4053380 con 0x7fc0000cd8e0 2023-03-03T14:09:22.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.018+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc00004e6e0 con 0x7fc0000cd8e0 2023-03-03T14:09:22.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.019+0000 7fbfe7fff700 1 -- 172.21.15.102:0/3159998944 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbff4020090 con 0x7fc0000cd8e0 2023-03-03T14:09:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:09:22.022 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:09:22.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.021+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfec0682f0 msgr2=0x7fbfec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.021+0000 7fc005dff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfec0682f0 0x7fbfec06a7a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbff0007fd0 tx=0x7fbff0007de0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.021+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0000cd8e0 msgr2=0x7fc000158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.021+0000 7fc005dff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0000cd8e0 0x7fc000158f90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009b90 tx=0x7fbff4004750 comp rx=0 tx=0).stop 2023-03-03T14:09:22.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 shutdown_connections 2023-03-03T14:09:22.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfec0682f0 0x7fbfec06a7a0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc000153290 0x7fc00014e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0000cd390 0x7fc0001f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 --2- 172.21.15.102:0/3159998944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0000cd8e0 0x7fc000158f90 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 >> 172.21.15.102:0/3159998944 conn(0x7fc000065fb0 msgr2=0x7fc0001575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:22.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 shutdown_connections 2023-03-03T14:09:22.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.022+0000 7fc005dff700 1 -- 172.21.15.102:0/3159998944 wait complete. 2023-03-03T14:09:22.026 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:09:22.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:09:22.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.303+0000 7f0cfa99d700 1 Processor -- start 2023-03-03T14:09:22.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.303+0000 7f0cfa99d700 1 -- start start 2023-03-03T14:09:22.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cfa99d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0cf40b3970 0x7f0cf40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:22.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cfa99d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40c41b0 0x7f0cf40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:22.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cfa99d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf40c3d20 con 0x7f0cf40ced00 2023-03-03T14:09:22.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cfa99d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf40c4930 con 0x7f0cf40b3970 2023-03-03T14:09:22.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cfa99d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40cf610 con 0x7f0cf40c41b0 2023-03-03T14:09:22.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cf999b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0cf40ced00 0x7f0cf4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34532/0 (socket says 172.21.15.102:34532) 2023-03-03T14:09:22.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cf999b700 1 -- 172.21.15.102:0/3228426414 learned_addr learned my addr 172.21.15.102:0/3228426414 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:09:22.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cfa19c700 1 --2- 172.21.15.102:0/3228426414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40c41b0 0x7f0cf40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:22.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.305+0000 7f0cf919a700 1 --2- 172.21.15.102:0/3228426414 >> v2:172.21.15.172:3300/0 conn(0x7f0cf40b3970 0x7f0cf40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:22.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cf8999700 1 -- 172.21.15.102:0/3228426414 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069731642 0 0) 0x7f0cf40c3d20 con 0x7f0cf40ced00 2023-03-03T14:09:22.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cf8999700 1 -- 172.21.15.102:0/3228426414 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00035b0 con 0x7f0cf40ced00 2023-03-03T14:09:22.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cfa19c700 1 -- 172.21.15.102:0/3228426414 >> v2:172.21.15.172:3300/0 conn(0x7f0cf40b3970 msgr2=0x7f0cf40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:09:22.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cfa19c700 1 --2- 172.21.15.102:0/3228426414 >> v2:172.21.15.172:3300/0 conn(0x7f0cf40b3970 0x7f0cf40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cfa19c700 1 -- 172.21.15.102:0/3228426414 >> v1:172.21.15.102:6789/0 conn(0x7f0cf40ced00 legacy=0x7f0cf4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cfa19c700 1 -- 172.21.15.102:0/3228426414 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf415cba0 con 0x7f0cf40c41b0 2023-03-03T14:09:22.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cfa19c700 1 --2- 172.21.15.102:0/3228426414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40c41b0 0x7f0cf40cf0d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009d90 tx=0x7f0cec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=542cf33cd577e393 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:22.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.306+0000 7f0cf8999700 1 -- 172.21.15.102:0/3228426414 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec00e070 con 0x7f0cf40c41b0 2023-03-03T14:09:22.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cf8999700 1 -- 172.21.15.102:0/3228426414 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec00bb20 con 0x7f0cf40c41b0 2023-03-03T14:09:22.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cf8999700 1 -- 172.21.15.102:0/3228426414 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec012430 con 0x7f0cf40c41b0 2023-03-03T14:09:22.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3228426414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40c41b0 msgr2=0x7f0cf40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3228426414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40c41b0 0x7f0cf40cf0d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009d90 tx=0x7f0cec00b330 comp rx=0 tx=0).stop 2023-03-03T14:09:22.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3228426414 shutdown_connections 2023-03-03T14:09:22.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3228426414 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40c41b0 0x7f0cf40cf0d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3228426414 >> v2:172.21.15.172:3300/0 conn(0x7f0cf40b3970 0x7f0cf40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3228426414 >> 172.21.15.102:0/3228426414 conn(0x7f0cf408b1c0 msgr2=0x7f0cf4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:22.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3228426414 shutdown_connections 2023-03-03T14:09:22.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.307+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3228426414 wait complete. 2023-03-03T14:09:22.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.310+0000 7f0cfa99d700 1 Processor -- start 2023-03-03T14:09:22.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.310+0000 7f0cfa99d700 1 -- start start 2023-03-03T14:09:22.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cfa99d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf40b3970 0x7f0cf41e28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:22.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cfa99d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf40c41b0 0x7f0cf41e2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cfa99d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40ced00 0x7f0cf41f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cfa99d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf414c210 con 0x7f0cf40c41b0 2023-03-03T14:09:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cfa99d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf41e1fa0 con 0x7f0cf40b3970 2023-03-03T14:09:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cfa99d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf41e4d80 con 0x7f0cf40ced00 2023-03-03T14:09:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cf999b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf40b3970 0x7f0cf41e28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cf919a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf40c41b0 0x7f0cf41e2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cfa19c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40ced00 0x7f0cf41f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cf919a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf40c41b0 0x7f0cf41e2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57260/0 (socket says 172.21.15.102:57260) 2023-03-03T14:09:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.312+0000 7f0cf919a700 1 -- 172.21.15.102:0/3674354061 learned_addr learned my addr 172.21.15.102:0/3674354061 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:09:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cfa19c700 1 -- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf40b3970 msgr2=0x7f0cf41e28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cfa19c700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf40b3970 0x7f0cf41e28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cfa19c700 1 -- 172.21.15.102:0/3674354061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf40c41b0 msgr2=0x7f0cf41e2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cfa19c700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf40c41b0 0x7f0cf41e2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cfa19c700 1 -- 172.21.15.102:0/3674354061 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec009a70 con 0x7f0cf40ced00 2023-03-03T14:09:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cf919a700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf40c41b0 0x7f0cf41e2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:09:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cfa19c700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40ced00 0x7f0cf41f1c80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009d90 tx=0x7f0cec004b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cea7fc700 1 -- 172.21.15.102:0/3674354061 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec012430 con 0x7f0cf40ced00 2023-03-03T14:09:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cea7fc700 1 -- 172.21.15.102:0/3674354061 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec020070 con 0x7f0cf40ced00 2023-03-03T14:09:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cea7fc700 1 -- 172.21.15.102:0/3674354061 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec004280 con 0x7f0cf40ced00 2023-03-03T14:09:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.313+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf406bf20 con 0x7f0cf40ced00 2023-03-03T14:09:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.314+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf415ce80 con 0x7f0cf40ced00 2023-03-03T14:09:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.315+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf406c200 con 0x7f0cf40ced00 2023-03-03T14:09:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.316+0000 7f0cea7fc700 1 -- 172.21.15.102:0/3674354061 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0cec02e070 con 0x7f0cf40ced00 2023-03-03T14:09:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.317+0000 7f0cea7fc700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cd80682f0 0x7f0cd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.317+0000 7f0cea7fc700 1 -- 172.21.15.102:0/3674354061 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0cec022070 con 0x7f0cf40ced00 2023-03-03T14:09:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.317+0000 7f0cf999b700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cd80682f0 0x7f0cd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.318+0000 7f0cf999b700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cd80682f0 0x7f0cd806a7a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0007fd0 tx=0x7f0cf0007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.321+0000 7f0cea7fc700 1 -- 172.21.15.102:0/3674354061 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0cec054010 con 0x7f0cf40ced00 2023-03-03T14:09:22.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.555+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cf404e6e0 con 0x7f0cf40ced00 2023-03-03T14:09:22.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.557+0000 7f0cea7fc700 1 -- 172.21.15.102:0/3674354061 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0cec02c090 con 0x7f0cf40ced00 2023-03-03T14:09:22.560 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:09:22.560 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:09:22.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cd80682f0 msgr2=0x7f0cd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cd80682f0 0x7f0cd806a7a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0007fd0 tx=0x7f0cf0007dc0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40ced00 msgr2=0x7f0cf41f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:22.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40ced00 0x7f0cf41f1c80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009d90 tx=0x7f0cec004b20 comp rx=0 tx=0).stop 2023-03-03T14:09:22.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 shutdown_connections 2023-03-03T14:09:22.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0cd80682f0 0x7f0cd806a7a0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0cf40ced00 0x7f0cf41f1c80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf40b3970 0x7f0cf41e28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 --2- 172.21.15.102:0/3674354061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf40c41b0 0x7f0cf41e2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 >> 172.21.15.102:0/3674354061 conn(0x7f0cf408b1c0 msgr2=0x7f0cf40ca960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 shutdown_connections 2023-03-03T14:09:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:09:22.559+0000 7f0cfa99d700 1 -- 172.21.15.102:0/3674354061 wait complete. 2023-03-03T14:09:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:09:22.589 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2023-03-03T14:09:34.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:34.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:00.002 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T14:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:10:04.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:04.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:34.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:34.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:05.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:05.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:19.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:11:19.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.842+0000 7f7a11b9d700 1 Processor -- start 2023-03-03T14:11:19.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.843+0000 7f7a11b9d700 1 -- start start 2023-03-03T14:11:19.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a11b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7a0c1564b0 0x7f7a0c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:19.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a11b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0cab50 0x7f7a0c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a11b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0c0c9ca0 con 0x7f7a0c0c9610 2023-03-03T14:11:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a11b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0c0b4920 con 0x7f7a0c1564b0 2023-03-03T14:11:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a11b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c0b4c00 con 0x7f7a0c0cab50 2023-03-03T14:11:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a1139c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0cab50 0x7f7a0c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a10b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7a0c1564b0 0x7f7a0c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a0bfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7a0c0c9610 0x7f7a0c0ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:53768/0 (socket says 172.21.15.102:53768) 2023-03-03T14:11:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.844+0000 7f7a0bfff700 1 -- 172.21.15.102:0/3572494916 learned_addr learned my addr 172.21.15.102:0/3572494916 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:11:19.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a10b9b700 1 -- 172.21.15.102:0/3572494916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0cab50 msgr2=0x7f7a0c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:11:19.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a10b9b700 1 --2- 172.21.15.102:0/3572494916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0cab50 0x7f7a0c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:19.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a10b9b700 1 -- 172.21.15.102:0/3572494916 >> v1:172.21.15.172:6789/0 conn(0x7f7a0c0c9610 legacy=0x7f7a0c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:19.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a10b9b700 1 -- 172.21.15.102:0/3572494916 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c0b46f0 con 0x7f7a0c1564b0 2023-03-03T14:11:19.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a10b9b700 1 --2- 172.21.15.102:0/3572494916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c1564b0 0x7f7a0c0cb660 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009840 tx=0x7f7a00009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8fe0b391930c822 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:19.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a0b7fe700 1 -- 172.21.15.102:0/3572494916 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00011830 con 0x7f7a0c1564b0 2023-03-03T14:11:19.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a0b7fe700 1 -- 172.21.15.102:0/3572494916 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a00011990 con 0x7f7a0c1564b0 2023-03-03T14:11:19.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.845+0000 7f7a0b7fe700 1 -- 172.21.15.102:0/3572494916 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00011d20 con 0x7f7a0c1564b0 2023-03-03T14:11:19.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 -- 172.21.15.102:0/3572494916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c1564b0 msgr2=0x7f7a0c0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:19.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/3572494916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c1564b0 0x7f7a0c0cb660 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009840 tx=0x7f7a00009b50 comp rx=0 tx=0).stop 2023-03-03T14:11:19.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 -- 172.21.15.102:0/3572494916 shutdown_connections 2023-03-03T14:11:19.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/3572494916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c0cab50 0x7f7a0c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:19.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/3572494916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c1564b0 0x7f7a0c0cb660 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:19.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 -- 172.21.15.102:0/3572494916 >> 172.21.15.102:0/3572494916 conn(0x7f7a0c065fb0 msgr2=0x7f7a0c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 -- 172.21.15.102:0/3572494916 shutdown_connections 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.846+0000 7f7a11b9d700 1 -- 172.21.15.102:0/3572494916 wait complete. 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.849+0000 7f7a11b9d700 1 Processor -- start 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.849+0000 7f7a11b9d700 1 -- start start 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a11b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0c9610 0x7f7a0c1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a11b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a0c0cab50 0x7f7a0c159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a11b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c1564b0 0x7f7a0c14e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a11b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c15b2e0 con 0x7f7a0c0cab50 2023-03-03T14:11:19.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a11b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c14e9d0 con 0x7f7a0c0c9610 2023-03-03T14:11:19.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a11b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c1f1f30 con 0x7f7a0c1564b0 2023-03-03T14:11:19.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0c9610 0x7f7a0c1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:19.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0c9610 0x7f7a0c1f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43960/0 (socket says 172.21.15.102:43960) 2023-03-03T14:11:19.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 -- 172.21.15.102:0/2050015834 learned_addr learned my addr 172.21.15.102:0/2050015834 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:11:19.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a1139c700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c1564b0 0x7f7a0c14e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:19.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a0bfff700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a0c0cab50 0x7f7a0c159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:19.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 -- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c1564b0 msgr2=0x7f7a0c14e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:19.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c1564b0 0x7f7a0c14e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:19.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 -- 172.21.15.102:0/2050015834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a0c0cab50 msgr2=0x7f7a0c159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:19.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a0c0cab50 0x7f7a0c159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.850+0000 7f7a10b9b700 1 -- 172.21.15.102:0/2050015834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a000094a0 con 0x7f7a0c0c9610 2023-03-03T14:11:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.851+0000 7f7a0bfff700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a0c0cab50 0x7f7a0c159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:11:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.851+0000 7f7a10b9b700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0c9610 0x7f7a0c1f23c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7a000090b0 tx=0x7f7a00004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.851+0000 7f7a097fa700 1 -- 172.21.15.102:0/2050015834 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a000036a0 con 0x7f7a0c0c9610 2023-03-03T14:11:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.851+0000 7f7a097fa700 1 -- 172.21.15.102:0/2050015834 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0001f070 con 0x7f7a0c0c9610 2023-03-03T14:11:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.851+0000 7f7a097fa700 1 -- 172.21.15.102:0/2050015834 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00003a60 con 0x7f7a0c0c9610 2023-03-03T14:11:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.851+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0c0b2210 con 0x7f7a0c0c9610 2023-03-03T14:11:19.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.851+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0c0b1f30 con 0x7f7a0c0c9610 2023-03-03T14:11:19.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.852+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0c156880 con 0x7f7a0c0c9610 2023-03-03T14:11:19.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.854+0000 7f7a097fa700 1 -- 172.21.15.102:0/2050015834 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7a0000f050 con 0x7f7a0c0c9610 2023-03-03T14:11:19.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.855+0000 7f7a097fa700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79f00682f0 0x7f79f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:19.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.855+0000 7f7a097fa700 1 -- 172.21.15.102:0/2050015834 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7a00021070 con 0x7f7a0c0c9610 2023-03-03T14:11:19.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.855+0000 7f7a0bfff700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79f00682f0 0x7f79f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:19.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.856+0000 7f7a0bfff700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79f00682f0 0x7f79f006a7a0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f79f8007fd0 tx=0x7f79f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:19.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:19.859+0000 7f7a097fa700 1 -- 172.21.15.102:0/2050015834 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7a00053d60 con 0x7f7a0c0c9610 2023-03-03T14:11:20.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.092+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a0c0c24d0 con 0x7f7a0c0c9610 2023-03-03T14:11:20.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.093+0000 7f7a097fa700 1 -- 172.21.15.102:0/2050015834 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7a00057380 con 0x7f7a0c0c9610 2023-03-03T14:11:20.095 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:11:20.095 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:11:20.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.095+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79f00682f0 msgr2=0x7f79f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:20.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.095+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79f00682f0 0x7f79f006a7a0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f79f8007fd0 tx=0x7f79f8009040 comp rx=0 tx=0).stop 2023-03-03T14:11:20.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.095+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0c9610 msgr2=0x7f7a0c1f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:20.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.095+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0c9610 0x7f7a0c1f23c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7a000090b0 tx=0x7f7a00004750 comp rx=0 tx=0).stop 2023-03-03T14:11:20.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.095+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 shutdown_connections 2023-03-03T14:11:20.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.095+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f79f00682f0 0x7f79f006a7a0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:20.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.095+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7a0c1564b0 0x7f7a0c14e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:20.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.096+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a0c0c9610 0x7f7a0c1f23c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:20.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.096+0000 7f7a11b9d700 1 --2- 172.21.15.102:0/2050015834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a0c0cab50 0x7f7a0c159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:20.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.096+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 >> 172.21.15.102:0/2050015834 conn(0x7f7a0c065fb0 msgr2=0x7f7a0c157520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:11:20.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.096+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 shutdown_connections 2023-03-03T14:11:20.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:11:20.096+0000 7f7a11b9d700 1 -- 172.21.15.102:0/2050015834 wait complete. 2023-03-03T14:11:20.105 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:11:20.126 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-03-03T14:11:35.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:35.246 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:05.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:05.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:35.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:35.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:41.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:12:41.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.386+0000 7fdbbb7ff700 1 Processor -- start 2023-03-03T14:12:41.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.387+0000 7fdbbb7ff700 1 -- start start 2023-03-03T14:12:41.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.388+0000 7fdbbb7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb4153290 0x7fdbb4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:41.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.388+0000 7fdbbb7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbb41593c0 0x7fdbb40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:41.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.388+0000 7fdbbb7ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb4153660 con 0x7fdbb40cd8e0 2023-03-03T14:12:41.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.388+0000 7fdbbb7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb406c360 con 0x7fdbb41593c0 2023-03-03T14:12:41.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.388+0000 7fdbbb7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb415cea0 con 0x7fdbb4153290 2023-03-03T14:12:41.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb8d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbb41593c0 0x7fdbb40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:41.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb8d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbb41593c0 0x7fdbb40cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35722/0 (socket says 172.21.15.102:35722) 2023-03-03T14:12:41.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb9d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdbb40cd8e0 0x7fdbb40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58502/0 (socket says 172.21.15.102:58502) 2023-03-03T14:12:41.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3667209407 learned_addr learned my addr 172.21.15.102:0/3667209407 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:12:41.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb959b700 1 --2- 172.21.15.102:0/3667209407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb4153290 0x7fdbb4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:41.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3667209407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb4153290 msgr2=0x7fdbb4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:12:41.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb8d9a700 1 --2- 172.21.15.102:0/3667209407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb4153290 0x7fdbb4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3667209407 >> v1:172.21.15.102:6789/0 conn(0x7fdbb40cd8e0 legacy=0x7fdbb40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.389+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3667209407 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb406c7f0 con 0x7fdbb41593c0 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.390+0000 7fdbb8d9a700 1 --2- 172.21.15.102:0/3667209407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb41593c0 0x7fdbb40cde20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fdba40098f0 tx=0x7fdba4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478d589a0335c7c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.390+0000 7fdbabfff700 1 -- 172.21.15.102:0/3667209407 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4011830 con 0x7fdbb41593c0 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.390+0000 7fdbabfff700 1 -- 172.21.15.102:0/3667209407 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba4011990 con 0x7fdbb41593c0 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.390+0000 7fdbabfff700 1 -- 172.21.15.102:0/3667209407 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4011d20 con 0x7fdbb41593c0 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.390+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3667209407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb41593c0 msgr2=0x7fdbb40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.390+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3667209407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb41593c0 0x7fdbb40cde20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fdba40098f0 tx=0x7fdba4009c00 comp rx=0 tx=0).stop 2023-03-03T14:12:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.391+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3667209407 shutdown_connections 2023-03-03T14:12:41.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.391+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3667209407 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb4153290 0x7fdbb4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.391+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3667209407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb41593c0 0x7fdbb40cde20 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.391+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3667209407 >> 172.21.15.102:0/3667209407 conn(0x7fdbb4065fb0 msgr2=0x7fdbb40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:12:41.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.391+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3667209407 shutdown_connections 2023-03-03T14:12:41.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.391+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3667209407 wait complete. 2023-03-03T14:12:41.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.393+0000 7fdbbb7ff700 1 Processor -- start 2023-03-03T14:12:41.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.393+0000 7fdbbb7ff700 1 -- start start 2023-03-03T14:12:41.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbbb7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbb40cd8e0 0x7fdbb40b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:41.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbbb7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb4153290 0x7fdbb40b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:41.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbbb7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb41593c0 0x7fdbb40b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbbb7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb41ee500 con 0x7fdbb4153290 2023-03-03T14:12:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbbb7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb40b4750 con 0x7fdbb40cd8e0 2023-03-03T14:12:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbbb7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb40b4a10 con 0x7fdbb41593c0 2023-03-03T14:12:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbb8d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb4153290 0x7fdbb40b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbb959b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbb40cd8e0 0x7fdbb40b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbb8d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb4153290 0x7fdbb40b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35724/0 (socket says 172.21.15.102:35724) 2023-03-03T14:12:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.394+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3147411893 learned_addr learned my addr 172.21.15.102:0/3147411893 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:12:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.395+0000 7fdbb9d9c700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb41593c0 0x7fdbb40b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.395+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb41593c0 msgr2=0x7fdbb40b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.395+0000 7fdbb8d9a700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb41593c0 0x7fdbb40b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.395+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbb40cd8e0 msgr2=0x7fdbb40b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.395+0000 7fdbb8d9a700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbb40cd8e0 0x7fdbb40b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.395+0000 7fdbb8d9a700 1 -- 172.21.15.102:0/3147411893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4009550 con 0x7fdbb4153290 2023-03-03T14:12:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.395+0000 7fdbb8d9a700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb4153290 0x7fdbb40b1c70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fdba4000c00 tx=0x7fdba4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.396+0000 7fdba9ffb700 1 -- 172.21.15.102:0/3147411893 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4004480 con 0x7fdbb4153290 2023-03-03T14:12:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.396+0000 7fdba9ffb700 1 -- 172.21.15.102:0/3147411893 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba401f070 con 0x7fdbb4153290 2023-03-03T14:12:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.396+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb40ccde0 con 0x7fdbb4153290 2023-03-03T14:12:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.396+0000 7fdba9ffb700 1 -- 172.21.15.102:0/3147411893 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba40038c0 con 0x7fdbb4153290 2023-03-03T14:12:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.396+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb415d160 con 0x7fdbb4153290 2023-03-03T14:12:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.397+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb40cd0c0 con 0x7fdbb4153290 2023-03-03T14:12:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.398+0000 7fdba9ffb700 1 -- 172.21.15.102:0/3147411893 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdba402a020 con 0x7fdbb4153290 2023-03-03T14:12:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.399+0000 7fdba9ffb700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb98068320 0x7fdb9806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.399+0000 7fdba9ffb700 1 -- 172.21.15.102:0/3147411893 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdba40873a0 con 0x7fdbb4153290 2023-03-03T14:12:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.400+0000 7fdbb959b700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb98068320 0x7fdb9806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.400+0000 7fdbb959b700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb98068320 0x7fdb9806a7d0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009be0 tx=0x7fdbb0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.404+0000 7fdba9ffb700 1 -- 172.21.15.102:0/3147411893 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdba4053ab0 con 0x7fdbb4153290 2023-03-03T14:12:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.638+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbb4155d30 con 0x7fdbb4153290 2023-03-03T14:12:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.639+0000 7fdba9ffb700 1 -- 172.21.15.102:0/3147411893 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdbb4155d30 con 0x7fdbb4153290 2023-03-03T14:12:41.642 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:12:41.642 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:12:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb98068320 msgr2=0x7fdb9806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb98068320 0x7fdb9806a7d0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009be0 tx=0x7fdbb0009210 comp rx=0 tx=0).stop 2023-03-03T14:12:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb4153290 msgr2=0x7fdbb40b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb4153290 0x7fdbb40b1c70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fdba4000c00 tx=0x7fdba4004750 comp rx=0 tx=0).stop 2023-03-03T14:12:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 shutdown_connections 2023-03-03T14:12:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdb98068320 0x7fdb9806a7d0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbb41593c0 0x7fdbb40b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbb40cd8e0 0x7fdbb40b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 --2- 172.21.15.102:0/3147411893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb4153290 0x7fdbb40b1c70 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 >> 172.21.15.102:0/3147411893 conn(0x7fdbb4065fb0 msgr2=0x7fdbb415c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:12:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 shutdown_connections 2023-03-03T14:12:41.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:12:41.642+0000 7fdbbb7ff700 1 -- 172.21.15.102:0/3147411893 wait complete. 2023-03-03T14:12:41.649 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:12:41.675 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T14:13:05.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:05.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:35.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:35.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:51.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:13:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.927+0000 7f5e23fff700 1 Processor -- start 2023-03-03T14:13:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.927+0000 7f5e23fff700 1 -- start start 2023-03-03T14:13:51.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e23fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5e240ced00 0x7f5e240c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e23fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e240b3970 0x7f5e240c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e23fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e240c3d20 con 0x7f5e240cf0d0 2023-03-03T14:13:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e23fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e240c4930 con 0x7f5e240b3970 2023-03-03T14:13:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e23fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e240cf570 con 0x7f5e240ced00 2023-03-03T14:13:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e227fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e240b3970 0x7f5e240c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e227fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e240b3970 0x7f5e240c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59900/0 (socket says 172.21.15.102:59900) 2023-03-03T14:13:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e227fc700 1 -- 172.21.15.102:0/1969553950 learned_addr learned my addr 172.21.15.102:0/1969553950 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:13:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e237fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5e240cf0d0 0x7f5e24158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37520/0 (socket says 172.21.15.102:37520) 2023-03-03T14:13:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.929+0000 7f5e22ffd700 1 --2- 172.21.15.102:0/1969553950 >> v2:172.21.15.172:3300/0 conn(0x7f5e240ced00 0x7f5e240c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e21ffb700 1 -- 172.21.15.102:0/1969553950 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187570772 0 0) 0x7f5e240c3d20 con 0x7f5e240cf0d0 2023-03-03T14:13:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e22ffd700 1 -- 172.21.15.102:0/1969553950 >> v2:172.21.15.102:3300/0 conn(0x7f5e240b3970 msgr2=0x7f5e240c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:13:51.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e22ffd700 1 --2- 172.21.15.102:0/1969553950 >> v2:172.21.15.102:3300/0 conn(0x7f5e240b3970 0x7f5e240c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e22ffd700 1 -- 172.21.15.102:0/1969553950 >> v1:172.21.15.102:6789/0 conn(0x7f5e240cf0d0 legacy=0x7f5e24158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e22ffd700 1 -- 172.21.15.102:0/1969553950 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2415cba0 con 0x7f5e240ced00 2023-03-03T14:13:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e22ffd700 1 --2- 172.21.15.102:0/1969553950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e240c5a90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5e240b1b40 tx=0x7f5e14009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18088056cc21f874 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e21ffb700 1 -- 172.21.15.102:0/1969553950 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14011830 con 0x7f5e240ced00 2023-03-03T14:13:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e21ffb700 1 -- 172.21.15.102:0/1969553950 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e14011990 con 0x7f5e240ced00 2023-03-03T14:13:51.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.930+0000 7f5e21ffb700 1 -- 172.21.15.102:0/1969553950 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14011d20 con 0x7f5e240ced00 2023-03-03T14:13:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 -- 172.21.15.102:0/1969553950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 msgr2=0x7f5e240c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 --2- 172.21.15.102:0/1969553950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e240c5a90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5e240b1b40 tx=0x7f5e14009b20 comp rx=0 tx=0).stop 2023-03-03T14:13:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 -- 172.21.15.102:0/1969553950 shutdown_connections 2023-03-03T14:13:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 --2- 172.21.15.102:0/1969553950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e240c5a90 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 --2- 172.21.15.102:0/1969553950 >> v2:172.21.15.102:3300/0 conn(0x7f5e240b3970 0x7f5e240c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 -- 172.21.15.102:0/1969553950 >> 172.21.15.102:0/1969553950 conn(0x7f5e2408b1c0 msgr2=0x7f5e24071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:13:51.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 -- 172.21.15.102:0/1969553950 shutdown_connections 2023-03-03T14:13:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.931+0000 7f5e23fff700 1 -- 172.21.15.102:0/1969553950 wait complete. 2023-03-03T14:13:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.933+0000 7f5e23fff700 1 Processor -- start 2023-03-03T14:13:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.933+0000 7f5e23fff700 1 -- start start 2023-03-03T14:13:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.934+0000 7f5e23fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e240b3970 0x7f5e241e4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.934+0000 7f5e23fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e241eed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.934+0000 7f5e23fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e240cf0d0 0x7f5e2415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.934+0000 7f5e23fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2415b0b0 con 0x7f5e240cf0d0 2023-03-03T14:13:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.934+0000 7f5e23fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241ee5a0 con 0x7f5e240ced00 2023-03-03T14:13:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.934+0000 7f5e23fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241e52d0 con 0x7f5e240b3970 2023-03-03T14:13:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e240cf0d0 0x7f5e2415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e240cf0d0 0x7f5e2415bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59914/0 (socket says 172.21.15.102:59914) 2023-03-03T14:13:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 -- 172.21.15.102:0/3609948167 learned_addr learned my addr 172.21.15.102:0/3609948167 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:13:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e227fc700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e241eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e22ffd700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e240b3970 0x7f5e241e4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 -- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e240b3970 msgr2=0x7f5e241e4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e240b3970 0x7f5e241e4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:51.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 -- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 msgr2=0x7f5e241eed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e241eed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 -- 172.21.15.102:0/3609948167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140094a0 con 0x7f5e240cf0d0 2023-03-03T14:13:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e227fc700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e241eed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:13:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.935+0000 7f5e237fe700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e240cf0d0 0x7f5e2415bc50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00cbf0 tx=0x7f5e1c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.936+0000 7f5e0b7fe700 1 -- 172.21.15.102:0/3609948167 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00aaa0 con 0x7f5e240cf0d0 2023-03-03T14:13:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.936+0000 7f5e0b7fe700 1 -- 172.21.15.102:0/3609948167 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c00ac20 con 0x7f5e240cf0d0 2023-03-03T14:13:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.936+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2406bf20 con 0x7f5e240cf0d0 2023-03-03T14:13:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.936+0000 7f5e0b7fe700 1 -- 172.21.15.102:0/3609948167 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0070d0 con 0x7f5e240cf0d0 2023-03-03T14:13:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.936+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2415ce80 con 0x7f5e240cf0d0 2023-03-03T14:13:51.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.937+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2406c200 con 0x7f5e240cf0d0 2023-03-03T14:13:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.938+0000 7f5e0b7fe700 1 -- 172.21.15.102:0/3609948167 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5e1c01a030 con 0x7f5e240cf0d0 2023-03-03T14:13:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.939+0000 7f5e0b7fe700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e100682f0 0x7f5e1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.940+0000 7f5e0b7fe700 1 -- 172.21.15.102:0/3609948167 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5e1c085960 con 0x7f5e240cf0d0 2023-03-03T14:13:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.940+0000 7f5e22ffd700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e100682f0 0x7f5e1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:51.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.941+0000 7f5e22ffd700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e100682f0 0x7f5e1006a7a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f5e140090b0 tx=0x7f5e1401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:51.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:51.944+0000 7f5e0b7fe700 1 -- 172.21.15.102:0/3609948167 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e1c052070 con 0x7f5e240cf0d0 2023-03-03T14:13:52.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.180+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e240b3d40 con 0x7f5e240cf0d0 2023-03-03T14:13:52.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.181+0000 7f5e0b7fe700 1 -- 172.21.15.102:0/3609948167 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5e1c018090 con 0x7f5e240cf0d0 2023-03-03T14:13:52.183 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:13:52.183 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:13:52.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e100682f0 msgr2=0x7f5e1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:52.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e100682f0 0x7f5e1006a7a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f5e140090b0 tx=0x7f5e1401c040 comp rx=0 tx=0).stop 2023-03-03T14:13:52.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e240cf0d0 msgr2=0x7f5e2415bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:52.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e240cf0d0 0x7f5e2415bc50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00cbf0 tx=0x7f5e1c00cf00 comp rx=0 tx=0).stop 2023-03-03T14:13:52.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 shutdown_connections 2023-03-03T14:13:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5e100682f0 0x7f5e1006a7a0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5e240b3970 0x7f5e241e4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e240cf0d0 0x7f5e2415bc50 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 --2- 172.21.15.102:0/3609948167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e240ced00 0x7f5e241eed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.183+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 >> 172.21.15.102:0/3609948167 conn(0x7f5e2408b1c0 msgr2=0x7f5e24071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:13:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.184+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 shutdown_connections 2023-03-03T14:13:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:13:52.184+0000 7f5e23fff700 1 -- 172.21.15.102:0/3609948167 wait complete. 2023-03-03T14:13:52.187 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:13:52.212 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-03-03T14:14:05.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:05.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:36.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:36.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:05.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:15:05.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.479+0000 7f0871bff700 1 Processor -- start 2023-03-03T14:15:05.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.479+0000 7f0871bff700 1 -- start start 2023-03-03T14:15:05.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f0871bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f086c0cd8e0 0x7f086c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:05.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f0871bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:05.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f0871bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c153660 con 0x7f086c153290 2023-03-03T14:15:05.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f0871bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086c06c360 con 0x7f086c0cd8e0 2023-03-03T14:15:05.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f0871bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c15cea0 con 0x7f086c0cd390 2023-03-03T14:15:05.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f086b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f086c153290 0x7f086c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54836/0 (socket says 172.21.15.102:54836) 2023-03-03T14:15:05.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f086b7fe700 1 -- 172.21.15.102:0/704696805 learned_addr learned my addr 172.21.15.102:0/704696805 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:15:05.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f086bfff700 1 --2- 172.21.15.102:0/704696805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:05.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.481+0000 7f086affd700 1 --2- 172.21.15.102:0/704696805 >> v2:172.21.15.172:3300/0 conn(0x7f086c0cd8e0 0x7f086c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:05.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086a7fc700 1 -- 172.21.15.102:0/704696805 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259473566 0 0) 0x7f086c153660 con 0x7f086c153290 2023-03-03T14:15:05.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086a7fc700 1 -- 172.21.15.102:0/704696805 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08540035b0 con 0x7f086c153290 2023-03-03T14:15:05.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086bfff700 1 -- 172.21.15.102:0/704696805 >> v2:172.21.15.172:3300/0 conn(0x7f086c0cd8e0 msgr2=0x7f086c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:05.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086bfff700 1 --2- 172.21.15.102:0/704696805 >> v2:172.21.15.172:3300/0 conn(0x7f086c0cd8e0 0x7f086c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086bfff700 1 -- 172.21.15.102:0/704696805 >> v1:172.21.15.102:6789/0 conn(0x7f086c153290 legacy=0x7f086c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:05.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086bfff700 1 -- 172.21.15.102:0/704696805 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c06c7f0 con 0x7f086c0cd390 2023-03-03T14:15:05.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086bfff700 1 --2- 172.21.15.102:0/704696805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c0ce360 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0864009d90 tx=0x7f086400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d243e9e60855b6a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:05.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.482+0000 7f086a7fc700 1 -- 172.21.15.102:0/704696805 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086400e070 con 0x7f086c0cd390 2023-03-03T14:15:05.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f086a7fc700 1 -- 172.21.15.102:0/704696805 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086400bb20 con 0x7f086c0cd390 2023-03-03T14:15:05.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f086a7fc700 1 -- 172.21.15.102:0/704696805 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864012430 con 0x7f086c0cd390 2023-03-03T14:15:05.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 -- 172.21.15.102:0/704696805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 msgr2=0x7f086c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:05.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 --2- 172.21.15.102:0/704696805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c0ce360 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0864009d90 tx=0x7f086400b330 comp rx=0 tx=0).stop 2023-03-03T14:15:05.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 -- 172.21.15.102:0/704696805 shutdown_connections 2023-03-03T14:15:05.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 --2- 172.21.15.102:0/704696805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c0ce360 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 --2- 172.21.15.102:0/704696805 >> v2:172.21.15.172:3300/0 conn(0x7f086c0cd8e0 0x7f086c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 -- 172.21.15.102:0/704696805 >> 172.21.15.102:0/704696805 conn(0x7f086c065fb0 msgr2=0x7f086c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:05.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 -- 172.21.15.102:0/704696805 shutdown_connections 2023-03-03T14:15:05.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.483+0000 7f0871bff700 1 -- 172.21.15.102:0/704696805 wait complete. 2023-03-03T14:15:05.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.485+0000 7f0871bff700 1 Processor -- start 2023-03-03T14:15:05.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.486+0000 7f0871bff700 1 -- start start 2023-03-03T14:15:05.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.486+0000 7f0871bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c1f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:05.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f0871bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f086c0cd8e0 0x7f086c0b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:05.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f0871bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f086c153290 0x7f086c14e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:05.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f0871bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c153fa0 con 0x7f086c0cd8e0 2023-03-03T14:15:05.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f0871bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0c78d0 con 0x7f086c153290 2023-03-03T14:15:05.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f0871bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0c70c0 con 0x7f086c0cd390 2023-03-03T14:15:05.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f086c0cd8e0 0x7f086c0b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:05.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f086c0cd8e0 0x7f086c0b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46200/0 (socket says 172.21.15.102:46200) 2023-03-03T14:15:05.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 -- 172.21.15.102:0/4038611130 learned_addr learned my addr 172.21.15.102:0/4038611130 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:15:05.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086b7fe700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c1f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:05.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086bfff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f086c153290 0x7f086c14e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:05.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 -- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 msgr2=0x7f086c1f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:05.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c1f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 -- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f086c153290 msgr2=0x7f086c14e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:05.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f086c153290 0x7f086c14e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.487+0000 7f086affd700 1 -- 172.21.15.102:0/4038611130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0864009a70 con 0x7f086c0cd8e0 2023-03-03T14:15:05.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.488+0000 7f086affd700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f086c0cd8e0 0x7f086c0b40b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f086000e8a0 tx=0x7f086000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:05.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.488+0000 7f0853fff700 1 -- 172.21.15.102:0/4038611130 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860009590 con 0x7f086c0cd8e0 2023-03-03T14:15:05.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.488+0000 7f0853fff700 1 -- 172.21.15.102:0/4038611130 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0860013680 con 0x7f086c0cd8e0 2023-03-03T14:15:05.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.488+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086c0ccde0 con 0x7f086c0cd8e0 2023-03-03T14:15:05.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.488+0000 7f0853fff700 1 -- 172.21.15.102:0/4038611130 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08600139f0 con 0x7f086c0cd8e0 2023-03-03T14:15:05.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.488+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c15d160 con 0x7f086c0cd8e0 2023-03-03T14:15:05.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.489+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086c0cd0c0 con 0x7f086c0cd8e0 2023-03-03T14:15:05.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.490+0000 7f0853fff700 1 -- 172.21.15.102:0/4038611130 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0860019030 con 0x7f086c0cd8e0 2023-03-03T14:15:05.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.492+0000 7f0853fff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08580682f0 0x7f085806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:05.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.492+0000 7f0853fff700 1 -- 172.21.15.102:0/4038611130 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0860085a90 con 0x7f086c0cd8e0 2023-03-03T14:15:05.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.493+0000 7f086b7fe700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08580682f0 0x7f085806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:05.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.493+0000 7f086b7fe700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08580682f0 0x7f085806a7a0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f085c007fd0 tx=0x7f085c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:05.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.497+0000 7f0853fff700 1 -- 172.21.15.102:0/4038611130 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08600521a0 con 0x7f086c0cd8e0 2023-03-03T14:15:05.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.731+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f086c152ad0 con 0x7f086c0cd8e0 2023-03-03T14:15:05.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.732+0000 7f0853fff700 1 -- 172.21.15.102:0/4038611130 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f08600557c0 con 0x7f086c0cd8e0 2023-03-03T14:15:05.733 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:15:05.734 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:15:05.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08580682f0 msgr2=0x7f085806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:05.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08580682f0 0x7f085806a7a0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f085c007fd0 tx=0x7f085c007dc0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f086c0cd8e0 msgr2=0x7f086c0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:05.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f086c0cd8e0 0x7f086c0b40b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f086000e8a0 tx=0x7f086000ebb0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 shutdown_connections 2023-03-03T14:15:05.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f08580682f0 0x7f085806a7a0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f086c0cd390 0x7f086c1f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f086c153290 0x7f086c14e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 --2- 172.21.15.102:0/4038611130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f086c0cd8e0 0x7f086c0b40b0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:05.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.734+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 >> 172.21.15.102:0/4038611130 conn(0x7f086c065fb0 msgr2=0x7f086c156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:05.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.735+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 shutdown_connections 2023-03-03T14:15:05.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:05.735+0000 7f0871bff700 1 -- 172.21.15.102:0/4038611130 wait complete. 2023-03-03T14:15:05.738 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:15:05.763 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-03-03T14:15:06.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:06.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:15.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:15:16.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.025+0000 7fe5017ff700 1 Processor -- start 2023-03-03T14:15:16.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.025+0000 7fe5017ff700 1 -- start start 2023-03-03T14:15:16.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe5017ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe4fc153290 0x7fe4fc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:16.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe5017ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc1593c0 0x7fe4fc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:16.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe5017ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe4fc0cd8e0 0x7fe4fc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:16.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe5017ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc153660 con 0x7fe4fc0cd8e0 2023-03-03T14:15:16.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe5017ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc06c360 con 0x7fe4fc153290 2023-03-03T14:15:16.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe5017ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc15cea0 con 0x7fe4fc1593c0 2023-03-03T14:15:16.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe4fb7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe4fc0cd8e0 0x7fe4fc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:16.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe4fb7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe4fc0cd8e0 0x7fe4fc0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46338/0 (socket says 172.21.15.102:46338) 2023-03-03T14:15:16.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe4fb7fe700 1 -- 172.21.15.102:0/1809851777 learned_addr learned my addr 172.21.15.102:0/1809851777 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:15:16.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe4fa7fc700 1 --2- 172.21.15.102:0/1809851777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc1593c0 0x7fe4fc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:16.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.027+0000 7fe4faffd700 1 --2- 172.21.15.102:0/1809851777 >> v2:172.21.15.172:3300/0 conn(0x7fe4fc153290 0x7fe4fc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4fb7fe700 1 -- 172.21.15.102:0/1809851777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc1593c0 msgr2=0x7fe4fc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4fb7fe700 1 --2- 172.21.15.102:0/1809851777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc1593c0 0x7fe4fc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4fb7fe700 1 -- 172.21.15.102:0/1809851777 >> v2:172.21.15.172:3300/0 conn(0x7fe4fc153290 msgr2=0x7fe4fc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4fb7fe700 1 --2- 172.21.15.102:0/1809851777 >> v2:172.21.15.172:3300/0 conn(0x7fe4fc153290 0x7fe4fc158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4fb7fe700 1 -- 172.21.15.102:0/1809851777 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc06c7f0 con 0x7fe4fc0cd8e0 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4faffd700 1 --2- 172.21.15.102:0/1809851777 >> v2:172.21.15.172:3300/0 conn(0x7fe4fc153290 0x7fe4fc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4fb7fe700 1 --2- 172.21.15.102:0/1809851777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc0cd8e0 0x7fe4fc0ce360 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc08cc70 tx=0x7fe4f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87635352f65be569 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:16.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4f9ffb700 1 -- 172.21.15.102:0/1809851777 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f400e070 con 0x7fe4fc0cd8e0 2023-03-03T14:15:16.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4f9ffb700 1 -- 172.21.15.102:0/1809851777 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f400b910 con 0x7fe4fc0cd8e0 2023-03-03T14:15:16.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.028+0000 7fe4f9ffb700 1 -- 172.21.15.102:0/1809851777 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f400bcc0 con 0x7fe4fc0cd8e0 2023-03-03T14:15:16.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 -- 172.21.15.102:0/1809851777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc0cd8e0 msgr2=0x7fe4fc0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:16.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1809851777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc0cd8e0 0x7fe4fc0ce360 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc08cc70 tx=0x7fe4f400b330 comp rx=0 tx=0).stop 2023-03-03T14:15:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 -- 172.21.15.102:0/1809851777 shutdown_connections 2023-03-03T14:15:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1809851777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc0cd8e0 0x7fe4fc0ce360 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1809851777 >> v2:172.21.15.172:3300/0 conn(0x7fe4fc153290 0x7fe4fc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1809851777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc1593c0 0x7fe4fc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 -- 172.21.15.102:0/1809851777 >> 172.21.15.102:0/1809851777 conn(0x7fe4fc065fb0 msgr2=0x7fe4fc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:16.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 -- 172.21.15.102:0/1809851777 shutdown_connections 2023-03-03T14:15:16.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.029+0000 7fe5017ff700 1 -- 172.21.15.102:0/1809851777 wait complete. 2023-03-03T14:15:16.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.031+0000 7fe5017ff700 1 Processor -- start 2023-03-03T14:15:16.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.031+0000 7fe5017ff700 1 -- start start 2023-03-03T14:15:16.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.032+0000 7fe5017ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc0cd8e0 0x7fe4fc0c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:16.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.032+0000 7fe5017ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc153290 0x7fe4fc0b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:16.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.032+0000 7fe5017ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4fc1593c0 0x7fe4fc0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:16.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.032+0000 7fe5017ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc1515e0 con 0x7fe4fc153290 2023-03-03T14:15:16.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.032+0000 7fe5017ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc1e5470 con 0x7fe4fc1593c0 2023-03-03T14:15:16.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.032+0000 7fe5017ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc06cce0 con 0x7fe4fc0cd8e0 2023-03-03T14:15:16.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc153290 0x7fe4fc0b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:16.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4faffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc0cd8e0 0x7fe4fc0c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:16.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc153290 0x7fe4fc0b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46348/0 (socket says 172.21.15.102:46348) 2023-03-03T14:15:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 -- 172.21.15.102:0/1068063319 learned_addr learned my addr 172.21.15.102:0/1068063319 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:15:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fb7fe700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4fc1593c0 0x7fe4fc0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 -- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc0cd8e0 msgr2=0x7fe4fc0c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc0cd8e0 0x7fe4fc0c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 -- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4fc1593c0 msgr2=0x7fe4fc0c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4fc1593c0 0x7fe4fc0c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 -- 172.21.15.102:0/1068063319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4009b30 con 0x7fe4fc153290 2023-03-03T14:15:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.033+0000 7fe4fa7fc700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc153290 0x7fe4fc0b1c70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000e9b0 tx=0x7fe4f000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.034+0000 7fe4e37fe700 1 -- 172.21.15.102:0/1068063319 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0009710 con 0x7fe4fc153290 2023-03-03T14:15:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.034+0000 7fe4e37fe700 1 -- 172.21.15.102:0/1068063319 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f000ee50 con 0x7fe4fc153290 2023-03-03T14:15:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.034+0000 7fe4e37fe700 1 -- 172.21.15.102:0/1068063319 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f000ca50 con 0x7fe4fc153290 2023-03-03T14:15:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.034+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc0ccde0 con 0x7fe4fc153290 2023-03-03T14:15:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.034+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc15d160 con 0x7fe4fc153290 2023-03-03T14:15:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.035+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc0cd0c0 con 0x7fe4fc153290 2023-03-03T14:15:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.036+0000 7fe4e37fe700 1 -- 172.21.15.102:0/1068063319 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe4f0019030 con 0x7fe4fc153290 2023-03-03T14:15:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.037+0000 7fe4e37fe700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe4e80682f0 0x7fe4e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.038+0000 7fe4e37fe700 1 -- 172.21.15.102:0/1068063319 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe4f0085c00 con 0x7fe4fc153290 2023-03-03T14:15:16.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.038+0000 7fe4faffd700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe4e80682f0 0x7fe4e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:16.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.039+0000 7fe4faffd700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe4e80682f0 0x7fe4e806a7a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00c940 tx=0x7fe4ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:16.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.041+0000 7fe4e37fe700 1 -- 172.21.15.102:0/1068063319 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe4f0052310 con 0x7fe4fc153290 2023-03-03T14:15:16.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.275+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4fc04e6e0 con 0x7fe4fc153290 2023-03-03T14:15:16.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.276+0000 7fe4e37fe700 1 -- 172.21.15.102:0/1068063319 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe4f0055930 con 0x7fe4fc153290 2023-03-03T14:15:16.277 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:15:16.278 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:15:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.278+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe4e80682f0 msgr2=0x7fe4e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.278+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe4e80682f0 0x7fe4e806a7a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00c940 tx=0x7fe4ec008040 comp rx=0 tx=0).stop 2023-03-03T14:15:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.278+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc153290 msgr2=0x7fe4fc0b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:16.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.278+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc153290 0x7fe4fc0b1c70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000e9b0 tx=0x7fe4f000ecc0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 shutdown_connections 2023-03-03T14:15:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe4e80682f0 0x7fe4e806a7a0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4fc0cd8e0 0x7fe4fc0c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4fc1593c0 0x7fe4fc0c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 --2- 172.21.15.102:0/1068063319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4fc153290 0x7fe4fc0b1c70 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 >> 172.21.15.102:0/1068063319 conn(0x7fe4fc065fb0 msgr2=0x7fe4fc0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 shutdown_connections 2023-03-03T14:15:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:15:16.279+0000 7fe5017ff700 1 -- 172.21.15.102:0/1068063319 wait complete. 2023-03-03T14:15:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:15:16.307 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 45 secs before thrashing 2023-03-03T14:15:36.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:36.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:01.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:16:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.556+0000 7fe493dff700 1 Processor -- start 2023-03-03T14:16:01.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.556+0000 7fe493dff700 1 -- start start 2023-03-03T14:16:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.557+0000 7fe493dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe48c0cd8e0 0x7fe48c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe493dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c153660 con 0x7fe48c153290 2023-03-03T14:16:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe493dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c06c360 con 0x7fe48c0cd390 2023-03-03T14:16:01.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe493dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c15cea0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe491b9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe48c153290 0x7fe48c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41772/0 (socket says 172.21.15.102:41772) 2023-03-03T14:16:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe491b9b700 1 -- 172.21.15.102:0/2183591597 learned_addr learned my addr 172.21.15.102:0/2183591597 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:16:01.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe49139a700 1 --2- 172.21.15.102:0/2183591597 >> v2:172.21.15.172:3300/0 conn(0x7fe48c0cd8e0 0x7fe48c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe490b99700 1 -- 172.21.15.102:0/2183591597 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287311074 0 0) 0x7fe48c153660 con 0x7fe48c153290 2023-03-03T14:16:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe490b99700 1 -- 172.21.15.102:0/2183591597 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4780035b0 con 0x7fe48c153290 2023-03-03T14:16:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.558+0000 7fe49139a700 1 -- 172.21.15.102:0/2183591597 >> v1:172.21.15.172:6789/0 conn(0x7fe48c0cd390 legacy=0x7fe48c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.559+0000 7fe490b99700 1 -- 172.21.15.102:0/2183591597 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857093091 0 0) 0x7fe48c06c360 con 0x7fe48c0cd390 2023-03-03T14:16:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.559+0000 7fe49139a700 1 -- 172.21.15.102:0/2183591597 >> v1:172.21.15.102:6789/0 conn(0x7fe48c153290 legacy=0x7fe48c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.559+0000 7fe49139a700 1 -- 172.21.15.102:0/2183591597 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c06c7f0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.559+0000 7fe49139a700 1 --2- 172.21.15.102:0/2183591597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c0cde20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009840 tx=0x7fe47c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ac8543581013c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:16:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.559+0000 7fe490b99700 1 -- 172.21.15.102:0/2183591597 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011830 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.559+0000 7fe490b99700 1 -- 172.21.15.102:0/2183591597 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c011990 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.559+0000 7fe490b99700 1 -- 172.21.15.102:0/2183591597 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011d20 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.560+0000 7fe493dff700 1 -- 172.21.15.102:0/2183591597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 msgr2=0x7fe48c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.560+0000 7fe493dff700 1 --2- 172.21.15.102:0/2183591597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c0cde20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009840 tx=0x7fe47c009b50 comp rx=0 tx=0).stop 2023-03-03T14:16:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.560+0000 7fe493dff700 1 -- 172.21.15.102:0/2183591597 shutdown_connections 2023-03-03T14:16:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.560+0000 7fe493dff700 1 --2- 172.21.15.102:0/2183591597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c0cde20 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.560+0000 7fe493dff700 1 -- 172.21.15.102:0/2183591597 >> 172.21.15.102:0/2183591597 conn(0x7fe48c065fb0 msgr2=0x7fe48c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:16:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.560+0000 7fe493dff700 1 -- 172.21.15.102:0/2183591597 shutdown_connections 2023-03-03T14:16:01.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.560+0000 7fe493dff700 1 -- 172.21.15.102:0/2183591597 wait complete. 2023-03-03T14:16:01.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.562+0000 7fe493dff700 1 Processor -- start 2023-03-03T14:16:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.562+0000 7fe493dff700 1 -- start start 2023-03-03T14:16:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe493dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c0cd390 0x7fe48c157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe493dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c1ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe493dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48c153290 0x7fe48c1eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:01.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe493dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c6df0 con 0x7fe48c0cd390 2023-03-03T14:16:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe493dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c8790 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe493dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c8a70 con 0x7fe48c153290 2023-03-03T14:16:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe49139a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c1ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe491b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c0cd390 0x7fe48c157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe49239c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48c153290 0x7fe48c1eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe49139a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c1ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:58654/0 (socket says 172.21.15.102:58654) 2023-03-03T14:16:01.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.563+0000 7fe49139a700 1 -- 172.21.15.102:0/1845702747 learned_addr learned my addr 172.21.15.102:0/1845702747 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:16:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe49139a700 1 -- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48c153290 msgr2=0x7fe48c1eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe49139a700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48c153290 0x7fe48c1eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe49139a700 1 -- 172.21.15.102:0/1845702747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c0cd390 msgr2=0x7fe48c157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe49139a700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c0cd390 0x7fe48c157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe49139a700 1 -- 172.21.15.102:0/1845702747 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0094a0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe491b9b700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c0cd390 0x7fe48c157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:16:01.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe49139a700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c1ee8c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0090b0 tx=0x7fe47c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:16:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe4827fc700 1 -- 172.21.15.102:0/1845702747 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c01a7e0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe4827fc700 1 -- 172.21.15.102:0/1845702747 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c01a940 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe4827fc700 1 -- 172.21.15.102:0/1845702747 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c01acd0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.564+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0ccde0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.565+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c15d160 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.566+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c0cd0c0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.567+0000 7fe4827fc700 1 -- 172.21.15.102:0/1845702747 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe47c00f050 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.568+0000 7fe4827fc700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.568+0000 7fe4827fc700 1 -- 172.21.15.102:0/1845702747 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe47c086d00 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.569+0000 7fe491b9b700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:01.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.569+0000 7fe491b9b700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fe488007fd0 tx=0x7fe488009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:16:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.573+0000 7fe4827fc700 1 -- 172.21.15.102:0/1845702747 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe47c053380 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.806+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe48c04e6e0 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.807+0000 7fe4827fc700 1 -- 172.21.15.102:0/1845702747 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe47c029070 con 0x7fe48c0cd8e0 2023-03-03T14:16:01.808 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:16:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:16:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 msgr2=0x7fe47006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fe488007fd0 tx=0x7fe488009040 comp rx=0 tx=0).stop 2023-03-03T14:16:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 msgr2=0x7fe48c1ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c1ee8c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0090b0 tx=0x7fe47c004750 comp rx=0 tx=0).stop 2023-03-03T14:16:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 shutdown_connections 2023-03-03T14:16:01.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe470068320 0x7fe47006a7d0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe48c153290 0x7fe48c1eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c0cd390 0x7fe48c157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 --2- 172.21.15.102:0/1845702747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe48c0cd8e0 0x7fe48c1ee8c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 >> 172.21.15.102:0/1845702747 conn(0x7fe48c065fb0 msgr2=0x7fe48c158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:16:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.809+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 shutdown_connections 2023-03-03T14:16:01.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:16:01.810+0000 7fe493dff700 1 -- 172.21.15.102:0/1845702747 wait complete. 2023-03-03T14:16:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:16:01.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T14:16:06.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:06.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:36.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:36.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:07.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:07.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:20.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:17:21.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.088+0000 7f49471ff700 1 Processor -- start 2023-03-03T14:17:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.088+0000 7f49471ff700 1 -- start start 2023-03-03T14:17:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f49471ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49400cd390 0x7f49400ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f49471ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940153660 con 0x7f49400cd8e0 2023-03-03T14:17:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f49471ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494006c360 con 0x7f4940153290 2023-03-03T14:17:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f49471ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494015cea0 con 0x7f49400cd390 2023-03-03T14:17:21.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f493ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f49400cd8e0 0x7f49400cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60654/0 (socket says 172.21.15.102:60654) 2023-03-03T14:17:21.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f4944f9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4940153290 0x7f4940158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33002/0 (socket says 172.21.15.102:33002) 2023-03-03T14:17:21.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f493ffff700 1 -- 172.21.15.102:0/2996334862 learned_addr learned my addr 172.21.15.102:0/2996334862 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:17:21.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f494579c700 1 --2- 172.21.15.102:0/2996334862 >> v2:172.21.15.102:3300/0 conn(0x7f49400cd390 0x7f49400ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:21.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.090+0000 7f493f7fe700 1 -- 172.21.15.102:0/2996334862 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742081815 0 0) 0x7f494006c360 con 0x7f4940153290 2023-03-03T14:17:21.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f493f7fe700 1 -- 172.21.15.102:0/2996334862 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49300035b0 con 0x7f4940153290 2023-03-03T14:17:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f493f7fe700 1 -- 172.21.15.102:0/2996334862 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035607546 0 0) 0x7f4940153660 con 0x7f49400cd8e0 2023-03-03T14:17:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f494579c700 1 -- 172.21.15.102:0/2996334862 >> v1:172.21.15.172:6789/0 conn(0x7f4940153290 legacy=0x7f4940158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f494579c700 1 -- 172.21.15.102:0/2996334862 >> v1:172.21.15.102:6789/0 conn(0x7f49400cd8e0 legacy=0x7f49400cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f494579c700 1 -- 172.21.15.102:0/2996334862 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494006c7f0 con 0x7f49400cd390 2023-03-03T14:17:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f494579c700 1 --2- 172.21.15.102:0/2996334862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400ce530 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4938009d90 tx=0x7f493800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b91e18cfe97addf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f493f7fe700 1 -- 172.21.15.102:0/2996334862 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493800e070 con 0x7f49400cd390 2023-03-03T14:17:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f493f7fe700 1 -- 172.21.15.102:0/2996334862 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493800bb20 con 0x7f49400cd390 2023-03-03T14:17:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.091+0000 7f493f7fe700 1 -- 172.21.15.102:0/2996334862 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493800bed0 con 0x7f49400cd390 2023-03-03T14:17:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.092+0000 7f49471ff700 1 -- 172.21.15.102:0/2996334862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 msgr2=0x7f49400ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.092+0000 7f49471ff700 1 --2- 172.21.15.102:0/2996334862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400ce530 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4938009d90 tx=0x7f493800b330 comp rx=0 tx=0).stop 2023-03-03T14:17:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.092+0000 7f49471ff700 1 -- 172.21.15.102:0/2996334862 shutdown_connections 2023-03-03T14:17:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.092+0000 7f49471ff700 1 --2- 172.21.15.102:0/2996334862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400ce530 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.092+0000 7f49471ff700 1 -- 172.21.15.102:0/2996334862 >> 172.21.15.102:0/2996334862 conn(0x7f4940065fb0 msgr2=0x7f49400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:17:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.092+0000 7f49471ff700 1 -- 172.21.15.102:0/2996334862 shutdown_connections 2023-03-03T14:17:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.092+0000 7f49471ff700 1 -- 172.21.15.102:0/2996334862 wait complete. 2023-03-03T14:17:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.094+0000 7f49471ff700 1 Processor -- start 2023-03-03T14:17:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.094+0000 7f49471ff700 1 -- start start 2023-03-03T14:17:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.095+0000 7f49471ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.095+0000 7f49471ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49400cd8e0 0x7f49400cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.095+0000 7f49471ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4940153290 0x7f49400cf380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.095+0000 7f49471ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49400b2650 con 0x7f49400cd390 2023-03-03T14:17:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.095+0000 7f49471ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49400c6410 con 0x7f4940153290 2023-03-03T14:17:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.095+0000 7f49471ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49401efe50 con 0x7f49400cd8e0 2023-03-03T14:17:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f494579c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4940153290 0x7f49400cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53226/0 (socket says 172.21.15.102:53226) 2023-03-03T14:17:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 -- 172.21.15.102:0/1537674028 learned_addr learned my addr 172.21.15.102:0/1537674028 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:17:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f493ffff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49400cd8e0 0x7f49400cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 -- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49400cd8e0 msgr2=0x7f49400cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49400cd8e0 0x7f49400cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 -- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4940153290 msgr2=0x7f49400cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4940153290 0x7f49400cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.096+0000 7f4944f9b700 1 -- 172.21.15.102:0/1537674028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4938009a70 con 0x7f49400cd390 2023-03-03T14:17:21.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.097+0000 7f4944f9b700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400b2110 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f493400c440 tx=0x7f493400c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:21.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.097+0000 7f493d7fa700 1 -- 172.21.15.102:0/1537674028 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934005cb0 con 0x7f49400cd390 2023-03-03T14:17:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.097+0000 7f493d7fa700 1 -- 172.21.15.102:0/1537674028 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4934005e10 con 0x7f49400cd390 2023-03-03T14:17:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.097+0000 7f493d7fa700 1 -- 172.21.15.102:0/1537674028 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934007310 con 0x7f49400cd390 2023-03-03T14:17:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.097+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49400ccde0 con 0x7f49400cd390 2023-03-03T14:17:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.097+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494015d160 con 0x7f49400cd390 2023-03-03T14:17:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.098+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49400cd0c0 con 0x7f49400cd390 2023-03-03T14:17:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.099+0000 7f493d7fa700 1 -- 172.21.15.102:0/1537674028 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4934019030 con 0x7f49400cd390 2023-03-03T14:17:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.101+0000 7f493d7fa700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4924068320 0x7f492406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.102+0000 7f493d7fa700 1 -- 172.21.15.102:0/1537674028 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4934084fe0 con 0x7f49400cd390 2023-03-03T14:17:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.102+0000 7f493ffff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4924068320 0x7f492406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.103+0000 7f493ffff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4924068320 0x7f492406a7d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f49401f2b00 tx=0x7f492c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:21.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.106+0000 7f493d7fa700 1 -- 172.21.15.102:0/1537674028 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49340517d0 con 0x7f49400cd390 2023-03-03T14:17:21.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.341+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4940152ad0 con 0x7f49400cd390 2023-03-03T14:17:21.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.342+0000 7f493d7fa700 1 -- 172.21.15.102:0/1537674028 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4934017090 con 0x7f49400cd390 2023-03-03T14:17:21.345 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:17:21.345 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:17:21.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4924068320 msgr2=0x7f492406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4924068320 0x7f492406a7d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f49401f2b00 tx=0x7f492c007de0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 msgr2=0x7f49400b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400b2110 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f493400c440 tx=0x7f493400c750 comp rx=0 tx=0).stop 2023-03-03T14:17:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 shutdown_connections 2023-03-03T14:17:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4940153290 0x7f49400cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd390 0x7f49400b2110 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4924068320 0x7f492406a7d0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 --2- 172.21.15.102:0/1537674028 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f49400cd8e0 0x7f49400cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 >> 172.21.15.102:0/1537674028 conn(0x7f4940065fb0 msgr2=0x7f4940158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:17:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 shutdown_connections 2023-03-03T14:17:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:17:21.345+0000 7f49471ff700 1 -- 172.21.15.102:0/1537674028 wait complete. 2023-03-03T14:17:21.351 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:17:21.375 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T14:17:37.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:37.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:01.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:18:01.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.622+0000 7f16b099d700 1 Processor -- start 2023-03-03T14:18:01.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.622+0000 7f16b099d700 1 -- start start 2023-03-03T14:18:01.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16b099d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f16ac0b3970 0x7f16ac0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:01.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16b099d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0c3d20 con 0x7f16ac0ced00 2023-03-03T14:18:01.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16b099d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac15cba0 con 0x7f16ac0c41b0 2023-03-03T14:18:01.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16b099d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac0c4930 con 0x7f16ac0b3970 2023-03-03T14:18:01.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16ab7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f16ac0ced00 0x7f16ac158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46472/0 (socket says 172.21.15.102:46472) 2023-03-03T14:18:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16ab7fe700 1 -- 172.21.15.102:0/4124662677 learned_addr learned my addr 172.21.15.102:0/4124662677 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:18:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16abfff700 1 --1- 172.21.15.102:0/4124662677 >> v1:172.21.15.172:6789/0 conn(0x7f16ac0c41b0 0x7f16ac163480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:43362/0 (socket says 172.21.15.102:43362) 2023-03-03T14:18:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.624+0000 7f16aaffd700 1 --2- 172.21.15.102:0/4124662677 >> v2:172.21.15.102:3300/0 conn(0x7f16ac0b3970 0x7f16ac0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aa7fc700 1 -- 172.21.15.102:0/4124662677 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699351676 0 0) 0x7f16ac15cba0 con 0x7f16ac0c41b0 2023-03-03T14:18:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aa7fc700 1 -- 172.21.15.102:0/4124662677 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16940035b0 con 0x7f16ac0c41b0 2023-03-03T14:18:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aa7fc700 1 -- 172.21.15.102:0/4124662677 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242930178 0 0) 0x7f16ac0c3d20 con 0x7f16ac0ced00 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aa7fc700 1 -- 172.21.15.102:0/4124662677 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac15cba0 con 0x7f16ac0ced00 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aaffd700 1 -- 172.21.15.102:0/4124662677 >> v1:172.21.15.172:6789/0 conn(0x7f16ac0c41b0 legacy=0x7f16ac163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aaffd700 1 -- 172.21.15.102:0/4124662677 >> v1:172.21.15.102:6789/0 conn(0x7f16ac0ced00 legacy=0x7f16ac158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aaffd700 1 -- 172.21.15.102:0/4124662677 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac15ce80 con 0x7f16ac0b3970 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.625+0000 7f16aaffd700 1 --2- 172.21.15.102:0/4124662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0b3970 0x7f16ac0c5a90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009840 tx=0x7f16a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d980f354dbce452a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16aa7fc700 1 -- 172.21.15.102:0/4124662677 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011830 con 0x7f16ac0b3970 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16aa7fc700 1 -- 172.21.15.102:0/4124662677 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0011990 con 0x7f16ac0b3970 2023-03-03T14:18:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16aa7fc700 1 -- 172.21.15.102:0/4124662677 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011d20 con 0x7f16ac0b3970 2023-03-03T14:18:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16b099d700 1 -- 172.21.15.102:0/4124662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0b3970 msgr2=0x7f16ac0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16b099d700 1 --2- 172.21.15.102:0/4124662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0b3970 0x7f16ac0c5a90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009840 tx=0x7f16a0009b50 comp rx=0 tx=0).stop 2023-03-03T14:18:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16b099d700 1 -- 172.21.15.102:0/4124662677 shutdown_connections 2023-03-03T14:18:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16b099d700 1 --2- 172.21.15.102:0/4124662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0b3970 0x7f16ac0c5a90 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16b099d700 1 -- 172.21.15.102:0/4124662677 >> 172.21.15.102:0/4124662677 conn(0x7f16ac08b1c0 msgr2=0x7f16ac071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16b099d700 1 -- 172.21.15.102:0/4124662677 shutdown_connections 2023-03-03T14:18:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.626+0000 7f16b099d700 1 -- 172.21.15.102:0/4124662677 wait complete. 2023-03-03T14:18:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.629+0000 7f16b099d700 1 Processor -- start 2023-03-03T14:18:01.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.629+0000 7f16b099d700 1 -- start start 2023-03-03T14:18:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16b099d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac0b3970 0x7f16ac15b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16b099d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16ac0c41b0 0x7f16ac1e5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16b099d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0ced00 0x7f16ac1501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16b099d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac15a3c0 con 0x7f16ac0ced00 2023-03-03T14:18:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16b099d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac15b8c0 con 0x7f16ac0c41b0 2023-03-03T14:18:01.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16b099d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac150f30 con 0x7f16ac0b3970 2023-03-03T14:18:01.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16abfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0ced00 0x7f16ac1501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:01.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16aaffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16ac0c41b0 0x7f16ac1e5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:01.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16ab7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac0b3970 0x7f16ac15b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:01.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16aaffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16ac0c41b0 0x7f16ac1e5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50270/0 (socket says 172.21.15.102:50270) 2023-03-03T14:18:01.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.631+0000 7f16aaffd700 1 -- 172.21.15.102:0/2052072432 learned_addr learned my addr 172.21.15.102:0/2052072432 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16abfff700 1 -- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac0b3970 msgr2=0x7f16ac15b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16abfff700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac0b3970 0x7f16ac15b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16abfff700 1 -- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16ac0c41b0 msgr2=0x7f16ac1e5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16abfff700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16ac0c41b0 0x7f16ac1e5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16abfff700 1 -- 172.21.15.102:0/2052072432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00094a0 con 0x7f16ac0ced00 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16aaffd700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16ac0c41b0 0x7f16ac1e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16abfff700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0ced00 0x7f16ac1501c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f16a400caa0 tx=0x7f16a400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f1693fff700 1 -- 172.21.15.102:0/2052072432 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a400a960 con 0x7f16ac0ced00 2023-03-03T14:18:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f1693fff700 1 -- 172.21.15.102:0/2052072432 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a400aae0 con 0x7f16ac0ced00 2023-03-03T14:18:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f1693fff700 1 -- 172.21.15.102:0/2052072432 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a400ae90 con 0x7f16ac0ced00 2023-03-03T14:18:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.632+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac06c200 con 0x7f16ac0ced00 2023-03-03T14:18:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.633+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac06bf20 con 0x7f16ac0ced00 2023-03-03T14:18:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.634+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac15c890 con 0x7f16ac0ced00 2023-03-03T14:18:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.635+0000 7f1693fff700 1 -- 172.21.15.102:0/2052072432 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f16a4019030 con 0x7f16ac0ced00 2023-03-03T14:18:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.636+0000 7f1693fff700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16980682f0 0x7f169806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.636+0000 7f1693fff700 1 -- 172.21.15.102:0/2052072432 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f16a4085840 con 0x7f16ac0ced00 2023-03-03T14:18:01.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.636+0000 7f16ab7fe700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16980682f0 0x7f169806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:01.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.637+0000 7f16ab7fe700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16980682f0 0x7f169806a7a0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f169c007fd0 tx=0x7f169c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:01.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.640+0000 7f1693fff700 1 -- 172.21.15.102:0/2052072432 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f16a4051fb0 con 0x7f16ac0ced00 2023-03-03T14:18:01.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.874+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16ac04e6e0 con 0x7f16ac0ced00 2023-03-03T14:18:01.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.874+0000 7f1693fff700 1 -- 172.21.15.102:0/2052072432 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f16a40555d0 con 0x7f16ac0ced00 2023-03-03T14:18:01.877 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:18:01.877 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:18:01.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16980682f0 msgr2=0x7f169806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:01.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16980682f0 0x7f169806a7a0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f169c007fd0 tx=0x7f169c007dc0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0ced00 msgr2=0x7f16ac1501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:01.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0ced00 0x7f16ac1501c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f16a400caa0 tx=0x7f16a400cdb0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 shutdown_connections 2023-03-03T14:18:01.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f16980682f0 0x7f169806a7a0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac0b3970 0x7f16ac15b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac0ced00 0x7f16ac1501c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 --2- 172.21.15.102:0/2052072432 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16ac0c41b0 0x7f16ac1e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:01.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 >> 172.21.15.102:0/2052072432 conn(0x7f16ac08b1c0 msgr2=0x7f16ac071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:01.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 shutdown_connections 2023-03-03T14:18:01.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:01.877+0000 7f16b099d700 1 -- 172.21.15.102:0/2052072432 wait complete. 2023-03-03T14:18:01.882 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:18:01.906 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-03-03T14:18:07.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:07.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:37.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:37.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:37.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:18:38.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.153+0000 7f9691bff700 1 Processor -- start 2023-03-03T14:18:38.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.153+0000 7f9691bff700 1 -- start start 2023-03-03T14:18:38.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.154+0000 7f9691bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f968c15c870 0x7f968c06cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:38.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.154+0000 7f9691bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c15cd00 con 0x7f968c0cea00 2023-03-03T14:18:38.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f9691bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c15cfa0 con 0x7f968c152eb0 2023-03-03T14:18:38.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f9691bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968c0ce5b0 con 0x7f968c15c870 2023-03-03T14:18:38.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f968c0cea00 0x7f968c15c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55096/0 (socket says 172.21.15.102:55096) 2023-03-03T14:18:38.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f968c152eb0 0x7f968c06b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:32808/0 (socket says 172.21.15.102:32808) 2023-03-03T14:18:38.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968affd700 1 -- 172.21.15.102:0/731923163 learned_addr learned my addr 172.21.15.102:0/731923163 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:18:38.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968bfff700 1 --2- 172.21.15.102:0/731923163 >> v2:172.21.15.102:3300/0 conn(0x7f968c15c870 0x7f968c06cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968a7fc700 1 -- 172.21.15.102:0/731923163 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198658901 0 0) 0x7f968c15cfa0 con 0x7f968c152eb0 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968a7fc700 1 -- 172.21.15.102:0/731923163 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96740035b0 con 0x7f968c152eb0 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968a7fc700 1 -- 172.21.15.102:0/731923163 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912713125 0 0) 0x7f968c15cd00 con 0x7f968c0cea00 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968a7fc700 1 -- 172.21.15.102:0/731923163 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f968c15cfa0 con 0x7f968c0cea00 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968bfff700 1 -- 172.21.15.102:0/731923163 >> v1:172.21.15.172:6789/0 conn(0x7f968c152eb0 legacy=0x7f968c06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968bfff700 1 -- 172.21.15.102:0/731923163 >> v1:172.21.15.102:6789/0 conn(0x7f968c0cea00 legacy=0x7f968c15c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.155+0000 7f968bfff700 1 -- 172.21.15.102:0/731923163 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c0cf750 con 0x7f968c15c870 2023-03-03T14:18:38.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.156+0000 7f968bfff700 1 --2- 172.21.15.102:0/731923163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 0x7f968c06cfd0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9684009d90 tx=0x7f968400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c102a6a1260be402 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:38.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.156+0000 7f968a7fc700 1 -- 172.21.15.102:0/731923163 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400e070 con 0x7f968c15c870 2023-03-03T14:18:38.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.156+0000 7f968a7fc700 1 -- 172.21.15.102:0/731923163 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968400bb20 con 0x7f968c15c870 2023-03-03T14:18:38.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.156+0000 7f968a7fc700 1 -- 172.21.15.102:0/731923163 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684012430 con 0x7f968c15c870 2023-03-03T14:18:38.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.157+0000 7f9691bff700 1 -- 172.21.15.102:0/731923163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 msgr2=0x7f968c06cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:38.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.157+0000 7f9691bff700 1 --2- 172.21.15.102:0/731923163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 0x7f968c06cfd0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9684009d90 tx=0x7f968400b330 comp rx=0 tx=0).stop 2023-03-03T14:18:38.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.157+0000 7f9691bff700 1 -- 172.21.15.102:0/731923163 shutdown_connections 2023-03-03T14:18:38.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.157+0000 7f9691bff700 1 --2- 172.21.15.102:0/731923163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 0x7f968c06cfd0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.157+0000 7f9691bff700 1 -- 172.21.15.102:0/731923163 >> 172.21.15.102:0/731923163 conn(0x7f968c066c40 msgr2=0x7f968c0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:38.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.157+0000 7f9691bff700 1 -- 172.21.15.102:0/731923163 shutdown_connections 2023-03-03T14:18:38.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.157+0000 7f9691bff700 1 -- 172.21.15.102:0/731923163 wait complete. 2023-03-03T14:18:38.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.159+0000 7f9691bff700 1 Processor -- start 2023-03-03T14:18:38.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.159+0000 7f9691bff700 1 -- start start 2023-03-03T14:18:38.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f9691bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f968c0cea00 0x7f968c1e5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:38.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f9691bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f968c152eb0 0x7f968c153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:38.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f9691bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 0x7f968c14e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:38.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f9691bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c14f820 con 0x7f968c15c870 2023-03-03T14:18:38.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f9691bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c0cc3f0 con 0x7f968c0cea00 2023-03-03T14:18:38.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f9691bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968c150ab0 con 0x7f968c152eb0 2023-03-03T14:18:38.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f968b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f968c0cea00 0x7f968c1e5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:38.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f968b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f968c0cea00 0x7f968c1e5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40502/0 (socket says 172.21.15.102:40502) 2023-03-03T14:18:38.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f968b7fe700 1 -- 172.21.15.102:0/2623826579 learned_addr learned my addr 172.21.15.102:0/2623826579 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:18:38.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f968affd700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f968c152eb0 0x7f968c153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:38.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.160+0000 7f968bfff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 0x7f968c14e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:38.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f968b7fe700 1 -- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f968c152eb0 msgr2=0x7f968c153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:38.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f968b7fe700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f968c152eb0 0x7f968c153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f968b7fe700 1 -- 172.21.15.102:0/2623826579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 msgr2=0x7f968c14e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:38.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f968b7fe700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 0x7f968c14e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f968b7fe700 1 -- 172.21.15.102:0/2623826579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9684009a70 con 0x7f968c0cea00 2023-03-03T14:18:38.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f968b7fe700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f968c0cea00 0x7f968c1e5220 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c530 tx=0x7f967c00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:38.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f9673fff700 1 -- 172.21.15.102:0/2623826579 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c005cb0 con 0x7f968c0cea00 2023-03-03T14:18:38.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f9673fff700 1 -- 172.21.15.102:0/2623826579 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c005e10 con 0x7f968c0cea00 2023-03-03T14:18:38.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f9673fff700 1 -- 172.21.15.102:0/2623826579 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c0072f0 con 0x7f968c0cea00 2023-03-03T14:18:38.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.161+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c153280 con 0x7f968c0cea00 2023-03-03T14:18:38.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.162+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c0cfa30 con 0x7f968c0cea00 2023-03-03T14:18:38.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.163+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968c0cfd10 con 0x7f968c0cea00 2023-03-03T14:18:38.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.164+0000 7f9673fff700 1 -- 172.21.15.102:0/2623826579 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f967c019030 con 0x7f968c0cea00 2023-03-03T14:18:38.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.165+0000 7f9673fff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96780682f0 0x7f967806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:38.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.165+0000 7f9673fff700 1 -- 172.21.15.102:0/2623826579 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f967c0858e0 con 0x7f968c0cea00 2023-03-03T14:18:38.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.166+0000 7f968affd700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96780682f0 0x7f967806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:38.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.166+0000 7f968affd700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96780682f0 0x7f967806a7a0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f968c15af70 tx=0x7f9680007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.168+0000 7f9673fff700 1 -- 172.21.15.102:0/2623826579 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f967c051fe0 con 0x7f968c0cea00 2023-03-03T14:18:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.401+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f968c04e6e0 con 0x7f968c0cea00 2023-03-03T14:18:38.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.402+0000 7f9673fff700 1 -- 172.21.15.102:0/2623826579 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f967c017020 con 0x7f968c0cea00 2023-03-03T14:18:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:18:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:18:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96780682f0 msgr2=0x7f967806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96780682f0 0x7f967806a7a0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f968c15af70 tx=0x7f9680007dc0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f968c0cea00 msgr2=0x7f968c1e5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f968c0cea00 0x7f968c1e5220 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c530 tx=0x7f967c00c840 comp rx=0 tx=0).stop 2023-03-03T14:18:38.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 shutdown_connections 2023-03-03T14:18:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c15c870 0x7f968c14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f968c0cea00 0x7f968c1e5220 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96780682f0 0x7f967806a7a0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 --2- 172.21.15.102:0/2623826579 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f968c152eb0 0x7f968c153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 >> 172.21.15.102:0/2623826579 conn(0x7f968c066c40 msgr2=0x7f968c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 shutdown_connections 2023-03-03T14:18:38.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:18:38.405+0000 7f9691bff700 1 -- 172.21.15.102:0/2623826579 wait complete. 2023-03-03T14:18:38.410 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:18:38.435 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T14:19:07.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:07.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:22.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:19:22.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.699+0000 7f51db1ff700 1 Processor -- start 2023-03-03T14:19:22.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.699+0000 7f51db1ff700 1 -- start start 2023-03-03T14:19:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.700+0000 7f51db1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d40ca130 0x7f51d415c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.700+0000 7f51db1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f51d40c8eb0 0x7f51d415c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.700+0000 7f51db1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40c9590 con 0x7f51d4152e50 2023-03-03T14:19:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51db1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d4159880 con 0x7f51d40c8eb0 2023-03-03T14:19:22.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51db1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40c7ae0 con 0x7f51d40ca130 2023-03-03T14:19:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d979c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f51d40c8eb0 0x7f51d415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d979c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f51d40c8eb0 0x7f51d415c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42092/0 (socket says 172.21.15.102:42092) 2023-03-03T14:19:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d979c700 1 -- 172.21.15.102:0/1563931286 learned_addr learned my addr 172.21.15.102:0/1563931286 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:19:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d3fff700 1 --2- 172.21.15.102:0/1563931286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d40ca130 0x7f51d415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:22.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d8f9b700 1 --1- 172.21.15.102:0/1563931286 >> v1:172.21.15.172:6789/0 conn(0x7f51d4152e50 0x7f51d40c9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50310/0 (socket says 172.21.15.102:50310) 2023-03-03T14:19:22.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d979c700 1 -- 172.21.15.102:0/1563931286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d40ca130 msgr2=0x7f51d415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:19:22.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d979c700 1 --2- 172.21.15.102:0/1563931286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d40ca130 0x7f51d415c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d979c700 1 -- 172.21.15.102:0/1563931286 >> v1:172.21.15.172:6789/0 conn(0x7f51d4152e50 legacy=0x7f51d40c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:22.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d979c700 1 -- 172.21.15.102:0/1563931286 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40c7dc0 con 0x7f51d40c8eb0 2023-03-03T14:19:22.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.701+0000 7f51d37fe700 1 -- 172.21.15.102:0/1563931286 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816551724 0 0) 0x7f51d40c9590 con 0x7f51d4152e50 2023-03-03T14:19:22.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.702+0000 7f51d979c700 1 --2- 172.21.15.102:0/1563931286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d415c790 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009e50 tx=0x7f51cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213afa5ac929530b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:19:22.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.702+0000 7f51d37fe700 1 -- 172.21.15.102:0/1563931286 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc00e070 con 0x7f51d40c8eb0 2023-03-03T14:19:22.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.702+0000 7f51d37fe700 1 -- 172.21.15.102:0/1563931286 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc00b910 con 0x7f51d40c8eb0 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.702+0000 7f51d37fe700 1 -- 172.21.15.102:0/1563931286 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc00bcc0 con 0x7f51d40c8eb0 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 -- 172.21.15.102:0/1563931286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 msgr2=0x7f51d415c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1563931286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d415c790 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009e50 tx=0x7f51cc00b330 comp rx=0 tx=0).stop 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 -- 172.21.15.102:0/1563931286 shutdown_connections 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1563931286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d415c790 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1563931286 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d40ca130 0x7f51d415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 -- 172.21.15.102:0/1563931286 >> 172.21.15.102:0/1563931286 conn(0x7f51d40704c0 msgr2=0x7f51d40708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:19:22.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 -- 172.21.15.102:0/1563931286 shutdown_connections 2023-03-03T14:19:22.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.703+0000 7f51db1ff700 1 -- 172.21.15.102:0/1563931286 wait complete. 2023-03-03T14:19:22.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.705+0000 7f51db1ff700 1 Processor -- start 2023-03-03T14:19:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.705+0000 7f51db1ff700 1 -- start start 2023-03-03T14:19:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51db1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d4157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51db1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d40ca130 0x7f51d4157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51db1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d4152e50 0x7f51d41f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51db1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d4157000 con 0x7f51d40c8eb0 2023-03-03T14:19:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51db1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d414faa0 con 0x7f51d40ca130 2023-03-03T14:19:22.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51db1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d414fd80 con 0x7f51d4152e50 2023-03-03T14:19:22.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51d8f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d4157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:22.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51d8f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d4157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42098/0 (socket says 172.21.15.102:42098) 2023-03-03T14:19:22.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.706+0000 7f51d8f9b700 1 -- 172.21.15.102:0/1617425759 learned_addr learned my addr 172.21.15.102:0/1617425759 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:19:22.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d3fff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d40ca130 0x7f51d4157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:22.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d979c700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d4152e50 0x7f51d41f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:22.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d8f9b700 1 -- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d4152e50 msgr2=0x7f51d41f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:22.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d8f9b700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d4152e50 0x7f51d41f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d8f9b700 1 -- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d40ca130 msgr2=0x7f51d4157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:22.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d8f9b700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d40ca130 0x7f51d4157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d8f9b700 1 -- 172.21.15.102:0/1617425759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc009b30 con 0x7f51d40c8eb0 2023-03-03T14:19:22.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.707+0000 7f51d8f9b700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d4157420 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c550 tx=0x7f51c800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:19:22.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.708+0000 7f51d17fa700 1 -- 172.21.15.102:0/1617425759 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8004c80 con 0x7f51d40c8eb0 2023-03-03T14:19:22.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.708+0000 7f51d17fa700 1 -- 172.21.15.102:0/1617425759 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8004e00 con 0x7f51d40c8eb0 2023-03-03T14:19:22.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.708+0000 7f51d17fa700 1 -- 172.21.15.102:0/1617425759 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8007310 con 0x7f51d40c8eb0 2023-03-03T14:19:22.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.708+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d4159b60 con 0x7f51d40c8eb0 2023-03-03T14:19:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.708+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d40c80a0 con 0x7f51d40c8eb0 2023-03-03T14:19:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.709+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d40c7690 con 0x7f51d40c8eb0 2023-03-03T14:19:22.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.710+0000 7f51d17fa700 1 -- 172.21.15.102:0/1617425759 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f51c801a020 con 0x7f51d40c8eb0 2023-03-03T14:19:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.711+0000 7f51d17fa700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51b80682f0 0x7f51b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.711+0000 7f51d17fa700 1 -- 172.21.15.102:0/1617425759 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f51c8085870 con 0x7f51d40c8eb0 2023-03-03T14:19:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.712+0000 7f51d3fff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51b80682f0 0x7f51b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:22.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.712+0000 7f51d3fff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51b80682f0 0x7f51b806a7a0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f51d40b2c70 tx=0x7f51c0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:19:22.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.716+0000 7f51d17fa700 1 -- 172.21.15.102:0/1617425759 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f51c8051f80 con 0x7f51d40c8eb0 2023-03-03T14:19:22.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.953+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51d404e6e0 con 0x7f51d40c8eb0 2023-03-03T14:19:22.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.954+0000 7f51d17fa700 1 -- 172.21.15.102:0/1617425759 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f51c80555a0 con 0x7f51d40c8eb0 2023-03-03T14:19:22.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:19:22.955 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:19:22.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51b80682f0 msgr2=0x7f51b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:22.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51b80682f0 0x7f51b806a7a0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f51d40b2c70 tx=0x7f51c0009210 comp rx=0 tx=0).stop 2023-03-03T14:19:22.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 msgr2=0x7f51d4157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:22.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d4157420 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c550 tx=0x7f51c800c860 comp rx=0 tx=0).stop 2023-03-03T14:19:22.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 shutdown_connections 2023-03-03T14:19:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51b80682f0 0x7f51b806a7a0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51d4152e50 0x7f51d41f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.956+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d40c8eb0 0x7f51d4157420 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.957+0000 7f51db1ff700 1 --2- 172.21.15.102:0/1617425759 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d40ca130 0x7f51d4157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.957+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 >> 172.21.15.102:0/1617425759 conn(0x7f51d40704c0 msgr2=0x7f51d406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:19:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.957+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 shutdown_connections 2023-03-03T14:19:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:19:22.957+0000 7f51db1ff700 1 -- 172.21.15.102:0/1617425759 wait complete. 2023-03-03T14:19:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:19:22.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2023-03-03T14:19:38.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:38.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T14:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:20:08.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:08.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:17.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:20:18.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.234+0000 7f4348b9d700 1 Processor -- start 2023-03-03T14:20:18.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.234+0000 7f4348b9d700 1 -- start start 2023-03-03T14:20:18.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.236+0000 7f4348b9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f43440ced00 0x7f43440c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:18.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.236+0000 7f4348b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440b3970 0x7f43440c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:18.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.236+0000 7f4348b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43440c3d20 con 0x7f43440cf0d0 2023-03-03T14:20:18.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.236+0000 7f4348b9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43440c4930 con 0x7f43440ced00 2023-03-03T14:20:18.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.236+0000 7f4348b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434415cba0 con 0x7f43440b3970 2023-03-03T14:20:18.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.236+0000 7f43437fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f43440ced00 0x7f43440c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:18.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f4342ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440b3970 0x7f43440c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:18.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f4343fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f43440cf0d0 0x7f4344158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54274/0 (socket says 172.21.15.102:54274) 2023-03-03T14:20:18.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f4343fff700 1 -- 172.21.15.102:0/3161085569 learned_addr learned my addr 172.21.15.102:0/3161085569 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:20:18.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f43437fe700 1 -- 172.21.15.102:0/3161085569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440b3970 msgr2=0x7f43440c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:20:18.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f43437fe700 1 --2- 172.21.15.102:0/3161085569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440b3970 0x7f43440c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f43437fe700 1 -- 172.21.15.102:0/3161085569 >> v1:172.21.15.172:6789/0 conn(0x7f43440cf0d0 legacy=0x7f4344158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:18.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f43437fe700 1 -- 172.21.15.102:0/3161085569 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434415ce80 con 0x7f43440ced00 2023-03-03T14:20:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.237+0000 7f43437fe700 1 --2- 172.21.15.102:0/3161085569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440ced00 0x7f43440c5a90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f4334009840 tx=0x7f4334009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60368f27bb3c933 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f43427fc700 1 -- 172.21.15.102:0/3161085569 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334004230 con 0x7f43440ced00 2023-03-03T14:20:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f43427fc700 1 -- 172.21.15.102:0/3161085569 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4334004390 con 0x7f43440ced00 2023-03-03T14:20:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f43427fc700 1 -- 172.21.15.102:0/3161085569 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43340036a0 con 0x7f43440ced00 2023-03-03T14:20:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 -- 172.21.15.102:0/3161085569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440ced00 msgr2=0x7f43440c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 --2- 172.21.15.102:0/3161085569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440ced00 0x7f43440c5a90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f4334009840 tx=0x7f4334009b50 comp rx=0 tx=0).stop 2023-03-03T14:20:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 -- 172.21.15.102:0/3161085569 shutdown_connections 2023-03-03T14:20:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 --2- 172.21.15.102:0/3161085569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440ced00 0x7f43440c5a90 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 --2- 172.21.15.102:0/3161085569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440b3970 0x7f43440c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 -- 172.21.15.102:0/3161085569 >> 172.21.15.102:0/3161085569 conn(0x7f434408b1c0 msgr2=0x7f4344071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 -- 172.21.15.102:0/3161085569 shutdown_connections 2023-03-03T14:20:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.238+0000 7f4348b9d700 1 -- 172.21.15.102:0/3161085569 wait complete. 2023-03-03T14:20:18.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.240+0000 7f4348b9d700 1 Processor -- start 2023-03-03T14:20:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.241+0000 7f4348b9d700 1 -- start start 2023-03-03T14:20:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.241+0000 7f4348b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440b3970 0x7f434415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f4348b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43440ced00 0x7f434414d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f4348b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440cf0d0 0x7f434414dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f4348b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434414d240 con 0x7f43440b3970 2023-03-03T14:20:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f4348b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43440c9bd0 con 0x7f43440ced00 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f4348b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4344150490 con 0x7f43440cf0d0 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440b3970 0x7f434415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440b3970 0x7f434415a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57220/0 (socket says 172.21.15.102:57220) 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 -- 172.21.15.102:0/1729738728 learned_addr learned my addr 172.21.15.102:0/1729738728 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f4342ffd700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43440ced00 0x7f434414d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f4343fff700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440cf0d0 0x7f434414dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 -- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440cf0d0 msgr2=0x7f434414dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440cf0d0 0x7f434414dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 -- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43440ced00 msgr2=0x7f434414d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:18.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43440ced00 0x7f434414d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.242+0000 7f43437fe700 1 -- 172.21.15.102:0/1729738728 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340094a0 con 0x7f43440b3970 2023-03-03T14:20:18.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.243+0000 7f43437fe700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440b3970 0x7f434415a3b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f43340090b0 tx=0x7f4334004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:18.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.243+0000 7f432bfff700 1 -- 172.21.15.102:0/1729738728 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43340045d0 con 0x7f43440b3970 2023-03-03T14:20:18.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.243+0000 7f432bfff700 1 -- 172.21.15.102:0/1729738728 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433401f070 con 0x7f43440b3970 2023-03-03T14:20:18.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.243+0000 7f432bfff700 1 -- 172.21.15.102:0/1729738728 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334011870 con 0x7f43440b3970 2023-03-03T14:20:18.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.243+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434406c200 con 0x7f43440b3970 2023-03-03T14:20:18.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.243+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434406bf20 con 0x7f43440b3970 2023-03-03T14:20:18.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.244+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43440cf540 con 0x7f43440b3970 2023-03-03T14:20:18.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.245+0000 7f432bfff700 1 -- 172.21.15.102:0/1729738728 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4334017020 con 0x7f43440b3970 2023-03-03T14:20:18.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.247+0000 7f432bfff700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43300682f0 0x7f433006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:18.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.247+0000 7f432bfff700 1 -- 172.21.15.102:0/1729738728 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4334021070 con 0x7f43440b3970 2023-03-03T14:20:18.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.247+0000 7f4342ffd700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43300682f0 0x7f433006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:18.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.248+0000 7f4342ffd700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43300682f0 0x7f433006a7a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f43440cd2e0 tx=0x7f4338009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:18.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.251+0000 7f432bfff700 1 -- 172.21.15.102:0/1729738728 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4334053e90 con 0x7f43440b3970 2023-03-03T14:20:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.486+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f434404e6e0 con 0x7f43440b3970 2023-03-03T14:20:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.486+0000 7f432bfff700 1 -- 172.21.15.102:0/1729738728 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f43340574b0 con 0x7f43440b3970 2023-03-03T14:20:18.489 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:20:18.489 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:20:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43300682f0 msgr2=0x7f433006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43300682f0 0x7f433006a7a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f43440cd2e0 tx=0x7f4338009210 comp rx=0 tx=0).stop 2023-03-03T14:20:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440b3970 msgr2=0x7f434415a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440b3970 0x7f434415a3b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f43340090b0 tx=0x7f4334004750 comp rx=0 tx=0).stop 2023-03-03T14:20:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 shutdown_connections 2023-03-03T14:20:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f43300682f0 0x7f433006a7a0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43440cf0d0 0x7f434414dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43440b3970 0x7f434415a3b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 --2- 172.21.15.102:0/1729738728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43440ced00 0x7f434414d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 >> 172.21.15.102:0/1729738728 conn(0x7f434408b1c0 msgr2=0x7f43440b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 shutdown_connections 2023-03-03T14:20:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:20:18.489+0000 7f4348b9d700 1 -- 172.21.15.102:0/1729738728 wait complete. 2023-03-03T14:20:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:20:18.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 106 secs before thrashing 2023-03-03T14:20:38.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:38.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:08.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:08.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:38.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:38.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:04.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:22:04.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.769+0000 7f02eb19d700 1 Processor -- start 2023-03-03T14:22:04.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.769+0000 7f02eb19d700 1 -- start start 2023-03-03T14:22:04.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.770+0000 7f02eb19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f02e40ced00 0x7f02e40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:04.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02eb19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f02e40c41b0 0x7f02e40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:04.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02eb19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40c3d20 con 0x7f02e40b3970 2023-03-03T14:22:04.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02eb19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e40c4930 con 0x7f02e40ced00 2023-03-03T14:22:04.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02eb19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e40cf610 con 0x7f02e40c41b0 2023-03-03T14:22:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02ea19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f02e40ced00 0x7f02e40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02ea99c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f02e40c41b0 0x7f02e40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02ea19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f02e40ced00 0x7f02e40c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53974/0 (socket says 172.21.15.102:53974) 2023-03-03T14:22:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02ea19b700 1 -- 172.21.15.102:0/2016492680 learned_addr learned my addr 172.21.15.102:0/2016492680 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:22:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02e999a700 1 --1- 172.21.15.102:0/2016492680 >> v1:172.21.15.172:6789/0 conn(0x7f02e40b3970 0x7f02e4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56666/0 (socket says 172.21.15.102:56666) 2023-03-03T14:22:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.771+0000 7f02e9199700 1 -- 172.21.15.102:0/2016492680 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217061406 0 0) 0x7f02e40c3d20 con 0x7f02e40b3970 2023-03-03T14:22:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02e9199700 1 -- 172.21.15.102:0/2016492680 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d00035b0 con 0x7f02e40b3970 2023-03-03T14:22:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02ea99c700 1 -- 172.21.15.102:0/2016492680 >> v2:172.21.15.102:3300/0 conn(0x7f02e40ced00 msgr2=0x7f02e40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:22:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02ea99c700 1 --2- 172.21.15.102:0/2016492680 >> v2:172.21.15.102:3300/0 conn(0x7f02e40ced00 0x7f02e40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02ea99c700 1 -- 172.21.15.102:0/2016492680 >> v1:172.21.15.172:6789/0 conn(0x7f02e40b3970 legacy=0x7f02e4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02ea99c700 1 -- 172.21.15.102:0/2016492680 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e415cba0 con 0x7f02e40c41b0 2023-03-03T14:22:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02ea19b700 1 --2- 172.21.15.102:0/2016492680 >> v2:172.21.15.102:3300/0 conn(0x7f02e40ced00 0x7f02e40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:22:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02ea99c700 1 --2- 172.21.15.102:0/2016492680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e40cf0d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f02e40ca080 tx=0x7f02dc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a00c84b0ab9fba8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02e9199700 1 -- 172.21.15.102:0/2016492680 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00e070 con 0x7f02e40c41b0 2023-03-03T14:22:04.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02e9199700 1 -- 172.21.15.102:0/2016492680 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc00bb20 con 0x7f02e40c41b0 2023-03-03T14:22:04.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.772+0000 7f02e9199700 1 -- 172.21.15.102:0/2016492680 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc012430 con 0x7f02e40c41b0 2023-03-03T14:22:04.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 -- 172.21.15.102:0/2016492680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 msgr2=0x7f02e40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:04.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 --2- 172.21.15.102:0/2016492680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e40cf0d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f02e40ca080 tx=0x7f02dc00b330 comp rx=0 tx=0).stop 2023-03-03T14:22:04.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 -- 172.21.15.102:0/2016492680 shutdown_connections 2023-03-03T14:22:04.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 --2- 172.21.15.102:0/2016492680 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e40cf0d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:04.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 --2- 172.21.15.102:0/2016492680 >> v2:172.21.15.102:3300/0 conn(0x7f02e40ced00 0x7f02e40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:04.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 -- 172.21.15.102:0/2016492680 >> 172.21.15.102:0/2016492680 conn(0x7f02e408b1c0 msgr2=0x7f02e4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:04.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 -- 172.21.15.102:0/2016492680 shutdown_connections 2023-03-03T14:22:04.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.773+0000 7f02eb19d700 1 -- 172.21.15.102:0/2016492680 wait complete. 2023-03-03T14:22:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.775+0000 7f02eb19d700 1 Processor -- start 2023-03-03T14:22:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.775+0000 7f02eb19d700 1 -- start start 2023-03-03T14:22:04.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.776+0000 7f02eb19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02e40b3970 0x7f02e414de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:04.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.776+0000 7f02eb19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e4148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:04.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.776+0000 7f02eb19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02e40ced00 0x7f02e4148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:04.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.776+0000 7f02eb19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e40cfc50 con 0x7f02e40ced00 2023-03-03T14:22:04.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.776+0000 7f02eb19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e41e2030 con 0x7f02e40c41b0 2023-03-03T14:22:04.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.776+0000 7f02eb19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e41eeeb0 con 0x7f02e40b3970 2023-03-03T14:22:04.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02e40b3970 0x7f02e414de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:04.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02e999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e4148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:04.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02e40b3970 0x7f02e414de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:56502/0 (socket says 172.21.15.102:56502) 2023-03-03T14:22:04.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02e999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e4148840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39648/0 (socket says 172.21.15.102:39648) 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 -- 172.21.15.102:0/1951010521 learned_addr learned my addr 172.21.15.102:0/1951010521 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea99c700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02e40ced00 0x7f02e4148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 -- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 msgr2=0x7f02e4148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e4148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 -- 172.21.15.102:0/1951010521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02e40ced00 msgr2=0x7f02e4148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02e40ced00 0x7f02e4148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 -- 172.21.15.102:0/1951010521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0009550 con 0x7f02e40b3970 2023-03-03T14:22:04.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea99c700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02e40ced00 0x7f02e4148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:22:04.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02ea19b700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02e40b3970 0x7f02e414de80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f02e000ebe0 tx=0x7f02e000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:04.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.777+0000 7f02daffd700 1 -- 172.21.15.102:0/1951010521 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e000c7f0 con 0x7f02e40b3970 2023-03-03T14:22:04.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.778+0000 7f02daffd700 1 -- 172.21.15.102:0/1951010521 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e000c950 con 0x7f02e40b3970 2023-03-03T14:22:04.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.778+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc009a70 con 0x7f02e40b3970 2023-03-03T14:22:04.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.778+0000 7f02daffd700 1 -- 172.21.15.102:0/1951010521 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e000ccc0 con 0x7f02e40b3970 2023-03-03T14:22:04.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.778+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e406bf20 con 0x7f02e40b3970 2023-03-03T14:22:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.779+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e415ce80 con 0x7f02e40b3970 2023-03-03T14:22:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.780+0000 7f02daffd700 1 -- 172.21.15.102:0/1951010521 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f02e0023070 con 0x7f02e40b3970 2023-03-03T14:22:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.781+0000 7f02daffd700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02c80682f0 0x7f02c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.782+0000 7f02daffd700 1 -- 172.21.15.102:0/1951010521 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f02e0086170 con 0x7f02e40b3970 2023-03-03T14:22:04.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.782+0000 7f02e999a700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02c80682f0 0x7f02c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:04.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.782+0000 7f02e999a700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02c80682f0 0x7f02c806a7a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f02e40ca640 tx=0x7f02d4007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:04.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:04.784+0000 7f02daffd700 1 -- 172.21.15.102:0/1951010521 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f02e0052870 con 0x7f02e40b3970 2023-03-03T14:22:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.017+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e406c200 con 0x7f02e40b3970 2023-03-03T14:22:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.019+0000 7f02daffd700 1 -- 172.21.15.102:0/1951010521 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f02e0055e90 con 0x7f02e40b3970 2023-03-03T14:22:05.020 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:22:05.021 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:22:05.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02c80682f0 msgr2=0x7f02c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02c80682f0 0x7f02c806a7a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f02e40ca640 tx=0x7f02d4007dc0 comp rx=0 tx=0).stop 2023-03-03T14:22:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02e40b3970 msgr2=0x7f02e414de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02e40b3970 0x7f02e414de80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f02e000ebe0 tx=0x7f02e000eef0 comp rx=0 tx=0).stop 2023-03-03T14:22:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 shutdown_connections 2023-03-03T14:22:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02c80682f0 0x7f02c806a7a0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02e40b3970 0x7f02e414de80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02e40ced00 0x7f02e4148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 --2- 172.21.15.102:0/1951010521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f02e40c41b0 0x7f02e4148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 >> 172.21.15.102:0/1951010521 conn(0x7f02e408b1c0 msgr2=0x7f02e40b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 shutdown_connections 2023-03-03T14:22:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:22:05.021+0000 7f02eb19d700 1 -- 172.21.15.102:0/1951010521 wait complete. 2023-03-03T14:22:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:22:05.050 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T14:22:09.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:09.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:39.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:39.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:07.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:23:07.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.315+0000 7fa1357ff700 1 Processor -- start 2023-03-03T14:23:07.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.315+0000 7fa1357ff700 1 -- start start 2023-03-03T14:23:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa1357ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa130153260 0x7fa1300ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa1357ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa1300c9cb0 0x7fa1300cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa1357ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1300cf940 con 0x7fa1300cac20 2023-03-03T14:23:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa1357ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1300cfe70 con 0x7fa130153260 2023-03-03T14:23:07.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa1357ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa130158d70 con 0x7fa1300c9cb0 2023-03-03T14:23:07.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa12effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa130153260 0x7fa1300ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa12e7fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa1300c9cb0 0x7fa1300cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa12f7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa1300cac20 0x7fa13015c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43866/0 (socket says 172.21.15.102:43866) 2023-03-03T14:23:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.317+0000 7fa12f7fe700 1 -- 172.21.15.102:0/2643117271 learned_addr learned my addr 172.21.15.102:0/2643117271 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:23:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12e7fc700 1 -- 172.21.15.102:0/2643117271 >> v2:172.21.15.102:3300/0 conn(0x7fa130153260 msgr2=0x7fa1300ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:23:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12e7fc700 1 --2- 172.21.15.102:0/2643117271 >> v2:172.21.15.102:3300/0 conn(0x7fa130153260 0x7fa1300ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12e7fc700 1 -- 172.21.15.102:0/2643117271 >> v1:172.21.15.102:6789/0 conn(0x7fa1300cac20 legacy=0x7fa13015c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12e7fc700 1 -- 172.21.15.102:0/2643117271 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1300cfb90 con 0x7fa1300c9cb0 2023-03-03T14:23:07.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12dffb700 1 -- 172.21.15.102:0/2643117271 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052190119 0 0) 0x7fa1300cf940 con 0x7fa1300cac20 2023-03-03T14:23:07.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12effd700 1 --2- 172.21.15.102:0/2643117271 >> v2:172.21.15.102:3300/0 conn(0x7fa130153260 0x7fa1300ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:23:07.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12e7fc700 1 --2- 172.21.15.102:0/2643117271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1300c9cb0 0x7fa1300cb880 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa13008cd80 tx=0x7fa124009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a162d9483bddf7d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:07.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12dffb700 1 -- 172.21.15.102:0/2643117271 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124011830 con 0x7fa1300c9cb0 2023-03-03T14:23:07.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12dffb700 1 -- 172.21.15.102:0/2643117271 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa124011990 con 0x7fa1300c9cb0 2023-03-03T14:23:07.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.318+0000 7fa12dffb700 1 -- 172.21.15.102:0/2643117271 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124011d20 con 0x7fa1300c9cb0 2023-03-03T14:23:07.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 -- 172.21.15.102:0/2643117271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1300c9cb0 msgr2=0x7fa1300cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:07.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 --2- 172.21.15.102:0/2643117271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1300c9cb0 0x7fa1300cb880 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa13008cd80 tx=0x7fa124009b20 comp rx=0 tx=0).stop 2023-03-03T14:23:07.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 -- 172.21.15.102:0/2643117271 shutdown_connections 2023-03-03T14:23:07.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 --2- 172.21.15.102:0/2643117271 >> v2:172.21.15.102:3300/0 conn(0x7fa130153260 0x7fa1300ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 --2- 172.21.15.102:0/2643117271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1300c9cb0 0x7fa1300cb880 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 -- 172.21.15.102:0/2643117271 >> 172.21.15.102:0/2643117271 conn(0x7fa130066c00 msgr2=0x7fa1300ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:07.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 -- 172.21.15.102:0/2643117271 shutdown_connections 2023-03-03T14:23:07.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.319+0000 7fa1357ff700 1 -- 172.21.15.102:0/2643117271 wait complete. 2023-03-03T14:23:07.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.321+0000 7fa1357ff700 1 Processor -- start 2023-03-03T14:23:07.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.321+0000 7fa1357ff700 1 -- start start 2023-03-03T14:23:07.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.322+0000 7fa1357ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1300cac20 0x7fa1300b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:07.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.322+0000 7fa1357ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa130153260 0x7fa1301e3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.322+0000 7fa1357ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa13014c670 0x7fa1301e3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.322+0000 7fa1357ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1301e4f80 con 0x7fa13014c670 2023-03-03T14:23:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.322+0000 7fa1357ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13014e1c0 con 0x7fa130153260 2023-03-03T14:23:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.322+0000 7fa1357ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13014e420 con 0x7fa1300cac20 2023-03-03T14:23:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1300cac20 0x7fa1300b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa13014c670 0x7fa1301e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12e7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa130153260 0x7fa1301e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1300cac20 0x7fa1300b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51376/0 (socket says 172.21.15.102:51376) 2023-03-03T14:23:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 -- 172.21.15.102:0/1330537974 learned_addr learned my addr 172.21.15.102:0/1330537974 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:23:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 -- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa130153260 msgr2=0x7fa1301e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa130153260 0x7fa1301e3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 -- 172.21.15.102:0/1330537974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa13014c670 msgr2=0x7fa1301e3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa13014c670 0x7fa1301e3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 -- 172.21.15.102:0/1330537974 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1200094a0 con 0x7fa1300cac20 2023-03-03T14:23:07.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12f7fe700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa13014c670 0x7fa1301e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:23:07.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.323+0000 7fa12effd700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1300cac20 0x7fa1300b4250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa12000ebe0 tx=0x7fa12000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.324+0000 7fa1177fe700 1 -- 172.21.15.102:0/1330537974 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12000c850 con 0x7fa1300cac20 2023-03-03T14:23:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.324+0000 7fa1177fe700 1 -- 172.21.15.102:0/1330537974 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12000c9b0 con 0x7fa1300cac20 2023-03-03T14:23:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.324+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1240094a0 con 0x7fa1300cac20 2023-03-03T14:23:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.324+0000 7fa1177fe700 1 -- 172.21.15.102:0/1330537974 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12000cd40 con 0x7fa1300cac20 2023-03-03T14:23:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.324+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa130159260 con 0x7fa1300cac20 2023-03-03T14:23:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.325+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa130158f80 con 0x7fa1300cac20 2023-03-03T14:23:07.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.326+0000 7fa1177fe700 1 -- 172.21.15.102:0/1330537974 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa120019030 con 0x7fa1300cac20 2023-03-03T14:23:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.328+0000 7fa1177fe700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa11c0682f0 0x7fa11c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.328+0000 7fa1177fe700 1 -- 172.21.15.102:0/1330537974 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa120085da0 con 0x7fa1300cac20 2023-03-03T14:23:07.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.328+0000 7fa12e7fc700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa11c0682f0 0x7fa11c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:07.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.329+0000 7fa12e7fc700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa11c0682f0 0x7fa11c06a7a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa1240090b0 tx=0x7fa12401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:07.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.332+0000 7fa1177fe700 1 -- 172.21.15.102:0/1330537974 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa120052420 con 0x7fa1300cac20 2023-03-03T14:23:07.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.565+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa130159540 con 0x7fa1300cac20 2023-03-03T14:23:07.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.566+0000 7fa1177fe700 1 -- 172.21.15.102:0/1330537974 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa120017090 con 0x7fa1300cac20 2023-03-03T14:23:07.567 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:23:07.567 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:23:07.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa11c0682f0 msgr2=0x7fa11c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:07.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa11c0682f0 0x7fa11c06a7a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa1240090b0 tx=0x7fa12401c040 comp rx=0 tx=0).stop 2023-03-03T14:23:07.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1300cac20 msgr2=0x7fa1300b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:07.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1300cac20 0x7fa1300b4250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa12000ebe0 tx=0x7fa12000eef0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 shutdown_connections 2023-03-03T14:23:07.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa11c0682f0 0x7fa11c06a7a0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1300cac20 0x7fa1300b4250 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa13014c670 0x7fa1301e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 --2- 172.21.15.102:0/1330537974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa130153260 0x7fa1301e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:07.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.568+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 >> 172.21.15.102:0/1330537974 conn(0x7fa1300c9cb0 msgr2=0x7fa1300b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:07.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.569+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 shutdown_connections 2023-03-03T14:23:07.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:23:07.569+0000 7fa1357ff700 1 -- 172.21.15.102:0/1330537974 wait complete. 2023-03-03T14:23:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:23:07.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2023-03-03T14:23:09.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:09.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:39.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:39.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:09.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:09.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:27.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:24:27.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.851+0000 7f15989fe700 1 Processor -- start 2023-03-03T14:24:27.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.851+0000 7f15989fe700 1 -- start start 2023-03-03T14:24:27.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f15989fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940b3970 0x7f15940c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:24:27.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f15989fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f15940c41b0 0x7f15940cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:24:27.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f15989fe700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940c3d20 con 0x7f15940ced00 2023-03-03T14:24:27.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f15989fe700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15940c4930 con 0x7f15940c41b0 2023-03-03T14:24:27.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f15989fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15940cf610 con 0x7f15940b3970 2023-03-03T14:24:27.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f159359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f15940c41b0 0x7f15940cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:24:27.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f1592d9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f15940ced00 0x7f1594158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57872/0 (socket says 172.21.15.102:57872) 2023-03-03T14:24:27.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f1592d9d700 1 -- 172.21.15.102:0/80948069 learned_addr learned my addr 172.21.15.102:0/80948069 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:24:27.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.853+0000 7f159259c700 1 --2- 172.21.15.102:0/80948069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940b3970 0x7f15940c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:24:27.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.854+0000 7f159359e700 1 -- 172.21.15.102:0/80948069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940b3970 msgr2=0x7f15940c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:24:27.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.854+0000 7f159359e700 1 --2- 172.21.15.102:0/80948069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940b3970 0x7f15940c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:27.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.854+0000 7f159359e700 1 -- 172.21.15.102:0/80948069 >> v1:172.21.15.102:6789/0 conn(0x7f15940ced00 legacy=0x7f1594158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:24:27.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.854+0000 7f159359e700 1 -- 172.21.15.102:0/80948069 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159415cba0 con 0x7f15940c41b0 2023-03-03T14:24:27.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.854+0000 7f159359e700 1 --2- 172.21.15.102:0/80948069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940c41b0 0x7f15940cf0d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f158c009dc0 tx=0x7f158c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3714138b33816f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:24:27.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f1591d9b700 1 -- 172.21.15.102:0/80948069 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c00e070 con 0x7f15940c41b0 2023-03-03T14:24:27.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f1591d9b700 1 -- 172.21.15.102:0/80948069 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c00bae0 con 0x7f15940c41b0 2023-03-03T14:24:27.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f1591d9b700 1 -- 172.21.15.102:0/80948069 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c00be90 con 0x7f15940c41b0 2023-03-03T14:24:27.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 -- 172.21.15.102:0/80948069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940c41b0 msgr2=0x7f15940cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:24:27.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 --2- 172.21.15.102:0/80948069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940c41b0 0x7f15940cf0d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f158c009dc0 tx=0x7f158c00b330 comp rx=0 tx=0).stop 2023-03-03T14:24:27.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 -- 172.21.15.102:0/80948069 shutdown_connections 2023-03-03T14:24:27.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 --2- 172.21.15.102:0/80948069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940c41b0 0x7f15940cf0d0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:27.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 --2- 172.21.15.102:0/80948069 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940b3970 0x7f15940c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:27.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 -- 172.21.15.102:0/80948069 >> 172.21.15.102:0/80948069 conn(0x7f159408b1c0 msgr2=0x7f1594071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:24:27.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 -- 172.21.15.102:0/80948069 shutdown_connections 2023-03-03T14:24:27.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.855+0000 7f15989fe700 1 -- 172.21.15.102:0/80948069 wait complete. 2023-03-03T14:24:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.857+0000 7f15989fe700 1 Processor -- start 2023-03-03T14:24:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.858+0000 7f15989fe700 1 -- start start 2023-03-03T14:24:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f15989fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15940b3970 0x7f15941efbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:24:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f15989fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940c41b0 0x7f15941e2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:24:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f15989fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ced00 0x7f15941e49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:24:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f15989fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15941e4520 con 0x7f15940ced00 2023-03-03T14:24:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f15989fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159415c3e0 con 0x7f15940b3970 2023-03-03T14:24:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f15989fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15941e3c50 con 0x7f15940c41b0 2023-03-03T14:24:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ced00 0x7f15941e49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:24:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ced00 0x7f15941e49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40926/0 (socket says 172.21.15.102:40926) 2023-03-03T14:24:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940c41b0 0x7f15941e2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:24:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 -- 172.21.15.102:0/2580754709 learned_addr learned my addr 172.21.15.102:0/2580754709 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:24:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f1592d9d700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15940b3970 0x7f15941efbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:24:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 -- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940c41b0 msgr2=0x7f15941e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:24:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940c41b0 0x7f15941e2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 -- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15940b3970 msgr2=0x7f15941efbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:24:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15940b3970 0x7f15941efbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.859+0000 7f159359e700 1 -- 172.21.15.102:0/2580754709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c009a70 con 0x7f15940ced00 2023-03-03T14:24:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.860+0000 7f159359e700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ced00 0x7f15941e49b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f158c013000 tx=0x7f158c004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:24:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.860+0000 7f157b7fe700 1 -- 172.21.15.102:0/2580754709 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c0045d0 con 0x7f15940ced00 2023-03-03T14:24:27.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.860+0000 7f157b7fe700 1 -- 172.21.15.102:0/2580754709 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c020070 con 0x7f15940ced00 2023-03-03T14:24:27.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.860+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159406bf20 con 0x7f15940ced00 2023-03-03T14:24:27.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.860+0000 7f157b7fe700 1 -- 172.21.15.102:0/2580754709 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c004970 con 0x7f15940ced00 2023-03-03T14:24:27.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.860+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159415ce80 con 0x7f15940ced00 2023-03-03T14:24:27.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.861+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159406c200 con 0x7f15940ced00 2023-03-03T14:24:27.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.863+0000 7f157b7fe700 1 -- 172.21.15.102:0/2580754709 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f158c012430 con 0x7f15940ced00 2023-03-03T14:24:27.865 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.864+0000 7f157b7fe700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1580068320 0x7f158006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:24:27.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.865+0000 7f157b7fe700 1 -- 172.21.15.102:0/2580754709 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f158c022070 con 0x7f15940ced00 2023-03-03T14:24:27.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.865+0000 7f1592d9d700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1580068320 0x7f158006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:24:27.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.865+0000 7f1592d9d700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1580068320 0x7f158006a7d0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1584005910 tx=0x7f1584009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:24:27.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:27.869+0000 7f157b7fe700 1 -- 172.21.15.102:0/2580754709 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f158c054760 con 0x7f15940ced00 2023-03-03T14:24:28.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.105+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15941e46d0 con 0x7f15940ced00 2023-03-03T14:24:28.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.106+0000 7f157b7fe700 1 -- 172.21.15.102:0/2580754709 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f15941e46d0 con 0x7f15940ced00 2023-03-03T14:24:28.107 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:24:28.107 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:24:28.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1580068320 msgr2=0x7f158006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:24:28.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1580068320 0x7f158006a7d0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1584005910 tx=0x7f1584009040 comp rx=0 tx=0).stop 2023-03-03T14:24:28.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ced00 msgr2=0x7f15941e49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:24:28.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ced00 0x7f15941e49b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f158c013000 tx=0x7f158c004b20 comp rx=0 tx=0).stop 2023-03-03T14:24:28.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 shutdown_connections 2023-03-03T14:24:28.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ced00 0x7f15941e49b0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:28.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f15940b3970 0x7f15941efbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:28.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1580068320 0x7f158006a7d0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:28.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 --2- 172.21.15.102:0/2580754709 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f15940c41b0 0x7f15941e2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:24:28.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.108+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 >> 172.21.15.102:0/2580754709 conn(0x7f159408b1c0 msgr2=0x7f1594071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:24:28.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.109+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 shutdown_connections 2023-03-03T14:24:28.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:24:28.109+0000 7f15989fe700 1 -- 172.21.15.102:0/2580754709 wait complete. 2023-03-03T14:24:28.112 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:24:28.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-03-03T14:24:39.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:39.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:01.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:25:01.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.385+0000 7efd0c9fe700 1 Processor -- start 2023-03-03T14:25:01.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.385+0000 7efd0c9fe700 1 -- start start 2023-03-03T14:25:01.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0c9fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efd080ced00 0x7efd080c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:01.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0c9fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080b3970 0x7efd080c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:01.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0c9fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd080cf0d0 0x7efd08158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:01.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0c9fe700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080c4930 con 0x7efd080ced00 2023-03-03T14:25:01.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0c9fe700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080c3d20 con 0x7efd080cf0d0 2023-03-03T14:25:01.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0c9fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0815cba0 con 0x7efd080b3970 2023-03-03T14:25:01.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0759e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd080cf0d0 0x7efd08158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:01.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd06d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efd080ced00 0x7efd080c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.387+0000 7efd0659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080b3970 0x7efd080c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd0759e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd080cf0d0 0x7efd08158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50518/0 (socket says 172.21.15.102:50518) 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd0759e700 1 -- 172.21.15.102:0/1327366275 learned_addr learned my addr 172.21.15.102:0/1327366275 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd06d9d700 1 -- 172.21.15.102:0/1327366275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080b3970 msgr2=0x7efd080c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd06d9d700 1 --2- 172.21.15.102:0/1327366275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080b3970 0x7efd080c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd06d9d700 1 -- 172.21.15.102:0/1327366275 >> v2:172.21.15.172:3300/0 conn(0x7efd080cf0d0 msgr2=0x7efd08158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd06d9d700 1 --2- 172.21.15.102:0/1327366275 >> v2:172.21.15.172:3300/0 conn(0x7efd080cf0d0 0x7efd08158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd06d9d700 1 -- 172.21.15.102:0/1327366275 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0815ce80 con 0x7efd080ced00 2023-03-03T14:25:01.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd0759e700 1 --2- 172.21.15.102:0/1327366275 >> v2:172.21.15.172:3300/0 conn(0x7efd080cf0d0 0x7efd08158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:25:01.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.388+0000 7efd06d9d700 1 --2- 172.21.15.102:0/1327366275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c5a90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7efd080b3920 tx=0x7efcf8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b33098cdaa53d7e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:01.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd05d9b700 1 -- 172.21.15.102:0/1327366275 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8011830 con 0x7efd080ced00 2023-03-03T14:25:01.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd05d9b700 1 -- 172.21.15.102:0/1327366275 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf8011990 con 0x7efd080ced00 2023-03-03T14:25:01.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd05d9b700 1 -- 172.21.15.102:0/1327366275 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8011d20 con 0x7efd080ced00 2023-03-03T14:25:01.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 -- 172.21.15.102:0/1327366275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 msgr2=0x7efd080c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:01.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/1327366275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c5a90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7efd080b3920 tx=0x7efcf8009b20 comp rx=0 tx=0).stop 2023-03-03T14:25:01.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 -- 172.21.15.102:0/1327366275 shutdown_connections 2023-03-03T14:25:01.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/1327366275 >> v2:172.21.15.172:3300/0 conn(0x7efd080cf0d0 0x7efd08158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/1327366275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c5a90 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/1327366275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080b3970 0x7efd080c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 -- 172.21.15.102:0/1327366275 >> 172.21.15.102:0/1327366275 conn(0x7efd0808b1c0 msgr2=0x7efd08071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:01.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.389+0000 7efd0c9fe700 1 -- 172.21.15.102:0/1327366275 shutdown_connections 2023-03-03T14:25:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.390+0000 7efd0c9fe700 1 -- 172.21.15.102:0/1327366275 wait complete. 2023-03-03T14:25:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.391+0000 7efd0c9fe700 1 Processor -- start 2023-03-03T14:25:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.392+0000 7efd0c9fe700 1 -- start start 2023-03-03T14:25:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.392+0000 7efd0c9fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd080b3970 0x7efd080c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0c9fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0c9fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080cf0d0 0x7efd080c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0c9fe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd08154f10 con 0x7efd080ced00 2023-03-03T14:25:01.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0c9fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd080c6a80 con 0x7efd080b3970 2023-03-03T14:25:01.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0c9fe700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd08073c80 con 0x7efd080cf0d0 2023-03-03T14:25:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53314/0 (socket says 172.21.15.102:53314) 2023-03-03T14:25:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 -- 172.21.15.102:0/818159994 learned_addr learned my addr 172.21.15.102:0/818159994 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:25:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd06d9d700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd080b3970 0x7efd080c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0759e700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080cf0d0 0x7efd080c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 -- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080cf0d0 msgr2=0x7efd080c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080cf0d0 0x7efd080c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 -- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd080b3970 msgr2=0x7efd080c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd080b3970 0x7efd080c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.393+0000 7efd0659c700 1 -- 172.21.15.102:0/818159994 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00009b30 con 0x7efd080ced00 2023-03-03T14:25:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.394+0000 7efd0659c700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c7b30 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e840 tx=0x7efcfc00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.394+0000 7efcef7fe700 1 -- 172.21.15.102:0/818159994 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc013680 con 0x7efd080ced00 2023-03-03T14:25:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.394+0000 7efcef7fe700 1 -- 172.21.15.102:0/818159994 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc0137e0 con 0x7efd080ced00 2023-03-03T14:25:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.394+0000 7efcef7fe700 1 -- 172.21.15.102:0/818159994 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc013b70 con 0x7efd080ced00 2023-03-03T14:25:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.394+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf80094a0 con 0x7efd080ced00 2023-03-03T14:25:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.394+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0806c200 con 0x7efd080ced00 2023-03-03T14:25:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.395+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0806bf20 con 0x7efd080ced00 2023-03-03T14:25:01.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.396+0000 7efcef7fe700 1 -- 172.21.15.102:0/818159994 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efcfc019030 con 0x7efd080ced00 2023-03-03T14:25:01.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.398+0000 7efcef7fe700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efcf40682f0 0x7efcf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:01.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.398+0000 7efcef7fe700 1 -- 172.21.15.102:0/818159994 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7efcfc085a90 con 0x7efd080ced00 2023-03-03T14:25:01.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.398+0000 7efd06d9d700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efcf40682f0 0x7efcf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:01.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.399+0000 7efd06d9d700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efcf40682f0 0x7efcf406a7a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7efcf80090b0 tx=0x7efcf801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:01.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.402+0000 7efcef7fe700 1 -- 172.21.15.102:0/818159994 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efcfc0521a0 con 0x7efd080ced00 2023-03-03T14:25:01.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.639+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd080c85b0 con 0x7efd080ced00 2023-03-03T14:25:01.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.640+0000 7efcef7fe700 1 -- 172.21.15.102:0/818159994 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7efcfc0557c0 con 0x7efd080ced00 2023-03-03T14:25:01.643 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:25:01.644 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:25:01.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efcf40682f0 msgr2=0x7efcf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:01.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efcf40682f0 0x7efcf406a7a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7efcf80090b0 tx=0x7efcf801c040 comp rx=0 tx=0).stop 2023-03-03T14:25:01.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 msgr2=0x7efd080c7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:01.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c7b30 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e840 tx=0x7efcfc00eb50 comp rx=0 tx=0).stop 2023-03-03T14:25:01.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 shutdown_connections 2023-03-03T14:25:01.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efcf40682f0 0x7efcf406a7a0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efd080cf0d0 0x7efd080c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd080b3970 0x7efd080c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 --2- 172.21.15.102:0/818159994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd080ced00 0x7efd080c7b30 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:01.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 >> 172.21.15.102:0/818159994 conn(0x7efd0808b1c0 msgr2=0x7efd0806b220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:01.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 shutdown_connections 2023-03-03T14:25:01.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:25:01.642+0000 7efd0c9fe700 1 -- 172.21.15.102:0/818159994 wait complete. 2023-03-03T14:25:01.647 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:25:01.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 76 secs before thrashing 2023-03-03T14:25:10.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:10.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:40.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:40.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:10.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:10.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:17.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:26:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.922+0000 7f4afcbff700 1 Processor -- start 2023-03-03T14:26:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.923+0000 7f4afcbff700 1 -- start start 2023-03-03T14:26:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.924+0000 7f4afcbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4af80cd390 0x7f4af80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.924+0000 7f4afcbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af8153660 con 0x7f4af8153290 2023-03-03T14:26:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.924+0000 7f4afcbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af806c360 con 0x7f4af80cd8e0 2023-03-03T14:26:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.924+0000 7f4afcbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af815cea0 con 0x7f4af80cd390 2023-03-03T14:26:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.924+0000 7f4af659c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4af8153290 0x7f4af8158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33496/0 (socket says 172.21.15.102:33496) 2023-03-03T14:26:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.924+0000 7f4af659c700 1 -- 172.21.15.102:0/150794052 learned_addr learned my addr 172.21.15.102:0/150794052 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:26:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.924+0000 7f4af6d9d700 1 --2- 172.21.15.102:0/150794052 >> v2:172.21.15.172:3300/0 conn(0x7f4af80cd390 0x7f4af80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af5d9b700 1 -- 172.21.15.102:0/150794052 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462575562 0 0) 0x7f4af8153660 con 0x7f4af8153290 2023-03-03T14:26:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af5d9b700 1 -- 172.21.15.102:0/150794052 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae40035b0 con 0x7f4af8153290 2023-03-03T14:26:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af5d9b700 1 -- 172.21.15.102:0/150794052 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882764049 0 0) 0x7f4af806c360 con 0x7f4af80cd8e0 2023-03-03T14:26:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af5d9b700 1 -- 172.21.15.102:0/150794052 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af8153660 con 0x7f4af80cd8e0 2023-03-03T14:26:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af6d9d700 1 -- 172.21.15.102:0/150794052 >> v1:172.21.15.172:6789/0 conn(0x7f4af80cd8e0 legacy=0x7f4af80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af6d9d700 1 -- 172.21.15.102:0/150794052 >> v1:172.21.15.102:6789/0 conn(0x7f4af8153290 legacy=0x7f4af8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af6d9d700 1 -- 172.21.15.102:0/150794052 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af806c7f0 con 0x7f4af80cd390 2023-03-03T14:26:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.925+0000 7f4af6d9d700 1 --2- 172.21.15.102:0/150794052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af80cd390 0x7f4af80ce530 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009d90 tx=0x7f4af000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b214209dc68925f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:26:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4af5d9b700 1 -- 172.21.15.102:0/150794052 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af000e070 con 0x7f4af80cd390 2023-03-03T14:26:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4af5d9b700 1 -- 172.21.15.102:0/150794052 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af000bb20 con 0x7f4af80cd390 2023-03-03T14:26:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4af5d9b700 1 -- 172.21.15.102:0/150794052 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0012430 con 0x7f4af80cd390 2023-03-03T14:26:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4afcbff700 1 -- 172.21.15.102:0/150794052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af80cd390 msgr2=0x7f4af80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4afcbff700 1 --2- 172.21.15.102:0/150794052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af80cd390 0x7f4af80ce530 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009d90 tx=0x7f4af000b330 comp rx=0 tx=0).stop 2023-03-03T14:26:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4afcbff700 1 -- 172.21.15.102:0/150794052 shutdown_connections 2023-03-03T14:26:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4afcbff700 1 --2- 172.21.15.102:0/150794052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af80cd390 0x7f4af80ce530 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4afcbff700 1 -- 172.21.15.102:0/150794052 >> 172.21.15.102:0/150794052 conn(0x7f4af8065fb0 msgr2=0x7f4af80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:26:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.926+0000 7f4afcbff700 1 -- 172.21.15.102:0/150794052 shutdown_connections 2023-03-03T14:26:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.927+0000 7f4afcbff700 1 -- 172.21.15.102:0/150794052 wait complete. 2023-03-03T14:26:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.929+0000 7f4afcbff700 1 Processor -- start 2023-03-03T14:26:17.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.930+0000 7f4afcbff700 1 -- start start 2023-03-03T14:26:17.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4afcbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4af80cd390 0x7f4af80c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4afcbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4af80cd8e0 0x7f4af81ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4afcbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af8153290 0x7f4af80c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4afcbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af80c9220 con 0x7f4af80cd8e0 2023-03-03T14:26:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4afcbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af81f2930 con 0x7f4af8153290 2023-03-03T14:26:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4afcbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af814eef0 con 0x7f4af80cd390 2023-03-03T14:26:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4af6d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af8153290 0x7f4af80c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4af659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4af80cd390 0x7f4af80c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4aeffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4af80cd8e0 0x7f4af81ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4af6d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af8153290 0x7f4af80c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36150/0 (socket says 172.21.15.102:36150) 2023-03-03T14:26:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.931+0000 7f4af6d9d700 1 -- 172.21.15.102:0/1482840417 learned_addr learned my addr 172.21.15.102:0/1482840417 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:26:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4af659c700 1 -- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af8153290 msgr2=0x7f4af80c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4af659c700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af8153290 0x7f4af80c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4af659c700 1 -- 172.21.15.102:0/1482840417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4af80cd8e0 msgr2=0x7f4af81ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:17.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4af659c700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4af80cd8e0 0x7f4af81ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4af659c700 1 -- 172.21.15.102:0/1482840417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af0009a70 con 0x7f4af80cd390 2023-03-03T14:26:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4af6d9d700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af8153290 0x7f4af80c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:26:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4af659c700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4af80cd390 0x7f4af80c8960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c410 tx=0x7f4ae800c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:26:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4aeeffd700 1 -- 172.21.15.102:0/1482840417 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8005ae0 con 0x7f4af80cd390 2023-03-03T14:26:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4aeeffd700 1 -- 172.21.15.102:0/1482840417 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae8005c40 con 0x7f4af80cd390 2023-03-03T14:26:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af80ccde0 con 0x7f4af80cd390 2023-03-03T14:26:17.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.932+0000 7f4aeeffd700 1 -- 172.21.15.102:0/1482840417 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae80070d0 con 0x7f4af80cd390 2023-03-03T14:26:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.933+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af815d160 con 0x7f4af80cd390 2023-03-03T14:26:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.934+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af80cd0c0 con 0x7f4af80cd390 2023-03-03T14:26:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.935+0000 7f4aeeffd700 1 -- 172.21.15.102:0/1482840417 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ae8019030 con 0x7f4af80cd390 2023-03-03T14:26:17.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.936+0000 7f4aeeffd700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ad80682f0 0x7f4ad806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:17.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.937+0000 7f4aeeffd700 1 -- 172.21.15.102:0/1482840417 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4ae80851b0 con 0x7f4af80cd390 2023-03-03T14:26:17.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.937+0000 7f4aeffff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ad80682f0 0x7f4ad806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:17.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.937+0000 7f4aeffff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ad80682f0 0x7f4ad806a7a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f4af80c9f20 tx=0x7f4ae0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:26:17.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:17.941+0000 7f4aeeffd700 1 -- 172.21.15.102:0/1482840417 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ae8051980 con 0x7f4af80cd390 2023-03-03T14:26:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.173+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4af804e6e0 con 0x7f4af80cd390 2023-03-03T14:26:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.174+0000 7f4aeeffd700 1 -- 172.21.15.102:0/1482840417 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4ae8054fa0 con 0x7f4af80cd390 2023-03-03T14:26:18.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:26:18.176 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:26:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ad80682f0 msgr2=0x7f4ad806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ad80682f0 0x7f4ad806a7a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f4af80c9f20 tx=0x7f4ae0009040 comp rx=0 tx=0).stop 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4af80cd390 msgr2=0x7f4af80c8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4af80cd390 0x7f4af80c8960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c410 tx=0x7f4ae800c720 comp rx=0 tx=0).stop 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 shutdown_connections 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ad80682f0 0x7f4ad806a7a0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4af80cd390 0x7f4af80c8960 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4af8153290 0x7f4af80c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 --2- 172.21.15.102:0/1482840417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4af80cd8e0 0x7f4af81ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 >> 172.21.15.102:0/1482840417 conn(0x7f4af8065fb0 msgr2=0x7f4af80cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:26:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 shutdown_connections 2023-03-03T14:26:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:18.177+0000 7f4afcbff700 1 -- 172.21.15.102:0/1482840417 wait complete. 2023-03-03T14:26:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:26:18.206 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T14:26:40.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:40.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:56.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:26:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.454+0000 7f594efff700 1 Processor -- start 2023-03-03T14:26:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.454+0000 7f594efff700 1 -- start start 2023-03-03T14:26:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594efff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5948153290 0x7f5948158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594efff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f59480cd8e0 0x7f59480ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594efff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5948153660 con 0x7f59481593c0 2023-03-03T14:26:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594efff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594806c360 con 0x7f59480cd8e0 2023-03-03T14:26:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594efff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594815cea0 con 0x7f5948153290 2023-03-03T14:26:56.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594cd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5948153290 0x7f5948158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594d59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f59480cd8e0 0x7f59480ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594cd9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5948153290 0x7f5948158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:37438/0 (socket says 172.21.15.102:37438) 2023-03-03T14:26:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.456+0000 7f594cd9b700 1 -- 172.21.15.102:0/2080830032 learned_addr learned my addr 172.21.15.102:0/2080830032 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:26:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f59477fe700 1 -- 172.21.15.102:0/2080830032 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422124968 0 0) 0x7f5948153660 con 0x7f59481593c0 2023-03-03T14:26:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f59477fe700 1 -- 172.21.15.102:0/2080830032 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59380035b0 con 0x7f59481593c0 2023-03-03T14:26:56.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f594cd9b700 1 -- 172.21.15.102:0/2080830032 >> v2:172.21.15.172:3300/0 conn(0x7f59480cd8e0 msgr2=0x7f59480ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f594cd9b700 1 --2- 172.21.15.102:0/2080830032 >> v2:172.21.15.172:3300/0 conn(0x7f59480cd8e0 0x7f59480ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f594cd9b700 1 -- 172.21.15.102:0/2080830032 >> v1:172.21.15.172:6789/0 conn(0x7f59481593c0 legacy=0x7f59480cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f594cd9b700 1 -- 172.21.15.102:0/2080830032 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594806c7f0 con 0x7f5948153290 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f594d59c700 1 --2- 172.21.15.102:0/2080830032 >> v2:172.21.15.172:3300/0 conn(0x7f59480cd8e0 0x7f59480ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f594cd9b700 1 --2- 172.21.15.102:0/2080830032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5948153290 0x7f5948158e80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f593c0098f0 tx=0x7f593c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64766d5f2434e9c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f59477fe700 1 -- 172.21.15.102:0/2080830032 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c011830 con 0x7f5948153290 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f59477fe700 1 -- 172.21.15.102:0/2080830032 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c011990 con 0x7f5948153290 2023-03-03T14:26:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.457+0000 7f59477fe700 1 -- 172.21.15.102:0/2080830032 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c011d20 con 0x7f5948153290 2023-03-03T14:26:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 -- 172.21.15.102:0/2080830032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5948153290 msgr2=0x7f5948158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 --2- 172.21.15.102:0/2080830032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5948153290 0x7f5948158e80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f593c0098f0 tx=0x7f593c009c00 comp rx=0 tx=0).stop 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 -- 172.21.15.102:0/2080830032 shutdown_connections 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 --2- 172.21.15.102:0/2080830032 >> v2:172.21.15.172:3300/0 conn(0x7f59480cd8e0 0x7f59480ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 --2- 172.21.15.102:0/2080830032 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5948153290 0x7f5948158e80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 -- 172.21.15.102:0/2080830032 >> 172.21.15.102:0/2080830032 conn(0x7f5948065fb0 msgr2=0x7f59480663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 -- 172.21.15.102:0/2080830032 shutdown_connections 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.458+0000 7f594efff700 1 -- 172.21.15.102:0/2080830032 wait complete. 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.460+0000 7f594efff700 1 Processor -- start 2023-03-03T14:26:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.460+0000 7f594efff700 1 -- start start 2023-03-03T14:26:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.461+0000 7f594efff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59480cd8e0 0x7f594814e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.461+0000 7f594efff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5948153290 0x7f59481ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.461+0000 7f594efff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59481593c0 0x7f59481eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.461+0000 7f594efff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59480c6b50 con 0x7f59480cd8e0 2023-03-03T14:26:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.461+0000 7f594efff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594814cf60 con 0x7f5948153290 2023-03-03T14:26:56.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.461+0000 7f594efff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59480c6580 con 0x7f59481593c0 2023-03-03T14:26:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f5947fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5948153290 0x7f59481ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59480cd8e0 0x7f594814e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59480cd8e0 0x7f594814e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36788/0 (socket says 172.21.15.102:36788) 2023-03-03T14:26:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 -- 172.21.15.102:0/539556678 learned_addr learned my addr 172.21.15.102:0/539556678 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:26:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f5947fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5948153290 0x7f59481ee730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:58394/0 (socket says 172.21.15.102:58394) 2023-03-03T14:26:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594d59c700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59481593c0 0x7f59481eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 -- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59481593c0 msgr2=0x7f59481eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59481593c0 0x7f59481eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 -- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5948153290 msgr2=0x7f59481ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5948153290 0x7f59481ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594cd9b700 1 -- 172.21.15.102:0/539556678 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5940009a70 con 0x7f59480cd8e0 2023-03-03T14:26:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f594d59c700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59481593c0 0x7f59481eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:26:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.462+0000 7f5947fff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5948153290 0x7f59481ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:26:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.463+0000 7f594cd9b700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59480cd8e0 0x7f594814e9b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f593c000c00 tx=0x7f593c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:26:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.463+0000 7f59457fa700 1 -- 172.21.15.102:0/539556678 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c0042c0 con 0x7f59480cd8e0 2023-03-03T14:26:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.463+0000 7f59457fa700 1 -- 172.21.15.102:0/539556678 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c01f070 con 0x7f59480cd8e0 2023-03-03T14:26:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.463+0000 7f59457fa700 1 -- 172.21.15.102:0/539556678 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c0036a0 con 0x7f59480cd8e0 2023-03-03T14:26:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.463+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c009550 con 0x7f59480cd8e0 2023-03-03T14:26:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.463+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59480ccde0 con 0x7f59480cd8e0 2023-03-03T14:26:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.464+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594815d160 con 0x7f59480cd8e0 2023-03-03T14:26:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.465+0000 7f59457fa700 1 -- 172.21.15.102:0/539556678 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f593c02a020 con 0x7f59480cd8e0 2023-03-03T14:26:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.467+0000 7f59457fa700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f592c0682f0 0x7f592c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:26:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.467+0000 7f59457fa700 1 -- 172.21.15.102:0/539556678 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f593c087270 con 0x7f59480cd8e0 2023-03-03T14:26:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.467+0000 7f5947fff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f592c0682f0 0x7f592c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:26:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.468+0000 7f5947fff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f592c0682f0 0x7f592c06a7a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5948073fc0 tx=0x7f5934009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:26:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.471+0000 7f59457fa700 1 -- 172.21.15.102:0/539556678 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f593c053980 con 0x7f59480cd8e0 2023-03-03T14:26:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.709+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59480cd0c0 con 0x7f59480cd8e0 2023-03-03T14:26:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.710+0000 7f59457fa700 1 -- 172.21.15.102:0/539556678 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f593c056fa0 con 0x7f59480cd8e0 2023-03-03T14:26:56.711 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:26:56.712 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:26:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.712+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f592c0682f0 msgr2=0x7f592c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.712+0000 7f594efff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f592c0682f0 0x7f592c06a7a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5948073fc0 tx=0x7f5934009040 comp rx=0 tx=0).stop 2023-03-03T14:26:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.712+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59480cd8e0 msgr2=0x7f594814e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:26:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.712+0000 7f594efff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59480cd8e0 0x7f594814e9b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f593c000c00 tx=0x7f593c004750 comp rx=0 tx=0).stop 2023-03-03T14:26:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 shutdown_connections 2023-03-03T14:26:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f592c0682f0 0x7f592c06a7a0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f59481593c0 0x7f59481eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59480cd8e0 0x7f594814e9b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 --2- 172.21.15.102:0/539556678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5948153290 0x7f59481ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:26:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 >> 172.21.15.102:0/539556678 conn(0x7f5948065fb0 msgr2=0x7f59480cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:26:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 shutdown_connections 2023-03-03T14:26:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:26:56.713+0000 7f594efff700 1 -- 172.21.15.102:0/539556678 wait complete. 2023-03-03T14:26:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:26:56.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 65 secs before thrashing 2023-03-03T14:27:10.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:10.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:40.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:40.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:01.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:28:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.990+0000 7f53217ff700 1 Processor -- start 2023-03-03T14:28:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.990+0000 7f53217ff700 1 -- start start 2023-03-03T14:28:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.991+0000 7f53217ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f531c153290 0x7f531c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f53217ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c1593c0 0x7f531c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f53217ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f531c0cd8e0 0x7f531c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f53217ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531c153660 con 0x7f531c0cd8e0 2023-03-03T14:28:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f53217ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531c06c360 con 0x7f531c153290 2023-03-03T14:28:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f53217ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c15cea0 con 0x7f531c1593c0 2023-03-03T14:28:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f531a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c1593c0 0x7f531c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f531affd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f531c153290 0x7f531c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f531b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f531c0cd8e0 0x7f531c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f531a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c1593c0 0x7f531c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55296/0 (socket says 172.21.15.102:55296) 2023-03-03T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.992+0000 7f531a7fc700 1 -- 172.21.15.102:0/194512170 learned_addr learned my addr 172.21.15.102:0/194512170 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531affd700 1 -- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c1593c0 msgr2=0x7f531c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531affd700 1 --2- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c1593c0 0x7f531c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531affd700 1 -- 172.21.15.102:0/194512170 >> v2:172.21.15.102:3300/0 conn(0x7f531c0cd8e0 msgr2=0x7f531c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531affd700 1 --2- 172.21.15.102:0/194512170 >> v2:172.21.15.102:3300/0 conn(0x7f531c0cd8e0 0x7f531c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531affd700 1 -- 172.21.15.102:0/194512170 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c06c7f0 con 0x7f531c153290 2023-03-03T14:28:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531a7fc700 1 --2- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c1593c0 0x7f531c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:28:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531b7fe700 1 --2- 172.21.15.102:0/194512170 >> v2:172.21.15.102:3300/0 conn(0x7f531c0cd8e0 0x7f531c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:28:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f531affd700 1 --2- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c153290 0x7f531c158e80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f531c08cc70 tx=0x7f530c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b28d60257513bfae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f5319ffb700 1 -- 172.21.15.102:0/194512170 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c011830 con 0x7f531c153290 2023-03-03T14:28:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f5319ffb700 1 -- 172.21.15.102:0/194512170 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c011990 con 0x7f531c153290 2023-03-03T14:28:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.993+0000 7f5319ffb700 1 -- 172.21.15.102:0/194512170 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c011d20 con 0x7f531c153290 2023-03-03T14:28:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 -- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c153290 msgr2=0x7f531c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 --2- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c153290 0x7f531c158e80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f531c08cc70 tx=0x7f530c009b20 comp rx=0 tx=0).stop 2023-03-03T14:28:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 -- 172.21.15.102:0/194512170 shutdown_connections 2023-03-03T14:28:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 --2- 172.21.15.102:0/194512170 >> v2:172.21.15.102:3300/0 conn(0x7f531c0cd8e0 0x7f531c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 --2- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c153290 0x7f531c158e80 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 --2- 172.21.15.102:0/194512170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c1593c0 0x7f531c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 -- 172.21.15.102:0/194512170 >> 172.21.15.102:0/194512170 conn(0x7f531c065fb0 msgr2=0x7f531c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:28:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 -- 172.21.15.102:0/194512170 shutdown_connections 2023-03-03T14:28:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.994+0000 7f53217ff700 1 -- 172.21.15.102:0/194512170 wait complete. 2023-03-03T14:28:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.996+0000 7f53217ff700 1 Processor -- start 2023-03-03T14:28:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.996+0000 7f53217ff700 1 -- start start 2023-03-03T14:28:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.997+0000 7f53217ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c0cd8e0 0x7f531c1506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.997+0000 7f53217ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c153290 0x7f531c0cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.997+0000 7f53217ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f531c1593c0 0x7f531c14e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.997+0000 7f53217ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0b2030 con 0x7f531c1593c0 2023-03-03T14:28:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.997+0000 7f53217ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c155370 con 0x7f531c0cd8e0 2023-03-03T14:28:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.997+0000 7f53217ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0ca040 con 0x7f531c153290 2023-03-03T14:28:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c0cd8e0 0x7f531c1506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c153290 0x7f531c0cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c0cd8e0 0x7f531c1506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36548/0 (socket says 172.21.15.102:36548) 2023-03-03T14:28:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 -- 172.21.15.102:0/2278127723 learned_addr learned my addr 172.21.15.102:0/2278127723 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:28:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531b7fe700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f531c1593c0 0x7f531c14e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 -- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c153290 msgr2=0x7f531c0cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c153290 0x7f531c0cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 -- 172.21.15.102:0/2278127723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f531c1593c0 msgr2=0x7f531c14e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f531c1593c0 0x7f531c14e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.998+0000 7f531affd700 1 -- 172.21.15.102:0/2278127723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5314009a70 con 0x7f531c0cd8e0 2023-03-03T14:28:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.999+0000 7f531affd700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c0cd8e0 0x7f531c1506f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f530c000c00 tx=0x7f530c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.999+0000 7f53037fe700 1 -- 172.21.15.102:0/2278127723 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c004450 con 0x7f531c0cd8e0 2023-03-03T14:28:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.999+0000 7f53037fe700 1 -- 172.21.15.102:0/2278127723 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c01e070 con 0x7f531c0cd8e0 2023-03-03T14:28:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.999+0000 7f53037fe700 1 -- 172.21.15.102:0/2278127723 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c0038e0 con 0x7f531c0cd8e0 2023-03-03T14:28:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.999+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5310009550 con 0x7f531c0cd8e0 2023-03-03T14:28:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:01.999+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c0094a0 con 0x7f531c0cd8e0 2023-03-03T14:28:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.000+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c0ccde0 con 0x7f531c0cd8e0 2023-03-03T14:28:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.002+0000 7f53037fe700 1 -- 172.21.15.102:0/2278127723 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f530c00f050 con 0x7f531c0cd8e0 2023-03-03T14:28:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.003+0000 7f53037fe700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53080682f0 0x7f530806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.003+0000 7f53037fe700 1 -- 172.21.15.102:0/2278127723 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f530c087300 con 0x7f531c0cd8e0 2023-03-03T14:28:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.003+0000 7f531a7fc700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53080682f0 0x7f530806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.004+0000 7f531a7fc700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53080682f0 0x7f530806a7a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f531c0b1880 tx=0x7f53100092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:02.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.006+0000 7f53037fe700 1 -- 172.21.15.102:0/2278127723 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f530c053a90 con 0x7f531c0cd8e0 2023-03-03T14:28:02.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.240+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f531c0cd0c0 con 0x7f531c0cd8e0 2023-03-03T14:28:02.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.241+0000 7f53037fe700 1 -- 172.21.15.102:0/2278127723 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f530c0570b0 con 0x7f531c0cd8e0 2023-03-03T14:28:02.242 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:28:02.243 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:28:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53080682f0 msgr2=0x7f530806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53080682f0 0x7f530806a7a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f531c0b1880 tx=0x7f53100092c0 comp rx=0 tx=0).stop 2023-03-03T14:28:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c0cd8e0 msgr2=0x7f531c1506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c0cd8e0 0x7f531c1506f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f530c000c00 tx=0x7f530c004750 comp rx=0 tx=0).stop 2023-03-03T14:28:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 shutdown_connections 2023-03-03T14:28:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f531c1593c0 0x7f531c14e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f531c0cd8e0 0x7f531c1506f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f53080682f0 0x7f530806a7a0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 --2- 172.21.15.102:0/2278127723 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f531c153290 0x7f531c0cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:02.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 >> 172.21.15.102:0/2278127723 conn(0x7f531c065fb0 msgr2=0x7f531c15b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:28:02.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 shutdown_connections 2023-03-03T14:28:02.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:28:02.244+0000 7f53217ff700 1 -- 172.21.15.102:0/2278127723 wait complete. 2023-03-03T14:28:02.247 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:28:02.273 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-03-03T14:28:10.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:10.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:41.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:41.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:11.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:11.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:26.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:29:26.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.522+0000 7f188c1ff700 1 Processor -- start 2023-03-03T14:29:26.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.522+0000 7f188c1ff700 1 -- start start 2023-03-03T14:29:26.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.523+0000 7f188c1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f1884158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:26.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.523+0000 7f188c1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f18841593c0 0x7f18840cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:26.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f188c1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1884153660 con 0x7f18840cd8e0 2023-03-03T14:29:26.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f188c1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f188406c360 con 0x7f18841593c0 2023-03-03T14:29:26.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f188c1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188415cea0 con 0x7f1884153290 2023-03-03T14:29:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f1889f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f1884158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f188a79c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f18840cd8e0 0x7f18840ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60928/0 (socket says 172.21.15.102:60928) 2023-03-03T14:29:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f188a79c700 1 -- 172.21.15.102:0/1109909718 learned_addr learned my addr 172.21.15.102:0/1109909718 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:29:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f188979a700 1 --2- 172.21.15.102:0/1109909718 >> v2:172.21.15.172:3300/0 conn(0x7f18841593c0 0x7f18840cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f1889f9b700 1 -- 172.21.15.102:0/1109909718 >> v2:172.21.15.172:3300/0 conn(0x7f18841593c0 msgr2=0x7f18840cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:29:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f1889f9b700 1 --2- 172.21.15.102:0/1109909718 >> v2:172.21.15.172:3300/0 conn(0x7f18841593c0 0x7f18840cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f1889f9b700 1 -- 172.21.15.102:0/1109909718 >> v1:172.21.15.172:6789/0 conn(0x7f18840cd8e0 legacy=0x7f18840ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.525+0000 7f1889f9b700 1 -- 172.21.15.102:0/1109909718 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188406c7f0 con 0x7f1884153290 2023-03-03T14:29:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.525+0000 7f188979a700 1 --2- 172.21.15.102:0/1109909718 >> v2:172.21.15.172:3300/0 conn(0x7f18841593c0 0x7f18840cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:29:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.524+0000 7f1888f99700 1 -- 172.21.15.102:0/1109909718 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784108771 0 0) 0x7f1884153660 con 0x7f18840cd8e0 2023-03-03T14:29:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.525+0000 7f1889f9b700 1 --2- 172.21.15.102:0/1109909718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f1884158e80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1880009840 tx=0x7f1880009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7e6b67a381438bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:26.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.525+0000 7f1888f99700 1 -- 172.21.15.102:0/1109909718 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880011830 con 0x7f1884153290 2023-03-03T14:29:26.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.525+0000 7f1888f99700 1 -- 172.21.15.102:0/1109909718 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1880011990 con 0x7f1884153290 2023-03-03T14:29:26.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.525+0000 7f1888f99700 1 -- 172.21.15.102:0/1109909718 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880011d20 con 0x7f1884153290 2023-03-03T14:29:26.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 -- 172.21.15.102:0/1109909718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 msgr2=0x7f1884158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:26.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1109909718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f1884158e80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1880009840 tx=0x7f1880009b50 comp rx=0 tx=0).stop 2023-03-03T14:29:26.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 -- 172.21.15.102:0/1109909718 shutdown_connections 2023-03-03T14:29:26.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1109909718 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f1884158e80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1109909718 >> v2:172.21.15.172:3300/0 conn(0x7f18841593c0 0x7f18840cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 -- 172.21.15.102:0/1109909718 >> 172.21.15.102:0/1109909718 conn(0x7f1884065fb0 msgr2=0x7f18840663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:29:26.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 -- 172.21.15.102:0/1109909718 shutdown_connections 2023-03-03T14:29:26.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.526+0000 7f188c1ff700 1 -- 172.21.15.102:0/1109909718 wait complete. 2023-03-03T14:29:26.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.528+0000 7f188c1ff700 1 Processor -- start 2023-03-03T14:29:26.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.528+0000 7f188c1ff700 1 -- start start 2023-03-03T14:29:26.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188c1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18840cd8e0 0x7f18840b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:26.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188c1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f18840c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:26.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188c1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18841593c0 0x7f18841e2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:26.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188c1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18840b2c90 con 0x7f18840cd8e0 2023-03-03T14:29:26.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188c1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18841e4920 con 0x7f18841593c0 2023-03-03T14:29:26.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188c1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18841e4ea0 con 0x7f1884153290 2023-03-03T14:29:26.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188979a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f18840c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:26.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188979a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f18840c6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38018/0 (socket says 172.21.15.102:38018) 2023-03-03T14:29:26.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188a79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18841593c0 0x7f18841e2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:26.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f1889f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18840cd8e0 0x7f18840b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:26.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.529+0000 7f188979a700 1 -- 172.21.15.102:0/1166419783 learned_addr learned my addr 172.21.15.102:0/1166419783 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:29:26.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f188979a700 1 -- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18841593c0 msgr2=0x7f18841e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:26.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f188979a700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18841593c0 0x7f18841e2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f188979a700 1 -- 172.21.15.102:0/1166419783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18840cd8e0 msgr2=0x7f18840b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:26.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f188979a700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18840cd8e0 0x7f18840b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f188979a700 1 -- 172.21.15.102:0/1166419783 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18740094a0 con 0x7f1884153290 2023-03-03T14:29:26.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f1889f9b700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18840cd8e0 0x7f18840b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:29:26.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f188979a700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f18840c6660 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f187400c4f0 tx=0x7f187400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:26.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f187a7fc700 1 -- 172.21.15.102:0/1166419783 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187400ce80 con 0x7f1884153290 2023-03-03T14:29:26.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f187a7fc700 1 -- 172.21.15.102:0/1166419783 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1874015be0 con 0x7f1884153290 2023-03-03T14:29:26.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f187a7fc700 1 -- 172.21.15.102:0/1166419783 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874016030 con 0x7f1884153290 2023-03-03T14:29:26.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.530+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18800094a0 con 0x7f1884153290 2023-03-03T14:29:26.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.531+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18840ccde0 con 0x7f1884153290 2023-03-03T14:29:26.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.532+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f188415d160 con 0x7f1884153290 2023-03-03T14:29:26.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.533+0000 7f187a7fc700 1 -- 172.21.15.102:0/1166419783 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1874025030 con 0x7f1884153290 2023-03-03T14:29:26.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.534+0000 7f187a7fc700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18680682f0 0x7f186806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:26.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.534+0000 7f187a7fc700 1 -- 172.21.15.102:0/1166419783 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f187408dbc0 con 0x7f1884153290 2023-03-03T14:29:26.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.534+0000 7f1889f9b700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18680682f0 0x7f186806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:26.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.535+0000 7f1889f9b700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18680682f0 0x7f186806a7a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f18800090b0 tx=0x7f188001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:26.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.538+0000 7f187a7fc700 1 -- 172.21.15.102:0/1166419783 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18740575e0 con 0x7f1884153290 2023-03-03T14:29:26.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.770+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18840cd0c0 con 0x7f1884153290 2023-03-03T14:29:26.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.772+0000 7f187a7fc700 1 -- 172.21.15.102:0/1166419783 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1874023090 con 0x7f1884153290 2023-03-03T14:29:26.773 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:29:26.773 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:29:26.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18680682f0 msgr2=0x7f186806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:26.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18680682f0 0x7f186806a7a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f18800090b0 tx=0x7f188001c040 comp rx=0 tx=0).stop 2023-03-03T14:29:26.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 msgr2=0x7f18840c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:26.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f18840c6660 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f187400c4f0 tx=0x7f187400c800 comp rx=0 tx=0).stop 2023-03-03T14:29:26.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 shutdown_connections 2023-03-03T14:29:26.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18841593c0 0x7f18841e2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18840cd8e0 0x7f18840b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18680682f0 0x7f186806a7a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 --2- 172.21.15.102:0/1166419783 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1884153290 0x7f18840c6660 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:26.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 >> 172.21.15.102:0/1166419783 conn(0x7f1884065fb0 msgr2=0x7f18840cf640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:29:26.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.774+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 shutdown_connections 2023-03-03T14:29:26.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:29:26.775+0000 7f188c1ff700 1 -- 172.21.15.102:0/1166419783 wait complete. 2023-03-03T14:29:26.777 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:29:26.802 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T14:29:41.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:41.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T14:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:30:11.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:11.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:41.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:41.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:11.847 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:11.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:20.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:31:21.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.053+0000 7f47f999d700 1 Processor -- start 2023-03-03T14:31:21.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.053+0000 7f47f999d700 1 -- start start 2023-03-03T14:31:21.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f999d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b3970 0x7f47f40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:21.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f999d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f47f40c41b0 0x7f47f40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:21.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f999d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f40c3d20 con 0x7f47f40ced00 2023-03-03T14:31:21.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f999d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f40c4930 con 0x7f47f40c41b0 2023-03-03T14:31:21.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f999d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40cf610 con 0x7f47f40b3970 2023-03-03T14:31:21.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f899b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f47f40ced00 0x7f47f4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48720/0 (socket says 172.21.15.102:48720) 2023-03-03T14:31:21.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f899b700 1 -- 172.21.15.102:0/937174109 learned_addr learned my addr 172.21.15.102:0/937174109 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:31:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f919c700 1 --2- 172.21.15.102:0/937174109 >> v2:172.21.15.172:3300/0 conn(0x7f47f40c41b0 0x7f47f40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.055+0000 7f47f3fff700 1 --2- 172.21.15.102:0/937174109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b3970 0x7f47f40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f37fe700 1 -- 172.21.15.102:0/937174109 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580998179 0 0) 0x7f47f40c3d20 con 0x7f47f40ced00 2023-03-03T14:31:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f37fe700 1 -- 172.21.15.102:0/937174109 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e40035b0 con 0x7f47f40ced00 2023-03-03T14:31:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f3fff700 1 -- 172.21.15.102:0/937174109 >> v2:172.21.15.172:3300/0 conn(0x7f47f40c41b0 msgr2=0x7f47f40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:31:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f3fff700 1 --2- 172.21.15.102:0/937174109 >> v2:172.21.15.172:3300/0 conn(0x7f47f40c41b0 0x7f47f40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f3fff700 1 -- 172.21.15.102:0/937174109 >> v1:172.21.15.102:6789/0 conn(0x7f47f40ced00 legacy=0x7f47f4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f3fff700 1 -- 172.21.15.102:0/937174109 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f415cba0 con 0x7f47f40b3970 2023-03-03T14:31:21.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f919c700 1 --2- 172.21.15.102:0/937174109 >> v2:172.21.15.172:3300/0 conn(0x7f47f40c41b0 0x7f47f40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:31:21.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f3fff700 1 --2- 172.21.15.102:0/937174109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b3970 0x7f47f40c5a90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009840 tx=0x7f47e0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cbfbe9fcdfb562b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:31:21.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.056+0000 7f47f37fe700 1 -- 172.21.15.102:0/937174109 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0011830 con 0x7f47f40b3970 2023-03-03T14:31:21.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f37fe700 1 -- 172.21.15.102:0/937174109 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e0011990 con 0x7f47f40b3970 2023-03-03T14:31:21.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f37fe700 1 -- 172.21.15.102:0/937174109 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0011d20 con 0x7f47f40b3970 2023-03-03T14:31:21.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 -- 172.21.15.102:0/937174109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b3970 msgr2=0x7f47f40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:21.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 --2- 172.21.15.102:0/937174109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b3970 0x7f47f40c5a90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009840 tx=0x7f47e0009b50 comp rx=0 tx=0).stop 2023-03-03T14:31:21.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 -- 172.21.15.102:0/937174109 shutdown_connections 2023-03-03T14:31:21.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 --2- 172.21.15.102:0/937174109 >> v2:172.21.15.172:3300/0 conn(0x7f47f40c41b0 0x7f47f40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 --2- 172.21.15.102:0/937174109 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40b3970 0x7f47f40c5a90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 -- 172.21.15.102:0/937174109 >> 172.21.15.102:0/937174109 conn(0x7f47f408b1c0 msgr2=0x7f47f4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:31:21.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 -- 172.21.15.102:0/937174109 shutdown_connections 2023-03-03T14:31:21.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.057+0000 7f47f999d700 1 -- 172.21.15.102:0/937174109 wait complete. 2023-03-03T14:31:21.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.059+0000 7f47f999d700 1 Processor -- start 2023-03-03T14:31:21.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.059+0000 7f47f999d700 1 -- start start 2023-03-03T14:31:21.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.060+0000 7f47f999d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47f40b3970 0x7f47f41484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:21.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.060+0000 7f47f999d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40c41b0 0x7f47f414fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:21.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.060+0000 7f47f999d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40ced00 0x7f47f4151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:21.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.060+0000 7f47f999d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415a750 con 0x7f47f40b3970 2023-03-03T14:31:21.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.060+0000 7f47f999d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415b5e0 con 0x7f47f40ced00 2023-03-03T14:31:21.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.060+0000 7f47f999d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f414f870 con 0x7f47f40c41b0 2023-03-03T14:31:21.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f899b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47f40b3970 0x7f47f41484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:21.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40ced00 0x7f47f4151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:21.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40c41b0 0x7f47f414fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:21.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40ced00 0x7f47f4151980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46182/0 (socket says 172.21.15.102:46182) 2023-03-03T14:31:21.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 -- 172.21.15.102:0/1778509170 learned_addr learned my addr 172.21.15.102:0/1778509170 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:31:21.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 -- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40c41b0 msgr2=0x7f47f414fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:21.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40c41b0 0x7f47f414fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 -- 172.21.15.102:0/1778509170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47f40b3970 msgr2=0x7f47f41484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:21.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47f40b3970 0x7f47f41484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.061+0000 7f47f919c700 1 -- 172.21.15.102:0/1778509170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec009a70 con 0x7f47f40ced00 2023-03-03T14:31:21.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.062+0000 7f47f919c700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40ced00 0x7f47f4151980 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00ef90 tx=0x7f47ec00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:31:21.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.062+0000 7f47f17fa700 1 -- 172.21.15.102:0/1778509170 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00cc90 con 0x7f47f40ced00 2023-03-03T14:31:21.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.062+0000 7f47f17fa700 1 -- 172.21.15.102:0/1778509170 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec00f040 con 0x7f47f40ced00 2023-03-03T14:31:21.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.062+0000 7f47f17fa700 1 -- 172.21.15.102:0/1778509170 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec0146c0 con 0x7f47f40ced00 2023-03-03T14:31:21.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.062+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e00094a0 con 0x7f47f40ced00 2023-03-03T14:31:21.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.062+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f406bf20 con 0x7f47f40ced00 2023-03-03T14:31:21.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.063+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f415ce80 con 0x7f47f40ced00 2023-03-03T14:31:21.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.064+0000 7f47f17fa700 1 -- 172.21.15.102:0/1778509170 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47ec024070 con 0x7f47f40ced00 2023-03-03T14:31:21.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.066+0000 7f47f17fa700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47d80682f0 0x7f47d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:21.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.066+0000 7f47f17fa700 1 -- 172.21.15.102:0/1778509170 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f47ec0869a0 con 0x7f47f40ced00 2023-03-03T14:31:21.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.066+0000 7f47f899b700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47d80682f0 0x7f47d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:21.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.067+0000 7f47f899b700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47d80682f0 0x7f47d806a7a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f47e8007fd0 tx=0x7f47e8007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:31:21.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.070+0000 7f47f17fa700 1 -- 172.21.15.102:0/1778509170 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47ec0530b0 con 0x7f47f40ced00 2023-03-03T14:31:21.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.306+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47f406c200 con 0x7f47f40ced00 2023-03-03T14:31:21.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.307+0000 7f47f17fa700 1 -- 172.21.15.102:0/1778509170 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f47ec0566d0 con 0x7f47f40ced00 2023-03-03T14:31:21.309 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:31:21.309 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:31:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47d80682f0 msgr2=0x7f47d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47d80682f0 0x7f47d806a7a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f47e8007fd0 tx=0x7f47e8007dc0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40ced00 msgr2=0x7f47f4151980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40ced00 0x7f47f4151980 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00ef90 tx=0x7f47ec00c3a0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 shutdown_connections 2023-03-03T14:31:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47f40ced00 0x7f47f4151980 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47f40b3970 0x7f47f41484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f47d80682f0 0x7f47d806a7a0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 --2- 172.21.15.102:0/1778509170 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47f40c41b0 0x7f47f414fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 >> 172.21.15.102:0/1778509170 conn(0x7f47f408b1c0 msgr2=0x7f47f41e1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:31:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.309+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 shutdown_connections 2023-03-03T14:31:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:31:21.310+0000 7f47f999d700 1 -- 172.21.15.102:0/1778509170 wait complete. 2023-03-03T14:31:21.314 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:31:21.338 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T14:31:42.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:42.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:12.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:12.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:31.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:32:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.588+0000 7f64703ff700 1 Processor -- start 2023-03-03T14:32:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.588+0000 7f64703ff700 1 -- start start 2023-03-03T14:32:31.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f64703ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f64680cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:31.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f64703ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6468153660 con 0x7f6468153290 2023-03-03T14:32:31.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f64703ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646806c360 con 0x7f64680cd390 2023-03-03T14:32:31.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f64703ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646815cea0 con 0x7f64680cd8e0 2023-03-03T14:32:31.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f646e99c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f64680cd390 0x7f64680ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:37670/0 (socket says 172.21.15.102:37670) 2023-03-03T14:32:31.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f646e99c700 1 -- 172.21.15.102:0/2969234563 learned_addr learned my addr 172.21.15.102:0/2969234563 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:32:31.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f646e19b700 1 --1- 172.21.15.102:0/2969234563 >> v1:172.21.15.102:6789/0 conn(0x7f6468153290 0x7f6468158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36840/0 (socket says 172.21.15.102:36840) 2023-03-03T14:32:31.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f646d99a700 1 --2- 172.21.15.102:0/2969234563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f64680cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:31.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.590+0000 7f646d199700 1 -- 172.21.15.102:0/2969234563 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736342727 0 0) 0x7f646806c360 con 0x7f64680cd390 2023-03-03T14:32:31.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d199700 1 -- 172.21.15.102:0/2969234563 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64540035b0 con 0x7f64680cd390 2023-03-03T14:32:31.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d99a700 1 -- 172.21.15.102:0/2969234563 >> v1:172.21.15.172:6789/0 conn(0x7f64680cd390 legacy=0x7f64680ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:31.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d199700 1 -- 172.21.15.102:0/2969234563 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956468944 0 0) 0x7f6468153660 con 0x7f6468153290 2023-03-03T14:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d99a700 1 -- 172.21.15.102:0/2969234563 >> v1:172.21.15.102:6789/0 conn(0x7f6468153290 legacy=0x7f6468158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d99a700 1 -- 172.21.15.102:0/2969234563 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646806c7f0 con 0x7f64680cd8e0 2023-03-03T14:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d99a700 1 --2- 172.21.15.102:0/2969234563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f64680cde20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6458009870 tx=0x7f6458009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecd0540f0f78ee0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d199700 1 -- 172.21.15.102:0/2969234563 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458011830 con 0x7f64680cd8e0 2023-03-03T14:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d199700 1 -- 172.21.15.102:0/2969234563 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6458011990 con 0x7f64680cd8e0 2023-03-03T14:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.591+0000 7f646d199700 1 -- 172.21.15.102:0/2969234563 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458011d20 con 0x7f64680cd8e0 2023-03-03T14:32:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.592+0000 7f64703ff700 1 -- 172.21.15.102:0/2969234563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 msgr2=0x7f64680cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.592+0000 7f64703ff700 1 --2- 172.21.15.102:0/2969234563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f64680cde20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6458009870 tx=0x7f6458009b80 comp rx=0 tx=0).stop 2023-03-03T14:32:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.592+0000 7f64703ff700 1 -- 172.21.15.102:0/2969234563 shutdown_connections 2023-03-03T14:32:31.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.592+0000 7f64703ff700 1 --2- 172.21.15.102:0/2969234563 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f64680cde20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:31.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.592+0000 7f64703ff700 1 -- 172.21.15.102:0/2969234563 >> 172.21.15.102:0/2969234563 conn(0x7f6468065fb0 msgr2=0x7f64680663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:32:31.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.592+0000 7f64703ff700 1 -- 172.21.15.102:0/2969234563 shutdown_connections 2023-03-03T14:32:31.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.592+0000 7f64703ff700 1 -- 172.21.15.102:0/2969234563 wait complete. 2023-03-03T14:32:31.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.594+0000 7f64703ff700 1 Processor -- start 2023-03-03T14:32:31.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.594+0000 7f64703ff700 1 -- start start 2023-03-03T14:32:31.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f64703ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64680cd390 0x7f64681563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:31.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f64703ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f646814f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:31.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f64703ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6468153290 0x7f646814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:31.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f64703ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680c8c40 con 0x7f64680cd390 2023-03-03T14:32:31.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f64703ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680c8f20 con 0x7f6468153290 2023-03-03T14:32:31.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f64703ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64680c8130 con 0x7f64680cd8e0 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f646e19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64680cd390 0x7f64681563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f646e19b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64680cd390 0x7f64681563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47502/0 (socket says 172.21.15.102:47502) 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f646e19b700 1 -- 172.21.15.102:0/787548765 learned_addr learned my addr 172.21.15.102:0/787548765 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f646e99c700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6468153290 0x7f646814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.595+0000 7f646d99a700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f646814f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f646e19b700 1 -- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 msgr2=0x7f646814f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f646e19b700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f646814f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:31.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f646e19b700 1 -- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6468153290 msgr2=0x7f646814fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:31.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f646e19b700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6468153290 0x7f646814fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:31.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f646e19b700 1 -- 172.21.15.102:0/787548765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64580094a0 con 0x7f64680cd390 2023-03-03T14:32:31.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f646e19b700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64680cd390 0x7f64681563b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f646400c550 tx=0x7f646400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:31.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f645effd700 1 -- 172.21.15.102:0/787548765 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464004c80 con 0x7f64680cd390 2023-03-03T14:32:31.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.596+0000 7f645effd700 1 -- 172.21.15.102:0/787548765 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6464004e00 con 0x7f64680cd390 2023-03-03T14:32:31.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.597+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64680ccde0 con 0x7f64680cd390 2023-03-03T14:32:31.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.597+0000 7f645effd700 1 -- 172.21.15.102:0/787548765 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464007310 con 0x7f64680cd390 2023-03-03T14:32:31.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.597+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646815d160 con 0x7f64680cd390 2023-03-03T14:32:31.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.598+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64680cd0c0 con 0x7f64680cd390 2023-03-03T14:32:31.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.599+0000 7f645effd700 1 -- 172.21.15.102:0/787548765 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f646401a020 con 0x7f64680cd390 2023-03-03T14:32:31.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.600+0000 7f645effd700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f644c068320 0x7f644c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:31.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.600+0000 7f645effd700 1 -- 172.21.15.102:0/787548765 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f64640858d0 con 0x7f64680cd390 2023-03-03T14:32:31.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.600+0000 7f646d99a700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f644c068320 0x7f644c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:31.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.601+0000 7f646d99a700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f644c068320 0x7f644c06a7d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f64580097c0 tx=0x7f645801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.604+0000 7f645effd700 1 -- 172.21.15.102:0/787548765 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6464052000 con 0x7f64680cd390 2023-03-03T14:32:31.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.836+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64680c7f90 con 0x7f64680cd390 2023-03-03T14:32:31.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.837+0000 7f645effd700 1 -- 172.21.15.102:0/787548765 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f64680c7f90 con 0x7f64680cd390 2023-03-03T14:32:31.838 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:32:31.839 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:32:31.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f644c068320 msgr2=0x7f644c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:31.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f644c068320 0x7f644c06a7d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f64580097c0 tx=0x7f645801c040 comp rx=0 tx=0).stop 2023-03-03T14:32:31.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64680cd390 msgr2=0x7f64681563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64680cd390 0x7f64681563b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f646400c550 tx=0x7f646400c860 comp rx=0 tx=0).stop 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 shutdown_connections 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6468153290 0x7f646814fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64680cd390 0x7f64681563b0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f644c068320 0x7f644c06a7d0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 --2- 172.21.15.102:0/787548765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f64680cd8e0 0x7f646814f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 >> 172.21.15.102:0/787548765 conn(0x7f6468065fb0 msgr2=0x7f64681584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:32:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 shutdown_connections 2023-03-03T14:32:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:32:31.840+0000 7f64703ff700 1 -- 172.21.15.102:0/787548765 wait complete. 2023-03-03T14:32:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:32:31.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T14:32:42.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:42.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:09.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:33:10.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.124+0000 7f11d45ff700 1 Processor -- start 2023-03-03T14:33:10.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.125+0000 7f11d45ff700 1 -- start start 2023-03-03T14:33:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d45ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11cc0cd8e0 0x7f11cc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d45ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd390 0x7f11cc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d45ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc153660 con 0x7f11cc153290 2023-03-03T14:33:10.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d45ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc06c360 con 0x7f11cc0cd8e0 2023-03-03T14:33:10.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d45ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc15cea0 con 0x7f11cc0cd390 2023-03-03T14:33:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d1b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11cc0cd8e0 0x7f11cc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d239b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f11cc153290 0x7f11cc158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54204/0 (socket says 172.21.15.102:54204) 2023-03-03T14:33:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d239b700 1 -- 172.21.15.102:0/235584713 learned_addr learned my addr 172.21.15.102:0/235584713 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:33:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.126+0000 7f11d2b9c700 1 --2- 172.21.15.102:0/235584713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd390 0x7f11cc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d2b9c700 1 -- 172.21.15.102:0/235584713 >> v2:172.21.15.102:3300/0 conn(0x7f11cc0cd8e0 msgr2=0x7f11cc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:33:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d2b9c700 1 --2- 172.21.15.102:0/235584713 >> v2:172.21.15.102:3300/0 conn(0x7f11cc0cd8e0 0x7f11cc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d2b9c700 1 -- 172.21.15.102:0/235584713 >> v1:172.21.15.172:6789/0 conn(0x7f11cc153290 legacy=0x7f11cc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d2b9c700 1 -- 172.21.15.102:0/235584713 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc06c7f0 con 0x7f11cc0cd390 2023-03-03T14:33:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d1399700 1 -- 172.21.15.102:0/235584713 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280657714 0 0) 0x7f11cc153660 con 0x7f11cc153290 2023-03-03T14:33:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d2b9c700 1 --2- 172.21.15.102:0/235584713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd390 0x7f11cc0ce360 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009d90 tx=0x7f11c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7e8d2ee2d3f7c38 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d1399700 1 -- 172.21.15.102:0/235584713 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c400e070 con 0x7f11cc0cd390 2023-03-03T14:33:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d1399700 1 -- 172.21.15.102:0/235584713 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c400bb20 con 0x7f11cc0cd390 2023-03-03T14:33:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.127+0000 7f11d1399700 1 -- 172.21.15.102:0/235584713 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4012430 con 0x7f11cc0cd390 2023-03-03T14:33:10.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 -- 172.21.15.102:0/235584713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd390 msgr2=0x7f11cc0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 --2- 172.21.15.102:0/235584713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd390 0x7f11cc0ce360 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009d90 tx=0x7f11c400b330 comp rx=0 tx=0).stop 2023-03-03T14:33:10.131 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 -- 172.21.15.102:0/235584713 shutdown_connections 2023-03-03T14:33:10.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 --2- 172.21.15.102:0/235584713 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd390 0x7f11cc0ce360 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 --2- 172.21.15.102:0/235584713 >> v2:172.21.15.102:3300/0 conn(0x7f11cc0cd8e0 0x7f11cc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 -- 172.21.15.102:0/235584713 >> 172.21.15.102:0/235584713 conn(0x7f11cc065fb0 msgr2=0x7f11cc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:10.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 -- 172.21.15.102:0/235584713 shutdown_connections 2023-03-03T14:33:10.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.128+0000 7f11d45ff700 1 -- 172.21.15.102:0/235584713 wait complete. 2023-03-03T14:33:10.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.130+0000 7f11d45ff700 1 Processor -- start 2023-03-03T14:33:10.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.130+0000 7f11d45ff700 1 -- start start 2023-03-03T14:33:10.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.131+0000 7f11d45ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc0cd390 0x7f11cc0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.131+0000 7f11d45ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd8e0 0x7f11cc1e5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d45ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11cc153290 0x7f11cc155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d45ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0b29b0 con 0x7f11cc0cd390 2023-03-03T14:33:10.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d45ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1f2330 con 0x7f11cc153290 2023-03-03T14:33:10.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d45ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0b2be0 con 0x7f11cc0cd8e0 2023-03-03T14:33:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc0cd390 0x7f11cc0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc0cd390 0x7f11cc0b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55460/0 (socket says 172.21.15.102:55460) 2023-03-03T14:33:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 -- 172.21.15.102:0/317226818 learned_addr learned my addr 172.21.15.102:0/317226818 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:33:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d1b9a700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd8e0 0x7f11cc1e5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d2b9c700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11cc153290 0x7f11cc155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 -- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd8e0 msgr2=0x7f11cc1e5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd8e0 0x7f11cc1e5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 -- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11cc153290 msgr2=0x7f11cc155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11cc153290 0x7f11cc155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d239b700 1 -- 172.21.15.102:0/317226818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c4009a70 con 0x7f11cc0cd390 2023-03-03T14:33:10.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.132+0000 7f11d1b9a700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd8e0 0x7f11cc1e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:33:10.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.133+0000 7f11d239b700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc0cd390 0x7f11cc0b2470 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f11c80011e0 tx=0x7f11c800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:10.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.133+0000 7f11c2ffd700 1 -- 172.21.15.102:0/317226818 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8004c80 con 0x7f11cc0cd390 2023-03-03T14:33:10.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.133+0000 7f11c2ffd700 1 -- 172.21.15.102:0/317226818 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c8004e00 con 0x7f11cc0cd390 2023-03-03T14:33:10.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.133+0000 7f11c2ffd700 1 -- 172.21.15.102:0/317226818 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8007310 con 0x7f11cc0cd390 2023-03-03T14:33:10.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.133+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc0ccde0 con 0x7f11cc0cd390 2023-03-03T14:33:10.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.133+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc15d160 con 0x7f11cc0cd390 2023-03-03T14:33:10.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.134+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc0cd0c0 con 0x7f11cc0cd390 2023-03-03T14:33:10.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.135+0000 7f11c2ffd700 1 -- 172.21.15.102:0/317226818 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11c801a020 con 0x7f11cc0cd390 2023-03-03T14:33:10.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.137+0000 7f11c2ffd700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11b00682f0 0x7f11b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.137+0000 7f11c2ffd700 1 -- 172.21.15.102:0/317226818 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f11c80858c0 con 0x7f11cc0cd390 2023-03-03T14:33:10.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.138+0000 7f11d1b9a700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11b00682f0 0x7f11b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.138+0000 7f11d1b9a700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11b00682f0 0x7f11b006a7a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f11cc159af0 tx=0x7f11bc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:10.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.142+0000 7f11c2ffd700 1 -- 172.21.15.102:0/317226818 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11c8051fd0 con 0x7f11cc0cd390 2023-03-03T14:33:10.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.376+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11cc04e6e0 con 0x7f11cc0cd390 2023-03-03T14:33:10.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.377+0000 7f11c2ffd700 1 -- 172.21.15.102:0/317226818 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f11c80555f0 con 0x7f11cc0cd390 2023-03-03T14:33:10.378 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:33:10.378 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:33:10.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11b00682f0 msgr2=0x7f11b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11b00682f0 0x7f11b006a7a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f11cc159af0 tx=0x7f11bc009210 comp rx=0 tx=0).stop 2023-03-03T14:33:10.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc0cd390 msgr2=0x7f11cc0b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc0cd390 0x7f11cc0b2470 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f11c80011e0 tx=0x7f11c800ca30 comp rx=0 tx=0).stop 2023-03-03T14:33:10.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 shutdown_connections 2023-03-03T14:33:10.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11cc153290 0x7f11cc155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc0cd390 0x7f11cc0b2470 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11b00682f0 0x7f11b006a7a0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 --2- 172.21.15.102:0/317226818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11cc0cd8e0 0x7f11cc1e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.379+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 >> 172.21.15.102:0/317226818 conn(0x7f11cc065fb0 msgr2=0x7f11cc1582f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:10.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.380+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 shutdown_connections 2023-03-03T14:33:10.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:33:10.380+0000 7f11d45ff700 1 -- 172.21.15.102:0/317226818 wait complete. 2023-03-03T14:33:10.383 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:33:10.408 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-03-03T14:33:12.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:12.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:42.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:42.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:12.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:12.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:16.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:34:16.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.663+0000 7f6643fff700 1 Processor -- start 2023-03-03T14:34:16.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.664+0000 7f6643fff700 1 -- start start 2023-03-03T14:34:16.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.665+0000 7f6643fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66440ced00 0x7f66440c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.665+0000 7f6643fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66440c41b0 0x7f66440cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.665+0000 7f6643fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66440c3d20 con 0x7f66440b3970 2023-03-03T14:34:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.665+0000 7f6643fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66440c4930 con 0x7f66440c41b0 2023-03-03T14:34:16.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.665+0000 7f6643fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66440cf610 con 0x7f66440ced00 2023-03-03T14:34:16.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66437fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66440c41b0 0x7f66440cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:16.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f6642ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66440ced00 0x7f66440c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:16.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66437fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66440c41b0 0x7f66440cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55106/0 (socket says 172.21.15.102:55106) 2023-03-03T14:34:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66437fe700 1 -- 172.21.15.102:0/373944315 learned_addr learned my addr 172.21.15.102:0/373944315 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:34:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f663bfff700 1 --1- 172.21.15.102:0/373944315 >> v1:172.21.15.172:6789/0 conn(0x7f66440b3970 0x7f6644158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:48936/0 (socket says 172.21.15.102:48936) 2023-03-03T14:34:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66427fc700 1 -- 172.21.15.102:0/373944315 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203456807 0 0) 0x7f66440c3d20 con 0x7f66440b3970 2023-03-03T14:34:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66437fe700 1 -- 172.21.15.102:0/373944315 >> v2:172.21.15.172:3300/0 conn(0x7f66440ced00 msgr2=0x7f66440c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:34:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66437fe700 1 --2- 172.21.15.102:0/373944315 >> v2:172.21.15.172:3300/0 conn(0x7f66440ced00 0x7f66440c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66437fe700 1 -- 172.21.15.102:0/373944315 >> v1:172.21.15.172:6789/0 conn(0x7f66440b3970 legacy=0x7f6644158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:16.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.666+0000 7f66437fe700 1 -- 172.21.15.102:0/373944315 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664415cba0 con 0x7f66440c41b0 2023-03-03T14:34:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f66437fe700 1 --2- 172.21.15.102:0/373944315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440cf0d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f66440b1b40 tx=0x7f663c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8695d2a9921df717 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f66427fc700 1 -- 172.21.15.102:0/373944315 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c00e070 con 0x7f66440c41b0 2023-03-03T14:34:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f66427fc700 1 -- 172.21.15.102:0/373944315 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c00b8d0 con 0x7f66440c41b0 2023-03-03T14:34:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f66427fc700 1 -- 172.21.15.102:0/373944315 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c00bc80 con 0x7f66440c41b0 2023-03-03T14:34:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f6643fff700 1 -- 172.21.15.102:0/373944315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 msgr2=0x7f66440cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f6643fff700 1 --2- 172.21.15.102:0/373944315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440cf0d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f66440b1b40 tx=0x7f663c00b330 comp rx=0 tx=0).stop 2023-03-03T14:34:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f6643fff700 1 -- 172.21.15.102:0/373944315 shutdown_connections 2023-03-03T14:34:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f6643fff700 1 --2- 172.21.15.102:0/373944315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440cf0d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.667+0000 7f6643fff700 1 --2- 172.21.15.102:0/373944315 >> v2:172.21.15.172:3300/0 conn(0x7f66440ced00 0x7f66440c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.668+0000 7f6643fff700 1 -- 172.21.15.102:0/373944315 >> 172.21.15.102:0/373944315 conn(0x7f664408b1c0 msgr2=0x7f6644071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.668+0000 7f6643fff700 1 -- 172.21.15.102:0/373944315 shutdown_connections 2023-03-03T14:34:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.668+0000 7f6643fff700 1 -- 172.21.15.102:0/373944315 wait complete. 2023-03-03T14:34:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.670+0000 7f6643fff700 1 Processor -- start 2023-03-03T14:34:16.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.670+0000 7f6643fff700 1 -- start start 2023-03-03T14:34:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6643fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66440b3970 0x7f66440c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6643fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6643fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66440ced00 0x7f6644150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6643fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66440c6910 con 0x7f66440c41b0 2023-03-03T14:34:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6643fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66440c7df0 con 0x7f66440b3970 2023-03-03T14:34:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6643fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66440c87e0 con 0x7f66440ced00 2023-03-03T14:34:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6642ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66440b3970 0x7f66440c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f663bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6642ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66440b3970 0x7f66440c82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55226/0 (socket says 172.21.15.102:55226) 2023-03-03T14:34:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f6642ffd700 1 -- 172.21.15.102:0/134266649 learned_addr learned my addr 172.21.15.102:0/134266649 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:34:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.671+0000 7f66437fe700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66440ced00 0x7f6644150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6642ffd700 1 -- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66440ced00 msgr2=0x7f6644150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6642ffd700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66440ced00 0x7f6644150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6642ffd700 1 -- 172.21.15.102:0/134266649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 msgr2=0x7f66440ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:16.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6642ffd700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6642ffd700 1 -- 172.21.15.102:0/134266649 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c009b30 con 0x7f66440b3970 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f663bfff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f66437fe700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66440ced00 0x7f6644150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6642ffd700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66440b3970 0x7f66440c82a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f663400ea10 tx=0x7f663400ed20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6640ff9700 1 -- 172.21.15.102:0/134266649 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634009710 con 0x7f66440b3970 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6640ff9700 1 -- 172.21.15.102:0/134266649 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663400eeb0 con 0x7f66440b3970 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6640ff9700 1 -- 172.21.15.102:0/134266649 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663400ca50 con 0x7f66440b3970 2023-03-03T14:34:16.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.672+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664406bf20 con 0x7f66440b3970 2023-03-03T14:34:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.673+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664415ce80 con 0x7f66440b3970 2023-03-03T14:34:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.674+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664406c200 con 0x7f66440b3970 2023-03-03T14:34:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.675+0000 7f6640ff9700 1 -- 172.21.15.102:0/134266649 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6634019030 con 0x7f66440b3970 2023-03-03T14:34:16.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.676+0000 7f6640ff9700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66240682f0 0x7f662406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:16.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.676+0000 7f6640ff9700 1 -- 172.21.15.102:0/134266649 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6634085c60 con 0x7f66440b3970 2023-03-03T14:34:16.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.677+0000 7f663bfff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66240682f0 0x7f662406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:16.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.677+0000 7f663bfff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66240682f0 0x7f662406a7a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f662c007c40 tx=0x7f662c007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:16.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.680+0000 7f6640ff9700 1 -- 172.21.15.102:0/134266649 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6634052370 con 0x7f66440b3970 2023-03-03T14:34:16.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.915+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664404e6e0 con 0x7f66440b3970 2023-03-03T14:34:16.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.916+0000 7f6640ff9700 1 -- 172.21.15.102:0/134266649 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6634055990 con 0x7f66440b3970 2023-03-03T14:34:16.918 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:34:16.918 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:34:16.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66240682f0 msgr2=0x7f662406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:16.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66240682f0 0x7f662406a7a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f662c007c40 tx=0x7f662c007b90 comp rx=0 tx=0).stop 2023-03-03T14:34:16.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66440b3970 msgr2=0x7f66440c82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:16.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66440b3970 0x7f66440c82a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f663400ea10 tx=0x7f663400ed20 comp rx=0 tx=0).stop 2023-03-03T14:34:16.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 shutdown_connections 2023-03-03T14:34:16.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66240682f0 0x7f662406a7a0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66440ced00 0x7f6644150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66440b3970 0x7f66440c82a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 --2- 172.21.15.102:0/134266649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66440c41b0 0x7f66440ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:16.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 >> 172.21.15.102:0/134266649 conn(0x7f664408b1c0 msgr2=0x7f6644071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:16.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.919+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 shutdown_connections 2023-03-03T14:34:16.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:16.920+0000 7f6643fff700 1 -- 172.21.15.102:0/134266649 wait complete. 2023-03-03T14:34:16.923 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:34:16.948 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 22 secs before thrashing 2023-03-03T14:34:38.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:34:39.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.213+0000 7f10cc1ff700 1 Processor -- start 2023-03-03T14:34:39.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.213+0000 7f10cc1ff700 1 -- start start 2023-03-03T14:34:39.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.214+0000 7f10cc1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c4153290 0x7f10c4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:39.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.214+0000 7f10cc1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10c41593c0 0x7f10c40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:39.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.214+0000 7f10cc1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c4153660 con 0x7f10c40cd8e0 2023-03-03T14:34:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10cc1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c406c360 con 0x7f10c41593c0 2023-03-03T14:34:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10cc1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c415cea0 con 0x7f10c4153290 2023-03-03T14:34:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c979a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10c41593c0 0x7f10c40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c9f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c4153290 0x7f10c4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c9f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c4153290 0x7f10c4158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:56442/0 (socket says 172.21.15.102:56442) 2023-03-03T14:34:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c979a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10c41593c0 0x7f10c40cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57058/0 (socket says 172.21.15.102:57058) 2023-03-03T14:34:39.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c9f9b700 1 -- 172.21.15.102:0/1163085520 learned_addr learned my addr 172.21.15.102:0/1163085520 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:34:39.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c979a700 1 -- 172.21.15.102:0/1163085520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c4153290 msgr2=0x7f10c4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:34:39.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c979a700 1 --2- 172.21.15.102:0/1163085520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c4153290 0x7f10c4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c979a700 1 -- 172.21.15.102:0/1163085520 >> v1:172.21.15.102:6789/0 conn(0x7f10c40cd8e0 legacy=0x7f10c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.215+0000 7f10c979a700 1 -- 172.21.15.102:0/1163085520 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c406c7f0 con 0x7f10c41593c0 2023-03-03T14:34:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.216+0000 7f10c979a700 1 --2- 172.21.15.102:0/1163085520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c41593c0 0x7f10c40cde20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009840 tx=0x7f10b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f62b3d3148d3d374 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.216+0000 7f10c8f99700 1 -- 172.21.15.102:0/1163085520 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4011830 con 0x7f10c41593c0 2023-03-03T14:34:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.216+0000 7f10c8f99700 1 -- 172.21.15.102:0/1163085520 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b4011990 con 0x7f10c41593c0 2023-03-03T14:34:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.216+0000 7f10c8f99700 1 -- 172.21.15.102:0/1163085520 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4011d20 con 0x7f10c41593c0 2023-03-03T14:34:39.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.216+0000 7f10cc1ff700 1 -- 172.21.15.102:0/1163085520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c41593c0 msgr2=0x7f10c40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.216+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/1163085520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c41593c0 0x7f10c40cde20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009840 tx=0x7f10b4009b50 comp rx=0 tx=0).stop 2023-03-03T14:34:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.217+0000 7f10cc1ff700 1 -- 172.21.15.102:0/1163085520 shutdown_connections 2023-03-03T14:34:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.217+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/1163085520 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c4153290 0x7f10c4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.217+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/1163085520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c41593c0 0x7f10c40cde20 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.217+0000 7f10cc1ff700 1 -- 172.21.15.102:0/1163085520 >> 172.21.15.102:0/1163085520 conn(0x7f10c4065fb0 msgr2=0x7f10c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.217+0000 7f10cc1ff700 1 -- 172.21.15.102:0/1163085520 shutdown_connections 2023-03-03T14:34:39.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.217+0000 7f10cc1ff700 1 -- 172.21.15.102:0/1163085520 wait complete. 2023-03-03T14:34:39.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.219+0000 7f10cc1ff700 1 Processor -- start 2023-03-03T14:34:39.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.219+0000 7f10cc1ff700 1 -- start start 2023-03-03T14:34:39.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10cc1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10c40cd8e0 0x7f10c40b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:39.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10cc1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c4153290 0x7f10c40b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:39.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10cc1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c41593c0 0x7f10c40b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:39.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10cc1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c41ee500 con 0x7f10c40cd8e0 2023-03-03T14:34:39.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10cc1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c40b4750 con 0x7f10c4153290 2023-03-03T14:34:39.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10cc1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c40b4a10 con 0x7f10c41593c0 2023-03-03T14:34:39.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10c9f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10c40cd8e0 0x7f10c40b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:39.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10ca79c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c41593c0 0x7f10c40b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:39.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10c9f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10c40cd8e0 0x7f10c40b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55342/0 (socket says 172.21.15.102:55342) 2023-03-03T14:34:39.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10c979a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c4153290 0x7f10c40b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:39.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.220+0000 7f10c9f9b700 1 -- 172.21.15.102:0/3224453690 learned_addr learned my addr 172.21.15.102:0/3224453690 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:34:39.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.221+0000 7f10c979a700 1 -- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c41593c0 msgr2=0x7f10c40b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:39.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.221+0000 7f10c979a700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c41593c0 0x7f10c40b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.221+0000 7f10c979a700 1 -- 172.21.15.102:0/3224453690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10c40cd8e0 msgr2=0x7f10c40b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:39.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.221+0000 7f10c979a700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10c40cd8e0 0x7f10c40b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.221+0000 7f10c979a700 1 -- 172.21.15.102:0/3224453690 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b40094a0 con 0x7f10c4153290 2023-03-03T14:34:39.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.221+0000 7f10c9f9b700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10c40cd8e0 0x7f10c40b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:34:39.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.221+0000 7f10c979a700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c4153290 0x7f10c40b1c70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f10b4016000 tx=0x7f10b4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:39.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.222+0000 7f10ba7fc700 1 -- 172.21.15.102:0/3224453690 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b40036a0 con 0x7f10c4153290 2023-03-03T14:34:39.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.222+0000 7f10ba7fc700 1 -- 172.21.15.102:0/3224453690 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b401f070 con 0x7f10c4153290 2023-03-03T14:34:39.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.222+0000 7f10ba7fc700 1 -- 172.21.15.102:0/3224453690 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4003a40 con 0x7f10c4153290 2023-03-03T14:34:39.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.222+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c40ccde0 con 0x7f10c4153290 2023-03-03T14:34:39.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.222+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c415d160 con 0x7f10c4153290 2023-03-03T14:34:39.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.223+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c40cd0c0 con 0x7f10c4153290 2023-03-03T14:34:39.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.224+0000 7f10ba7fc700 1 -- 172.21.15.102:0/3224453690 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f10b4048070 con 0x7f10c4153290 2023-03-03T14:34:39.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.225+0000 7f10ba7fc700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10a8068320 0x7f10a806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:39.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.225+0000 7f10ba7fc700 1 -- 172.21.15.102:0/3224453690 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f10b4021070 con 0x7f10c4153290 2023-03-03T14:34:39.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.226+0000 7f10c9f9b700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10a8068320 0x7f10a806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:39.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.226+0000 7f10c9f9b700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10a8068320 0x7f10a806a7d0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009520 tx=0x7f10c00092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:39.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.229+0000 7f10ba7fc700 1 -- 172.21.15.102:0/3224453690 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10b4054850 con 0x7f10c4153290 2023-03-03T14:34:39.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.461+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10c4155d30 con 0x7f10c4153290 2023-03-03T14:34:39.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.462+0000 7f10ba7fc700 1 -- 172.21.15.102:0/3224453690 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f10c4155d30 con 0x7f10c4153290 2023-03-03T14:34:39.464 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:34:39.464 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:34:39.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10a8068320 msgr2=0x7f10a806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:39.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10a8068320 0x7f10a806a7d0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009520 tx=0x7f10c00092c0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c4153290 msgr2=0x7f10c40b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:39.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c4153290 0x7f10c40b1c70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f10b4016000 tx=0x7f10b4004750 comp rx=0 tx=0).stop 2023-03-03T14:34:39.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 shutdown_connections 2023-03-03T14:34:39.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f10a8068320 0x7f10a806a7d0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10c41593c0 0x7f10c40b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.464+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10c40cd8e0 0x7f10c40b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.465+0000 7f10cc1ff700 1 --2- 172.21.15.102:0/3224453690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10c4153290 0x7f10c40b1c70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:39.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.465+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 >> 172.21.15.102:0/3224453690 conn(0x7f10c4065fb0 msgr2=0x7f10c415c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:39.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.465+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 shutdown_connections 2023-03-03T14:34:39.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:34:39.465+0000 7f10cc1ff700 1 -- 172.21.15.102:0/3224453690 wait complete. 2023-03-03T14:34:39.469 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:34:39.494 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 101 secs before thrashing 2023-03-03T14:34:42.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:42.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:13.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:13.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:43.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:43.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:13.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:13.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:20.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:36:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.766+0000 7effbadff700 1 Processor -- start 2023-03-03T14:36:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.766+0000 7effbadff700 1 -- start start 2023-03-03T14:36:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.767+0000 7effbadff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7effb40cd8e0 0x7effb40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.767+0000 7effbadff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7effb40cd390 0x7effb40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.767+0000 7effbadff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb4153660 con 0x7effb4153290 2023-03-03T14:36:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effbadff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb406c360 con 0x7effb40cd390 2023-03-03T14:36:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effbadff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb415cea0 con 0x7effb40cd8e0 2023-03-03T14:36:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb939c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7effb40cd390 0x7effb40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb939c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7effb40cd390 0x7effb40ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57202/0 (socket says 172.21.15.102:57202) 2023-03-03T14:36:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7effb40cd8e0 0x7effb40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb939c700 1 -- 172.21.15.102:0/3539994815 learned_addr learned my addr 172.21.15.102:0/3539994815 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:36:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb939c700 1 -- 172.21.15.102:0/3539994815 >> v2:172.21.15.172:3300/0 conn(0x7effb40cd8e0 msgr2=0x7effb40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb939c700 1 --2- 172.21.15.102:0/3539994815 >> v2:172.21.15.172:3300/0 conn(0x7effb40cd8e0 0x7effb40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb939c700 1 -- 172.21.15.102:0/3539994815 >> v1:172.21.15.172:6789/0 conn(0x7effb4153290 legacy=0x7effb4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb939c700 1 -- 172.21.15.102:0/3539994815 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb406c7f0 con 0x7effb40cd390 2023-03-03T14:36:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.768+0000 7effb37fe700 1 -- 172.21.15.102:0/3539994815 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945330933 0 0) 0x7effb4153660 con 0x7effb4153290 2023-03-03T14:36:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.769+0000 7effb939c700 1 --2- 172.21.15.102:0/3539994815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd390 0x7effb40ce360 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7effb4159fb0 tx=0x7effac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=371ba59c13bd6eeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.769+0000 7effb37fe700 1 -- 172.21.15.102:0/3539994815 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00e070 con 0x7effb40cd390 2023-03-03T14:36:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.769+0000 7effb37fe700 1 -- 172.21.15.102:0/3539994815 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac00b910 con 0x7effb40cd390 2023-03-03T14:36:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.769+0000 7effb37fe700 1 -- 172.21.15.102:0/3539994815 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00bcc0 con 0x7effb40cd390 2023-03-03T14:36:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 -- 172.21.15.102:0/3539994815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd390 msgr2=0x7effb40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 --2- 172.21.15.102:0/3539994815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd390 0x7effb40ce360 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7effb4159fb0 tx=0x7effac00b330 comp rx=0 tx=0).stop 2023-03-03T14:36:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 -- 172.21.15.102:0/3539994815 shutdown_connections 2023-03-03T14:36:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 --2- 172.21.15.102:0/3539994815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd390 0x7effb40ce360 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 --2- 172.21.15.102:0/3539994815 >> v2:172.21.15.172:3300/0 conn(0x7effb40cd8e0 0x7effb40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 -- 172.21.15.102:0/3539994815 >> 172.21.15.102:0/3539994815 conn(0x7effb4065fb0 msgr2=0x7effb40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 -- 172.21.15.102:0/3539994815 shutdown_connections 2023-03-03T14:36:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.770+0000 7effbadff700 1 -- 172.21.15.102:0/3539994815 wait complete. 2023-03-03T14:36:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.773+0000 7effbadff700 1 Processor -- start 2023-03-03T14:36:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.773+0000 7effbadff700 1 -- start start 2023-03-03T14:36:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.774+0000 7effbadff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effb40cd390 0x7effb4150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.774+0000 7effbadff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd8e0 0x7effb4157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.774+0000 7effbadff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb4153290 0x7effb41efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.774+0000 7effbadff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb40cfd20 con 0x7effb40cd8e0 2023-03-03T14:36:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.774+0000 7effbadff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb41585b0 con 0x7effb4153290 2023-03-03T14:36:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.774+0000 7effbadff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb41582d0 con 0x7effb40cd390 2023-03-03T14:36:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb8b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effb40cd390 0x7effb4150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd8e0 0x7effb4157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb8b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effb40cd390 0x7effb4150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57200/0 (socket says 172.21.15.102:57200) 2023-03-03T14:36:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb8b9b700 1 -- 172.21.15.102:0/1004130699 learned_addr learned my addr 172.21.15.102:0/1004130699 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:36:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb939c700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb4153290 0x7effb41efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb3fff700 1 -- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effb40cd390 msgr2=0x7effb4150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb3fff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effb40cd390 0x7effb4150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb3fff700 1 -- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb4153290 msgr2=0x7effb41efaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb3fff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb4153290 0x7effb41efaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb3fff700 1 -- 172.21.15.102:0/1004130699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac009b30 con 0x7effb40cd8e0 2023-03-03T14:36:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb8b9b700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effb40cd390 0x7effb4150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:36:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb939c700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb4153290 0x7effb41efaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:36:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.775+0000 7effb3fff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd8e0 0x7effb4157270 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7effa000e900 tx=0x7effa000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.776+0000 7effb17fa700 1 -- 172.21.15.102:0/1004130699 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0013680 con 0x7effb40cd8e0 2023-03-03T14:36:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.776+0000 7effb17fa700 1 -- 172.21.15.102:0/1004130699 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa00137e0 con 0x7effb40cd8e0 2023-03-03T14:36:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.776+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb40ccde0 con 0x7effb40cd8e0 2023-03-03T14:36:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.776+0000 7effb17fa700 1 -- 172.21.15.102:0/1004130699 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0013b50 con 0x7effb40cd8e0 2023-03-03T14:36:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.776+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb415d160 con 0x7effb40cd8e0 2023-03-03T14:36:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.777+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb40cd0c0 con 0x7effb40cd8e0 2023-03-03T14:36:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.778+0000 7effb17fa700 1 -- 172.21.15.102:0/1004130699 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7effa0019030 con 0x7effb40cd8e0 2023-03-03T14:36:20.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.780+0000 7effb17fa700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff980682f0 0x7eff9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:20.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.780+0000 7effb17fa700 1 -- 172.21.15.102:0/1004130699 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7effa0085f00 con 0x7effb40cd8e0 2023-03-03T14:36:20.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.780+0000 7effb8b9b700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff980682f0 0x7eff9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:20.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.781+0000 7effb8b9b700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff980682f0 0x7eff9806a7a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7effa8000f60 tx=0x7effa8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:20.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:20.784+0000 7effb17fa700 1 -- 172.21.15.102:0/1004130699 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effa0052610 con 0x7effb40cd8e0 2023-03-03T14:36:21.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.018+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effb415a5d0 con 0x7effb40cd8e0 2023-03-03T14:36:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.019+0000 7effb17fa700 1 -- 172.21.15.102:0/1004130699 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7effa0017020 con 0x7effb40cd8e0 2023-03-03T14:36:21.020 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:36:21.020 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:36:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff980682f0 msgr2=0x7eff9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff980682f0 0x7eff9806a7a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7effa8000f60 tx=0x7effa8009040 comp rx=0 tx=0).stop 2023-03-03T14:36:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd8e0 msgr2=0x7effb4157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd8e0 0x7effb4157270 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7effa000e900 tx=0x7effa000ec10 comp rx=0 tx=0).stop 2023-03-03T14:36:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 shutdown_connections 2023-03-03T14:36:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7eff980682f0 0x7eff9806a7a0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7effb40cd390 0x7effb4150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effb4153290 0x7effb41efaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.021+0000 7effbadff700 1 --2- 172.21.15.102:0/1004130699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effb40cd8e0 0x7effb4157270 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.022+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 >> 172.21.15.102:0/1004130699 conn(0x7effb4065fb0 msgr2=0x7effb4150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.022+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 shutdown_connections 2023-03-03T14:36:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:36:21.022+0000 7effbadff700 1 -- 172.21.15.102:0/1004130699 wait complete. 2023-03-03T14:36:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:36:21.050 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 103 secs before thrashing 2023-03-03T14:36:43.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:43.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:13.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:13.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:43.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:43.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:04.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:38:04.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.319+0000 7ffaeb9ff700 1 Processor -- start 2023-03-03T14:38:04.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.319+0000 7ffaeb9ff700 1 -- start start 2023-03-03T14:38:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.320+0000 7ffaeb9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffae4152e50 0x7ffae40c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.320+0000 7ffaeb9ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae40c9590 con 0x7ffae40c8eb0 2023-03-03T14:38:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffaeb9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae40ca400 con 0x7ffae40c9f60 2023-03-03T14:38:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffaeb9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae40cda80 con 0x7ffae4152e50 2023-03-03T14:38:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffae979b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffae4152e50 0x7ffae40c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffae8f9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffae40c8eb0 0x7ffae40ccc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42576/0 (socket says 172.21.15.102:42576) 2023-03-03T14:38:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffae8f9a700 1 -- 172.21.15.102:0/2613956191 learned_addr learned my addr 172.21.15.102:0/2613956191 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:38:04.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffae979b700 1 -- 172.21.15.102:0/2613956191 >> v1:172.21.15.172:6789/0 conn(0x7ffae40c9f60 legacy=0x7ffae40cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:04.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffae979b700 1 -- 172.21.15.102:0/2613956191 >> v1:172.21.15.102:6789/0 conn(0x7ffae40c8eb0 legacy=0x7ffae40ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:04.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffae979b700 1 -- 172.21.15.102:0/2613956191 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae4159880 con 0x7ffae4152e50 2023-03-03T14:38:04.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.321+0000 7ffadbfff700 1 -- 172.21.15.102:0/2613956191 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768325133 0 0) 0x7ffae40c9590 con 0x7ffae40c8eb0 2023-03-03T14:38:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.322+0000 7ffae979b700 1 --2- 172.21.15.102:0/2613956191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae4152e50 0x7ffae40c9a20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009840 tx=0x7ffae0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41967d8777bda7e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.322+0000 7ffadbfff700 1 -- 172.21.15.102:0/2613956191 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0011830 con 0x7ffae4152e50 2023-03-03T14:38:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.322+0000 7ffadbfff700 1 -- 172.21.15.102:0/2613956191 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0011990 con 0x7ffae4152e50 2023-03-03T14:38:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.322+0000 7ffadbfff700 1 -- 172.21.15.102:0/2613956191 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0011d20 con 0x7ffae4152e50 2023-03-03T14:38:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.322+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/2613956191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae4152e50 msgr2=0x7ffae40c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.322+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/2613956191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae4152e50 0x7ffae40c9a20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009840 tx=0x7ffae0009b50 comp rx=0 tx=0).stop 2023-03-03T14:38:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.323+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/2613956191 shutdown_connections 2023-03-03T14:38:04.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.323+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/2613956191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae4152e50 0x7ffae40c9a20 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:04.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.323+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/2613956191 >> 172.21.15.102:0/2613956191 conn(0x7ffae40704c0 msgr2=0x7ffae40708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:38:04.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.323+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/2613956191 shutdown_connections 2023-03-03T14:38:04.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.323+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/2613956191 wait complete. 2023-03-03T14:38:04.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.325+0000 7ffaeb9ff700 1 Processor -- start 2023-03-03T14:38:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.325+0000 7ffaeb9ff700 1 -- start start 2023-03-03T14:38:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffaeb9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae40c8eb0 0x7ffae414bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffaeb9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae40c9f60 0x7ffae41e5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffaeb9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae4152e50 0x7ffae41e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffaeb9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae40c9950 con 0x7ffae40c8eb0 2023-03-03T14:38:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffaeb9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae414c070 con 0x7ffae40c9f60 2023-03-03T14:38:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffaeb9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae40ac960 con 0x7ffae4152e50 2023-03-03T14:38:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffae979b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae40c8eb0 0x7ffae414bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:04.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffae8f9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae40c9f60 0x7ffae41e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffae979b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae40c8eb0 0x7ffae414bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60954/0 (socket says 172.21.15.102:60954) 2023-03-03T14:38:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffae979b700 1 -- 172.21.15.102:0/1272595095 learned_addr learned my addr 172.21.15.102:0/1272595095 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:38:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.326+0000 7ffae9f9c700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae4152e50 0x7ffae41e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffae979b700 1 -- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae4152e50 msgr2=0x7ffae41e5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffae979b700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae4152e50 0x7ffae41e5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffae979b700 1 -- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae40c9f60 msgr2=0x7ffae41e5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffae979b700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae40c9f60 0x7ffae41e5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffae979b700 1 -- 172.21.15.102:0/1272595095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae00094a0 con 0x7ffae40c8eb0 2023-03-03T14:38:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffae8f9a700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae40c9f60 0x7ffae41e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:38:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffae979b700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae40c8eb0 0x7ffae414bb30 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ffae00090b0 tx=0x7ffae0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.327+0000 7ffad9ffb700 1 -- 172.21.15.102:0/1272595095 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0024620 con 0x7ffae40c8eb0 2023-03-03T14:38:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.328+0000 7ffad9ffb700 1 -- 172.21.15.102:0/1272595095 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0024780 con 0x7ffae40c8eb0 2023-03-03T14:38:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.328+0000 7ffad9ffb700 1 -- 172.21.15.102:0/1272595095 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0024af0 con 0x7ffae40c8eb0 2023-03-03T14:38:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.328+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae40c7dc0 con 0x7ffae40c8eb0 2023-03-03T14:38:04.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.328+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae40c7ae0 con 0x7ffae40c8eb0 2023-03-03T14:38:04.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.329+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae40c80a0 con 0x7ffae40c8eb0 2023-03-03T14:38:04.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.330+0000 7ffad9ffb700 1 -- 172.21.15.102:0/1272595095 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffae002b030 con 0x7ffae40c8eb0 2023-03-03T14:38:04.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.331+0000 7ffad9ffb700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffac8068320 0x7ffac806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:04.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.331+0000 7ffad9ffb700 1 -- 172.21.15.102:0/1272595095 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ffae0086e40 con 0x7ffae40c8eb0 2023-03-03T14:38:04.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.332+0000 7ffae8f9a700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffac8068320 0x7ffac806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:04.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.332+0000 7ffae8f9a700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffac8068320 0x7ffac806a7d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7ffae41e8aa0 tx=0x7ffad4007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:04.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.335+0000 7ffad9ffb700 1 -- 172.21.15.102:0/1272595095 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffae0053600 con 0x7ffae40c8eb0 2023-03-03T14:38:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.571+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffae404e6e0 con 0x7ffae40c8eb0 2023-03-03T14:38:04.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.571+0000 7ffad9ffb700 1 -- 172.21.15.102:0/1272595095 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ffae0029090 con 0x7ffae40c8eb0 2023-03-03T14:38:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:38:04.573 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:38:04.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffac8068320 msgr2=0x7ffac806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:04.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffac8068320 0x7ffac806a7d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7ffae41e8aa0 tx=0x7ffad4007a60 comp rx=0 tx=0).stop 2023-03-03T14:38:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae40c8eb0 msgr2=0x7ffae414bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae40c8eb0 0x7ffae414bb30 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ffae00090b0 tx=0x7ffae0004750 comp rx=0 tx=0).stop 2023-03-03T14:38:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 shutdown_connections 2023-03-03T14:38:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffac8068320 0x7ffac806a7d0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffae4152e50 0x7ffae41e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffae40c8eb0 0x7ffae414bb30 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 --2- 172.21.15.102:0/1272595095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffae40c9f60 0x7ffae41e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:04.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 >> 172.21.15.102:0/1272595095 conn(0x7ffae40704c0 msgr2=0x7ffae40cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:38:04.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 shutdown_connections 2023-03-03T14:38:04.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:38:04.574+0000 7ffaeb9ff700 1 -- 172.21.15.102:0/1272595095 wait complete. 2023-03-03T14:38:04.577 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:38:04.603 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-03-03T14:38:13.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:13.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:43.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:43.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:14.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:14.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:27.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:39:27.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.866+0000 7f35ff1ff700 1 Processor -- start 2023-03-03T14:39:27.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.866+0000 7f35ff1ff700 1 -- start start 2023-03-03T14:39:27.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.867+0000 7f35ff1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f35f8153290 0x7f35f8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:27.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.867+0000 7f35ff1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f80cd8e0 0x7f35f80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:27.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.867+0000 7f35ff1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f8153660 con 0x7f35f81593c0 2023-03-03T14:39:27.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.867+0000 7f35ff1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f806c360 con 0x7f35f8153290 2023-03-03T14:39:27.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.867+0000 7f35ff1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f815cea0 con 0x7f35f80cd8e0 2023-03-03T14:39:27.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35fcf9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f35f8153290 0x7f35f8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:27.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35f7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f35f81593c0 0x7f35f80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38350/0 (socket says 172.21.15.102:38350) 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35f7fff700 1 -- 172.21.15.102:0/3394251541 learned_addr learned my addr 172.21.15.102:0/3394251541 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35fd79c700 1 --2- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f80cd8e0 0x7f35f80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35fcf9b700 1 -- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f80cd8e0 msgr2=0x7f35f80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35fcf9b700 1 --2- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f80cd8e0 0x7f35f80ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35fcf9b700 1 -- 172.21.15.102:0/3394251541 >> v1:172.21.15.102:6789/0 conn(0x7f35f81593c0 legacy=0x7f35f80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.868+0000 7f35fcf9b700 1 -- 172.21.15.102:0/3394251541 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f806c7f0 con 0x7f35f8153290 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.869+0000 7f35fcf9b700 1 --2- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f8153290 0x7f35f8158e80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009840 tx=0x7f35ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e5bfeb0fa110d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:27.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.869+0000 7f35f77fe700 1 -- 172.21.15.102:0/3394251541 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec011830 con 0x7f35f8153290 2023-03-03T14:39:27.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.869+0000 7f35f77fe700 1 -- 172.21.15.102:0/3394251541 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ec011990 con 0x7f35f8153290 2023-03-03T14:39:27.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.869+0000 7f35f77fe700 1 -- 172.21.15.102:0/3394251541 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec011d20 con 0x7f35f8153290 2023-03-03T14:39:27.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.869+0000 7f35ff1ff700 1 -- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f8153290 msgr2=0x7f35f8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:27.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.869+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f8153290 0x7f35f8158e80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009840 tx=0x7f35ec009b50 comp rx=0 tx=0).stop 2023-03-03T14:39:27.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.870+0000 7f35ff1ff700 1 -- 172.21.15.102:0/3394251541 shutdown_connections 2023-03-03T14:39:27.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.870+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f80cd8e0 0x7f35f80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:27.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.870+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/3394251541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f8153290 0x7f35f8158e80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:27.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.870+0000 7f35ff1ff700 1 -- 172.21.15.102:0/3394251541 >> 172.21.15.102:0/3394251541 conn(0x7f35f8065fb0 msgr2=0x7f35f80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:27.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.870+0000 7f35ff1ff700 1 -- 172.21.15.102:0/3394251541 shutdown_connections 2023-03-03T14:39:27.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.870+0000 7f35ff1ff700 1 -- 172.21.15.102:0/3394251541 wait complete. 2023-03-03T14:39:27.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.872+0000 7f35ff1ff700 1 Processor -- start 2023-03-03T14:39:27.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.872+0000 7f35ff1ff700 1 -- start start 2023-03-03T14:39:27.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35ff1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f80cd8e0 0x7f35f80c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:27.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35ff1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f8153290 0x7f35f8150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:27.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35ff1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f81593c0 0x7f35f80c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:27.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35ff1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c44b0 con 0x7f35f8153290 2023-03-03T14:39:27.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35ff1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c73f0 con 0x7f35f80cd8e0 2023-03-03T14:39:27.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35ff1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f8150a40 con 0x7f35f81593c0 2023-03-03T14:39:27.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35fcf9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f80cd8e0 0x7f35f80c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:27.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35f7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f8153290 0x7f35f8150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:27.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35fcf9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f80cd8e0 0x7f35f80c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34636/0 (socket says 172.21.15.102:34636) 2023-03-03T14:39:27.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35fcf9b700 1 -- 172.21.15.102:0/436906537 learned_addr learned my addr 172.21.15.102:0/436906537 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:39:27.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.873+0000 7f35fd79c700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f81593c0 0x7f35f80c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:27.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35fcf9b700 1 -- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f81593c0 msgr2=0x7f35f80c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:27.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35fcf9b700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f81593c0 0x7f35f80c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:27.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35fcf9b700 1 -- 172.21.15.102:0/436906537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f8153290 msgr2=0x7f35f8150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:27.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35fcf9b700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f8153290 0x7f35f8150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:27.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35fcf9b700 1 -- 172.21.15.102:0/436906537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0094a0 con 0x7f35f80cd8e0 2023-03-03T14:39:27.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35fcf9b700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f80cd8e0 0x7f35f80c3f70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009fd0 tx=0x7f35ec004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:27.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35f57fa700 1 -- 172.21.15.102:0/436906537 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec0036a0 con 0x7f35f80cd8e0 2023-03-03T14:39:27.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35f57fa700 1 -- 172.21.15.102:0/436906537 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ec01e070 con 0x7f35f80cd8e0 2023-03-03T14:39:27.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35f57fa700 1 -- 172.21.15.102:0/436906537 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec003a40 con 0x7f35f80cd8e0 2023-03-03T14:39:27.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.874+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f80ccde0 con 0x7f35f80cd8e0 2023-03-03T14:39:27.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.875+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f815d160 con 0x7f35f80cd8e0 2023-03-03T14:39:27.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.876+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f80cd0c0 con 0x7f35f80cd8e0 2023-03-03T14:39:27.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.877+0000 7f35f57fa700 1 -- 172.21.15.102:0/436906537 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f35ec00f050 con 0x7f35f80cd8e0 2023-03-03T14:39:27.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.878+0000 7f35f57fa700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35dc0682f0 0x7f35dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:27.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.878+0000 7f35f57fa700 1 -- 172.21.15.102:0/436906537 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f35ec086cc0 con 0x7f35f80cd8e0 2023-03-03T14:39:27.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.878+0000 7f35f7fff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35dc0682f0 0x7f35dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:27.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.879+0000 7f35f7fff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35dc0682f0 0x7f35dc06a7a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f35f80cbb60 tx=0x7f35e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:27.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:27.883+0000 7f35f57fa700 1 -- 172.21.15.102:0/436906537 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35ec08d080 con 0x7f35f80cd8e0 2023-03-03T14:39:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.116+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35f815a8e0 con 0x7f35f80cd8e0 2023-03-03T14:39:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.117+0000 7f35f57fa700 1 -- 172.21.15.102:0/436906537 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f35ec015020 con 0x7f35f80cd8e0 2023-03-03T14:39:28.119 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:39:28.119 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:39:28.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35dc0682f0 msgr2=0x7f35dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35dc0682f0 0x7f35dc06a7a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f35f80cbb60 tx=0x7f35e4009040 comp rx=0 tx=0).stop 2023-03-03T14:39:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f80cd8e0 msgr2=0x7f35f80c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f80cd8e0 0x7f35f80c3f70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009fd0 tx=0x7f35ec004750 comp rx=0 tx=0).stop 2023-03-03T14:39:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 shutdown_connections 2023-03-03T14:39:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35dc0682f0 0x7f35dc06a7a0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35f81593c0 0x7f35f80c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35f80cd8e0 0x7f35f80c3f70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 --2- 172.21.15.102:0/436906537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f8153290 0x7f35f8150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 >> 172.21.15.102:0/436906537 conn(0x7f35f8065fb0 msgr2=0x7f35f80b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 shutdown_connections 2023-03-03T14:39:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:28.119+0000 7f35ff1ff700 1 -- 172.21.15.102:0/436906537 wait complete. 2023-03-03T14:39:28.124 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:39:28.149 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2023-03-03T14:39:44.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:44.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:45.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:39:45.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.397+0000 7fd631d9d700 1 Processor -- start 2023-03-03T14:39:45.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.397+0000 7fd631d9d700 1 -- start start 2023-03-03T14:39:45.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.399+0000 7fd631d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd62c0ca200 0x7fd62c164370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:45.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.399+0000 7fd631d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c0c8b40 con 0x7fd62c152ea0 2023-03-03T14:39:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.399+0000 7fd631d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c0b4580 con 0x7fd62c0cf910 2023-03-03T14:39:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.399+0000 7fd631d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62c0c9b40 con 0x7fd62c0ca200 2023-03-03T14:39:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.399+0000 7fd62bfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd62c0cf910 0x7fd62c0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52694/0 (socket says 172.21.15.102:52694) 2023-03-03T14:39:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.399+0000 7fd62bfff700 1 -- 172.21.15.102:0/1982748445 learned_addr learned my addr 172.21.15.102:0/1982748445 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:39:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.399+0000 7fd63159c700 1 --2- 172.21.15.102:0/1982748445 >> v2:172.21.15.102:3300/0 conn(0x7fd62c0ca200 0x7fd62c164370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:45.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd62b7fe700 1 -- 172.21.15.102:0/1982748445 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129714735 0 0) 0x7fd62c0b4580 con 0x7fd62c0cf910 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd62b7fe700 1 -- 172.21.15.102:0/1982748445 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c0035b0 con 0x7fd62c0cf910 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd62b7fe700 1 -- 172.21.15.102:0/1982748445 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440382000 0 0) 0x7fd62c0c8b40 con 0x7fd62c152ea0 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd62b7fe700 1 -- 172.21.15.102:0/1982748445 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd62c0b4580 con 0x7fd62c152ea0 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd63159c700 1 -- 172.21.15.102:0/1982748445 >> v1:172.21.15.172:6789/0 conn(0x7fd62c0cf910 legacy=0x7fd62c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd63159c700 1 -- 172.21.15.102:0/1982748445 >> v1:172.21.15.102:6789/0 conn(0x7fd62c152ea0 legacy=0x7fd62c158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd63159c700 1 -- 172.21.15.102:0/1982748445 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c0c8fd0 con 0x7fd62c0ca200 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd63159c700 1 --2- 172.21.15.102:0/1982748445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c0ca200 0x7fd62c164370 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd624009d90 tx=0x7fd62400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d5669eed33b218 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:45.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd62b7fe700 1 -- 172.21.15.102:0/1982748445 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62400e070 con 0x7fd62c0ca200 2023-03-03T14:39:45.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.400+0000 7fd62b7fe700 1 -- 172.21.15.102:0/1982748445 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62400bb20 con 0x7fd62c0ca200 2023-03-03T14:39:45.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd62b7fe700 1 -- 172.21.15.102:0/1982748445 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624012430 con 0x7fd62c0ca200 2023-03-03T14:39:45.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd631d9d700 1 -- 172.21.15.102:0/1982748445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c0ca200 msgr2=0x7fd62c164370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:45.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd631d9d700 1 --2- 172.21.15.102:0/1982748445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c0ca200 0x7fd62c164370 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd624009d90 tx=0x7fd62400b330 comp rx=0 tx=0).stop 2023-03-03T14:39:45.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd631d9d700 1 -- 172.21.15.102:0/1982748445 shutdown_connections 2023-03-03T14:39:45.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd631d9d700 1 --2- 172.21.15.102:0/1982748445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c0ca200 0x7fd62c164370 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd631d9d700 1 -- 172.21.15.102:0/1982748445 >> 172.21.15.102:0/1982748445 conn(0x7fd62c066c40 msgr2=0x7fd62c0cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd631d9d700 1 -- 172.21.15.102:0/1982748445 shutdown_connections 2023-03-03T14:39:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.401+0000 7fd631d9d700 1 -- 172.21.15.102:0/1982748445 wait complete. 2023-03-03T14:39:45.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.403+0000 7fd631d9d700 1 Processor -- start 2023-03-03T14:39:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.403+0000 7fd631d9d700 1 -- start start 2023-03-03T14:39:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.404+0000 7fd631d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd62c0ca200 0x7fd62c1f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.404+0000 7fd631d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd62c0cf910 0x7fd62c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.404+0000 7fd631d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c152ea0 0x7fd62c14cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.404+0000 7fd631d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c1ed3b0 con 0x7fd62c152ea0 2023-03-03T14:39:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.404+0000 7fd631d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c156960 con 0x7fd62c0ca200 2023-03-03T14:39:45.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.404+0000 7fd631d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c1f23e0 con 0x7fd62c0cf910 2023-03-03T14:39:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd62c0ca200 0x7fd62c1f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd62bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd62c0cf910 0x7fd62c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd62c0ca200 0x7fd62c1f1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52470/0 (socket says 172.21.15.102:52470) 2023-03-03T14:39:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 -- 172.21.15.102:0/2752743942 learned_addr learned my addr 172.21.15.102:0/2752743942 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:39:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd63159c700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c152ea0 0x7fd62c14cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 -- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd62c0cf910 msgr2=0x7fd62c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:45.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd62c0cf910 0x7fd62c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 -- 172.21.15.102:0/2752743942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c152ea0 msgr2=0x7fd62c14cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c152ea0 0x7fd62c14cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.405+0000 7fd630d9b700 1 -- 172.21.15.102:0/2752743942 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624009a70 con 0x7fd62c0ca200 2023-03-03T14:39:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.406+0000 7fd630d9b700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd62c0ca200 0x7fd62c1f1ab0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd62000c500 tx=0x7fd62000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.406+0000 7fd6297fa700 1 -- 172.21.15.102:0/2752743942 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6200052d0 con 0x7fd62c0ca200 2023-03-03T14:39:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.406+0000 7fd6297fa700 1 -- 172.21.15.102:0/2752743942 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd620005450 con 0x7fd62c0ca200 2023-03-03T14:39:45.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.406+0000 7fd6297fa700 1 -- 172.21.15.102:0/2752743942 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6200038c0 con 0x7fd62c0ca200 2023-03-03T14:39:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.406+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c0c9580 con 0x7fd62c0ca200 2023-03-03T14:39:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.406+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c0b4840 con 0x7fd62c0ca200 2023-03-03T14:39:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.407+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62c0c9860 con 0x7fd62c0ca200 2023-03-03T14:39:45.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.408+0000 7fd6297fa700 1 -- 172.21.15.102:0/2752743942 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd620019030 con 0x7fd62c0ca200 2023-03-03T14:39:45.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.410+0000 7fd6297fa700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6100682f0 0x7fd61006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:45.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.410+0000 7fd6297fa700 1 -- 172.21.15.102:0/2752743942 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd620085020 con 0x7fd62c0ca200 2023-03-03T14:39:45.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.410+0000 7fd62bfff700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6100682f0 0x7fd61006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:45.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.411+0000 7fd62bfff700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6100682f0 0x7fd61006a7a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd62c156f50 tx=0x7fd618007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:45.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.414+0000 7fd6297fa700 1 -- 172.21.15.102:0/2752743942 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd620051730 con 0x7fd62c0ca200 2023-03-03T14:39:45.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.645+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd62c0ca870 con 0x7fd62c0ca200 2023-03-03T14:39:45.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.646+0000 7fd6297fa700 1 -- 172.21.15.102:0/2752743942 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd620017090 con 0x7fd62c0ca200 2023-03-03T14:39:45.648 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:39:45.648 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:39:45.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6100682f0 msgr2=0x7fd61006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:45.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6100682f0 0x7fd61006a7a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd62c156f50 tx=0x7fd618007de0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd62c0ca200 msgr2=0x7fd62c1f1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:45.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd62c0ca200 0x7fd62c1f1ab0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd62000c500 tx=0x7fd62000c810 comp rx=0 tx=0).stop 2023-03-03T14:39:45.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 shutdown_connections 2023-03-03T14:39:45.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd62c152ea0 0x7fd62c14cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd62c0ca200 0x7fd62c1f1ab0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6100682f0 0x7fd61006a7a0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 --2- 172.21.15.102:0/2752743942 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd62c0cf910 0x7fd62c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:45.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 >> 172.21.15.102:0/2752743942 conn(0x7fd62c066c40 msgr2=0x7fd62c153270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:45.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 shutdown_connections 2023-03-03T14:39:45.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:39:45.649+0000 7fd631d9d700 1 -- 172.21.15.102:0/2752743942 wait complete. 2023-03-03T14:39:45.652 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:39:45.678 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T14:40:00.002 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T14:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:40:14.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:14.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:44.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:44.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:14.825 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:14.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:23.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:41:23.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.947+0000 7ff2727ff700 1 Processor -- start 2023-03-03T14:41:23.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.947+0000 7ff2727ff700 1 -- start start 2023-03-03T14:41:23.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.948+0000 7ff2727ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff26c153290 0x7ff26c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:23.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.948+0000 7ff2727ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c1593c0 0x7ff26c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:23.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.948+0000 7ff2727ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c153660 con 0x7ff26c0cd8e0 2023-03-03T14:41:23.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.948+0000 7ff2727ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26c06c360 con 0x7ff26c153290 2023-03-03T14:41:23.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.948+0000 7ff2727ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c15cea0 con 0x7ff26c1593c0 2023-03-03T14:41:23.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff26bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff26c153290 0x7ff26c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:23.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff270d9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff26c0cd8e0 0x7ff26c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47070/0 (socket says 172.21.15.102:47070) 2023-03-03T14:41:23.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff270d9c700 1 -- 172.21.15.102:0/3755582801 learned_addr learned my addr 172.21.15.102:0/3755582801 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:41:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff263fff700 1 --2- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c1593c0 0x7ff26c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff26bfff700 1 -- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c1593c0 msgr2=0x7ff26c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff26bfff700 1 --2- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c1593c0 0x7ff26c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff26bfff700 1 -- 172.21.15.102:0/3755582801 >> v1:172.21.15.172:6789/0 conn(0x7ff26c0cd8e0 legacy=0x7ff26c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff26bfff700 1 -- 172.21.15.102:0/3755582801 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c06c7f0 con 0x7ff26c153290 2023-03-03T14:41:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff26b7fe700 1 -- 172.21.15.102:0/3755582801 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958623574 0 0) 0x7ff26c153660 con 0x7ff26c0cd8e0 2023-03-03T14:41:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.949+0000 7ff26bfff700 1 --2- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c158e80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009840 tx=0x7ff25c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3d900b9881cee62 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff26b7fe700 1 -- 172.21.15.102:0/3755582801 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c011830 con 0x7ff26c153290 2023-03-03T14:41:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff26b7fe700 1 -- 172.21.15.102:0/3755582801 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c011990 con 0x7ff26c153290 2023-03-03T14:41:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff26b7fe700 1 -- 172.21.15.102:0/3755582801 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c011d20 con 0x7ff26c153290 2023-03-03T14:41:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff2727ff700 1 -- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 msgr2=0x7ff26c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff2727ff700 1 --2- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c158e80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009840 tx=0x7ff25c009b50 comp rx=0 tx=0).stop 2023-03-03T14:41:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff2727ff700 1 -- 172.21.15.102:0/3755582801 shutdown_connections 2023-03-03T14:41:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff2727ff700 1 --2- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c158e80 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff2727ff700 1 --2- 172.21.15.102:0/3755582801 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c1593c0 0x7ff26c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff2727ff700 1 -- 172.21.15.102:0/3755582801 >> 172.21.15.102:0/3755582801 conn(0x7ff26c065fb0 msgr2=0x7ff26c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.950+0000 7ff2727ff700 1 -- 172.21.15.102:0/3755582801 shutdown_connections 2023-03-03T14:41:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.951+0000 7ff2727ff700 1 -- 172.21.15.102:0/3755582801 wait complete. 2023-03-03T14:41:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.952+0000 7ff2727ff700 1 Processor -- start 2023-03-03T14:41:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.953+0000 7ff2727ff700 1 -- start start 2023-03-03T14:41:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff2727ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c0cd8e0 0x7ff26c0c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff2727ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c14e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff2727ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26c1593c0 0x7ff26c14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff2727ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c14dad0 con 0x7ff26c1593c0 2023-03-03T14:41:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff2727ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c14c100 con 0x7ff26c153290 2023-03-03T14:41:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff2727ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c14c3a0 con 0x7ff26c0cd8e0 2023-03-03T14:41:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff270d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26c1593c0 0x7ff26c14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff263fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c14e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff26bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c0cd8e0 0x7ff26c0c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff263fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c14e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45642/0 (socket says 172.21.15.102:45642) 2023-03-03T14:41:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.954+0000 7ff263fff700 1 -- 172.21.15.102:0/2245517402 learned_addr learned my addr 172.21.15.102:0/2245517402 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:41:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff263fff700 1 -- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c0cd8e0 msgr2=0x7ff26c0c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:23.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff263fff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c0cd8e0 0x7ff26c0c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:23.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff263fff700 1 -- 172.21.15.102:0/2245517402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26c1593c0 msgr2=0x7ff26c14e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:23.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff263fff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26c1593c0 0x7ff26c14e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:23.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff263fff700 1 -- 172.21.15.102:0/2245517402 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0094a0 con 0x7ff26c153290 2023-03-03T14:41:23.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff26bfff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c0cd8e0 0x7ff26c0c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:41:23.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff263fff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c14e260 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff25400e9b0 tx=0x7ff25400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:23.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff2697fa700 1 -- 172.21.15.102:0/2245517402 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254009710 con 0x7ff26c153290 2023-03-03T14:41:23.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff2697fa700 1 -- 172.21.15.102:0/2245517402 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25400ee50 con 0x7ff26c153290 2023-03-03T14:41:23.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff2697fa700 1 -- 172.21.15.102:0/2245517402 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400ca50 con 0x7ff26c153290 2023-03-03T14:41:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.955+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c0ccde0 con 0x7ff26c153290 2023-03-03T14:41:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.956+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c15d160 con 0x7ff26c153290 2023-03-03T14:41:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.957+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26c0cd0c0 con 0x7ff26c153290 2023-03-03T14:41:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.958+0000 7ff2697fa700 1 -- 172.21.15.102:0/2245517402 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff254019030 con 0x7ff26c153290 2023-03-03T14:41:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.959+0000 7ff2697fa700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff24c0682f0 0x7ff24c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.959+0000 7ff2697fa700 1 -- 172.21.15.102:0/2245517402 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff254085d10 con 0x7ff26c153290 2023-03-03T14:41:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.959+0000 7ff26bfff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff24c0682f0 0x7ff24c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.960+0000 7ff26bfff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff24c0682f0 0x7ff24c06a7a0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0090b0 tx=0x7ff25c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:23.963+0000 7ff2697fa700 1 -- 172.21.15.102:0/2245517402 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff25408b020 con 0x7ff26c153290 2023-03-03T14:41:24.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.199+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff26c04e6e0 con 0x7ff26c153290 2023-03-03T14:41:24.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.200+0000 7ff2697fa700 1 -- 172.21.15.102:0/2245517402 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff254021090 con 0x7ff26c153290 2023-03-03T14:41:24.201 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:41:24.201 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:41:24.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff24c0682f0 msgr2=0x7ff24c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:24.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff24c0682f0 0x7ff24c06a7a0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0090b0 tx=0x7ff25c008030 comp rx=0 tx=0).stop 2023-03-03T14:41:24.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 msgr2=0x7ff26c14e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:24.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c14e260 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff25400e9b0 tx=0x7ff25400ecc0 comp rx=0 tx=0).stop 2023-03-03T14:41:24.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 shutdown_connections 2023-03-03T14:41:24.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff24c0682f0 0x7ff24c06a7a0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:24.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff26c0cd8e0 0x7ff26c0c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:24.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26c1593c0 0x7ff26c14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:24.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 --2- 172.21.15.102:0/2245517402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff26c153290 0x7ff26c14e260 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:24.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 >> 172.21.15.102:0/2245517402 conn(0x7ff26c065fb0 msgr2=0x7ff26c0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:24.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.202+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 shutdown_connections 2023-03-03T14:41:24.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:24.203+0000 7ff2727ff700 1 -- 172.21.15.102:0/2245517402 wait complete. 2023-03-03T14:41:24.206 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:41:24.231 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T14:41:44.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:44.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:59.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:41:59.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.480+0000 7f3d1759e700 1 Processor -- start 2023-03-03T14:41:59.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.481+0000 7f3d1759e700 1 -- start start 2023-03-03T14:41:59.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d1759e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3d180c9610 0x7f3d180ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:59.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d1759e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d180c9ca0 con 0x7f3d180ca980 2023-03-03T14:41:59.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d1759e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d180b46f0 con 0x7f3d181564b0 2023-03-03T14:41:59.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d1759e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d180b4920 con 0x7f3d180c9610 2023-03-03T14:41:59.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d15d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3d180c9610 0x7f3d180ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:59.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d1659c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3d181564b0 0x7f3d180cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50626/0 (socket says 172.21.15.102:50626) 2023-03-03T14:41:59.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d1659c700 1 -- 172.21.15.102:0/3882198961 learned_addr learned my addr 172.21.15.102:0/3882198961 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:41:59.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.482+0000 7f3d16d9d700 1 --1- 172.21.15.102:0/3882198961 >> v1:172.21.15.102:6789/0 conn(0x7f3d180ca980 0x7f3d181636a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36640/0 (socket says 172.21.15.102:36640) 2023-03-03T14:41:59.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.483+0000 7f3d1559a700 1 -- 172.21.15.102:0/3882198961 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063734545 0 0) 0x7f3d180b46f0 con 0x7f3d181564b0 2023-03-03T14:41:59.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.483+0000 7f3d1559a700 1 -- 172.21.15.102:0/3882198961 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d000035b0 con 0x7f3d181564b0 2023-03-03T14:41:59.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.483+0000 7f3d15d9b700 1 -- 172.21.15.102:0/3882198961 >> v1:172.21.15.172:6789/0 conn(0x7f3d181564b0 legacy=0x7f3d180cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:59.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.483+0000 7f3d15d9b700 1 -- 172.21.15.102:0/3882198961 >> v1:172.21.15.102:6789/0 conn(0x7f3d180ca980 legacy=0x7f3d181636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:59.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.483+0000 7f3d15d9b700 1 -- 172.21.15.102:0/3882198961 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d180b4c00 con 0x7f3d180c9610 2023-03-03T14:41:59.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.483+0000 7f3d15d9b700 1 --2- 172.21.15.102:0/3882198961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d180c9610 0x7f3d180ca440 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009840 tx=0x7f3d0c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7edba1bc8f424201 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:59.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.483+0000 7f3d1559a700 1 -- 172.21.15.102:0/3882198961 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c011830 con 0x7f3d180c9610 2023-03-03T14:41:59.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1559a700 1 -- 172.21.15.102:0/3882198961 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c011990 con 0x7f3d180c9610 2023-03-03T14:41:59.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1559a700 1 -- 172.21.15.102:0/3882198961 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c011d20 con 0x7f3d180c9610 2023-03-03T14:41:59.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1759e700 1 -- 172.21.15.102:0/3882198961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d180c9610 msgr2=0x7f3d180ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:59.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1759e700 1 --2- 172.21.15.102:0/3882198961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d180c9610 0x7f3d180ca440 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009840 tx=0x7f3d0c009b50 comp rx=0 tx=0).stop 2023-03-03T14:41:59.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1759e700 1 -- 172.21.15.102:0/3882198961 shutdown_connections 2023-03-03T14:41:59.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1759e700 1 --2- 172.21.15.102:0/3882198961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d180c9610 0x7f3d180ca440 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1759e700 1 -- 172.21.15.102:0/3882198961 >> 172.21.15.102:0/3882198961 conn(0x7f3d18065fb0 msgr2=0x7f3d180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:59.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1759e700 1 -- 172.21.15.102:0/3882198961 shutdown_connections 2023-03-03T14:41:59.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.484+0000 7f3d1759e700 1 -- 172.21.15.102:0/3882198961 wait complete. 2023-03-03T14:41:59.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.486+0000 7f3d1759e700 1 Processor -- start 2023-03-03T14:41:59.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.486+0000 7f3d1759e700 1 -- start start 2023-03-03T14:41:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.487+0000 7f3d1759e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180c9610 0x7f3d18158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.487+0000 7f3d1759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180ca980 0x7f3d18156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.487+0000 7f3d1759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d181564b0 0x7f3d180c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1759e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d181574a0 con 0x7f3d180ca980 2023-03-03T14:41:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d181ee0b0 con 0x7f3d181564b0 2023-03-03T14:41:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1759e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d181ee390 con 0x7f3d180c9610 2023-03-03T14:41:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180c9610 0x7f3d18158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d16d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d181564b0 0x7f3d180c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180c9610 0x7f3d18158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:47710/0 (socket says 172.21.15.102:47710) 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 -- 172.21.15.102:0/992977078 learned_addr learned my addr 172.21.15.102:0/992977078 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d15d9b700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180ca980 0x7f3d18156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 -- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d181564b0 msgr2=0x7f3d180c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d181564b0 0x7f3d180c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 -- 172.21.15.102:0/992977078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180ca980 msgr2=0x7f3d18156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180ca980 0x7f3d18156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.488+0000 7f3d1659c700 1 -- 172.21.15.102:0/992977078 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0094a0 con 0x7f3d180c9610 2023-03-03T14:41:59.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.489+0000 7f3d1659c700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180c9610 0x7f3d18158e70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3d08001eb0 tx=0x7f3d0800c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:59.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.489+0000 7f3d06ffd700 1 -- 172.21.15.102:0/992977078 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08005d70 con 0x7f3d180c9610 2023-03-03T14:41:59.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.489+0000 7f3d06ffd700 1 -- 172.21.15.102:0/992977078 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d080070d0 con 0x7f3d180c9610 2023-03-03T14:41:59.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.489+0000 7f3d06ffd700 1 -- 172.21.15.102:0/992977078 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08007460 con 0x7f3d180c9610 2023-03-03T14:41:59.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.489+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d180b2210 con 0x7f3d180c9610 2023-03-03T14:41:59.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.489+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d180b1f30 con 0x7f3d180c9610 2023-03-03T14:41:59.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.490+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d18156880 con 0x7f3d180c9610 2023-03-03T14:41:59.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.491+0000 7f3d06ffd700 1 -- 172.21.15.102:0/992977078 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d08019030 con 0x7f3d180c9610 2023-03-03T14:41:59.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.493+0000 7f3d06ffd700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cf80682f0 0x7f3cf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:59.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.493+0000 7f3d06ffd700 1 -- 172.21.15.102:0/992977078 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3d08085120 con 0x7f3d180c9610 2023-03-03T14:41:59.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.493+0000 7f3d15d9b700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cf80682f0 0x7f3cf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.494+0000 7f3d15d9b700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cf80682f0 0x7f3cf806a7a0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f3d180c9a30 tx=0x7f3d0c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:59.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.497+0000 7f3d06ffd700 1 -- 172.21.15.102:0/992977078 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d080517a0 con 0x7f3d180c9610 2023-03-03T14:41:59.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.731+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d181ee4d0 con 0x7f3d180c9610 2023-03-03T14:41:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.732+0000 7f3d06ffd700 1 -- 172.21.15.102:0/992977078 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3d181ee4d0 con 0x7f3d180c9610 2023-03-03T14:41:59.734 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:41:59.734 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:41:59.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cf80682f0 msgr2=0x7f3cf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:59.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cf80682f0 0x7f3cf806a7a0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f3d180c9a30 tx=0x7f3d0c008030 comp rx=0 tx=0).stop 2023-03-03T14:41:59.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180c9610 msgr2=0x7f3d18158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:59.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180c9610 0x7f3d18158e70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3d08001eb0 tx=0x7f3d0800c7b0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 shutdown_connections 2023-03-03T14:41:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cf80682f0 0x7f3cf806a7a0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180c9610 0x7f3d18158e70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d181564b0 0x7f3d180c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 --2- 172.21.15.102:0/992977078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180ca980 0x7f3d18156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 >> 172.21.15.102:0/992977078 conn(0x7f3d18065fb0 msgr2=0x7f3d18153510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 shutdown_connections 2023-03-03T14:41:59.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:41:59.735+0000 7f3d1759e700 1 -- 172.21.15.102:0/992977078 wait complete. 2023-03-03T14:41:59.739 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:41:59.763 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-03-03T14:42:15.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:15.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:45.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:45.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:15.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:15.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:45.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:45.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:48.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:43:49.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.019+0000 7f04ea7ff700 1 Processor -- start 2023-03-03T14:43:49.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.019+0000 7f04ea7ff700 1 -- start start 2023-03-03T14:43:49.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.020+0000 7f04ea7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04e40cd8e0 0x7f04e40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:49.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.020+0000 7f04ea7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd390 0x7f04e40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:49.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.020+0000 7f04ea7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e4153660 con 0x7f04e4153290 2023-03-03T14:43:49.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.020+0000 7f04ea7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e406c360 con 0x7f04e40cd8e0 2023-03-03T14:43:49.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.020+0000 7f04ea7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e415cea0 con 0x7f04e40cd390 2023-03-03T14:43:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e37fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04e40cd8e0 0x7f04e40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e3fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f04e4153290 0x7f04e4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33168/0 (socket says 172.21.15.102:33168) 2023-03-03T14:43:49.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e8d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd390 0x7f04e40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e3fff700 1 -- 172.21.15.102:0/2856499282 learned_addr learned my addr 172.21.15.102:0/2856499282 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e8d9c700 1 -- 172.21.15.102:0/2856499282 >> v2:172.21.15.102:3300/0 conn(0x7f04e40cd8e0 msgr2=0x7f04e40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e8d9c700 1 --2- 172.21.15.102:0/2856499282 >> v2:172.21.15.102:3300/0 conn(0x7f04e40cd8e0 0x7f04e40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e8d9c700 1 -- 172.21.15.102:0/2856499282 >> v1:172.21.15.172:6789/0 conn(0x7f04e4153290 legacy=0x7f04e4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e2ffd700 1 -- 172.21.15.102:0/2856499282 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153537381 0 0) 0x7f04e4153660 con 0x7f04e4153290 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e8d9c700 1 -- 172.21.15.102:0/2856499282 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e406c7f0 con 0x7f04e40cd390 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.021+0000 7f04e8d9c700 1 --2- 172.21.15.102:0/2856499282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd390 0x7f04e40ce360 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009d90 tx=0x7f04dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb69a9cdde72e0b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:49.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04e2ffd700 1 -- 172.21.15.102:0/2856499282 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00e070 con 0x7f04e40cd390 2023-03-03T14:43:49.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04e2ffd700 1 -- 172.21.15.102:0/2856499282 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc00bb20 con 0x7f04e40cd390 2023-03-03T14:43:49.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04e2ffd700 1 -- 172.21.15.102:0/2856499282 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc012430 con 0x7f04e40cd390 2023-03-03T14:43:49.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 -- 172.21.15.102:0/2856499282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd390 msgr2=0x7f04e40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:49.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/2856499282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd390 0x7f04e40ce360 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009d90 tx=0x7f04dc00b330 comp rx=0 tx=0).stop 2023-03-03T14:43:49.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 -- 172.21.15.102:0/2856499282 shutdown_connections 2023-03-03T14:43:49.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/2856499282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd390 0x7f04e40ce360 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/2856499282 >> v2:172.21.15.102:3300/0 conn(0x7f04e40cd8e0 0x7f04e40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 -- 172.21.15.102:0/2856499282 >> 172.21.15.102:0/2856499282 conn(0x7f04e4065fb0 msgr2=0x7f04e40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:49.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 -- 172.21.15.102:0/2856499282 shutdown_connections 2023-03-03T14:43:49.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.022+0000 7f04ea7ff700 1 -- 172.21.15.102:0/2856499282 wait complete. 2023-03-03T14:43:49.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.024+0000 7f04ea7ff700 1 Processor -- start 2023-03-03T14:43:49.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.024+0000 7f04ea7ff700 1 -- start start 2023-03-03T14:43:49.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.025+0000 7f04ea7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e40cd390 0x7f04e40b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.025+0000 7f04ea7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd8e0 0x7f04e41e5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.025+0000 7f04ea7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e4153290 0x7f04e4155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.025+0000 7f04ea7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e40b29b0 con 0x7f04e4153290 2023-03-03T14:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.025+0000 7f04ea7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e41f2330 con 0x7f04e40cd390 2023-03-03T14:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04ea7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e40b2be0 con 0x7f04e40cd8e0 2023-03-03T14:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e4153290 0x7f04e4155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e40cd390 0x7f04e40b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:49.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e4153290 0x7f04e4155f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48632/0 (socket says 172.21.15.102:48632) 2023-03-03T14:43:49.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd8e0 0x7f04e41e5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:49.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 -- 172.21.15.102:0/3799838870 learned_addr learned my addr 172.21.15.102:0/3799838870 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:43:49.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 -- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd8e0 msgr2=0x7f04e41e5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:49.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd8e0 0x7f04e41e5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 -- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e40cd390 msgr2=0x7f04e40b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:49.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e40cd390 0x7f04e40b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.026+0000 7f04e8d9c700 1 -- 172.21.15.102:0/3799838870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc009a70 con 0x7f04e4153290 2023-03-03T14:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.027+0000 7f04e8d9c700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e4153290 0x7f04e4155f10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f04dc014040 tx=0x7f04dc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.027+0000 7f04e0ff9700 1 -- 172.21.15.102:0/3799838870 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc012430 con 0x7f04e4153290 2023-03-03T14:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.027+0000 7f04e0ff9700 1 -- 172.21.15.102:0/3799838870 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc020070 con 0x7f04e4153290 2023-03-03T14:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.027+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e40ccde0 con 0x7f04e4153290 2023-03-03T14:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.027+0000 7f04e0ff9700 1 -- 172.21.15.102:0/3799838870 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc004280 con 0x7f04e4153290 2023-03-03T14:43:49.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.027+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e415d160 con 0x7f04e4153290 2023-03-03T14:43:49.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.028+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e40cd0c0 con 0x7f04e4153290 2023-03-03T14:43:49.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.029+0000 7f04e0ff9700 1 -- 172.21.15.102:0/3799838870 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f04dc02e070 con 0x7f04e4153290 2023-03-03T14:43:49.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.031+0000 7f04e0ff9700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04d00682f0 0x7f04d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:49.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.031+0000 7f04e0ff9700 1 -- 172.21.15.102:0/3799838870 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f04dc022070 con 0x7f04e4153290 2023-03-03T14:43:49.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.031+0000 7f04e3fff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04d00682f0 0x7f04d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:49.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.032+0000 7f04e3fff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04d00682f0 0x7f04d006a7a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f04d4007fd0 tx=0x7f04d4007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:49.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.036+0000 7f04e0ff9700 1 -- 172.21.15.102:0/3799838870 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04dc0540d0 con 0x7f04e4153290 2023-03-03T14:43:49.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.268+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04e41f24a0 con 0x7f04e4153290 2023-03-03T14:43:49.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.268+0000 7f04e0ff9700 1 -- 172.21.15.102:0/3799838870 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f04dc0576f0 con 0x7f04e4153290 2023-03-03T14:43:49.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:43:49.270 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:43:49.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04d00682f0 msgr2=0x7f04d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:49.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04d00682f0 0x7f04d006a7a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f04d4007fd0 tx=0x7f04d4007de0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e4153290 msgr2=0x7f04e4155f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:49.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e4153290 0x7f04e4155f10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f04dc014040 tx=0x7f04dc004ad0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 shutdown_connections 2023-03-03T14:43:49.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e4153290 0x7f04e4155f10 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04e40cd390 0x7f04e40b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f04d00682f0 0x7f04d006a7a0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 --2- 172.21.15.102:0/3799838870 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f04e40cd8e0 0x7f04e41e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:49.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 >> 172.21.15.102:0/3799838870 conn(0x7f04e4065fb0 msgr2=0x7f04e41582f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:49.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 shutdown_connections 2023-03-03T14:43:49.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:43:49.271+0000 7f04ea7ff700 1 -- 172.21.15.102:0/3799838870 wait complete. 2023-03-03T14:43:49.274 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:43:49.301 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 11 secs before thrashing 2023-03-03T14:44:00.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:44:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.552+0000 7ff9b17ff700 1 Processor -- start 2023-03-03T14:44:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.553+0000 7ff9b17ff700 1 -- start start 2023-03-03T14:44:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9b17ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9ac153290 0x7ff9ac158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:00.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9b17ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac1593c0 0x7ff9ac0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9b17ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac153660 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9b17ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac06c360 con 0x7ff9ac153290 2023-03-03T14:44:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9b17ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac15cea0 con 0x7ff9ac1593c0 2023-03-03T14:44:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9aaffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9ac153290 0x7ff9ac158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9aaffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9ac153290 0x7ff9ac158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59260/0 (socket says 172.21.15.102:59260) 2023-03-03T14:44:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1298877716 learned_addr learned my addr 172.21.15.102:0/1298877716 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:44:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.554+0000 7ff9a3fff700 1 --2- 172.21.15.102:0/1298877716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac1593c0 0x7ff9ac0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.555+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1298877716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac1593c0 msgr2=0x7ff9ac0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:44:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.555+0000 7ff9aaffd700 1 --2- 172.21.15.102:0/1298877716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac1593c0 0x7ff9ac0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.555+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1298877716 >> v1:172.21.15.172:6789/0 conn(0x7ff9ac0cd8e0 legacy=0x7ff9ac0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.555+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1298877716 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac06c7f0 con 0x7ff9ac153290 2023-03-03T14:44:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.555+0000 7ff9aa7fc700 1 -- 172.21.15.102:0/1298877716 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824184099 0 0) 0x7ff9ac153660 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.555+0000 7ff9aaffd700 1 --2- 172.21.15.102:0/1298877716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac153290 0x7ff9ac158e80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7ff99c0098f0 tx=0x7ff99c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1988eac471df4aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:44:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.555+0000 7ff9aa7fc700 1 -- 172.21.15.102:0/1298877716 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c011830 con 0x7ff9ac153290 2023-03-03T14:44:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9aa7fc700 1 -- 172.21.15.102:0/1298877716 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c011990 con 0x7ff9ac153290 2023-03-03T14:44:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9aa7fc700 1 -- 172.21.15.102:0/1298877716 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c011d20 con 0x7ff9ac153290 2023-03-03T14:44:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1298877716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac153290 msgr2=0x7ff9ac158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1298877716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac153290 0x7ff9ac158e80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7ff99c0098f0 tx=0x7ff99c009c00 comp rx=0 tx=0).stop 2023-03-03T14:44:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1298877716 shutdown_connections 2023-03-03T14:44:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1298877716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac153290 0x7ff9ac158e80 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1298877716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac1593c0 0x7ff9ac0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1298877716 >> 172.21.15.102:0/1298877716 conn(0x7ff9ac065fb0 msgr2=0x7ff9ac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:44:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1298877716 shutdown_connections 2023-03-03T14:44:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.556+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1298877716 wait complete. 2023-03-03T14:44:00.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.558+0000 7ff9b17ff700 1 Processor -- start 2023-03-03T14:44:00.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.559+0000 7ff9b17ff700 1 -- start start 2023-03-03T14:44:00.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.559+0000 7ff9b17ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac0cd8e0 0x7ff9ac0c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:00.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.559+0000 7ff9b17ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac153290 0x7ff9ac14e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:00.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9b17ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac1593c0 0x7ff9ac14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:00.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9b17ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac14dad0 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9b17ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac14c100 con 0x7ff9ac1593c0 2023-03-03T14:44:00.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9b17ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac14c3a0 con 0x7ff9ac153290 2023-03-03T14:44:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac0cd8e0 0x7ff9ac0c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9ab7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac1593c0 0x7ff9ac14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac0cd8e0 0x7ff9ac0c88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59262/0 (socket says 172.21.15.102:59262) 2023-03-03T14:44:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1243217417 learned_addr learned my addr 172.21.15.102:0/1243217417 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:44:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9a3fff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac153290 0x7ff9ac14e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac153290 msgr2=0x7ff9ac14e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac153290 0x7ff9ac14e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac1593c0 msgr2=0x7ff9ac14e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac1593c0 0x7ff9ac14e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.560+0000 7ff9aaffd700 1 -- 172.21.15.102:0/1243217417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c009550 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.561+0000 7ff9aaffd700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac0cd8e0 0x7ff9ac0c88f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac074540 tx=0x7ff99c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:44:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.561+0000 7ff9a37fe700 1 -- 172.21.15.102:0/1243217417 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c004400 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.561+0000 7ff9a37fe700 1 -- 172.21.15.102:0/1243217417 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c01f070 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.561+0000 7ff9a37fe700 1 -- 172.21.15.102:0/1243217417 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0038e0 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.561+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac0ccde0 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.561+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac15d160 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.562+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac0cd0c0 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.563+0000 7ff9a37fe700 1 -- 172.21.15.102:0/1243217417 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff99c02a020 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.564+0000 7ff9a37fe700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff98c0682f0 0x7ff98c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:00.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.564+0000 7ff9a37fe700 1 -- 172.21.15.102:0/1243217417 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff99c087330 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.565+0000 7ff9a3fff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff98c0682f0 0x7ff98c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:00.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.566+0000 7ff9a3fff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff98c0682f0 0x7ff98c06a7a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0c5a20 tx=0x7ff994009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:44:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.569+0000 7ff9a37fe700 1 -- 172.21.15.102:0/1243217417 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff99c053a40 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.807+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9ac04e6e0 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.808+0000 7ff9a37fe700 1 -- 172.21.15.102:0/1243217417 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff99c057060 con 0x7ff9ac0cd8e0 2023-03-03T14:44:00.810 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:44:00.811 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:44:00.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.810+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff98c0682f0 msgr2=0x7ff98c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:00.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.810+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff98c0682f0 0x7ff98c06a7a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0c5a20 tx=0x7ff994009210 comp rx=0 tx=0).stop 2023-03-03T14:44:00.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.810+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac0cd8e0 msgr2=0x7ff9ac0c88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:00.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.810+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac0cd8e0 0x7ff9ac0c88f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac074540 tx=0x7ff99c004750 comp rx=0 tx=0).stop 2023-03-03T14:44:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 shutdown_connections 2023-03-03T14:44:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9ac1593c0 0x7ff9ac14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9ac0cd8e0 0x7ff9ac0c88f0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff98c0682f0 0x7ff98c06a7a0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 --2- 172.21.15.102:0/1243217417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9ac153290 0x7ff9ac14e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 >> 172.21.15.102:0/1243217417 conn(0x7ff9ac065fb0 msgr2=0x7ff9ac0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:44:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 shutdown_connections 2023-03-03T14:44:00.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:00.811+0000 7ff9b17ff700 1 -- 172.21.15.102:0/1243217417 wait complete. 2023-03-03T14:44:00.815 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:44:00.841 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2023-03-03T14:44:13.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:44:14.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.097+0000 7f6294dff700 1 Processor -- start 2023-03-03T14:44:14.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.098+0000 7f6294dff700 1 -- start start 2023-03-03T14:44:14.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.099+0000 7f6294dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6290153290 0x7f6290158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:14.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.099+0000 7f6294dff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6290153660 con 0x7f62901593c0 2023-03-03T14:44:14.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.099+0000 7f6294dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629006c360 con 0x7f62900cd8e0 2023-03-03T14:44:14.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.099+0000 7f6294dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62900cec40 con 0x7f6290153290 2023-03-03T14:44:14.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.099+0000 7f628dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f62901593c0 0x7f62900cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49464/0 (socket says 172.21.15.102:49464) 2023-03-03T14:44:14.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.099+0000 7f628e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6290153290 0x7f6290158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:14.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.100+0000 7f628dd9b700 1 -- 172.21.15.102:0/2645875523 learned_addr learned my addr 172.21.15.102:0/2645875523 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:44:14.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.100+0000 7f628e59c700 1 -- 172.21.15.102:0/2645875523 >> v1:172.21.15.172:6789/0 conn(0x7f62900cd8e0 legacy=0x7f62900ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:14.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.100+0000 7f628e59c700 1 -- 172.21.15.102:0/2645875523 >> v1:172.21.15.102:6789/0 conn(0x7f62901593c0 legacy=0x7f62900cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:14.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.100+0000 7f628d59a700 1 -- 172.21.15.102:0/2645875523 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580417041 0 0) 0x7f6290153660 con 0x7f62901593c0 2023-03-03T14:44:14.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.100+0000 7f628e59c700 1 -- 172.21.15.102:0/2645875523 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629015cea0 con 0x7f6290153290 2023-03-03T14:44:14.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.100+0000 7f628e59c700 1 --2- 172.21.15.102:0/2645875523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6290153290 0x7f6290158e80 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f62800097c0 tx=0x7f6280009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ff8e99ffd96867 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:44:14.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f628d59a700 1 -- 172.21.15.102:0/2645875523 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280004230 con 0x7f6290153290 2023-03-03T14:44:14.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f628d59a700 1 -- 172.21.15.102:0/2645875523 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6280004390 con 0x7f6290153290 2023-03-03T14:44:14.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f628d59a700 1 -- 172.21.15.102:0/2645875523 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62800036a0 con 0x7f6290153290 2023-03-03T14:44:14.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f6294dff700 1 -- 172.21.15.102:0/2645875523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6290153290 msgr2=0x7f6290158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:14.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f6294dff700 1 --2- 172.21.15.102:0/2645875523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6290153290 0x7f6290158e80 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f62800097c0 tx=0x7f6280009ad0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f6294dff700 1 -- 172.21.15.102:0/2645875523 shutdown_connections 2023-03-03T14:44:14.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f6294dff700 1 --2- 172.21.15.102:0/2645875523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6290153290 0x7f6290158e80 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f6294dff700 1 -- 172.21.15.102:0/2645875523 >> 172.21.15.102:0/2645875523 conn(0x7f6290065fb0 msgr2=0x7f62900663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:44:14.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f6294dff700 1 -- 172.21.15.102:0/2645875523 shutdown_connections 2023-03-03T14:44:14.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.101+0000 7f6294dff700 1 -- 172.21.15.102:0/2645875523 wait complete. 2023-03-03T14:44:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.103+0000 7f6294dff700 1 Processor -- start 2023-03-03T14:44:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.103+0000 7f6294dff700 1 -- start start 2023-03-03T14:44:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.104+0000 7f6294dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62900cd8e0 0x7f629015c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:14.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.104+0000 7f6294dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6290153290 0x7f62900b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:14.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.104+0000 7f6294dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62901593c0 0x7f6290152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:14.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.104+0000 7f6294dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62901f2360 con 0x7f62901593c0 2023-03-03T14:44:14.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.104+0000 7f6294dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62900b3120 con 0x7f6290153290 2023-03-03T14:44:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.104+0000 7f6294dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62900b2b60 con 0x7f62900cd8e0 2023-03-03T14:44:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62901593c0 0x7f6290152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6290153290 0x7f62900b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62901593c0 0x7f6290152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51230/0 (socket says 172.21.15.102:51230) 2023-03-03T14:44:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 -- 172.21.15.102:0/1621628888 learned_addr learned my addr 172.21.15.102:0/1621628888 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:44:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628e59c700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62900cd8e0 0x7f629015c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 -- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62900cd8e0 msgr2=0x7f629015c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62900cd8e0 0x7f629015c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 -- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6290153290 msgr2=0x7f62900b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6290153290 0x7f62900b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628ed9d700 1 -- 172.21.15.102:0/1621628888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800094a0 con 0x7f62901593c0 2023-03-03T14:44:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.105+0000 7f628dd9b700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6290153290 0x7f62900b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:44:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.106+0000 7f628ed9d700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62901593c0 0x7f6290152180 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f628800c870 tx=0x7f628800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:44:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.106+0000 7f627effd700 1 -- 172.21.15.102:0/1621628888 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288003e30 con 0x7f62901593c0 2023-03-03T14:44:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.106+0000 7f627effd700 1 -- 172.21.15.102:0/1621628888 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6288005070 con 0x7f62901593c0 2023-03-03T14:44:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.106+0000 7f627effd700 1 -- 172.21.15.102:0/1621628888 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288005420 con 0x7f62901593c0 2023-03-03T14:44:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.106+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629015d160 con 0x7f62901593c0 2023-03-03T14:44:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.106+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629006c7f0 con 0x7f62901593c0 2023-03-03T14:44:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.107+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62900ccde0 con 0x7f62901593c0 2023-03-03T14:44:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.108+0000 7f627effd700 1 -- 172.21.15.102:0/1621628888 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6288010070 con 0x7f62901593c0 2023-03-03T14:44:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.110+0000 7f627effd700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62700682f0 0x7f627006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:44:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.110+0000 7f627effd700 1 -- 172.21.15.102:0/1621628888 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6288084f10 con 0x7f62901593c0 2023-03-03T14:44:14.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.110+0000 7f628e59c700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62700682f0 0x7f627006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:44:14.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.111+0000 7f628e59c700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62700682f0 0x7f627006a7a0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f62800090b0 tx=0x7f6280008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:44:14.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.114+0000 7f627effd700 1 -- 172.21.15.102:0/1621628888 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6288051590 con 0x7f62901593c0 2023-03-03T14:44:14.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.349+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f629004e6e0 con 0x7f62901593c0 2023-03-03T14:44:14.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.350+0000 7f627effd700 1 -- 172.21.15.102:0/1621628888 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6288017020 con 0x7f62901593c0 2023-03-03T14:44:14.351 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:44:14.352 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:44:14.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62700682f0 msgr2=0x7f627006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:14.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62700682f0 0x7f627006a7a0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f62800090b0 tx=0x7f6280008030 comp rx=0 tx=0).stop 2023-03-03T14:44:14.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62901593c0 msgr2=0x7f6290152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:44:14.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62901593c0 0x7f6290152180 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f628800c870 tx=0x7f628800cb80 comp rx=0 tx=0).stop 2023-03-03T14:44:14.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 shutdown_connections 2023-03-03T14:44:14.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f62700682f0 0x7f627006a7a0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f62900cd8e0 0x7f629015c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62901593c0 0x7f6290152180 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 --2- 172.21.15.102:0/1621628888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6290153290 0x7f62900b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:44:14.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.352+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 >> 172.21.15.102:0/1621628888 conn(0x7f6290065fb0 msgr2=0x7f62900cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:44:14.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.353+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 shutdown_connections 2023-03-03T14:44:14.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:44:14.353+0000 7f6294dff700 1 -- 172.21.15.102:0/1621628888 wait complete. 2023-03-03T14:44:14.356 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:44:14.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-03T14:44:15.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:15.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:46.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:46.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:16.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:16.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:25.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:45:25.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.630+0000 7f52ca9ff700 1 Processor -- start 2023-03-03T14:45:25.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.631+0000 7f52ca9ff700 1 -- start start 2023-03-03T14:45:25.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52ca9ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f52c40cd8e0 0x7f52c40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52ca9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52ca9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c4153660 con 0x7f52c4153290 2023-03-03T14:45:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52ca9ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c406c360 con 0x7f52c40cd8e0 2023-03-03T14:45:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52ca9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c415cea0 con 0x7f52c40cd390 2023-03-03T14:45:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52c37fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f52c40cd8e0 0x7f52c40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52c3fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f52c4153290 0x7f52c4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:39928/0 (socket says 172.21.15.102:39928) 2023-03-03T14:45:25.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52c3fff700 1 -- 172.21.15.102:0/3283791605 learned_addr learned my addr 172.21.15.102:0/3283791605 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:45:25.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.632+0000 7f52c8f9c700 1 --2- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:25.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c37fe700 1 -- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 msgr2=0x7f52c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:45:25.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c37fe700 1 --2- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c37fe700 1 -- 172.21.15.102:0/3283791605 >> v1:172.21.15.172:6789/0 conn(0x7f52c4153290 legacy=0x7f52c4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:25.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c37fe700 1 -- 172.21.15.102:0/3283791605 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c406c7f0 con 0x7f52c40cd8e0 2023-03-03T14:45:25.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c37fe700 1 --2- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c40cd8e0 0x7f52c40cde20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009840 tx=0x7f52b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45e21aeee72522b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:25.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c2ffd700 1 -- 172.21.15.102:0/3283791605 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8011830 con 0x7f52c40cd8e0 2023-03-03T14:45:25.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c2ffd700 1 -- 172.21.15.102:0/3283791605 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b8011990 con 0x7f52c40cd8e0 2023-03-03T14:45:25.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.633+0000 7f52c2ffd700 1 -- 172.21.15.102:0/3283791605 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8011d20 con 0x7f52c40cd8e0 2023-03-03T14:45:25.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 -- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c40cd8e0 msgr2=0x7f52c40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:25.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c40cd8e0 0x7f52c40cde20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009840 tx=0x7f52b8009b50 comp rx=0 tx=0).stop 2023-03-03T14:45:25.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 -- 172.21.15.102:0/3283791605 shutdown_connections 2023-03-03T14:45:25.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/3283791605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c40cd8e0 0x7f52c40cde20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 -- 172.21.15.102:0/3283791605 >> 172.21.15.102:0/3283791605 conn(0x7f52c4065fb0 msgr2=0x7f52c40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:45:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 -- 172.21.15.102:0/3283791605 shutdown_connections 2023-03-03T14:45:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.634+0000 7f52ca9ff700 1 -- 172.21.15.102:0/3283791605 wait complete. 2023-03-03T14:45:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.636+0000 7f52ca9ff700 1 Processor -- start 2023-03-03T14:45:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.636+0000 7f52ca9ff700 1 -- start start 2023-03-03T14:45:25.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.637+0000 7f52ca9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:25.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52ca9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52c40cd8e0 0x7f52c40c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52ca9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c4153290 0x7f52c41f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52ca9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40b4410 con 0x7f52c40cd8e0 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52ca9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40b2ac0 con 0x7f52c4153290 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52ca9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40b36c0 con 0x7f52c40cd390 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52c40cd8e0 0x7f52c40c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c4153290 0x7f52c41f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:25.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c4153290 0x7f52c41f2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50496/0 (socket says 172.21.15.102:50496) 2023-03-03T14:45:25.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 -- 172.21.15.102:0/2970593634 learned_addr learned my addr 172.21.15.102:0/2970593634 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:45:25.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 -- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 msgr2=0x7f52c40b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:25.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 -- 172.21.15.102:0/2970593634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52c40cd8e0 msgr2=0x7f52c40c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:25.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52c40cd8e0 0x7f52c40c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.638+0000 7f52c8f9c700 1 -- 172.21.15.102:0/2970593634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80094a0 con 0x7f52c4153290 2023-03-03T14:45:25.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.639+0000 7f52c8f9c700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c4153290 0x7f52c41f2040 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009d30 tx=0x7f52bc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:25.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.639+0000 7f52c0ff9700 1 -- 172.21.15.102:0/2970593634 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00cc90 con 0x7f52c4153290 2023-03-03T14:45:25.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.639+0000 7f52c0ff9700 1 -- 172.21.15.102:0/2970593634 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc00f040 con 0x7f52c4153290 2023-03-03T14:45:25.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.639+0000 7f52c0ff9700 1 -- 172.21.15.102:0/2970593634 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc009e80 con 0x7f52c4153290 2023-03-03T14:45:25.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.639+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c40ccde0 con 0x7f52c4153290 2023-03-03T14:45:25.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.639+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c415d160 con 0x7f52c4153290 2023-03-03T14:45:25.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.640+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c40cd0c0 con 0x7f52c4153290 2023-03-03T14:45:25.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.642+0000 7f52c0ff9700 1 -- 172.21.15.102:0/2970593634 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f52bc014480 con 0x7f52c4153290 2023-03-03T14:45:25.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.643+0000 7f52c0ff9700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f52b00682f0 0x7f52b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:25.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.643+0000 7f52c0ff9700 1 -- 172.21.15.102:0/2970593634 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f52bc0866d0 con 0x7f52c4153290 2023-03-03T14:45:25.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.643+0000 7f52c3fff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f52b00682f0 0x7f52b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:25.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.644+0000 7f52c3fff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f52b00682f0 0x7f52b006a7a0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f52b4005910 tx=0x7f52b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.647+0000 7f52c0ff9700 1 -- 172.21.15.102:0/2970593634 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f52bc052d80 con 0x7f52c4153290 2023-03-03T14:45:25.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.882+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52c40b4550 con 0x7f52c4153290 2023-03-03T14:45:25.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.883+0000 7f52c0ff9700 1 -- 172.21.15.102:0/2970593634 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f52c40b4550 con 0x7f52c4153290 2023-03-03T14:45:25.885 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:45:25.885 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:45:25.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f52b00682f0 msgr2=0x7f52b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:25.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f52b00682f0 0x7f52b006a7a0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f52b4005910 tx=0x7f52b4009040 comp rx=0 tx=0).stop 2023-03-03T14:45:25.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c4153290 msgr2=0x7f52c41f2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:25.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c4153290 0x7f52c41f2040 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009d30 tx=0x7f52bc00c3a0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 shutdown_connections 2023-03-03T14:45:25.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f52b00682f0 0x7f52b006a7a0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f52c40cd390 0x7f52c40b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52c4153290 0x7f52c41f2040 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 --2- 172.21.15.102:0/2970593634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52c40cd8e0 0x7f52c40c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:25.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 >> 172.21.15.102:0/2970593634 conn(0x7f52c4065fb0 msgr2=0x7f52c40cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:45:25.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 shutdown_connections 2023-03-03T14:45:25.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:45:25.886+0000 7f52ca9ff700 1 -- 172.21.15.102:0/2970593634 wait complete. 2023-03-03T14:45:25.889 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:45:25.914 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 54 secs before thrashing 2023-03-03T14:45:46.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:46.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:16.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:16.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:19.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:46:20.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.162+0000 7f47bebff700 1 Processor -- start 2023-03-03T14:46:20.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.162+0000 7f47bebff700 1 -- start start 2023-03-03T14:46:20.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.163+0000 7f47bebff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f47b80cd390 0x7f47b80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:20.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.163+0000 7f47bebff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b8153660 con 0x7f47b80cd8e0 2023-03-03T14:46:20.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.163+0000 7f47bebff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b806c360 con 0x7f47b8153290 2023-03-03T14:46:20.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.163+0000 7f47bebff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b815cea0 con 0x7f47b80cd390 2023-03-03T14:46:20.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47b7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f47b80cd8e0 0x7f47b80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45198/0 (socket says 172.21.15.102:45198) 2023-03-03T14:46:20.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47b7fff700 1 -- 172.21.15.102:0/2884171796 learned_addr learned my addr 172.21.15.102:0/2884171796 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:46:20.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47bd19c700 1 --2- 172.21.15.102:0/2884171796 >> v2:172.21.15.172:3300/0 conn(0x7f47b80cd390 0x7f47b80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:20.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47b77fe700 1 -- 172.21.15.102:0/2884171796 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832689368 0 0) 0x7f47b806c360 con 0x7f47b8153290 2023-03-03T14:46:20.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47b77fe700 1 -- 172.21.15.102:0/2884171796 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47a80035b0 con 0x7f47b8153290 2023-03-03T14:46:20.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47b77fe700 1 -- 172.21.15.102:0/2884171796 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856557621 0 0) 0x7f47b8153660 con 0x7f47b80cd8e0 2023-03-03T14:46:20.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47bd19c700 1 -- 172.21.15.102:0/2884171796 >> v1:172.21.15.172:6789/0 conn(0x7f47b8153290 legacy=0x7f47b8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:20.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47bd19c700 1 -- 172.21.15.102:0/2884171796 >> v1:172.21.15.102:6789/0 conn(0x7f47b80cd8e0 legacy=0x7f47b80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:20.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.164+0000 7f47bd19c700 1 -- 172.21.15.102:0/2884171796 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b806c7f0 con 0x7f47b80cd390 2023-03-03T14:46:20.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.165+0000 7f47bd19c700 1 --2- 172.21.15.102:0/2884171796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd390 0x7f47b80ce530 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009d90 tx=0x7f47b000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27f8deab67e2cc60 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:20.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.165+0000 7f47b77fe700 1 -- 172.21.15.102:0/2884171796 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b000e070 con 0x7f47b80cd390 2023-03-03T14:46:20.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.165+0000 7f47b77fe700 1 -- 172.21.15.102:0/2884171796 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b000bb20 con 0x7f47b80cd390 2023-03-03T14:46:20.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.165+0000 7f47b77fe700 1 -- 172.21.15.102:0/2884171796 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b000be90 con 0x7f47b80cd390 2023-03-03T14:46:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.166+0000 7f47bebff700 1 -- 172.21.15.102:0/2884171796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd390 msgr2=0x7f47b80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.166+0000 7f47bebff700 1 --2- 172.21.15.102:0/2884171796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd390 0x7f47b80ce530 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009d90 tx=0x7f47b000b330 comp rx=0 tx=0).stop 2023-03-03T14:46:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.166+0000 7f47bebff700 1 -- 172.21.15.102:0/2884171796 shutdown_connections 2023-03-03T14:46:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.166+0000 7f47bebff700 1 --2- 172.21.15.102:0/2884171796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd390 0x7f47b80ce530 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.166+0000 7f47bebff700 1 -- 172.21.15.102:0/2884171796 >> 172.21.15.102:0/2884171796 conn(0x7f47b8065fb0 msgr2=0x7f47b80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.166+0000 7f47bebff700 1 -- 172.21.15.102:0/2884171796 shutdown_connections 2023-03-03T14:46:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.166+0000 7f47bebff700 1 -- 172.21.15.102:0/2884171796 wait complete. 2023-03-03T14:46:20.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.168+0000 7f47bebff700 1 Processor -- start 2023-03-03T14:46:20.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.168+0000 7f47bebff700 1 -- start start 2023-03-03T14:46:20.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bebff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47b80cd390 0x7f47b80b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:20.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bebff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd8e0 0x7f47b80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:20.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bebff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47b8153290 0x7f47b81e29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:20.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bebff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80b4780 con 0x7f47b8153290 2023-03-03T14:46:20.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bebff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80c6a50 con 0x7f47b80cd8e0 2023-03-03T14:46:20.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bebff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80b4300 con 0x7f47b80cd390 2023-03-03T14:46:20.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bd19c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47b8153290 0x7f47b81e29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:20.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47b7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd8e0 0x7f47b80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bc99b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47b80cd390 0x7f47b80b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bc99b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47b80cd390 0x7f47b80b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:43082/0 (socket says 172.21.15.102:43082) 2023-03-03T14:46:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bd19c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47b8153290 0x7f47b81e29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51684/0 (socket says 172.21.15.102:51684) 2023-03-03T14:46:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47bc99b700 1 -- 172.21.15.102:0/3444420191 learned_addr learned my addr 172.21.15.102:0/3444420191 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:46:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.169+0000 7f47b7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd8e0 0x7f47b80b3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35446/0 (socket says 172.21.15.102:35446) 2023-03-03T14:46:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b7fff700 1 -- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47b80cd390 msgr2=0x7f47b80b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b7fff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47b80cd390 0x7f47b80b2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b7fff700 1 -- 172.21.15.102:0/3444420191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47b8153290 msgr2=0x7f47b81e29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:20.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b7fff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47b8153290 0x7f47b81e29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b7fff700 1 -- 172.21.15.102:0/3444420191 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b0009a70 con 0x7f47b80cd8e0 2023-03-03T14:46:20.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b7fff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd8e0 0x7f47b80b3060 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c4b0 tx=0x7f47a400c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:20.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b57fa700 1 -- 172.21.15.102:0/3444420191 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4004c80 con 0x7f47b80cd8e0 2023-03-03T14:46:20.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b57fa700 1 -- 172.21.15.102:0/3444420191 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a4004e00 con 0x7f47b80cd8e0 2023-03-03T14:46:20.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47b57fa700 1 -- 172.21.15.102:0/3444420191 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4007310 con 0x7f47b80cd8e0 2023-03-03T14:46:20.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.170+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b80ccde0 con 0x7f47b80cd8e0 2023-03-03T14:46:20.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.171+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b815d160 con 0x7f47b80cd8e0 2023-03-03T14:46:20.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.172+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b80cd0c0 con 0x7f47b80cd8e0 2023-03-03T14:46:20.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.173+0000 7f47b57fa700 1 -- 172.21.15.102:0/3444420191 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47a401a020 con 0x7f47b80cd8e0 2023-03-03T14:46:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.174+0000 7f47b57fa700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f479c0682c0 0x7f479c06a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.174+0000 7f47b57fa700 1 -- 172.21.15.102:0/3444420191 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f47a4085880 con 0x7f47b80cd8e0 2023-03-03T14:46:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.175+0000 7f47bc99b700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f479c0682c0 0x7f479c06a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:20.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.175+0000 7f47bc99b700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f479c0682c0 0x7f479c06a770 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f47ac001890 tx=0x7f47ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:20.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.179+0000 7f47b57fa700 1 -- 172.21.15.102:0/3444420191 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47a4052050 con 0x7f47b80cd8e0 2023-03-03T14:46:20.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.414+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47b80b3ae0 con 0x7f47b80cd8e0 2023-03-03T14:46:20.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.416+0000 7f47b57fa700 1 -- 172.21.15.102:0/3444420191 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f47a4055670 con 0x7f47b80cd8e0 2023-03-03T14:46:20.418 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:46:20.418 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:46:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f479c0682c0 msgr2=0x7f479c06a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f479c0682c0 0x7f479c06a770 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f47ac001890 tx=0x7f47ac009040 comp rx=0 tx=0).stop 2023-03-03T14:46:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd8e0 msgr2=0x7f47b80b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd8e0 0x7f47b80b3060 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c4b0 tx=0x7f47a400c7c0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 shutdown_connections 2023-03-03T14:46:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f479c0682c0 0x7f479c06a770 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f47b80cd390 0x7f47b80b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47b8153290 0x7f47b81e29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 --2- 172.21.15.102:0/3444420191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47b80cd8e0 0x7f47b80b3060 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 >> 172.21.15.102:0/3444420191 conn(0x7f47b8065fb0 msgr2=0x7f47b8150ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 shutdown_connections 2023-03-03T14:46:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:46:20.418+0000 7f47bebff700 1 -- 172.21.15.102:0/3444420191 wait complete. 2023-03-03T14:46:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:46:20.448 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2023-03-03T14:46:46.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:46.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:16.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:16.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:47.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:47.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:05.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:48:05.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.698+0000 7fd9e65ff700 1 Processor -- start 2023-03-03T14:48:05.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.698+0000 7fd9e65ff700 1 -- start start 2023-03-03T14:48:05.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.700+0000 7fd9e65ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9e0153290 0x7fd9e0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:05.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9e65ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e0153660 con 0x7fd9e01593c0 2023-03-03T14:48:05.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9dffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9e0153290 0x7fd9e0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:05.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9e65ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e006c360 con 0x7fd9e00cd8e0 2023-03-03T14:48:05.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9dffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9e0153290 0x7fd9e0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49552/0 (socket says 172.21.15.102:49552) 2023-03-03T14:48:05.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9dffff700 1 -- 172.21.15.102:0/622010700 learned_addr learned my addr 172.21.15.102:0/622010700 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:48:05.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9e65ff700 1 -- 172.21.15.102:0/622010700 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e00cec40 con 0x7fd9e0153290 2023-03-03T14:48:05.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9dffff700 1 -- 172.21.15.102:0/622010700 >> v1:172.21.15.172:6789/0 conn(0x7fd9e00cd8e0 legacy=0x7fd9e00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:05.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9dffff700 1 -- 172.21.15.102:0/622010700 >> v1:172.21.15.102:6789/0 conn(0x7fd9e01593c0 legacy=0x7fd9e00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9dffff700 1 -- 172.21.15.102:0/622010700 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e015cea0 con 0x7fd9e0153290 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.701+0000 7fd9deffd700 1 -- 172.21.15.102:0/622010700 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656173057 0 0) 0x7fd9e006c360 con 0x7fd9e00cd8e0 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.702+0000 7fd9dffff700 1 --2- 172.21.15.102:0/622010700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0158e80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098f0 tx=0x7fd9d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c106923a2af1f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.702+0000 7fd9deffd700 1 -- 172.21.15.102:0/622010700 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d000b8a0 con 0x7fd9e0153290 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.702+0000 7fd9deffd700 1 -- 172.21.15.102:0/622010700 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d000ba00 con 0x7fd9e0153290 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.702+0000 7fd9deffd700 1 -- 172.21.15.102:0/622010700 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d000bd90 con 0x7fd9e0153290 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.703+0000 7fd9e65ff700 1 -- 172.21.15.102:0/622010700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 msgr2=0x7fd9e0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:05.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.703+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/622010700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0158e80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098f0 tx=0x7fd9d0009c00 comp rx=0 tx=0).stop 2023-03-03T14:48:05.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.703+0000 7fd9e65ff700 1 -- 172.21.15.102:0/622010700 shutdown_connections 2023-03-03T14:48:05.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.703+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/622010700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0158e80 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:05.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.703+0000 7fd9e65ff700 1 -- 172.21.15.102:0/622010700 >> 172.21.15.102:0/622010700 conn(0x7fd9e0065fb0 msgr2=0x7fd9e00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:48:05.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.703+0000 7fd9e65ff700 1 -- 172.21.15.102:0/622010700 shutdown_connections 2023-03-03T14:48:05.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.703+0000 7fd9e65ff700 1 -- 172.21.15.102:0/622010700 wait complete. 2023-03-03T14:48:05.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.705+0000 7fd9e65ff700 1 Processor -- start 2023-03-03T14:48:05.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.705+0000 7fd9e65ff700 1 -- start start 2023-03-03T14:48:05.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9e65ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e00cd8e0 0x7fd9e01ef490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:05.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9e65ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:05.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9e65ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e01593c0 0x7fd9e015b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:05.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9e65ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e015ad20 con 0x7fd9e0153290 2023-03-03T14:48:05.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9e65ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e015b000 con 0x7fd9e01593c0 2023-03-03T14:48:05.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9e65ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e01546b0 con 0x7fd9e00cd8e0 2023-03-03T14:48:05.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9df7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:05.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9df7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49558/0 (socket says 172.21.15.102:49558) 2023-03-03T14:48:05.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9df7fe700 1 -- 172.21.15.102:0/1870560293 learned_addr learned my addr 172.21.15.102:0/1870560293 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:48:05.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9dffff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e00cd8e0 0x7fd9e01ef490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:05.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.706+0000 7fd9e4b9c700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e01593c0 0x7fd9e015b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:05.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9df7fe700 1 -- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e00cd8e0 msgr2=0x7fd9e01ef490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:05.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9df7fe700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e00cd8e0 0x7fd9e01ef490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:05.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9df7fe700 1 -- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e01593c0 msgr2=0x7fd9e015b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:05.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9df7fe700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e01593c0 0x7fd9e015b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:05.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9df7fe700 1 -- 172.21.15.102:0/1870560293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0009550 con 0x7fd9e0153290 2023-03-03T14:48:05.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9df7fe700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0157ba0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c510 tx=0x7fd9d400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:05.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9dcff9700 1 -- 172.21.15.102:0/1870560293 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40052d0 con 0x7fd9e0153290 2023-03-03T14:48:05.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9dcff9700 1 -- 172.21.15.102:0/1870560293 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d4005430 con 0x7fd9e0153290 2023-03-03T14:48:05.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9dcff9700 1 -- 172.21.15.102:0/1870560293 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40038c0 con 0x7fd9e0153290 2023-03-03T14:48:05.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.707+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e015d160 con 0x7fd9e0153290 2023-03-03T14:48:05.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.708+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e006c7f0 con 0x7fd9e0153290 2023-03-03T14:48:05.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.709+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e00ccde0 con 0x7fd9e0153290 2023-03-03T14:48:05.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.710+0000 7fd9dcff9700 1 -- 172.21.15.102:0/1870560293 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd9d4019030 con 0x7fd9e0153290 2023-03-03T14:48:05.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.711+0000 7fd9dcff9700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9cc068320 0x7fd9cc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:05.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.711+0000 7fd9dcff9700 1 -- 172.21.15.102:0/1870560293 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd9d4084fe0 con 0x7fd9e0153290 2023-03-03T14:48:05.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.712+0000 7fd9dffff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9cc068320 0x7fd9cc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:05.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.712+0000 7fd9dffff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9cc068320 0x7fd9cc06a7d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0014040 tx=0x7fd9d001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.716+0000 7fd9dcff9700 1 -- 172.21.15.102:0/1870560293 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9d4051730 con 0x7fd9e0153290 2023-03-03T14:48:05.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.951+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9e004e6e0 con 0x7fd9e0153290 2023-03-03T14:48:05.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.952+0000 7fd9dcff9700 1 -- 172.21.15.102:0/1870560293 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd9d4017090 con 0x7fd9e0153290 2023-03-03T14:48:05.954 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:48:05.955 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:48:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.954+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9cc068320 msgr2=0x7fd9cc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.954+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9cc068320 0x7fd9cc06a7d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0014040 tx=0x7fd9d001c040 comp rx=0 tx=0).stop 2023-03-03T14:48:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.954+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 msgr2=0x7fd9e0157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.954+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0157ba0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c510 tx=0x7fd9d400c820 comp rx=0 tx=0).stop 2023-03-03T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 shutdown_connections 2023-03-03T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd9cc068320 0x7fd9cc06a7d0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd9e00cd8e0 0x7fd9e01ef490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9e01593c0 0x7fd9e015b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 --2- 172.21.15.102:0/1870560293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9e0153290 0x7fd9e0157ba0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 >> 172.21.15.102:0/1870560293 conn(0x7fd9e0065fb0 msgr2=0x7fd9e00b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 shutdown_connections 2023-03-03T14:48:05.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:48:05.955+0000 7fd9e65ff700 1 -- 172.21.15.102:0/1870560293 wait complete. 2023-03-03T14:48:05.959 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:48:05.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-03-03T14:48:17.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:17.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:47.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:47.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:17.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:17.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:47.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:47.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:54.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:49:55.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.249+0000 7fd8acdff700 1 Processor -- start 2023-03-03T14:49:55.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.249+0000 7fd8acdff700 1 -- start start 2023-03-03T14:49:55.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8acdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a80ced80 0x7fd8a80cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:55.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8acdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8a81597f0 0x7fd8a80cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8acdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a80cb9f0 con 0x7fd8a80cf9a0 2023-03-03T14:49:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8acdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a80ca750 con 0x7fd8a81597f0 2023-03-03T14:49:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8acdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a815a230 con 0x7fd8a80ced80 2023-03-03T14:49:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8a659c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a80ced80 0x7fd8a80cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8a6d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd8a80cf9a0 0x7fd8a815c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47010/0 (socket says 172.21.15.102:47010) 2023-03-03T14:49:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8a5d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8a81597f0 0x7fd8a80cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:55.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8a6d9d700 1 -- 172.21.15.102:0/3711045553 learned_addr learned my addr 172.21.15.102:0/3711045553 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:49:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.251+0000 7fd8a559a700 1 -- 172.21.15.102:0/3711045553 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702651690 0 0) 0x7fd8a80cb9f0 con 0x7fd8a80cf9a0 2023-03-03T14:49:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a559a700 1 -- 172.21.15.102:0/3711045553 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8900035b0 con 0x7fd8a80cf9a0 2023-03-03T14:49:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a659c700 1 -- 172.21.15.102:0/3711045553 >> v2:172.21.15.102:3300/0 conn(0x7fd8a81597f0 msgr2=0x7fd8a80cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:49:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a659c700 1 --2- 172.21.15.102:0/3711045553 >> v2:172.21.15.102:3300/0 conn(0x7fd8a81597f0 0x7fd8a80cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a659c700 1 -- 172.21.15.102:0/3711045553 >> v1:172.21.15.172:6789/0 conn(0x7fd8a80cf9a0 legacy=0x7fd8a815c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:55.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a659c700 1 -- 172.21.15.102:0/3711045553 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a8159c80 con 0x7fd8a80ced80 2023-03-03T14:49:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a5d9b700 1 --2- 172.21.15.102:0/3711045553 >> v2:172.21.15.102:3300/0 conn(0x7fd8a81597f0 0x7fd8a80cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:49:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a659c700 1 --2- 172.21.15.102:0/3711045553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a80ced80 0x7fd8a80cf150 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd898009840 tx=0x7fd898009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aadb8933bab4291 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:49:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.252+0000 7fd8a559a700 1 -- 172.21.15.102:0/3711045553 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898011830 con 0x7fd8a80ced80 2023-03-03T14:49:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8a559a700 1 -- 172.21.15.102:0/3711045553 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898011990 con 0x7fd8a80ced80 2023-03-03T14:49:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8a559a700 1 -- 172.21.15.102:0/3711045553 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898011d20 con 0x7fd8a80ced80 2023-03-03T14:49:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 -- 172.21.15.102:0/3711045553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a80ced80 msgr2=0x7fd8a80cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:55.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3711045553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a80ced80 0x7fd8a80cf150 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd898009840 tx=0x7fd898009b50 comp rx=0 tx=0).stop 2023-03-03T14:49:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 -- 172.21.15.102:0/3711045553 shutdown_connections 2023-03-03T14:49:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3711045553 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a80ced80 0x7fd8a80cf150 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3711045553 >> v2:172.21.15.102:3300/0 conn(0x7fd8a81597f0 0x7fd8a80cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 -- 172.21.15.102:0/3711045553 >> 172.21.15.102:0/3711045553 conn(0x7fd8a8065fb0 msgr2=0x7fd8a80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:49:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 -- 172.21.15.102:0/3711045553 shutdown_connections 2023-03-03T14:49:55.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.253+0000 7fd8acdff700 1 -- 172.21.15.102:0/3711045553 wait complete. 2023-03-03T14:49:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.255+0000 7fd8acdff700 1 Processor -- start 2023-03-03T14:49:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.256+0000 7fd8acdff700 1 -- start start 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.256+0000 7fd8acdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8a80ced80 0x7fd8a81ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.256+0000 7fd8acdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8a80cf9a0 0x7fd8a80b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8acdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a81597f0 0x7fd8a81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8acdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a80b30f0 con 0x7fd8a80ced80 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8acdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a8158190 con 0x7fd8a80cf9a0 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8acdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a81eed30 con 0x7fd8a81597f0 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8a80ced80 0x7fd8a81ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:55.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8a80ced80 0x7fd8a81ee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55750/0 (socket says 172.21.15.102:55750) 2023-03-03T14:49:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a6d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a81597f0 0x7fd8a81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 -- 172.21.15.102:0/3147723561 learned_addr learned my addr 172.21.15.102:0/3147723561 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:49:55.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a5d9b700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8a80cf9a0 0x7fd8a80b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 -- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a81597f0 msgr2=0x7fd8a81576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a81597f0 0x7fd8a81576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 -- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8a80cf9a0 msgr2=0x7fd8a80b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8a80cf9a0 0x7fd8a80b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.257+0000 7fd8a659c700 1 -- 172.21.15.102:0/3147723561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c0094a0 con 0x7fd8a80ced80 2023-03-03T14:49:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.258+0000 7fd8a659c700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8a80ced80 0x7fd8a81ee7f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd898000c00 tx=0x7fd898004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:49:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.258+0000 7fd896ffd700 1 -- 172.21.15.102:0/3147723561 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898004590 con 0x7fd8a80ced80 2023-03-03T14:49:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.258+0000 7fd896ffd700 1 -- 172.21.15.102:0/3147723561 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89801e070 con 0x7fd8a80ced80 2023-03-03T14:49:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.258+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8980094a0 con 0x7fd8a80ced80 2023-03-03T14:49:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.258+0000 7fd896ffd700 1 -- 172.21.15.102:0/3147723561 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980038e0 con 0x7fd8a80ced80 2023-03-03T14:49:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.258+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a80cb0c0 con 0x7fd8a80ced80 2023-03-03T14:49:55.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.259+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a80cade0 con 0x7fd8a80ced80 2023-03-03T14:49:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.261+0000 7fd896ffd700 1 -- 172.21.15.102:0/3147723561 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd89800f050 con 0x7fd8a80ced80 2023-03-03T14:49:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.262+0000 7fd896ffd700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd8880682f0 0x7fd88806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:55.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.262+0000 7fd896ffd700 1 -- 172.21.15.102:0/3147723561 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd898086cb0 con 0x7fd8a80ced80 2023-03-03T14:49:55.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.262+0000 7fd8a5d9b700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd8880682f0 0x7fd88806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:55.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.263+0000 7fd8a5d9b700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd8880682f0 0x7fd88806a7a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fd8a81e86b0 tx=0x7fd89c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:49:55.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.266+0000 7fd896ffd700 1 -- 172.21.15.102:0/3147723561 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd898053340 con 0x7fd8a80ced80 2023-03-03T14:49:55.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.499+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8a80cb3a0 con 0x7fd8a80ced80 2023-03-03T14:49:55.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.499+0000 7fd896ffd700 1 -- 172.21.15.102:0/3147723561 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd898020020 con 0x7fd8a80ced80 2023-03-03T14:49:55.501 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:49:55.501 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:49:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd8880682f0 msgr2=0x7fd88806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd8880682f0 0x7fd88806a7a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fd8a81e86b0 tx=0x7fd89c009210 comp rx=0 tx=0).stop 2023-03-03T14:49:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8a80ced80 msgr2=0x7fd8a81ee7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8a80ced80 0x7fd8a81ee7f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd898000c00 tx=0x7fd898004750 comp rx=0 tx=0).stop 2023-03-03T14:49:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 shutdown_connections 2023-03-03T14:49:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd8880682f0 0x7fd88806a7a0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd8a81597f0 0x7fd8a81576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8a80ced80 0x7fd8a81ee7f0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 --2- 172.21.15.102:0/3147723561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8a80cf9a0 0x7fd8a80b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:55.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 >> 172.21.15.102:0/3147723561 conn(0x7fd8a8065fb0 msgr2=0x7fd8a81585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:49:55.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 shutdown_connections 2023-03-03T14:49:55.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:49:55.502+0000 7fd8acdff700 1 -- 172.21.15.102:0/3147723561 wait complete. 2023-03-03T14:49:55.505 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:49:55.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T14:50:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T14:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:50:18.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:18.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:48.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:48.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:04.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:51:04.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.781+0000 7fa1673ff700 1 Processor -- start 2023-03-03T14:51:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.781+0000 7fa1673ff700 1 -- start start 2023-03-03T14:51:04.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.782+0000 7fa1673ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa1600cd8e0 0x7fa1600cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.782+0000 7fa1673ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa160153660 con 0x7fa1600cd390 2023-03-03T14:51:04.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.782+0000 7fa1673ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16006c360 con 0x7fa160153290 2023-03-03T14:51:04.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.782+0000 7fa1673ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16015cea0 con 0x7fa1600cd8e0 2023-03-03T14:51:04.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa16519b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa160153290 0x7fa160158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36086/0 (socket says 172.21.15.102:36086) 2023-03-03T14:51:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa16519b700 1 -- 172.21.15.102:0/985831624 learned_addr learned my addr 172.21.15.102:0/985831624 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:51:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa16499a700 1 --2- 172.21.15.102:0/985831624 >> v2:172.21.15.102:3300/0 conn(0x7fa1600cd8e0 0x7fa1600cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa157fff700 1 -- 172.21.15.102:0/985831624 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022690235 0 0) 0x7fa16006c360 con 0x7fa160153290 2023-03-03T14:51:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa157fff700 1 -- 172.21.15.102:0/985831624 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c0035b0 con 0x7fa160153290 2023-03-03T14:51:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa16499a700 1 -- 172.21.15.102:0/985831624 >> v1:172.21.15.172:6789/0 conn(0x7fa160153290 legacy=0x7fa160158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa16499a700 1 -- 172.21.15.102:0/985831624 >> v1:172.21.15.102:6789/0 conn(0x7fa1600cd390 legacy=0x7fa1600ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.783+0000 7fa16499a700 1 -- 172.21.15.102:0/985831624 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16006c7f0 con 0x7fa1600cd8e0 2023-03-03T14:51:04.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.784+0000 7fa16499a700 1 --2- 172.21.15.102:0/985831624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd8e0 0x7fa1600cde20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa150009840 tx=0x7fa150009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52ae460d1466e10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:04.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.784+0000 7fa157fff700 1 -- 172.21.15.102:0/985831624 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150011830 con 0x7fa1600cd8e0 2023-03-03T14:51:04.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.784+0000 7fa157fff700 1 -- 172.21.15.102:0/985831624 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa150011990 con 0x7fa1600cd8e0 2023-03-03T14:51:04.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.784+0000 7fa157fff700 1 -- 172.21.15.102:0/985831624 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150011d20 con 0x7fa1600cd8e0 2023-03-03T14:51:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.784+0000 7fa1673ff700 1 -- 172.21.15.102:0/985831624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd8e0 msgr2=0x7fa1600cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.784+0000 7fa1673ff700 1 --2- 172.21.15.102:0/985831624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd8e0 0x7fa1600cde20 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa150009840 tx=0x7fa150009b50 comp rx=0 tx=0).stop 2023-03-03T14:51:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.785+0000 7fa1673ff700 1 -- 172.21.15.102:0/985831624 shutdown_connections 2023-03-03T14:51:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.785+0000 7fa1673ff700 1 --2- 172.21.15.102:0/985831624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd8e0 0x7fa1600cde20 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.785+0000 7fa1673ff700 1 -- 172.21.15.102:0/985831624 >> 172.21.15.102:0/985831624 conn(0x7fa160065fb0 msgr2=0x7fa1600663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:51:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.785+0000 7fa1673ff700 1 -- 172.21.15.102:0/985831624 shutdown_connections 2023-03-03T14:51:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.785+0000 7fa1673ff700 1 -- 172.21.15.102:0/985831624 wait complete. 2023-03-03T14:51:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.787+0000 7fa1673ff700 1 Processor -- start 2023-03-03T14:51:04.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.787+0000 7fa1673ff700 1 -- start start 2023-03-03T14:51:04.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa1673ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd390 0x7fa1600c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa1673ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1600cd8e0 0x7fa160151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa1673ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa160153290 0x7fa1601f1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa1673ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1600b2360 con 0x7fa1600cd390 2023-03-03T14:51:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa1673ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1601f2920 con 0x7fa1600cd8e0 2023-03-03T14:51:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa1673ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16014f1c0 con 0x7fa160153290 2023-03-03T14:51:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa16499a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1600cd8e0 0x7fa160151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa16519b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd390 0x7fa1600c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa16499a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1600cd8e0 0x7fa160151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55800/0 (socket says 172.21.15.102:55800) 2023-03-03T14:51:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa16499a700 1 -- 172.21.15.102:0/1226029149 learned_addr learned my addr 172.21.15.102:0/1226029149 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:51:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.788+0000 7fa16599c700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa160153290 0x7fa1601f1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa16499a700 1 -- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa160153290 msgr2=0x7fa1601f1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa16499a700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa160153290 0x7fa1601f1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa16499a700 1 -- 172.21.15.102:0/1226029149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd390 msgr2=0x7fa1600c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa16499a700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd390 0x7fa1600c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa16499a700 1 -- 172.21.15.102:0/1226029149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1500094a0 con 0x7fa1600cd8e0 2023-03-03T14:51:04.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa16499a700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1600cd8e0 0x7fa160151750 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa150009fd0 tx=0x7fa150004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa155ffb700 1 -- 172.21.15.102:0/1226029149 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1500036a0 con 0x7fa1600cd8e0 2023-03-03T14:51:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa155ffb700 1 -- 172.21.15.102:0/1226029149 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15001f070 con 0x7fa1600cd8e0 2023-03-03T14:51:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa155ffb700 1 -- 172.21.15.102:0/1226029149 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150003a60 con 0x7fa1600cd8e0 2023-03-03T14:51:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.789+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1600ccde0 con 0x7fa1600cd8e0 2023-03-03T14:51:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.790+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16015d160 con 0x7fa1600cd8e0 2023-03-03T14:51:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.791+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1600cd0c0 con 0x7fa1600cd8e0 2023-03-03T14:51:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.792+0000 7fa155ffb700 1 -- 172.21.15.102:0/1226029149 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa15000f050 con 0x7fa1600cd8e0 2023-03-03T14:51:04.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.793+0000 7fa155ffb700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1440682f0 0x7fa14406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.793+0000 7fa155ffb700 1 -- 172.21.15.102:0/1226029149 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa150021070 con 0x7fa1600cd8e0 2023-03-03T14:51:04.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.793+0000 7fa16519b700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1440682f0 0x7fa14406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.794+0000 7fa16519b700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1440682f0 0x7fa14406a7a0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa15c007fd0 tx=0x7fa15c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:04.796+0000 7fa155ffb700 1 -- 172.21.15.102:0/1226029149 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa150053f60 con 0x7fa1600cd8e0 2023-03-03T14:51:05.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.028+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa16004e6e0 con 0x7fa1600cd8e0 2023-03-03T14:51:05.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.029+0000 7fa155ffb700 1 -- 172.21.15.102:0/1226029149 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa150057580 con 0x7fa1600cd8e0 2023-03-03T14:51:05.032 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:51:05.032 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:51:05.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1440682f0 msgr2=0x7fa14406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:05.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1440682f0 0x7fa14406a7a0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa15c007fd0 tx=0x7fa15c007dc0 comp rx=0 tx=0).stop 2023-03-03T14:51:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1600cd8e0 msgr2=0x7fa160151750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1600cd8e0 0x7fa160151750 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa150009fd0 tx=0x7fa150004750 comp rx=0 tx=0).stop 2023-03-03T14:51:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 shutdown_connections 2023-03-03T14:51:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1440682f0 0x7fa14406a7a0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa160153290 0x7fa1601f1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1600cd390 0x7fa1600c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 --2- 172.21.15.102:0/1226029149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1600cd8e0 0x7fa160151750 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:05.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 >> 172.21.15.102:0/1226029149 conn(0x7fa160065fb0 msgr2=0x7fa160158160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:51:05.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 shutdown_connections 2023-03-03T14:51:05.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:51:05.032+0000 7fa1673ff700 1 -- 172.21.15.102:0/1226029149 wait complete. 2023-03-03T14:51:05.036 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:51:05.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T14:51:18.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:18.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:48.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:48.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:18.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:18.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:37.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:52:37.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.324+0000 7f4b777ff700 1 Processor -- start 2023-03-03T14:52:37.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.325+0000 7f4b777ff700 1 -- start start 2023-03-03T14:52:37.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.326+0000 7f4b777ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b700c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:37.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.326+0000 7f4b777ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b7014ffb0 0x7f4b700c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:37.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.326+0000 7f4b777ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b700c3c80 con 0x7f4b700716c0 2023-03-03T14:52:37.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.326+0000 7f4b777ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700c41a0 con 0x7f4b7014ffb0 2023-03-03T14:52:37.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.326+0000 7f4b777ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70071cd0 con 0x7f4b7014fbe0 2023-03-03T14:52:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b7014ffb0 0x7f4b700c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b7559b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b700c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:37.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b7014ffb0 0x7f4b700c45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41802/0 (socket says 172.21.15.102:41802) 2023-03-03T14:52:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 -- 172.21.15.102:0/284325537 learned_addr learned my addr 172.21.15.102:0/284325537 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:52:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b7559b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b700c3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59630/0 (socket says 172.21.15.102:59630) 2023-03-03T14:52:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 -- 172.21.15.102:0/284325537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 msgr2=0x7f4b700c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:52:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 --2- 172.21.15.102:0/284325537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b700c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 -- 172.21.15.102:0/284325537 >> v1:172.21.15.102:6789/0 conn(0x7f4b700716c0 legacy=0x7f4b700706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 -- 172.21.15.102:0/284325537 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700c3ec0 con 0x7f4b7014ffb0 2023-03-03T14:52:37.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.327+0000 7f4b74d9a700 1 --2- 172.21.15.102:0/284325537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 0x7f4b700c45a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4b600098f0 tx=0x7f4b60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce0403719af55e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:37.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b67fff700 1 -- 172.21.15.102:0/284325537 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60011830 con 0x7f4b7014ffb0 2023-03-03T14:52:37.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b67fff700 1 -- 172.21.15.102:0/284325537 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60011990 con 0x7f4b7014ffb0 2023-03-03T14:52:37.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b67fff700 1 -- 172.21.15.102:0/284325537 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60011d20 con 0x7f4b7014ffb0 2023-03-03T14:52:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b777ff700 1 -- 172.21.15.102:0/284325537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 msgr2=0x7f4b700c45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b777ff700 1 --2- 172.21.15.102:0/284325537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 0x7f4b700c45a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4b600098f0 tx=0x7f4b60009c00 comp rx=0 tx=0).stop 2023-03-03T14:52:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b777ff700 1 -- 172.21.15.102:0/284325537 shutdown_connections 2023-03-03T14:52:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b777ff700 1 --2- 172.21.15.102:0/284325537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b700c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b777ff700 1 --2- 172.21.15.102:0/284325537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 0x7f4b700c45a0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.328+0000 7f4b777ff700 1 -- 172.21.15.102:0/284325537 >> 172.21.15.102:0/284325537 conn(0x7f4b7006af90 msgr2=0x7f4b7006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.329+0000 7f4b777ff700 1 -- 172.21.15.102:0/284325537 shutdown_connections 2023-03-03T14:52:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.329+0000 7f4b777ff700 1 -- 172.21.15.102:0/284325537 wait complete. 2023-03-03T14:52:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.331+0000 7f4b777ff700 1 Processor -- start 2023-03-03T14:52:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.332+0000 7f4b777ff700 1 -- start start 2023-03-03T14:52:37.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b777ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b700716c0 0x7f4b70156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:37.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b777ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b7014ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b777ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 0x7f4b700c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b777ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014e400 con 0x7f4b7014ffb0 2023-03-03T14:52:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b777ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014cf70 con 0x7f4b700716c0 2023-03-03T14:52:37.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b777ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70145a10 con 0x7f4b7014fbe0 2023-03-03T14:52:37.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b75d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 0x7f4b700c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:37.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b7559b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b700716c0 0x7f4b70156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b7559b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b700716c0 0x7f4b70156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:44088/0 (socket says 172.21.15.102:44088) 2023-03-03T14:52:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b7559b700 1 -- 172.21.15.102:0/2609480891 learned_addr learned my addr 172.21.15.102:0/2609480891 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:52:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.333+0000 7f4b74d9a700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b7014ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b7559b700 1 -- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 msgr2=0x7f4b7014ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b7559b700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b7014ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b7559b700 1 -- 172.21.15.102:0/2609480891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 msgr2=0x7f4b700c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:37.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b7559b700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 0x7f4b700c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b7559b700 1 -- 172.21.15.102:0/2609480891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b60009550 con 0x7f4b700716c0 2023-03-03T14:52:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b7559b700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b700716c0 0x7f4b70156530 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9b0 tx=0x7f4b6c00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b65ffb700 1 -- 172.21.15.102:0/2609480891 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c009710 con 0x7f4b700716c0 2023-03-03T14:52:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b65ffb700 1 -- 172.21.15.102:0/2609480891 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c00ee50 con 0x7f4b700716c0 2023-03-03T14:52:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b65ffb700 1 -- 172.21.15.102:0/2609480891 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c00ca50 con 0x7f4b700716c0 2023-03-03T14:52:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.334+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b70072250 con 0x7f4b700716c0 2023-03-03T14:52:37.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.335+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b70071f70 con 0x7f4b700716c0 2023-03-03T14:52:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.336+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b70072530 con 0x7f4b700716c0 2023-03-03T14:52:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.337+0000 7f4b65ffb700 1 -- 172.21.15.102:0/2609480891 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b6c023070 con 0x7f4b700716c0 2023-03-03T14:52:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.338+0000 7f4b65ffb700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b54068320 0x7f4b5406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.338+0000 7f4b65ffb700 1 -- 172.21.15.102:0/2609480891 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4b6c085f80 con 0x7f4b700716c0 2023-03-03T14:52:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.338+0000 7f4b74d9a700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b54068320 0x7f4b5406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:37.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.339+0000 7f4b74d9a700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b54068320 0x7f4b5406a7d0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4b60014040 tx=0x7f4b60008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:37.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.342+0000 7f4b65ffb700 1 -- 172.21.15.102:0/2609480891 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b6c052630 con 0x7f4b700716c0 2023-03-03T14:52:37.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.577+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b7004e6e0 con 0x7f4b700716c0 2023-03-03T14:52:37.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.577+0000 7f4b65ffb700 1 -- 172.21.15.102:0/2609480891 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4b6c055c50 con 0x7f4b700716c0 2023-03-03T14:52:37.580 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:52:37.581 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:52:37.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b54068320 msgr2=0x7f4b5406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:37.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b54068320 0x7f4b5406a7d0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4b60014040 tx=0x7f4b60008030 comp rx=0 tx=0).stop 2023-03-03T14:52:37.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b700716c0 msgr2=0x7f4b70156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:37.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b700716c0 0x7f4b70156530 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9b0 tx=0x7f4b6c00ecc0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 shutdown_connections 2023-03-03T14:52:37.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b7014ffb0 0x7f4b700c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b700716c0 0x7f4b70156530 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4b54068320 0x7f4b5406a7d0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 --2- 172.21.15.102:0/2609480891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4b7014fbe0 0x7f4b7014ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:37.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 >> 172.21.15.102:0/2609480891 conn(0x7f4b7006af90 msgr2=0x7f4b7006c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:37.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 shutdown_connections 2023-03-03T14:52:37.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:52:37.580+0000 7f4b777ff700 1 -- 172.21.15.102:0/2609480891 wait complete. 2023-03-03T14:52:37.585 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:52:37.610 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-03-03T14:52:49.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:49.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:19.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:19.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:49.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:49.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:50.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:53:50.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.865+0000 7f11ffbff700 1 Processor -- start 2023-03-03T14:53:50.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.866+0000 7f11ffbff700 1 -- start start 2023-03-03T14:53:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.867+0000 7f11ffbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11f8153290 0x7f11f8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.867+0000 7f11ffbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11f81593c0 0x7f11f80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.867+0000 7f11ffbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f8153660 con 0x7f11f80cd8e0 2023-03-03T14:53:50.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.867+0000 7f11ffbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f806c360 con 0x7f11f8153290 2023-03-03T14:53:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.867+0000 7f11ffbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f815cea0 con 0x7f11f81593c0 2023-03-03T14:53:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.867+0000 7f11fd99b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11f8153290 0x7f11f8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11f81593c0 0x7f11f80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11f81593c0 0x7f11f80cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34552/0 (socket says 172.21.15.102:34552) 2023-03-03T14:53:50.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 -- 172.21.15.102:0/1090557654 learned_addr learned my addr 172.21.15.102:0/1090557654 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:53:50.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd99b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11f8153290 0x7f11f8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46768/0 (socket says 172.21.15.102:46768) 2023-03-03T14:53:50.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fe19c700 1 --1- 172.21.15.102:0/1090557654 >> v1:172.21.15.172:6789/0 conn(0x7f11f80cd8e0 0x7f11f80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42334/0 (socket says 172.21.15.102:42334) 2023-03-03T14:53:50.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 -- 172.21.15.102:0/1090557654 >> v2:172.21.15.102:3300/0 conn(0x7f11f8153290 msgr2=0x7f11f8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:53:50.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 --2- 172.21.15.102:0/1090557654 >> v2:172.21.15.102:3300/0 conn(0x7f11f8153290 0x7f11f8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 -- 172.21.15.102:0/1090557654 >> v1:172.21.15.172:6789/0 conn(0x7f11f80cd8e0 legacy=0x7f11f80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 -- 172.21.15.102:0/1090557654 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f806c7f0 con 0x7f11f81593c0 2023-03-03T14:53:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.868+0000 7f11fd19a700 1 --2- 172.21.15.102:0/1090557654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f81593c0 0x7f11f80cde20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f11f8159fb0 tx=0x7f11e8009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=465dc0b8d0b96d9b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11fc999700 1 -- 172.21.15.102:0/1090557654 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8011830 con 0x7f11f81593c0 2023-03-03T14:53:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11fc999700 1 -- 172.21.15.102:0/1090557654 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e8011990 con 0x7f11f81593c0 2023-03-03T14:53:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11fc999700 1 -- 172.21.15.102:0/1090557654 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8011d20 con 0x7f11f81593c0 2023-03-03T14:53:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 -- 172.21.15.102:0/1090557654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f81593c0 msgr2=0x7f11f80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 --2- 172.21.15.102:0/1090557654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f81593c0 0x7f11f80cde20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f11f8159fb0 tx=0x7f11e8009bd0 comp rx=0 tx=0).stop 2023-03-03T14:53:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 -- 172.21.15.102:0/1090557654 shutdown_connections 2023-03-03T14:53:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 --2- 172.21.15.102:0/1090557654 >> v2:172.21.15.102:3300/0 conn(0x7f11f8153290 0x7f11f8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 --2- 172.21.15.102:0/1090557654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f81593c0 0x7f11f80cde20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 -- 172.21.15.102:0/1090557654 >> 172.21.15.102:0/1090557654 conn(0x7f11f8065fb0 msgr2=0x7f11f80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:53:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 -- 172.21.15.102:0/1090557654 shutdown_connections 2023-03-03T14:53:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.869+0000 7f11ffbff700 1 -- 172.21.15.102:0/1090557654 wait complete. 2023-03-03T14:53:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.871+0000 7f11ffbff700 1 Processor -- start 2023-03-03T14:53:50.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.872+0000 7f11ffbff700 1 -- start start 2023-03-03T14:53:50.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11ffbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f80cd8e0 0x7f11f80b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:50.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11ffbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f8153290 0x7f11f814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:50.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11ffbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11f81593c0 0x7f11f814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11ffbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f814d970 con 0x7f11f8153290 2023-03-03T14:53:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11ffbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80b24f0 con 0x7f11f80cd8e0 2023-03-03T14:53:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11ffbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f814ed90 con 0x7f11f81593c0 2023-03-03T14:53:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f8153290 0x7f11f814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f8153290 0x7f11f814f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46782/0 (socket says 172.21.15.102:46782) 2023-03-03T14:53:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 -- 172.21.15.102:0/4175685450 learned_addr learned my addr 172.21.15.102:0/4175685450 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd99b700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f80cd8e0 0x7f11f80b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fe19c700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11f81593c0 0x7f11f814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 -- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11f81593c0 msgr2=0x7f11f814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11f81593c0 0x7f11f814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 -- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f80cd8e0 msgr2=0x7f11f80b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f80cd8e0 0x7f11f80b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.873+0000 7f11fd19a700 1 -- 172.21.15.102:0/4175685450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e8009550 con 0x7f11f8153290 2023-03-03T14:53:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.874+0000 7f11fd19a700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f8153290 0x7f11f814f220 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f11e8000c00 tx=0x7f11e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:50.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.874+0000 7f11ee7fc700 1 -- 172.21.15.102:0/4175685450 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8004380 con 0x7f11f8153290 2023-03-03T14:53:50.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.874+0000 7f11ee7fc700 1 -- 172.21.15.102:0/4175685450 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e801f070 con 0x7f11f8153290 2023-03-03T14:53:50.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.874+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f80ccde0 con 0x7f11f8153290 2023-03-03T14:53:50.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.874+0000 7f11ee7fc700 1 -- 172.21.15.102:0/4175685450 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e80036a0 con 0x7f11f8153290 2023-03-03T14:53:50.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.874+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f815d160 con 0x7f11f8153290 2023-03-03T14:53:50.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.875+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f80cd0c0 con 0x7f11f8153290 2023-03-03T14:53:50.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.876+0000 7f11ee7fc700 1 -- 172.21.15.102:0/4175685450 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11e802a020 con 0x7f11f8153290 2023-03-03T14:53:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.878+0000 7f11ee7fc700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11dc0682f0 0x7f11dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.878+0000 7f11ee7fc700 1 -- 172.21.15.102:0/4175685450 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f11e80872e0 con 0x7f11f8153290 2023-03-03T14:53:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.878+0000 7f11fd99b700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11dc0682f0 0x7f11dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.879+0000 7f11fd99b700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11dc0682f0 0x7f11dc06a7a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009be0 tx=0x7f11f4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:50.882+0000 7f11ee7fc700 1 -- 172.21.15.102:0/4175685450 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11e8053970 con 0x7f11f8153290 2023-03-03T14:53:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.118+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11f804e6e0 con 0x7f11f8153290 2023-03-03T14:53:51.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.118+0000 7f11ee7fc700 1 -- 172.21.15.102:0/4175685450 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f11e8056f90 con 0x7f11f8153290 2023-03-03T14:53:51.120 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:53:51.120 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:53:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11dc0682f0 msgr2=0x7f11dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11dc0682f0 0x7f11dc06a7a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009be0 tx=0x7f11f4009210 comp rx=0 tx=0).stop 2023-03-03T14:53:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f8153290 msgr2=0x7f11f814f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f8153290 0x7f11f814f220 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f11e8000c00 tx=0x7f11e8004750 comp rx=0 tx=0).stop 2023-03-03T14:53:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 shutdown_connections 2023-03-03T14:53:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f11dc0682f0 0x7f11dc06a7a0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f11f81593c0 0x7f11f814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11f80cd8e0 0x7f11f80b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 --2- 172.21.15.102:0/4175685450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f8153290 0x7f11f814f220 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 >> 172.21.15.102:0/4175685450 conn(0x7f11f8065fb0 msgr2=0x7f11f815c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:53:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 shutdown_connections 2023-03-03T14:53:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:53:51.121+0000 7f11ffbff700 1 -- 172.21.15.102:0/4175685450 wait complete. 2023-03-03T14:53:51.125 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:53:51.150 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-03-03T14:54:01.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:54:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.406+0000 7ffbc0f9d700 1 Processor -- start 2023-03-03T14:54:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.406+0000 7ffbc0f9d700 1 -- start start 2023-03-03T14:54:01.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbc0f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffbbc0c9610 0x7ffbbc0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:01.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbc0f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc0ca980 0x7ffbbc1636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:01.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbc0f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc0c9ca0 con 0x7ffbbc1564b0 2023-03-03T14:54:01.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbc0f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbbc0b4920 con 0x7ffbbc0c9610 2023-03-03T14:54:01.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbc0f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc0b4c00 con 0x7ffbbc0ca980 2023-03-03T14:54:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbbbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc0ca980 0x7ffbbc1636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbbaffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffbbc0c9610 0x7ffbbc0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbbb7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ffbbc1564b0 0x7ffbbc0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51706/0 (socket says 172.21.15.102:51706) 2023-03-03T14:54:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.408+0000 7ffbbb7fe700 1 -- 172.21.15.102:0/1272976996 learned_addr learned my addr 172.21.15.102:0/1272976996 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:54:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbbbfff700 1 -- 172.21.15.102:0/1272976996 >> v2:172.21.15.172:3300/0 conn(0x7ffbbc0c9610 msgr2=0x7ffbbc0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:54:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbbbfff700 1 --2- 172.21.15.102:0/1272976996 >> v2:172.21.15.172:3300/0 conn(0x7ffbbc0c9610 0x7ffbbc0ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbbbfff700 1 -- 172.21.15.102:0/1272976996 >> v1:172.21.15.172:6789/0 conn(0x7ffbbc1564b0 legacy=0x7ffbbc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbbbfff700 1 -- 172.21.15.102:0/1272976996 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc0b46f0 con 0x7ffbbc0ca980 2023-03-03T14:54:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbbbfff700 1 --2- 172.21.15.102:0/1272976996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc0ca980 0x7ffbbc1636a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009d90 tx=0x7ffbb400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cdeac710a9c6d66 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbba7fc700 1 -- 172.21.15.102:0/1272976996 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb400e070 con 0x7ffbbc0ca980 2023-03-03T14:54:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbba7fc700 1 -- 172.21.15.102:0/1272976996 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb400bb20 con 0x7ffbbc0ca980 2023-03-03T14:54:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.409+0000 7ffbba7fc700 1 -- 172.21.15.102:0/1272976996 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4012430 con 0x7ffbbc0ca980 2023-03-03T14:54:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/1272976996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc0ca980 msgr2=0x7ffbbc1636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/1272976996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc0ca980 0x7ffbbc1636a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009d90 tx=0x7ffbb400b330 comp rx=0 tx=0).stop 2023-03-03T14:54:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/1272976996 shutdown_connections 2023-03-03T14:54:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/1272976996 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc0ca980 0x7ffbbc1636a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/1272976996 >> v2:172.21.15.172:3300/0 conn(0x7ffbbc0c9610 0x7ffbbc0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/1272976996 >> 172.21.15.102:0/1272976996 conn(0x7ffbbc065fb0 msgr2=0x7ffbbc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/1272976996 shutdown_connections 2023-03-03T14:54:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.410+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/1272976996 wait complete. 2023-03-03T14:54:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.413+0000 7ffbc0f9d700 1 Processor -- start 2023-03-03T14:54:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.413+0000 7ffbc0f9d700 1 -- start start 2023-03-03T14:54:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.414+0000 7ffbc0f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c9610 0x7ffbbc0c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.414+0000 7ffbc0f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0ca980 0x7ffbbc1f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.414+0000 7ffbc0f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc1564b0 0x7ffbbc14f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.414+0000 7ffbc0f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc150e10 con 0x7ffbbc0ca980 2023-03-03T14:54:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.414+0000 7ffbc0f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1f2700 con 0x7ffbbc0c9610 2023-03-03T14:54:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.414+0000 7ffbc0f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc150b30 con 0x7ffbbc1564b0 2023-03-03T14:54:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c9610 0x7ffbbc0c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c9610 0x7ffbbc0c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48610/0 (socket says 172.21.15.102:48610) 2023-03-03T14:54:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 -- 172.21.15.102:0/258749478 learned_addr learned my addr 172.21.15.102:0/258749478 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:54:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbbfff700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc1564b0 0x7ffbbc14f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:01.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbaffd700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0ca980 0x7ffbbc1f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:01.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 -- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc1564b0 msgr2=0x7ffbbc14f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc1564b0 0x7ffbbc14f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 -- 172.21.15.102:0/258749478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0ca980 msgr2=0x7ffbbc1f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0ca980 0x7ffbbc1f1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 -- 172.21.15.102:0/258749478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb4009a70 con 0x7ffbbc0c9610 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.415+0000 7ffbbb7fe700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c9610 0x7ffbbc0c32c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00c480 tx=0x7ffbac00c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.416+0000 7ffba3fff700 1 -- 172.21.15.102:0/258749478 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac005d70 con 0x7ffbbc0c9610 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.416+0000 7ffba3fff700 1 -- 172.21.15.102:0/258749478 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac00ce90 con 0x7ffbbc0c9610 2023-03-03T14:54:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.416+0000 7ffba3fff700 1 -- 172.21.15.102:0/258749478 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac0072d0 con 0x7ffbbc0c9610 2023-03-03T14:54:01.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.416+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc0b2210 con 0x7ffbbc0c9610 2023-03-03T14:54:01.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.416+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc0b1f30 con 0x7ffbbc0c9610 2023-03-03T14:54:01.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.417+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc156880 con 0x7ffbbc0c9610 2023-03-03T14:54:01.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.418+0000 7ffba3fff700 1 -- 172.21.15.102:0/258749478 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffbac019030 con 0x7ffbbc0c9610 2023-03-03T14:54:01.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.419+0000 7ffba3fff700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffba80682f0 0x7ffba806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:01.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.420+0000 7ffba3fff700 1 -- 172.21.15.102:0/258749478 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ffbac084e40 con 0x7ffbbc0c9610 2023-03-03T14:54:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.420+0000 7ffbbaffd700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffba80682f0 0x7ffba806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.420+0000 7ffbbaffd700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffba80682f0 0x7ffba806a7a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc14e650 tx=0x7ffbb0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:01.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.424+0000 7ffba3fff700 1 -- 172.21.15.102:0/258749478 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffbac04b5e0 con 0x7ffbbc0c9610 2023-03-03T14:54:01.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.660+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbbc14f790 con 0x7ffbbc0c9610 2023-03-03T14:54:01.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.661+0000 7ffba3fff700 1 -- 172.21.15.102:0/258749478 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ffbac017090 con 0x7ffbbc0c9610 2023-03-03T14:54:01.663 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:54:01.663 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:54:01.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffba80682f0 msgr2=0x7ffba806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:01.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffba80682f0 0x7ffba806a7a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc14e650 tx=0x7ffbb0009210 comp rx=0 tx=0).stop 2023-03-03T14:54:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c9610 msgr2=0x7ffbbc0c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c9610 0x7ffbbc0c32c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00c480 tx=0x7ffbac00c790 comp rx=0 tx=0).stop 2023-03-03T14:54:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 shutdown_connections 2023-03-03T14:54:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffba80682f0 0x7ffba806a7a0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbbc1564b0 0x7ffbbc14f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c9610 0x7ffbbc0c32c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 --2- 172.21.15.102:0/258749478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0ca980 0x7ffbbc1f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 >> 172.21.15.102:0/258749478 conn(0x7ffbbc065fb0 msgr2=0x7ffbbc158470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 shutdown_connections 2023-03-03T14:54:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:01.664+0000 7ffbc0f9d700 1 -- 172.21.15.102:0/258749478 wait complete. 2023-03-03T14:54:01.667 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:54:01.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-03-03T14:54:09.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:54:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.943+0000 7f9756f9d700 1 Processor -- start 2023-03-03T14:54:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.943+0000 7f9756f9d700 1 -- start start 2023-03-03T14:54:09.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f9756f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:09.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f9756f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97500c4780 0x7f9750161310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:09.947 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f9756f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500c1bc0 con 0x7f9750150c30 2023-03-03T14:54:09.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f9756f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97500c36a0 con 0x7f97500c4780 2023-03-03T14:54:09.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f9756f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9750156990 con 0x7f97500c6c40 2023-03-03T14:54:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f975579a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f975679c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97500c4780 0x7f9750161310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f9755f9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9750150c30 0x7f97500c5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:35734/0 (socket says 172.21.15.102:35734) 2023-03-03T14:54:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.945+0000 7f9755f9b700 1 -- 172.21.15.102:0/16444768 learned_addr learned my addr 172.21.15.102:0/16444768 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:54:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f9754f99700 1 -- 172.21.15.102:0/16444768 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375876197 0 0) 0x7f97500c1bc0 con 0x7f9750150c30 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f975679c700 1 -- 172.21.15.102:0/16444768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 msgr2=0x7f97500c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f975679c700 1 --2- 172.21.15.102:0/16444768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f975679c700 1 -- 172.21.15.102:0/16444768 >> v1:172.21.15.172:6789/0 conn(0x7f9750150c30 legacy=0x7f97500c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f975679c700 1 -- 172.21.15.102:0/16444768 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500c2740 con 0x7f97500c4780 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f975579a700 1 --2- 172.21.15.102:0/16444768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f975679c700 1 --2- 172.21.15.102:0/16444768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97500c4780 0x7f9750161310 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9748009d90 tx=0x7f974800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46546ff389eaeb6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f9754f99700 1 -- 172.21.15.102:0/16444768 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974800e070 con 0x7f97500c4780 2023-03-03T14:54:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f9754f99700 1 -- 172.21.15.102:0/16444768 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974800bb20 con 0x7f97500c4780 2023-03-03T14:54:09.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.946+0000 7f9754f99700 1 -- 172.21.15.102:0/16444768 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748012430 con 0x7f97500c4780 2023-03-03T14:54:09.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 -- 172.21.15.102:0/16444768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97500c4780 msgr2=0x7f9750161310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:09.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 --2- 172.21.15.102:0/16444768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97500c4780 0x7f9750161310 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9748009d90 tx=0x7f974800b330 comp rx=0 tx=0).stop 2023-03-03T14:54:09.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 -- 172.21.15.102:0/16444768 shutdown_connections 2023-03-03T14:54:09.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 --2- 172.21.15.102:0/16444768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97500c4780 0x7f9750161310 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:09.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 --2- 172.21.15.102:0/16444768 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:09.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 -- 172.21.15.102:0/16444768 >> 172.21.15.102:0/16444768 conn(0x7f97500ca7c0 msgr2=0x7f97500cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:09.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 -- 172.21.15.102:0/16444768 shutdown_connections 2023-03-03T14:54:09.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.947+0000 7f9756f9d700 1 -- 172.21.15.102:0/16444768 wait complete. 2023-03-03T14:54:09.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.950+0000 7f9756f9d700 1 Processor -- start 2023-03-03T14:54:09.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.950+0000 7f9756f9d700 1 -- start start 2023-03-03T14:54:09.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f9756f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97500c4780 0x7f9750071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:09.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f9756f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f9756f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9750150c30 0x7f9750157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f9756f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97500719d0 con 0x7f9750150c30 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f9756f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975015a420 con 0x7f97500c4780 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f9756f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975015a700 con 0x7f97500c6c40 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f9755f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97500c4780 0x7f9750071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:32872/0 (socket says 172.21.15.102:32872) 2023-03-03T14:54:09.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 -- 172.21.15.102:0/1384242332 learned_addr learned my addr 172.21.15.102:0/1384242332 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:54:09.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975679c700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9750150c30 0x7f9750157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 -- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97500c4780 msgr2=0x7f9750071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97500c4780 0x7f9750071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 -- 172.21.15.102:0/1384242332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9750150c30 msgr2=0x7f9750157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9750150c30 0x7f9750157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.952+0000 7f975579a700 1 -- 172.21.15.102:0/1384242332 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97400094a0 con 0x7f97500c6c40 2023-03-03T14:54:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.953+0000 7f975579a700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500cae00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f974000ebe0 tx=0x7f974000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.953+0000 7f97467fc700 1 -- 172.21.15.102:0/1384242332 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974000c850 con 0x7f97500c6c40 2023-03-03T14:54:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.953+0000 7f97467fc700 1 -- 172.21.15.102:0/1384242332 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974000c9b0 con 0x7f97500c6c40 2023-03-03T14:54:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.953+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9748009a70 con 0x7f97500c6c40 2023-03-03T14:54:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.953+0000 7f97467fc700 1 -- 172.21.15.102:0/1384242332 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974000cd40 con 0x7f97500c6c40 2023-03-03T14:54:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.953+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9750156ba0 con 0x7f97500c6c40 2023-03-03T14:54:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.954+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97500c5730 con 0x7f97500c6c40 2023-03-03T14:54:09.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.956+0000 7f97467fc700 1 -- 172.21.15.102:0/1384242332 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9740019030 con 0x7f97500c6c40 2023-03-03T14:54:09.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.957+0000 7f97467fc700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97340682f0 0x7f973406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:09.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.957+0000 7f97467fc700 1 -- 172.21.15.102:0/1384242332 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9740085d80 con 0x7f97500c6c40 2023-03-03T14:54:09.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.957+0000 7f9755f9b700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97340682f0 0x7f973406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:09.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.958+0000 7f9755f9b700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97340682f0 0x7f973406a7a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f974c000f60 tx=0x7f974c007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:09.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:09.962+0000 7f97467fc700 1 -- 172.21.15.102:0/1384242332 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9740052400 con 0x7f97500c6c40 2023-03-03T14:54:10.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.197+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97500ca630 con 0x7f97500c6c40 2023-03-03T14:54:10.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.199+0000 7f97467fc700 1 -- 172.21.15.102:0/1384242332 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9740017090 con 0x7f97500c6c40 2023-03-03T14:54:10.201 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:54:10.201 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:54:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97340682f0 msgr2=0x7f973406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97340682f0 0x7f973406a7a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f974c000f60 tx=0x7f974c007e70 comp rx=0 tx=0).stop 2023-03-03T14:54:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 msgr2=0x7f97500cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500cae00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f974000ebe0 tx=0x7f974000eef0 comp rx=0 tx=0).stop 2023-03-03T14:54:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 shutdown_connections 2023-03-03T14:54:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9750150c30 0x7f9750157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97500c4780 0x7f9750071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f97340682f0 0x7f973406a7a0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 --2- 172.21.15.102:0/1384242332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97500c6c40 0x7f97500cae00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 >> 172.21.15.102:0/1384242332 conn(0x7f97500ca7c0 msgr2=0x7f97500c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:10.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 shutdown_connections 2023-03-03T14:54:10.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:54:10.201+0000 7f9756f9d700 1 -- 172.21.15.102:0/1384242332 wait complete. 2023-03-03T14:54:10.206 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:54:10.230 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T14:54:19.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:19.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:49.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:49.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:19.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:55:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.487+0000 7f1fab9ff700 1 Processor -- start 2023-03-03T14:55:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.488+0000 7f1fab9ff700 1 -- start start 2023-03-03T14:55:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fab9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fab9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1fa41593c0 0x7f1fa40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fab9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa4153660 con 0x7f1fa40cd8e0 2023-03-03T14:55:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fab9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa406c360 con 0x7f1fa41593c0 2023-03-03T14:55:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fab9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa415cea0 con 0x7f1fa4153290 2023-03-03T14:55:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fa8f9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1fa41593c0 0x7f1fa40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fa979b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fa9f9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1fa40cd8e0 0x7f1fa40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36730/0 (socket says 172.21.15.102:36730) 2023-03-03T14:55:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.489+0000 7f1fa9f9c700 1 -- 172.21.15.102:0/2838976496 learned_addr learned my addr 172.21.15.102:0/2838976496 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:55:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.490+0000 7f1f9bfff700 1 -- 172.21.15.102:0/2838976496 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253302733 0 0) 0x7f1fa4153660 con 0x7f1fa40cd8e0 2023-03-03T14:55:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.490+0000 7f1fa8f9a700 1 -- 172.21.15.102:0/2838976496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 msgr2=0x7f1fa4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:55:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.490+0000 7f1fa8f9a700 1 --2- 172.21.15.102:0/2838976496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.490+0000 7f1fa8f9a700 1 -- 172.21.15.102:0/2838976496 >> v1:172.21.15.172:6789/0 conn(0x7f1fa40cd8e0 legacy=0x7f1fa40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.490+0000 7f1fa8f9a700 1 -- 172.21.15.102:0/2838976496 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa406c7f0 con 0x7f1fa41593c0 2023-03-03T14:55:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.490+0000 7f1fa8f9a700 1 --2- 172.21.15.102:0/2838976496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 0x7f1fa40cde20 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1f94009840 tx=0x7f1f94009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f504cddc4e855 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:55:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.490+0000 7f1f9bfff700 1 -- 172.21.15.102:0/2838976496 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94011830 con 0x7f1fa41593c0 2023-03-03T14:55:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1f9bfff700 1 -- 172.21.15.102:0/2838976496 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f94011990 con 0x7f1fa41593c0 2023-03-03T14:55:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1f9bfff700 1 -- 172.21.15.102:0/2838976496 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94011d20 con 0x7f1fa41593c0 2023-03-03T14:55:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 -- 172.21.15.102:0/2838976496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 msgr2=0x7f1fa40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/2838976496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 0x7f1fa40cde20 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1f94009840 tx=0x7f1f94009b50 comp rx=0 tx=0).stop 2023-03-03T14:55:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 -- 172.21.15.102:0/2838976496 shutdown_connections 2023-03-03T14:55:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/2838976496 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/2838976496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 0x7f1fa40cde20 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 -- 172.21.15.102:0/2838976496 >> 172.21.15.102:0/2838976496 conn(0x7f1fa4065fb0 msgr2=0x7f1fa40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:55:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 -- 172.21.15.102:0/2838976496 shutdown_connections 2023-03-03T14:55:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.491+0000 7f1fab9ff700 1 -- 172.21.15.102:0/2838976496 wait complete. 2023-03-03T14:55:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.493+0000 7f1fab9ff700 1 Processor -- start 2023-03-03T14:55:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.493+0000 7f1fab9ff700 1 -- start start 2023-03-03T14:55:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.494+0000 7f1fab9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cd8e0 0x7f1fa40cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.494+0000 7f1fab9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.494+0000 7f1fab9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 0x7f1fa414fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.494+0000 7f1fab9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa4153f50 con 0x7f1fa41593c0 2023-03-03T14:55:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.494+0000 7f1fab9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa41501a0 con 0x7f1fa40cd8e0 2023-03-03T14:55:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.494+0000 7f1fab9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40b2740 con 0x7f1fa4153290 2023-03-03T14:55:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa979b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cd8e0 0x7f1fa40cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa9f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 0x7f1fa414fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:56246/0 (socket says 172.21.15.102:56246) 2023-03-03T14:55:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 -- 172.21.15.102:0/1449485343 learned_addr learned my addr 172.21.15.102:0/1449485343 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:55:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 -- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cd8e0 msgr2=0x7f1fa40cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cd8e0 0x7f1fa40cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 -- 172.21.15.102:0/1449485343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 msgr2=0x7f1fa414fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:19.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 0x7f1fa414fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa8f9a700 1 -- 172.21.15.102:0/1449485343 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f940094a0 con 0x7f1fa4153290 2023-03-03T14:55:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.495+0000 7f1fa979b700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cd8e0 0x7f1fa40cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:55:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.496+0000 7f1fa8f9a700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4154370 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1f94009b90 tx=0x7f1f94004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:55:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.496+0000 7f1f99ffb700 1 -- 172.21.15.102:0/1449485343 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f940036a0 con 0x7f1fa4153290 2023-03-03T14:55:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.496+0000 7f1f99ffb700 1 -- 172.21.15.102:0/1449485343 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9401f070 con 0x7f1fa4153290 2023-03-03T14:55:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.496+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa40ccde0 con 0x7f1fa4153290 2023-03-03T14:55:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.496+0000 7f1f99ffb700 1 -- 172.21.15.102:0/1449485343 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94003a40 con 0x7f1fa4153290 2023-03-03T14:55:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.496+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa415d160 con 0x7f1fa4153290 2023-03-03T14:55:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.497+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa40cd0c0 con 0x7f1fa4153290 2023-03-03T14:55:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.498+0000 7f1f99ffb700 1 -- 172.21.15.102:0/1449485343 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f9400f050 con 0x7f1fa4153290 2023-03-03T14:55:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.499+0000 7f1f99ffb700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f880682f0 0x7f1f8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.500+0000 7f1f99ffb700 1 -- 172.21.15.102:0/1449485343 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1f94021070 con 0x7f1fa4153290 2023-03-03T14:55:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.500+0000 7f1fa979b700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f880682f0 0x7f1f8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.500+0000 7f1fa979b700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f880682f0 0x7f1f8806a7a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009fd0 tx=0x7f1fa0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:55:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.504+0000 7f1f99ffb700 1 -- 172.21.15.102:0/1449485343 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f94053d60 con 0x7f1fa4153290 2023-03-03T14:55:19.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.741+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fa404e6e0 con 0x7f1fa4153290 2023-03-03T14:55:19.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.742+0000 7f1f99ffb700 1 -- 172.21.15.102:0/1449485343 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1f94057380 con 0x7f1fa4153290 2023-03-03T14:55:19.743 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:55:19.743 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:55:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f880682f0 msgr2=0x7f1f8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f880682f0 0x7f1f8806a7a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009fd0 tx=0x7f1fa0009210 comp rx=0 tx=0).stop 2023-03-03T14:55:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 msgr2=0x7f1fa4154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4154370 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1f94009b90 tx=0x7f1f94004750 comp rx=0 tx=0).stop 2023-03-03T14:55:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 shutdown_connections 2023-03-03T14:55:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fa41593c0 0x7f1fa414fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa40cd8e0 0x7f1fa40cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f880682f0 0x7f1f8806a7a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 --2- 172.21.15.102:0/1449485343 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1fa4153290 0x7f1fa4154370 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 >> 172.21.15.102:0/1449485343 conn(0x7f1fa4065fb0 msgr2=0x7f1fa41e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:55:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 shutdown_connections 2023-03-03T14:55:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:55:19.744+0000 7f1fab9ff700 1 -- 172.21.15.102:0/1449485343 wait complete. 2023-03-03T14:55:19.748 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:55:19.773 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-03-03T14:55:20.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:20.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:50.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:50.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:06.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:56:07.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.022+0000 7f3d205ff700 1 Processor -- start 2023-03-03T14:56:07.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.022+0000 7f3d205ff700 1 -- start start 2023-03-03T14:56:07.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d205ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3d180cd8e0 0x7f3d180cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d205ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd390 0x7f3d180ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d205ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d18153660 con 0x7f3d18153290 2023-03-03T14:56:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d205ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1806c360 con 0x7f3d180cd8e0 2023-03-03T14:56:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d205ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1815cea0 con 0x7f3d180cd390 2023-03-03T14:56:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d1db9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3d180cd8e0 0x7f3d180cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d1db9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3d180cd8e0 0x7f3d180cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42574/0 (socket says 172.21.15.102:42574) 2023-03-03T14:56:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d1db9a700 1 -- 172.21.15.102:0/1715602524 learned_addr learned my addr 172.21.15.102:0/1715602524 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:56:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.024+0000 7f3d1eb9c700 1 --2- 172.21.15.102:0/1715602524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd390 0x7f3d180ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1db9a700 1 -- 172.21.15.102:0/1715602524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd390 msgr2=0x7f3d180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1db9a700 1 --2- 172.21.15.102:0/1715602524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd390 0x7f3d180ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1db9a700 1 -- 172.21.15.102:0/1715602524 >> v1:172.21.15.102:6789/0 conn(0x7f3d18153290 legacy=0x7f3d18158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1db9a700 1 -- 172.21.15.102:0/1715602524 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1806c7f0 con 0x7f3d180cd8e0 2023-03-03T14:56:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1d399700 1 -- 172.21.15.102:0/1715602524 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323781190 0 0) 0x7f3d18153660 con 0x7f3d18153290 2023-03-03T14:56:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1db9a700 1 --2- 172.21.15.102:0/1715602524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd8e0 0x7f3d180cde20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3d080098f0 tx=0x7f3d08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a76848ecc18e1ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1d399700 1 -- 172.21.15.102:0/1715602524 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08011830 con 0x7f3d180cd8e0 2023-03-03T14:56:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1d399700 1 -- 172.21.15.102:0/1715602524 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08011990 con 0x7f3d180cd8e0 2023-03-03T14:56:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.025+0000 7f3d1d399700 1 -- 172.21.15.102:0/1715602524 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08011d20 con 0x7f3d180cd8e0 2023-03-03T14:56:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 -- 172.21.15.102:0/1715602524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd8e0 msgr2=0x7f3d180cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 --2- 172.21.15.102:0/1715602524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd8e0 0x7f3d180cde20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3d080098f0 tx=0x7f3d08009c00 comp rx=0 tx=0).stop 2023-03-03T14:56:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 -- 172.21.15.102:0/1715602524 shutdown_connections 2023-03-03T14:56:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 --2- 172.21.15.102:0/1715602524 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd390 0x7f3d180ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 --2- 172.21.15.102:0/1715602524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd8e0 0x7f3d180cde20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 -- 172.21.15.102:0/1715602524 >> 172.21.15.102:0/1715602524 conn(0x7f3d18065fb0 msgr2=0x7f3d180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 -- 172.21.15.102:0/1715602524 shutdown_connections 2023-03-03T14:56:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.026+0000 7f3d205ff700 1 -- 172.21.15.102:0/1715602524 wait complete. 2023-03-03T14:56:07.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.028+0000 7f3d205ff700 1 Processor -- start 2023-03-03T14:56:07.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.028+0000 7f3d205ff700 1 -- start start 2023-03-03T14:56:07.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.029+0000 7f3d205ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd390 0x7f3d18154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:07.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.029+0000 7f3d205ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd8e0 0x7f3d181ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:07.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.029+0000 7f3d205ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d18153290 0x7f3d180c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:07.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.029+0000 7f3d205ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18154d40 con 0x7f3d180cd390 2023-03-03T14:56:07.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.029+0000 7f3d205ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18156370 con 0x7f3d18153290 2023-03-03T14:56:07.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.029+0000 7f3d205ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18153d70 con 0x7f3d180cd8e0 2023-03-03T14:56:07.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1e39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd390 0x7f3d18154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:07.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1e39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd390 0x7f3d18154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42586/0 (socket says 172.21.15.102:42586) 2023-03-03T14:56:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1e39b700 1 -- 172.21.15.102:0/2614166604 learned_addr learned my addr 172.21.15.102:0/2614166604 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:56:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1db9a700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd8e0 0x7f3d181ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1eb9c700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d18153290 0x7f3d180c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1db9a700 1 -- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d18153290 msgr2=0x7f3d180c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1db9a700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d18153290 0x7f3d180c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1db9a700 1 -- 172.21.15.102:0/2614166604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd390 msgr2=0x7f3d18154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1db9a700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd390 0x7f3d18154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1db9a700 1 -- 172.21.15.102:0/2614166604 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08009550 con 0x7f3d180cd8e0 2023-03-03T14:56:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.030+0000 7f3d1e39b700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd390 0x7f3d18154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:56:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.031+0000 7f3d1db9a700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd8e0 0x7f3d181ef370 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3d1808cc70 tx=0x7f3d08004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.031+0000 7f3d0effd700 1 -- 172.21.15.102:0/2614166604 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08004510 con 0x7f3d180cd8e0 2023-03-03T14:56:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.031+0000 7f3d0effd700 1 -- 172.21.15.102:0/2614166604 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0801e070 con 0x7f3d180cd8e0 2023-03-03T14:56:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.031+0000 7f3d0effd700 1 -- 172.21.15.102:0/2614166604 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d080038e0 con 0x7f3d180cd8e0 2023-03-03T14:56:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.031+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d180ccde0 con 0x7f3d180cd8e0 2023-03-03T14:56:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.031+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1815d160 con 0x7f3d180cd8e0 2023-03-03T14:56:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.032+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d180cd0c0 con 0x7f3d180cd8e0 2023-03-03T14:56:07.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.033+0000 7f3d0effd700 1 -- 172.21.15.102:0/2614166604 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d0800f050 con 0x7f3d180cd8e0 2023-03-03T14:56:07.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.035+0000 7f3d0effd700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:07.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.035+0000 7f3d0effd700 1 -- 172.21.15.102:0/2614166604 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3d08086d30 con 0x7f3d180cd8e0 2023-03-03T14:56:07.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.035+0000 7f3d1e39b700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:07.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.036+0000 7f3d1e39b700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3d14000f60 tx=0x7f3d14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:07.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.039+0000 7f3d0effd700 1 -- 172.21.15.102:0/2614166604 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d08053330 con 0x7f3d180cd8e0 2023-03-03T14:56:07.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.273+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d180ca380 con 0x7f3d180cd8e0 2023-03-03T14:56:07.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.275+0000 7f3d0effd700 1 -- 172.21.15.102:0/2614166604 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3d08015020 con 0x7f3d180cd8e0 2023-03-03T14:56:07.277 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:56:07.277 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:56:07.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cfc0682f0 msgr2=0x7f3cfc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:07.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3d14000f60 tx=0x7f3d14009040 comp rx=0 tx=0).stop 2023-03-03T14:56:07.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd8e0 msgr2=0x7f3d181ef370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:07.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd8e0 0x7f3d181ef370 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3d1808cc70 tx=0x7f3d08004750 comp rx=0 tx=0).stop 2023-03-03T14:56:07.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 shutdown_connections 2023-03-03T14:56:07.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d18153290 0x7f3d180c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d180cd390 0x7f3d18154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 --2- 172.21.15.102:0/2614166604 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3d180cd8e0 0x7f3d181ef370 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:07.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 >> 172.21.15.102:0/2614166604 conn(0x7f3d18065fb0 msgr2=0x7f3d181513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:07.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 shutdown_connections 2023-03-03T14:56:07.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:56:07.277+0000 7f3d205ff700 1 -- 172.21.15.102:0/2614166604 wait complete. 2023-03-03T14:56:07.282 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:56:07.306 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T14:56:20.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:20.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:50.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:50.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:14.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:57:14.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.555+0000 7f1e089ff700 1 Processor -- start 2023-03-03T14:57:14.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.555+0000 7f1e089ff700 1 -- start start 2023-03-03T14:57:14.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e089ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e040ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:14.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e089ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04153660 con 0x7f1e04153290 2023-03-03T14:57:14.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e089ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0406c360 con 0x7f1e040cd8e0 2023-03-03T14:57:14.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e089ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0415cea0 con 0x7f1e040cd390 2023-03-03T14:57:14.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e01d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1e040cd8e0 0x7f1e040cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:32872/0 (socket says 172.21.15.102:32872) 2023-03-03T14:57:14.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e02d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e040ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:14.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e01d9b700 1 -- 172.21.15.102:0/213433229 learned_addr learned my addr 172.21.15.102:0/213433229 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:57:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e0259c700 1 --1- 172.21.15.102:0/213433229 >> v1:172.21.15.102:6789/0 conn(0x7f1e04153290 0x7f1e04158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55028/0 (socket says 172.21.15.102:55028) 2023-03-03T14:57:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e02d9d700 1 -- 172.21.15.102:0/213433229 >> v1:172.21.15.172:6789/0 conn(0x7f1e040cd8e0 legacy=0x7f1e040cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.558+0000 7f1e02d9d700 1 -- 172.21.15.102:0/213433229 >> v1:172.21.15.102:6789/0 conn(0x7f1e04153290 legacy=0x7f1e04158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.557+0000 7f1e0159a700 1 -- 172.21.15.102:0/213433229 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584714970 0 0) 0x7f1e0406c360 con 0x7f1e040cd8e0 2023-03-03T14:57:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.558+0000 7f1e02d9d700 1 -- 172.21.15.102:0/213433229 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0406c7f0 con 0x7f1e040cd390 2023-03-03T14:57:14.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.558+0000 7f1e02d9d700 1 --2- 172.21.15.102:0/213433229 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e040ce530 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009dc0 tx=0x7f1dfc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2017c4f370e2635 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:14.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.558+0000 7f1e0159a700 1 -- 172.21.15.102:0/213433229 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc00e070 con 0x7f1e040cd390 2023-03-03T14:57:14.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.558+0000 7f1e0159a700 1 -- 172.21.15.102:0/213433229 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc00bbd0 con 0x7f1e040cd390 2023-03-03T14:57:14.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.558+0000 7f1e0159a700 1 -- 172.21.15.102:0/213433229 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc012430 con 0x7f1e040cd390 2023-03-03T14:57:14.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.559+0000 7f1e089ff700 1 -- 172.21.15.102:0/213433229 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 msgr2=0x7f1e040ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.559+0000 7f1e089ff700 1 --2- 172.21.15.102:0/213433229 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e040ce530 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009dc0 tx=0x7f1dfc00b330 comp rx=0 tx=0).stop 2023-03-03T14:57:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.559+0000 7f1e089ff700 1 -- 172.21.15.102:0/213433229 shutdown_connections 2023-03-03T14:57:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.559+0000 7f1e089ff700 1 --2- 172.21.15.102:0/213433229 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e040ce530 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.559+0000 7f1e089ff700 1 -- 172.21.15.102:0/213433229 >> 172.21.15.102:0/213433229 conn(0x7f1e04065fb0 msgr2=0x7f1e040663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:57:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.559+0000 7f1e089ff700 1 -- 172.21.15.102:0/213433229 shutdown_connections 2023-03-03T14:57:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.559+0000 7f1e089ff700 1 -- 172.21.15.102:0/213433229 wait complete. 2023-03-03T14:57:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.561+0000 7f1e089ff700 1 Processor -- start 2023-03-03T14:57:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.561+0000 7f1e089ff700 1 -- start start 2023-03-03T14:57:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e089ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e04154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e089ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e040cd8e0 0x7f1e041eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e089ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e04153290 0x7f1e041ef010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e089ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e041ee680 con 0x7f1e040cd8e0 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e089ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e040c6140 con 0x7f1e04153290 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e089ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04153890 con 0x7f1e040cd390 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e01d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e040cd8e0 0x7f1e041eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e0259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e04154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e02d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e04153290 0x7f1e041ef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:14.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e0259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e04154e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48590/0 (socket says 172.21.15.102:48590) 2023-03-03T14:57:14.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.562+0000 7f1e0259c700 1 -- 172.21.15.102:0/546450127 learned_addr learned my addr 172.21.15.102:0/546450127 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:57:14.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1e0259c700 1 -- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e04153290 msgr2=0x7f1e041ef010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:14.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1e0259c700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e04153290 0x7f1e041ef010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1e0259c700 1 -- 172.21.15.102:0/546450127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e040cd8e0 msgr2=0x7f1e041eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1e0259c700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e040cd8e0 0x7f1e041eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1e0259c700 1 -- 172.21.15.102:0/546450127 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc009a70 con 0x7f1e040cd390 2023-03-03T14:57:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1e0259c700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e04154e20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1df400c520 tx=0x7f1df400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1df2ffd700 1 -- 172.21.15.102:0/546450127 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df4005cb0 con 0x7f1e040cd390 2023-03-03T14:57:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1df2ffd700 1 -- 172.21.15.102:0/546450127 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df4005e10 con 0x7f1e040cd390 2023-03-03T14:57:14.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e040ccde0 con 0x7f1e040cd390 2023-03-03T14:57:14.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.563+0000 7f1df2ffd700 1 -- 172.21.15.102:0/546450127 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df40072d0 con 0x7f1e040cd390 2023-03-03T14:57:14.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.564+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0415d160 con 0x7f1e040cd390 2023-03-03T14:57:14.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.565+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e040cd0c0 con 0x7f1e040cd390 2023-03-03T14:57:14.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.566+0000 7f1df2ffd700 1 -- 172.21.15.102:0/546450127 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1df4019030 con 0x7f1e040cd390 2023-03-03T14:57:14.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.567+0000 7f1df2ffd700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de40682f0 0x7f1de406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:14.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.568+0000 7f1df2ffd700 1 -- 172.21.15.102:0/546450127 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1df4084fa0 con 0x7f1e040cd390 2023-03-03T14:57:14.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.568+0000 7f1e01d9b700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de40682f0 0x7f1de406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:14.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.568+0000 7f1e01d9b700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de40682f0 0x7f1de406a7a0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f1e04151f10 tx=0x7f1df8004a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:14.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.572+0000 7f1df2ffd700 1 -- 172.21.15.102:0/546450127 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1df404b5e0 con 0x7f1e040cd390 2023-03-03T14:57:14.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.803+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e041ee920 con 0x7f1e040cd390 2023-03-03T14:57:14.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.804+0000 7f1df2ffd700 1 -- 172.21.15.102:0/546450127 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1df4017070 con 0x7f1e040cd390 2023-03-03T14:57:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:57:14.807 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:57:14.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de40682f0 msgr2=0x7f1de406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:14.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de40682f0 0x7f1de406a7a0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f1e04151f10 tx=0x7f1df8004a30 comp rx=0 tx=0).stop 2023-03-03T14:57:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 msgr2=0x7f1e04154e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e04154e20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1df400c520 tx=0x7f1df400c830 comp rx=0 tx=0).stop 2023-03-03T14:57:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 shutdown_connections 2023-03-03T14:57:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1de40682f0 0x7f1de406a7a0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1e040cd390 0x7f1e04154e20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e04153290 0x7f1e041ef010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 --2- 172.21.15.102:0/546450127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e040cd8e0 0x7f1e041eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:14.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 >> 172.21.15.102:0/546450127 conn(0x7f1e04065fb0 msgr2=0x7f1e04158b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:57:14.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 shutdown_connections 2023-03-03T14:57:14.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:14.807+0000 7f1e089ff700 1 -- 172.21.15.102:0/546450127 wait complete. 2023-03-03T14:57:14.811 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:57:14.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2023-03-03T14:57:20.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:20.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:45.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:57:46.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.093+0000 7f423e5ff700 1 Processor -- start 2023-03-03T14:57:46.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.093+0000 7f423e5ff700 1 -- start start 2023-03-03T14:57:46.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.094+0000 7f423e5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42380cd390 0x7f42380ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:46.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.094+0000 7f423e5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4238153660 con 0x7f4238153290 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.094+0000 7f423e5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423806c360 con 0x7f42380cd8e0 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.094+0000 7f423e5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423815cea0 con 0x7f42380cd390 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f4237fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4238153290 0x7f4238158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35680/0 (socket says 172.21.15.102:35680) 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f4237fff700 1 -- 172.21.15.102:0/3436232067 learned_addr learned my addr 172.21.15.102:0/3436232067 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f423cb9c700 1 --2- 172.21.15.102:0/3436232067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42380cd390 0x7f42380ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f4236ffd700 1 -- 172.21.15.102:0/3436232067 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122719760 0 0) 0x7f4238153660 con 0x7f4238153290 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f4236ffd700 1 -- 172.21.15.102:0/3436232067 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42200035b0 con 0x7f4238153290 2023-03-03T14:57:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f4236ffd700 1 -- 172.21.15.102:0/3436232067 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842701603 0 0) 0x7f423806c360 con 0x7f42380cd8e0 2023-03-03T14:57:46.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f423cb9c700 1 -- 172.21.15.102:0/3436232067 >> v1:172.21.15.172:6789/0 conn(0x7f42380cd8e0 legacy=0x7f42380cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:46.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f423cb9c700 1 -- 172.21.15.102:0/3436232067 >> v1:172.21.15.102:6789/0 conn(0x7f4238153290 legacy=0x7f4238158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:46.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.095+0000 7f423cb9c700 1 -- 172.21.15.102:0/3436232067 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423806c7f0 con 0x7f42380cd390 2023-03-03T14:57:46.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f423cb9c700 1 --2- 172.21.15.102:0/3436232067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42380cd390 0x7f42380ce530 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4230009dc0 tx=0x7f423000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3365a529df1185d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:46.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f4236ffd700 1 -- 172.21.15.102:0/3436232067 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423000e070 con 0x7f42380cd390 2023-03-03T14:57:46.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f4236ffd700 1 -- 172.21.15.102:0/3436232067 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423000bb20 con 0x7f42380cd390 2023-03-03T14:57:46.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f4236ffd700 1 -- 172.21.15.102:0/3436232067 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423000bed0 con 0x7f42380cd390 2023-03-03T14:57:46.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f423e5ff700 1 -- 172.21.15.102:0/3436232067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42380cd390 msgr2=0x7f42380ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:46.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3436232067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42380cd390 0x7f42380ce530 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4230009dc0 tx=0x7f423000b330 comp rx=0 tx=0).stop 2023-03-03T14:57:46.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f423e5ff700 1 -- 172.21.15.102:0/3436232067 shutdown_connections 2023-03-03T14:57:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3436232067 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f42380cd390 0x7f42380ce530 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.096+0000 7f423e5ff700 1 -- 172.21.15.102:0/3436232067 >> 172.21.15.102:0/3436232067 conn(0x7f4238065fb0 msgr2=0x7f42380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:57:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.097+0000 7f423e5ff700 1 -- 172.21.15.102:0/3436232067 shutdown_connections 2023-03-03T14:57:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.097+0000 7f423e5ff700 1 -- 172.21.15.102:0/3436232067 wait complete. 2023-03-03T14:57:46.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.099+0000 7f423e5ff700 1 Processor -- start 2023-03-03T14:57:46.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.099+0000 7f423e5ff700 1 -- start start 2023-03-03T14:57:46.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f423e5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42380cd390 0x7f423814e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:46.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f423e5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42380cd8e0 0x7f42380c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:46.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f423e5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4238153290 0x7f42380c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:46.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f423e5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423814ebb0 con 0x7f42380cd390 2023-03-03T14:57:46.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f423e5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4238158380 con 0x7f42380cd8e0 2023-03-03T14:57:46.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f423e5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4238153de0 con 0x7f4238153290 2023-03-03T14:57:46.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f4237fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42380cd390 0x7f423814e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:46.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f4237fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42380cd390 0x7f423814e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45420/0 (socket says 172.21.15.102:45420) 2023-03-03T14:57:46.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f4237fff700 1 -- 172.21.15.102:0/3936772930 learned_addr learned my addr 172.21.15.102:0/3936772930 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:57:46.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f423cb9c700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4238153290 0x7f42380c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:46.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.100+0000 7f42377fe700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42380cd8e0 0x7f42380c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:46.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4237fff700 1 -- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4238153290 msgr2=0x7f42380c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:46.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4237fff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4238153290 0x7f42380c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4237fff700 1 -- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42380cd8e0 msgr2=0x7f42380c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4237fff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42380cd8e0 0x7f42380c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4237fff700 1 -- 172.21.15.102:0/3936772930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4230009a70 con 0x7f42380cd390 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4237fff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42380cd390 0x7f423814e670 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f422800c4d0 tx=0x7f422800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4234ff9700 1 -- 172.21.15.102:0/3936772930 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228005d70 con 0x7f42380cd390 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4234ff9700 1 -- 172.21.15.102:0/3936772930 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42280070d0 con 0x7f42380cd390 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f4234ff9700 1 -- 172.21.15.102:0/3936772930 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228007440 con 0x7f42380cd390 2023-03-03T14:57:46.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.101+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42380ccde0 con 0x7f42380cd390 2023-03-03T14:57:46.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.102+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423815d160 con 0x7f42380cd390 2023-03-03T14:57:46.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.103+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42380cd0c0 con 0x7f42380cd390 2023-03-03T14:57:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.104+0000 7f4234ff9700 1 -- 172.21.15.102:0/3936772930 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4228019030 con 0x7f42380cd390 2023-03-03T14:57:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.106+0000 7f4234ff9700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42240682f0 0x7f422406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.106+0000 7f4234ff9700 1 -- 172.21.15.102:0/3936772930 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4228084f70 con 0x7f42380cd390 2023-03-03T14:57:46.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.106+0000 7f42377fe700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42240682f0 0x7f422406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:46.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.107+0000 7f42377fe700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42240682f0 0x7f422406a7a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f42380c5720 tx=0x7f422c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:46.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.111+0000 7f4234ff9700 1 -- 172.21.15.102:0/3936772930 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f422804b5e0 con 0x7f42380cd390 2023-03-03T14:57:46.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.343+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4238153890 con 0x7f42380cd390 2023-03-03T14:57:46.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.344+0000 7f4234ff9700 1 -- 172.21.15.102:0/3936772930 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4228017020 con 0x7f42380cd390 2023-03-03T14:57:46.346 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:57:46.346 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:57:46.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42240682f0 msgr2=0x7f422406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:46.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42240682f0 0x7f422406a7a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f42380c5720 tx=0x7f422c009040 comp rx=0 tx=0).stop 2023-03-03T14:57:46.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42380cd390 msgr2=0x7f423814e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:46.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42380cd390 0x7f423814e670 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f422800c4d0 tx=0x7f422800c7e0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 shutdown_connections 2023-03-03T14:57:46.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f42240682f0 0x7f422406a7a0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4238153290 0x7f42380c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42380cd390 0x7f423814e670 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 --2- 172.21.15.102:0/3936772930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42380cd8e0 0x7f42380c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:46.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 >> 172.21.15.102:0/3936772930 conn(0x7f4238065fb0 msgr2=0x7f4238150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:57:46.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 shutdown_connections 2023-03-03T14:57:46.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:57:46.347+0000 7f423e5ff700 1 -- 172.21.15.102:0/3936772930 wait complete. 2023-03-03T14:57:46.353 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:57:46.376 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T14:57:50.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:50.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:20.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:20.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:34.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:58:34.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.630+0000 7f12d6fff700 1 Processor -- start 2023-03-03T14:58:34.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.631+0000 7f12d6fff700 1 -- start start 2023-03-03T14:58:34.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.632+0000 7f12d6fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f12d00cd390 0x7f12d00ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:34.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.632+0000 7f12d6fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d0153660 con 0x7f12d0153290 2023-03-03T14:58:34.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.632+0000 7f12d6fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d006c360 con 0x7f12d00cd8e0 2023-03-03T14:58:34.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.632+0000 7f12d6fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d015cea0 con 0x7f12d00cd390 2023-03-03T14:58:34.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.632+0000 7f12d4d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f12d0153290 0x7f12d0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58406/0 (socket says 172.21.15.102:58406) 2023-03-03T14:58:34.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.632+0000 7f12d4d9b700 1 -- 172.21.15.102:0/2346090883 learned_addr learned my addr 172.21.15.102:0/2346090883 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T14:58:34.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.632+0000 7f12d559c700 1 --2- 172.21.15.102:0/2346090883 >> v2:172.21.15.102:3300/0 conn(0x7f12d00cd390 0x7f12d00ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:34.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12cf7fe700 1 -- 172.21.15.102:0/2346090883 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172611329 0 0) 0x7f12d006c360 con 0x7f12d00cd8e0 2023-03-03T14:58:34.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12cf7fe700 1 -- 172.21.15.102:0/2346090883 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c00035b0 con 0x7f12d00cd8e0 2023-03-03T14:58:34.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12cf7fe700 1 -- 172.21.15.102:0/2346090883 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680855321 0 0) 0x7f12d0153660 con 0x7f12d0153290 2023-03-03T14:58:34.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12cf7fe700 1 -- 172.21.15.102:0/2346090883 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d006c360 con 0x7f12d0153290 2023-03-03T14:58:34.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12d559c700 1 -- 172.21.15.102:0/2346090883 >> v1:172.21.15.172:6789/0 conn(0x7f12d00cd8e0 legacy=0x7f12d00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:34.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12d559c700 1 -- 172.21.15.102:0/2346090883 >> v1:172.21.15.102:6789/0 conn(0x7f12d0153290 legacy=0x7f12d0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:34.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12d559c700 1 -- 172.21.15.102:0/2346090883 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d006c7f0 con 0x7f12d00cd390 2023-03-03T14:58:34.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.633+0000 7f12d559c700 1 --2- 172.21.15.102:0/2346090883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d00cd390 0x7f12d00ce530 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009d90 tx=0x7f12c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cce4459409aa21b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12cf7fe700 1 -- 172.21.15.102:0/2346090883 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c800e070 con 0x7f12d00cd390 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12cf7fe700 1 -- 172.21.15.102:0/2346090883 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c800bb20 con 0x7f12d00cd390 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12cf7fe700 1 -- 172.21.15.102:0/2346090883 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8012430 con 0x7f12d00cd390 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12d6fff700 1 -- 172.21.15.102:0/2346090883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d00cd390 msgr2=0x7f12d00ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12d6fff700 1 --2- 172.21.15.102:0/2346090883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d00cd390 0x7f12d00ce530 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009d90 tx=0x7f12c800b330 comp rx=0 tx=0).stop 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12d6fff700 1 -- 172.21.15.102:0/2346090883 shutdown_connections 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12d6fff700 1 --2- 172.21.15.102:0/2346090883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d00cd390 0x7f12d00ce530 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12d6fff700 1 -- 172.21.15.102:0/2346090883 >> 172.21.15.102:0/2346090883 conn(0x7f12d0065fb0 msgr2=0x7f12d00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:58:34.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.634+0000 7f12d6fff700 1 -- 172.21.15.102:0/2346090883 shutdown_connections 2023-03-03T14:58:34.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.635+0000 7f12d6fff700 1 -- 172.21.15.102:0/2346090883 wait complete. 2023-03-03T14:58:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.636+0000 7f12d6fff700 1 Processor -- start 2023-03-03T14:58:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.637+0000 7f12d6fff700 1 -- start start 2023-03-03T14:58:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d6fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12d00cd390 0x7f12d0153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d6fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12d00cd8e0 0x7f12d00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d6fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d0153290 0x7f12d01f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d6fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d00c5cf0 con 0x7f12d0153290 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d6fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d0156120 con 0x7f12d00cd8e0 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d6fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d01e4790 con 0x7f12d00cd390 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d559c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d0153290 0x7f12d01f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d4d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12d00cd390 0x7f12d0153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12cffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12d00cd8e0 0x7f12d00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d4d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12d00cd390 0x7f12d0153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39328/0 (socket says 172.21.15.102:39328) 2023-03-03T14:58:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d559c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d0153290 0x7f12d01f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35736/0 (socket says 172.21.15.102:35736) 2023-03-03T14:58:34.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.638+0000 7f12d4d9b700 1 -- 172.21.15.102:0/79101187 learned_addr learned my addr 172.21.15.102:0/79101187 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T14:58:34.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d559c700 1 -- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12d00cd390 msgr2=0x7f12d0153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:34.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d559c700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12d00cd390 0x7f12d0153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d559c700 1 -- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12d00cd8e0 msgr2=0x7f12d00cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:34.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d559c700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12d00cd8e0 0x7f12d00cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d559c700 1 -- 172.21.15.102:0/79101187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8009a70 con 0x7f12d0153290 2023-03-03T14:58:34.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12cffff700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12d00cd8e0 0x7f12d00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:58:34.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d4d9b700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12d00cd390 0x7f12d0153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:58:34.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d559c700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d0153290 0x7f12d01f2090 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f12c8016000 tx=0x7f12c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:34.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12cd7fa700 1 -- 172.21.15.102:0/79101187 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8012430 con 0x7f12d0153290 2023-03-03T14:58:34.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12cd7fa700 1 -- 172.21.15.102:0/79101187 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c8020070 con 0x7f12d0153290 2023-03-03T14:58:34.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d00ccde0 con 0x7f12d0153290 2023-03-03T14:58:34.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12cd7fa700 1 -- 172.21.15.102:0/79101187 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c801ba50 con 0x7f12d0153290 2023-03-03T14:58:34.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.639+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d015d160 con 0x7f12d0153290 2023-03-03T14:58:34.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.641+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d0154640 con 0x7f12d0153290 2023-03-03T14:58:34.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.642+0000 7f12cd7fa700 1 -- 172.21.15.102:0/79101187 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f12c802e070 con 0x7f12d0153290 2023-03-03T14:58:34.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.643+0000 7f12cd7fa700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f12b40682f0 0x7f12b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:34.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.643+0000 7f12cd7fa700 1 -- 172.21.15.102:0/79101187 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f12c8022070 con 0x7f12d0153290 2023-03-03T14:58:34.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.643+0000 7f12d4d9b700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f12b40682f0 0x7f12b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:34.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.644+0000 7f12d4d9b700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f12b40682f0 0x7f12b406a7a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f12c40011e0 tx=0x7f12c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:34.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.647+0000 7f12cd7fa700 1 -- 172.21.15.102:0/79101187 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12c8054330 con 0x7f12d0153290 2023-03-03T14:58:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.880+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12d00cd0c0 con 0x7f12d0153290 2023-03-03T14:58:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.881+0000 7f12cd7fa700 1 -- 172.21.15.102:0/79101187 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f12c802c070 con 0x7f12d0153290 2023-03-03T14:58:34.883 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T14:58:34.883 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T14:58:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f12b40682f0 msgr2=0x7f12b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:34.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f12b40682f0 0x7f12b406a7a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f12c40011e0 tx=0x7f12c4009040 comp rx=0 tx=0).stop 2023-03-03T14:58:34.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d0153290 msgr2=0x7f12d01f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:34.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d0153290 0x7f12d01f2090 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f12c8016000 tx=0x7f12c8004ad0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 shutdown_connections 2023-03-03T14:58:34.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f12b40682f0 0x7f12b406a7a0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f12d00cd390 0x7f12d0153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12d0153290 0x7f12d01f2090 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 --2- 172.21.15.102:0/79101187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12d00cd8e0 0x7f12d00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:34.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 >> 172.21.15.102:0/79101187 conn(0x7f12d0065fb0 msgr2=0x7f12d0158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:58:34.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 shutdown_connections 2023-03-03T14:58:34.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T14:58:34.884+0000 7f12d6fff700 1 -- 172.21.15.102:0/79101187 wait complete. 2023-03-03T14:58:34.889 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T14:58:34.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T14:58:50.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:51.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:21.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:21.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:51.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:51.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T14:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:00:21.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:21.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:29.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:00:30.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.165+0000 7f86f219d700 1 Processor -- start 2023-03-03T15:00:30.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.165+0000 7f86f219d700 1 -- start start 2023-03-03T15:00:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f219d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f86ec0ced00 0x7f86ec0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f219d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f86ec0c41b0 0x7f86ec0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f219d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ec0c3d20 con 0x7f86ec0b3970 2023-03-03T15:00:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f219d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ec0c4930 con 0x7f86ec0c41b0 2023-03-03T15:00:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f219d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ec0cf610 con 0x7f86ec0ced00 2023-03-03T15:00:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f199c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f86ec0c41b0 0x7f86ec0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:30.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f119b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f86ec0ced00 0x7f86ec0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:30.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f099a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f86ec0b3970 0x7f86ec158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38634/0 (socket says 172.21.15.102:38634) 2023-03-03T15:00:30.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.167+0000 7f86f099a700 1 -- 172.21.15.102:0/1674531503 learned_addr learned my addr 172.21.15.102:0/1674531503 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:00:30.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86f119b700 1 -- 172.21.15.102:0/1674531503 >> v2:172.21.15.102:3300/0 conn(0x7f86ec0c41b0 msgr2=0x7f86ec0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:00:30.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86f119b700 1 --2- 172.21.15.102:0/1674531503 >> v2:172.21.15.102:3300/0 conn(0x7f86ec0c41b0 0x7f86ec0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86f119b700 1 -- 172.21.15.102:0/1674531503 >> v1:172.21.15.172:6789/0 conn(0x7f86ec0b3970 legacy=0x7f86ec158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86f119b700 1 -- 172.21.15.102:0/1674531503 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec15cba0 con 0x7f86ec0ced00 2023-03-03T15:00:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86f199c700 1 --2- 172.21.15.102:0/1674531503 >> v2:172.21.15.102:3300/0 conn(0x7f86ec0c41b0 0x7f86ec0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:00:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86e3fff700 1 -- 172.21.15.102:0/1674531503 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895904086 0 0) 0x7f86ec0c3d20 con 0x7f86ec0b3970 2023-03-03T15:00:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86f119b700 1 --2- 172.21.15.102:0/1674531503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0ced00 0x7f86ec0c5a90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0b1b40 tx=0x7f86e8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec0e9ee7b7a01d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.168+0000 7f86e3fff700 1 -- 172.21.15.102:0/1674531503 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8011830 con 0x7f86ec0ced00 2023-03-03T15:00:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86e3fff700 1 -- 172.21.15.102:0/1674531503 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e8011990 con 0x7f86ec0ced00 2023-03-03T15:00:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86e3fff700 1 -- 172.21.15.102:0/1674531503 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8011d20 con 0x7f86ec0ced00 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 -- 172.21.15.102:0/1674531503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0ced00 msgr2=0x7f86ec0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 --2- 172.21.15.102:0/1674531503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0ced00 0x7f86ec0c5a90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0b1b40 tx=0x7f86e8009b20 comp rx=0 tx=0).stop 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 -- 172.21.15.102:0/1674531503 shutdown_connections 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 --2- 172.21.15.102:0/1674531503 >> v2:172.21.15.102:3300/0 conn(0x7f86ec0c41b0 0x7f86ec0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 --2- 172.21.15.102:0/1674531503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0ced00 0x7f86ec0c5a90 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 -- 172.21.15.102:0/1674531503 >> 172.21.15.102:0/1674531503 conn(0x7f86ec08b1c0 msgr2=0x7f86ec071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 -- 172.21.15.102:0/1674531503 shutdown_connections 2023-03-03T15:00:30.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.169+0000 7f86f219d700 1 -- 172.21.15.102:0/1674531503 wait complete. 2023-03-03T15:00:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.171+0000 7f86f219d700 1 Processor -- start 2023-03-03T15:00:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.171+0000 7f86f219d700 1 -- start start 2023-03-03T15:00:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.172+0000 7f86f219d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ec0b3970 0x7f86ec1e3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.172+0000 7f86f219d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0c41b0 0x7f86ec15a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.172+0000 7f86f219d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ec0ced00 0x7f86ec1e2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.172+0000 7f86f219d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1e4220 con 0x7f86ec0ced00 2023-03-03T15:00:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.172+0000 7f86f219d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1eef40 con 0x7f86ec0c41b0 2023-03-03T15:00:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.172+0000 7f86f219d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1e4860 con 0x7f86ec0b3970 2023-03-03T15:00:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ec0b3970 0x7f86ec1e3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f099a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0c41b0 0x7f86ec15a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f199c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ec0ced00 0x7f86ec1e2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ec0b3970 0x7f86ec1e3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:34818/0 (socket says 172.21.15.102:34818) 2023-03-03T15:00:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 -- 172.21.15.102:0/1042141997 learned_addr learned my addr 172.21.15.102:0/1042141997 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:00:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 -- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0c41b0 msgr2=0x7f86ec15a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0c41b0 0x7f86ec15a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 -- 172.21.15.102:0/1042141997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ec0ced00 msgr2=0x7f86ec1e2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ec0ced00 0x7f86ec1e2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 -- 172.21.15.102:0/1042141997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4009a70 con 0x7f86ec0b3970 2023-03-03T15:00:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f199c700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ec0ced00 0x7f86ec1e2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:00:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.173+0000 7f86f119b700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ec0b3970 0x7f86ec1e3ce0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f86e80090b0 tx=0x7f86e8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.174+0000 7f86e1ffb700 1 -- 172.21.15.102:0/1042141997 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e801a7a0 con 0x7f86ec0b3970 2023-03-03T15:00:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.174+0000 7f86e1ffb700 1 -- 172.21.15.102:0/1042141997 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e801a900 con 0x7f86ec0b3970 2023-03-03T15:00:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.174+0000 7f86e1ffb700 1 -- 172.21.15.102:0/1042141997 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e801ac70 con 0x7f86ec0b3970 2023-03-03T15:00:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.174+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e80094a0 con 0x7f86ec0b3970 2023-03-03T15:00:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.174+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec06bf20 con 0x7f86ec0b3970 2023-03-03T15:00:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.175+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec15ce80 con 0x7f86ec0b3970 2023-03-03T15:00:30.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.176+0000 7f86e1ffb700 1 -- 172.21.15.102:0/1042141997 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f86e802b030 con 0x7f86ec0b3970 2023-03-03T15:00:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.177+0000 7f86e1ffb700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86d00682f0 0x7f86d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.178+0000 7f86e1ffb700 1 -- 172.21.15.102:0/1042141997 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f86e8087280 con 0x7f86ec0b3970 2023-03-03T15:00:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.178+0000 7f86f099a700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86d00682f0 0x7f86d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:30.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.178+0000 7f86f099a700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86d00682f0 0x7f86d006a7a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f86ec14c350 tx=0x7f86dc007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:30.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.181+0000 7f86e1ffb700 1 -- 172.21.15.102:0/1042141997 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f86e8053a60 con 0x7f86ec0b3970 2023-03-03T15:00:30.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.413+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ec06c200 con 0x7f86ec0b3970 2023-03-03T15:00:30.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.414+0000 7f86e1ffb700 1 -- 172.21.15.102:0/1042141997 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f86e8057080 con 0x7f86ec0b3970 2023-03-03T15:00:30.421 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:00:30.421 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:00:30.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86d00682f0 msgr2=0x7f86d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86d00682f0 0x7f86d006a7a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f86ec14c350 tx=0x7f86dc007e70 comp rx=0 tx=0).stop 2023-03-03T15:00:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ec0b3970 msgr2=0x7f86ec1e3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ec0b3970 0x7f86ec1e3ce0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f86e80090b0 tx=0x7f86e8004750 comp rx=0 tx=0).stop 2023-03-03T15:00:30.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 shutdown_connections 2023-03-03T15:00:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86d00682f0 0x7f86d006a7a0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f86ec0b3970 0x7f86ec1e3ce0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ec0ced00 0x7f86ec1e2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 --2- 172.21.15.102:0/1042141997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec0c41b0 0x7f86ec15a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 >> 172.21.15.102:0/1042141997 conn(0x7f86ec08b1c0 msgr2=0x7f86ec0cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:00:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 shutdown_connections 2023-03-03T15:00:30.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:00:30.417+0000 7f86f219d700 1 -- 172.21.15.102:0/1042141997 wait complete. 2023-03-03T15:00:30.425 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:00:30.448 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2023-03-03T15:00:51.585 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:51.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:01.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:01:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.708+0000 7f65eb59d700 1 Processor -- start 2023-03-03T15:01:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.708+0000 7f65eb59d700 1 -- start start 2023-03-03T15:01:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65eb59d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65e4150c30 0x7f65e40c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65eb59d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65e40c5560 0x7f65e40c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65eb59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e40c4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65eb59d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e40c36a0 con 0x7f65e4150c30 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65eb59d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e4156990 con 0x7f65e40c5560 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65eb59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40c1bc0 con 0x7f65e40c6c40 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65ea59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65e4150c30 0x7f65e40c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65e9d9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65e40c5560 0x7f65e40c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65ea59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65e4150c30 0x7f65e40c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34606/0 (socket says 172.21.15.102:34606) 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65ea59b700 1 -- 172.21.15.102:0/1209915319 learned_addr learned my addr 172.21.15.102:0/1209915319 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.710+0000 7f65ead9c700 1 --2- 172.21.15.102:0/1209915319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e40c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65ea59b700 1 -- 172.21.15.102:0/1209915319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 msgr2=0x7f65e40c4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:01:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65ea59b700 1 --2- 172.21.15.102:0/1209915319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e40c4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65ea59b700 1 -- 172.21.15.102:0/1209915319 >> v2:172.21.15.172:3300/0 conn(0x7f65e40c5560 msgr2=0x7f65e40c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:01:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65ea59b700 1 --2- 172.21.15.102:0/1209915319 >> v2:172.21.15.172:3300/0 conn(0x7f65e40c5560 0x7f65e40c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65ea59b700 1 -- 172.21.15.102:0/1209915319 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e40c2740 con 0x7f65e4150c30 2023-03-03T15:01:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65ea59b700 1 --2- 172.21.15.102:0/1209915319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e40c5020 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f65e406d410 tx=0x7f65e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b03997d8697f0ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65e9599700 1 -- 172.21.15.102:0/1209915319 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0011830 con 0x7f65e4150c30 2023-03-03T15:01:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.711+0000 7f65e9599700 1 -- 172.21.15.102:0/1209915319 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0011990 con 0x7f65e4150c30 2023-03-03T15:01:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65e9599700 1 -- 172.21.15.102:0/1209915319 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0011d20 con 0x7f65e4150c30 2023-03-03T15:01:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 -- 172.21.15.102:0/1209915319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 msgr2=0x7f65e40c5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1209915319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e40c5020 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f65e406d410 tx=0x7f65e0009bd0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 -- 172.21.15.102:0/1209915319 shutdown_connections 2023-03-03T15:01:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1209915319 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e40c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1209915319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e40c5020 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1209915319 >> v2:172.21.15.172:3300/0 conn(0x7f65e40c5560 0x7f65e40c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 -- 172.21.15.102:0/1209915319 >> 172.21.15.102:0/1209915319 conn(0x7f65e40ca7c0 msgr2=0x7f65e40cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 -- 172.21.15.102:0/1209915319 shutdown_connections 2023-03-03T15:01:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.712+0000 7f65eb59d700 1 -- 172.21.15.102:0/1209915319 wait complete. 2023-03-03T15:01:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.714+0000 7f65eb59d700 1 Processor -- start 2023-03-03T15:01:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.714+0000 7f65eb59d700 1 -- start start 2023-03-03T15:01:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.715+0000 7f65eb59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c5560 0x7f65e40bb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.715+0000 7f65eb59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e4071c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.715+0000 7f65eb59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e4149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.715+0000 7f65eb59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e41ee3d0 con 0x7f65e4150c30 2023-03-03T15:01:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.715+0000 7f65eb59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e41eec30 con 0x7f65e40c5560 2023-03-03T15:01:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.715+0000 7f65eb59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e4070620 con 0x7f65e40c6c40 2023-03-03T15:01:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65ead9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e4149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65ea59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c5560 0x7f65e40bb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65e9d9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e4071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65ead9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e4149790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34620/0 (socket says 172.21.15.102:34620) 2023-03-03T15:01:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65ead9c700 1 -- 172.21.15.102:0/1488740264 learned_addr learned my addr 172.21.15.102:0/1488740264 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:01:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65ea59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c5560 0x7f65e40bb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56642/0 (socket says 172.21.15.102:56642) 2023-03-03T15:01:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65e9d9a700 1 -- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c5560 msgr2=0x7f65e40bb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65e9d9a700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c5560 0x7f65e40bb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65e9d9a700 1 -- 172.21.15.102:0/1488740264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 msgr2=0x7f65e4149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65e9d9a700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e4149790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65e9d9a700 1 -- 172.21.15.102:0/1488740264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0009550 con 0x7f65e40c6c40 2023-03-03T15:01:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.716+0000 7f65ead9c700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e4149790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:01:01.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.717+0000 7f65e9d9a700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e4071c20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f65d400e870 tx=0x7f65d400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:01.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.717+0000 7f65daffd700 1 -- 172.21.15.102:0/1488740264 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d4013680 con 0x7f65e40c6c40 2023-03-03T15:01:01.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.717+0000 7f65daffd700 1 -- 172.21.15.102:0/1488740264 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d40137e0 con 0x7f65e40c6c40 2023-03-03T15:01:01.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.717+0000 7f65daffd700 1 -- 172.21.15.102:0/1488740264 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d4013b50 con 0x7f65e40c6c40 2023-03-03T15:01:01.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.717+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e40c1e10 con 0x7f65e40c6c40 2023-03-03T15:01:01.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.717+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e4156ba0 con 0x7f65e40c6c40 2023-03-03T15:01:01.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.718+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e40c20f0 con 0x7f65e40c6c40 2023-03-03T15:01:01.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.720+0000 7f65daffd700 1 -- 172.21.15.102:0/1488740264 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f65d4023070 con 0x7f65e40c6c40 2023-03-03T15:01:01.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.721+0000 7f65daffd700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65c80682f0 0x7f65c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:01.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.721+0000 7f65daffd700 1 -- 172.21.15.102:0/1488740264 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f65d4085e30 con 0x7f65e40c6c40 2023-03-03T15:01:01.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.721+0000 7f65ea59b700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65c80682f0 0x7f65c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:01.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.722+0000 7f65ea59b700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65c80682f0 0x7f65c806a7a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009c10 tx=0x7f65e0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:01.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.725+0000 7f65daffd700 1 -- 172.21.15.102:0/1488740264 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f65d40523d0 con 0x7f65e40c6c40 2023-03-03T15:01:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.962+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65e404e6e0 con 0x7f65e40c6c40 2023-03-03T15:01:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.963+0000 7f65daffd700 1 -- 172.21.15.102:0/1488740264 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f65d4017090 con 0x7f65e40c6c40 2023-03-03T15:01:01.964 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:01:01.965 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:01:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65c80682f0 msgr2=0x7f65c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:01.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65c80682f0 0x7f65c806a7a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009c10 tx=0x7f65e0008030 comp rx=0 tx=0).stop 2023-03-03T15:01:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 msgr2=0x7f65e4071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e4071c20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f65d400e870 tx=0x7f65d400eb80 comp rx=0 tx=0).stop 2023-03-03T15:01:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 shutdown_connections 2023-03-03T15:01:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65e4150c30 0x7f65e4149790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65e40c5560 0x7f65e40bb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f65c80682f0 0x7f65c806a7a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 --2- 172.21.15.102:0/1488740264 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f65e40c6c40 0x7f65e4071c20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:01.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.965+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 >> 172.21.15.102:0/1488740264 conn(0x7f65e40ca7c0 msgr2=0x7f65e406dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:01.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.966+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 shutdown_connections 2023-03-03T15:01:01.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:01:01.966+0000 7f65eb59d700 1 -- 172.21.15.102:0/1488740264 wait complete. 2023-03-03T15:01:01.970 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:01:01.994 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 93 secs before thrashing 2023-03-03T15:01:21.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:21.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:51.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:51.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:21.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:22.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:34.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:02:35.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.247+0000 7f72e2dff700 1 Processor -- start 2023-03-03T15:02:35.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.248+0000 7f72e2dff700 1 -- start start 2023-03-03T15:02:35.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e2dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72dc153290 0x7f72dc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:35.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e2dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f72dc1593c0 0x7f72dc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:35.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e2dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc153660 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e2dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72dc06c360 con 0x7f72dc153290 2023-03-03T15:02:35.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e2dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72dc15cea0 con 0x7f72dc1593c0 2023-03-03T15:02:35.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e0b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72dc153290 0x7f72dc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:35.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e0b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72dc153290 0x7f72dc158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54106/0 (socket says 172.21.15.102:54106) 2023-03-03T15:02:35.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3910812066 learned_addr learned my addr 172.21.15.102:0/3910812066 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:02:35.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.249+0000 7f72dbfff700 1 --2- 172.21.15.102:0/3910812066 >> v2:172.21.15.172:3300/0 conn(0x7f72dc1593c0 0x7f72dc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:35.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.250+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3910812066 >> v2:172.21.15.172:3300/0 conn(0x7f72dc1593c0 msgr2=0x7f72dc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:02:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.250+0000 7f72e0b9b700 1 --2- 172.21.15.102:0/3910812066 >> v2:172.21.15.172:3300/0 conn(0x7f72dc1593c0 0x7f72dc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.250+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3910812066 >> v1:172.21.15.172:6789/0 conn(0x7f72dc0cd8e0 legacy=0x7f72dc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.250+0000 7f72db7fe700 1 -- 172.21.15.102:0/3910812066 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251953566 0 0) 0x7f72dc153660 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.250+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3910812066 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc06c7f0 con 0x7f72dc153290 2023-03-03T15:02:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.250+0000 7f72e0b9b700 1 --2- 172.21.15.102:0/3910812066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc153290 0x7f72dc158e80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f72dc159fb0 tx=0x7f72d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aeb5b5ad7c8a8a1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:02:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72db7fe700 1 -- 172.21.15.102:0/3910812066 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0011830 con 0x7f72dc153290 2023-03-03T15:02:35.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72db7fe700 1 -- 172.21.15.102:0/3910812066 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d0011990 con 0x7f72dc153290 2023-03-03T15:02:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72db7fe700 1 -- 172.21.15.102:0/3910812066 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0011d20 con 0x7f72dc153290 2023-03-03T15:02:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 -- 172.21.15.102:0/3910812066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc153290 msgr2=0x7f72dc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3910812066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc153290 0x7f72dc158e80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f72dc159fb0 tx=0x7f72d0009bd0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 -- 172.21.15.102:0/3910812066 shutdown_connections 2023-03-03T15:02:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3910812066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc153290 0x7f72dc158e80 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3910812066 >> v2:172.21.15.172:3300/0 conn(0x7f72dc1593c0 0x7f72dc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 -- 172.21.15.102:0/3910812066 >> 172.21.15.102:0/3910812066 conn(0x7f72dc065fb0 msgr2=0x7f72dc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:02:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 -- 172.21.15.102:0/3910812066 shutdown_connections 2023-03-03T15:02:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.251+0000 7f72e2dff700 1 -- 172.21.15.102:0/3910812066 wait complete. 2023-03-03T15:02:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.253+0000 7f72e2dff700 1 Processor -- start 2023-03-03T15:02:35.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.253+0000 7f72e2dff700 1 -- start start 2023-03-03T15:02:35.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.254+0000 7f72e2dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72dc0cd8e0 0x7f72dc1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:35.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.254+0000 7f72e2dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72dc153290 0x7f72dc0cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:35.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.254+0000 7f72e2dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc1593c0 0x7f72dc14d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:35.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e2dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc0b4350 con 0x7f72dc1593c0 2023-03-03T15:02:35.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e2dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc0b4820 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e2dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc0b2460 con 0x7f72dc153290 2023-03-03T15:02:35.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72dc0cd8e0 0x7f72dc1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:35.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72dbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72dc153290 0x7f72dc0cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:35.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72dc0cd8e0 0x7f72dc1537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:33412/0 (socket says 172.21.15.102:33412) 2023-03-03T15:02:35.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3012144914 learned_addr learned my addr 172.21.15.102:0/3012144914 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:02:35.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e139c700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc1593c0 0x7f72dc14d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:35.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72dc153290 msgr2=0x7f72dc0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:35.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72dc153290 0x7f72dc0cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3012144914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc1593c0 msgr2=0x7f72dc14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:35.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc1593c0 0x7f72dc14d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e0b9b700 1 -- 172.21.15.102:0/3012144914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0009550 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72dbfff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72dc153290 0x7f72dc0cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:02:35.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.255+0000 7f72e139c700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc1593c0 0x7f72dc14d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:02:35.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.256+0000 7f72e0b9b700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72dc0cd8e0 0x7f72dc1537f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f72dc074540 tx=0x7f72d0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:02:35.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.256+0000 7f72d97fa700 1 -- 172.21.15.102:0/3012144914 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0004510 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.256+0000 7f72d97fa700 1 -- 172.21.15.102:0/3012144914 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d00036a0 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.256+0000 7f72d97fa700 1 -- 172.21.15.102:0/3012144914 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0003a10 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.256+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc0ccde0 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.256+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc15d160 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.257+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc0cd0c0 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.258+0000 7f72d97fa700 1 -- 172.21.15.102:0/3012144914 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72d002d070 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.260+0000 7f72d97fa700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72c00682f0 0x7f72c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:35.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.260+0000 7f72d97fa700 1 -- 172.21.15.102:0/3012144914 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f72d0021030 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.260+0000 7f72dbfff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72c00682f0 0x7f72c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:35.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.261+0000 7f72dbfff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72c00682f0 0x7f72c006a7a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0b4d90 tx=0x7f72c8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:02:35.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.264+0000 7f72d97fa700 1 -- 172.21.15.102:0/3012144914 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72d0053ac0 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.496+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72dc153d30 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.497+0000 7f72d97fa700 1 -- 172.21.15.102:0/3012144914 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f72d0015090 con 0x7f72dc0cd8e0 2023-03-03T15:02:35.498 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:02:35.499 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:02:35.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72c00682f0 msgr2=0x7f72c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:35.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72c00682f0 0x7f72c006a7a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0b4d90 tx=0x7f72c8009210 comp rx=0 tx=0).stop 2023-03-03T15:02:35.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72dc0cd8e0 msgr2=0x7f72dc1537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:35.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72dc0cd8e0 0x7f72dc1537f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f72dc074540 tx=0x7f72d0004750 comp rx=0 tx=0).stop 2023-03-03T15:02:35.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 shutdown_connections 2023-03-03T15:02:35.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc1593c0 0x7f72dc14d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72dc0cd8e0 0x7f72dc1537f0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f72c00682f0 0x7f72c006a7a0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 --2- 172.21.15.102:0/3012144914 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f72dc153290 0x7f72dc0cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:35.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.499+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 >> 172.21.15.102:0/3012144914 conn(0x7f72dc065fb0 msgr2=0x7f72dc157700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:02:35.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.500+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 shutdown_connections 2023-03-03T15:02:35.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:02:35.500+0000 7f72e2dff700 1 -- 172.21.15.102:0/3012144914 wait complete. 2023-03-03T15:02:35.503 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:02:35.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 50 secs before thrashing 2023-03-03T15:02:52.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:52.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:22.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:22.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:25.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:03:25.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.781+0000 7f0b42f9d700 1 Processor -- start 2023-03-03T15:03:25.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.781+0000 7f0b42f9d700 1 -- start start 2023-03-03T15:03:25.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b42f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0b3c0b3970 0x7f0b3c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b42f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0b3c0c41b0 0x7f0b3c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b42f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c0c3d20 con 0x7f0b3c0ced00 2023-03-03T15:03:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b42f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c0c4930 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b42f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c0cf610 con 0x7f0b3c0b3970 2023-03-03T15:03:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b41f9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0b3c0ced00 0x7f0b3c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59748/0 (socket says 172.21.15.102:59748) 2023-03-03T15:03:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b41f9b700 1 -- 172.21.15.102:0/1391043631 learned_addr learned my addr 172.21.15.102:0/1391043631 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:03:25.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b4279c700 1 --2- 172.21.15.102:0/1391043631 >> v2:172.21.15.102:3300/0 conn(0x7f0b3c0c41b0 0x7f0b3c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:25.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.783+0000 7f0b4179a700 1 --2- 172.21.15.102:0/1391043631 >> v2:172.21.15.172:3300/0 conn(0x7f0b3c0b3970 0x7f0b3c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:25.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b40f99700 1 -- 172.21.15.102:0/1391043631 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113152001 0 0) 0x7f0b3c0c3d20 con 0x7f0b3c0ced00 2023-03-03T15:03:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b40f99700 1 -- 172.21.15.102:0/1391043631 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b280035b0 con 0x7f0b3c0ced00 2023-03-03T15:03:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b4279c700 1 -- 172.21.15.102:0/1391043631 >> v2:172.21.15.172:3300/0 conn(0x7f0b3c0b3970 msgr2=0x7f0b3c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:03:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b4279c700 1 --2- 172.21.15.102:0/1391043631 >> v2:172.21.15.172:3300/0 conn(0x7f0b3c0b3970 0x7f0b3c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b4279c700 1 -- 172.21.15.102:0/1391043631 >> v1:172.21.15.102:6789/0 conn(0x7f0b3c0ced00 legacy=0x7f0b3c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b4279c700 1 -- 172.21.15.102:0/1391043631 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c15cba0 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b4179a700 1 --2- 172.21.15.102:0/1391043631 >> v2:172.21.15.172:3300/0 conn(0x7f0b3c0b3970 0x7f0b3c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:03:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b4279c700 1 --2- 172.21.15.102:0/1391043631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c0cf0d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0c4ba0 tx=0x7f0b3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733fda0efd0c73cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:03:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.784+0000 7f0b40f99700 1 -- 172.21.15.102:0/1391043631 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3400e070 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b40f99700 1 -- 172.21.15.102:0/1391043631 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3400bae0 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b40f99700 1 -- 172.21.15.102:0/1391043631 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3400be50 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1391043631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0c41b0 msgr2=0x7f0b3c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1391043631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c0cf0d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0c4ba0 tx=0x7f0b3400b330 comp rx=0 tx=0).stop 2023-03-03T15:03:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1391043631 shutdown_connections 2023-03-03T15:03:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1391043631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c0cf0d0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1391043631 >> v2:172.21.15.172:3300/0 conn(0x7f0b3c0b3970 0x7f0b3c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1391043631 >> 172.21.15.102:0/1391043631 conn(0x7f0b3c08b1c0 msgr2=0x7f0b3c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:03:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1391043631 shutdown_connections 2023-03-03T15:03:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.785+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1391043631 wait complete. 2023-03-03T15:03:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.788+0000 7f0b42f9d700 1 Processor -- start 2023-03-03T15:03:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.788+0000 7f0b42f9d700 1 -- start start 2023-03-03T15:03:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b42f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b3c0b3970 0x7f0b3c14dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b42f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c14e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b42f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0ced00 0x7f0b3c1e2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b42f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c1e1fa0 con 0x7f0b3c0ced00 2023-03-03T15:03:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b42f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c1ee650 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b42f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c1ee930 con 0x7f0b3c0b3970 2023-03-03T15:03:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b4279c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0ced00 0x7f0b3c1e2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b4179a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c14e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b4179a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c14e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:51930/0 (socket says 172.21.15.102:51930) 2023-03-03T15:03:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b4179a700 1 -- 172.21.15.102:0/1502952207 learned_addr learned my addr 172.21.15.102:0/1502952207 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:03:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.790+0000 7f0b41f9b700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b3c0b3970 0x7f0b3c14dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b4179a700 1 -- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b3c0b3970 msgr2=0x7f0b3c14dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b4179a700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b3c0b3970 0x7f0b3c14dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b4179a700 1 -- 172.21.15.102:0/1502952207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0ced00 msgr2=0x7f0b3c1e2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b4179a700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0ced00 0x7f0b3c1e2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b4179a700 1 -- 172.21.15.102:0/1502952207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0094a0 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b4279c700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0ced00 0x7f0b3c1e2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:03:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b4179a700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c14e2b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ebe0 tx=0x7f0b2c00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:03:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b327fc700 1 -- 172.21.15.102:0/1502952207 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c009760 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b327fc700 1 -- 172.21.15.102:0/1502952207 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c00c850 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b34009a70 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b327fc700 1 -- 172.21.15.102:0/1502952207 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c00cbe0 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.791+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c06bf20 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.792+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c15ce80 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.798 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.794+0000 7f0b327fc700 1 -- 172.21.15.102:0/1502952207 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b2c019030 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.795+0000 7f0b327fc700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b200682f0 0x7f0b2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:25.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.795+0000 7f0b327fc700 1 -- 172.21.15.102:0/1502952207 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0b2c085d00 con 0x7f0b3c0c41b0 2023-03-03T15:03:25.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.795+0000 7f0b41f9b700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b200682f0 0x7f0b2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:25.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.796+0000 7f0b41f9b700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b200682f0 0x7f0b2006a7a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0b38007fd0 tx=0x7f0b38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:03:25.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:25.799+0000 7f0b327fc700 1 -- 172.21.15.102:0/1502952207 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b2c052380 con 0x7f0b3c0c41b0 2023-03-03T15:03:26.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.032+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b3c06c200 con 0x7f0b3c0c41b0 2023-03-03T15:03:26.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.033+0000 7f0b327fc700 1 -- 172.21.15.102:0/1502952207 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0b2c017090 con 0x7f0b3c0c41b0 2023-03-03T15:03:26.034 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:03:26.035 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:03:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b200682f0 msgr2=0x7f0b2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b200682f0 0x7f0b2006a7a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0b38007fd0 tx=0x7f0b38009040 comp rx=0 tx=0).stop 2023-03-03T15:03:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3c0c41b0 msgr2=0x7f0b3c14e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c14e2b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ebe0 tx=0x7f0b2c00eef0 comp rx=0 tx=0).stop 2023-03-03T15:03:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 shutdown_connections 2023-03-03T15:03:26.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0b200682f0 0x7f0b2006a7a0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:26.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0b3c0b3970 0x7f0b3c14dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:26.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c0ced00 0x7f0b3c1e2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:26.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 --2- 172.21.15.102:0/1502952207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3c0c41b0 0x7f0b3c14e2b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:26.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 >> 172.21.15.102:0/1502952207 conn(0x7f0b3c08b1c0 msgr2=0x7f0b3c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:03:26.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 shutdown_connections 2023-03-03T15:03:26.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:03:26.035+0000 7f0b42f9d700 1 -- 172.21.15.102:0/1502952207 wait complete. 2023-03-03T15:03:26.039 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:03:26.064 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-03-03T15:03:52.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:52.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:22.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:22.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:52.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:52.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:19.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:05:19.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.314+0000 7fe73cfff700 1 Processor -- start 2023-03-03T15:05:19.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.314+0000 7fe73cfff700 1 -- start start 2023-03-03T15:05:19.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.315+0000 7fe73cfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe7380cd390 0x7fe7380ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:19.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.315+0000 7fe73cfff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe738153660 con 0x7fe7380cd8e0 2023-03-03T15:05:19.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.315+0000 7fe73cfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73806c360 con 0x7fe738153290 2023-03-03T15:05:19.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.315+0000 7fe73cfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73815cea0 con 0x7fe7380cd390 2023-03-03T15:05:19.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.316+0000 7fe735d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe7380cd8e0 0x7fe7380cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34714/0 (socket says 172.21.15.102:34714) 2023-03-03T15:05:19.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.316+0000 7fe735d9b700 1 -- 172.21.15.102:0/3216542601 learned_addr learned my addr 172.21.15.102:0/3216542601 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:05:19.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.316+0000 7fe736d9d700 1 --2- 172.21.15.102:0/3216542601 >> v2:172.21.15.172:3300/0 conn(0x7fe7380cd390 0x7fe7380ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:19.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.316+0000 7fe736d9d700 1 -- 172.21.15.102:0/3216542601 >> v1:172.21.15.172:6789/0 conn(0x7fe738153290 legacy=0x7fe738158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:19.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.316+0000 7fe736d9d700 1 -- 172.21.15.102:0/3216542601 >> v1:172.21.15.102:6789/0 conn(0x7fe7380cd8e0 legacy=0x7fe7380cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:19.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.316+0000 7fe73559a700 1 -- 172.21.15.102:0/3216542601 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052330813 0 0) 0x7fe738153660 con 0x7fe7380cd8e0 2023-03-03T15:05:19.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.316+0000 7fe736d9d700 1 -- 172.21.15.102:0/3216542601 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73806c7f0 con 0x7fe7380cd390 2023-03-03T15:05:19.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe736d9d700 1 --2- 172.21.15.102:0/3216542601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd390 0x7fe7380ce530 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe730009d90 tx=0x7fe73000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e7503efaed75445 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73559a700 1 -- 172.21.15.102:0/3216542601 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73000e070 con 0x7fe7380cd390 2023-03-03T15:05:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73559a700 1 -- 172.21.15.102:0/3216542601 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73000bbd0 con 0x7fe7380cd390 2023-03-03T15:05:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73559a700 1 -- 172.21.15.102:0/3216542601 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730012430 con 0x7fe7380cd390 2023-03-03T15:05:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73cfff700 1 -- 172.21.15.102:0/3216542601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd390 msgr2=0x7fe7380ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73cfff700 1 --2- 172.21.15.102:0/3216542601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd390 0x7fe7380ce530 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe730009d90 tx=0x7fe73000b330 comp rx=0 tx=0).stop 2023-03-03T15:05:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73cfff700 1 -- 172.21.15.102:0/3216542601 shutdown_connections 2023-03-03T15:05:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73cfff700 1 --2- 172.21.15.102:0/3216542601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd390 0x7fe7380ce530 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.317+0000 7fe73cfff700 1 -- 172.21.15.102:0/3216542601 >> 172.21.15.102:0/3216542601 conn(0x7fe738065fb0 msgr2=0x7fe7380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:05:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.318+0000 7fe73cfff700 1 -- 172.21.15.102:0/3216542601 shutdown_connections 2023-03-03T15:05:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.318+0000 7fe73cfff700 1 -- 172.21.15.102:0/3216542601 wait complete. 2023-03-03T15:05:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.319+0000 7fe73cfff700 1 Processor -- start 2023-03-03T15:05:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.320+0000 7fe73cfff700 1 -- start start 2023-03-03T15:05:19.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe73cfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7380cd390 0x7fe7381ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:19.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe73cfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd8e0 0x7fe7380b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:19.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe73cfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe738153290 0x7fe7380b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:19.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe73cfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7381eed70 con 0x7fe7380cd390 2023-03-03T15:05:19.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe73cfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7381ef810 con 0x7fe7380cd8e0 2023-03-03T15:05:19.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe73cfff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe738159c40 con 0x7fe738153290 2023-03-03T15:05:19.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe73659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7380cd390 0x7fe7381ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:19.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd8e0 0x7fe7380b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:19.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd8e0 0x7fe7380b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:43104/0 (socket says 172.21.15.102:43104) 2023-03-03T15:05:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe736d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe738153290 0x7fe7380b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 -- 172.21.15.102:0/1775114257 learned_addr learned my addr 172.21.15.102:0/1775114257 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:05:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 -- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe738153290 msgr2=0x7fe7380b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe738153290 0x7fe7380b4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 -- 172.21.15.102:0/1775114257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7380cd390 msgr2=0x7fe7381ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7380cd390 0x7fe7381ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.321+0000 7fe735d9b700 1 -- 172.21.15.102:0/1775114257 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe730009a70 con 0x7fe7380cd8e0 2023-03-03T15:05:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.322+0000 7fe735d9b700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd8e0 0x7fe7380b1ee0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00c520 tx=0x7fe72c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.322+0000 7fe726ffd700 1 -- 172.21.15.102:0/1775114257 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c004c80 con 0x7fe7380cd8e0 2023-03-03T15:05:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.322+0000 7fe726ffd700 1 -- 172.21.15.102:0/1775114257 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c004e00 con 0x7fe7380cd8e0 2023-03-03T15:05:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.322+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7380ccde0 con 0x7fe7380cd8e0 2023-03-03T15:05:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.322+0000 7fe726ffd700 1 -- 172.21.15.102:0/1775114257 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c007310 con 0x7fe7380cd8e0 2023-03-03T15:05:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.322+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73815d160 con 0x7fe7380cd8e0 2023-03-03T15:05:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.323+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7380cd0c0 con 0x7fe7380cd8e0 2023-03-03T15:05:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.324+0000 7fe726ffd700 1 -- 172.21.15.102:0/1775114257 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe72c01a020 con 0x7fe7380cd8e0 2023-03-03T15:05:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.326+0000 7fe726ffd700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7180682f0 0x7fe71806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.326+0000 7fe726ffd700 1 -- 172.21.15.102:0/1775114257 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe72c0858c0 con 0x7fe7380cd8e0 2023-03-03T15:05:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.326+0000 7fe73659c700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7180682f0 0x7fe71806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:19.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.326+0000 7fe73659c700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7180682f0 0x7fe71806a7a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe728007fd0 tx=0x7fe728009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:19.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.330+0000 7fe726ffd700 1 -- 172.21.15.102:0/1775114257 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe72c051f40 con 0x7fe7380cd8e0 2023-03-03T15:05:19.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.563+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe73804e6e0 con 0x7fe7380cd8e0 2023-03-03T15:05:19.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.564+0000 7fe726ffd700 1 -- 172.21.15.102:0/1775114257 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe72c055560 con 0x7fe7380cd8e0 2023-03-03T15:05:19.566 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:05:19.567 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:05:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7180682f0 msgr2=0x7fe71806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7180682f0 0x7fe71806a7a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe728007fd0 tx=0x7fe728009040 comp rx=0 tx=0).stop 2023-03-03T15:05:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd8e0 msgr2=0x7fe7380b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd8e0 0x7fe7380b1ee0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00c520 tx=0x7fe72c00c830 comp rx=0 tx=0).stop 2023-03-03T15:05:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 shutdown_connections 2023-03-03T15:05:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7180682f0 0x7fe71806a7a0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe738153290 0x7fe7380b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7380cd390 0x7fe7381ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 --2- 172.21.15.102:0/1775114257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7380cd8e0 0x7fe7380b1ee0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 >> 172.21.15.102:0/1775114257 conn(0x7fe738065fb0 msgr2=0x7fe7380cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:05:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.566+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 shutdown_connections 2023-03-03T15:05:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:05:19.567+0000 7fe73cfff700 1 -- 172.21.15.102:0/1775114257 wait complete. 2023-03-03T15:05:19.571 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:05:19.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-03-03T15:05:22.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:22.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:53.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:53.014 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:23.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:23.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:53.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:53.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:54.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:06:54.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.845+0000 7fed901ff700 1 Processor -- start 2023-03-03T15:06:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.845+0000 7fed901ff700 1 -- start start 2023-03-03T15:06:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.846+0000 7fed901ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fed88153290 0x7fed88158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.846+0000 7fed901ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed880ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.846+0000 7fed901ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed88153660 con 0x7fed881593c0 2023-03-03T15:06:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed901ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8806c360 con 0x7fed88153290 2023-03-03T15:06:54.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed901ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8815cea0 con 0x7fed880cd8e0 2023-03-03T15:06:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed8e79c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed880ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed8d79a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fed881593c0 0x7fed880cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33854/0 (socket says 172.21.15.102:33854) 2023-03-03T15:06:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed8d79a700 1 -- 172.21.15.102:0/635027615 learned_addr learned my addr 172.21.15.102:0/635027615 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:06:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed8df9b700 1 --2- 172.21.15.102:0/635027615 >> v2:172.21.15.172:3300/0 conn(0x7fed88153290 0x7fed88158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed8cf99700 1 -- 172.21.15.102:0/635027615 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720163699 0 0) 0x7fed88153660 con 0x7fed881593c0 2023-03-03T15:06:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.847+0000 7fed8cf99700 1 -- 172.21.15.102:0/635027615 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed740035b0 con 0x7fed881593c0 2023-03-03T15:06:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8e79c700 1 -- 172.21.15.102:0/635027615 >> v2:172.21.15.172:3300/0 conn(0x7fed88153290 msgr2=0x7fed88158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8e79c700 1 --2- 172.21.15.102:0/635027615 >> v2:172.21.15.172:3300/0 conn(0x7fed88153290 0x7fed88158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:54.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8e79c700 1 -- 172.21.15.102:0/635027615 >> v1:172.21.15.102:6789/0 conn(0x7fed881593c0 legacy=0x7fed880cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:54.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8e79c700 1 -- 172.21.15.102:0/635027615 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8806c7f0 con 0x7fed880cd8e0 2023-03-03T15:06:54.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8e79c700 1 --2- 172.21.15.102:0/635027615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed880ce530 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fed80009d90 tx=0x7fed8000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba738ec387781b74 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:54.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8cf99700 1 -- 172.21.15.102:0/635027615 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8000e070 con 0x7fed880cd8e0 2023-03-03T15:06:54.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8cf99700 1 -- 172.21.15.102:0/635027615 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8000bb20 con 0x7fed880cd8e0 2023-03-03T15:06:54.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.848+0000 7fed8cf99700 1 -- 172.21.15.102:0/635027615 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80012430 con 0x7fed880cd8e0 2023-03-03T15:06:54.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 -- 172.21.15.102:0/635027615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 msgr2=0x7fed880ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:54.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 --2- 172.21.15.102:0/635027615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed880ce530 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fed80009d90 tx=0x7fed8000b330 comp rx=0 tx=0).stop 2023-03-03T15:06:54.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 -- 172.21.15.102:0/635027615 shutdown_connections 2023-03-03T15:06:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 --2- 172.21.15.102:0/635027615 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed880ce530 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 --2- 172.21.15.102:0/635027615 >> v2:172.21.15.172:3300/0 conn(0x7fed88153290 0x7fed88158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 -- 172.21.15.102:0/635027615 >> 172.21.15.102:0/635027615 conn(0x7fed88065fb0 msgr2=0x7fed880663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 -- 172.21.15.102:0/635027615 shutdown_connections 2023-03-03T15:06:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.849+0000 7fed901ff700 1 -- 172.21.15.102:0/635027615 wait complete. 2023-03-03T15:06:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.851+0000 7fed901ff700 1 Processor -- start 2023-03-03T15:06:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.851+0000 7fed901ff700 1 -- start start 2023-03-03T15:06:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed901ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed88155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed901ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed88153290 0x7fed880c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed901ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed881593c0 0x7fed881efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed901ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88155fe0 con 0x7fed88153290 2023-03-03T15:06:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed901ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88151580 con 0x7fed881593c0 2023-03-03T15:06:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed901ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed88154ea0 con 0x7fed880cd8e0 2023-03-03T15:06:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed8d79a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed88153290 0x7fed880c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed8d79a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed88153290 0x7fed880c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60764/0 (socket says 172.21.15.102:60764) 2023-03-03T15:06:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.852+0000 7fed8d79a700 1 -- 172.21.15.102:0/3939280910 learned_addr learned my addr 172.21.15.102:0/3939280910 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:06:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8df9b700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed88155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8e79c700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed881593c0 0x7fed881efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:54.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8d79a700 1 -- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 msgr2=0x7fed88155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8d79a700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed88155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8d79a700 1 -- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed881593c0 msgr2=0x7fed881efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8d79a700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed881593c0 0x7fed881efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8d79a700 1 -- 172.21.15.102:0/3939280910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80009a70 con 0x7fed88153290 2023-03-03T15:06:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed8d79a700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed88153290 0x7fed880c89f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fed7800c700 tx=0x7fed7800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed7e7fc700 1 -- 172.21.15.102:0/3939280910 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed78005d70 con 0x7fed88153290 2023-03-03T15:06:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed7e7fc700 1 -- 172.21.15.102:0/3939280910 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed780070d0 con 0x7fed88153290 2023-03-03T15:06:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.853+0000 7fed7e7fc700 1 -- 172.21.15.102:0/3939280910 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed78007460 con 0x7fed88153290 2023-03-03T15:06:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.854+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed880ccde0 con 0x7fed88153290 2023-03-03T15:06:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.854+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8815d160 con 0x7fed88153290 2023-03-03T15:06:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.855+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed880cd0c0 con 0x7fed88153290 2023-03-03T15:06:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.856+0000 7fed7e7fc700 1 -- 172.21.15.102:0/3939280910 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed78019030 con 0x7fed88153290 2023-03-03T15:06:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.857+0000 7fed7e7fc700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed6c0682f0 0x7fed6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.857+0000 7fed7e7fc700 1 -- 172.21.15.102:0/3939280910 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fed78085090 con 0x7fed88153290 2023-03-03T15:06:54.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.858+0000 7fed8df9b700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed6c0682f0 0x7fed6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:54.861 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.858+0000 7fed8df9b700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed6c0682f0 0x7fed6c06a7a0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fed84009b40 tx=0x7fed84009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:54.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:54.861+0000 7fed7e7fc700 1 -- 172.21.15.102:0/3939280910 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed78051740 con 0x7fed88153290 2023-03-03T15:06:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.096+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed8804e6e0 con 0x7fed88153290 2023-03-03T15:06:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.097+0000 7fed7e7fc700 1 -- 172.21.15.102:0/3939280910 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fed78017020 con 0x7fed88153290 2023-03-03T15:06:55.100 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:06:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:06:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.099+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed6c0682f0 msgr2=0x7fed6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.099+0000 7fed901ff700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed6c0682f0 0x7fed6c06a7a0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fed84009b40 tx=0x7fed84009210 comp rx=0 tx=0).stop 2023-03-03T15:06:55.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.099+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed88153290 msgr2=0x7fed880c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.099+0000 7fed901ff700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed88153290 0x7fed880c89f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fed7800c700 tx=0x7fed7800ca10 comp rx=0 tx=0).stop 2023-03-03T15:06:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 shutdown_connections 2023-03-03T15:06:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed6c0682f0 0x7fed6c06a7a0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed880cd8e0 0x7fed88155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed881593c0 0x7fed881efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 --2- 172.21.15.102:0/3939280910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed88153290 0x7fed880c89f0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 >> 172.21.15.102:0/3939280910 conn(0x7fed88065fb0 msgr2=0x7fed8815ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 shutdown_connections 2023-03-03T15:06:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:06:55.100+0000 7fed901ff700 1 -- 172.21.15.102:0/3939280910 wait complete. 2023-03-03T15:06:55.104 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:06:55.129 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2023-03-03T15:07:23.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:23.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:53.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:53.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:59.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:07:59.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.377+0000 7f01107ff700 1 Processor -- start 2023-03-03T15:07:59.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.377+0000 7f01107ff700 1 -- start start 2023-03-03T15:07:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.378+0000 7f01107ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0108153290 0x7f0108158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.378+0000 7f01107ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.378+0000 7f01107ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0108153660 con 0x7f01081593c0 2023-03-03T15:07:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.378+0000 7f01107ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010806c360 con 0x7f0108153290 2023-03-03T15:07:59.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.378+0000 7f01107ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010815cea0 con 0x7f01080cd8e0 2023-03-03T15:07:59.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010e59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0108153290 0x7f0108158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:59.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010dd9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f01081593c0 0x7f01080cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42566/0 (socket says 172.21.15.102:42566) 2023-03-03T15:07:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010e59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0108153290 0x7f0108158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38740/0 (socket says 172.21.15.102:38740) 2023-03-03T15:07:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010dd9a700 1 -- 172.21.15.102:0/2148002260 learned_addr learned my addr 172.21.15.102:0/2148002260 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:07:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010ed9c700 1 --2- 172.21.15.102:0/2148002260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010e59b700 1 -- 172.21.15.102:0/2148002260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 msgr2=0x7f01080ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:07:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010e59b700 1 --2- 172.21.15.102:0/2148002260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010d599700 1 -- 172.21.15.102:0/2148002260 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885271969 0 0) 0x7f0108153660 con 0x7f01081593c0 2023-03-03T15:07:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010e59b700 1 -- 172.21.15.102:0/2148002260 >> v1:172.21.15.172:6789/0 conn(0x7f01081593c0 legacy=0x7f01080cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.379+0000 7f010e59b700 1 -- 172.21.15.102:0/2148002260 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010806c7f0 con 0x7f0108153290 2023-03-03T15:07:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f010e59b700 1 --2- 172.21.15.102:0/2148002260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f0108158e80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0104009840 tx=0x7f0104009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6594e9c693e91e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f010d599700 1 -- 172.21.15.102:0/2148002260 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104011830 con 0x7f0108153290 2023-03-03T15:07:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f010d599700 1 -- 172.21.15.102:0/2148002260 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0104011990 con 0x7f0108153290 2023-03-03T15:07:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f010d599700 1 -- 172.21.15.102:0/2148002260 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104011d20 con 0x7f0108153290 2023-03-03T15:07:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f01107ff700 1 -- 172.21.15.102:0/2148002260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 msgr2=0x7f0108158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:59.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f01107ff700 1 --2- 172.21.15.102:0/2148002260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f0108158e80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0104009840 tx=0x7f0104009b50 comp rx=0 tx=0).stop 2023-03-03T15:07:59.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f01107ff700 1 -- 172.21.15.102:0/2148002260 shutdown_connections 2023-03-03T15:07:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f01107ff700 1 --2- 172.21.15.102:0/2148002260 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f01107ff700 1 --2- 172.21.15.102:0/2148002260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f0108158e80 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.380+0000 7f01107ff700 1 -- 172.21.15.102:0/2148002260 >> 172.21.15.102:0/2148002260 conn(0x7f0108065fb0 msgr2=0x7f01080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:07:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.381+0000 7f01107ff700 1 -- 172.21.15.102:0/2148002260 shutdown_connections 2023-03-03T15:07:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.381+0000 7f01107ff700 1 -- 172.21.15.102:0/2148002260 wait complete. 2023-03-03T15:07:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.383+0000 7f01107ff700 1 Processor -- start 2023-03-03T15:07:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.383+0000 7f01107ff700 1 -- start start 2023-03-03T15:07:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f01107ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f01107ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f010814e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f01107ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01081593c0 0x7f010814e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f01107ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01080b1f90 con 0x7f0108153290 2023-03-03T15:07:59.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f01107ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01080b2860 con 0x7f01081593c0 2023-03-03T15:07:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f01107ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01080b2180 con 0x7f01080cd8e0 2023-03-03T15:07:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f010814e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f010814e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38752/0 (socket says 172.21.15.102:38752) 2023-03-03T15:07:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 -- 172.21.15.102:0/1022510527 learned_addr learned my addr 172.21.15.102:0/1022510527 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:07:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010ed9c700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01081593c0 0x7f010814e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010e59b700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:59.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 -- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 msgr2=0x7f01080c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 -- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01081593c0 msgr2=0x7f010814e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01081593c0 0x7f010814e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.384+0000 7f010dd9a700 1 -- 172.21.15.102:0/1022510527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01040094a0 con 0x7f0108153290 2023-03-03T15:07:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.385+0000 7f010dd9a700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f010814e270 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f00f800c750 tx=0x7f00f800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.385+0000 7f00feffd700 1 -- 172.21.15.102:0/1022510527 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8004c80 con 0x7f0108153290 2023-03-03T15:07:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.385+0000 7f00feffd700 1 -- 172.21.15.102:0/1022510527 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f8004e00 con 0x7f0108153290 2023-03-03T15:07:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.385+0000 7f00feffd700 1 -- 172.21.15.102:0/1022510527 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8007310 con 0x7f0108153290 2023-03-03T15:07:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.385+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01080ccde0 con 0x7f0108153290 2023-03-03T15:07:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.385+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010815d160 con 0x7f0108153290 2023-03-03T15:07:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.386+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01080cd0c0 con 0x7f0108153290 2023-03-03T15:07:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.387+0000 7f00feffd700 1 -- 172.21.15.102:0/1022510527 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f00f801a020 con 0x7f0108153290 2023-03-03T15:07:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.389+0000 7f00feffd700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00ec068320 0x7f00ec06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.389+0000 7f00feffd700 1 -- 172.21.15.102:0/1022510527 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f00f8085870 con 0x7f0108153290 2023-03-03T15:07:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.389+0000 7f010e59b700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00ec068320 0x7f00ec06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.390+0000 7f010e59b700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00ec068320 0x7f00ec06a7d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f01040090b0 tx=0x7f010401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:59.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.393+0000 7f00feffd700 1 -- 172.21.15.102:0/1022510527 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f00f8051f20 con 0x7f0108153290 2023-03-03T15:07:59.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.627+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f010804e6e0 con 0x7f0108153290 2023-03-03T15:07:59.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.627+0000 7f00feffd700 1 -- 172.21.15.102:0/1022510527 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f00f8055540 con 0x7f0108153290 2023-03-03T15:07:59.629 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:07:59.629 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:07:59.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00ec068320 msgr2=0x7f00ec06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:59.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00ec068320 0x7f00ec06a7d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f01040090b0 tx=0x7f010401c040 comp rx=0 tx=0).stop 2023-03-03T15:07:59.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 msgr2=0x7f010814e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:59.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f010814e270 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f00f800c750 tx=0x7f00f800ca60 comp rx=0 tx=0).stop 2023-03-03T15:07:59.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 shutdown_connections 2023-03-03T15:07:59.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f00ec068320 0x7f00ec06a7d0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f01080cd8e0 0x7f01080c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01081593c0 0x7f010814e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 --2- 172.21.15.102:0/1022510527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0108153290 0x7f010814e270 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:59.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 >> 172.21.15.102:0/1022510527 conn(0x7f0108065fb0 msgr2=0x7f0108150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:07:59.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 shutdown_connections 2023-03-03T15:07:59.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:07:59.630+0000 7f01107ff700 1 -- 172.21.15.102:0/1022510527 wait complete. 2023-03-03T15:07:59.633 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:07:59.659 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2023-03-03T15:08:24.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:24.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:54.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:54.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:56.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:08:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.906+0000 7ff66e79d700 1 Processor -- start 2023-03-03T15:08:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.906+0000 7ff66e79d700 1 -- start start 2023-03-03T15:08:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66e79d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff6680ced00 0x7ff6680c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff6680c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66e79d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6680c3d20 con 0x7ff6680cf0d0 2023-03-03T15:08:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66e79d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6680c4930 con 0x7ff6680ced00 2023-03-03T15:08:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66815cba0 con 0x7ff6680b3970 2023-03-03T15:08:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66d79b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff6680ced00 0x7ff6680c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66df9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff6680cf0d0 0x7ff668158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45780/0 (socket says 172.21.15.102:45780) 2023-03-03T15:08:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66cf9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff6680c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.908+0000 7ff66df9c700 1 -- 172.21.15.102:0/2955916117 learned_addr learned my addr 172.21.15.102:0/2955916117 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:08:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff65ffff700 1 -- 172.21.15.102:0/2955916117 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435462466 0 0) 0x7ff6680c3d20 con 0x7ff6680cf0d0 2023-03-03T15:08:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff65ffff700 1 -- 172.21.15.102:0/2955916117 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6540035b0 con 0x7ff6680cf0d0 2023-03-03T15:08:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff66d79b700 1 -- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 msgr2=0x7ff6680c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:08:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff66d79b700 1 --2- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff6680c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff66d79b700 1 -- 172.21.15.102:0/2955916117 >> v1:172.21.15.172:6789/0 conn(0x7ff6680cf0d0 legacy=0x7ff668158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff66d79b700 1 -- 172.21.15.102:0/2955916117 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66815ce80 con 0x7ff6680ced00 2023-03-03T15:08:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff66cf9a700 1 --2- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff6680c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:08:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff66d79b700 1 --2- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680ced00 0x7ff6680c5a90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff664009840 tx=0x7ff664009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c31a15c6bed5596 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:08:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.909+0000 7ff65ffff700 1 -- 172.21.15.102:0/2955916117 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664011830 con 0x7ff6680ced00 2023-03-03T15:08:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff65ffff700 1 -- 172.21.15.102:0/2955916117 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff664011990 con 0x7ff6680ced00 2023-03-03T15:08:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff65ffff700 1 -- 172.21.15.102:0/2955916117 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664011d20 con 0x7ff6680ced00 2023-03-03T15:08:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 -- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680ced00 msgr2=0x7ff6680c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 --2- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680ced00 0x7ff6680c5a90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff664009840 tx=0x7ff664009b50 comp rx=0 tx=0).stop 2023-03-03T15:08:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 -- 172.21.15.102:0/2955916117 shutdown_connections 2023-03-03T15:08:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 --2- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680ced00 0x7ff6680c5a90 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 --2- 172.21.15.102:0/2955916117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff6680c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 -- 172.21.15.102:0/2955916117 >> 172.21.15.102:0/2955916117 conn(0x7ff66808b1c0 msgr2=0x7ff668071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:08:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 -- 172.21.15.102:0/2955916117 shutdown_connections 2023-03-03T15:08:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.910+0000 7ff66e79d700 1 -- 172.21.15.102:0/2955916117 wait complete. 2023-03-03T15:08:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.912+0000 7ff66e79d700 1 Processor -- start 2023-03-03T15:08:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.912+0000 7ff66e79d700 1 -- start start 2023-03-03T15:08:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.913+0000 7ff66e79d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff66815a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.913+0000 7ff66e79d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6680ced00 0x7ff6681f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.913+0000 7ff66e79d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680cf0d0 0x7ff6681f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.913+0000 7ff66e79d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681515d0 con 0x7ff6680ced00 2023-03-03T15:08:56.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.913+0000 7ff66e79d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66815b410 con 0x7ff6680cf0d0 2023-03-03T15:08:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.913+0000 7ff66e79d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66814f3f0 con 0x7ff6680b3970 2023-03-03T15:08:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff66815a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff66815a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:53522/0 (socket says 172.21.15.102:53522) 2023-03-03T15:08:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 -- 172.21.15.102:0/1311386445 learned_addr learned my addr 172.21.15.102:0/1311386445 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:08:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66df9c700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680cf0d0 0x7ff6681f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66cf9a700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6680ced00 0x7ff6681f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 -- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680cf0d0 msgr2=0x7ff6681f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680cf0d0 0x7ff6681f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 -- 172.21.15.102:0/1311386445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6680ced00 msgr2=0x7ff6681f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6680ced00 0x7ff6681f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 -- 172.21.15.102:0/1311386445 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6580094a0 con 0x7ff6680b3970 2023-03-03T15:08:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66cf9a700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6680ced00 0x7ff6681f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:08:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.914+0000 7ff66d79b700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff66815a540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff6640090b0 tx=0x7ff664004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:08:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.915+0000 7ff65dffb700 1 -- 172.21.15.102:0/1311386445 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6640036a0 con 0x7ff6680b3970 2023-03-03T15:08:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.915+0000 7ff65dffb700 1 -- 172.21.15.102:0/1311386445 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66401f070 con 0x7ff6680b3970 2023-03-03T15:08:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.915+0000 7ff65dffb700 1 -- 172.21.15.102:0/1311386445 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664003a20 con 0x7ff6680b3970 2023-03-03T15:08:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.915+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6640094a0 con 0x7ff6680b3970 2023-03-03T15:08:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.915+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66806c200 con 0x7ff6680b3970 2023-03-03T15:08:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.916+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66806bf20 con 0x7ff6680b3970 2023-03-03T15:08:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.917+0000 7ff65dffb700 1 -- 172.21.15.102:0/1311386445 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff66400f050 con 0x7ff6680b3970 2023-03-03T15:08:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.919+0000 7ff65dffb700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.919+0000 7ff65dffb700 1 -- 172.21.15.102:0/1311386445 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff664021070 con 0x7ff6680b3970 2023-03-03T15:08:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.919+0000 7ff66cf9a700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.920+0000 7ff66cf9a700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff668067fb0 tx=0x7ff658008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:08:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:56.922+0000 7ff65dffb700 1 -- 172.21.15.102:0/1311386445 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff664053ff0 con 0x7ff6680b3970 2023-03-03T15:08:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.161+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6680cf540 con 0x7ff6680b3970 2023-03-03T15:08:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.162+0000 7ff65dffb700 1 -- 172.21.15.102:0/1311386445 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff664057610 con 0x7ff6680b3970 2023-03-03T15:08:57.164 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:08:57.164 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:08:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 msgr2=0x7ff64c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff668067fb0 tx=0x7ff658008040 comp rx=0 tx=0).stop 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 msgr2=0x7ff66815a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff66815a540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff6640090b0 tx=0x7ff664004750 comp rx=0 tx=0).stop 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 shutdown_connections 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff64c0682f0 0x7ff64c06a7a0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff6680b3970 0x7ff66815a540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6680cf0d0 0x7ff6681f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 --2- 172.21.15.102:0/1311386445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6680ced00 0x7ff6681f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 >> 172.21.15.102:0/1311386445 conn(0x7ff66808b1c0 msgr2=0x7ff6680ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:08:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 shutdown_connections 2023-03-03T15:08:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:08:57.165+0000 7ff66e79d700 1 -- 172.21.15.102:0/1311386445 wait complete. 2023-03-03T15:08:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:08:57.193 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-03-03T15:09:05.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:09:05.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.458+0000 7f6975fff700 1 Processor -- start 2023-03-03T15:09:05.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.458+0000 7f6975fff700 1 -- start start 2023-03-03T15:09:05.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.459+0000 7f6975fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69700cd8e0 0x7f69700cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:05.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.459+0000 7f6975fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69700cd390 0x7f69700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:05.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.459+0000 7f6975fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6970153660 con 0x7f6970153290 2023-03-03T15:09:05.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.459+0000 7f6975fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697006c360 con 0x7f69700cd390 2023-03-03T15:09:05.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.459+0000 7f6975fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697015cea0 con 0x7f69700cd8e0 2023-03-03T15:09:05.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69700cd390 0x7f69700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:05.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69700cd8e0 0x7f69700cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:05.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6970153290 0x7f6970158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45796/0 (socket says 172.21.15.102:45796) 2023-03-03T15:09:05.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696f7fe700 1 -- 172.21.15.102:0/103727477 learned_addr learned my addr 172.21.15.102:0/103727477 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:09:05.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696effd700 1 -- 172.21.15.102:0/103727477 >> v2:172.21.15.102:3300/0 conn(0x7f69700cd390 msgr2=0x7f69700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:09:05.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696effd700 1 --2- 172.21.15.102:0/103727477 >> v2:172.21.15.102:3300/0 conn(0x7f69700cd390 0x7f69700ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696effd700 1 -- 172.21.15.102:0/103727477 >> v1:172.21.15.172:6789/0 conn(0x7f6970153290 legacy=0x7f6970158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:05.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696e7fc700 1 -- 172.21.15.102:0/103727477 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079275879 0 0) 0x7f6970153660 con 0x7f6970153290 2023-03-03T15:09:05.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696effd700 1 -- 172.21.15.102:0/103727477 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697006c7f0 con 0x7f69700cd8e0 2023-03-03T15:09:05.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.460+0000 7f696ffff700 1 --2- 172.21.15.102:0/103727477 >> v2:172.21.15.102:3300/0 conn(0x7f69700cd390 0x7f69700ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:09:05.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f696effd700 1 --2- 172.21.15.102:0/103727477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69700cd8e0 0x7f69700cde20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6964009840 tx=0x7f6964009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ca05fbf775c6386 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:05.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f696e7fc700 1 -- 172.21.15.102:0/103727477 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964011830 con 0x7f69700cd8e0 2023-03-03T15:09:05.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f696e7fc700 1 -- 172.21.15.102:0/103727477 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6964011990 con 0x7f69700cd8e0 2023-03-03T15:09:05.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f696e7fc700 1 -- 172.21.15.102:0/103727477 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964011d20 con 0x7f69700cd8e0 2023-03-03T15:09:05.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f6975fff700 1 -- 172.21.15.102:0/103727477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69700cd8e0 msgr2=0x7f69700cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:05.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f6975fff700 1 --2- 172.21.15.102:0/103727477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69700cd8e0 0x7f69700cde20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6964009840 tx=0x7f6964009b50 comp rx=0 tx=0).stop 2023-03-03T15:09:05.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f6975fff700 1 -- 172.21.15.102:0/103727477 shutdown_connections 2023-03-03T15:09:05.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f6975fff700 1 --2- 172.21.15.102:0/103727477 >> v2:172.21.15.102:3300/0 conn(0x7f69700cd390 0x7f69700ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f6975fff700 1 --2- 172.21.15.102:0/103727477 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f69700cd8e0 0x7f69700cde20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.461+0000 7f6975fff700 1 -- 172.21.15.102:0/103727477 >> 172.21.15.102:0/103727477 conn(0x7f6970065fb0 msgr2=0x7f69700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:09:05.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.462+0000 7f6975fff700 1 -- 172.21.15.102:0/103727477 shutdown_connections 2023-03-03T15:09:05.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.462+0000 7f6975fff700 1 -- 172.21.15.102:0/103727477 wait complete. 2023-03-03T15:09:05.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.464+0000 7f6975fff700 1 Processor -- start 2023-03-03T15:09:05.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.464+0000 7f6975fff700 1 -- start start 2023-03-03T15:09:05.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f6975fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69700cd390 0x7f69700c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:05.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f6975fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700cd8e0 0x7f69700c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:05.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f6975fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6970153290 0x7f6970150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:05.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f6975fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700c9160 con 0x7f69700cd390 2023-03-03T15:09:05.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f6975fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701efc10 con 0x7f69700cd8e0 2023-03-03T15:09:05.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f6975fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69700c7b80 con 0x7f6970153290 2023-03-03T15:09:05.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f696f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69700cd390 0x7f69700c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:05.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f696f7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69700cd390 0x7f69700c8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48920/0 (socket says 172.21.15.102:48920) 2023-03-03T15:09:05.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f696f7fe700 1 -- 172.21.15.102:0/2459794277 learned_addr learned my addr 172.21.15.102:0/2459794277 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:09:05.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f696effd700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700cd8e0 0x7f69700c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:05.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.465+0000 7f696ffff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6970153290 0x7f6970150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:05.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f696f7fe700 1 -- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6970153290 msgr2=0x7f6970150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:05.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f696f7fe700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6970153290 0x7f6970150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f696f7fe700 1 -- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700cd8e0 msgr2=0x7f69700c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:05.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f696f7fe700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700cd8e0 0x7f69700c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f696f7fe700 1 -- 172.21.15.102:0/2459794277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968009a70 con 0x7f69700cd390 2023-03-03T15:09:05.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f696f7fe700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69700cd390 0x7f69700c8c20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f69600011e0 tx=0x7f696000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:05.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f6957fff700 1 -- 172.21.15.102:0/2459794277 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960004c80 con 0x7f69700cd390 2023-03-03T15:09:05.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f6957fff700 1 -- 172.21.15.102:0/2459794277 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960004de0 con 0x7f69700cd390 2023-03-03T15:09:05.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f6957fff700 1 -- 172.21.15.102:0/2459794277 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960007310 con 0x7f69700cd390 2023-03-03T15:09:05.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69640094a0 con 0x7f69700cd390 2023-03-03T15:09:05.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.466+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69700ccde0 con 0x7f69700cd390 2023-03-03T15:09:05.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.467+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697015d160 con 0x7f69700cd390 2023-03-03T15:09:05.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.469+0000 7f6957fff700 1 -- 172.21.15.102:0/2459794277 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f696001a020 con 0x7f69700cd390 2023-03-03T15:09:05.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.471+0000 7f6957fff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f695c0682f0 0x7f695c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:05.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.471+0000 7f6957fff700 1 -- 172.21.15.102:0/2459794277 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6960085940 con 0x7f69700cd390 2023-03-03T15:09:05.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.471+0000 7f696effd700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f695c0682f0 0x7f695c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:05.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.472+0000 7f696effd700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f695c0682f0 0x7f695c06a7a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f69640090b0 tx=0x7f6964008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:05.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.475+0000 7f6957fff700 1 -- 172.21.15.102:0/2459794277 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6960051fc0 con 0x7f69700cd390 2023-03-03T15:09:05.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.711+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69700cd0c0 con 0x7f69700cd390 2023-03-03T15:09:05.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.712+0000 7f6957fff700 1 -- 172.21.15.102:0/2459794277 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f69600555e0 con 0x7f69700cd390 2023-03-03T15:09:05.714 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:09:05.715 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:09:05.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f695c0682f0 msgr2=0x7f695c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:05.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f695c0682f0 0x7f695c06a7a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f69640090b0 tx=0x7f6964008030 comp rx=0 tx=0).stop 2023-03-03T15:09:05.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69700cd390 msgr2=0x7f69700c8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69700cd390 0x7f69700c8c20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f69600011e0 tx=0x7f696000ca30 comp rx=0 tx=0).stop 2023-03-03T15:09:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 shutdown_connections 2023-03-03T15:09:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f695c0682f0 0x7f695c06a7a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6970153290 0x7f6970150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69700cd390 0x7f69700c8c20 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 --2- 172.21.15.102:0/2459794277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f69700cd8e0 0x7f69700c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:05.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 >> 172.21.15.102:0/2459794277 conn(0x7f6970065fb0 msgr2=0x7f69700cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:09:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 shutdown_connections 2023-03-03T15:09:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:09:05.715+0000 7f6975fff700 1 -- 172.21.15.102:0/2459794277 wait complete. 2023-03-03T15:09:05.719 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:09:05.744 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2023-03-03T15:09:24.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:24.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:54.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:54.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T15:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:10:24.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:24.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:39.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:10:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.008+0000 7f3aff5ff700 1 Processor -- start 2023-03-03T15:10:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.008+0000 7f3aff5ff700 1 -- start start 2023-03-03T15:10:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.009+0000 7f3aff5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af80c9cb0 0x7f3af80cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:40.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.009+0000 7f3aff5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af80cf940 con 0x7f3af80ca190 2023-03-03T15:10:40.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.009+0000 7f3aff5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af80cfb90 con 0x7f3af8153260 2023-03-03T15:10:40.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.009+0000 7f3aff5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af80cfe70 con 0x7f3af80c9cb0 2023-03-03T15:10:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3afdb9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3af80ca190 0x7f3af8163610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48922/0 (socket says 172.21.15.102:48922) 2023-03-03T15:10:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3afd39b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3af8153260 0x7f3af80ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54504/0 (socket says 172.21.15.102:54504) 2023-03-03T15:10:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3afd39b700 1 -- 172.21.15.102:0/1146068295 learned_addr learned my addr 172.21.15.102:0/1146068295 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:10:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3afcb9a700 1 --2- 172.21.15.102:0/1146068295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af80c9cb0 0x7f3af80cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3aeffff700 1 -- 172.21.15.102:0/1146068295 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180162278 0 0) 0x7f3af80cfb90 con 0x7f3af8153260 2023-03-03T15:10:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3aeffff700 1 -- 172.21.15.102:0/1146068295 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae40035b0 con 0x7f3af8153260 2023-03-03T15:10:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3afcb9a700 1 -- 172.21.15.102:0/1146068295 >> v1:172.21.15.172:6789/0 conn(0x7f3af8153260 legacy=0x7f3af80ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3afcb9a700 1 -- 172.21.15.102:0/1146068295 >> v1:172.21.15.102:6789/0 conn(0x7f3af80ca190 legacy=0x7f3af8163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3aeffff700 1 -- 172.21.15.102:0/1146068295 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552673277 0 0) 0x7f3af80cf940 con 0x7f3af80ca190 2023-03-03T15:10:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.010+0000 7f3afcb9a700 1 -- 172.21.15.102:0/1146068295 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af8158f80 con 0x7f3af80c9cb0 2023-03-03T15:10:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3afcb9a700 1 --2- 172.21.15.102:0/1146068295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af80c9cb0 0x7f3af80cb880 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009870 tx=0x7f3ae8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2c9acba21b28478 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3aeffff700 1 -- 172.21.15.102:0/1146068295 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8011830 con 0x7f3af80c9cb0 2023-03-03T15:10:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3aeffff700 1 -- 172.21.15.102:0/1146068295 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae8011990 con 0x7f3af80c9cb0 2023-03-03T15:10:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3aeffff700 1 -- 172.21.15.102:0/1146068295 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8011d20 con 0x7f3af80c9cb0 2023-03-03T15:10:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3aff5ff700 1 -- 172.21.15.102:0/1146068295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af80c9cb0 msgr2=0x7f3af80cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/1146068295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af80c9cb0 0x7f3af80cb880 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009870 tx=0x7f3ae8009b80 comp rx=0 tx=0).stop 2023-03-03T15:10:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3aff5ff700 1 -- 172.21.15.102:0/1146068295 shutdown_connections 2023-03-03T15:10:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.011+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/1146068295 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af80c9cb0 0x7f3af80cb880 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.012+0000 7f3aff5ff700 1 -- 172.21.15.102:0/1146068295 >> 172.21.15.102:0/1146068295 conn(0x7f3af8066c00 msgr2=0x7f3af80ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:10:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.012+0000 7f3aff5ff700 1 -- 172.21.15.102:0/1146068295 shutdown_connections 2023-03-03T15:10:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.012+0000 7f3aff5ff700 1 -- 172.21.15.102:0/1146068295 wait complete. 2023-03-03T15:10:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.014+0000 7f3aff5ff700 1 Processor -- start 2023-03-03T15:10:40.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.015+0000 7f3aff5ff700 1 -- start start 2023-03-03T15:10:40.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3aff5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af80ca190 0x7f3af80b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3aff5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af8153260 0x7f3af80c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3aff5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3af80c6180 0x7f3af80c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3aff5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af80c78b0 con 0x7f3af80ca190 2023-03-03T15:10:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3aff5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af815be80 con 0x7f3af80c6180 2023-03-03T15:10:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3aff5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af815c140 con 0x7f3af8153260 2023-03-03T15:10:40.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3afd39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af80ca190 0x7f3af80b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:40.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3afd39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af80ca190 0x7f3af80b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53776/0 (socket says 172.21.15.102:53776) 2023-03-03T15:10:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.016+0000 7f3afd39b700 1 -- 172.21.15.102:0/73382578 learned_addr learned my addr 172.21.15.102:0/73382578 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:10:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afcb9a700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af8153260 0x7f3af80c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afdb9c700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3af80c6180 0x7f3af80c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afd39b700 1 -- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af8153260 msgr2=0x7f3af80c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afd39b700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af8153260 0x7f3af80c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afd39b700 1 -- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3af80c6180 msgr2=0x7f3af80c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afd39b700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3af80c6180 0x7f3af80c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afd39b700 1 -- 172.21.15.102:0/73382578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae80094a0 con 0x7f3af80ca190 2023-03-03T15:10:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.017+0000 7f3afd39b700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af80ca190 0x7f3af80b4120 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3af400c420 tx=0x7f3af400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.018+0000 7f3aedffb700 1 -- 172.21.15.102:0/73382578 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4005d70 con 0x7f3af80ca190 2023-03-03T15:10:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.018+0000 7f3aedffb700 1 -- 172.21.15.102:0/73382578 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af400ce30 con 0x7f3af80ca190 2023-03-03T15:10:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.018+0000 7f3aedffb700 1 -- 172.21.15.102:0/73382578 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af40072d0 con 0x7f3af80ca190 2023-03-03T15:10:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.018+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af8159540 con 0x7f3af80ca190 2023-03-03T15:10:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.018+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af8159260 con 0x7f3af80ca190 2023-03-03T15:10:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.019+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af8159820 con 0x7f3af80ca190 2023-03-03T15:10:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.020+0000 7f3aedffb700 1 -- 172.21.15.102:0/73382578 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3af4019030 con 0x7f3af80ca190 2023-03-03T15:10:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.021+0000 7f3aedffb700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3adc0682f0 0x7f3adc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.021+0000 7f3aedffb700 1 -- 172.21.15.102:0/73382578 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3af4084ec0 con 0x7f3af80ca190 2023-03-03T15:10:40.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.022+0000 7f3afcb9a700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3adc0682f0 0x7f3adc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:40.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.023+0000 7f3afcb9a700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3adc0682f0 0x7f3adc06a7a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80097c0 tx=0x7f3ae801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:40.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.026+0000 7f3aedffb700 1 -- 172.21.15.102:0/73382578 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3af404b5e0 con 0x7f3af80ca190 2023-03-03T15:10:40.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.261+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3af80c7b70 con 0x7f3af80ca190 2023-03-03T15:10:40.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.262+0000 7f3aedffb700 1 -- 172.21.15.102:0/73382578 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3af4017020 con 0x7f3af80ca190 2023-03-03T15:10:40.265 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:10:40.265 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:10:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.264+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3adc0682f0 msgr2=0x7f3adc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.264+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3adc0682f0 0x7f3adc06a7a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80097c0 tx=0x7f3ae801c040 comp rx=0 tx=0).stop 2023-03-03T15:10:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.264+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af80ca190 msgr2=0x7f3af80b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.264+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af80ca190 0x7f3af80b4120 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3af400c420 tx=0x7f3af400c730 comp rx=0 tx=0).stop 2023-03-03T15:10:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 shutdown_connections 2023-03-03T15:10:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3af80c6180 0x7f3af80c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af80ca190 0x7f3af80b4120 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:40.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3adc0682f0 0x7f3adc06a7a0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 --2- 172.21.15.102:0/73382578 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3af8153260 0x7f3af80c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 >> 172.21.15.102:0/73382578 conn(0x7f3af80c9cb0 msgr2=0x7f3af8157c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:10:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 shutdown_connections 2023-03-03T15:10:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:10:40.265+0000 7f3aff5ff700 1 -- 172.21.15.102:0/73382578 wait complete. 2023-03-03T15:10:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:10:40.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-03-03T15:10:54.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:54.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:12.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:11:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.546+0000 7fa50b3ff700 1 Processor -- start 2023-03-03T15:11:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.547+0000 7fa50b3ff700 1 -- start start 2023-03-03T15:11:12.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.548+0000 7fa50b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5040ca130 0x7fa50415c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:12.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.548+0000 7fa50b3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa5040c8eb0 0x7fa50415c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:12.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.548+0000 7fa50b3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5040c9590 con 0x7fa504152e50 2023-03-03T15:11:12.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.548+0000 7fa50b3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa504159880 con 0x7fa5040c8eb0 2023-03-03T15:11:12.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.548+0000 7fa50b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040c7ae0 con 0x7fa5040ca130 2023-03-03T15:11:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50919b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa504152e50 0x7fa5040c9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44782/0 (socket says 172.21.15.102:44782) 2023-03-03T15:11:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50919b700 1 -- 172.21.15.102:0/1742008885 learned_addr learned my addr 172.21.15.102:0/1742008885 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:11:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50999c700 1 --2- 172.21.15.102:0/1742008885 >> v2:172.21.15.102:3300/0 conn(0x7fa5040c8eb0 0x7fa50415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50899a700 1 --2- 172.21.15.102:0/1742008885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5040ca130 0x7fa50415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50999c700 1 -- 172.21.15.102:0/1742008885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5040ca130 msgr2=0x7fa50415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:11:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50999c700 1 --2- 172.21.15.102:0/1742008885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5040ca130 0x7fa50415c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50999c700 1 -- 172.21.15.102:0/1742008885 >> v1:172.21.15.102:6789/0 conn(0x7fa504152e50 legacy=0x7fa5040c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50999c700 1 -- 172.21.15.102:0/1742008885 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5040c7dc0 con 0x7fa5040c8eb0 2023-03-03T15:11:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa4fbfff700 1 -- 172.21.15.102:0/1742008885 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183849039 0 0) 0x7fa5040c9590 con 0x7fa504152e50 2023-03-03T15:11:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.549+0000 7fa50999c700 1 --2- 172.21.15.102:0/1742008885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa50415c790 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009d90 tx=0x7fa4fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef15e6e5517ff638 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa4fbfff700 1 -- 172.21.15.102:0/1742008885 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc00e070 con 0x7fa5040c8eb0 2023-03-03T15:11:12.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa4fbfff700 1 -- 172.21.15.102:0/1742008885 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc00bb20 con 0x7fa5040c8eb0 2023-03-03T15:11:12.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa4fbfff700 1 -- 172.21.15.102:0/1742008885 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc012430 con 0x7fa5040c8eb0 2023-03-03T15:11:12.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 -- 172.21.15.102:0/1742008885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 msgr2=0x7fa50415c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:12.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/1742008885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa50415c790 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009d90 tx=0x7fa4fc00b330 comp rx=0 tx=0).stop 2023-03-03T15:11:12.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 -- 172.21.15.102:0/1742008885 shutdown_connections 2023-03-03T15:11:12.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/1742008885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa50415c790 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/1742008885 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa5040ca130 0x7fa50415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 -- 172.21.15.102:0/1742008885 >> 172.21.15.102:0/1742008885 conn(0x7fa5040704c0 msgr2=0x7fa5040708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:11:12.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 -- 172.21.15.102:0/1742008885 shutdown_connections 2023-03-03T15:11:12.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.550+0000 7fa50b3ff700 1 -- 172.21.15.102:0/1742008885 wait complete. 2023-03-03T15:11:12.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.552+0000 7fa50b3ff700 1 Processor -- start 2023-03-03T15:11:12.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.553+0000 7fa50b3ff700 1 -- start start 2023-03-03T15:11:12.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50b3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa504157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:12.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50b3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5040ca130 0x7fa504157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa504152e50 0x7fa5041f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50b3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa504157000 con 0x7fa5040c8eb0 2023-03-03T15:11:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50b3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50414faa0 con 0x7fa5040ca130 2023-03-03T15:11:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50414fd80 con 0x7fa504152e50 2023-03-03T15:11:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa504157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa504157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56112/0 (socket says 172.21.15.102:56112) 2023-03-03T15:11:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 -- 172.21.15.102:0/2707939006 learned_addr learned my addr 172.21.15.102:0/2707939006 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50999c700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa504152e50 0x7fa5041f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50899a700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5040ca130 0x7fa504157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 -- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa504152e50 msgr2=0x7fa5041f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa504152e50 0x7fa5041f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 -- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5040ca130 msgr2=0x7fa504157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5040ca130 0x7fa504157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.554+0000 7fa50919b700 1 -- 172.21.15.102:0/2707939006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc009a70 con 0x7fa5040c8eb0 2023-03-03T15:11:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.555+0000 7fa50919b700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa504157420 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fa50000c550 tx=0x7fa50000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:12.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.555+0000 7fa4f9ffb700 1 -- 172.21.15.102:0/2707939006 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500004c80 con 0x7fa5040c8eb0 2023-03-03T15:11:12.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.555+0000 7fa4f9ffb700 1 -- 172.21.15.102:0/2707939006 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa500004e00 con 0x7fa5040c8eb0 2023-03-03T15:11:12.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.555+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa504159b60 con 0x7fa5040c8eb0 2023-03-03T15:11:12.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.555+0000 7fa4f9ffb700 1 -- 172.21.15.102:0/2707939006 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500007310 con 0x7fa5040c8eb0 2023-03-03T15:11:12.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.555+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5040c80a0 con 0x7fa5040c8eb0 2023-03-03T15:11:12.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.556+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5040c7690 con 0x7fa5040c8eb0 2023-03-03T15:11:12.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.557+0000 7fa4f9ffb700 1 -- 172.21.15.102:0/2707939006 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa50001a020 con 0x7fa5040c8eb0 2023-03-03T15:11:12.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.559+0000 7fa4f9ffb700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4e80682f0 0x7fa4e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:12.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.559+0000 7fa4f9ffb700 1 -- 172.21.15.102:0/2707939006 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa500085870 con 0x7fa5040c8eb0 2023-03-03T15:11:12.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.559+0000 7fa50899a700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4e80682f0 0x7fa4e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:12.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.560+0000 7fa50899a700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4e80682f0 0x7fa4e806a7a0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fa5040b2c70 tx=0x7fa4f4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:12.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.563+0000 7fa4f9ffb700 1 -- 172.21.15.102:0/2707939006 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa500051f80 con 0x7fa5040c8eb0 2023-03-03T15:11:12.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.799+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa50404e6e0 con 0x7fa5040c8eb0 2023-03-03T15:11:12.801 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.800+0000 7fa4f9ffb700 1 -- 172.21.15.102:0/2707939006 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa5000555a0 con 0x7fa5040c8eb0 2023-03-03T15:11:12.802 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:11:12.802 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:11:12.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4e80682f0 msgr2=0x7fa4e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:12.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4e80682f0 0x7fa4e806a7a0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fa5040b2c70 tx=0x7fa4f4009210 comp rx=0 tx=0).stop 2023-03-03T15:11:12.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 msgr2=0x7fa504157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:12.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa504157420 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fa50000c550 tx=0x7fa50000c860 comp rx=0 tx=0).stop 2023-03-03T15:11:12.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 shutdown_connections 2023-03-03T15:11:12.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa4e80682f0 0x7fa4e806a7a0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa504152e50 0x7fa5041f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5040c8eb0 0x7fa504157420 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 --2- 172.21.15.102:0/2707939006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5040ca130 0x7fa504157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:12.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 >> 172.21.15.102:0/2707939006 conn(0x7fa5040704c0 msgr2=0x7fa50406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:11:12.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 shutdown_connections 2023-03-03T15:11:12.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:11:12.803+0000 7fa50b3ff700 1 -- 172.21.15.102:0/2707939006 wait complete. 2023-03-03T15:11:12.807 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:11:12.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2023-03-03T15:11:25.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:25.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:55.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:55.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:25.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:25.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:46.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:12:47.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.081+0000 7fae757ff700 1 Processor -- start 2023-03-03T15:12:47.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.082+0000 7fae757ff700 1 -- start start 2023-03-03T15:12:47.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.083+0000 7fae757ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae70153290 0x7fae70158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.083+0000 7fae757ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae701593c0 0x7fae700cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.083+0000 7fae757ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fae700cd8e0 0x7fae700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.083+0000 7fae757ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae70153660 con 0x7fae70153290 2023-03-03T15:12:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.083+0000 7fae757ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7006c360 con 0x7fae700cd8e0 2023-03-03T15:12:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.083+0000 7fae757ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7015cea0 con 0x7fae701593c0 2023-03-03T15:12:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae70153290 0x7fae70158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae70153290 0x7fae70158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42258/0 (socket says 172.21.15.102:42258) 2023-03-03T15:12:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6e7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae701593c0 0x7fae700cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 -- 172.21.15.102:0/663576830 learned_addr learned my addr 172.21.15.102:0/663576830 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:12:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6f7fe700 1 --2- 172.21.15.102:0/663576830 >> v2:172.21.15.172:3300/0 conn(0x7fae700cd8e0 0x7fae700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 -- 172.21.15.102:0/663576830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae701593c0 msgr2=0x7fae700cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:12:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 --2- 172.21.15.102:0/663576830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae701593c0 0x7fae700cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 -- 172.21.15.102:0/663576830 >> v2:172.21.15.172:3300/0 conn(0x7fae700cd8e0 msgr2=0x7fae700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:12:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 --2- 172.21.15.102:0/663576830 >> v2:172.21.15.172:3300/0 conn(0x7fae700cd8e0 0x7fae700ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 -- 172.21.15.102:0/663576830 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7006c7f0 con 0x7fae70153290 2023-03-03T15:12:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.084+0000 7fae6effd700 1 --2- 172.21.15.102:0/663576830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70158e80 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fae7006d0e0 tx=0x7fae60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71b7c409e8a48de server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:12:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae6dffb700 1 -- 172.21.15.102:0/663576830 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60011830 con 0x7fae70153290 2023-03-03T15:12:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae6dffb700 1 -- 172.21.15.102:0/663576830 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae60011990 con 0x7fae70153290 2023-03-03T15:12:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae6dffb700 1 -- 172.21.15.102:0/663576830 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60011d20 con 0x7fae70153290 2023-03-03T15:12:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae757ff700 1 -- 172.21.15.102:0/663576830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 msgr2=0x7fae70158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae757ff700 1 --2- 172.21.15.102:0/663576830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70158e80 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fae7006d0e0 tx=0x7fae60009bd0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae757ff700 1 -- 172.21.15.102:0/663576830 shutdown_connections 2023-03-03T15:12:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae757ff700 1 --2- 172.21.15.102:0/663576830 >> v2:172.21.15.172:3300/0 conn(0x7fae700cd8e0 0x7fae700ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae757ff700 1 --2- 172.21.15.102:0/663576830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70158e80 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae757ff700 1 --2- 172.21.15.102:0/663576830 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae701593c0 0x7fae700cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.085+0000 7fae757ff700 1 -- 172.21.15.102:0/663576830 >> 172.21.15.102:0/663576830 conn(0x7fae70065fb0 msgr2=0x7fae700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:12:47.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.086+0000 7fae757ff700 1 -- 172.21.15.102:0/663576830 shutdown_connections 2023-03-03T15:12:47.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.086+0000 7fae757ff700 1 -- 172.21.15.102:0/663576830 wait complete. 2023-03-03T15:12:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.088+0000 7fae757ff700 1 Processor -- start 2023-03-03T15:12:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.088+0000 7fae757ff700 1 -- start start 2023-03-03T15:12:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae757ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae700cd8e0 0x7fae7014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae757ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae757ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae701593c0 0x7fae70155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae757ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae70155100 con 0x7fae70153290 2023-03-03T15:12:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae757ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae701f2480 con 0x7fae701593c0 2023-03-03T15:12:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae757ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7015b9d0 con 0x7fae700cd8e0 2023-03-03T15:12:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70155590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42262/0 (socket says 172.21.15.102:42262) 2023-03-03T15:12:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 -- 172.21.15.102:0/2581602379 learned_addr learned my addr 172.21.15.102:0/2581602379 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:12:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6effd700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae700cd8e0 0x7fae7014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6f7fe700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae701593c0 0x7fae70155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 -- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae700cd8e0 msgr2=0x7fae7014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae700cd8e0 0x7fae7014dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 -- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae701593c0 msgr2=0x7fae70155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae701593c0 0x7fae70155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.089+0000 7fae6e7fc700 1 -- 172.21.15.102:0/2581602379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae60009550 con 0x7fae70153290 2023-03-03T15:12:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.090+0000 7fae6e7fc700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70155590 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fae6400e9b0 tx=0x7fae6400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:12:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.090+0000 7fae577fe700 1 -- 172.21.15.102:0/2581602379 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64009710 con 0x7fae70153290 2023-03-03T15:12:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.090+0000 7fae577fe700 1 -- 172.21.15.102:0/2581602379 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6400ee50 con 0x7fae70153290 2023-03-03T15:12:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.090+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae700ccde0 con 0x7fae70153290 2023-03-03T15:12:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.090+0000 7fae577fe700 1 -- 172.21.15.102:0/2581602379 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6400ca50 con 0x7fae70153290 2023-03-03T15:12:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.090+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7015d160 con 0x7fae70153290 2023-03-03T15:12:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.091+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae700cd0c0 con 0x7fae70153290 2023-03-03T15:12:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.093+0000 7fae577fe700 1 -- 172.21.15.102:0/2581602379 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae64019030 con 0x7fae70153290 2023-03-03T15:12:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.094+0000 7fae577fe700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae5c0682f0 0x7fae5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.094+0000 7fae577fe700 1 -- 172.21.15.102:0/2581602379 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fae64085c90 con 0x7fae70153290 2023-03-03T15:12:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.094+0000 7fae6effd700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae5c0682f0 0x7fae5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.095+0000 7fae6effd700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae5c0682f0 0x7fae5c06a7a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fae60014040 tx=0x7fae60008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:12:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.098+0000 7fae577fe700 1 -- 172.21.15.102:0/2581602379 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae64052310 con 0x7fae70153290 2023-03-03T15:12:47.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.330+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae7004e6e0 con 0x7fae70153290 2023-03-03T15:12:47.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.331+0000 7fae577fe700 1 -- 172.21.15.102:0/2581602379 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fae64021090 con 0x7fae70153290 2023-03-03T15:12:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:12:47.333 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:12:47.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae5c0682f0 msgr2=0x7fae5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:47.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae5c0682f0 0x7fae5c06a7a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fae60014040 tx=0x7fae60008030 comp rx=0 tx=0).stop 2023-03-03T15:12:47.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 msgr2=0x7fae70155590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:47.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70155590 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fae6400e9b0 tx=0x7fae6400ecc0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 shutdown_connections 2023-03-03T15:12:47.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae5c0682f0 0x7fae5c06a7a0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae700cd8e0 0x7fae7014dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae701593c0 0x7fae70155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 --2- 172.21.15.102:0/2581602379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae70153290 0x7fae70155590 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:47.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 >> 172.21.15.102:0/2581602379 conn(0x7fae70065fb0 msgr2=0x7fae700cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:12:47.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 shutdown_connections 2023-03-03T15:12:47.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:12:47.334+0000 7fae757ff700 1 -- 172.21.15.102:0/2581602379 wait complete. 2023-03-03T15:12:47.339 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:12:47.362 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-03-03T15:12:55.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:55.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:25.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:25.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:56.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:56.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:26.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:26.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:31.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:14:31.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.613+0000 7f6dcd1ff700 1 Processor -- start 2023-03-03T15:14:31.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.613+0000 7f6dcd1ff700 1 -- start start 2023-03-03T15:14:31.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.614+0000 7f6dcd1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6dc80cd390 0x7f6dc80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:31.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.614+0000 7f6dcd1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc8153660 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.614+0000 7f6dcd1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc806c360 con 0x7f6dc8153290 2023-03-03T15:14:31.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.614+0000 7f6dcd1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc815cea0 con 0x7f6dc80cd390 2023-03-03T15:14:31.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc759e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6dc80cd390 0x7f6dc80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:31.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc6d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6dc8153290 0x7f6dc8158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38390/0 (socket says 172.21.15.102:38390) 2023-03-03T15:14:31.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc6d9d700 1 -- 172.21.15.102:0/1968512489 learned_addr learned my addr 172.21.15.102:0/1968512489 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:14:31.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc659c700 1 -- 172.21.15.102:0/1968512489 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210544977 0 0) 0x7f6dc806c360 con 0x7f6dc8153290 2023-03-03T15:14:31.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc659c700 1 -- 172.21.15.102:0/1968512489 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db40035b0 con 0x7f6dc8153290 2023-03-03T15:14:31.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc759e700 1 -- 172.21.15.102:0/1968512489 >> v1:172.21.15.172:6789/0 conn(0x7f6dc8153290 legacy=0x7f6dc8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:31.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc759e700 1 -- 172.21.15.102:0/1968512489 >> v1:172.21.15.102:6789/0 conn(0x7f6dc80cd8e0 legacy=0x7f6dc80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:31.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc659c700 1 -- 172.21.15.102:0/1968512489 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564450457 0 0) 0x7f6dc8153660 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.615+0000 7f6dc759e700 1 -- 172.21.15.102:0/1968512489 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc806c7f0 con 0x7f6dc80cd390 2023-03-03T15:14:31.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.616+0000 7f6dc759e700 1 --2- 172.21.15.102:0/1968512489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd390 0x7f6dc80ce530 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009d90 tx=0x7f6dc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1838746be5bae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:31.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.616+0000 7f6dc659c700 1 -- 172.21.15.102:0/1968512489 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000e070 con 0x7f6dc80cd390 2023-03-03T15:14:31.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.616+0000 7f6dc659c700 1 -- 172.21.15.102:0/1968512489 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc000bb20 con 0x7f6dc80cd390 2023-03-03T15:14:31.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.616+0000 7f6dc659c700 1 -- 172.21.15.102:0/1968512489 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000bed0 con 0x7f6dc80cd390 2023-03-03T15:14:31.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.616+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/1968512489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd390 msgr2=0x7f6dc80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:31.619 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.616+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/1968512489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd390 0x7f6dc80ce530 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009d90 tx=0x7f6dc000b330 comp rx=0 tx=0).stop 2023-03-03T15:14:31.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.617+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/1968512489 shutdown_connections 2023-03-03T15:14:31.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.617+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/1968512489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd390 0x7f6dc80ce530 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.617+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/1968512489 >> 172.21.15.102:0/1968512489 conn(0x7f6dc8065fb0 msgr2=0x7f6dc80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:31.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.617+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/1968512489 shutdown_connections 2023-03-03T15:14:31.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.617+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/1968512489 wait complete. 2023-03-03T15:14:31.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.620+0000 7f6dcd1ff700 1 Processor -- start 2023-03-03T15:14:31.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.620+0000 7f6dcd1ff700 1 -- start start 2023-03-03T15:14:31.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.621+0000 7f6dcd1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dc80cd390 0x7f6dc80c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:31.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.621+0000 7f6dcd1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd8e0 0x7f6dc81eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:31.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.621+0000 7f6dcd1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dc8153290 0x7f6dc81ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:31.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.621+0000 7f6dcd1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc81537f0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.621+0000 7f6dcd1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80c7340 con 0x7f6dc80cd390 2023-03-03T15:14:31.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.621+0000 7f6dcd1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc8153d20 con 0x7f6dc8153290 2023-03-03T15:14:31.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.621+0000 7f6dbffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd8e0 0x7f6dc81eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:31.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dc6d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dc80cd390 0x7f6dc80c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:31.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd8e0 0x7f6dc81eeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36710/0 (socket says 172.21.15.102:36710) 2023-03-03T15:14:31.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 -- 172.21.15.102:0/4261855596 learned_addr learned my addr 172.21.15.102:0/4261855596 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:14:31.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dc759e700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dc8153290 0x7f6dc81ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:31.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 -- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dc8153290 msgr2=0x7f6dc81ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:31.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dc8153290 0x7f6dc81ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 -- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dc80cd390 msgr2=0x7f6dc80c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:31.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dc80cd390 0x7f6dc80c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 -- 172.21.15.102:0/4261855596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0009a70 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.622+0000 7f6dbffff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd8e0 0x7f6dc81eeb00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6db00011e0 tx=0x7f6db000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:31.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.623+0000 7f6dbf7fe700 1 -- 172.21.15.102:0/4261855596 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db00054e0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.623+0000 7f6dbf7fe700 1 -- 172.21.15.102:0/4261855596 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db0003680 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.623+0000 7f6dbf7fe700 1 -- 172.21.15.102:0/4261855596 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0003a30 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.623+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc80ccde0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.623+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc815d160 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.624+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc80cd0c0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.625+0000 7f6dbf7fe700 1 -- 172.21.15.102:0/4261855596 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6db0019030 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.626+0000 7f6dbf7fe700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6da8068320 0x7f6da806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:31.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.626+0000 7f6dbf7fe700 1 -- 172.21.15.102:0/4261855596 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6db00850a0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.627+0000 7f6dc6d9d700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6da8068320 0x7f6da806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:31.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.627+0000 7f6dc6d9d700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6da8068320 0x7f6da806a7d0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f6db8007fd0 tx=0x7f6db8007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:31.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.631+0000 7f6dbf7fe700 1 -- 172.21.15.102:0/4261855596 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6db00517b0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.863 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.862+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dc80c71a0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.863+0000 7f6dbf7fe700 1 -- 172.21.15.102:0/4261855596 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6dc80c71a0 con 0x7f6dc80cd8e0 2023-03-03T15:14:31.865 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:14:31.865 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:14:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.865+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6da8068320 msgr2=0x7f6da806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.865+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6da8068320 0x7f6da806a7d0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f6db8007fd0 tx=0x7f6db8007dc0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.865+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd8e0 msgr2=0x7f6dc81eeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.865+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd8e0 0x7f6dc81eeb00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6db00011e0 tx=0x7f6db000ca30 comp rx=0 tx=0).stop 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 shutdown_connections 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6da8068320 0x7f6da806a7d0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6dc8153290 0x7f6dc81ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dc80cd390 0x7f6dc80c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 --2- 172.21.15.102:0/4261855596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dc80cd8e0 0x7f6dc81eeb00 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 >> 172.21.15.102:0/4261855596 conn(0x7f6dc8065fb0 msgr2=0x7f6dc81575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 shutdown_connections 2023-03-03T15:14:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:14:31.866+0000 7f6dcd1ff700 1 -- 172.21.15.102:0/4261855596 wait complete. 2023-03-03T15:14:31.869 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:14:31.894 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-03-03T15:14:56.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:56.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:26.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:26.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:56.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:56.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:06.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:16:07.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.147+0000 7fba201ff700 1 Processor -- start 2023-03-03T15:16:07.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.147+0000 7fba201ff700 1 -- start start 2023-03-03T15:16:07.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.148+0000 7fba201ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fba180cd8e0 0x7fba180cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:07.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.148+0000 7fba201ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba18153660 con 0x7fba180cd390 2023-03-03T15:16:07.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.148+0000 7fba201ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1806c360 con 0x7fba18153290 2023-03-03T15:16:07.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba201ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1815cea0 con 0x7fba180cd8e0 2023-03-03T15:16:07.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1d79a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fba180cd8e0 0x7fba180cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:07.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1df9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fba18153290 0x7fba18158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33670/0 (socket says 172.21.15.102:33670) 2023-03-03T15:16:07.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1df9b700 1 -- 172.21.15.102:0/2266723897 learned_addr learned my addr 172.21.15.102:0/2266723897 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:16:07.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1cf99700 1 -- 172.21.15.102:0/2266723897 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255530172 0 0) 0x7fba1806c360 con 0x7fba18153290 2023-03-03T15:16:07.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1cf99700 1 -- 172.21.15.102:0/2266723897 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba040035b0 con 0x7fba18153290 2023-03-03T15:16:07.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1d79a700 1 -- 172.21.15.102:0/2266723897 >> v1:172.21.15.172:6789/0 conn(0x7fba18153290 legacy=0x7fba18158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:07.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1d79a700 1 -- 172.21.15.102:0/2266723897 >> v1:172.21.15.102:6789/0 conn(0x7fba180cd390 legacy=0x7fba180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.149+0000 7fba1d79a700 1 -- 172.21.15.102:0/2266723897 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1806c7f0 con 0x7fba180cd8e0 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.150+0000 7fba1d79a700 1 --2- 172.21.15.102:0/2266723897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba180cde20 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fba08009840 tx=0x7fba08009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dee96adfe135b1f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.150+0000 7fba1cf99700 1 -- 172.21.15.102:0/2266723897 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08011830 con 0x7fba180cd8e0 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.150+0000 7fba1cf99700 1 -- 172.21.15.102:0/2266723897 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba08011990 con 0x7fba180cd8e0 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.150+0000 7fba1cf99700 1 -- 172.21.15.102:0/2266723897 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08011d20 con 0x7fba180cd8e0 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.150+0000 7fba201ff700 1 -- 172.21.15.102:0/2266723897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 msgr2=0x7fba180cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.150+0000 7fba201ff700 1 --2- 172.21.15.102:0/2266723897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba180cde20 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fba08009840 tx=0x7fba08009b50 comp rx=0 tx=0).stop 2023-03-03T15:16:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.151+0000 7fba201ff700 1 -- 172.21.15.102:0/2266723897 shutdown_connections 2023-03-03T15:16:07.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.151+0000 7fba201ff700 1 --2- 172.21.15.102:0/2266723897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba180cde20 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.151+0000 7fba201ff700 1 -- 172.21.15.102:0/2266723897 >> 172.21.15.102:0/2266723897 conn(0x7fba18065fb0 msgr2=0x7fba180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:16:07.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.151+0000 7fba201ff700 1 -- 172.21.15.102:0/2266723897 shutdown_connections 2023-03-03T15:16:07.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.151+0000 7fba201ff700 1 -- 172.21.15.102:0/2266723897 wait complete. 2023-03-03T15:16:07.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.153+0000 7fba201ff700 1 Processor -- start 2023-03-03T15:16:07.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.153+0000 7fba201ff700 1 -- start start 2023-03-03T15:16:07.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba201ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba180cd390 0x7fba181f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba201ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba181f1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba201ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba18153290 0x7fba1814df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba201ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba181f07a0 con 0x7fba180cd8e0 2023-03-03T15:16:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba201ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b2100 con 0x7fba180cd390 2023-03-03T15:16:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba201ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba181e4f50 con 0x7fba18153290 2023-03-03T15:16:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba1d79a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba181f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba1d79a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba181f1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48640/0 (socket says 172.21.15.102:48640) 2023-03-03T15:16:07.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba1d79a700 1 -- 172.21.15.102:0/4265405860 learned_addr learned my addr 172.21.15.102:0/4265405860 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:16:07.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba1df9b700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba180cd390 0x7fba181f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:07.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.154+0000 7fba1e79c700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba18153290 0x7fba1814df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba1d79a700 1 -- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba18153290 msgr2=0x7fba1814df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba1d79a700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba18153290 0x7fba1814df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba1d79a700 1 -- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba180cd390 msgr2=0x7fba181f0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba1d79a700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba180cd390 0x7fba181f0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba1d79a700 1 -- 172.21.15.102:0/4265405860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba080094a0 con 0x7fba180cd8e0 2023-03-03T15:16:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba1e79c700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba18153290 0x7fba1814df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:16:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba1d79a700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba181f1100 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fba08009fd0 tx=0x7fba08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba0e7fc700 1 -- 172.21.15.102:0/4265405860 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba080036a0 con 0x7fba180cd8e0 2023-03-03T15:16:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba0e7fc700 1 -- 172.21.15.102:0/4265405860 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0801f070 con 0x7fba180cd8e0 2023-03-03T15:16:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba0e7fc700 1 -- 172.21.15.102:0/4265405860 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08003a60 con 0x7fba180cd8e0 2023-03-03T15:16:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.155+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba180ccde0 con 0x7fba180cd8e0 2023-03-03T15:16:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.156+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1815d160 con 0x7fba180cd8e0 2023-03-03T15:16:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.157+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba180cd0c0 con 0x7fba180cd8e0 2023-03-03T15:16:07.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.158+0000 7fba0e7fc700 1 -- 172.21.15.102:0/4265405860 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba0800f050 con 0x7fba180cd8e0 2023-03-03T15:16:07.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.159+0000 7fba0e7fc700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9fc0682f0 0x7fb9fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:07.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.159+0000 7fba0e7fc700 1 -- 172.21.15.102:0/4265405860 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fba08021070 con 0x7fba180cd8e0 2023-03-03T15:16:07.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.159+0000 7fba1df9b700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9fc0682f0 0x7fb9fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:07.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.160+0000 7fba1df9b700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9fc0682f0 0x7fb9fc06a7a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fba14007b50 tx=0x7fba14007ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:07.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.164+0000 7fba0e7fc700 1 -- 172.21.15.102:0/4265405860 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba08053cb0 con 0x7fba180cd8e0 2023-03-03T15:16:07.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.398+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba181f08e0 con 0x7fba180cd8e0 2023-03-03T15:16:07.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.398+0000 7fba0e7fc700 1 -- 172.21.15.102:0/4265405860 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fba181f08e0 con 0x7fba180cd8e0 2023-03-03T15:16:07.401 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:16:07.401 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:16:07.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9fc0682f0 msgr2=0x7fb9fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:07.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9fc0682f0 0x7fb9fc06a7a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fba14007b50 tx=0x7fba14007ac0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 msgr2=0x7fba181f1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:07.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba181f1100 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fba08009fd0 tx=0x7fba08004750 comp rx=0 tx=0).stop 2023-03-03T15:16:07.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 shutdown_connections 2023-03-03T15:16:07.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb9fc0682f0 0x7fb9fc06a7a0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fba18153290 0x7fba1814df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba180cd390 0x7fba181f0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 --2- 172.21.15.102:0/4265405860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba180cd8e0 0x7fba181f1100 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:07.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 >> 172.21.15.102:0/4265405860 conn(0x7fba18065fb0 msgr2=0x7fba181513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:16:07.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 shutdown_connections 2023-03-03T15:16:07.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:16:07.401+0000 7fba201ff700 1 -- 172.21.15.102:0/4265405860 wait complete. 2023-03-03T15:16:07.406 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:16:07.430 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-03-03T15:16:26.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:26.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:57.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:57.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:27.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:27.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:57.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:57.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:01.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:18:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.681+0000 7fed115ff700 1 Processor -- start 2023-03-03T15:18:01.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.681+0000 7fed115ff700 1 -- start start 2023-03-03T15:18:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.682+0000 7fed115ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd8e0 0x7fed0c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.682+0000 7fed115ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed0c0cd390 0x7fed0c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.682+0000 7fed115ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c153660 con 0x7fed0c153290 2023-03-03T15:18:01.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.682+0000 7fed115ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c06c360 con 0x7fed0c0cd390 2023-03-03T15:18:01.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.682+0000 7fed115ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c15cea0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd8e0 0x7fed0c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed0c0cd390 0x7fed0c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0affd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fed0c153290 0x7fed0c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48768/0 (socket says 172.21.15.102:48768) 2023-03-03T15:18:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0affd700 1 -- 172.21.15.102:0/3396735655 learned_addr learned my addr 172.21.15.102:0/3396735655 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:18:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0a7fc700 1 -- 172.21.15.102:0/3396735655 >> v2:172.21.15.102:3300/0 conn(0x7fed0c0cd390 msgr2=0x7fed0c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:18:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0a7fc700 1 --2- 172.21.15.102:0/3396735655 >> v2:172.21.15.102:3300/0 conn(0x7fed0c0cd390 0x7fed0c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0a7fc700 1 -- 172.21.15.102:0/3396735655 >> v1:172.21.15.102:6789/0 conn(0x7fed0c153290 legacy=0x7fed0c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed09ffb700 1 -- 172.21.15.102:0/3396735655 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881559083 0 0) 0x7fed0c153660 con 0x7fed0c153290 2023-03-03T15:18:01.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.683+0000 7fed0a7fc700 1 -- 172.21.15.102:0/3396735655 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c06c7f0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed0a7fc700 1 --2- 172.21.15.102:0/3396735655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd8e0 0x7fed0c0cde20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fed00009840 tx=0x7fed00009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec64b903dfcd86e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:01.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed09ffb700 1 -- 172.21.15.102:0/3396735655 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00011830 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed09ffb700 1 -- 172.21.15.102:0/3396735655 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00011990 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed09ffb700 1 -- 172.21.15.102:0/3396735655 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00011d20 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed115ff700 1 -- 172.21.15.102:0/3396735655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd8e0 msgr2=0x7fed0c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed115ff700 1 --2- 172.21.15.102:0/3396735655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd8e0 0x7fed0c0cde20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fed00009840 tx=0x7fed00009b50 comp rx=0 tx=0).stop 2023-03-03T15:18:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed115ff700 1 -- 172.21.15.102:0/3396735655 shutdown_connections 2023-03-03T15:18:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed115ff700 1 --2- 172.21.15.102:0/3396735655 >> v2:172.21.15.102:3300/0 conn(0x7fed0c0cd390 0x7fed0c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed115ff700 1 --2- 172.21.15.102:0/3396735655 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd8e0 0x7fed0c0cde20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.684+0000 7fed115ff700 1 -- 172.21.15.102:0/3396735655 >> 172.21.15.102:0/3396735655 conn(0x7fed0c065fb0 msgr2=0x7fed0c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.685+0000 7fed115ff700 1 -- 172.21.15.102:0/3396735655 shutdown_connections 2023-03-03T15:18:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.685+0000 7fed115ff700 1 -- 172.21.15.102:0/3396735655 wait complete. 2023-03-03T15:18:01.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.687+0000 7fed115ff700 1 Processor -- start 2023-03-03T15:18:01.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.687+0000 7fed115ff700 1 -- start start 2023-03-03T15:18:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed115ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd390 0x7fed0c1ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed115ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0c0cd8e0 0x7fed0c14f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed115ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c153290 0x7fed0c14fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed115ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c3a40 con 0x7fed0c153290 2023-03-03T15:18:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed115ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c151600 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed115ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1518a0 con 0x7fed0c0cd390 2023-03-03T15:18:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed0a7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0c0cd8e0 0x7fed0c14f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed0affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd390 0x7fed0c1ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed0a7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0c0cd8e0 0x7fed0c14f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56194/0 (socket says 172.21.15.102:56194) 2023-03-03T15:18:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed0a7fc700 1 -- 172.21.15.102:0/4118759974 learned_addr learned my addr 172.21.15.102:0/4118759974 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:18:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.688+0000 7fed0b7fe700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c153290 0x7fed0c14fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:01.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fed0a7fc700 1 -- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd390 msgr2=0x7fed0c1ec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:01.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fed0a7fc700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd390 0x7fed0c1ec5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fed0a7fc700 1 -- 172.21.15.102:0/4118759974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c153290 msgr2=0x7fed0c14fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:01.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fed0a7fc700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c153290 0x7fed0c14fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fed0a7fc700 1 -- 172.21.15.102:0/4118759974 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed000094a0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fed0a7fc700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0c0cd8e0 0x7fed0c14f740 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fed00009fd0 tx=0x7fed00004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:01.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fecf37fe700 1 -- 172.21.15.102:0/4118759974 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed000036a0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fecf37fe700 1 -- 172.21.15.102:0/4118759974 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0001f070 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c0ccde0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.689+0000 7fecf37fe700 1 -- 172.21.15.102:0/4118759974 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00003a60 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.690+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c15d160 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.691+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c0cd0c0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.692+0000 7fecf37fe700 1 -- 172.21.15.102:0/4118759974 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed0000f050 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.693+0000 7fecf37fe700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecf8068320 0x7fecf806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:01.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.693+0000 7fecf37fe700 1 -- 172.21.15.102:0/4118759974 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fed00021070 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.693+0000 7fed0affd700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecf8068320 0x7fecf806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:01.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.694+0000 7fed0affd700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecf8068320 0x7fecf806a7d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fecfc007fd0 tx=0x7fecfc007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:01.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.696+0000 7fecf37fe700 1 -- 172.21.15.102:0/4118759974 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed00053cf0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.929+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c04e6e0 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.930+0000 7fecf37fe700 1 -- 172.21.15.102:0/4118759974 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fed00057310 con 0x7fed0c0cd8e0 2023-03-03T15:18:01.931 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:18:01.932 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:18:01.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecf8068320 msgr2=0x7fecf806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecf8068320 0x7fecf806a7d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fecfc007fd0 tx=0x7fecfc007de0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0c0cd8e0 msgr2=0x7fed0c14f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0c0cd8e0 0x7fed0c14f740 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fed00009fd0 tx=0x7fed00004750 comp rx=0 tx=0).stop 2023-03-03T15:18:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 shutdown_connections 2023-03-03T15:18:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fecf8068320 0x7fecf806a7d0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed0c0cd390 0x7fed0c1ec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c153290 0x7fed0c14fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 --2- 172.21.15.102:0/4118759974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed0c0cd8e0 0x7fed0c14f740 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 >> 172.21.15.102:0/4118759974 conn(0x7fed0c065fb0 msgr2=0x7fed0c157460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 shutdown_connections 2023-03-03T15:18:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:01.933+0000 7fed115ff700 1 -- 172.21.15.102:0/4118759974 wait complete. 2023-03-03T15:18:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:18:01.962 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-03-03T15:18:13.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:18:14.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.214+0000 7f8456bff700 1 Processor -- start 2023-03-03T15:18:14.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.214+0000 7f8456bff700 1 -- start start 2023-03-03T15:18:14.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f8456bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f8456bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f84500cd8e0 0x7f84500ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f8456bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8450153660 con 0x7f84501593c0 2023-03-03T15:18:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f8456bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845006c360 con 0x7f84500cd8e0 2023-03-03T15:18:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f8456bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845015cea0 con 0x7f8450153290 2023-03-03T15:18:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f844ffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f84501593c0 0x7f84500cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:48748/0 (socket says 172.21.15.102:48748) 2023-03-03T15:18:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f845499b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f844ffff700 1 -- 172.21.15.102:0/2705360981 learned_addr learned my addr 172.21.15.102:0/2705360981 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:18:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.216+0000 7f845519c700 1 --2- 172.21.15.102:0/2705360981 >> v2:172.21.15.172:3300/0 conn(0x7f84500cd8e0 0x7f84500ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:14.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f844f7fe700 1 -- 172.21.15.102:0/2705360981 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858040192 0 0) 0x7f8450153660 con 0x7f84501593c0 2023-03-03T15:18:14.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f844f7fe700 1 -- 172.21.15.102:0/2705360981 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84400035b0 con 0x7f84501593c0 2023-03-03T15:18:14.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f845499b700 1 -- 172.21.15.102:0/2705360981 >> v2:172.21.15.172:3300/0 conn(0x7f84500cd8e0 msgr2=0x7f84500ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:18:14.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f845499b700 1 --2- 172.21.15.102:0/2705360981 >> v2:172.21.15.172:3300/0 conn(0x7f84500cd8e0 0x7f84500ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f845499b700 1 -- 172.21.15.102:0/2705360981 >> v1:172.21.15.172:6789/0 conn(0x7f84501593c0 legacy=0x7f84500cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:14.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f845499b700 1 -- 172.21.15.102:0/2705360981 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845006c7f0 con 0x7f8450153290 2023-03-03T15:18:14.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f845499b700 1 --2- 172.21.15.102:0/2705360981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450158e80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8444009840 tx=0x7f8444009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8760c706a7d431c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:14.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f844f7fe700 1 -- 172.21.15.102:0/2705360981 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444011830 con 0x7f8450153290 2023-03-03T15:18:14.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f844f7fe700 1 -- 172.21.15.102:0/2705360981 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444011990 con 0x7f8450153290 2023-03-03T15:18:14.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.217+0000 7f844f7fe700 1 -- 172.21.15.102:0/2705360981 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444011d20 con 0x7f8450153290 2023-03-03T15:18:14.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 -- 172.21.15.102:0/2705360981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 msgr2=0x7f8450158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:14.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 --2- 172.21.15.102:0/2705360981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450158e80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8444009840 tx=0x7f8444009b50 comp rx=0 tx=0).stop 2023-03-03T15:18:14.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 -- 172.21.15.102:0/2705360981 shutdown_connections 2023-03-03T15:18:14.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 --2- 172.21.15.102:0/2705360981 >> v2:172.21.15.172:3300/0 conn(0x7f84500cd8e0 0x7f84500ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 --2- 172.21.15.102:0/2705360981 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450158e80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 -- 172.21.15.102:0/2705360981 >> 172.21.15.102:0/2705360981 conn(0x7f8450065fb0 msgr2=0x7f84500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:14.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 -- 172.21.15.102:0/2705360981 shutdown_connections 2023-03-03T15:18:14.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.218+0000 7f8456bff700 1 -- 172.21.15.102:0/2705360981 wait complete. 2023-03-03T15:18:14.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.220+0000 7f8456bff700 1 Processor -- start 2023-03-03T15:18:14.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.220+0000 7f8456bff700 1 -- start start 2023-03-03T15:18:14.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.221+0000 7f8456bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84500cd8e0 0x7f84501f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.221+0000 7f8456bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f8456bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84501593c0 0x7f84501561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f8456bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450151570 con 0x7f84501593c0 2023-03-03T15:18:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f8456bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84500c8ec0 con 0x7f84500cd8e0 2023-03-03T15:18:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f8456bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84501f23b0 con 0x7f8450153290 2023-03-03T15:18:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f845499b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84500cd8e0 0x7f84501f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58156/0 (socket says 172.21.15.102:58156) 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f845499b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84500cd8e0 0x7f84501f1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55330/0 (socket says 172.21.15.102:55330) 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 -- 172.21.15.102:0/790910360 learned_addr learned my addr 172.21.15.102:0/790910360 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f845519c700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84501593c0 0x7f84501561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 -- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84500cd8e0 msgr2=0x7f84501f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84500cd8e0 0x7f84501f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 -- 172.21.15.102:0/790910360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84501593c0 msgr2=0x7f84501561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:14.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84501593c0 0x7f84501561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.222+0000 7f844ffff700 1 -- 172.21.15.102:0/790910360 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84440094a0 con 0x7f8450153290 2023-03-03T15:18:14.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.223+0000 7f844ffff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450155c70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f843c00c700 tx=0x7f843c00ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:14.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.223+0000 7f844d7fa700 1 -- 172.21.15.102:0/790910360 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c005d70 con 0x7f8450153290 2023-03-03T15:18:14.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.223+0000 7f844d7fa700 1 -- 172.21.15.102:0/790910360 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c0070d0 con 0x7f8450153290 2023-03-03T15:18:14.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.223+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84500ccde0 con 0x7f8450153290 2023-03-03T15:18:14.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.223+0000 7f844d7fa700 1 -- 172.21.15.102:0/790910360 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c007460 con 0x7f8450153290 2023-03-03T15:18:14.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.223+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845015d160 con 0x7f8450153290 2023-03-03T15:18:14.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.224+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84500cd0c0 con 0x7f8450153290 2023-03-03T15:18:14.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.225+0000 7f844d7fa700 1 -- 172.21.15.102:0/790910360 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f843c019030 con 0x7f8450153290 2023-03-03T15:18:14.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.227+0000 7f844d7fa700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84340682f0 0x7f843406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:14.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.227+0000 7f844d7fa700 1 -- 172.21.15.102:0/790910360 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f843c0850f0 con 0x7f8450153290 2023-03-03T15:18:14.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.227+0000 7f845499b700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84340682f0 0x7f843406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:14.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.228+0000 7f845499b700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84340682f0 0x7f843406a7a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f8444000c00 tx=0x7f8444008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:14.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.231+0000 7f844d7fa700 1 -- 172.21.15.102:0/790910360 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f843c0517a0 con 0x7f8450153290 2023-03-03T15:18:14.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.467+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f845004e6e0 con 0x7f8450153290 2023-03-03T15:18:14.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.468+0000 7f844d7fa700 1 -- 172.21.15.102:0/790910360 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f843c054dc0 con 0x7f8450153290 2023-03-03T15:18:14.470 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:18:14.470 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:18:14.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84340682f0 msgr2=0x7f843406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:14.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84340682f0 0x7f843406a7a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f8444000c00 tx=0x7f8444008030 comp rx=0 tx=0).stop 2023-03-03T15:18:14.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 msgr2=0x7f8450155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450155c70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f843c00c700 tx=0x7f843c00ca10 comp rx=0 tx=0).stop 2023-03-03T15:18:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 shutdown_connections 2023-03-03T15:18:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84501593c0 0x7f84501561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84500cd8e0 0x7f84501f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f84340682f0 0x7f843406a7a0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 --2- 172.21.15.102:0/790910360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8450153290 0x7f8450155c70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 >> 172.21.15.102:0/790910360 conn(0x7f8450065fb0 msgr2=0x7f84500cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:14.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 shutdown_connections 2023-03-03T15:18:14.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:18:14.471+0000 7f8456bff700 1 -- 172.21.15.102:0/790910360 wait complete. 2023-03-03T15:18:14.474 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:18:14.498 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 106 secs before thrashing 2023-03-03T15:18:27.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:27.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:57.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:57.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:27.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:27.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:57.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:57.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T15:20:00.000+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:20:00.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:00.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.757+0000 7f18ca99d700 1 Processor -- start 2023-03-03T15:20:00.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.757+0000 7f18ca99d700 1 -- start start 2023-03-03T15:20:00.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18ca99d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f18c40b3970 0x7f18c40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:00.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18ca99d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f18c40c41b0 0x7f18c40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:00.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18ca99d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c40c3d20 con 0x7f18c40ced00 2023-03-03T15:20:00.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18ca99d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c40c4930 con 0x7f18c40c41b0 2023-03-03T15:20:00.761 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18ca99d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c40cf610 con 0x7f18c40b3970 2023-03-03T15:20:00.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18ca19c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f18c40c41b0 0x7f18c40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:00.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18c999b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f18c40ced00 0x7f18c4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60002/0 (socket says 172.21.15.102:60002) 2023-03-03T15:20:00.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18c999b700 1 -- 172.21.15.102:0/3409769368 learned_addr learned my addr 172.21.15.102:0/3409769368 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:20:00.764 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.759+0000 7f18c919a700 1 --2- 172.21.15.102:0/3409769368 >> v2:172.21.15.172:3300/0 conn(0x7f18c40b3970 0x7f18c40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:00.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18c8999700 1 -- 172.21.15.102:0/3409769368 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575126515 0 0) 0x7f18c40c3d20 con 0x7f18c40ced00 2023-03-03T15:20:00.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18ca19c700 1 -- 172.21.15.102:0/3409769368 >> v2:172.21.15.172:3300/0 conn(0x7f18c40b3970 msgr2=0x7f18c40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:00.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18ca19c700 1 --2- 172.21.15.102:0/3409769368 >> v2:172.21.15.172:3300/0 conn(0x7f18c40b3970 0x7f18c40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:00.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18ca19c700 1 -- 172.21.15.102:0/3409769368 >> v1:172.21.15.172:6789/0 conn(0x7f18c40ced00 legacy=0x7f18c4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:00.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18ca19c700 1 -- 172.21.15.102:0/3409769368 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c415cba0 con 0x7f18c40c41b0 2023-03-03T15:20:00.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18ca19c700 1 --2- 172.21.15.102:0/3409769368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40c41b0 0x7f18c40cf0d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f18c40c4ba0 tx=0x7f18bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638a122bed37316b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:00.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18c8999700 1 -- 172.21.15.102:0/3409769368 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18bc00e070 con 0x7f18c40c41b0 2023-03-03T15:20:00.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18c8999700 1 -- 172.21.15.102:0/3409769368 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18bc00bae0 con 0x7f18c40c41b0 2023-03-03T15:20:00.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.760+0000 7f18c8999700 1 -- 172.21.15.102:0/3409769368 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18bc00be90 con 0x7f18c40c41b0 2023-03-03T15:20:00.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 -- 172.21.15.102:0/3409769368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40c41b0 msgr2=0x7f18c40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:00.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3409769368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40c41b0 0x7f18c40cf0d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f18c40c4ba0 tx=0x7f18bc00b330 comp rx=0 tx=0).stop 2023-03-03T15:20:00.767 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 -- 172.21.15.102:0/3409769368 shutdown_connections 2023-03-03T15:20:00.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3409769368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40c41b0 0x7f18c40cf0d0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:00.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3409769368 >> v2:172.21.15.172:3300/0 conn(0x7f18c40b3970 0x7f18c40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:00.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 -- 172.21.15.102:0/3409769368 >> 172.21.15.102:0/3409769368 conn(0x7f18c408b1c0 msgr2=0x7f18c4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:00.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 -- 172.21.15.102:0/3409769368 shutdown_connections 2023-03-03T15:20:00.768 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.761+0000 7f18ca99d700 1 -- 172.21.15.102:0/3409769368 wait complete. 2023-03-03T15:20:00.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.763+0000 7f18ca99d700 1 Processor -- start 2023-03-03T15:20:00.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.763+0000 7f18ca99d700 1 -- start start 2023-03-03T15:20:00.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.764+0000 7f18ca99d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40b3970 0x7f18c4159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:00.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.764+0000 7f18ca99d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18c40c41b0 0x7f18c4151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:00.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.764+0000 7f18ca99d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18c40ced00 0x7f18c414f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:00.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.764+0000 7f18ca99d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c414d100 con 0x7f18c40b3970 2023-03-03T15:20:00.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.764+0000 7f18ca99d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c415a300 con 0x7f18c40ced00 2023-03-03T15:20:00.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.764+0000 7f18ca99d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c415ba60 con 0x7f18c40c41b0 2023-03-03T15:20:00.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40b3970 0x7f18c4159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:00.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18ca19c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18c40ced00 0x7f18c414f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:00.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c919a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18c40c41b0 0x7f18c4151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:00.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40b3970 0x7f18c4159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52426/0 (socket says 172.21.15.102:52426) 2023-03-03T15:20:00.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18ca19c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18c40ced00 0x7f18c414f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:38764/0 (socket says 172.21.15.102:38764) 2023-03-03T15:20:00.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 -- 172.21.15.102:0/3975273454 learned_addr learned my addr 172.21.15.102:0/3975273454 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:20:00.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 -- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18c40c41b0 msgr2=0x7f18c4151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:00.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18c40c41b0 0x7f18c4151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:00.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 -- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18c40ced00 msgr2=0x7f18c414f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:00.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18c40ced00 0x7f18c414f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:00.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 -- 172.21.15.102:0/3975273454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc009a70 con 0x7f18c40b3970 2023-03-03T15:20:00.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c919a700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18c40c41b0 0x7f18c4151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:20:00.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.765+0000 7f18c999b700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40b3970 0x7f18c4159cc0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f18c00011e0 tx=0x7f18c000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:00.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.766+0000 7f18ba7fc700 1 -- 172.21.15.102:0/3975273454 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0004c80 con 0x7f18c40b3970 2023-03-03T15:20:00.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.766+0000 7f18ba7fc700 1 -- 172.21.15.102:0/3975273454 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c0004e00 con 0x7f18c40b3970 2023-03-03T15:20:00.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.766+0000 7f18ba7fc700 1 -- 172.21.15.102:0/3975273454 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0007310 con 0x7f18c40b3970 2023-03-03T15:20:00.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.766+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c406bf20 con 0x7f18c40b3970 2023-03-03T15:20:00.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.766+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c415ce80 con 0x7f18c40b3970 2023-03-03T15:20:00.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.767+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c406c200 con 0x7f18c40b3970 2023-03-03T15:20:00.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.768+0000 7f18ba7fc700 1 -- 172.21.15.102:0/3975273454 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18c001a020 con 0x7f18c40b3970 2023-03-03T15:20:00.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.769+0000 7f18ba7fc700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18a80682f0 0x7f18a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:00.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.769+0000 7f18ba7fc700 1 -- 172.21.15.102:0/3975273454 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f18c00858e0 con 0x7f18c40b3970 2023-03-03T15:20:00.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.770+0000 7f18c919a700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18a80682f0 0x7f18a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:00.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.770+0000 7f18c919a700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18a80682f0 0x7f18a806a7a0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f18c414dd30 tx=0x7f18b4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:00.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:00.773+0000 7f18ba7fc700 1 -- 172.21.15.102:0/3975273454 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18c0051f90 con 0x7f18c40b3970 2023-03-03T15:20:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.008+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18c404e6e0 con 0x7f18c40b3970 2023-03-03T15:20:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.008+0000 7f18ba7fc700 1 -- 172.21.15.102:0/3975273454 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f18c00555b0 con 0x7f18c40b3970 2023-03-03T15:20:01.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:20:01.011 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:20:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18a80682f0 msgr2=0x7f18a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18a80682f0 0x7f18a806a7a0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f18c414dd30 tx=0x7f18b4009210 comp rx=0 tx=0).stop 2023-03-03T15:20:01.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40b3970 msgr2=0x7f18c4159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40b3970 0x7f18c4159cc0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f18c00011e0 tx=0x7f18c000ca30 comp rx=0 tx=0).stop 2023-03-03T15:20:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 shutdown_connections 2023-03-03T15:20:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f18c40ced00 0x7f18c414f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c40b3970 0x7f18c4159cc0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f18a80682f0 0x7f18a806a7a0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 --2- 172.21.15.102:0/3975273454 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f18c40c41b0 0x7f18c4151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 >> 172.21.15.102:0/3975273454 conn(0x7f18c408b1c0 msgr2=0x7f18c4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.011+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 shutdown_connections 2023-03-03T15:20:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:01.012+0000 7f18ca99d700 1 -- 172.21.15.102:0/3975273454 wait complete. 2023-03-03T15:20:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:20:01.042 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before thrashing 2023-03-03T15:20:02.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.294+0000 7f5ab139d700 1 Processor -- start 2023-03-03T15:20:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.294+0000 7f5ab139d700 1 -- start start 2023-03-03T15:20:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5ab139d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0ced00 0x7f5aac0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5ab139d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5aac0c41b0 0x7f5aac0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5ab139d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac0c3d20 con 0x7f5aac0b3970 2023-03-03T15:20:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5ab139d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aac0c4930 con 0x7f5aac0c41b0 2023-03-03T15:20:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5ab139d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0cf610 con 0x7f5aac0ced00 2023-03-03T15:20:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5aabfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0ced00 0x7f5aac0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5aab7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5aac0b3970 0x7f5aac158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60006/0 (socket says 172.21.15.102:60006) 2023-03-03T15:20:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5ab0b9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5aac0c41b0 0x7f5aac0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.296+0000 7f5aab7fe700 1 -- 172.21.15.102:0/1731565533 learned_addr learned my addr 172.21.15.102:0/1731565533 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:20:02.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aaaffd700 1 -- 172.21.15.102:0/1731565533 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017587242 0 0) 0x7f5aac0c3d20 con 0x7f5aac0b3970 2023-03-03T15:20:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aaaffd700 1 -- 172.21.15.102:0/1731565533 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a940035b0 con 0x7f5aac0b3970 2023-03-03T15:20:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aabfff700 1 -- 172.21.15.102:0/1731565533 >> v2:172.21.15.172:3300/0 conn(0x7f5aac0c41b0 msgr2=0x7f5aac0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aabfff700 1 --2- 172.21.15.102:0/1731565533 >> v2:172.21.15.172:3300/0 conn(0x7f5aac0c41b0 0x7f5aac0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aabfff700 1 -- 172.21.15.102:0/1731565533 >> v1:172.21.15.172:6789/0 conn(0x7f5aac0b3970 legacy=0x7f5aac158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aabfff700 1 -- 172.21.15.102:0/1731565533 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac15cba0 con 0x7f5aac0ced00 2023-03-03T15:20:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5ab0b9c700 1 --2- 172.21.15.102:0/1731565533 >> v2:172.21.15.172:3300/0 conn(0x7f5aac0c41b0 0x7f5aac0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:20:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aabfff700 1 --2- 172.21.15.102:0/1731565533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0ced00 0x7f5aac0c5a90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009840 tx=0x7f5a9c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f87422ed3f99dc4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.297+0000 7f5aaaffd700 1 -- 172.21.15.102:0/1731565533 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c011830 con 0x7f5aac0ced00 2023-03-03T15:20:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5aaaffd700 1 -- 172.21.15.102:0/1731565533 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c011990 con 0x7f5aac0ced00 2023-03-03T15:20:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5aaaffd700 1 -- 172.21.15.102:0/1731565533 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c011d20 con 0x7f5aac0ced00 2023-03-03T15:20:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 -- 172.21.15.102:0/1731565533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0ced00 msgr2=0x7f5aac0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:02.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 --2- 172.21.15.102:0/1731565533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0ced00 0x7f5aac0c5a90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009840 tx=0x7f5a9c009b50 comp rx=0 tx=0).stop 2023-03-03T15:20:02.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 -- 172.21.15.102:0/1731565533 shutdown_connections 2023-03-03T15:20:02.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 --2- 172.21.15.102:0/1731565533 >> v2:172.21.15.172:3300/0 conn(0x7f5aac0c41b0 0x7f5aac0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 --2- 172.21.15.102:0/1731565533 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0ced00 0x7f5aac0c5a90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 -- 172.21.15.102:0/1731565533 >> 172.21.15.102:0/1731565533 conn(0x7f5aac08b1c0 msgr2=0x7f5aac071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:02.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 -- 172.21.15.102:0/1731565533 shutdown_connections 2023-03-03T15:20:02.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.298+0000 7f5ab139d700 1 -- 172.21.15.102:0/1731565533 wait complete. 2023-03-03T15:20:02.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.300+0000 7f5ab139d700 1 Processor -- start 2023-03-03T15:20:02.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.300+0000 7f5ab139d700 1 -- start start 2023-03-03T15:20:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.301+0000 7f5ab139d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aac0b3970 0x7f5aac1e3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.301+0000 7f5ab139d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0c41b0 0x7f5aac151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5ab139d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aac0ced00 0x7f5aac1f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5ab139d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac1e4e40 con 0x7f5aac0b3970 2023-03-03T15:20:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5ab139d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac148530 con 0x7f5aac0ced00 2023-03-03T15:20:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5ab139d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac1e40f0 con 0x7f5aac0c41b0 2023-03-03T15:20:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aac0b3970 0x7f5aac1e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aac0b3970 0x7f5aac1e3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52442/0 (socket says 172.21.15.102:52442) 2023-03-03T15:20:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 -- 172.21.15.102:0/2046699660 learned_addr learned my addr 172.21.15.102:0/2046699660 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:20:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aab7fe700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0c41b0 0x7f5aac151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5ab0b9c700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aac0ced00 0x7f5aac1f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 -- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0c41b0 msgr2=0x7f5aac151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0c41b0 0x7f5aac151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 -- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aac0ced00 msgr2=0x7f5aac1f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aac0ced00 0x7f5aac1f1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.302+0000 7f5aabfff700 1 -- 172.21.15.102:0/2046699660 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4009a70 con 0x7f5aac0b3970 2023-03-03T15:20:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.303+0000 7f5aabfff700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aac0b3970 0x7f5aac1e3430 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0090b0 tx=0x7f5a9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.303+0000 7f5aa8ff9700 1 -- 172.21.15.102:0/2046699660 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c0044d0 con 0x7f5aac0b3970 2023-03-03T15:20:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.303+0000 7f5aa8ff9700 1 -- 172.21.15.102:0/2046699660 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c01f070 con 0x7f5aac0b3970 2023-03-03T15:20:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.303+0000 7f5aa8ff9700 1 -- 172.21.15.102:0/2046699660 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c0038c0 con 0x7f5aac0b3970 2023-03-03T15:20:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.303+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c0094a0 con 0x7f5aac0b3970 2023-03-03T15:20:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.303+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac06bf20 con 0x7f5aac0b3970 2023-03-03T15:20:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.304+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aac15ce80 con 0x7f5aac0b3970 2023-03-03T15:20:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.306+0000 7f5aa8ff9700 1 -- 172.21.15.102:0/2046699660 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a9c00f050 con 0x7f5aac0b3970 2023-03-03T15:20:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.307+0000 7f5aa8ff9700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a980682f0 0x7f5a9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.307+0000 7f5aa8ff9700 1 -- 172.21.15.102:0/2046699660 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5a9c021070 con 0x7f5aac0b3970 2023-03-03T15:20:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.307+0000 7f5aab7fe700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a980682f0 0x7f5a9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.308+0000 7f5aab7fe700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a980682f0 0x7f5a9806a7a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f5aac1f2700 tx=0x7f5aa0007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.312+0000 7f5aa8ff9700 1 -- 172.21.15.102:0/2046699660 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a9c053c60 con 0x7f5aac0b3970 2023-03-03T15:20:02.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.548+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5aac06c200 con 0x7f5aac0b3970 2023-03-03T15:20:02.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.548+0000 7f5aa8ff9700 1 -- 172.21.15.102:0/2046699660 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5a9c057280 con 0x7f5aac0b3970 2023-03-03T15:20:02.551 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:20:02.551 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:20:02.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a980682f0 msgr2=0x7f5a9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:02.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a980682f0 0x7f5a9806a7a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f5aac1f2700 tx=0x7f5aa0007dc0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aac0b3970 msgr2=0x7f5aac1e3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:02.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aac0b3970 0x7f5aac1e3430 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0090b0 tx=0x7f5a9c004750 comp rx=0 tx=0).stop 2023-03-03T15:20:02.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 shutdown_connections 2023-03-03T15:20:02.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5aac0ced00 0x7f5aac1f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5aac0b3970 0x7f5aac1e3430 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5a980682f0 0x7f5a9806a7a0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 --2- 172.21.15.102:0/2046699660 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5aac0c41b0 0x7f5aac151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:02.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 >> 172.21.15.102:0/2046699660 conn(0x7f5aac08b1c0 msgr2=0x7f5aac0b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:02.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 shutdown_connections 2023-03-03T15:20:02.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:02.551+0000 7f5ab139d700 1 -- 172.21.15.102:0/2046699660 wait complete. 2023-03-03T15:20:02.555 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:20:02.580 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 15 secs before thrashing 2023-03-03T15:20:17.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:17.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.831+0000 7ff2e99ff700 1 Processor -- start 2023-03-03T15:20:17.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.831+0000 7ff2e99ff700 1 -- start start 2023-03-03T15:20:17.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.832+0000 7ff2e99ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff2e4153290 0x7ff2e4158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.832+0000 7ff2e99ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2e40cd8e0 0x7ff2e40ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e99ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4153660 con 0x7ff2e41593c0 2023-03-03T15:20:17.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e99ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e406c360 con 0x7ff2e40cd8e0 2023-03-03T15:20:17.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e99ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e415cea0 con 0x7ff2e4153290 2023-03-03T15:20:17.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e27fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff2e41593c0 0x7ff2e40cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54336/0 (socket says 172.21.15.102:54336) 2023-03-03T15:20:17.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e27fc700 1 -- 172.21.15.102:0/467367519 learned_addr learned my addr 172.21.15.102:0/467367519 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:20:17.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e2ffd700 1 --2- 172.21.15.102:0/467367519 >> v2:172.21.15.172:3300/0 conn(0x7ff2e4153290 0x7ff2e4158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e37fe700 1 --2- 172.21.15.102:0/467367519 >> v2:172.21.15.102:3300/0 conn(0x7ff2e40cd8e0 0x7ff2e40ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/467367519 >> v2:172.21.15.102:3300/0 conn(0x7ff2e40cd8e0 msgr2=0x7ff2e40ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:17.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e1ffb700 1 -- 172.21.15.102:0/467367519 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698238203 0 0) 0x7ff2e4153660 con 0x7ff2e41593c0 2023-03-03T15:20:17.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e2ffd700 1 --2- 172.21.15.102:0/467367519 >> v2:172.21.15.102:3300/0 conn(0x7ff2e40cd8e0 0x7ff2e40ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.833+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/467367519 >> v1:172.21.15.102:6789/0 conn(0x7ff2e41593c0 legacy=0x7ff2e40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.834+0000 7ff2e2ffd700 1 -- 172.21.15.102:0/467367519 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e406c7f0 con 0x7ff2e4153290 2023-03-03T15:20:17.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.834+0000 7ff2e2ffd700 1 --2- 172.21.15.102:0/467367519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e4158e80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff2e408cc70 tx=0x7ff2d4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c9877edc973e0f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:17.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.834+0000 7ff2e1ffb700 1 -- 172.21.15.102:0/467367519 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4011830 con 0x7ff2e4153290 2023-03-03T15:20:17.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.834+0000 7ff2e1ffb700 1 -- 172.21.15.102:0/467367519 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d4011990 con 0x7ff2e4153290 2023-03-03T15:20:17.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.834+0000 7ff2e1ffb700 1 -- 172.21.15.102:0/467367519 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4011d20 con 0x7ff2e4153290 2023-03-03T15:20:17.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 -- 172.21.15.102:0/467367519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 msgr2=0x7ff2e4158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/467367519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e4158e80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff2e408cc70 tx=0x7ff2d4009b20 comp rx=0 tx=0).stop 2023-03-03T15:20:17.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 -- 172.21.15.102:0/467367519 shutdown_connections 2023-03-03T15:20:17.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/467367519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e4158e80 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/467367519 >> v2:172.21.15.102:3300/0 conn(0x7ff2e40cd8e0 0x7ff2e40ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 -- 172.21.15.102:0/467367519 >> 172.21.15.102:0/467367519 conn(0x7ff2e4065fb0 msgr2=0x7ff2e40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 -- 172.21.15.102:0/467367519 shutdown_connections 2023-03-03T15:20:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.835+0000 7ff2e99ff700 1 -- 172.21.15.102:0/467367519 wait complete. 2023-03-03T15:20:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.837+0000 7ff2e99ff700 1 Processor -- start 2023-03-03T15:20:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.837+0000 7ff2e99ff700 1 -- start start 2023-03-03T15:20:17.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e99ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e40cd8e0 0x7ff2e41ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e99ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e99ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2e41593c0 0x7ff2e40c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e99ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40c8970 con 0x7ff2e40cd8e0 2023-03-03T15:20:17.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e99ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40c78d0 con 0x7ff2e4153290 2023-03-03T15:20:17.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e99ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40c6eb0 con 0x7ff2e41593c0 2023-03-03T15:20:17.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e40cd8e0 0x7ff2e41ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e27fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e37fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2e41593c0 0x7ff2e40c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e27fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e40cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59410/0 (socket says 172.21.15.102:59410) 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.838+0000 7ff2e27fc700 1 -- 172.21.15.102:0/952212464 learned_addr learned my addr 172.21.15.102:0/952212464 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e27fc700 1 -- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2e41593c0 msgr2=0x7ff2e40c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e27fc700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2e41593c0 0x7ff2e40c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e27fc700 1 -- 172.21.15.102:0/952212464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e40cd8e0 msgr2=0x7ff2e41ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e27fc700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e40cd8e0 0x7ff2e41ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e27fc700 1 -- 172.21.15.102:0/952212464 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d40094a0 con 0x7ff2e4153290 2023-03-03T15:20:17.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e27fc700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e40cf0b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80011e0 tx=0x7ff2d800ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:17.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2cbfff700 1 -- 172.21.15.102:0/952212464 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8004c80 con 0x7ff2e4153290 2023-03-03T15:20:17.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2cbfff700 1 -- 172.21.15.102:0/952212464 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d8004e00 con 0x7ff2e4153290 2023-03-03T15:20:17.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e40ccde0 con 0x7ff2e4153290 2023-03-03T15:20:17.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2cbfff700 1 -- 172.21.15.102:0/952212464 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8007310 con 0x7ff2e4153290 2023-03-03T15:20:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.839+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e415d160 con 0x7ff2e4153290 2023-03-03T15:20:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.840+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e40cd0c0 con 0x7ff2e4153290 2023-03-03T15:20:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.842+0000 7ff2cbfff700 1 -- 172.21.15.102:0/952212464 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff2d801a020 con 0x7ff2e4153290 2023-03-03T15:20:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.843+0000 7ff2cbfff700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2d00682f0 0x7ff2d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.843+0000 7ff2cbfff700 1 -- 172.21.15.102:0/952212464 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff2d8085950 con 0x7ff2e4153290 2023-03-03T15:20:17.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.843+0000 7ff2e2ffd700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2d00682f0 0x7ff2d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.844+0000 7ff2e2ffd700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2d00682f0 0x7ff2d006a7a0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7ff2d40090b0 tx=0x7ff2d4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:17.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:17.847+0000 7ff2cbfff700 1 -- 172.21.15.102:0/952212464 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff2d8052000 con 0x7ff2e4153290 2023-03-03T15:20:18.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.084+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2e404e6e0 con 0x7ff2e4153290 2023-03-03T15:20:18.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.085+0000 7ff2cbfff700 1 -- 172.21.15.102:0/952212464 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff2d8055620 con 0x7ff2e4153290 2023-03-03T15:20:18.088 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:20:18.088 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:20:18.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2d00682f0 msgr2=0x7ff2d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2d00682f0 0x7ff2d006a7a0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7ff2d40090b0 tx=0x7ff2d4008030 comp rx=0 tx=0).stop 2023-03-03T15:20:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 msgr2=0x7ff2e40cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e40cf0b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80011e0 tx=0x7ff2d800ca30 comp rx=0 tx=0).stop 2023-03-03T15:20:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 shutdown_connections 2023-03-03T15:20:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff2d00682f0 0x7ff2d006a7a0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff2e41593c0 0x7ff2e40c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e40cd8e0 0x7ff2e41ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:18.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 --2- 172.21.15.102:0/952212464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff2e4153290 0x7ff2e40cf0b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:18.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 >> 172.21.15.102:0/952212464 conn(0x7ff2e4065fb0 msgr2=0x7ff2e4150f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:18.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 shutdown_connections 2023-03-03T15:20:18.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:18.088+0000 7ff2e99ff700 1 -- 172.21.15.102:0/952212464 wait complete. 2023-03-03T15:20:18.092 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:20:18.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T15:20:28.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:28.048 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:56.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:56.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.385+0000 7ffbe85ff700 1 Processor -- start 2023-03-03T15:20:56.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.385+0000 7ffbe85ff700 1 -- start start 2023-03-03T15:20:56.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.386+0000 7ffbe85ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffbe0153290 0x7ffbe0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:56.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.386+0000 7ffbe85ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe0153660 con 0x7ffbe01593c0 2023-03-03T15:20:56.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.386+0000 7ffbe85ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe006c360 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.386+0000 7ffbe85ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe00cec40 con 0x7ffbe0153290 2023-03-03T15:20:56.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe5b9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffbe01593c0 0x7ffbe00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35316/0 (socket says 172.21.15.102:35316) 2023-03-03T15:20:56.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe5b9a700 1 -- 172.21.15.102:0/378980813 learned_addr learned my addr 172.21.15.102:0/378980813 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:20:56.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe639b700 1 --2- 172.21.15.102:0/378980813 >> v2:172.21.15.172:3300/0 conn(0x7ffbe0153290 0x7ffbe0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:56.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe5399700 1 -- 172.21.15.102:0/378980813 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216668793 0 0) 0x7ffbe0153660 con 0x7ffbe01593c0 2023-03-03T15:20:56.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe5399700 1 -- 172.21.15.102:0/378980813 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc0035b0 con 0x7ffbe01593c0 2023-03-03T15:20:56.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe5399700 1 -- 172.21.15.102:0/378980813 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842939857 0 0) 0x7ffbe006c360 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe5399700 1 -- 172.21.15.102:0/378980813 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbe0153660 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe639b700 1 -- 172.21.15.102:0/378980813 >> v1:172.21.15.172:6789/0 conn(0x7ffbe00cd8e0 legacy=0x7ffbe00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:56.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe639b700 1 -- 172.21.15.102:0/378980813 >> v1:172.21.15.102:6789/0 conn(0x7ffbe01593c0 legacy=0x7ffbe00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:56.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.387+0000 7ffbe639b700 1 -- 172.21.15.102:0/378980813 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe015cea0 con 0x7ffbe0153290 2023-03-03T15:20:56.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe639b700 1 --2- 172.21.15.102:0/378980813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe0153290 0x7ffbe0158e80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009840 tx=0x7ffbdc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2b791a9189d60f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:56.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe5399700 1 -- 172.21.15.102:0/378980813 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc011830 con 0x7ffbe0153290 2023-03-03T15:20:56.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe5399700 1 -- 172.21.15.102:0/378980813 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc011990 con 0x7ffbe0153290 2023-03-03T15:20:56.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe5399700 1 -- 172.21.15.102:0/378980813 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc011d20 con 0x7ffbe0153290 2023-03-03T15:20:56.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe85ff700 1 -- 172.21.15.102:0/378980813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe0153290 msgr2=0x7ffbe0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:56.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/378980813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe0153290 0x7ffbe0158e80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009840 tx=0x7ffbdc009b50 comp rx=0 tx=0).stop 2023-03-03T15:20:56.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe85ff700 1 -- 172.21.15.102:0/378980813 shutdown_connections 2023-03-03T15:20:56.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/378980813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe0153290 0x7ffbe0158e80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:56.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.388+0000 7ffbe85ff700 1 -- 172.21.15.102:0/378980813 >> 172.21.15.102:0/378980813 conn(0x7ffbe0065fb0 msgr2=0x7ffbe00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:56.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.389+0000 7ffbe85ff700 1 -- 172.21.15.102:0/378980813 shutdown_connections 2023-03-03T15:20:56.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.389+0000 7ffbe85ff700 1 -- 172.21.15.102:0/378980813 wait complete. 2023-03-03T15:20:56.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.391+0000 7ffbe85ff700 1 Processor -- start 2023-03-03T15:20:56.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.391+0000 7ffbe85ff700 1 -- start start 2023-03-03T15:20:56.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe85ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe00cd8e0 0x7ffbe00c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:56.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe85ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe0153290 0x7ffbe01520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:56.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe85ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe01593c0 0x7ffbe0152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:56.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe85ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe00c6d40 con 0x7ffbe01593c0 2023-03-03T15:20:56.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe85ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe0152b70 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe85ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe00c62c0 con 0x7ffbe0153290 2023-03-03T15:20:56.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe5b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe0153290 0x7ffbe01520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:56.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe639b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe00cd8e0 0x7ffbe00c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:56.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe5b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe0153290 0x7ffbe01520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:58738/0 (socket says 172.21.15.102:58738) 2023-03-03T15:20:56.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe639b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe00cd8e0 0x7ffbe00c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39344/0 (socket says 172.21.15.102:39344) 2023-03-03T15:20:56.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe5b9a700 1 -- 172.21.15.102:0/3199173935 learned_addr learned my addr 172.21.15.102:0/3199173935 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:20:56.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.392+0000 7ffbe6b9c700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe01593c0 0x7ffbe0152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:56.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe639b700 1 -- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe0153290 msgr2=0x7ffbe01520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:56.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe639b700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe0153290 0x7ffbe01520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:56.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe639b700 1 -- 172.21.15.102:0/3199173935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe01593c0 msgr2=0x7ffbe0152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:56.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe639b700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe01593c0 0x7ffbe0152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:56.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe639b700 1 -- 172.21.15.102:0/3199173935 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc0094a0 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe5b9a700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe0153290 0x7ffbe01520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:20:56.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe639b700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe00cd8e0 0x7ffbe00c8b00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc016000 tx=0x7ffbdc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:56.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbd6ffd700 1 -- 172.21.15.102:0/3199173935 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc01a6e0 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbd6ffd700 1 -- 172.21.15.102:0/3199173935 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc01a860 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe015d160 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbd6ffd700 1 -- 172.21.15.102:0/3199173935 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc01ac10 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.393+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe006c7f0 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.394+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe00ccde0 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.396+0000 7ffbd6ffd700 1 -- 172.21.15.102:0/3199173935 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffbdc02b030 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.397+0000 7ffbd6ffd700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffbc4068320 0x7ffbc406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.397+0000 7ffbd6ffd700 1 -- 172.21.15.102:0/3199173935 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ffbdc0873a0 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.397+0000 7ffbe5b9a700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffbc4068320 0x7ffbc406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:56.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.398+0000 7ffbe5b9a700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffbc4068320 0x7ffbc406a7d0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00b42f0 tx=0x7ffbd0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:56.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.400+0000 7ffbd6ffd700 1 -- 172.21.15.102:0/3199173935 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffbdc053b30 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.636+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbe004e6e0 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.637+0000 7ffbd6ffd700 1 -- 172.21.15.102:0/3199173935 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ffbdc057150 con 0x7ffbe00cd8e0 2023-03-03T15:20:56.639 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:20:56.639 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:20:56.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffbc4068320 msgr2=0x7ffbc406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:56.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffbc4068320 0x7ffbc406a7d0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00b42f0 tx=0x7ffbd0009040 comp rx=0 tx=0).stop 2023-03-03T15:20:56.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe00cd8e0 msgr2=0x7ffbe00c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:56.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe00cd8e0 0x7ffbe00c8b00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc016000 tx=0x7ffbdc004750 comp rx=0 tx=0).stop 2023-03-03T15:20:56.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 shutdown_connections 2023-03-03T15:20:56.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe01593c0 0x7ffbe0152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:56.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbe00cd8e0 0x7ffbe00c8b00 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:56.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffbc4068320 0x7ffbc406a7d0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:56.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 --2- 172.21.15.102:0/3199173935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffbe0153290 0x7ffbe01520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:56.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.639+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 >> 172.21.15.102:0/3199173935 conn(0x7ffbe0065fb0 msgr2=0x7ffbe00cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:56.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.640+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 shutdown_connections 2023-03-03T15:20:56.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:20:56.640+0000 7ffbe85ff700 1 -- 172.21.15.102:0/3199173935 wait complete. 2023-03-03T15:20:56.644 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:20:56.669 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-03-03T15:20:58.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:58.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:28.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:28.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:32.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:21:32.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.920+0000 7f2caffff700 1 Processor -- start 2023-03-03T15:21:32.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.920+0000 7f2caffff700 1 -- start start 2023-03-03T15:21:32.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.922+0000 7f2caffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2cb01564b0 0x7f2cb00cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:21:32.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.922+0000 7f2caffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2cb00cab50 0x7f2cb00cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:21:32.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.922+0000 7f2caffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb00c9ca0 con 0x7f2cb00c9610 2023-03-03T15:21:32.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caffff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb00b4920 con 0x7f2cb00cab50 2023-03-03T15:21:32.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb00b4c00 con 0x7f2cb01564b0 2023-03-03T15:21:32.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2cae7fc700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2cb00c9610 0x7f2cb00ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38738/0 (socket says 172.21.15.102:38738) 2023-03-03T15:21:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2cae7fc700 1 -- 172.21.15.102:0/3793508632 learned_addr learned my addr 172.21.15.102:0/3793508632 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:21:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caeffd700 1 --2- 172.21.15.102:0/3793508632 >> v2:172.21.15.172:3300/0 conn(0x7f2cb01564b0 0x7f2cb00cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:21:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caf7fe700 1 --2- 172.21.15.102:0/3793508632 >> v2:172.21.15.102:3300/0 conn(0x7f2cb00cab50 0x7f2cb00cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:21:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2cadffb700 1 -- 172.21.15.102:0/3793508632 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235355523 0 0) 0x7f2cb00c9ca0 con 0x7f2cb00c9610 2023-03-03T15:21:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2cadffb700 1 -- 172.21.15.102:0/3793508632 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c980035b0 con 0x7f2cb00c9610 2023-03-03T15:21:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caeffd700 1 -- 172.21.15.102:0/3793508632 >> v2:172.21.15.102:3300/0 conn(0x7f2cb00cab50 msgr2=0x7f2cb00cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:21:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caeffd700 1 --2- 172.21.15.102:0/3793508632 >> v2:172.21.15.102:3300/0 conn(0x7f2cb00cab50 0x7f2cb00cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caeffd700 1 -- 172.21.15.102:0/3793508632 >> v1:172.21.15.172:6789/0 conn(0x7f2cb00c9610 legacy=0x7f2cb00ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.923+0000 7f2caeffd700 1 -- 172.21.15.102:0/3793508632 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00b46f0 con 0x7f2cb01564b0 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.924+0000 7f2caeffd700 1 --2- 172.21.15.102:0/3793508632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 0x7f2cb00cb660 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2cb008c600 tx=0x7f2ca0009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b45db4c616b504e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.924+0000 7f2cadffb700 1 -- 172.21.15.102:0/3793508632 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0011830 con 0x7f2cb01564b0 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.924+0000 7f2cadffb700 1 -- 172.21.15.102:0/3793508632 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca0011990 con 0x7f2cb01564b0 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.924+0000 7f2cadffb700 1 -- 172.21.15.102:0/3793508632 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0011d20 con 0x7f2cb01564b0 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.924+0000 7f2caffff700 1 -- 172.21.15.102:0/3793508632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 msgr2=0x7f2cb00cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:21:32.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.924+0000 7f2caffff700 1 --2- 172.21.15.102:0/3793508632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 0x7f2cb00cb660 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2cb008c600 tx=0x7f2ca0009b20 comp rx=0 tx=0).stop 2023-03-03T15:21:32.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.925+0000 7f2caffff700 1 -- 172.21.15.102:0/3793508632 shutdown_connections 2023-03-03T15:21:32.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.925+0000 7f2caffff700 1 --2- 172.21.15.102:0/3793508632 >> v2:172.21.15.102:3300/0 conn(0x7f2cb00cab50 0x7f2cb00cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:32.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.925+0000 7f2caffff700 1 --2- 172.21.15.102:0/3793508632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 0x7f2cb00cb660 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:32.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.925+0000 7f2caffff700 1 -- 172.21.15.102:0/3793508632 >> 172.21.15.102:0/3793508632 conn(0x7f2cb0065fb0 msgr2=0x7f2cb00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:21:32.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.925+0000 7f2caffff700 1 -- 172.21.15.102:0/3793508632 shutdown_connections 2023-03-03T15:21:32.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.925+0000 7f2caffff700 1 -- 172.21.15.102:0/3793508632 wait complete. 2023-03-03T15:21:32.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.927+0000 7f2caffff700 1 Processor -- start 2023-03-03T15:21:32.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.927+0000 7f2caffff700 1 -- start start 2023-03-03T15:21:32.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2caffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cb00c9610 0x7f2cb00c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:21:32.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2caffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cb00cab50 0x7f2cb015a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:21:32.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2caffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 0x7f2cb015a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:21:32.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2caffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00c7ed0 con 0x7f2cb00cab50 2023-03-03T15:21:32.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2caffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0159a40 con 0x7f2cb01564b0 2023-03-03T15:21:32.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2caffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00c80f0 con 0x7f2cb00c9610 2023-03-03T15:21:32.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2cae7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cb00cab50 0x7f2cb015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:21:32.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2cae7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cb00cab50 0x7f2cb015a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36494/0 (socket says 172.21.15.102:36494) 2023-03-03T15:21:32.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2cae7fc700 1 -- 172.21.15.102:0/3104362595 learned_addr learned my addr 172.21.15.102:0/3104362595 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:21:32.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.928+0000 7f2caeffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cb00c9610 0x7f2cb00c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:21:32.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2caf7fe700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 0x7f2cb015a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:21:32.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2cae7fc700 1 -- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cb00c9610 msgr2=0x7f2cb00c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:21:32.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2cae7fc700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cb00c9610 0x7f2cb00c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2cae7fc700 1 -- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 msgr2=0x7f2cb015a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:21:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2cae7fc700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 0x7f2cb015a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2cae7fc700 1 -- 172.21.15.102:0/3104362595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00094a0 con 0x7f2cb00cab50 2023-03-03T15:21:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2cae7fc700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cb00cab50 0x7f2cb015a020 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c700 tx=0x7f2ca400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:21:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2c977fe700 1 -- 172.21.15.102:0/3104362595 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4005d70 con 0x7f2cb00cab50 2023-03-03T15:21:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2c977fe700 1 -- 172.21.15.102:0/3104362595 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca40070d0 con 0x7f2cb00cab50 2023-03-03T15:21:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2c977fe700 1 -- 172.21.15.102:0/3104362595 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4007460 con 0x7f2cb00cab50 2023-03-03T15:21:32.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.929+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb00b2210 con 0x7f2cb00cab50 2023-03-03T15:21:32.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.930+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb00b1f30 con 0x7f2cb00cab50 2023-03-03T15:21:32.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.931+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb0156880 con 0x7f2cb00cab50 2023-03-03T15:21:32.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.932+0000 7f2c977fe700 1 -- 172.21.15.102:0/3104362595 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ca4019030 con 0x7f2cb00cab50 2023-03-03T15:21:32.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.933+0000 7f2c977fe700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c9c0682f0 0x7f2c9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:21:32.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.933+0000 7f2c977fe700 1 -- 172.21.15.102:0/3104362595 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2ca4085140 con 0x7f2cb00cab50 2023-03-03T15:21:32.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.934+0000 7f2caeffd700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c9c0682f0 0x7f2c9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:21:32.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.934+0000 7f2caeffd700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c9c0682f0 0x7f2c9c06a7a0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00090b0 tx=0x7f2ca001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:21:32.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:32.937+0000 7f2c977fe700 1 -- 172.21.15.102:0/3104362595 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ca40517c0 con 0x7f2cb00cab50 2023-03-03T15:21:33.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.172+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cb00677f0 con 0x7f2cb00cab50 2023-03-03T15:21:33.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.173+0000 7f2c977fe700 1 -- 172.21.15.102:0/3104362595 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2ca4017090 con 0x7f2cb00cab50 2023-03-03T15:21:33.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:21:33.175 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:21:33.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.175+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c9c0682f0 msgr2=0x7f2c9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:21:33.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c9c0682f0 0x7f2c9c06a7a0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00090b0 tx=0x7f2ca001c040 comp rx=0 tx=0).stop 2023-03-03T15:21:33.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cb00cab50 msgr2=0x7f2cb015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:21:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cb00cab50 0x7f2cb015a020 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c700 tx=0x7f2ca400ca10 comp rx=0 tx=0).stop 2023-03-03T15:21:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 shutdown_connections 2023-03-03T15:21:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2c9c0682f0 0x7f2c9c06a7a0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2cb00c9610 0x7f2cb00c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cb01564b0 0x7f2cb015a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 --2- 172.21.15.102:0/3104362595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cb00cab50 0x7f2cb015a020 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:21:33.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 >> 172.21.15.102:0/3104362595 conn(0x7f2cb0065fb0 msgr2=0x7f2cb01553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:21:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 shutdown_connections 2023-03-03T15:21:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:21:33.176+0000 7f2caffff700 1 -- 172.21.15.102:0/3104362595 wait complete. 2023-03-03T15:21:33.179 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:21:33.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 61 secs before thrashing 2023-03-03T15:21:58.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:58.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:28.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:28.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:34.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:22:34.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.460+0000 7fcedabff700 1 Processor -- start 2023-03-03T15:22:34.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.461+0000 7fcedabff700 1 -- start start 2023-03-03T15:22:34.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.462+0000 7fcedabff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced40cd8e0 0x7fced40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:34.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.462+0000 7fcedabff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced4153660 con 0x7fced4153290 2023-03-03T15:22:34.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.462+0000 7fcedabff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced406c360 con 0x7fced40cd390 2023-03-03T15:22:34.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.462+0000 7fcedabff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced415cea0 con 0x7fced40cd8e0 2023-03-03T15:22:34.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.462+0000 7fced899b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fced4153290 0x7fced4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48326/0 (socket says 172.21.15.102:48326) 2023-03-03T15:22:34.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.462+0000 7fced899b700 1 -- 172.21.15.102:0/609475163 learned_addr learned my addr 172.21.15.102:0/609475163 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:22:34.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced3fff700 1 --2- 172.21.15.102:0/609475163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced40cd8e0 0x7fced40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:34.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced37fe700 1 -- 172.21.15.102:0/609475163 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227126066 0 0) 0x7fced4153660 con 0x7fced4153290 2023-03-03T15:22:34.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced37fe700 1 -- 172.21.15.102:0/609475163 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec40035b0 con 0x7fced4153290 2023-03-03T15:22:34.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced37fe700 1 -- 172.21.15.102:0/609475163 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381152444 0 0) 0x7fced406c360 con 0x7fced40cd390 2023-03-03T15:22:34.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced37fe700 1 -- 172.21.15.102:0/609475163 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced4153660 con 0x7fced40cd390 2023-03-03T15:22:34.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced3fff700 1 -- 172.21.15.102:0/609475163 >> v1:172.21.15.172:6789/0 conn(0x7fced40cd390 legacy=0x7fced40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:34.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced3fff700 1 -- 172.21.15.102:0/609475163 >> v1:172.21.15.102:6789/0 conn(0x7fced4153290 legacy=0x7fced4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:34.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced3fff700 1 -- 172.21.15.102:0/609475163 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced406c7f0 con 0x7fced40cd8e0 2023-03-03T15:22:34.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.463+0000 7fced3fff700 1 --2- 172.21.15.102:0/609475163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced40cd8e0 0x7fced40cde20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009870 tx=0x7fcec0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac09c79cbbfb8665 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:34.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fced37fe700 1 -- 172.21.15.102:0/609475163 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0011830 con 0x7fced40cd8e0 2023-03-03T15:22:34.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fced37fe700 1 -- 172.21.15.102:0/609475163 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec0011990 con 0x7fced40cd8e0 2023-03-03T15:22:34.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fced37fe700 1 -- 172.21.15.102:0/609475163 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0011d20 con 0x7fced40cd8e0 2023-03-03T15:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fcedabff700 1 -- 172.21.15.102:0/609475163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced40cd8e0 msgr2=0x7fced40cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fcedabff700 1 --2- 172.21.15.102:0/609475163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced40cd8e0 0x7fced40cde20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009870 tx=0x7fcec0009b80 comp rx=0 tx=0).stop 2023-03-03T15:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fcedabff700 1 -- 172.21.15.102:0/609475163 shutdown_connections 2023-03-03T15:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fcedabff700 1 --2- 172.21.15.102:0/609475163 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced40cd8e0 0x7fced40cde20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fcedabff700 1 -- 172.21.15.102:0/609475163 >> 172.21.15.102:0/609475163 conn(0x7fced4065fb0 msgr2=0x7fced40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fcedabff700 1 -- 172.21.15.102:0/609475163 shutdown_connections 2023-03-03T15:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.464+0000 7fcedabff700 1 -- 172.21.15.102:0/609475163 wait complete. 2023-03-03T15:22:34.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.466+0000 7fcedabff700 1 Processor -- start 2023-03-03T15:22:34.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.466+0000 7fcedabff700 1 -- start start 2023-03-03T15:22:34.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.467+0000 7fcedabff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fced40cd390 0x7fced41ef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:34.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.467+0000 7fcedabff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fced40cd8e0 0x7fced414ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:34.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.467+0000 7fcedabff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced4153290 0x7fced40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.467+0000 7fcedabff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41eeb70 con 0x7fced40cd390 2023-03-03T15:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.467+0000 7fcedabff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4150500 con 0x7fced40cd8e0 2023-03-03T15:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.467+0000 7fcedabff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced41eede0 con 0x7fced4153290 2023-03-03T15:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced899b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fced40cd390 0x7fced41ef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fced40cd8e0 0x7fced414ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced919c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced4153290 0x7fced40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fced40cd8e0 0x7fced414ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:40212/0 (socket says 172.21.15.102:40212) 2023-03-03T15:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced3fff700 1 -- 172.21.15.102:0/4159391185 learned_addr learned my addr 172.21.15.102:0/4159391185 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced919c700 1 -- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fced40cd8e0 msgr2=0x7fced414ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced919c700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fced40cd8e0 0x7fced414ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced919c700 1 -- 172.21.15.102:0/4159391185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fced40cd390 msgr2=0x7fced41ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced919c700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fced40cd390 0x7fced41ef5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced919c700 1 -- 172.21.15.102:0/4159391185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec00094a0 con 0x7fced4153290 2023-03-03T15:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced3fff700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fced40cd8e0 0x7fced414ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.468+0000 7fced899b700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fced40cd390 0x7fced41ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.469+0000 7fced919c700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced4153290 0x7fced40cf0b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00cb10 tx=0x7fcecc00ce20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.469+0000 7fced17fa700 1 -- 172.21.15.102:0/4159391185 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc003c00 con 0x7fced4153290 2023-03-03T15:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.469+0000 7fced17fa700 1 -- 172.21.15.102:0/4159391185 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc003d60 con 0x7fced4153290 2023-03-03T15:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.469+0000 7fced17fa700 1 -- 172.21.15.102:0/4159391185 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc00aa20 con 0x7fced4153290 2023-03-03T15:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.469+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced40ccde0 con 0x7fced4153290 2023-03-03T15:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.469+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced415d160 con 0x7fced4153290 2023-03-03T15:22:34.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.470+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced40cd0c0 con 0x7fced4153290 2023-03-03T15:22:34.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.471+0000 7fced17fa700 1 -- 172.21.15.102:0/4159391185 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcecc010050 con 0x7fced4153290 2023-03-03T15:22:34.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.473+0000 7fced17fa700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fceb80682f0 0x7fceb806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:34.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.473+0000 7fced17fa700 1 -- 172.21.15.102:0/4159391185 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcecc085c10 con 0x7fced4153290 2023-03-03T15:22:34.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.473+0000 7fced899b700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fceb80682f0 0x7fceb806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:34.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.474+0000 7fced899b700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fceb80682f0 0x7fceb806a7a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fcec8007fd0 tx=0x7fcec8007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:34.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.477+0000 7fced17fa700 1 -- 172.21.15.102:0/4159391185 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcecc052290 con 0x7fced4153290 2023-03-03T15:22:34.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.710+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fced41ef2d0 con 0x7fced4153290 2023-03-03T15:22:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.712+0000 7fced17fa700 1 -- 172.21.15.102:0/4159391185 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcecc0558b0 con 0x7fced4153290 2023-03-03T15:22:34.714 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:22:34.714 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:22:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fceb80682f0 msgr2=0x7fceb806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fceb80682f0 0x7fceb806a7a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fcec8007fd0 tx=0x7fcec8007e00 comp rx=0 tx=0).stop 2023-03-03T15:22:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced4153290 msgr2=0x7fced40cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced4153290 0x7fced40cf0b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00cb10 tx=0x7fcecc00ce20 comp rx=0 tx=0).stop 2023-03-03T15:22:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 shutdown_connections 2023-03-03T15:22:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fceb80682f0 0x7fceb806a7a0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fced4153290 0x7fced40cf0b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fced40cd390 0x7fced41ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 --2- 172.21.15.102:0/4159391185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fced40cd8e0 0x7fced414ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 >> 172.21.15.102:0/4159391185 conn(0x7fced4065fb0 msgr2=0x7fced4151430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 shutdown_connections 2023-03-03T15:22:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:22:34.714+0000 7fcedabff700 1 -- 172.21.15.102:0/4159391185 wait complete. 2023-03-03T15:22:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:22:34.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 63 secs before thrashing 2023-03-03T15:22:58.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:58.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:28.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:28.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:37.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:23:37.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.993+0000 7ff9f659d700 1 Processor -- start 2023-03-03T15:23:37.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.993+0000 7ff9f659d700 1 -- start start 2023-03-03T15:23:37.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f659d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff9f00ced00 0x7ff9f00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:37.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f659d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f00cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:37.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f659d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f00c3d20 con 0x7ff9f00b3970 2023-03-03T15:23:37.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f659d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f00c4930 con 0x7ff9f00ced00 2023-03-03T15:23:37.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f659d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f00cf610 con 0x7ff9f00c41b0 2023-03-03T15:23:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f5d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f00cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f559b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff9f00ced00 0x7ff9f00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:37.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f4d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff9f00b3970 0x7ff9f0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:54094/0 (socket says 172.21.15.102:54094) 2023-03-03T15:23:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.995+0000 7ff9f4d9a700 1 -- 172.21.15.102:0/2486398958 learned_addr learned my addr 172.21.15.102:0/2486398958 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:23:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9f5d9c700 1 -- 172.21.15.102:0/2486398958 >> v2:172.21.15.172:3300/0 conn(0x7ff9f00ced00 msgr2=0x7ff9f00c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:23:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9f5d9c700 1 --2- 172.21.15.102:0/2486398958 >> v2:172.21.15.172:3300/0 conn(0x7ff9f00ced00 0x7ff9f00c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9e7fff700 1 -- 172.21.15.102:0/2486398958 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876516075 0 0) 0x7ff9f00c3d20 con 0x7ff9f00b3970 2023-03-03T15:23:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9f5d9c700 1 -- 172.21.15.102:0/2486398958 >> v1:172.21.15.172:6789/0 conn(0x7ff9f00b3970 legacy=0x7ff9f0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9f5d9c700 1 -- 172.21.15.102:0/2486398958 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f015cba0 con 0x7ff9f00c41b0 2023-03-03T15:23:37.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9f5d9c700 1 --2- 172.21.15.102:0/2486398958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f00cf0d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8009d90 tx=0x7ff9e800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=209bc9b1a53ec68e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9e7fff700 1 -- 172.21.15.102:0/2486398958 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e800e070 con 0x7ff9f00c41b0 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9e7fff700 1 -- 172.21.15.102:0/2486398958 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e800bb20 con 0x7ff9f00c41b0 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.996+0000 7ff9e7fff700 1 -- 172.21.15.102:0/2486398958 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8012430 con 0x7ff9f00c41b0 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 -- 172.21.15.102:0/2486398958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 msgr2=0x7ff9f00cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 --2- 172.21.15.102:0/2486398958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f00cf0d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8009d90 tx=0x7ff9e800b330 comp rx=0 tx=0).stop 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 -- 172.21.15.102:0/2486398958 shutdown_connections 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 --2- 172.21.15.102:0/2486398958 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f00cf0d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 --2- 172.21.15.102:0/2486398958 >> v2:172.21.15.172:3300/0 conn(0x7ff9f00ced00 0x7ff9f00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 -- 172.21.15.102:0/2486398958 >> 172.21.15.102:0/2486398958 conn(0x7ff9f008b1c0 msgr2=0x7ff9f0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 -- 172.21.15.102:0/2486398958 shutdown_connections 2023-03-03T15:23:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.997+0000 7ff9f659d700 1 -- 172.21.15.102:0/2486398958 wait complete. 2023-03-03T15:23:38.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.999+0000 7ff9f659d700 1 Processor -- start 2023-03-03T15:23:38.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:37.999+0000 7ff9f659d700 1 -- start start 2023-03-03T15:23:38.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f659d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9f00b3970 0x7ff9f014e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:38.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f659d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f014e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f659d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9f00ced00 0x7ff9f00c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f659d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f01494e0 con 0x7ff9f00b3970 2023-03-03T15:23:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f659d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f015afd0 con 0x7ff9f00ced00 2023-03-03T15:23:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f659d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f01520a0 con 0x7ff9f00c41b0 2023-03-03T15:23:38.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f559b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9f00b3970 0x7ff9f014e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f559b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9f00b3970 0x7ff9f014e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47392/0 (socket says 172.21.15.102:47392) 2023-03-03T15:23:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f559b700 1 -- 172.21.15.102:0/3072845219 learned_addr learned my addr 172.21.15.102:0/3072845219 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:23:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f4d9a700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f014e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.001+0000 7ff9f5d9c700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9f00ced00 0x7ff9f00c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9f559b700 1 -- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 msgr2=0x7ff9f014e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9f559b700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f014e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9f559b700 1 -- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9f00ced00 msgr2=0x7ff9f00c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9f559b700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9f00ced00 0x7ff9f00c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9f559b700 1 -- 172.21.15.102:0/3072845219 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e8009a70 con 0x7ff9f00b3970 2023-03-03T15:23:38.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9f559b700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9f00b3970 0x7ff9f014e0e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00e9b0 tx=0x7ff9ec00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9e5ffb700 1 -- 172.21.15.102:0/3072845219 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec009710 con 0x7ff9f00b3970 2023-03-03T15:23:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9e5ffb700 1 -- 172.21.15.102:0/3072845219 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec00ee50 con 0x7ff9f00b3970 2023-03-03T15:23:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9e5ffb700 1 -- 172.21.15.102:0/3072845219 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec00ca50 con 0x7ff9f00b3970 2023-03-03T15:23:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.002+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f006bf20 con 0x7ff9f00b3970 2023-03-03T15:23:38.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.003+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f015ce80 con 0x7ff9f00b3970 2023-03-03T15:23:38.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.004+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f006c200 con 0x7ff9f00b3970 2023-03-03T15:23:38.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.005+0000 7ff9e5ffb700 1 -- 172.21.15.102:0/3072845219 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff9ec019030 con 0x7ff9f00b3970 2023-03-03T15:23:38.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.007+0000 7ff9e5ffb700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff9d40682f0 0x7ff9d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:38.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.007+0000 7ff9e5ffb700 1 -- 172.21.15.102:0/3072845219 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff9ec085c20 con 0x7ff9f00b3970 2023-03-03T15:23:38.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.007+0000 7ff9f4d9a700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff9d40682f0 0x7ff9d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:38.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.008+0000 7ff9f4d9a700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff9d40682f0 0x7ff9d406a7a0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff9f01498b0 tx=0x7ff9e0007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:38.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.011+0000 7ff9e5ffb700 1 -- 172.21.15.102:0/3072845219 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff9ec052330 con 0x7ff9f00b3970 2023-03-03T15:23:38.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.249+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9f00c6130 con 0x7ff9f00b3970 2023-03-03T15:23:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.249+0000 7ff9e5ffb700 1 -- 172.21.15.102:0/3072845219 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff9ec055950 con 0x7ff9f00b3970 2023-03-03T15:23:38.251 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:23:38.251 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:23:38.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff9d40682f0 msgr2=0x7ff9d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:38.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff9d40682f0 0x7ff9d406a7a0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff9f01498b0 tx=0x7ff9e0007e70 comp rx=0 tx=0).stop 2023-03-03T15:23:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9f00b3970 msgr2=0x7ff9f014e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9f00b3970 0x7ff9f014e0e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00e9b0 tx=0x7ff9ec00ecc0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 shutdown_connections 2023-03-03T15:23:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9f00ced00 0x7ff9f00c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9f00b3970 0x7ff9f014e0e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff9d40682f0 0x7ff9d406a7a0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 --2- 172.21.15.102:0/3072845219 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff9f00c41b0 0x7ff9f014e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:38.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 >> 172.21.15.102:0/3072845219 conn(0x7ff9f008b1c0 msgr2=0x7ff9f00cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:38.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 shutdown_connections 2023-03-03T15:23:38.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:23:38.252+0000 7ff9f659d700 1 -- 172.21.15.102:0/3072845219 wait complete. 2023-03-03T15:23:38.255 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:23:38.280 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T15:23:58.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:59.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:29.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:29.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:59.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:59.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:29.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:29.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:33.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:25:33.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.538+0000 7efca11ff700 1 Processor -- start 2023-03-03T15:25:33.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.538+0000 7efca11ff700 1 -- start start 2023-03-03T15:25:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.539+0000 7efca11ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c153290 0x7efc9c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.539+0000 7efca11ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efc9c0cd8e0 0x7efc9c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.539+0000 7efca11ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c153660 con 0x7efc9c1593c0 2023-03-03T15:25:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.539+0000 7efca11ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9c06c360 con 0x7efc9c0cd8e0 2023-03-03T15:25:33.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.539+0000 7efca11ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c15cea0 con 0x7efc9c153290 2023-03-03T15:25:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9a59c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc9c1593c0 0x7efc9c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54340/0 (socket says 172.21.15.102:54340) 2023-03-03T15:25:33.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9a59c700 1 -- 172.21.15.102:0/3204076221 learned_addr learned my addr 172.21.15.102:0/3204076221 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:25:33.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9b59e700 1 --2- 172.21.15.102:0/3204076221 >> v2:172.21.15.172:3300/0 conn(0x7efc9c0cd8e0 0x7efc9c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:33.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9ad9d700 1 --2- 172.21.15.102:0/3204076221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c153290 0x7efc9c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc99d9b700 1 -- 172.21.15.102:0/3204076221 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048185973 0 0) 0x7efc9c153660 con 0x7efc9c1593c0 2023-03-03T15:25:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc99d9b700 1 -- 172.21.15.102:0/3204076221 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc840035b0 con 0x7efc9c1593c0 2023-03-03T15:25:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9ad9d700 1 -- 172.21.15.102:0/3204076221 >> v2:172.21.15.172:3300/0 conn(0x7efc9c0cd8e0 msgr2=0x7efc9c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:25:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9ad9d700 1 --2- 172.21.15.102:0/3204076221 >> v2:172.21.15.172:3300/0 conn(0x7efc9c0cd8e0 0x7efc9c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9ad9d700 1 -- 172.21.15.102:0/3204076221 >> v1:172.21.15.102:6789/0 conn(0x7efc9c1593c0 legacy=0x7efc9c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9ad9d700 1 -- 172.21.15.102:0/3204076221 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c06c7f0 con 0x7efc9c153290 2023-03-03T15:25:33.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.540+0000 7efc9b59e700 1 --2- 172.21.15.102:0/3204076221 >> v2:172.21.15.172:3300/0 conn(0x7efc9c0cd8e0 0x7efc9c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:25:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.541+0000 7efc9ad9d700 1 --2- 172.21.15.102:0/3204076221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c153290 0x7efc9c158e80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009840 tx=0x7efc8c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10345c92fa824dc4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:25:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.541+0000 7efc99d9b700 1 -- 172.21.15.102:0/3204076221 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c011830 con 0x7efc9c153290 2023-03-03T15:25:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.541+0000 7efc99d9b700 1 -- 172.21.15.102:0/3204076221 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c011990 con 0x7efc9c153290 2023-03-03T15:25:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.541+0000 7efc99d9b700 1 -- 172.21.15.102:0/3204076221 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c011d20 con 0x7efc9c153290 2023-03-03T15:25:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.541+0000 7efca11ff700 1 -- 172.21.15.102:0/3204076221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c153290 msgr2=0x7efc9c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.541+0000 7efca11ff700 1 --2- 172.21.15.102:0/3204076221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c153290 0x7efc9c158e80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009840 tx=0x7efc8c009b50 comp rx=0 tx=0).stop 2023-03-03T15:25:33.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.542+0000 7efca11ff700 1 -- 172.21.15.102:0/3204076221 shutdown_connections 2023-03-03T15:25:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.542+0000 7efca11ff700 1 --2- 172.21.15.102:0/3204076221 >> v2:172.21.15.172:3300/0 conn(0x7efc9c0cd8e0 0x7efc9c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.542+0000 7efca11ff700 1 --2- 172.21.15.102:0/3204076221 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c153290 0x7efc9c158e80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.542+0000 7efca11ff700 1 -- 172.21.15.102:0/3204076221 >> 172.21.15.102:0/3204076221 conn(0x7efc9c065fb0 msgr2=0x7efc9c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:25:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.542+0000 7efca11ff700 1 -- 172.21.15.102:0/3204076221 shutdown_connections 2023-03-03T15:25:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.542+0000 7efca11ff700 1 -- 172.21.15.102:0/3204076221 wait complete. 2023-03-03T15:25:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.544+0000 7efca11ff700 1 Processor -- start 2023-03-03T15:25:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.544+0000 7efca11ff700 1 -- start start 2023-03-03T15:25:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efca11ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc9c0cd8e0 0x7efc9c14e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efca11ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc9c153290 0x7efc9c1ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efca11ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c1593c0 0x7efc9c1eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efca11ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0c6b50 con 0x7efc9c153290 2023-03-03T15:25:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efca11ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c14cf60 con 0x7efc9c0cd8e0 2023-03-03T15:25:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efca11ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0c6580 con 0x7efc9c1593c0 2023-03-03T15:25:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efc9a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc9c153290 0x7efc9c1ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efc9a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc9c153290 0x7efc9c1ee730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54140/0 (socket says 172.21.15.102:54140) 2023-03-03T15:25:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efc9ad9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc9c0cd8e0 0x7efc9c14e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:33.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efc9a59c700 1 -- 172.21.15.102:0/1658968825 learned_addr learned my addr 172.21.15.102:0/1658968825 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:25:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.545+0000 7efc9b59e700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c1593c0 0x7efc9c1eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc9a59c700 1 -- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c1593c0 msgr2=0x7efc9c1eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc9a59c700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c1593c0 0x7efc9c1eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc9a59c700 1 -- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc9c0cd8e0 msgr2=0x7efc9c14e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc9a59c700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc9c0cd8e0 0x7efc9c14e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc9a59c700 1 -- 172.21.15.102:0/1658968825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc94009a70 con 0x7efc9c153290 2023-03-03T15:25:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc9a59c700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc9c153290 0x7efc9c1ee730 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7efc9000c700 tx=0x7efc9000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:25:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc837fe700 1 -- 172.21.15.102:0/1658968825 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90005d70 con 0x7efc9c153290 2023-03-03T15:25:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc837fe700 1 -- 172.21.15.102:0/1658968825 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc900070d0 con 0x7efc9c153290 2023-03-03T15:25:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efc837fe700 1 -- 172.21.15.102:0/1658968825 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90007460 con 0x7efc9c153290 2023-03-03T15:25:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.546+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c0094a0 con 0x7efc9c153290 2023-03-03T15:25:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.547+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c0ccde0 con 0x7efc9c153290 2023-03-03T15:25:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.548+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c15d160 con 0x7efc9c153290 2023-03-03T15:25:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.549+0000 7efc837fe700 1 -- 172.21.15.102:0/1658968825 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc90019030 con 0x7efc9c153290 2023-03-03T15:25:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.550+0000 7efc837fe700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efc880682f0 0x7efc8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.550+0000 7efc837fe700 1 -- 172.21.15.102:0/1658968825 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7efc900850d0 con 0x7efc9c153290 2023-03-03T15:25:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.550+0000 7efc9ad9d700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efc880682f0 0x7efc8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:33.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.551+0000 7efc9ad9d700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efc880682f0 0x7efc8806a7a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000c00 tx=0x7efc8c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:25:33.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.554+0000 7efc837fe700 1 -- 172.21.15.102:0/1658968825 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc900517e0 con 0x7efc9c153290 2023-03-03T15:25:33.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.787+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc9c0cd0c0 con 0x7efc9c153290 2023-03-03T15:25:33.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.788+0000 7efc837fe700 1 -- 172.21.15.102:0/1658968825 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7efc90054e00 con 0x7efc9c153290 2023-03-03T15:25:33.790 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:25:33.791 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:25:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efc880682f0 msgr2=0x7efc8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efc880682f0 0x7efc8806a7a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000c00 tx=0x7efc8c008030 comp rx=0 tx=0).stop 2023-03-03T15:25:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc9c153290 msgr2=0x7efc9c1ee730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc9c153290 0x7efc9c1ee730 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7efc9000c700 tx=0x7efc9000ca10 comp rx=0 tx=0).stop 2023-03-03T15:25:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 shutdown_connections 2023-03-03T15:25:33.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7efc880682f0 0x7efc8806a7a0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7efc9c1593c0 0x7efc9c1eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc9c0cd8e0 0x7efc9c14e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 --2- 172.21.15.102:0/1658968825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc9c153290 0x7efc9c1ee730 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 >> 172.21.15.102:0/1658968825 conn(0x7efc9c065fb0 msgr2=0x7efc9c0cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:25:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 shutdown_connections 2023-03-03T15:25:33.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:25:33.790+0000 7efca11ff700 1 -- 172.21.15.102:0/1658968825 wait complete. 2023-03-03T15:25:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:25:33.821 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 50 secs before thrashing 2023-03-03T15:25:59.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:59.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:23.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:26:24.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.071+0000 7fc5f7bff700 1 Processor -- start 2023-03-03T15:26:24.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.072+0000 7fc5f7bff700 1 -- start start 2023-03-03T15:26:24.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.073+0000 7fc5f7bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:24.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.073+0000 7fc5f7bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f0153660 con 0x7fc5f0153290 2023-03-03T15:26:24.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.073+0000 7fc5f7bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f006c360 con 0x7fc5f00cd390 2023-03-03T15:26:24.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.073+0000 7fc5f7bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f015cea0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.073+0000 7fc5f599b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc5f0153290 0x7fc5f0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45782/0 (socket says 172.21.15.102:45782) 2023-03-03T15:26:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.073+0000 7fc5f599b700 1 -- 172.21.15.102:0/3057369980 learned_addr learned my addr 172.21.15.102:0/3057369980 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:26:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.073+0000 7fc5f519a700 1 --2- 172.21.15.102:0/3057369980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.074+0000 7fc5f4999700 1 -- 172.21.15.102:0/3057369980 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202308967 0 0) 0x7fc5f0153660 con 0x7fc5f0153290 2023-03-03T15:26:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.074+0000 7fc5f4999700 1 -- 172.21.15.102:0/3057369980 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5dc0035b0 con 0x7fc5f0153290 2023-03-03T15:26:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.074+0000 7fc5f4999700 1 -- 172.21.15.102:0/3057369980 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060131493 0 0) 0x7fc5f006c360 con 0x7fc5f00cd390 2023-03-03T15:26:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.074+0000 7fc5f519a700 1 -- 172.21.15.102:0/3057369980 >> v1:172.21.15.172:6789/0 conn(0x7fc5f00cd390 legacy=0x7fc5f00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.074+0000 7fc5f519a700 1 -- 172.21.15.102:0/3057369980 >> v1:172.21.15.102:6789/0 conn(0x7fc5f0153290 legacy=0x7fc5f0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.074+0000 7fc5f519a700 1 -- 172.21.15.102:0/3057369980 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f006c7f0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.074+0000 7fc5f519a700 1 --2- 172.21.15.102:0/3057369980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f00cde20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009870 tx=0x7fc5e0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26fc532863b4071 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f4999700 1 -- 172.21.15.102:0/3057369980 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0011830 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f4999700 1 -- 172.21.15.102:0/3057369980 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e0011990 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f4999700 1 -- 172.21.15.102:0/3057369980 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0011d20 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3057369980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 msgr2=0x7fc5f00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3057369980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f00cde20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009870 tx=0x7fc5e0009b80 comp rx=0 tx=0).stop 2023-03-03T15:26:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3057369980 shutdown_connections 2023-03-03T15:26:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3057369980 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f00cde20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3057369980 >> 172.21.15.102:0/3057369980 conn(0x7fc5f0065fb0 msgr2=0x7fc5f00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:26:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3057369980 shutdown_connections 2023-03-03T15:26:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.075+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3057369980 wait complete. 2023-03-03T15:26:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.077+0000 7fc5f7bff700 1 Processor -- start 2023-03-03T15:26:24.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.078+0000 7fc5f7bff700 1 -- start start 2023-03-03T15:26:24.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f7bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5f00cd390 0x7fc5f00b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f7bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f01f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f7bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5f0153290 0x7fc5f01f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f7bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0151aa0 con 0x7fc5f0153290 2023-03-03T15:26:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f7bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f01f2580 con 0x7fc5f00cd390 2023-03-03T15:26:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f7bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f00b2340 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f599b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5f00cd390 0x7fc5f00b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f519a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f01f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f599b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5f00cd390 0x7fc5f00b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46364/0 (socket says 172.21.15.102:46364) 2023-03-03T15:26:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f519a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f01f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39526/0 (socket says 172.21.15.102:39526) 2023-03-03T15:26:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f599b700 1 -- 172.21.15.102:0/3211913455 learned_addr learned my addr 172.21.15.102:0/3211913455 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:26:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f619c700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5f0153290 0x7fc5f01f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f519a700 1 -- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5f00cd390 msgr2=0x7fc5f00b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f519a700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5f00cd390 0x7fc5f00b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f519a700 1 -- 172.21.15.102:0/3211913455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5f0153290 msgr2=0x7fc5f01f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f519a700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5f0153290 0x7fc5f01f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.079+0000 7fc5f519a700 1 -- 172.21.15.102:0/3211913455 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00094a0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.080+0000 7fc5f519a700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f01f1a20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0016000 tx=0x7fc5e0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.080+0000 7fc5e67fc700 1 -- 172.21.15.102:0/3211913455 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e001a7e0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.080+0000 7fc5e67fc700 1 -- 172.21.15.102:0/3211913455 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e001a960 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.080+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f00ccde0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.080+0000 7fc5e67fc700 1 -- 172.21.15.102:0/3211913455 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e001ad10 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.080+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f015d160 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.081+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f00cd0c0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.083+0000 7fc5e67fc700 1 -- 172.21.15.102:0/3211913455 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5e000f050 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.084+0000 7fc5e67fc700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5d4068320 0x7fc5d406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.084+0000 7fc5e67fc700 1 -- 172.21.15.102:0/3211913455 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc5e0087330 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.084+0000 7fc5f599b700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5d4068320 0x7fc5d406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.085+0000 7fc5f599b700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5d4068320 0x7fc5d406a7d0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec005910 tx=0x7fc5ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:24.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.087+0000 7fc5e67fc700 1 -- 172.21.15.102:0/3211913455 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5e0053ac0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.322+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5f0151c10 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.323+0000 7fc5e67fc700 1 -- 172.21.15.102:0/3211913455 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc5e00570e0 con 0x7fc5f00cd8e0 2023-03-03T15:26:24.325 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:26:24.328 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:26:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5d4068320 msgr2=0x7fc5d406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5d4068320 0x7fc5d406a7d0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec005910 tx=0x7fc5ec009040 comp rx=0 tx=0).stop 2023-03-03T15:26:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 msgr2=0x7fc5f01f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f01f1a20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0016000 tx=0x7fc5e0004750 comp rx=0 tx=0).stop 2023-03-03T15:26:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 shutdown_connections 2023-03-03T15:26:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5f0153290 0x7fc5f01f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:24.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5f00cd390 0x7fc5f00b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:24.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc5d4068320 0x7fc5d406a7d0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:24.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 --2- 172.21.15.102:0/3211913455 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc5f00cd8e0 0x7fc5f01f1a20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:24.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 >> 172.21.15.102:0/3211913455 conn(0x7fc5f0065fb0 msgr2=0x7fc5f0158910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:26:24.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.325+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 shutdown_connections 2023-03-03T15:26:24.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:26:24.326+0000 7fc5f7bff700 1 -- 172.21.15.102:0/3211913455 wait complete. 2023-03-03T15:26:24.331 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:26:24.355 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T15:26:29.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:29.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:00.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:00.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:16.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:27:16.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.617+0000 7f957b1ff700 1 Processor -- start 2023-03-03T15:27:16.620 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.618+0000 7f957b1ff700 1 -- start start 2023-03-03T15:27:16.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.619+0000 7f957b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:16.621 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.619+0000 7f957b1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9574153660 con 0x7f95740cd8e0 2023-03-03T15:27:16.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.619+0000 7f957b1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957406c360 con 0x7f9574153290 2023-03-03T15:27:16.622 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.619+0000 7f957b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957415cea0 con 0x7f95740cd390 2023-03-03T15:27:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.619+0000 7f957979c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.619+0000 7f9578f9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9574153290 0x7f9574158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59558/0 (socket says 172.21.15.102:59558) 2023-03-03T15:27:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.619+0000 7f9578f9b700 1 -- 172.21.15.102:0/1316567760 learned_addr learned my addr 172.21.15.102:0/1316567760 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:27:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.620+0000 7f95737fe700 1 -- 172.21.15.102:0/1316567760 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311711702 0 0) 0x7f957406c360 con 0x7f9574153290 2023-03-03T15:27:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.620+0000 7f957979c700 1 -- 172.21.15.102:0/1316567760 >> v1:172.21.15.172:6789/0 conn(0x7f9574153290 legacy=0x7f9574158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.620+0000 7f957979c700 1 -- 172.21.15.102:0/1316567760 >> v1:172.21.15.102:6789/0 conn(0x7f95740cd8e0 legacy=0x7f95740cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.620+0000 7f957979c700 1 -- 172.21.15.102:0/1316567760 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957406c7f0 con 0x7f95740cd390 2023-03-03T15:27:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.620+0000 7f957979c700 1 --2- 172.21.15.102:0/1316567760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740ce530 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f956c009dc0 tx=0x7f956c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d50acab01af3337 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:27:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.620+0000 7f95737fe700 1 -- 172.21.15.102:0/1316567760 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c00e070 con 0x7f95740cd390 2023-03-03T15:27:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f95737fe700 1 -- 172.21.15.102:0/1316567760 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c00bbd0 con 0x7f95740cd390 2023-03-03T15:27:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f95737fe700 1 -- 172.21.15.102:0/1316567760 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c012430 con 0x7f95740cd390 2023-03-03T15:27:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f957b1ff700 1 -- 172.21.15.102:0/1316567760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 msgr2=0x7f95740ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f957b1ff700 1 --2- 172.21.15.102:0/1316567760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740ce530 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f956c009dc0 tx=0x7f956c00b330 comp rx=0 tx=0).stop 2023-03-03T15:27:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f957b1ff700 1 -- 172.21.15.102:0/1316567760 shutdown_connections 2023-03-03T15:27:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f957b1ff700 1 --2- 172.21.15.102:0/1316567760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740ce530 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f957b1ff700 1 -- 172.21.15.102:0/1316567760 >> 172.21.15.102:0/1316567760 conn(0x7f9574065fb0 msgr2=0x7f95740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:27:16.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f957b1ff700 1 -- 172.21.15.102:0/1316567760 shutdown_connections 2023-03-03T15:27:16.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.621+0000 7f957b1ff700 1 -- 172.21.15.102:0/1316567760 wait complete. 2023-03-03T15:27:16.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.623+0000 7f957b1ff700 1 Processor -- start 2023-03-03T15:27:16.626 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.623+0000 7f957b1ff700 1 -- start start 2023-03-03T15:27:16.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.624+0000 7f957b1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.624+0000 7f957b1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95740cd8e0 0x7f95740c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.624+0000 7f957b1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9574153290 0x7f95740c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.624+0000 7f957b1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9574156160 con 0x7f95740cd8e0 2023-03-03T15:27:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.624+0000 7f957b1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95740c6140 con 0x7f9574153290 2023-03-03T15:27:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.624+0000 7f957b1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95740c6420 con 0x7f95740cd390 2023-03-03T15:27:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95740cd8e0 0x7f95740c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95740cd8e0 0x7f95740c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54458/0 (socket says 172.21.15.102:54458) 2023-03-03T15:27:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 -- 172.21.15.102:0/4241672196 learned_addr learned my addr 172.21.15.102:0/4241672196 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:27:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f957979c700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9574153290 0x7f95740c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9578f9b700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 -- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 msgr2=0x7f95740c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 -- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9574153290 msgr2=0x7f95740c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9574153290 0x7f95740c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 -- 172.21.15.102:0/4241672196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c009a70 con 0x7f95740cd8e0 2023-03-03T15:27:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.625+0000 7f9573fff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95740cd8e0 0x7f95740c68b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f956000c4e0 tx=0x7f956000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:27:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.626+0000 7f95717fa700 1 -- 172.21.15.102:0/4241672196 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95600052d0 con 0x7f95740cd8e0 2023-03-03T15:27:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.626+0000 7f95717fa700 1 -- 172.21.15.102:0/4241672196 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560005430 con 0x7f95740cd8e0 2023-03-03T15:27:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.626+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95740ccde0 con 0x7f95740cd8e0 2023-03-03T15:27:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.626+0000 7f95717fa700 1 -- 172.21.15.102:0/4241672196 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95600038c0 con 0x7f95740cd8e0 2023-03-03T15:27:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.626+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957415d160 con 0x7f95740cd8e0 2023-03-03T15:27:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.627+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95740cd0c0 con 0x7f95740cd8e0 2023-03-03T15:27:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.628+0000 7f95717fa700 1 -- 172.21.15.102:0/4241672196 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9560019030 con 0x7f95740cd8e0 2023-03-03T15:27:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.629+0000 7f95717fa700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9558068320 0x7f955806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.630+0000 7f95717fa700 1 -- 172.21.15.102:0/4241672196 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9560084ff0 con 0x7f95740cd8e0 2023-03-03T15:27:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.630+0000 7f9578f9b700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9558068320 0x7f955806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:16.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.630+0000 7f9578f9b700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9558068320 0x7f955806a7d0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f9568007fd0 tx=0x7f9568007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:27:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.634+0000 7f95717fa700 1 -- 172.21.15.102:0/4241672196 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95600516a0 con 0x7f95740cd8e0 2023-03-03T15:27:16.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.867+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f957404e6e0 con 0x7f95740cd8e0 2023-03-03T15:27:16.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.868+0000 7f95717fa700 1 -- 172.21.15.102:0/4241672196 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9560017090 con 0x7f95740cd8e0 2023-03-03T15:27:16.871 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:27:16.871 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:27:16.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9558068320 msgr2=0x7f955806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:16.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9558068320 0x7f955806a7d0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f9568007fd0 tx=0x7f9568007d80 comp rx=0 tx=0).stop 2023-03-03T15:27:16.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95740cd8e0 msgr2=0x7f95740c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:16.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95740cd8e0 0x7f95740c68b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f956000c4e0 tx=0x7f956000c7f0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 shutdown_connections 2023-03-03T15:27:16.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9558068320 0x7f955806a7d0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f95740cd390 0x7f95740c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9574153290 0x7f95740c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 --2- 172.21.15.102:0/4241672196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95740cd8e0 0x7f95740c68b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:16.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 >> 172.21.15.102:0/4241672196 conn(0x7f9574065fb0 msgr2=0x7f9574157660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:27:16.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 shutdown_connections 2023-03-03T15:27:16.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:27:16.871+0000 7f957b1ff700 1 -- 172.21.15.102:0/4241672196 wait complete. 2023-03-03T15:27:16.875 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:27:16.901 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-03-03T15:27:30.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:30.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:00.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:00.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:02.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:28:03.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.149+0000 7f1969bff700 1 Processor -- start 2023-03-03T15:28:03.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.149+0000 7f1969bff700 1 -- start start 2023-03-03T15:28:03.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f1969bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1964152b70 0x7f19640c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:03.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f1969bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f196406ccc0 0x7f19640c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:03.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f1969bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640c7e10 con 0x7f196406d1d0 2023-03-03T15:28:03.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f1969bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1964072460 con 0x7f196406ccc0 2023-03-03T15:28:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f1969bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196406f180 con 0x7f1964152b70 2023-03-03T15:28:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f19637fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1964152b70 0x7f19640c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f195bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f196406ccc0 0x7f19640c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f19637fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1964152b70 0x7f19640c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42198/0 (socket says 172.21.15.102:42198) 2023-03-03T15:28:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f19637fe700 1 -- 172.21.15.102:0/1748320955 learned_addr learned my addr 172.21.15.102:0/1748320955 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:28:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.151+0000 7f1963fff700 1 --1- 172.21.15.102:0/1748320955 >> v1:172.21.15.102:6789/0 conn(0x7f196406d1d0 0x7f196415c190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43680/0 (socket says 172.21.15.102:43680) 2023-03-03T15:28:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f19637fe700 1 -- 172.21.15.102:0/1748320955 >> v2:172.21.15.172:3300/0 conn(0x7f196406ccc0 msgr2=0x7f19640c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:28:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f19637fe700 1 --2- 172.21.15.102:0/1748320955 >> v2:172.21.15.172:3300/0 conn(0x7f196406ccc0 0x7f19640c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f19637fe700 1 -- 172.21.15.102:0/1748320955 >> v1:172.21.15.102:6789/0 conn(0x7f196406d1d0 legacy=0x7f196415c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:03.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f19637fe700 1 -- 172.21.15.102:0/1748320955 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640c5a40 con 0x7f1964152b70 2023-03-03T15:28:03.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f19637fe700 1 --2- 172.21.15.102:0/1748320955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1964152b70 0x7f19640c5ed0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f19540098f0 tx=0x7f1954009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c9dc5530194e1ff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:03.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f1962ffd700 1 -- 172.21.15.102:0/1748320955 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954011830 con 0x7f1964152b70 2023-03-03T15:28:03.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f1962ffd700 1 -- 172.21.15.102:0/1748320955 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954011990 con 0x7f1964152b70 2023-03-03T15:28:03.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.152+0000 7f1962ffd700 1 -- 172.21.15.102:0/1748320955 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954011d20 con 0x7f1964152b70 2023-03-03T15:28:03.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 -- 172.21.15.102:0/1748320955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1964152b70 msgr2=0x7f19640c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:03.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 --2- 172.21.15.102:0/1748320955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1964152b70 0x7f19640c5ed0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f19540098f0 tx=0x7f1954009c00 comp rx=0 tx=0).stop 2023-03-03T15:28:03.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 -- 172.21.15.102:0/1748320955 shutdown_connections 2023-03-03T15:28:03.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 --2- 172.21.15.102:0/1748320955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1964152b70 0x7f19640c5ed0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 --2- 172.21.15.102:0/1748320955 >> v2:172.21.15.172:3300/0 conn(0x7f196406ccc0 0x7f19640c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 -- 172.21.15.102:0/1748320955 >> 172.21.15.102:0/1748320955 conn(0x7f196406af90 msgr2=0x7f196406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:03.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 -- 172.21.15.102:0/1748320955 shutdown_connections 2023-03-03T15:28:03.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.153+0000 7f1969bff700 1 -- 172.21.15.102:0/1748320955 wait complete. 2023-03-03T15:28:03.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.155+0000 7f1969bff700 1 Processor -- start 2023-03-03T15:28:03.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.155+0000 7f1969bff700 1 -- start start 2023-03-03T15:28:03.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.156+0000 7f1969bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196406ccc0 0x7f19641482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:03.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.156+0000 7f1969bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196406d1d0 0x7f1964151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.156+0000 7f1969bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1964152b70 0x7f19641f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.156+0000 7f1969bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19641523b0 con 0x7f1964152b70 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.156+0000 7f1969bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964145610 con 0x7f196406ccc0 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.156+0000 7f1969bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19641f18a0 con 0x7f196406d1d0 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f195bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196406d1d0 0x7f1964151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196406ccc0 0x7f19641482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f195bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196406d1d0 0x7f1964151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42208/0 (socket says 172.21.15.102:42208) 2023-03-03T15:28:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f195bfff700 1 -- 172.21.15.102:0/2510458548 learned_addr learned my addr 172.21.15.102:0/2510458548 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:28:03.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196406ccc0 0x7f19641482a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60438/0 (socket says 172.21.15.102:60438) 2023-03-03T15:28:03.161 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f1963fff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1964152b70 0x7f19641f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:03.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 -- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196406d1d0 msgr2=0x7f1964151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:03.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196406d1d0 0x7f1964151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 -- 172.21.15.102:0/2510458548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1964152b70 msgr2=0x7f19641f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:03.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1964152b70 0x7f19641f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 -- 172.21.15.102:0/2510458548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954009550 con 0x7f196406ccc0 2023-03-03T15:28:03.162 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.157+0000 7f19637fe700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196406ccc0 0x7f19641482a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1954016000 tx=0x7f1954004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.158+0000 7f1960ff9700 1 -- 172.21.15.102:0/2510458548 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954004300 con 0x7f196406ccc0 2023-03-03T15:28:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.158+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19640c8060 con 0x7f196406ccc0 2023-03-03T15:28:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.158+0000 7f1960ff9700 1 -- 172.21.15.102:0/2510458548 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195401f070 con 0x7f196406ccc0 2023-03-03T15:28:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.158+0000 7f1960ff9700 1 -- 172.21.15.102:0/2510458548 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19540036a0 con 0x7f196406ccc0 2023-03-03T15:28:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.158+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19640c5430 con 0x7f196406ccc0 2023-03-03T15:28:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.159+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19640c56d0 con 0x7f196406ccc0 2023-03-03T15:28:03.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.161+0000 7f1960ff9700 1 -- 172.21.15.102:0/2510458548 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f195402a020 con 0x7f196406ccc0 2023-03-03T15:28:03.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.162+0000 7f1960ff9700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:03.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.162+0000 7f1960ff9700 1 -- 172.21.15.102:0/2510458548 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1954087ba0 con 0x7f196406ccc0 2023-03-03T15:28:03.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.162+0000 7f195bfff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:03.165 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.163+0000 7f195bfff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f1964145090 tx=0x7f194c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:03.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.165+0000 7f1960ff9700 1 -- 172.21.15.102:0/2510458548 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1954054230 con 0x7f196406ccc0 2023-03-03T15:28:03.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.398+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19641f1a60 con 0x7f196406ccc0 2023-03-03T15:28:03.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.399+0000 7f1960ff9700 1 -- 172.21.15.102:0/2510458548 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1954057850 con 0x7f196406ccc0 2023-03-03T15:28:03.402 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:28:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:28:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 msgr2=0x7f194406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f1964145090 tx=0x7f194c009210 comp rx=0 tx=0).stop 2023-03-03T15:28:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196406ccc0 msgr2=0x7f19641482a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196406ccc0 0x7f19641482a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1954016000 tx=0x7f1954004750 comp rx=0 tx=0).stop 2023-03-03T15:28:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 shutdown_connections 2023-03-03T15:28:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1964152b70 0x7f19641f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f196406ccc0 0x7f19641482a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f19440682f0 0x7f194406a7a0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 --2- 172.21.15.102:0/2510458548 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f196406d1d0 0x7f1964151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 >> 172.21.15.102:0/2510458548 conn(0x7f196406af90 msgr2=0x7f19640c89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 shutdown_connections 2023-03-03T15:28:03.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:03.402+0000 7f1969bff700 1 -- 172.21.15.102:0/2510458548 wait complete. 2023-03-03T15:28:03.407 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:28:03.431 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before thrashing 2023-03-03T15:28:04.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:28:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.693+0000 7f58b2d9d700 1 Processor -- start 2023-03-03T15:28:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.694+0000 7f58b2d9d700 1 -- start start 2023-03-03T15:28:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac1564b0 0x7f58ac0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b2d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58ac0c9610 0x7f58ac0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b2d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac0c9ca0 con 0x7f58ac0ca440 2023-03-03T15:28:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b2d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac0b4920 con 0x7f58ac0c9610 2023-03-03T15:28:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b2d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0b4c00 con 0x7f58ac1564b0 2023-03-03T15:28:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b1d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac1564b0 0x7f58ac0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b159a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58ac0c9610 0x7f58ac0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.695+0000 7f58b259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f58ac0ca440 0x7f58ac0ca880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43692/0 (socket says 172.21.15.102:43692) 2023-03-03T15:28:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.696+0000 7f58b259c700 1 -- 172.21.15.102:0/3301234908 learned_addr learned my addr 172.21.15.102:0/3301234908 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:28:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.696+0000 7f58b1d9b700 1 -- 172.21.15.102:0/3301234908 >> v2:172.21.15.172:3300/0 conn(0x7f58ac0c9610 msgr2=0x7f58ac0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:28:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.696+0000 7f58b1d9b700 1 --2- 172.21.15.102:0/3301234908 >> v2:172.21.15.172:3300/0 conn(0x7f58ac0c9610 0x7f58ac0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.696+0000 7f58b1d9b700 1 -- 172.21.15.102:0/3301234908 >> v1:172.21.15.102:6789/0 conn(0x7f58ac0ca440 legacy=0x7f58ac0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.696+0000 7f58b1d9b700 1 -- 172.21.15.102:0/3301234908 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0b46f0 con 0x7f58ac1564b0 2023-03-03T15:28:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.696+0000 7f58b0d99700 1 -- 172.21.15.102:0/3301234908 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289524326 0 0) 0x7f58ac0c9ca0 con 0x7f58ac0ca440 2023-03-03T15:28:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.696+0000 7f58b1d9b700 1 --2- 172.21.15.102:0/3301234908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac1564b0 0x7f58ac0cb660 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009840 tx=0x7f58a8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=217cb4217e39dbd5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b0d99700 1 -- 172.21.15.102:0/3301234908 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8011830 con 0x7f58ac1564b0 2023-03-03T15:28:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b0d99700 1 -- 172.21.15.102:0/3301234908 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a8011990 con 0x7f58ac1564b0 2023-03-03T15:28:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b0d99700 1 -- 172.21.15.102:0/3301234908 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8011d20 con 0x7f58ac1564b0 2023-03-03T15:28:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 -- 172.21.15.102:0/3301234908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac1564b0 msgr2=0x7f58ac0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/3301234908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac1564b0 0x7f58ac0cb660 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009840 tx=0x7f58a8009b50 comp rx=0 tx=0).stop 2023-03-03T15:28:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 -- 172.21.15.102:0/3301234908 shutdown_connections 2023-03-03T15:28:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/3301234908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac1564b0 0x7f58ac0cb660 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/3301234908 >> v2:172.21.15.172:3300/0 conn(0x7f58ac0c9610 0x7f58ac0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 -- 172.21.15.102:0/3301234908 >> 172.21.15.102:0/3301234908 conn(0x7f58ac065fb0 msgr2=0x7f58ac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 -- 172.21.15.102:0/3301234908 shutdown_connections 2023-03-03T15:28:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.697+0000 7f58b2d9d700 1 -- 172.21.15.102:0/3301234908 wait complete. 2023-03-03T15:28:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.699+0000 7f58b2d9d700 1 Processor -- start 2023-03-03T15:28:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.700+0000 7f58b2d9d700 1 -- start start 2023-03-03T15:28:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58ac0c9610 0x7f58ac159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac0ca440 0x7f58ac1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:04.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b2d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ac1564b0 0x7f58ac0ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:04.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b2d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1f1fe0 con 0x7f58ac0c9610 2023-03-03T15:28:04.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b2d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1f1680 con 0x7f58ac1564b0 2023-03-03T15:28:04.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b2d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1eec50 con 0x7f58ac0ca440 2023-03-03T15:28:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58ac0c9610 0x7f58ac159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58ac0c9610 0x7f58ac159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38070/0 (socket says 172.21.15.102:38070) 2023-03-03T15:28:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b159a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac0ca440 0x7f58ac1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:04.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 -- 172.21.15.102:0/2745363694 learned_addr learned my addr 172.21.15.102:0/2745363694 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:28:04.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b259c700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ac1564b0 0x7f58ac0ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:04.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 -- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac0ca440 msgr2=0x7f58ac1522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:04.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac0ca440 0x7f58ac1522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 -- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ac1564b0 msgr2=0x7f58ac0ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:04.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ac1564b0 0x7f58ac0ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b1d9b700 1 -- 172.21.15.102:0/2745363694 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80094a0 con 0x7f58ac0c9610 2023-03-03T15:28:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.701+0000 7f58b159a700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac0ca440 0x7f58ac1522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:28:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.702+0000 7f58b1d9b700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58ac0c9610 0x7f58ac159b50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009fd0 tx=0x7f58a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.702+0000 7f58a27fc700 1 -- 172.21.15.102:0/2745363694 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80036a0 con 0x7f58ac0c9610 2023-03-03T15:28:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.702+0000 7f58a27fc700 1 -- 172.21.15.102:0/2745363694 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a801f070 con 0x7f58ac0c9610 2023-03-03T15:28:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.702+0000 7f58a27fc700 1 -- 172.21.15.102:0/2745363694 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8003a60 con 0x7f58ac0c9610 2023-03-03T15:28:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.702+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac0b2210 con 0x7f58ac0c9610 2023-03-03T15:28:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.702+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac0b1f30 con 0x7f58ac0c9610 2023-03-03T15:28:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.704+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac156880 con 0x7f58ac0c9610 2023-03-03T15:28:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.705+0000 7f58a27fc700 1 -- 172.21.15.102:0/2745363694 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f58a800f050 con 0x7f58ac0c9610 2023-03-03T15:28:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.706+0000 7f58a27fc700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58900682f0 0x7f589006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.706+0000 7f58a27fc700 1 -- 172.21.15.102:0/2745363694 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f58a8021070 con 0x7f58ac0c9610 2023-03-03T15:28:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.706+0000 7f58b159a700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58900682f0 0x7f589006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.707+0000 7f58b159a700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58900682f0 0x7f589006a7a0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0c23e0 tx=0x7f589c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:04.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.710+0000 7f58a27fc700 1 -- 172.21.15.102:0/2745363694 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f58a8053920 con 0x7f58ac0c9610 2023-03-03T15:28:04.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.943+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58ac0677f0 con 0x7f58ac0c9610 2023-03-03T15:28:04.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.944+0000 7f58a27fc700 1 -- 172.21.15.102:0/2745363694 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f58a8015020 con 0x7f58ac0c9610 2023-03-03T15:28:04.946 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:28:04.946 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:28:04.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.946+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58900682f0 msgr2=0x7f589006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:04.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.946+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58900682f0 0x7f589006a7a0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0c23e0 tx=0x7f589c009210 comp rx=0 tx=0).stop 2023-03-03T15:28:04.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.946+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58ac0c9610 msgr2=0x7f58ac159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.946+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58ac0c9610 0x7f58ac159b50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009fd0 tx=0x7f58a8004750 comp rx=0 tx=0).stop 2023-03-03T15:28:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 shutdown_connections 2023-03-03T15:28:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ac1564b0 0x7f58ac0ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58ac0c9610 0x7f58ac159b50 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f58900682f0 0x7f589006a7a0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 --2- 172.21.15.102:0/2745363694 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f58ac0ca440 0x7f58ac1522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:04.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 >> 172.21.15.102:0/2745363694 conn(0x7f58ac065fb0 msgr2=0x7f58ac0c6140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 shutdown_connections 2023-03-03T15:28:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:28:04.947+0000 7f58b2d9d700 1 -- 172.21.15.102:0/2745363694 wait complete. 2023-03-03T15:28:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:28:04.976 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-03T15:28:30.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:30.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:00.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:00.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:15.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:29:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.239+0000 7f5fc57ff700 1 Processor -- start 2023-03-03T15:29:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.239+0000 7f5fc57ff700 1 -- start start 2023-03-03T15:29:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fc57ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fc014ffb0 0x7f5fc00c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fc57ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5fc00716c0 0x7f5fc00706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fc57ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc00c3c80 con 0x7f5fc014fbe0 2023-03-03T15:29:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fc57ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc00c41a0 con 0x7f5fc00716c0 2023-03-03T15:29:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fc57ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc0071cd0 con 0x7f5fc014ffb0 2023-03-03T15:29:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fbeffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5fc014fbe0 0x7f5fc00c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42754/0 (socket says 172.21.15.102:42754) 2023-03-03T15:29:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fbeffd700 1 -- 172.21.15.102:0/2264452122 learned_addr learned my addr 172.21.15.102:0/2264452122 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:29:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fbf7fe700 1 --2- 172.21.15.102:0/2264452122 >> v2:172.21.15.102:3300/0 conn(0x7f5fc00716c0 0x7f5fc00706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.241+0000 7f5fbe7fc700 1 --2- 172.21.15.102:0/2264452122 >> v2:172.21.15.172:3300/0 conn(0x7f5fc014ffb0 0x7f5fc00c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.242+0000 7f5fbdffb700 1 -- 172.21.15.102:0/2264452122 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554903181 0 0) 0x7f5fb0001110 con 0x7f5fc014fbe0 2023-03-03T15:29:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.242+0000 7f5fbdffb700 1 -- 172.21.15.102:0/2264452122 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa80035b0 con 0x7f5fc014fbe0 2023-03-03T15:29:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.242+0000 7f5fbe7fc700 1 -- 172.21.15.102:0/2264452122 >> v2:172.21.15.102:3300/0 conn(0x7f5fc00716c0 msgr2=0x7f5fc00706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.242+0000 7f5fbe7fc700 1 --2- 172.21.15.102:0/2264452122 >> v2:172.21.15.102:3300/0 conn(0x7f5fc00716c0 0x7f5fc00706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.242+0000 7f5fbe7fc700 1 -- 172.21.15.102:0/2264452122 >> v1:172.21.15.102:6789/0 conn(0x7f5fc014fbe0 legacy=0x7f5fc00c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.242+0000 7f5fbe7fc700 1 -- 172.21.15.102:0/2264452122 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc00c3ec0 con 0x7f5fc014ffb0 2023-03-03T15:29:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.242+0000 7f5fbe7fc700 1 --2- 172.21.15.102:0/2264452122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014ffb0 0x7f5fc00c3740 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00c2850 tx=0x7f5fb4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e8d37b27854cedd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fbdffb700 1 -- 172.21.15.102:0/2264452122 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4011830 con 0x7f5fc014ffb0 2023-03-03T15:29:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fbdffb700 1 -- 172.21.15.102:0/2264452122 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4011990 con 0x7f5fc014ffb0 2023-03-03T15:29:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fbdffb700 1 -- 172.21.15.102:0/2264452122 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4011d00 con 0x7f5fc014ffb0 2023-03-03T15:29:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 -- 172.21.15.102:0/2264452122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014ffb0 msgr2=0x7f5fc00c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/2264452122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014ffb0 0x7f5fc00c3740 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00c2850 tx=0x7f5fb4009b20 comp rx=0 tx=0).stop 2023-03-03T15:29:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 -- 172.21.15.102:0/2264452122 shutdown_connections 2023-03-03T15:29:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/2264452122 >> v2:172.21.15.102:3300/0 conn(0x7f5fc00716c0 0x7f5fc00706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/2264452122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014ffb0 0x7f5fc00c3740 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 -- 172.21.15.102:0/2264452122 >> 172.21.15.102:0/2264452122 conn(0x7f5fc006af90 msgr2=0x7f5fc006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 -- 172.21.15.102:0/2264452122 shutdown_connections 2023-03-03T15:29:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.243+0000 7f5fc57ff700 1 -- 172.21.15.102:0/2264452122 wait complete. 2023-03-03T15:29:16.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.245+0000 7f5fc57ff700 1 Processor -- start 2023-03-03T15:29:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.246+0000 7f5fc57ff700 1 -- start start 2023-03-03T15:29:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.246+0000 7f5fc57ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fc00716c0 0x7f5fc01ed190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fc57ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014fbe0 0x7f5fc0151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fc57ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fc014ffb0 0x7f5fc0151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fc57ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc01513d0 con 0x7f5fc014ffb0 2023-03-03T15:29:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fc57ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc014bcd0 con 0x7f5fc014fbe0 2023-03-03T15:29:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fc57ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc014bf70 con 0x7f5fc00716c0 2023-03-03T15:29:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fc014ffb0 0x7f5fc0151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbe7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014fbe0 0x7f5fc0151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbf7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fc014ffb0 0x7f5fc0151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52940/0 (socket says 172.21.15.102:52940) 2023-03-03T15:29:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbf7fe700 1 -- 172.21.15.102:0/4235311765 learned_addr learned my addr 172.21.15.102:0/4235311765 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:29:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbeffd700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fc00716c0 0x7f5fc01ed190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbf7fe700 1 -- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fc00716c0 msgr2=0x7f5fc01ed190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbf7fe700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fc00716c0 0x7f5fc01ed190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbf7fe700 1 -- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014fbe0 msgr2=0x7f5fc0151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.247+0000 7f5fbf7fe700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014fbe0 0x7f5fc0151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.248+0000 7f5fbf7fe700 1 -- 172.21.15.102:0/4235311765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40094a0 con 0x7f5fc014ffb0 2023-03-03T15:29:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.248+0000 7f5fbf7fe700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fc014ffb0 0x7f5fc0151da0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800ef90 tx=0x7f5fb800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.248+0000 7f5fa77fe700 1 -- 172.21.15.102:0/4235311765 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb800cc90 con 0x7f5fc014ffb0 2023-03-03T15:29:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.248+0000 7f5fa77fe700 1 -- 172.21.15.102:0/4235311765 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb800f040 con 0x7f5fc014ffb0 2023-03-03T15:29:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.248+0000 7f5fa77fe700 1 -- 172.21.15.102:0/4235311765 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8014480 con 0x7f5fc014ffb0 2023-03-03T15:29:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.248+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc0072250 con 0x7f5fc014ffb0 2023-03-03T15:29:16.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.248+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc0071f70 con 0x7f5fc014ffb0 2023-03-03T15:29:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.249+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc0072530 con 0x7f5fc014ffb0 2023-03-03T15:29:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.251+0000 7f5fa77fe700 1 -- 172.21.15.102:0/4235311765 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5fb80145e0 con 0x7f5fc014ffb0 2023-03-03T15:29:16.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.252+0000 7f5fa77fe700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5fac0682f0 0x7f5fac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.253+0000 7f5fa77fe700 1 -- 172.21.15.102:0/4235311765 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5fb8086b30 con 0x7f5fc014ffb0 2023-03-03T15:29:16.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.253+0000 7f5fbeffd700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5fac0682f0 0x7f5fac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.254+0000 7f5fbeffd700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5fac0682f0 0x7f5fac06a7a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0001370 tx=0x7f5fb000d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.257+0000 7f5fa77fe700 1 -- 172.21.15.102:0/4235311765 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5fb80531b0 con 0x7f5fc014ffb0 2023-03-03T15:29:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.491+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fc004e6e0 con 0x7f5fc014ffb0 2023-03-03T15:29:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.492+0000 7f5fa77fe700 1 -- 172.21.15.102:0/4235311765 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5fb8018070 con 0x7f5fc014ffb0 2023-03-03T15:29:16.493 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:29:16.493 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:29:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5fac0682f0 msgr2=0x7f5fac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5fac0682f0 0x7f5fac06a7a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0001370 tx=0x7f5fb000d040 comp rx=0 tx=0).stop 2023-03-03T15:29:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fc014ffb0 msgr2=0x7f5fc0151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fc014ffb0 0x7f5fc0151da0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800ef90 tx=0x7f5fb800c3a0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 shutdown_connections 2023-03-03T15:29:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5fac0682f0 0x7f5fac06a7a0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5fc00716c0 0x7f5fc01ed190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fc014ffb0 0x7f5fc0151da0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 --2- 172.21.15.102:0/4235311765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fc014fbe0 0x7f5fc0151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 >> 172.21.15.102:0/4235311765 conn(0x7f5fc006af90 msgr2=0x7f5fc014dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 shutdown_connections 2023-03-03T15:29:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:29:16.494+0000 7f5fc57ff700 1 -- 172.21.15.102:0/4235311765 wait complete. 2023-03-03T15:29:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:29:16.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T15:29:30.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:30.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T15:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:30:01.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:01.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:12.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:30:12.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.769+0000 7f949e7ff700 1 Processor -- start 2023-03-03T15:30:12.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.770+0000 7f949e7ff700 1 -- start start 2023-03-03T15:30:12.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.771+0000 7f949e7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9498153290 0x7f9498158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:12.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.771+0000 7f949e7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94981593c0 0x7f94980cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:12.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.771+0000 7f949e7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94980cd8e0 0x7f94980ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:12.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.771+0000 7f949e7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9498153660 con 0x7f9498153290 2023-03-03T15:30:12.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.771+0000 7f949e7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949806c360 con 0x7f94980cd8e0 2023-03-03T15:30:12.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.771+0000 7f949e7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949815cea0 con 0x7f94981593c0 2023-03-03T15:30:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.771+0000 7f9497fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9498153290 0x7f9498158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9498153290 0x7f9498158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50302/0 (socket says 172.21.15.102:50302) 2023-03-03T15:30:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 -- 172.21.15.102:0/3670713464 learned_addr learned my addr 172.21.15.102:0/3670713464 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:30:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f949cd9c700 1 --2- 172.21.15.102:0/3670713464 >> v2:172.21.15.172:3300/0 conn(0x7f94980cd8e0 0x7f94980ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f94977fe700 1 --2- 172.21.15.102:0/3670713464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94981593c0 0x7f94980cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 -- 172.21.15.102:0/3670713464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94981593c0 msgr2=0x7f94980cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:12.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 --2- 172.21.15.102:0/3670713464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94981593c0 0x7f94980cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 -- 172.21.15.102:0/3670713464 >> v2:172.21.15.172:3300/0 conn(0x7f94980cd8e0 msgr2=0x7f94980ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:12.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 --2- 172.21.15.102:0/3670713464 >> v2:172.21.15.172:3300/0 conn(0x7f94980cd8e0 0x7f94980ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 -- 172.21.15.102:0/3670713464 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949806c7f0 con 0x7f9498153290 2023-03-03T15:30:12.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.772+0000 7f9497fff700 1 --2- 172.21.15.102:0/3670713464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9498153290 0x7f9498158e80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f949806d0e0 tx=0x7f9488009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31d0720a10d7c67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:12.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f9496ffd700 1 -- 172.21.15.102:0/3670713464 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488011830 con 0x7f9498153290 2023-03-03T15:30:12.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f9496ffd700 1 -- 172.21.15.102:0/3670713464 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9488011990 con 0x7f9498153290 2023-03-03T15:30:12.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f9496ffd700 1 -- 172.21.15.102:0/3670713464 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488011d20 con 0x7f9498153290 2023-03-03T15:30:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 -- 172.21.15.102:0/3670713464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9498153290 msgr2=0x7f9498158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 --2- 172.21.15.102:0/3670713464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9498153290 0x7f9498158e80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f949806d0e0 tx=0x7f9488009bd0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 -- 172.21.15.102:0/3670713464 shutdown_connections 2023-03-03T15:30:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 --2- 172.21.15.102:0/3670713464 >> v2:172.21.15.172:3300/0 conn(0x7f94980cd8e0 0x7f94980ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 --2- 172.21.15.102:0/3670713464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9498153290 0x7f9498158e80 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 --2- 172.21.15.102:0/3670713464 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94981593c0 0x7f94980cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 -- 172.21.15.102:0/3670713464 >> 172.21.15.102:0/3670713464 conn(0x7f9498065fb0 msgr2=0x7f94980663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:12.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 -- 172.21.15.102:0/3670713464 shutdown_connections 2023-03-03T15:30:12.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.773+0000 7f949e7ff700 1 -- 172.21.15.102:0/3670713464 wait complete. 2023-03-03T15:30:12.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.775+0000 7f949e7ff700 1 Processor -- start 2023-03-03T15:30:12.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.775+0000 7f949e7ff700 1 -- start start 2023-03-03T15:30:12.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.776+0000 7f949e7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94980cd8e0 0x7f949814dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:12.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.776+0000 7f949e7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9498153290 0x7f9498155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:12.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.776+0000 7f949e7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94981593c0 0x7f9498155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.776+0000 7f949e7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9498155100 con 0x7f94981593c0 2023-03-03T15:30:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.776+0000 7f949e7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94981f2480 con 0x7f9498153290 2023-03-03T15:30:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.776+0000 7f949e7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949815b9d0 con 0x7f94980cd8e0 2023-03-03T15:30:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f949cd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94981593c0 0x7f9498155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94980cd8e0 0x7f949814dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f94977fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9498153290 0x7f9498155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94980cd8e0 0x7f949814dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36988/0 (socket says 172.21.15.102:36988) 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 -- 172.21.15.102:0/2448466063 learned_addr learned my addr 172.21.15.102:0/2448466063 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 -- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9498153290 msgr2=0x7f9498155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9498153290 0x7f9498155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 -- 172.21.15.102:0/2448466063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94981593c0 msgr2=0x7f9498155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94981593c0 0x7f9498155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f9497fff700 1 -- 172.21.15.102:0/2448466063 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488009550 con 0x7f94980cd8e0 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.777+0000 7f949cd9c700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94981593c0 0x7f9498155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:30:12.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.778+0000 7f9497fff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94980cd8e0 0x7f949814dbb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9488009c10 tx=0x7f9488004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.778+0000 7f9494ff9700 1 -- 172.21.15.102:0/2448466063 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488004450 con 0x7f94980cd8e0 2023-03-03T15:30:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.778+0000 7f9494ff9700 1 -- 172.21.15.102:0/2448466063 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948801e070 con 0x7f94980cd8e0 2023-03-03T15:30:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.778+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94980ccde0 con 0x7f94980cd8e0 2023-03-03T15:30:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.778+0000 7f9494ff9700 1 -- 172.21.15.102:0/2448466063 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94880038c0 con 0x7f94980cd8e0 2023-03-03T15:30:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.778+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949815d160 con 0x7f94980cd8e0 2023-03-03T15:30:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.779+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94980cd0c0 con 0x7f94980cd8e0 2023-03-03T15:30:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.781+0000 7f9494ff9700 1 -- 172.21.15.102:0/2448466063 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f948800f050 con 0x7f94980cd8e0 2023-03-03T15:30:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.782+0000 7f9494ff9700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f94840682f0 0x7f948406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.782+0000 7f9494ff9700 1 -- 172.21.15.102:0/2448466063 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9488086ba0 con 0x7f94980cd8e0 2023-03-03T15:30:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.782+0000 7f94977fe700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f94840682f0 0x7f948406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.783+0000 7f94977fe700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f94840682f0 0x7f948406a7a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f94980b4d90 tx=0x7f948c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:12.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:12.786+0000 7f9494ff9700 1 -- 172.21.15.102:0/2448466063 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94880532b0 con 0x7f94980cd8e0 2023-03-03T15:30:13.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.023+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f949804e6e0 con 0x7f94980cd8e0 2023-03-03T15:30:13.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.023+0000 7f9494ff9700 1 -- 172.21.15.102:0/2448466063 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9488015090 con 0x7f94980cd8e0 2023-03-03T15:30:13.026 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:30:13.026 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:30:13.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f94840682f0 msgr2=0x7f948406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:13.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f94840682f0 0x7f948406a7a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f94980b4d90 tx=0x7f948c008040 comp rx=0 tx=0).stop 2023-03-03T15:30:13.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94980cd8e0 msgr2=0x7f949814dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:13.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94980cd8e0 0x7f949814dbb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9488009c10 tx=0x7f9488004750 comp rx=0 tx=0).stop 2023-03-03T15:30:13.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 shutdown_connections 2023-03-03T15:30:13.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f94840682f0 0x7f948406a7a0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:13.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94980cd8e0 0x7f949814dbb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:13.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94981593c0 0x7f9498155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:13.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 --2- 172.21.15.102:0/2448466063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9498153290 0x7f9498155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:13.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.026+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 >> 172.21.15.102:0/2448466063 conn(0x7f9498065fb0 msgr2=0x7f94980cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:13.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.027+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 shutdown_connections 2023-03-03T15:30:13.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:30:13.027+0000 7f949e7ff700 1 -- 172.21.15.102:0/2448466063 wait complete. 2023-03-03T15:30:13.031 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:30:13.056 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2023-03-03T15:30:31.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:31.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:01.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:01.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:02.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:31:02.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.301+0000 7f0d20b9d700 1 Processor -- start 2023-03-03T15:31:02.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.301+0000 7f0d20b9d700 1 -- start start 2023-03-03T15:31:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d20b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0ced00 0x7f0d1c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d20b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d1c0c41b0 0x7f0d1c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d20b9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0c3d20 con 0x7f0d1c0b3970 2023-03-03T15:31:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d20b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c0c4930 con 0x7f0d1c0c41b0 2023-03-03T15:31:02.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d20b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0cf610 con 0x7f0d1c0ced00 2023-03-03T15:31:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d1bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d1c0c41b0 0x7f0d1c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d1b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0ced00 0x7f0d1c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d1affd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0d1c0b3970 0x7f0d1c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45292/0 (socket says 172.21.15.102:45292) 2023-03-03T15:31:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.303+0000 7f0d1affd700 1 -- 172.21.15.102:0/3025808749 learned_addr learned my addr 172.21.15.102:0/3025808749 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:31:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.304+0000 7f0d1bfff700 1 -- 172.21.15.102:0/3025808749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0ced00 msgr2=0x7f0d1c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:31:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.304+0000 7f0d1bfff700 1 --2- 172.21.15.102:0/3025808749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0ced00 0x7f0d1c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.304+0000 7f0d1bfff700 1 -- 172.21.15.102:0/3025808749 >> v1:172.21.15.172:6789/0 conn(0x7f0d1c0b3970 legacy=0x7f0d1c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.304+0000 7f0d1bfff700 1 -- 172.21.15.102:0/3025808749 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c15cba0 con 0x7f0d1c0c41b0 2023-03-03T15:31:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.304+0000 7f0d1bfff700 1 --2- 172.21.15.102:0/3025808749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0c41b0 0x7f0d1c0cf0d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009dc0 tx=0x7f0d1400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79afc3311db285e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.304+0000 7f0d1a7fc700 1 -- 172.21.15.102:0/3025808749 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1400e070 con 0x7f0d1c0c41b0 2023-03-03T15:31:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.304+0000 7f0d1a7fc700 1 -- 172.21.15.102:0/3025808749 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1400bae0 con 0x7f0d1c0c41b0 2023-03-03T15:31:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d1a7fc700 1 -- 172.21.15.102:0/3025808749 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1400be90 con 0x7f0d1c0c41b0 2023-03-03T15:31:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 -- 172.21.15.102:0/3025808749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0c41b0 msgr2=0x7f0d1c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/3025808749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0c41b0 0x7f0d1c0cf0d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009dc0 tx=0x7f0d1400b330 comp rx=0 tx=0).stop 2023-03-03T15:31:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 -- 172.21.15.102:0/3025808749 shutdown_connections 2023-03-03T15:31:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/3025808749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0c41b0 0x7f0d1c0cf0d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/3025808749 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0ced00 0x7f0d1c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 -- 172.21.15.102:0/3025808749 >> 172.21.15.102:0/3025808749 conn(0x7f0d1c08b1c0 msgr2=0x7f0d1c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 -- 172.21.15.102:0/3025808749 shutdown_connections 2023-03-03T15:31:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.305+0000 7f0d20b9d700 1 -- 172.21.15.102:0/3025808749 wait complete. 2023-03-03T15:31:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.308+0000 7f0d20b9d700 1 Processor -- start 2023-03-03T15:31:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.308+0000 7f0d20b9d700 1 -- start start 2023-03-03T15:31:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d20b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0b3970 0x7f0d1c0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d20b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0c41b0 0x7f0d1c0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d20b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c0ced00 0x7f0d1c0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:02.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d20b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0c8210 con 0x7f0d1c0ced00 2023-03-03T15:31:02.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d20b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c15c460 con 0x7f0d1c0b3970 2023-03-03T15:31:02.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d20b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c15bea0 con 0x7f0d1c0c41b0 2023-03-03T15:31:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d1affd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0c41b0 0x7f0d1c0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d1b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0b3970 0x7f0d1c0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d1b7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0b3970 0x7f0d1c0c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42732/0 (socket says 172.21.15.102:42732) 2023-03-03T15:31:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d1b7fe700 1 -- 172.21.15.102:0/2957164183 learned_addr learned my addr 172.21.15.102:0/2957164183 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:31:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.310+0000 7f0d1bfff700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c0ced00 0x7f0d1c0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d1b7fe700 1 -- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0c41b0 msgr2=0x7f0d1c0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d1b7fe700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0c41b0 0x7f0d1c0c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d1b7fe700 1 -- 172.21.15.102:0/2957164183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c0ced00 msgr2=0x7f0d1c0c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d1b7fe700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c0ced00 0x7f0d1c0c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d1b7fe700 1 -- 172.21.15.102:0/2957164183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d14009a70 con 0x7f0d1c0b3970 2023-03-03T15:31:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d1b7fe700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0b3970 0x7f0d1c0c6cf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9b0 tx=0x7f0d0c00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d03fff700 1 -- 172.21.15.102:0/2957164183 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c009710 con 0x7f0d1c0b3970 2023-03-03T15:31:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d03fff700 1 -- 172.21.15.102:0/2957164183 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c00ee50 con 0x7f0d1c0b3970 2023-03-03T15:31:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d03fff700 1 -- 172.21.15.102:0/2957164183 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00ca50 con 0x7f0d1c0b3970 2023-03-03T15:31:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c06bf20 con 0x7f0d1c0b3970 2023-03-03T15:31:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.311+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c15ce80 con 0x7f0d1c0b3970 2023-03-03T15:31:02.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.312+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c06c200 con 0x7f0d1c0b3970 2023-03-03T15:31:02.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.313+0000 7f0d03fff700 1 -- 172.21.15.102:0/2957164183 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d0c019030 con 0x7f0d1c0b3970 2023-03-03T15:31:02.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.315+0000 7f0d03fff700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d08068320 0x7f0d0806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.315+0000 7f0d03fff700 1 -- 172.21.15.102:0/2957164183 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0d0c085c30 con 0x7f0d1c0b3970 2023-03-03T15:31:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.315+0000 7f0d1affd700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d08068320 0x7f0d0806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.316+0000 7f0d1affd700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d08068320 0x7f0d0806a7d0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c14ec90 tx=0x7f0d10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:02.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.319+0000 7f0d03fff700 1 -- 172.21.15.102:0/2957164183 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d0c0522e0 con 0x7f0d1c0b3970 2023-03-03T15:31:02.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.556+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d1c04e6e0 con 0x7f0d1c0b3970 2023-03-03T15:31:02.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.557+0000 7f0d03fff700 1 -- 172.21.15.102:0/2957164183 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0d0c055900 con 0x7f0d1c0b3970 2023-03-03T15:31:02.560 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:31:02.560 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:31:02.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.559+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d08068320 msgr2=0x7f0d0806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:02.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.559+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d08068320 0x7f0d0806a7d0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c14ec90 tx=0x7f0d10009040 comp rx=0 tx=0).stop 2023-03-03T15:31:02.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.559+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0b3970 msgr2=0x7f0d1c0c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:02.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.559+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0b3970 0x7f0d1c0c6cf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9b0 tx=0x7f0d0c00ecc0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 shutdown_connections 2023-03-03T15:31:02.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1c0ced00 0x7f0d1c0c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d1c0b3970 0x7f0d1c0c6cf0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0d08068320 0x7f0d0806a7d0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 --2- 172.21.15.102:0/2957164183 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0d1c0c41b0 0x7f0d1c0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:02.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 >> 172.21.15.102:0/2957164183 conn(0x7f0d1c08b1c0 msgr2=0x7f0d1c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:02.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 shutdown_connections 2023-03-03T15:31:02.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:02.560+0000 7f0d20b9d700 1 -- 172.21.15.102:0/2957164183 wait complete. 2023-03-03T15:31:02.564 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:31:02.590 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-03-03T15:31:05.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:31:05.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.842+0000 7f91007ff700 1 Processor -- start 2023-03-03T15:31:05.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.842+0000 7f91007ff700 1 -- start start 2023-03-03T15:31:05.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.843+0000 7f91007ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f8153290 0x7f90f8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:05.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.843+0000 7f91007ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90f80cd8e0 0x7f90f80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:05.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f91007ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f8153660 con 0x7f90f81593c0 2023-03-03T15:31:05.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f91007ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f806c360 con 0x7f90f80cd8e0 2023-03-03T15:31:05.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f91007ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f815cea0 con 0x7f90f8153290 2023-03-03T15:31:05.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fed9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90f80cd8e0 0x7f90f80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:05.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fdd9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f90f81593c0 0x7f90f80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39870/0 (socket says 172.21.15.102:39870) 2023-03-03T15:31:05.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fdd9a700 1 -- 172.21.15.102:0/3960555340 learned_addr learned my addr 172.21.15.102:0/3960555340 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:31:05.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fe59b700 1 --2- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f8153290 0x7f90f8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:05.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fed9c700 1 -- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f8153290 msgr2=0x7f90f8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:31:05.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fed9c700 1 --2- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f8153290 0x7f90f8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:05.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fed9c700 1 -- 172.21.15.102:0/3960555340 >> v1:172.21.15.102:6789/0 conn(0x7f90f81593c0 legacy=0x7f90f80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:05.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.844+0000 7f90fed9c700 1 -- 172.21.15.102:0/3960555340 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f806c7f0 con 0x7f90f80cd8e0 2023-03-03T15:31:05.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.845+0000 7f90fed9c700 1 --2- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f80ce530 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009d90 tx=0x7f90f000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84a8299ee5ea76ee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:05.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.845+0000 7f90fd599700 1 -- 172.21.15.102:0/3960555340 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f000e070 con 0x7f90f80cd8e0 2023-03-03T15:31:05.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.845+0000 7f90fd599700 1 -- 172.21.15.102:0/3960555340 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f000bb20 con 0x7f90f80cd8e0 2023-03-03T15:31:05.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.845+0000 7f90fd599700 1 -- 172.21.15.102:0/3960555340 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0012430 con 0x7f90f80cd8e0 2023-03-03T15:31:05.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.845+0000 7f91007ff700 1 -- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 msgr2=0x7f90f80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:05.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.845+0000 7f91007ff700 1 --2- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f80ce530 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009d90 tx=0x7f90f000b330 comp rx=0 tx=0).stop 2023-03-03T15:31:05.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.846+0000 7f91007ff700 1 -- 172.21.15.102:0/3960555340 shutdown_connections 2023-03-03T15:31:05.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.846+0000 7f91007ff700 1 --2- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f80ce530 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:05.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.846+0000 7f91007ff700 1 --2- 172.21.15.102:0/3960555340 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f8153290 0x7f90f8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:05.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.846+0000 7f91007ff700 1 -- 172.21.15.102:0/3960555340 >> 172.21.15.102:0/3960555340 conn(0x7f90f8065fb0 msgr2=0x7f90f80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:05.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.846+0000 7f91007ff700 1 -- 172.21.15.102:0/3960555340 shutdown_connections 2023-03-03T15:31:05.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.846+0000 7f91007ff700 1 -- 172.21.15.102:0/3960555340 wait complete. 2023-03-03T15:31:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.848+0000 7f91007ff700 1 Processor -- start 2023-03-03T15:31:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.848+0000 7f91007ff700 1 -- start start 2023-03-03T15:31:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f91007ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f81ef260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f91007ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f8153290 0x7f90f80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f91007ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f81593c0 0x7f90f80c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f91007ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f8156ad0 con 0x7f90f8153290 2023-03-03T15:31:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f91007ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f8156d70 con 0x7f90f80cd8e0 2023-03-03T15:31:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f91007ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f81ee780 con 0x7f90f81593c0 2023-03-03T15:31:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f90fdd9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f8153290 0x7f90f80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f90fe59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f81ef260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f90fe59b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f81ef260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:42748/0 (socket says 172.21.15.102:42748) 2023-03-03T15:31:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f90fe59b700 1 -- 172.21.15.102:0/1566671113 learned_addr learned my addr 172.21.15.102:0/1566671113 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:31:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.849+0000 7f90fed9c700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f81593c0 0x7f90f80c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90fe59b700 1 -- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f81593c0 msgr2=0x7f90f80c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90fe59b700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f81593c0 0x7f90f80c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90fe59b700 1 -- 172.21.15.102:0/1566671113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f8153290 msgr2=0x7f90f80b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90fe59b700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f8153290 0x7f90f80b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90fe59b700 1 -- 172.21.15.102:0/1566671113 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f0009a70 con 0x7f90f80cd8e0 2023-03-03T15:31:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90fe59b700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f81ef260 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e9b0 tx=0x7f90f400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90eeffd700 1 -- 172.21.15.102:0/1566671113 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4009710 con 0x7f90f80cd8e0 2023-03-03T15:31:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90eeffd700 1 -- 172.21.15.102:0/1566671113 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f400ee50 con 0x7f90f80cd8e0 2023-03-03T15:31:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f80ccde0 con 0x7f90f80cd8e0 2023-03-03T15:31:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f90eeffd700 1 -- 172.21.15.102:0/1566671113 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400ca50 con 0x7f90f80cd8e0 2023-03-03T15:31:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.850+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f815d160 con 0x7f90f80cd8e0 2023-03-03T15:31:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.852+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f80cd0c0 con 0x7f90f80cd8e0 2023-03-03T15:31:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.853+0000 7f90eeffd700 1 -- 172.21.15.102:0/1566671113 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90f4019030 con 0x7f90f80cd8e0 2023-03-03T15:31:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.854+0000 7f90eeffd700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90dc0682f0 0x7f90dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:05.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.854+0000 7f90eeffd700 1 -- 172.21.15.102:0/1566671113 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f90f4085c80 con 0x7f90f80cd8e0 2023-03-03T15:31:05.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.854+0000 7f90fdd9a700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90dc0682f0 0x7f90dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:05.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.855+0000 7f90fdd9a700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90dc0682f0 0x7f90dc06a7a0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f90f80b4bf0 tx=0x7f90e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:05.858+0000 7f90eeffd700 1 -- 172.21.15.102:0/1566671113 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90f4052390 con 0x7f90f80cd8e0 2023-03-03T15:31:06.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.090+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f815c4c0 con 0x7f90f80cd8e0 2023-03-03T15:31:06.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.092+0000 7f90eeffd700 1 -- 172.21.15.102:0/1566671113 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f90f40559b0 con 0x7f90f80cd8e0 2023-03-03T15:31:06.094 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:31:06.094 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:31:06.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.093+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90dc0682f0 msgr2=0x7f90dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:06.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.093+0000 7f91007ff700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90dc0682f0 0x7f90dc06a7a0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f90f80b4bf0 tx=0x7f90e8009040 comp rx=0 tx=0).stop 2023-03-03T15:31:06.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.093+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 msgr2=0x7f90f81ef260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.093+0000 7f91007ff700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f81ef260 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e9b0 tx=0x7f90f400ecc0 comp rx=0 tx=0).stop 2023-03-03T15:31:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 shutdown_connections 2023-03-03T15:31:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f90dc0682f0 0x7f90dc06a7a0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90f81593c0 0x7f90f80c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f80cd8e0 0x7f90f81ef260 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 --2- 172.21.15.102:0/1566671113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90f8153290 0x7f90f80b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 >> 172.21.15.102:0/1566671113 conn(0x7f90f8065fb0 msgr2=0x7f90f80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:06.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 shutdown_connections 2023-03-03T15:31:06.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:06.094+0000 7f91007ff700 1 -- 172.21.15.102:0/1566671113 wait complete. 2023-03-03T15:31:06.098 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:31:06.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-03-03T15:31:31.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:31.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:40.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:31:40.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.375+0000 7fcaadbff700 1 Processor -- start 2023-03-03T15:31:40.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.376+0000 7fcaadbff700 1 -- start start 2023-03-03T15:31:40.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaadbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcaa8153290 0x7fcaa8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:40.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaadbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa81593c0 0x7fcaa80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:40.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaadbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa8153660 con 0x7fcaa80cd8e0 2023-03-03T15:31:40.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaadbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa806c360 con 0x7fcaa8153290 2023-03-03T15:31:40.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaadbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa815cea0 con 0x7fcaa81593c0 2023-03-03T15:31:40.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaa77fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcaa8153290 0x7fcaa8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:40.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaa7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcaa80cd8e0 0x7fcaa80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49928/0 (socket says 172.21.15.102:49928) 2023-03-03T15:31:40.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.377+0000 7fcaa7fff700 1 -- 172.21.15.102:0/1317304868 learned_addr learned my addr 172.21.15.102:0/1317304868 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:31:40.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.378+0000 7fcaa6ffd700 1 --2- 172.21.15.102:0/1317304868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa81593c0 0x7fcaa80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:40.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.378+0000 7fcaa77fe700 1 -- 172.21.15.102:0/1317304868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa81593c0 msgr2=0x7fcaa80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:31:40.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.378+0000 7fcaa77fe700 1 --2- 172.21.15.102:0/1317304868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa81593c0 0x7fcaa80cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.378+0000 7fcaa77fe700 1 -- 172.21.15.102:0/1317304868 >> v1:172.21.15.102:6789/0 conn(0x7fcaa80cd8e0 legacy=0x7fcaa80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.378+0000 7fcaa77fe700 1 -- 172.21.15.102:0/1317304868 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa806c7f0 con 0x7fcaa8153290 2023-03-03T15:31:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.378+0000 7fcaa67fc700 1 -- 172.21.15.102:0/1317304868 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122506570 0 0) 0x7fcaa8153660 con 0x7fcaa80cd8e0 2023-03-03T15:31:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.378+0000 7fcaa77fe700 1 --2- 172.21.15.102:0/1317304868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa8153290 0x7fcaa8158e80 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fca98009840 tx=0x7fca98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9e72872e6a3bcb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaa67fc700 1 -- 172.21.15.102:0/1317304868 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98011830 con 0x7fcaa8153290 2023-03-03T15:31:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaa67fc700 1 -- 172.21.15.102:0/1317304868 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca98011990 con 0x7fcaa8153290 2023-03-03T15:31:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaa67fc700 1 -- 172.21.15.102:0/1317304868 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98011d20 con 0x7fcaa8153290 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaadbff700 1 -- 172.21.15.102:0/1317304868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa8153290 msgr2=0x7fcaa8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaadbff700 1 --2- 172.21.15.102:0/1317304868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa8153290 0x7fcaa8158e80 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fca98009840 tx=0x7fca98009b50 comp rx=0 tx=0).stop 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaadbff700 1 -- 172.21.15.102:0/1317304868 shutdown_connections 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaadbff700 1 --2- 172.21.15.102:0/1317304868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa8153290 0x7fcaa8158e80 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaadbff700 1 --2- 172.21.15.102:0/1317304868 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa81593c0 0x7fcaa80cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaadbff700 1 -- 172.21.15.102:0/1317304868 >> 172.21.15.102:0/1317304868 conn(0x7fcaa8065fb0 msgr2=0x7fcaa80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.379+0000 7fcaadbff700 1 -- 172.21.15.102:0/1317304868 shutdown_connections 2023-03-03T15:31:40.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.380+0000 7fcaadbff700 1 -- 172.21.15.102:0/1317304868 wait complete. 2023-03-03T15:31:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.382+0000 7fcaadbff700 1 Processor -- start 2023-03-03T15:31:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.382+0000 7fcaadbff700 1 -- start start 2023-03-03T15:31:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaadbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa80cd8e0 0x7fcaa80c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaadbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa8153290 0x7fcaa814e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaadbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa81593c0 0x7fcaa814e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaadbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa814dad0 con 0x7fcaa81593c0 2023-03-03T15:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaadbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa814c100 con 0x7fcaa80cd8e0 2023-03-03T15:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaadbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa814c3a0 con 0x7fcaa8153290 2023-03-03T15:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaa7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa81593c0 0x7fcaa814e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaa77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa80cd8e0 0x7fcaa80c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaa7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa81593c0 0x7fcaa814e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52062/0 (socket says 172.21.15.102:52062) 2023-03-03T15:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaa7fff700 1 -- 172.21.15.102:0/840431748 learned_addr learned my addr 172.21.15.102:0/840431748 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.383+0000 7fcaa6ffd700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa8153290 0x7fcaa814e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:40.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa7fff700 1 -- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa8153290 msgr2=0x7fcaa814e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:40.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa7fff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa8153290 0x7fcaa814e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa7fff700 1 -- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa80cd8e0 msgr2=0x7fcaa80c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa7fff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa80cd8e0 0x7fcaa80c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa7fff700 1 -- 172.21.15.102:0/840431748 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca980094a0 con 0x7fcaa81593c0 2023-03-03T15:31:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa77fe700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa80cd8e0 0x7fcaa80c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:31:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa6ffd700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa8153290 0x7fcaa814e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:31:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaa7fff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa81593c0 0x7fcaa814e7a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcaa000cba0 tx=0x7fcaa000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fca8ffff700 1 -- 172.21.15.102:0/840431748 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa000aaa0 con 0x7fcaa81593c0 2023-03-03T15:31:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fca8ffff700 1 -- 172.21.15.102:0/840431748 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa000ac20 con 0x7fcaa81593c0 2023-03-03T15:31:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fca8ffff700 1 -- 172.21.15.102:0/840431748 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa00070d0 con 0x7fcaa81593c0 2023-03-03T15:31:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.384+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa80ccde0 con 0x7fcaa81593c0 2023-03-03T15:31:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.385+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa815d160 con 0x7fcaa81593c0 2023-03-03T15:31:40.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.386+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa80cd0c0 con 0x7fcaa81593c0 2023-03-03T15:31:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.387+0000 7fca8ffff700 1 -- 172.21.15.102:0/840431748 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcaa0010070 con 0x7fcaa81593c0 2023-03-03T15:31:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.388+0000 7fca8ffff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fca940682f0 0x7fca9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.388+0000 7fca8ffff700 1 -- 172.21.15.102:0/840431748 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcaa0085790 con 0x7fcaa81593c0 2023-03-03T15:31:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.389+0000 7fcaa77fe700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fca940682f0 0x7fca9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.389+0000 7fcaa77fe700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fca940682f0 0x7fca9406a7a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fca980090b0 tx=0x7fca98008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.393+0000 7fca8ffff700 1 -- 172.21.15.102:0/840431748 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcaa0051e10 con 0x7fcaa81593c0 2023-03-03T15:31:40.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.631+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaa804e6e0 con 0x7fcaa81593c0 2023-03-03T15:31:40.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.632+0000 7fca8ffff700 1 -- 172.21.15.102:0/840431748 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcaa0056e60 con 0x7fcaa81593c0 2023-03-03T15:31:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:31:40.634 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:31:40.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.634+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fca940682f0 msgr2=0x7fca9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:40.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fca940682f0 0x7fca9406a7a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fca980090b0 tx=0x7fca98008030 comp rx=0 tx=0).stop 2023-03-03T15:31:40.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa81593c0 msgr2=0x7fcaa814e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:40.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa81593c0 0x7fcaa814e7a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcaa000cba0 tx=0x7fcaa000ceb0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 shutdown_connections 2023-03-03T15:31:40.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa81593c0 0x7fcaa814e7a0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa80cd8e0 0x7fcaa80c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fca940682f0 0x7fca9406a7a0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 --2- 172.21.15.102:0/840431748 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcaa8153290 0x7fcaa814e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:40.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 >> 172.21.15.102:0/840431748 conn(0x7fcaa8065fb0 msgr2=0x7fcaa80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:40.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 shutdown_connections 2023-03-03T15:31:40.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:31:40.635+0000 7fcaadbff700 1 -- 172.21.15.102:0/840431748 wait complete. 2023-03-03T15:31:40.638 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:31:40.663 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T15:32:01.966 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:01.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:32.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:32.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:02.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:02.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:17.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:33:17.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.912+0000 7f43605ff700 1 Processor -- start 2023-03-03T15:33:17.914 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.913+0000 7f43605ff700 1 -- start start 2023-03-03T15:33:17.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f43605ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4358153290 0x7f4358158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:17.915 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f43605ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43580cd8e0 0x7f43580ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:17.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f43605ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4358153660 con 0x7f43581593c0 2023-03-03T15:33:17.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f43605ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435806c360 con 0x7f4358153290 2023-03-03T15:33:17.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f43605ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435815cea0 con 0x7f43580cd8e0 2023-03-03T15:33:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f435e39b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4358153290 0x7f4358158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f435db9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f43581593c0 0x7f43580cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59980/0 (socket says 172.21.15.102:59980) 2023-03-03T15:33:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f435db9a700 1 -- 172.21.15.102:0/815630688 learned_addr learned my addr 172.21.15.102:0/815630688 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:33:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.914+0000 7f435eb9c700 1 --2- 172.21.15.102:0/815630688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43580cd8e0 0x7f43580ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.915+0000 7f435e39b700 1 -- 172.21.15.102:0/815630688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43580cd8e0 msgr2=0x7f43580ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:33:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.915+0000 7f435e39b700 1 --2- 172.21.15.102:0/815630688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43580cd8e0 0x7f43580ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.915+0000 7f435e39b700 1 -- 172.21.15.102:0/815630688 >> v1:172.21.15.102:6789/0 conn(0x7f43581593c0 legacy=0x7f43580cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.915+0000 7f435e39b700 1 -- 172.21.15.102:0/815630688 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435806c7f0 con 0x7f4358153290 2023-03-03T15:33:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.915+0000 7f435e39b700 1 --2- 172.21.15.102:0/815630688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4358153290 0x7f4358158e80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4354009840 tx=0x7f4354009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7899625559d0ebc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.915+0000 7f435d399700 1 -- 172.21.15.102:0/815630688 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354011830 con 0x7f4358153290 2023-03-03T15:33:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f435d399700 1 -- 172.21.15.102:0/815630688 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4354011990 con 0x7f4358153290 2023-03-03T15:33:17.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f435d399700 1 -- 172.21.15.102:0/815630688 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354011d20 con 0x7f4358153290 2023-03-03T15:33:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 -- 172.21.15.102:0/815630688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4358153290 msgr2=0x7f4358158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 --2- 172.21.15.102:0/815630688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4358153290 0x7f4358158e80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4354009840 tx=0x7f4354009b50 comp rx=0 tx=0).stop 2023-03-03T15:33:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 -- 172.21.15.102:0/815630688 shutdown_connections 2023-03-03T15:33:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 --2- 172.21.15.102:0/815630688 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43580cd8e0 0x7f43580ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 --2- 172.21.15.102:0/815630688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4358153290 0x7f4358158e80 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 -- 172.21.15.102:0/815630688 >> 172.21.15.102:0/815630688 conn(0x7f4358065fb0 msgr2=0x7f43580663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 -- 172.21.15.102:0/815630688 shutdown_connections 2023-03-03T15:33:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.916+0000 7f43605ff700 1 -- 172.21.15.102:0/815630688 wait complete. 2023-03-03T15:33:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.918+0000 7f43605ff700 1 Processor -- start 2023-03-03T15:33:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.918+0000 7f43605ff700 1 -- start start 2023-03-03T15:33:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.919+0000 7f43605ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43580cd8e0 0x7f4358156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.919+0000 7f43605ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4358153290 0x7f435815b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.919+0000 7f43605ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43581593c0 0x7f4358150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.919+0000 7f43605ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43580b44e0 con 0x7f43581593c0 2023-03-03T15:33:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.919+0000 7f43605ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43581ee5d0 con 0x7f43580cd8e0 2023-03-03T15:33:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.919+0000 7f43605ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43581ee8b0 con 0x7f4358153290 2023-03-03T15:33:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435db9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4358153290 0x7f435815b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43580cd8e0 0x7f4358156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43580cd8e0 0x7f4358156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48842/0 (socket says 172.21.15.102:48842) 2023-03-03T15:33:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435db9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4358153290 0x7f435815b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:33850/0 (socket says 172.21.15.102:33850) 2023-03-03T15:33:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 -- 172.21.15.102:0/2094929211 learned_addr learned my addr 172.21.15.102:0/2094929211 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:33:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435eb9c700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43581593c0 0x7f4358150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 -- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4358153290 msgr2=0x7f435815b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4358153290 0x7f435815b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 -- 172.21.15.102:0/2094929211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43581593c0 msgr2=0x7f4358150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:17.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43581593c0 0x7f4358150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435e39b700 1 -- 172.21.15.102:0/2094929211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43540094a0 con 0x7f43580cd8e0 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.920+0000 7f435eb9c700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43581593c0 0x7f4358150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.921+0000 7f435e39b700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43580cd8e0 0x7f4358156290 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fd0 tx=0x7f4354004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.921+0000 7f434effd700 1 -- 172.21.15.102:0/2094929211 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43540036a0 con 0x7f43580cd8e0 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.921+0000 7f434effd700 1 -- 172.21.15.102:0/2094929211 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435401e070 con 0x7f43580cd8e0 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.921+0000 7f434effd700 1 -- 172.21.15.102:0/2094929211 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354003a40 con 0x7f43580cd8e0 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.921+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43580ccde0 con 0x7f43580cd8e0 2023-03-03T15:33:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.921+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435815d160 con 0x7f43580cd8e0 2023-03-03T15:33:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.922+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43580cd0c0 con 0x7f43580cd8e0 2023-03-03T15:33:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.924+0000 7f434effd700 1 -- 172.21.15.102:0/2094929211 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f435400f050 con 0x7f43580cd8e0 2023-03-03T15:33:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.925+0000 7f434effd700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f433c0682f0 0x7f433c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.925+0000 7f434effd700 1 -- 172.21.15.102:0/2094929211 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4354086cf0 con 0x7f43580cd8e0 2023-03-03T15:33:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.925+0000 7f435db9a700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f433c0682f0 0x7f433c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.926+0000 7f435db9a700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f433c0682f0 0x7f433c06a7a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f435815b890 tx=0x7f4348009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:17.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:17.929+0000 7f434effd700 1 -- 172.21.15.102:0/2094929211 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4354053380 con 0x7f43580cd8e0 2023-03-03T15:33:18.163 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.162+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43581ee710 con 0x7f43580cd8e0 2023-03-03T15:33:18.164 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.163+0000 7f434effd700 1 -- 172.21.15.102:0/2094929211 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f43581ee710 con 0x7f43580cd8e0 2023-03-03T15:33:18.164 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:33:18.165 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:33:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f433c0682f0 msgr2=0x7f433c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f433c0682f0 0x7f433c06a7a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f435815b890 tx=0x7f4348009040 comp rx=0 tx=0).stop 2023-03-03T15:33:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43580cd8e0 msgr2=0x7f4358156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43580cd8e0 0x7f4358156290 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fd0 tx=0x7f4354004750 comp rx=0 tx=0).stop 2023-03-03T15:33:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 shutdown_connections 2023-03-03T15:33:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43581593c0 0x7f4358150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43580cd8e0 0x7f4358156290 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f433c0682f0 0x7f433c06a7a0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 --2- 172.21.15.102:0/2094929211 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4358153290 0x7f435815b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 >> 172.21.15.102:0/2094929211 conn(0x7f4358065fb0 msgr2=0x7f4358156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:18.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 shutdown_connections 2023-03-03T15:33:18.169 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:33:18.166+0000 7f43605ff700 1 -- 172.21.15.102:0/2094929211 wait complete. 2023-03-03T15:33:18.169 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:33:18.194 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-03-03T15:33:32.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:32.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:02.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:02.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:11.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:34:11.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.447+0000 7fd6e01ff700 1 Processor -- start 2023-03-03T15:34:11.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.448+0000 7fd6e01ff700 1 -- start start 2023-03-03T15:34:11.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6e01ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6d8153290 0x7fd6d8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:11.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6e01ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:11.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6e01ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d8153660 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6e01ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d806c360 con 0x7fd6d8153290 2023-03-03T15:34:11.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6e01ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d815cea0 con 0x7fd6d81593c0 2023-03-03T15:34:11.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6ddf9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6d8153290 0x7fd6d8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:11.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6de79c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd6d80cd8e0 0x7fd6d80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58366/0 (socket says 172.21.15.102:58366) 2023-03-03T15:34:11.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6de79c700 1 -- 172.21.15.102:0/2103946941 learned_addr learned my addr 172.21.15.102:0/2103946941 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:34:11.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.449+0000 7fd6dd79a700 1 --2- 172.21.15.102:0/2103946941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.450+0000 7fd6ddf9b700 1 -- 172.21.15.102:0/2103946941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 msgr2=0x7fd6d80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:34:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.450+0000 7fd6ddf9b700 1 --2- 172.21.15.102:0/2103946941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.450+0000 7fd6ddf9b700 1 -- 172.21.15.102:0/2103946941 >> v1:172.21.15.102:6789/0 conn(0x7fd6d80cd8e0 legacy=0x7fd6d80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.450+0000 7fd6ddf9b700 1 -- 172.21.15.102:0/2103946941 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d806c7f0 con 0x7fd6d8153290 2023-03-03T15:34:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.450+0000 7fd6dcf99700 1 -- 172.21.15.102:0/2103946941 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930032722 0 0) 0x7fd6d8153660 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.450+0000 7fd6ddf9b700 1 --2- 172.21.15.102:0/2103946941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d8153290 0x7fd6d8158e80 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009840 tx=0x7fd6d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ec7654b6e9c6d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:34:11.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6dcf99700 1 -- 172.21.15.102:0/2103946941 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4011830 con 0x7fd6d8153290 2023-03-03T15:34:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6dcf99700 1 -- 172.21.15.102:0/2103946941 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4011990 con 0x7fd6d8153290 2023-03-03T15:34:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6dcf99700 1 -- 172.21.15.102:0/2103946941 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4011d20 con 0x7fd6d8153290 2023-03-03T15:34:11.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2103946941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d8153290 msgr2=0x7fd6d8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2103946941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d8153290 0x7fd6d8158e80 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009840 tx=0x7fd6d4009b50 comp rx=0 tx=0).stop 2023-03-03T15:34:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2103946941 shutdown_connections 2023-03-03T15:34:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2103946941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d8153290 0x7fd6d8158e80 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2103946941 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2103946941 >> 172.21.15.102:0/2103946941 conn(0x7fd6d8065fb0 msgr2=0x7fd6d80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:34:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2103946941 shutdown_connections 2023-03-03T15:34:11.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.451+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2103946941 wait complete. 2023-03-03T15:34:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.453+0000 7fd6e01ff700 1 Processor -- start 2023-03-03T15:34:11.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.454+0000 7fd6e01ff700 1 -- start start 2023-03-03T15:34:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6e01ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d80cd8e0 0x7fd6d81e3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6e01ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6d8153290 0x7fd6d80cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6e01ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6e01ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80b4c60 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6e01ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d815bf90 con 0x7fd6d8153290 2023-03-03T15:34:11.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6e01ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80b1d10 con 0x7fd6d81593c0 2023-03-03T15:34:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6ddf9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d80cd8e0 0x7fd6d81e3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6dd79a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6d8153290 0x7fd6d80cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6dd79a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6d8153290 0x7fd6d80cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56678/0 (socket says 172.21.15.102:56678) 2023-03-03T15:34:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6de79c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.455+0000 7fd6dd79a700 1 -- 172.21.15.102:0/2080361047 learned_addr learned my addr 172.21.15.102:0/2080361047 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:34:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ddf9b700 1 -- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 msgr2=0x7fd6d80cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ddf9b700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ddf9b700 1 -- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6d8153290 msgr2=0x7fd6d80cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:11.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ddf9b700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6d8153290 0x7fd6d80cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ddf9b700 1 -- 172.21.15.102:0/2080361047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d40094a0 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6dd79a700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6d8153290 0x7fd6d80cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ddf9b700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d80cd8e0 0x7fd6d81e3ec0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4016000 tx=0x7fd6d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ce7fc700 1 -- 172.21.15.102:0/2080361047 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d40036a0 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ce7fc700 1 -- 172.21.15.102:0/2080361047 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d401e070 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d80ccde0 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.456+0000 7fd6ce7fc700 1 -- 172.21.15.102:0/2080361047 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4003a60 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.457+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d815d160 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.458+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d80cd0c0 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.459+0000 7fd6ce7fc700 1 -- 172.21.15.102:0/2080361047 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd6d400f050 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.460+0000 7fd6ce7fc700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6bc0682f0 0x7fd6bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.460+0000 7fd6ce7fc700 1 -- 172.21.15.102:0/2080361047 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd6d4086eb0 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.460+0000 7fd6dd79a700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6bc0682f0 0x7fd6bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.461+0000 7fd6dd79a700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6bc0682f0 0x7fd6bc06a7a0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8066ed0 tx=0x7fd6c8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:34:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.464+0000 7fd6ce7fc700 1 -- 172.21.15.102:0/2080361047 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd6d4053540 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.703 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.701+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6d804e6e0 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.702+0000 7fd6ce7fc700 1 -- 172.21.15.102:0/2080361047 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd6d4020090 con 0x7fd6d80cd8e0 2023-03-03T15:34:11.704 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:34:11.704 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:34:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6bc0682f0 msgr2=0x7fd6bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6bc0682f0 0x7fd6bc06a7a0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8066ed0 tx=0x7fd6c8009210 comp rx=0 tx=0).stop 2023-03-03T15:34:11.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d80cd8e0 msgr2=0x7fd6d81e3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d80cd8e0 0x7fd6d81e3ec0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4016000 tx=0x7fd6d4004750 comp rx=0 tx=0).stop 2023-03-03T15:34:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 shutdown_connections 2023-03-03T15:34:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd6bc0682f0 0x7fd6bc06a7a0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd6d81593c0 0x7fd6d80cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6d80cd8e0 0x7fd6d81e3ec0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 --2- 172.21.15.102:0/2080361047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd6d8153290 0x7fd6d80cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:11.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 >> 172.21.15.102:0/2080361047 conn(0x7fd6d8065fb0 msgr2=0x7fd6d815b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:34:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 shutdown_connections 2023-03-03T15:34:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:34:11.705+0000 7fd6e01ff700 1 -- 172.21.15.102:0/2080361047 wait complete. 2023-03-03T15:34:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:34:11.733 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 72 secs before thrashing 2023-03-03T15:34:32.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:32.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:02.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:02.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:23.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:35:23.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.985+0000 7f2bb47ff700 1 Processor -- start 2023-03-03T15:35:23.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.985+0000 7f2bb47ff700 1 -- start start 2023-03-03T15:35:23.987 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.986+0000 7f2bb47ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac153290 0x7f2bac158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:23.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.986+0000 7f2bb47ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2bac0cd8e0 0x7f2bac0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:23.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.986+0000 7f2bb47ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac153660 con 0x7f2bac1593c0 2023-03-03T15:35:23.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.986+0000 7f2bb47ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac06c360 con 0x7f2bac0cd8e0 2023-03-03T15:35:23.988 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.986+0000 7f2bb47ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac15cea0 con 0x7f2bac153290 2023-03-03T15:35:23.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb259b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac153290 0x7f2bac158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:23.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb1d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2bac1593c0 0x7f2bac0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44028/0 (socket says 172.21.15.102:44028) 2023-03-03T15:35:23.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb2d9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2bac0cd8e0 0x7f2bac0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:23.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb1d9a700 1 -- 172.21.15.102:0/3738411755 learned_addr learned my addr 172.21.15.102:0/3738411755 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:35:23.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb1599700 1 -- 172.21.15.102:0/3738411755 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844462324 0 0) 0x7f2bac153660 con 0x7f2bac1593c0 2023-03-03T15:35:23.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb1599700 1 -- 172.21.15.102:0/3738411755 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b980035b0 con 0x7f2bac1593c0 2023-03-03T15:35:23.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb2d9c700 1 -- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac153290 msgr2=0x7f2bac158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb2d9c700 1 --2- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac153290 0x7f2bac158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb2d9c700 1 -- 172.21.15.102:0/3738411755 >> v1:172.21.15.172:6789/0 conn(0x7f2bac1593c0 legacy=0x7f2bac0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb2d9c700 1 -- 172.21.15.102:0/3738411755 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac06c7f0 con 0x7f2bac0cd8e0 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb259b700 1 --2- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac153290 0x7f2bac158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.987+0000 7f2bb2d9c700 1 --2- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac0cd8e0 0x7f2bac0ce530 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009d90 tx=0x7f2ba400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5f5279b4c257168 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb1599700 1 -- 172.21.15.102:0/3738411755 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400e070 con 0x7f2bac0cd8e0 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb1599700 1 -- 172.21.15.102:0/3738411755 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba400bb20 con 0x7f2bac0cd8e0 2023-03-03T15:35:23.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb1599700 1 -- 172.21.15.102:0/3738411755 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4012430 con 0x7f2bac0cd8e0 2023-03-03T15:35:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 -- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac0cd8e0 msgr2=0x7f2bac0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac0cd8e0 0x7f2bac0ce530 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009d90 tx=0x7f2ba400b330 comp rx=0 tx=0).stop 2023-03-03T15:35:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 -- 172.21.15.102:0/3738411755 shutdown_connections 2023-03-03T15:35:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac0cd8e0 0x7f2bac0ce530 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/3738411755 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac153290 0x7f2bac158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 -- 172.21.15.102:0/3738411755 >> 172.21.15.102:0/3738411755 conn(0x7f2bac065fb0 msgr2=0x7f2bac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:35:23.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 -- 172.21.15.102:0/3738411755 shutdown_connections 2023-03-03T15:35:23.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.988+0000 7f2bb47ff700 1 -- 172.21.15.102:0/3738411755 wait complete. 2023-03-03T15:35:23.993 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.991+0000 7f2bb47ff700 1 Processor -- start 2023-03-03T15:35:23.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.991+0000 7f2bb47ff700 1 -- start start 2023-03-03T15:35:23.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb47ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac0cd8e0 0x7f2bac0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:23.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb47ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac153290 0x7f2bac153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:23.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb47ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac1593c0 0x7f2bac0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:23.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb47ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0c3d20 con 0x7f2bac0cd8e0 2023-03-03T15:35:23.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb47ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0c8380 con 0x7f2bac153290 2023-03-03T15:35:23.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb47ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0c8660 con 0x7f2bac1593c0 2023-03-03T15:35:23.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb259b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac0cd8e0 0x7f2bac0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:23.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb1d9a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac153290 0x7f2bac153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:23.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb2d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac1593c0 0x7f2bac0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:23.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb2d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac1593c0 0x7f2bac0c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39872/0 (socket says 172.21.15.102:39872) 2023-03-03T15:35:23.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.992+0000 7f2bb2d9c700 1 -- 172.21.15.102:0/741391595 learned_addr learned my addr 172.21.15.102:0/741391595 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:35:23.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb2d9c700 1 -- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac153290 msgr2=0x7f2bac153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:23.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb2d9c700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac153290 0x7f2bac153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb2d9c700 1 -- 172.21.15.102:0/741391595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac0cd8e0 msgr2=0x7f2bac0c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb2d9c700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac0cd8e0 0x7f2bac0c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb2d9c700 1 -- 172.21.15.102:0/741391595 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba80094a0 con 0x7f2bac1593c0 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb2d9c700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac1593c0 0x7f2bac0c41b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4014040 tx=0x7f2ba4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2ba2ffd700 1 -- 172.21.15.102:0/741391595 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba40129f0 con 0x7f2bac1593c0 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2ba2ffd700 1 -- 172.21.15.102:0/741391595 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4020070 con 0x7f2bac1593c0 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2ba2ffd700 1 -- 172.21.15.102:0/741391595 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4012db0 con 0x7f2bac1593c0 2023-03-03T15:35:23.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba4009a70 con 0x7f2bac1593c0 2023-03-03T15:35:23.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.993+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac0ccde0 con 0x7f2bac1593c0 2023-03-03T15:35:23.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.995+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bac15d160 con 0x7f2bac1593c0 2023-03-03T15:35:23.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.996+0000 7f2ba2ffd700 1 -- 172.21.15.102:0/741391595 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ba402e070 con 0x7f2bac1593c0 2023-03-03T15:35:23.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.997+0000 7f2ba2ffd700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b900682f0 0x7f2b9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:23.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.997+0000 7f2ba2ffd700 1 -- 172.21.15.102:0/741391595 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2ba4022070 con 0x7f2bac1593c0 2023-03-03T15:35:23.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.997+0000 7f2bb259b700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b900682f0 0x7f2b9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:23.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:23.998+0000 7f2bb259b700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b900682f0 0x7f2b9006a7a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009d70 tx=0x7f2ba8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:35:24.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.002+0000 7f2ba2ffd700 1 -- 172.21.15.102:0/741391595 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ba4054050 con 0x7f2bac1593c0 2023-03-03T15:35:24.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.234+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bac0cd0c0 con 0x7f2bac1593c0 2023-03-03T15:35:24.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.235+0000 7f2ba2ffd700 1 -- 172.21.15.102:0/741391595 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2ba402c090 con 0x7f2bac1593c0 2023-03-03T15:35:24.237 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:35:24.237 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:35:24.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.237+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b900682f0 msgr2=0x7f2b9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:24.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.237+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b900682f0 0x7f2b9006a7a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009d70 tx=0x7f2ba8009210 comp rx=0 tx=0).stop 2023-03-03T15:35:24.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac1593c0 msgr2=0x7f2bac0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac1593c0 0x7f2bac0c41b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4014040 tx=0x7f2ba4004ad0 comp rx=0 tx=0).stop 2023-03-03T15:35:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 shutdown_connections 2023-03-03T15:35:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b900682f0 0x7f2b9006a7a0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2bac1593c0 0x7f2bac0c41b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac0cd8e0 0x7f2bac0c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 --2- 172.21.15.102:0/741391595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2bac153290 0x7f2bac153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 >> 172.21.15.102:0/741391595 conn(0x7f2bac065fb0 msgr2=0x7f2bac15be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:35:24.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 shutdown_connections 2023-03-03T15:35:24.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:35:24.238+0000 7f2bb47ff700 1 -- 172.21.15.102:0/741391595 wait complete. 2023-03-03T15:35:24.241 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:35:24.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-03-03T15:35:32.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:32.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:03.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:03.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:33.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:33.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:03.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:03.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:11.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:37:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.515+0000 7fb2f119d700 1 Processor -- start 2023-03-03T15:37:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.515+0000 7fb2f119d700 1 -- start start 2023-03-03T15:37:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2f119d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0ced00 0x7fb2ec0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2f119d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb2ec0b3970 0x7fb2ec0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2f119d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec0c3d20 con 0x7fb2ec0cf0d0 2023-03-03T15:37:11.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2f119d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec0c4930 con 0x7fb2ec0b3970 2023-03-03T15:37:11.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2f119d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec15cba0 con 0x7fb2ec0ced00 2023-03-03T15:37:11.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2f099c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb2ec0cf0d0 0x7fb2ec158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45014/0 (socket says 172.21.15.102:45014) 2023-03-03T15:37:11.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2eb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb2ec0b3970 0x7fb2ec0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:11.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2f099c700 1 -- 172.21.15.102:0/235842476 learned_addr learned my addr 172.21.15.102:0/235842476 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:37:11.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.517+0000 7fb2ebfff700 1 --2- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0ced00 0x7fb2ec0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:11.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eaffd700 1 -- 172.21.15.102:0/235842476 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442684284 0 0) 0x7fb2ec0c3d20 con 0x7fb2ec0cf0d0 2023-03-03T15:37:11.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eb7fe700 1 -- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0ced00 msgr2=0x7fb2ec0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:37:11.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eb7fe700 1 --2- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0ced00 0x7fb2ec0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eb7fe700 1 -- 172.21.15.102:0/235842476 >> v1:172.21.15.102:6789/0 conn(0x7fb2ec0cf0d0 legacy=0x7fb2ec158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eb7fe700 1 -- 172.21.15.102:0/235842476 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec15ce80 con 0x7fb2ec0b3970 2023-03-03T15:37:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eb7fe700 1 --2- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0b3970 0x7fb2ec0c41b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009840 tx=0x7fb2e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b00ed280fa039e12 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eaffd700 1 -- 172.21.15.102:0/235842476 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0011830 con 0x7fb2ec0b3970 2023-03-03T15:37:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.518+0000 7fb2eaffd700 1 -- 172.21.15.102:0/235842476 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e0011990 con 0x7fb2ec0b3970 2023-03-03T15:37:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2eaffd700 1 -- 172.21.15.102:0/235842476 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0011d20 con 0x7fb2ec0b3970 2023-03-03T15:37:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 -- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0b3970 msgr2=0x7fb2ec0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 --2- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0b3970 0x7fb2ec0c41b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009840 tx=0x7fb2e0009b50 comp rx=0 tx=0).stop 2023-03-03T15:37:11.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 -- 172.21.15.102:0/235842476 shutdown_connections 2023-03-03T15:37:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 --2- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0ced00 0x7fb2ec0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 --2- 172.21.15.102:0/235842476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0b3970 0x7fb2ec0c41b0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 -- 172.21.15.102:0/235842476 >> 172.21.15.102:0/235842476 conn(0x7fb2ec08b1c0 msgr2=0x7fb2ec071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 -- 172.21.15.102:0/235842476 shutdown_connections 2023-03-03T15:37:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.519+0000 7fb2f119d700 1 -- 172.21.15.102:0/235842476 wait complete. 2023-03-03T15:37:11.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.521+0000 7fb2f119d700 1 Processor -- start 2023-03-03T15:37:11.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.521+0000 7fb2f119d700 1 -- start start 2023-03-03T15:37:11.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.522+0000 7fb2f119d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0b3970 0x7fb2ec145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.522+0000 7fb2f119d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0ced00 0x7fb2ec14e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.522+0000 7fb2f119d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2ec0cf0d0 0x7fb2ec1eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.522+0000 7fb2f119d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec145fc0 con 0x7fb2ec0cf0d0 2023-03-03T15:37:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.522+0000 7fb2f119d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1ee960 con 0x7fb2ec0ced00 2023-03-03T15:37:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.522+0000 7fb2f119d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec14d8a0 con 0x7fb2ec0b3970 2023-03-03T15:37:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2f099c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2ec0cf0d0 0x7fb2ec1eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:11.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0ced00 0x7fb2ec14e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2ebfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0b3970 0x7fb2ec145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2ebfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0b3970 0x7fb2ec145a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54964/0 (socket says 172.21.15.102:54964) 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2eb7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0ced00 0x7fb2ec14e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59236/0 (socket says 172.21.15.102:59236) 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2ebfff700 1 -- 172.21.15.102:0/2668001916 learned_addr learned my addr 172.21.15.102:0/2668001916 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2eb7fe700 1 -- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0b3970 msgr2=0x7fb2ec145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2eb7fe700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0b3970 0x7fb2ec145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2eb7fe700 1 -- 172.21.15.102:0/2668001916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2ec0cf0d0 msgr2=0x7fb2ec1eedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:11.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2eb7fe700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2ec0cf0d0 0x7fb2ec1eedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2eb7fe700 1 -- 172.21.15.102:0/2668001916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00094a0 con 0x7fb2ec0ced00 2023-03-03T15:37:11.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.523+0000 7fb2ebfff700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0b3970 0x7fb2ec145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:37:11.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.524+0000 7fb2eb7fe700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0ced00 0x7fb2ec14e5d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0016000 tx=0x7fb2e0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.524+0000 7fb2e8ff9700 1 -- 172.21.15.102:0/2668001916 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00045d0 con 0x7fb2ec0ced00 2023-03-03T15:37:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.524+0000 7fb2e8ff9700 1 -- 172.21.15.102:0/2668001916 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e001e070 con 0x7fb2ec0ced00 2023-03-03T15:37:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.524+0000 7fb2e8ff9700 1 -- 172.21.15.102:0/2668001916 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00038a0 con 0x7fb2ec0ced00 2023-03-03T15:37:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.524+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec06c200 con 0x7fb2ec0ced00 2023-03-03T15:37:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.524+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec06bf20 con 0x7fb2ec0ced00 2023-03-03T15:37:11.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.525+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec15c890 con 0x7fb2ec0ced00 2023-03-03T15:37:11.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.526+0000 7fb2e8ff9700 1 -- 172.21.15.102:0/2668001916 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb2e000f050 con 0x7fb2ec0ced00 2023-03-03T15:37:11.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.528+0000 7fb2e8ff9700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2d80682f0 0x7fb2d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:11.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.528+0000 7fb2e8ff9700 1 -- 172.21.15.102:0/2668001916 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb2e0086c90 con 0x7fb2ec0ced00 2023-03-03T15:37:11.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.528+0000 7fb2ebfff700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2d80682f0 0x7fb2d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:11.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.529+0000 7fb2ebfff700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2d80682f0 0x7fb2d806a7a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0113d0 tx=0x7fb2dc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:11.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.532+0000 7fb2e8ff9700 1 -- 172.21.15.102:0/2668001916 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2e00533a0 con 0x7fb2ec0ced00 2023-03-03T15:37:11.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.768+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2ec0caf00 con 0x7fb2ec0ced00 2023-03-03T15:37:11.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.769+0000 7fb2e8ff9700 1 -- 172.21.15.102:0/2668001916 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb2e0020090 con 0x7fb2ec0ced00 2023-03-03T15:37:11.770 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:37:11.770 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:37:11.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2d80682f0 msgr2=0x7fb2d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:11.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2d80682f0 0x7fb2d806a7a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0113d0 tx=0x7fb2dc009210 comp rx=0 tx=0).stop 2023-03-03T15:37:11.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0ced00 msgr2=0x7fb2ec14e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:11.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0ced00 0x7fb2ec14e5d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0016000 tx=0x7fb2e0004750 comp rx=0 tx=0).stop 2023-03-03T15:37:11.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 shutdown_connections 2023-03-03T15:37:11.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb2d80682f0 0x7fb2d806a7a0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2ec0b3970 0x7fb2ec145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2ec0cf0d0 0x7fb2ec1eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 --2- 172.21.15.102:0/2668001916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2ec0ced00 0x7fb2ec14e5d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:11.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 >> 172.21.15.102:0/2668001916 conn(0x7fb2ec08b1c0 msgr2=0x7fb2ec071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:11.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 shutdown_connections 2023-03-03T15:37:11.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:11.771+0000 7fb2f119d700 1 -- 172.21.15.102:0/2668001916 wait complete. 2023-03-03T15:37:11.775 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:37:11.800 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2023-03-03T15:37:28.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:37:29.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.057+0000 7fc70e3ff700 1 Processor -- start 2023-03-03T15:37:29.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.057+0000 7fc70e3ff700 1 -- start start 2023-03-03T15:37:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.058+0000 7fc70e3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc708153290 0x7fc708158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:29.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.058+0000 7fc70e3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc7081593c0 0x7fc7080cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:29.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.058+0000 7fc70e3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc708153660 con 0x7fc7080cd8e0 2023-03-03T15:37:29.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.058+0000 7fc70e3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70806c360 con 0x7fc708153290 2023-03-03T15:37:29.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.058+0000 7fc70e3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70815cea0 con 0x7fc7081593c0 2023-03-03T15:37:29.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc707fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc708153290 0x7fc708158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:29.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc7077fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc7081593c0 0x7fc7080cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:29.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc707fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc708153290 0x7fc708158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42074/0 (socket says 172.21.15.102:42074) 2023-03-03T15:37:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc707fff700 1 -- 172.21.15.102:0/258536056 learned_addr learned my addr 172.21.15.102:0/258536056 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:37:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc707fff700 1 -- 172.21.15.102:0/258536056 >> v2:172.21.15.172:3300/0 conn(0x7fc7081593c0 msgr2=0x7fc7080cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:37:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc707fff700 1 --2- 172.21.15.102:0/258536056 >> v2:172.21.15.172:3300/0 conn(0x7fc7081593c0 0x7fc7080cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc707fff700 1 -- 172.21.15.102:0/258536056 >> v1:172.21.15.102:6789/0 conn(0x7fc7080cd8e0 legacy=0x7fc7080ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc707fff700 1 -- 172.21.15.102:0/258536056 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70806c7f0 con 0x7fc708153290 2023-03-03T15:37:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.059+0000 7fc7077fe700 1 --2- 172.21.15.102:0/258536056 >> v2:172.21.15.172:3300/0 conn(0x7fc7081593c0 0x7fc7080cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:37:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.060+0000 7fc707fff700 1 --2- 172.21.15.102:0/258536056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc708153290 0x7fc708158e80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc708159fb0 tx=0x7fc6f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afc5621ad06123a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.060+0000 7fc706ffd700 1 -- 172.21.15.102:0/258536056 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8011830 con 0x7fc708153290 2023-03-03T15:37:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.060+0000 7fc706ffd700 1 -- 172.21.15.102:0/258536056 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f8011990 con 0x7fc708153290 2023-03-03T15:37:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.060+0000 7fc706ffd700 1 -- 172.21.15.102:0/258536056 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8011d20 con 0x7fc708153290 2023-03-03T15:37:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.060+0000 7fc70e3ff700 1 -- 172.21.15.102:0/258536056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc708153290 msgr2=0x7fc708158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.060+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/258536056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc708153290 0x7fc708158e80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc708159fb0 tx=0x7fc6f8009bd0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.061+0000 7fc70e3ff700 1 -- 172.21.15.102:0/258536056 shutdown_connections 2023-03-03T15:37:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.061+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/258536056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc708153290 0x7fc708158e80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.061+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/258536056 >> v2:172.21.15.172:3300/0 conn(0x7fc7081593c0 0x7fc7080cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.061+0000 7fc70e3ff700 1 -- 172.21.15.102:0/258536056 >> 172.21.15.102:0/258536056 conn(0x7fc708065fb0 msgr2=0x7fc7080663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.061+0000 7fc70e3ff700 1 -- 172.21.15.102:0/258536056 shutdown_connections 2023-03-03T15:37:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.061+0000 7fc70e3ff700 1 -- 172.21.15.102:0/258536056 wait complete. 2023-03-03T15:37:29.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.063+0000 7fc70e3ff700 1 Processor -- start 2023-03-03T15:37:29.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.063+0000 7fc70e3ff700 1 -- start start 2023-03-03T15:37:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70e3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7080cd8e0 0x7fc70814c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70e3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc708153290 0x7fc708155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70e3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7081593c0 0x7fc708156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70e3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7081ee930 con 0x7fc7081593c0 2023-03-03T15:37:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70e3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7080b45f0 con 0x7fc708153290 2023-03-03T15:37:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70e3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7080b1d10 con 0x7fc7080cd8e0 2023-03-03T15:37:29.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70c99c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7081593c0 0x7fc708156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:29.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc7077fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc708153290 0x7fc708155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc707fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7080cd8e0 0x7fc70814c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc7077fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc708153290 0x7fc708155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:54366/0 (socket says 172.21.15.102:54366) 2023-03-03T15:37:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc70c99c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7081593c0 0x7fc708156200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42080/0 (socket says 172.21.15.102:42080) 2023-03-03T15:37:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.064+0000 7fc7077fe700 1 -- 172.21.15.102:0/246716985 learned_addr learned my addr 172.21.15.102:0/246716985 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:37:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc707fff700 1 -- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc708153290 msgr2=0x7fc708155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc707fff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc708153290 0x7fc708155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc707fff700 1 -- 172.21.15.102:0/246716985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7081593c0 msgr2=0x7fc708156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc707fff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7081593c0 0x7fc708156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc707fff700 1 -- 172.21.15.102:0/246716985 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0094a0 con 0x7fc7080cd8e0 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc70c99c700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7081593c0 0x7fc708156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc7077fe700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc708153290 0x7fc708155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.065+0000 7fc707fff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7080cd8e0 0x7fc70814c770 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8016000 tx=0x7fc6f8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.066+0000 7fc704ff9700 1 -- 172.21.15.102:0/246716985 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8004380 con 0x7fc7080cd8e0 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.066+0000 7fc704ff9700 1 -- 172.21.15.102:0/246716985 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f801f070 con 0x7fc7080cd8e0 2023-03-03T15:37:29.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.066+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f8009550 con 0x7fc7080cd8e0 2023-03-03T15:37:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.066+0000 7fc704ff9700 1 -- 172.21.15.102:0/246716985 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80036a0 con 0x7fc7080cd8e0 2023-03-03T15:37:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.066+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7080ccde0 con 0x7fc7080cd8e0 2023-03-03T15:37:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.067+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70815d160 con 0x7fc7080cd8e0 2023-03-03T15:37:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.068+0000 7fc704ff9700 1 -- 172.21.15.102:0/246716985 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc6f802a020 con 0x7fc7080cd8e0 2023-03-03T15:37:29.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.069+0000 7fc704ff9700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6f40682f0 0x7fc6f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:29.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.069+0000 7fc704ff9700 1 -- 172.21.15.102:0/246716985 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc6f8087c00 con 0x7fc7080cd8e0 2023-03-03T15:37:29.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.070+0000 7fc7077fe700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6f40682f0 0x7fc6f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:29.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.071+0000 7fc7077fe700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6f40682f0 0x7fc6f406a7a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fc70815a4d0 tx=0x7fc6fc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:29.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.072+0000 7fc704ff9700 1 -- 172.21.15.102:0/246716985 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6f8054310 con 0x7fc7080cd8e0 2023-03-03T15:37:29.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.311+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7080cd0c0 con 0x7fc7080cd8e0 2023-03-03T15:37:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.312+0000 7fc704ff9700 1 -- 172.21.15.102:0/246716985 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc6f8015090 con 0x7fc7080cd8e0 2023-03-03T15:37:29.315 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:37:29.315 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:37:29.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6f40682f0 msgr2=0x7fc6f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6f40682f0 0x7fc6f406a7a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fc70815a4d0 tx=0x7fc6fc009210 comp rx=0 tx=0).stop 2023-03-03T15:37:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7080cd8e0 msgr2=0x7fc70814c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7080cd8e0 0x7fc70814c770 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8016000 tx=0x7fc6f8004750 comp rx=0 tx=0).stop 2023-03-03T15:37:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 shutdown_connections 2023-03-03T15:37:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc6f40682f0 0x7fc6f406a7a0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc7080cd8e0 0x7fc70814c770 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7081593c0 0x7fc708156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 --2- 172.21.15.102:0/246716985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc708153290 0x7fc708155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 >> 172.21.15.102:0/246716985 conn(0x7fc708065fb0 msgr2=0x7fc7080cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 shutdown_connections 2023-03-03T15:37:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:37:29.315+0000 7fc70e3ff700 1 -- 172.21.15.102:0/246716985 wait complete. 2023-03-03T15:37:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:37:29.345 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T15:37:33.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:33.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:03.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:03.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:34.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:34.189 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:36.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:38:36.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.595+0000 7fb83fbff700 1 Processor -- start 2023-03-03T15:38:36.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.595+0000 7fb83fbff700 1 -- start start 2023-03-03T15:38:36.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.596+0000 7fb83fbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8380cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.596+0000 7fb83fbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb838153660 con 0x7fb838153290 2023-03-03T15:38:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.596+0000 7fb83fbff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83806c360 con 0x7fb8380cd390 2023-03-03T15:38:36.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.596+0000 7fb83fbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83815cea0 con 0x7fb8380cd8e0 2023-03-03T15:38:36.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83d99b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb838153290 0x7fb838158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35940/0 (socket says 172.21.15.102:35940) 2023-03-03T15:38:36.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83d99b700 1 -- 172.21.15.102:0/1922455597 learned_addr learned my addr 172.21.15.102:0/1922455597 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:38:36.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83d19a700 1 --2- 172.21.15.102:0/1922455597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8380cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:36.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83c999700 1 -- 172.21.15.102:0/1922455597 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923749417 0 0) 0x7fb83806c360 con 0x7fb8380cd390 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83c999700 1 -- 172.21.15.102:0/1922455597 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8240035b0 con 0x7fb8380cd390 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83c999700 1 -- 172.21.15.102:0/1922455597 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43114458 0 0) 0x7fb838153660 con 0x7fb838153290 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83c999700 1 -- 172.21.15.102:0/1922455597 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb83806c360 con 0x7fb838153290 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83d19a700 1 -- 172.21.15.102:0/1922455597 >> v1:172.21.15.172:6789/0 conn(0x7fb8380cd390 legacy=0x7fb8380ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83d19a700 1 -- 172.21.15.102:0/1922455597 >> v1:172.21.15.102:6789/0 conn(0x7fb838153290 legacy=0x7fb838158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.597+0000 7fb83d19a700 1 -- 172.21.15.102:0/1922455597 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83806c7f0 con 0x7fb8380cd8e0 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.598+0000 7fb83d19a700 1 --2- 172.21.15.102:0/1922455597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8380cde20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb828009870 tx=0x7fb828009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8775ef28017af00e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:36.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.598+0000 7fb83c999700 1 -- 172.21.15.102:0/1922455597 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb828011830 con 0x7fb8380cd8e0 2023-03-03T15:38:36.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.598+0000 7fb83c999700 1 -- 172.21.15.102:0/1922455597 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb828011990 con 0x7fb8380cd8e0 2023-03-03T15:38:36.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.598+0000 7fb83c999700 1 -- 172.21.15.102:0/1922455597 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb828011d20 con 0x7fb8380cd8e0 2023-03-03T15:38:36.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.598+0000 7fb83fbff700 1 -- 172.21.15.102:0/1922455597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 msgr2=0x7fb8380cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:36.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.598+0000 7fb83fbff700 1 --2- 172.21.15.102:0/1922455597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8380cde20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb828009870 tx=0x7fb828009b80 comp rx=0 tx=0).stop 2023-03-03T15:38:36.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.599+0000 7fb83fbff700 1 -- 172.21.15.102:0/1922455597 shutdown_connections 2023-03-03T15:38:36.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.599+0000 7fb83fbff700 1 --2- 172.21.15.102:0/1922455597 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8380cde20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.599+0000 7fb83fbff700 1 -- 172.21.15.102:0/1922455597 >> 172.21.15.102:0/1922455597 conn(0x7fb838065fb0 msgr2=0x7fb8380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:36.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.599+0000 7fb83fbff700 1 -- 172.21.15.102:0/1922455597 shutdown_connections 2023-03-03T15:38:36.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.599+0000 7fb83fbff700 1 -- 172.21.15.102:0/1922455597 wait complete. 2023-03-03T15:38:36.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.601+0000 7fb83fbff700 1 Processor -- start 2023-03-03T15:38:36.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.601+0000 7fb83fbff700 1 -- start start 2023-03-03T15:38:36.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83fbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8380cd390 0x7fb8380cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:36.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83fbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8381e4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:36.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83fbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb838153290 0x7fb83815abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:36.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83fbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8381e3c10 con 0x7fb8380cd390 2023-03-03T15:38:36.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83fbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380cfce0 con 0x7fb838153290 2023-03-03T15:38:36.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83fbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8381ef470 con 0x7fb8380cd8e0 2023-03-03T15:38:36.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83d99b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8380cd390 0x7fb8380cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:36.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83d99b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8380cd390 0x7fb8380cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55810/0 (socket says 172.21.15.102:55810) 2023-03-03T15:38:36.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83d99b700 1 -- 172.21.15.102:0/570413125 learned_addr learned my addr 172.21.15.102:0/570413125 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:38:36.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.602+0000 7fb83d19a700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8381e4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83e19c700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb838153290 0x7fb83815abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83d99b700 1 -- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 msgr2=0x7fb8381e4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83d99b700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8381e4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83d99b700 1 -- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb838153290 msgr2=0x7fb83815abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83d99b700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb838153290 0x7fb83815abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83d99b700 1 -- 172.21.15.102:0/570413125 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8280094a0 con 0x7fb8380cd390 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83e19c700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb838153290 0x7fb83815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:38:36.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83d99b700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8380cd390 0x7fb8380cf7a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb834001eb0 tx=0x7fb83400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:36.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb82e7fc700 1 -- 172.21.15.102:0/570413125 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834005d70 con 0x7fb8380cd390 2023-03-03T15:38:36.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb82e7fc700 1 -- 172.21.15.102:0/570413125 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8340070d0 con 0x7fb8380cd390 2023-03-03T15:38:36.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb82e7fc700 1 -- 172.21.15.102:0/570413125 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834007460 con 0x7fb8380cd390 2023-03-03T15:38:36.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.603+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8380ccde0 con 0x7fb8380cd390 2023-03-03T15:38:36.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.604+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83815d160 con 0x7fb8380cd390 2023-03-03T15:38:36.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.605+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8380cd0c0 con 0x7fb8380cd390 2023-03-03T15:38:36.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.606+0000 7fb82e7fc700 1 -- 172.21.15.102:0/570413125 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb834019030 con 0x7fb8380cd390 2023-03-03T15:38:36.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.607+0000 7fb82e7fc700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb81c0682f0 0x7fb81c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:36.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.607+0000 7fb82e7fc700 1 -- 172.21.15.102:0/570413125 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb834085160 con 0x7fb8380cd390 2023-03-03T15:38:36.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.607+0000 7fb83d19a700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb81c0682f0 0x7fb81c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:36.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.608+0000 7fb83d19a700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb81c0682f0 0x7fb81c06a7a0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb8280097c0 tx=0x7fb82801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:36.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.611+0000 7fb82e7fc700 1 -- 172.21.15.102:0/570413125 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb834051780 con 0x7fb8380cd390 2023-03-03T15:38:36.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.846+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8381e3e80 con 0x7fb8380cd390 2023-03-03T15:38:36.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.847+0000 7fb82e7fc700 1 -- 172.21.15.102:0/570413125 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb834017090 con 0x7fb8380cd390 2023-03-03T15:38:36.848 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:38:36.849 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:38:36.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.849+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb81c0682f0 msgr2=0x7fb81c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:36.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.849+0000 7fb83fbff700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb81c0682f0 0x7fb81c06a7a0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb8280097c0 tx=0x7fb82801c040 comp rx=0 tx=0).stop 2023-03-03T15:38:36.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.849+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8380cd390 msgr2=0x7fb8380cf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:36.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.849+0000 7fb83fbff700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8380cd390 0x7fb8380cf7a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb834001eb0 tx=0x7fb83400c7b0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 shutdown_connections 2023-03-03T15:38:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb838153290 0x7fb83815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8380cd390 0x7fb8380cf7a0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb81c0682f0 0x7fb81c06a7a0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 --2- 172.21.15.102:0/570413125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb8380cd8e0 0x7fb8381e4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 >> 172.21.15.102:0/570413125 conn(0x7fb838065fb0 msgr2=0x7fb838158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 shutdown_connections 2023-03-03T15:38:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:36.850+0000 7fb83fbff700 1 -- 172.21.15.102:0/570413125 wait complete. 2023-03-03T15:38:36.853 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:38:36.877 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T15:38:45.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:38:46.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.139+0000 7f24189ff700 1 Processor -- start 2023-03-03T15:38:46.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.139+0000 7f24189ff700 1 -- start start 2023-03-03T15:38:46.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.140+0000 7f24189ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f24140cd8e0 0x7f24140cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.140+0000 7f24189ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24140cd390 0x7f24140ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.140+0000 7f24189ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2414153660 con 0x7f2414153290 2023-03-03T15:38:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.140+0000 7f24189ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241406c360 con 0x7f24140cd390 2023-03-03T15:38:46.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.140+0000 7f24189ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241415cea0 con 0x7f24140cd8e0 2023-03-03T15:38:46.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2412d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24140cd390 0x7f24140ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:46.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2411d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f24140cd8e0 0x7f24140cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:46.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2412d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24140cd390 0x7f24140ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35664/0 (socket says 172.21.15.102:35664) 2023-03-03T15:38:46.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2412d9d700 1 -- 172.21.15.102:0/493105840 learned_addr learned my addr 172.21.15.102:0/493105840 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:38:46.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2411d9b700 1 -- 172.21.15.102:0/493105840 >> v2:172.21.15.102:3300/0 conn(0x7f24140cd390 msgr2=0x7f24140ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:38:46.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2411d9b700 1 --2- 172.21.15.102:0/493105840 >> v2:172.21.15.102:3300/0 conn(0x7f24140cd390 0x7f24140ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2411d9b700 1 -- 172.21.15.102:0/493105840 >> v1:172.21.15.172:6789/0 conn(0x7f2414153290 legacy=0x7f2414158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:46.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f2411d9b700 1 -- 172.21.15.102:0/493105840 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241406c7f0 con 0x7f24140cd8e0 2023-03-03T15:38:46.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.141+0000 7f241159a700 1 -- 172.21.15.102:0/493105840 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460255502 0 0) 0x7f2414153660 con 0x7f2414153290 2023-03-03T15:38:46.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.142+0000 7f2411d9b700 1 --2- 172.21.15.102:0/493105840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd8e0 0x7f24140cde20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2414159fb0 tx=0x7f2408009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f390af1f717c41a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:46.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.142+0000 7f241159a700 1 -- 172.21.15.102:0/493105840 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408011830 con 0x7f24140cd8e0 2023-03-03T15:38:46.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.142+0000 7f241159a700 1 -- 172.21.15.102:0/493105840 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2408011990 con 0x7f24140cd8e0 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.142+0000 7f241159a700 1 -- 172.21.15.102:0/493105840 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408011d20 con 0x7f24140cd8e0 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.142+0000 7f24189ff700 1 -- 172.21.15.102:0/493105840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd8e0 msgr2=0x7f24140cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.142+0000 7f24189ff700 1 --2- 172.21.15.102:0/493105840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd8e0 0x7f24140cde20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2414159fb0 tx=0x7f2408009b20 comp rx=0 tx=0).stop 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.143+0000 7f24189ff700 1 -- 172.21.15.102:0/493105840 shutdown_connections 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.143+0000 7f24189ff700 1 --2- 172.21.15.102:0/493105840 >> v2:172.21.15.102:3300/0 conn(0x7f24140cd390 0x7f24140ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.143+0000 7f24189ff700 1 --2- 172.21.15.102:0/493105840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd8e0 0x7f24140cde20 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.143+0000 7f24189ff700 1 -- 172.21.15.102:0/493105840 >> 172.21.15.102:0/493105840 conn(0x7f2414065fb0 msgr2=0x7f24140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:46.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.143+0000 7f24189ff700 1 -- 172.21.15.102:0/493105840 shutdown_connections 2023-03-03T15:38:46.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.143+0000 7f24189ff700 1 -- 172.21.15.102:0/493105840 wait complete. 2023-03-03T15:38:46.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.145+0000 7f24189ff700 1 Processor -- start 2023-03-03T15:38:46.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.145+0000 7f24189ff700 1 -- start start 2023-03-03T15:38:46.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.146+0000 7f24189ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd390 0x7f241414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:46.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.146+0000 7f24189ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24140cd8e0 0x7f24141ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:46.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.146+0000 7f24189ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414153290 0x7f24140c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:46.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.146+0000 7f24189ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241414ea90 con 0x7f2414153290 2023-03-03T15:38:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.146+0000 7f24189ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414151890 con 0x7f24140cd390 2023-03-03T15:38:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.146+0000 7f24189ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24141efe20 con 0x7f24140cd8e0 2023-03-03T15:38:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.146+0000 7f2412d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414153290 0x7f24140c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2411d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24140cd8e0 0x7f24141ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2412d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414153290 0x7f24140c2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35668/0 (socket says 172.21.15.102:35668) 2023-03-03T15:38:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f241259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd390 0x7f241414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2412d9d700 1 -- 172.21.15.102:0/998088008 learned_addr learned my addr 172.21.15.102:0/998088008 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:38:46.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2411d9b700 1 -- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd390 msgr2=0x7f241414e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:46.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2411d9b700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd390 0x7f241414e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2411d9b700 1 -- 172.21.15.102:0/998088008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414153290 msgr2=0x7f24140c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:46.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2411d9b700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414153290 0x7f24140c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2411d9b700 1 -- 172.21.15.102:0/998088008 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24080094a0 con 0x7f24140cd8e0 2023-03-03T15:38:46.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f241259c700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd390 0x7f241414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:38:46.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2412d9d700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414153290 0x7f24140c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:38:46.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.147+0000 7f2411d9b700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24140cd8e0 0x7f24141ef8e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2408016000 tx=0x7f2408004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:46.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.148+0000 7f2402ffd700 1 -- 172.21.15.102:0/998088008 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24080036a0 con 0x7f24140cd8e0 2023-03-03T15:38:46.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.148+0000 7f2402ffd700 1 -- 172.21.15.102:0/998088008 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240801f070 con 0x7f24140cd8e0 2023-03-03T15:38:46.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.148+0000 7f2402ffd700 1 -- 172.21.15.102:0/998088008 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408003a40 con 0x7f24140cd8e0 2023-03-03T15:38:46.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.148+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24140ccde0 con 0x7f24140cd8e0 2023-03-03T15:38:46.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.148+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241415d160 con 0x7f24140cd8e0 2023-03-03T15:38:46.152 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.149+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24140cd0c0 con 0x7f24140cd8e0 2023-03-03T15:38:46.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.150+0000 7f2402ffd700 1 -- 172.21.15.102:0/998088008 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f240800f050 con 0x7f24140cd8e0 2023-03-03T15:38:46.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.151+0000 7f2402ffd700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23f40682f0 0x7f23f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:46.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.151+0000 7f2402ffd700 1 -- 172.21.15.102:0/998088008 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2408021070 con 0x7f24140cd8e0 2023-03-03T15:38:46.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.152+0000 7f241259c700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23f40682f0 0x7f23f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:46.154 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.152+0000 7f241259c700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23f40682f0 0x7f23f406a7a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2404000f60 tx=0x7f2404009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:46.157 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.156+0000 7f2402ffd700 1 -- 172.21.15.102:0/998088008 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2408053ca0 con 0x7f24140cd8e0 2023-03-03T15:38:46.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.387+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f241404e710 con 0x7f24140cd8e0 2023-03-03T15:38:46.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.388+0000 7f2402ffd700 1 -- 172.21.15.102:0/998088008 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2408015090 con 0x7f24140cd8e0 2023-03-03T15:38:46.390 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:38:46.390 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:38:46.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23f40682f0 msgr2=0x7f23f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:46.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23f40682f0 0x7f23f406a7a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2404000f60 tx=0x7f2404009040 comp rx=0 tx=0).stop 2023-03-03T15:38:46.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24140cd8e0 msgr2=0x7f24141ef8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:46.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24140cd8e0 0x7f24141ef8e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2408016000 tx=0x7f2408004750 comp rx=0 tx=0).stop 2023-03-03T15:38:46.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 shutdown_connections 2023-03-03T15:38:46.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414153290 0x7f24140c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24140cd390 0x7f241414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23f40682f0 0x7f23f406a7a0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 --2- 172.21.15.102:0/998088008 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24140cd8e0 0x7f24141ef8e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:46.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 >> 172.21.15.102:0/998088008 conn(0x7f2414065fb0 msgr2=0x7f24140cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:46.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 shutdown_connections 2023-03-03T15:38:46.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:46.391+0000 7f24189ff700 1 -- 172.21.15.102:0/998088008 wait complete. 2023-03-03T15:38:46.395 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:38:46.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-03-03T15:38:54.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:38:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.683+0000 7f49487ff700 1 Processor -- start 2023-03-03T15:38:54.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.683+0000 7f49487ff700 1 -- start start 2023-03-03T15:38:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.684+0000 7f49487ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.684+0000 7f49487ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49401593c0 0x7f49400cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.684+0000 7f49487ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940153660 con 0x7f49400cd8e0 2023-03-03T15:38:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.684+0000 7f49487ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494006c360 con 0x7f49401593c0 2023-03-03T15:38:54.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.684+0000 7f49487ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494015cea0 con 0x7f4940153290 2023-03-03T15:38:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4945d9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49401593c0 0x7f49400cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4946d9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f49400cd8e0 0x7f49400ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43674/0 (socket says 172.21.15.102:43674) 2023-03-03T15:38:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4946d9c700 1 -- 172.21.15.102:0/652831138 learned_addr learned my addr 172.21.15.102:0/652831138 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:38:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f494659b700 1 --2- 172.21.15.102:0/652831138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4945d9a700 1 -- 172.21.15.102:0/652831138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 msgr2=0x7f4940158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:38:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4945d9a700 1 --2- 172.21.15.102:0/652831138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4945d9a700 1 -- 172.21.15.102:0/652831138 >> v1:172.21.15.102:6789/0 conn(0x7f49400cd8e0 legacy=0x7f49400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4945d9a700 1 -- 172.21.15.102:0/652831138 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494006c7f0 con 0x7f49401593c0 2023-03-03T15:38:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4945599700 1 -- 172.21.15.102:0/652831138 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943642315 0 0) 0x7f4940153660 con 0x7f49400cd8e0 2023-03-03T15:38:54.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.685+0000 7f4945d9a700 1 --2- 172.21.15.102:0/652831138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49401593c0 0x7f49400cde20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f4930009840 tx=0x7f4930009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6f83ed1ec22d24 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.686+0000 7f4945599700 1 -- 172.21.15.102:0/652831138 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4930011830 con 0x7f49401593c0 2023-03-03T15:38:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.686+0000 7f4945599700 1 -- 172.21.15.102:0/652831138 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4930011990 con 0x7f49401593c0 2023-03-03T15:38:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.686+0000 7f4945599700 1 -- 172.21.15.102:0/652831138 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4930011d20 con 0x7f49401593c0 2023-03-03T15:38:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.686+0000 7f49487ff700 1 -- 172.21.15.102:0/652831138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49401593c0 msgr2=0x7f49400cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.686+0000 7f49487ff700 1 --2- 172.21.15.102:0/652831138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49401593c0 0x7f49400cde20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f4930009840 tx=0x7f4930009b50 comp rx=0 tx=0).stop 2023-03-03T15:38:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.687+0000 7f49487ff700 1 -- 172.21.15.102:0/652831138 shutdown_connections 2023-03-03T15:38:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.687+0000 7f49487ff700 1 --2- 172.21.15.102:0/652831138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.687+0000 7f49487ff700 1 --2- 172.21.15.102:0/652831138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49401593c0 0x7f49400cde20 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.687+0000 7f49487ff700 1 -- 172.21.15.102:0/652831138 >> 172.21.15.102:0/652831138 conn(0x7f4940065fb0 msgr2=0x7f49400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.687+0000 7f49487ff700 1 -- 172.21.15.102:0/652831138 shutdown_connections 2023-03-03T15:38:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.687+0000 7f49487ff700 1 -- 172.21.15.102:0/652831138 wait complete. 2023-03-03T15:38:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.689+0000 7f49487ff700 1 Processor -- start 2023-03-03T15:38:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.689+0000 7f49487ff700 1 -- start start 2023-03-03T15:38:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f49487ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd8e0 0x7f49400cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f49487ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f49487ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49401593c0 0x7f494014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f49487ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4940153f50 con 0x7f49400cd8e0 2023-03-03T15:38:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f49487ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49401501a0 con 0x7f49401593c0 2023-03-03T15:38:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f49487ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49400b2740 con 0x7f4940153290 2023-03-03T15:38:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f494659b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd8e0 0x7f49400cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f494659b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd8e0 0x7f49400cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53718/0 (socket says 172.21.15.102:53718) 2023-03-03T15:38:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f494659b700 1 -- 172.21.15.102:0/1790589624 learned_addr learned my addr 172.21.15.102:0/1790589624 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:38:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.690+0000 7f4946d9c700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49401593c0 0x7f494014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:54.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f4945d9a700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f494659b700 1 -- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 msgr2=0x7f4940154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f494659b700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f494659b700 1 -- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49401593c0 msgr2=0x7f494014fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f494659b700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49401593c0 0x7f494014fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f494659b700 1 -- 172.21.15.102:0/1790589624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300094a0 con 0x7f49400cd8e0 2023-03-03T15:38:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f494659b700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd8e0 0x7f49400cad00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e9b0 tx=0x7f493c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.691+0000 7f4936ffd700 1 -- 172.21.15.102:0/1790589624 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c009710 con 0x7f49400cd8e0 2023-03-03T15:38:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.692+0000 7f4936ffd700 1 -- 172.21.15.102:0/1790589624 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c00ee50 con 0x7f49400cd8e0 2023-03-03T15:38:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.692+0000 7f4936ffd700 1 -- 172.21.15.102:0/1790589624 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c00ca50 con 0x7f49400cd8e0 2023-03-03T15:38:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.692+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49400ccde0 con 0x7f49400cd8e0 2023-03-03T15:38:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.692+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494015d160 con 0x7f49400cd8e0 2023-03-03T15:38:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.693+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49400cd0c0 con 0x7f49400cd8e0 2023-03-03T15:38:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.694+0000 7f4936ffd700 1 -- 172.21.15.102:0/1790589624 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f493c019030 con 0x7f49400cd8e0 2023-03-03T15:38:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.695+0000 7f4936ffd700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49240682f0 0x7f492406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.695+0000 7f4936ffd700 1 -- 172.21.15.102:0/1790589624 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f493c085c60 con 0x7f49400cd8e0 2023-03-03T15:38:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.696+0000 7f4945d9a700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49240682f0 0x7f492406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:54.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.696+0000 7f4945d9a700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49240682f0 0x7f492406a7a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f49300090b0 tx=0x7f4930008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:54.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.700+0000 7f4936ffd700 1 -- 172.21.15.102:0/1790589624 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f493c052370 con 0x7f49400cd8e0 2023-03-03T15:38:54.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.935+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f494004e6e0 con 0x7f49400cd8e0 2023-03-03T15:38:54.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.936+0000 7f4936ffd700 1 -- 172.21.15.102:0/1790589624 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f493c055990 con 0x7f49400cd8e0 2023-03-03T15:38:54.938 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:38:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:38:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49240682f0 msgr2=0x7f492406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49240682f0 0x7f492406a7a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f49300090b0 tx=0x7f4930008030 comp rx=0 tx=0).stop 2023-03-03T15:38:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd8e0 msgr2=0x7f49400cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd8e0 0x7f49400cad00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e9b0 tx=0x7f493c00ecc0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 shutdown_connections 2023-03-03T15:38:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49401593c0 0x7f494014fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49400cd8e0 0x7f49400cad00 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49240682f0 0x7f492406a7a0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 --2- 172.21.15.102:0/1790589624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4940153290 0x7f4940154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 >> 172.21.15.102:0/1790589624 conn(0x7f4940065fb0 msgr2=0x7f49401e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 shutdown_connections 2023-03-03T15:38:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:38:54.939+0000 7f49487ff700 1 -- 172.21.15.102:0/1790589624 wait complete. 2023-03-03T15:38:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:38:54.967 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 25 secs before thrashing 2023-03-03T15:39:04.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:04.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:19.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:39:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.218+0000 7ffb193ff700 1 Processor -- start 2023-03-03T15:39:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.218+0000 7ffb193ff700 1 -- start start 2023-03-03T15:39:20.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb193ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb140cd8e0 0x7ffb140cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:20.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb193ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffb140cd390 0x7ffb140ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:20.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb193ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb14153660 con 0x7ffb14153290 2023-03-03T15:39:20.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb193ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1406c360 con 0x7ffb140cd390 2023-03-03T15:39:20.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb193ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1415cea0 con 0x7ffb140cd8e0 2023-03-03T15:39:20.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb12ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffb14153290 0x7ffb14158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40762/0 (socket says 172.21.15.102:40762) 2023-03-03T15:39:20.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb12ffd700 1 -- 172.21.15.102:0/1552279701 learned_addr learned my addr 172.21.15.102:0/1552279701 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:39:20.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb137fe700 1 --2- 172.21.15.102:0/1552279701 >> v2:172.21.15.102:3300/0 conn(0x7ffb140cd390 0x7ffb140ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:20.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.220+0000 7ffb127fc700 1 --2- 172.21.15.102:0/1552279701 >> v2:172.21.15.172:3300/0 conn(0x7ffb140cd8e0 0x7ffb140cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:20.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb11ffb700 1 -- 172.21.15.102:0/1552279701 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147363284 0 0) 0x7ffb14153660 con 0x7ffb14153290 2023-03-03T15:39:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb11ffb700 1 -- 172.21.15.102:0/1552279701 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc0035b0 con 0x7ffb14153290 2023-03-03T15:39:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb137fe700 1 -- 172.21.15.102:0/1552279701 >> v2:172.21.15.172:3300/0 conn(0x7ffb140cd8e0 msgr2=0x7ffb140cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:39:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb137fe700 1 --2- 172.21.15.102:0/1552279701 >> v2:172.21.15.172:3300/0 conn(0x7ffb140cd8e0 0x7ffb140cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb137fe700 1 -- 172.21.15.102:0/1552279701 >> v1:172.21.15.102:6789/0 conn(0x7ffb14153290 legacy=0x7ffb14158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb137fe700 1 -- 172.21.15.102:0/1552279701 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1406c7f0 con 0x7ffb140cd390 2023-03-03T15:39:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb137fe700 1 --2- 172.21.15.102:0/1552279701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd390 0x7ffb140ce360 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7ffb14159fb0 tx=0x7ffb0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70d60ca22691787 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb11ffb700 1 -- 172.21.15.102:0/1552279701 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c00e070 con 0x7ffb140cd390 2023-03-03T15:39:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb11ffb700 1 -- 172.21.15.102:0/1552279701 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c00bb20 con 0x7ffb140cd390 2023-03-03T15:39:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.221+0000 7ffb11ffb700 1 -- 172.21.15.102:0/1552279701 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c012430 con 0x7ffb140cd390 2023-03-03T15:39:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 -- 172.21.15.102:0/1552279701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd390 msgr2=0x7ffb140ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 --2- 172.21.15.102:0/1552279701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd390 0x7ffb140ce360 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7ffb14159fb0 tx=0x7ffb0c00b330 comp rx=0 tx=0).stop 2023-03-03T15:39:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 -- 172.21.15.102:0/1552279701 shutdown_connections 2023-03-03T15:39:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 --2- 172.21.15.102:0/1552279701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd390 0x7ffb140ce360 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 --2- 172.21.15.102:0/1552279701 >> v2:172.21.15.172:3300/0 conn(0x7ffb140cd8e0 0x7ffb140cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 -- 172.21.15.102:0/1552279701 >> 172.21.15.102:0/1552279701 conn(0x7ffb14065fb0 msgr2=0x7ffb140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 -- 172.21.15.102:0/1552279701 shutdown_connections 2023-03-03T15:39:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.222+0000 7ffb193ff700 1 -- 172.21.15.102:0/1552279701 wait complete. 2023-03-03T15:39:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.224+0000 7ffb193ff700 1 Processor -- start 2023-03-03T15:39:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.224+0000 7ffb193ff700 1 -- start start 2023-03-03T15:39:20.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb193ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb140cd390 0x7ffb141e2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb193ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd8e0 0x7ffb141e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb193ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb14153290 0x7ffb1414e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb193ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140b25b0 con 0x7ffb140cd8e0 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb193ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140b20a0 con 0x7ffb14153290 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb193ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb141e22c0 con 0x7ffb140cd390 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd8e0 0x7ffb141e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd8e0 0x7ffb141e2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58924/0 (socket says 172.21.15.102:58924) 2023-03-03T15:39:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 -- 172.21.15.102:0/3839798807 learned_addr learned my addr 172.21.15.102:0/3839798807 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:39:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb12ffd700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb140cd390 0x7ffb141e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb137fe700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb14153290 0x7ffb1414e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 -- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb140cd390 msgr2=0x7ffb141e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb140cd390 0x7ffb141e2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 -- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb14153290 msgr2=0x7ffb1414e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:20.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb14153290 0x7ffb1414e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.226+0000 7ffb127fc700 1 -- 172.21.15.102:0/3839798807 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c009a70 con 0x7ffb140cd8e0 2023-03-03T15:39:20.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.227+0000 7ffb127fc700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd8e0 0x7ffb141e2c90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800e9b0 tx=0x7ffb0800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.227+0000 7ffafb7fe700 1 -- 172.21.15.102:0/3839798807 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08009710 con 0x7ffb140cd8e0 2023-03-03T15:39:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.227+0000 7ffafb7fe700 1 -- 172.21.15.102:0/3839798807 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0800ee50 con 0x7ffb140cd8e0 2023-03-03T15:39:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.227+0000 7ffafb7fe700 1 -- 172.21.15.102:0/3839798807 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0800ca50 con 0x7ffb140cd8e0 2023-03-03T15:39:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.227+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb140ccde0 con 0x7ffb140cd8e0 2023-03-03T15:39:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.227+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1415d160 con 0x7ffb140cd8e0 2023-03-03T15:39:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.228+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb140cd0c0 con 0x7ffb140cd8e0 2023-03-03T15:39:20.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.230+0000 7ffafb7fe700 1 -- 172.21.15.102:0/3839798807 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb08019030 con 0x7ffb140cd8e0 2023-03-03T15:39:20.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.231+0000 7ffafb7fe700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb000682f0 0x7ffb0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:20.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.231+0000 7ffafb7fe700 1 -- 172.21.15.102:0/3839798807 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ffb08085c80 con 0x7ffb140cd8e0 2023-03-03T15:39:20.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.231+0000 7ffb12ffd700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb000682f0 0x7ffb0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:20.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.232+0000 7ffb12ffd700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb000682f0 0x7ffb0006a7a0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7ffb04007fd0 tx=0x7ffb04007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:20.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.235+0000 7ffafb7fe700 1 -- 172.21.15.102:0/3839798807 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb08052390 con 0x7ffb140cd8e0 2023-03-03T15:39:20.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.466+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb141e2090 con 0x7ffb140cd8e0 2023-03-03T15:39:20.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.466+0000 7ffafb7fe700 1 -- 172.21.15.102:0/3839798807 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ffb141e2090 con 0x7ffb140cd8e0 2023-03-03T15:39:20.468 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:39:20.468 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:39:20.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb000682f0 msgr2=0x7ffb0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:20.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb000682f0 0x7ffb0006a7a0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7ffb04007fd0 tx=0x7ffb04007dc0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd8e0 msgr2=0x7ffb141e2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:20.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd8e0 0x7ffb141e2c90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800e9b0 tx=0x7ffb0800ecc0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 shutdown_connections 2023-03-03T15:39:20.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ffb000682f0 0x7ffb0006a7a0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ffb140cd390 0x7ffb141e2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb14153290 0x7ffb1414e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 --2- 172.21.15.102:0/3839798807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb140cd8e0 0x7ffb141e2c90 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:20.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 >> 172.21.15.102:0/3839798807 conn(0x7ffb14065fb0 msgr2=0x7ffb14158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:20.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 shutdown_connections 2023-03-03T15:39:20.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:39:20.469+0000 7ffb193ff700 1 -- 172.21.15.102:0/3839798807 wait complete. 2023-03-03T15:39:20.472 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:39:20.497 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-03-03T15:39:34.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:34.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:00.001 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T15:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:40:04.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:04.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:33.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:40:33.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.745+0000 7f1cbc7ff700 1 Processor -- start 2023-03-03T15:40:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.745+0000 7f1cbc7ff700 1 -- start start 2023-03-03T15:40:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cbc7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1cb41531c0 0x7f1cb415c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cbc7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1cb415c890 0x7f1cb40c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cbc7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb40723d0 con 0x7f1cb40c86f0 2023-03-03T15:40:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cbc7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb4153590 con 0x7f1cb41531c0 2023-03-03T15:40:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cbc7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb4072860 con 0x7f1cb415c890 2023-03-03T15:40:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cba59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1cb41531c0 0x7f1cb415c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cb9d9a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1cb415c890 0x7f1cb40c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cba59b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1cb41531c0 0x7f1cb415c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41238/0 (socket says 172.21.15.102:41238) 2023-03-03T15:40:33.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cbad9c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1cb40c86f0 0x7f1cb40c6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38640/0 (socket says 172.21.15.102:38640) 2023-03-03T15:40:33.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.747+0000 7f1cba59b700 1 -- 172.21.15.102:0/935583807 learned_addr learned my addr 172.21.15.102:0/935583807 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:40:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cba59b700 1 -- 172.21.15.102:0/935583807 >> v2:172.21.15.172:3300/0 conn(0x7f1cb415c890 msgr2=0x7f1cb40c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:40:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cba59b700 1 --2- 172.21.15.102:0/935583807 >> v2:172.21.15.172:3300/0 conn(0x7f1cb415c890 0x7f1cb40c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cba59b700 1 -- 172.21.15.102:0/935583807 >> v1:172.21.15.172:6789/0 conn(0x7f1cb40c86f0 legacy=0x7f1cb40c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cba59b700 1 -- 172.21.15.102:0/935583807 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb40c8c50 con 0x7f1cb41531c0 2023-03-03T15:40:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cba59b700 1 --2- 172.21.15.102:0/935583807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb41531c0 0x7f1cb415c350 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40c3170 tx=0x7f1cb0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ca878ba5e9f4fd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:40:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cb9599700 1 -- 172.21.15.102:0/935583807 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0011830 con 0x7f1cb41531c0 2023-03-03T15:40:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cb9599700 1 -- 172.21.15.102:0/935583807 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb0011990 con 0x7f1cb41531c0 2023-03-03T15:40:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.748+0000 7f1cb9599700 1 -- 172.21.15.102:0/935583807 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0011d20 con 0x7f1cb41531c0 2023-03-03T15:40:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/935583807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb41531c0 msgr2=0x7f1cb415c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/935583807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb41531c0 0x7f1cb415c350 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40c3170 tx=0x7f1cb0009bd0 comp rx=0 tx=0).stop 2023-03-03T15:40:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/935583807 shutdown_connections 2023-03-03T15:40:33.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/935583807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb41531c0 0x7f1cb415c350 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:33.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/935583807 >> v2:172.21.15.172:3300/0 conn(0x7f1cb415c890 0x7f1cb40c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:33.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/935583807 >> 172.21.15.102:0/935583807 conn(0x7f1cb406b410 msgr2=0x7f1cb406b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:40:33.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/935583807 shutdown_connections 2023-03-03T15:40:33.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.749+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/935583807 wait complete. 2023-03-03T15:40:33.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.752+0000 7f1cbc7ff700 1 Processor -- start 2023-03-03T15:40:33.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.752+0000 7f1cbc7ff700 1 -- start start 2023-03-03T15:40:33.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbc7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cb40c86f0 0x7f1cb415a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:33.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbc7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cb41531c0 0x7f1cb41e3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:33.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbc7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb415c890 0x7f1cb41474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:33.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbc7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb41e2790 con 0x7f1cb415c890 2023-03-03T15:40:33.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbc7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb41e47d0 con 0x7f1cb40c86f0 2023-03-03T15:40:33.756 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbc7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb41e2420 con 0x7f1cb41531c0 2023-03-03T15:40:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbad9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb415c890 0x7f1cb41474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbad9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb415c890 0x7f1cb41474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41246/0 (socket says 172.21.15.102:41246) 2023-03-03T15:40:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cbad9c700 1 -- 172.21.15.102:0/2485531147 learned_addr learned my addr 172.21.15.102:0/2485531147 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:40:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cba59b700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cb40c86f0 0x7f1cb415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.754+0000 7f1cb9d9a700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cb41531c0 0x7f1cb41e3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1cbad9c700 1 -- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cb41531c0 msgr2=0x7f1cb41e3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1cbad9c700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cb41531c0 0x7f1cb41e3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:33.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1cbad9c700 1 -- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cb40c86f0 msgr2=0x7f1cb415a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:33.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1cbad9c700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cb40c86f0 0x7f1cb415a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:33.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1cbad9c700 1 -- 172.21.15.102:0/2485531147 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb0009550 con 0x7f1cb415c890 2023-03-03T15:40:33.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1cbad9c700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb415c890 0x7f1cb41474c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00cbd0 tx=0x7f1cac00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:40:33.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1caaffd700 1 -- 172.21.15.102:0/2485531147 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac00aaa0 con 0x7f1cb415c890 2023-03-03T15:40:33.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1caaffd700 1 -- 172.21.15.102:0/2485531147 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac00ac00 con 0x7f1cb415c890 2023-03-03T15:40:33.759 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb4071aa0 con 0x7f1cb415c890 2023-03-03T15:40:33.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.755+0000 7f1caaffd700 1 -- 172.21.15.102:0/2485531147 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac007070 con 0x7f1cb415c890 2023-03-03T15:40:33.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.756+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb40c8f30 con 0x7f1cb415c890 2023-03-03T15:40:33.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.757+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb4071d80 con 0x7f1cb415c890 2023-03-03T15:40:33.760 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.758+0000 7f1caaffd700 1 -- 172.21.15.102:0/2485531147 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1cac019030 con 0x7f1cb415c890 2023-03-03T15:40:33.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.760+0000 7f1caaffd700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c980682f0 0x7f1c9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:33.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.760+0000 7f1caaffd700 1 -- 172.21.15.102:0/2485531147 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1cac085890 con 0x7f1cb415c890 2023-03-03T15:40:33.762 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.760+0000 7f1cba59b700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c980682f0 0x7f1c9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:33.763 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.761+0000 7f1cba59b700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c980682f0 0x7f1c9806a7a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009c10 tx=0x7f1cb0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:40:33.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.764+0000 7f1caaffd700 1 -- 172.21.15.102:0/2485531147 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1cac04b5e0 con 0x7f1cb415c890 2023-03-03T15:40:34.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.998+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cb40c5f50 con 0x7f1cb415c890 2023-03-03T15:40:34.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:33.998+0000 7f1caaffd700 1 -- 172.21.15.102:0/2485531147 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1cac017090 con 0x7f1cb415c890 2023-03-03T15:40:34.001 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:40:34.001 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:40:34.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c980682f0 msgr2=0x7f1c9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:34.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c980682f0 0x7f1c9806a7a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009c10 tx=0x7f1cb0008030 comp rx=0 tx=0).stop 2023-03-03T15:40:34.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb415c890 msgr2=0x7f1cb41474c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:34.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb415c890 0x7f1cb41474c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00cbd0 tx=0x7f1cac00cee0 comp rx=0 tx=0).stop 2023-03-03T15:40:34.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 shutdown_connections 2023-03-03T15:40:34.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cb415c890 0x7f1cb41474c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:34.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1cb40c86f0 0x7f1cb415a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:34.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c980682f0 0x7f1c9806a7a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:34.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 --2- 172.21.15.102:0/2485531147 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1cb41531c0 0x7f1cb41e3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 >> 172.21.15.102:0/2485531147 conn(0x7f1cb406b410 msgr2=0x7f1cb40c5290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:40:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 shutdown_connections 2023-03-03T15:40:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:40:34.001+0000 7f1cbc7ff700 1 -- 172.21.15.102:0/2485531147 wait complete. 2023-03-03T15:40:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:40:34.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-03-03T15:40:34.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:34.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:05.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:05.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:32.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:41:32.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.277+0000 7f77b6fff700 1 Processor -- start 2023-03-03T15:41:32.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.278+0000 7f77b6fff700 1 -- start start 2023-03-03T15:41:32.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b6fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b00cd8e0 0x7f77b00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:32.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b6fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77b00cd390 0x7f77b00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:32.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b6fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b0153660 con 0x7f77b0153290 2023-03-03T15:41:32.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b6fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b006c360 con 0x7f77b00cd390 2023-03-03T15:41:32.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b6fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b015cea0 con 0x7f77b00cd8e0 2023-03-03T15:41:32.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b4d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f77b0153290 0x7f77b0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42686/0 (socket says 172.21.15.102:42686) 2023-03-03T15:41:32.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b4d9b700 1 -- 172.21.15.102:0/2761168376 learned_addr learned my addr 172.21.15.102:0/2761168376 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:41:32.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77affff700 1 --2- 172.21.15.102:0/2761168376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b00cd8e0 0x7f77b00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:32.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.279+0000 7f77b559c700 1 --2- 172.21.15.102:0/2761168376 >> v2:172.21.15.172:3300/0 conn(0x7f77b00cd390 0x7f77b00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:32.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77af7fe700 1 -- 172.21.15.102:0/2761168376 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310463138 0 0) 0x7f77b0153660 con 0x7f77b0153290 2023-03-03T15:41:32.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77af7fe700 1 -- 172.21.15.102:0/2761168376 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a00035b0 con 0x7f77b0153290 2023-03-03T15:41:32.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77affff700 1 -- 172.21.15.102:0/2761168376 >> v2:172.21.15.172:3300/0 conn(0x7f77b00cd390 msgr2=0x7f77b00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:41:32.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77affff700 1 --2- 172.21.15.102:0/2761168376 >> v2:172.21.15.172:3300/0 conn(0x7f77b00cd390 0x7f77b00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77affff700 1 -- 172.21.15.102:0/2761168376 >> v1:172.21.15.172:6789/0 conn(0x7f77b0153290 legacy=0x7f77b0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:32.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77affff700 1 -- 172.21.15.102:0/2761168376 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b006c7f0 con 0x7f77b00cd8e0 2023-03-03T15:41:32.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77b559c700 1 --2- 172.21.15.102:0/2761168376 >> v2:172.21.15.172:3300/0 conn(0x7f77b00cd390 0x7f77b00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:41:32.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.280+0000 7f77affff700 1 --2- 172.21.15.102:0/2761168376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b00cd8e0 0x7f77b00cde20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f779c009840 tx=0x7f779c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e7433da44213425 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:32.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77af7fe700 1 -- 172.21.15.102:0/2761168376 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011830 con 0x7f77b00cd8e0 2023-03-03T15:41:32.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77af7fe700 1 -- 172.21.15.102:0/2761168376 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c011990 con 0x7f77b00cd8e0 2023-03-03T15:41:32.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77af7fe700 1 -- 172.21.15.102:0/2761168376 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011d20 con 0x7f77b00cd8e0 2023-03-03T15:41:32.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 -- 172.21.15.102:0/2761168376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b00cd8e0 msgr2=0x7f77b00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:32.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 --2- 172.21.15.102:0/2761168376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b00cd8e0 0x7f77b00cde20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f779c009840 tx=0x7f779c009b50 comp rx=0 tx=0).stop 2023-03-03T15:41:32.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 -- 172.21.15.102:0/2761168376 shutdown_connections 2023-03-03T15:41:32.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 --2- 172.21.15.102:0/2761168376 >> v2:172.21.15.172:3300/0 conn(0x7f77b00cd390 0x7f77b00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 --2- 172.21.15.102:0/2761168376 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b00cd8e0 0x7f77b00cde20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 -- 172.21.15.102:0/2761168376 >> 172.21.15.102:0/2761168376 conn(0x7f77b0065fb0 msgr2=0x7f77b00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:41:32.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 -- 172.21.15.102:0/2761168376 shutdown_connections 2023-03-03T15:41:32.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.281+0000 7f77b6fff700 1 -- 172.21.15.102:0/2761168376 wait complete. 2023-03-03T15:41:32.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.283+0000 7f77b6fff700 1 Processor -- start 2023-03-03T15:41:32.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.284+0000 7f77b6fff700 1 -- start start 2023-03-03T15:41:32.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b6fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77b00cd390 0x7f77b00c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:32.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b6fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b00cd8e0 0x7f77b014e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:32.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b6fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b0153290 0x7f77b00c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:32.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b6fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b00c7e00 con 0x7f77b00cd8e0 2023-03-03T15:41:32.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b6fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b014de10 con 0x7f77b00cd390 2023-03-03T15:41:32.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b6fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b00c78b0 con 0x7f77b0153290 2023-03-03T15:41:32.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77b00cd390 0x7f77b00c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:32.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77b00cd390 0x7f77b00c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36146/0 (socket says 172.21.15.102:36146) 2023-03-03T15:41:32.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 -- 172.21.15.102:0/151522383 learned_addr learned my addr 172.21.15.102:0/151522383 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:41:32.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77affff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b00cd8e0 0x7f77b014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:32.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b559c700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b0153290 0x7f77b00c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 -- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b0153290 msgr2=0x7f77b00c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b0153290 0x7f77b00c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 -- 172.21.15.102:0/151522383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b00cd8e0 msgr2=0x7f77b014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b00cd8e0 0x7f77b014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.285+0000 7f77b4d9b700 1 -- 172.21.15.102:0/151522383 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a8009a70 con 0x7f77b00cd390 2023-03-03T15:41:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.286+0000 7f77b4d9b700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77b00cd390 0x7f77b00c7160 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f77a4001eb0 tx=0x7f77a400c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.286+0000 7f77ad7fa700 1 -- 172.21.15.102:0/151522383 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400cea0 con 0x7f77b00cd390 2023-03-03T15:41:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.286+0000 7f77ad7fa700 1 -- 172.21.15.102:0/151522383 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a4005d70 con 0x7f77b00cd390 2023-03-03T15:41:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.286+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c0094a0 con 0x7f77b00cd390 2023-03-03T15:41:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.286+0000 7f77ad7fa700 1 -- 172.21.15.102:0/151522383 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a40072f0 con 0x7f77b00cd390 2023-03-03T15:41:32.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.286+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b00ccde0 con 0x7f77b00cd390 2023-03-03T15:41:32.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.287+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b015d160 con 0x7f77b00cd390 2023-03-03T15:41:32.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.289+0000 7f77ad7fa700 1 -- 172.21.15.102:0/151522383 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f77a4019030 con 0x7f77b00cd390 2023-03-03T15:41:32.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.290+0000 7f77ad7fa700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77940682f0 0x7f779406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:32.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.290+0000 7f77ad7fa700 1 -- 172.21.15.102:0/151522383 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f77a40850d0 con 0x7f77b00cd390 2023-03-03T15:41:32.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.290+0000 7f77affff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77940682f0 0x7f779406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:32.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.291+0000 7f77affff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77940682f0 0x7f779406a7a0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f779c000c00 tx=0x7f779c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:32.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.294+0000 7f77ad7fa700 1 -- 172.21.15.102:0/151522383 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f77a4051750 con 0x7f77b00cd390 2023-03-03T15:41:32.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.528+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77b00c80a0 con 0x7f77b00cd390 2023-03-03T15:41:32.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.529+0000 7f77ad7fa700 1 -- 172.21.15.102:0/151522383 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f77a4017020 con 0x7f77b00cd390 2023-03-03T15:41:32.531 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:41:32.532 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:41:32.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77940682f0 msgr2=0x7f779406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:32.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77940682f0 0x7f779406a7a0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f779c000c00 tx=0x7f779c008030 comp rx=0 tx=0).stop 2023-03-03T15:41:32.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77b00cd390 msgr2=0x7f77b00c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:32.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77b00cd390 0x7f77b00c7160 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f77a4001eb0 tx=0x7f77a400c7b0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 shutdown_connections 2023-03-03T15:41:32.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f77940682f0 0x7f779406a7a0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f77b0153290 0x7f77b00c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77b00cd390 0x7f77b00c7160 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 --2- 172.21.15.102:0/151522383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b00cd8e0 0x7f77b014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:32.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 >> 172.21.15.102:0/151522383 conn(0x7f77b0065fb0 msgr2=0x7f77b00cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:41:32.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.531+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 shutdown_connections 2023-03-03T15:41:32.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:41:32.532+0000 7f77b6fff700 1 -- 172.21.15.102:0/151522383 wait complete. 2023-03-03T15:41:32.537 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:41:32.561 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T15:41:35.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:35.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:05.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:05.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:34.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:42:34.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.811+0000 7f1d98dff700 1 Processor -- start 2023-03-03T15:42:34.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.811+0000 7f1d98dff700 1 -- start start 2023-03-03T15:42:34.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d98dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1d940cd8e0 0x7f1d940cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:34.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d98dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d940cd390 0x7f1d940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:34.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d98dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d94153660 con 0x7f1d94153290 2023-03-03T15:42:34.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d98dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9406c360 con 0x7f1d940cd390 2023-03-03T15:42:34.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d98dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9415cea0 con 0x7f1d940cd8e0 2023-03-03T15:42:34.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d92d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d940cd390 0x7f1d940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:34.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d92d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d940cd390 0x7f1d940ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60626/0 (socket says 172.21.15.102:60626) 2023-03-03T15:42:34.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d92d9d700 1 -- 172.21.15.102:0/571188674 learned_addr learned my addr 172.21.15.102:0/571188674 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:42:34.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.813+0000 7f1d91d9b700 1 --2- 172.21.15.102:0/571188674 >> v2:172.21.15.172:3300/0 conn(0x7f1d940cd8e0 0x7f1d940cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:34.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d91d9b700 1 -- 172.21.15.102:0/571188674 >> v2:172.21.15.102:3300/0 conn(0x7f1d940cd390 msgr2=0x7f1d940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:34.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d91d9b700 1 --2- 172.21.15.102:0/571188674 >> v2:172.21.15.102:3300/0 conn(0x7f1d940cd390 0x7f1d940ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:34.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d91d9b700 1 -- 172.21.15.102:0/571188674 >> v1:172.21.15.172:6789/0 conn(0x7f1d94153290 legacy=0x7f1d94158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:34.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d91d9b700 1 -- 172.21.15.102:0/571188674 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9406c7f0 con 0x7f1d940cd8e0 2023-03-03T15:42:34.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d91d9b700 1 --2- 172.21.15.102:0/571188674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d940cd8e0 0x7f1d940cde20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1d94159fb0 tx=0x7f1d88009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ae826b272e9da88 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:34.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d9159a700 1 -- 172.21.15.102:0/571188674 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88011830 con 0x7f1d940cd8e0 2023-03-03T15:42:34.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d9159a700 1 -- 172.21.15.102:0/571188674 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88011990 con 0x7f1d940cd8e0 2023-03-03T15:42:34.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.814+0000 7f1d9159a700 1 -- 172.21.15.102:0/571188674 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88011d20 con 0x7f1d940cd8e0 2023-03-03T15:42:34.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 -- 172.21.15.102:0/571188674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d940cd8e0 msgr2=0x7f1d940cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:34.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 --2- 172.21.15.102:0/571188674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d940cd8e0 0x7f1d940cde20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1d94159fb0 tx=0x7f1d88009b20 comp rx=0 tx=0).stop 2023-03-03T15:42:34.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 -- 172.21.15.102:0/571188674 shutdown_connections 2023-03-03T15:42:34.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 --2- 172.21.15.102:0/571188674 >> v2:172.21.15.102:3300/0 conn(0x7f1d940cd390 0x7f1d940ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:34.818 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 --2- 172.21.15.102:0/571188674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d940cd8e0 0x7f1d940cde20 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:34.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 -- 172.21.15.102:0/571188674 >> 172.21.15.102:0/571188674 conn(0x7f1d94065fb0 msgr2=0x7f1d940663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:34.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 -- 172.21.15.102:0/571188674 shutdown_connections 2023-03-03T15:42:34.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.815+0000 7f1d98dff700 1 -- 172.21.15.102:0/571188674 wait complete. 2023-03-03T15:42:34.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.817+0000 7f1d98dff700 1 Processor -- start 2023-03-03T15:42:34.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.817+0000 7f1d98dff700 1 -- start start 2023-03-03T15:42:34.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.818+0000 7f1d98dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d940cd390 0x7f1d940c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.818+0000 7f1d98dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d940cd8e0 0x7f1d941f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.818+0000 7f1d98dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d94153290 0x7f1d941e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.818+0000 7f1d98dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940c6950 con 0x7f1d940cd8e0 2023-03-03T15:42:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.818+0000 7f1d98dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940c6140 con 0x7f1d94153290 2023-03-03T15:42:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.818+0000 7f1d98dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940c85d0 con 0x7f1d940cd390 2023-03-03T15:42:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d940cd8e0 0x7f1d941f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d9259c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d940cd390 0x7f1d940c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d92d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d94153290 0x7f1d941e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d940cd8e0 0x7f1d941f1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60642/0 (socket says 172.21.15.102:60642) 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 -- 172.21.15.102:0/491891035 learned_addr learned my addr 172.21.15.102:0/491891035 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 -- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d940cd390 msgr2=0x7f1d940c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d940cd390 0x7f1d940c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 -- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d94153290 msgr2=0x7f1d941e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d94153290 0x7f1d941e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:34.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 -- 172.21.15.102:0/491891035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d880094a0 con 0x7f1d940cd8e0 2023-03-03T15:42:34.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.819+0000 7f1d91d9b700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d940cd8e0 0x7f1d941f1ea0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009b60 tx=0x7f1d88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:34.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.820+0000 7f1d82ffd700 1 -- 172.21.15.102:0/491891035 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880036a0 con 0x7f1d940cd8e0 2023-03-03T15:42:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.820+0000 7f1d82ffd700 1 -- 172.21.15.102:0/491891035 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8801f070 con 0x7f1d940cd8e0 2023-03-03T15:42:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.820+0000 7f1d82ffd700 1 -- 172.21.15.102:0/491891035 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88003a60 con 0x7f1d940cd8e0 2023-03-03T15:42:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.820+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d940ccde0 con 0x7f1d940cd8e0 2023-03-03T15:42:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.820+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9415d160 con 0x7f1d940cd8e0 2023-03-03T15:42:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.821+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d940cd0c0 con 0x7f1d940cd8e0 2023-03-03T15:42:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.822+0000 7f1d82ffd700 1 -- 172.21.15.102:0/491891035 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1d8800f050 con 0x7f1d940cd8e0 2023-03-03T15:42:34.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.824+0000 7f1d82ffd700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d74068320 0x7f1d7406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:34.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.824+0000 7f1d82ffd700 1 -- 172.21.15.102:0/491891035 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1d88021070 con 0x7f1d940cd8e0 2023-03-03T15:42:34.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.824+0000 7f1d9259c700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d74068320 0x7f1d7406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:34.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.825+0000 7f1d9259c700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d74068320 0x7f1d7406a7d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f1d84007fd0 tx=0x7f1d84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:34.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:34.828+0000 7f1d82ffd700 1 -- 172.21.15.102:0/491891035 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1d8808d080 con 0x7f1d940cd8e0 2023-03-03T15:42:35.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.062+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d940c8430 con 0x7f1d940cd8e0 2023-03-03T15:42:35.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.063+0000 7f1d82ffd700 1 -- 172.21.15.102:0/491891035 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1d940c8430 con 0x7f1d940cd8e0 2023-03-03T15:42:35.064 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:42:35.064 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:42:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d74068320 msgr2=0x7f1d7406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d74068320 0x7f1d7406a7d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f1d84007fd0 tx=0x7f1d84009040 comp rx=0 tx=0).stop 2023-03-03T15:42:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d940cd8e0 msgr2=0x7f1d941f1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d940cd8e0 0x7f1d941f1ea0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009b60 tx=0x7f1d88004750 comp rx=0 tx=0).stop 2023-03-03T15:42:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 shutdown_connections 2023-03-03T15:42:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d74068320 0x7f1d7406a7d0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d940cd390 0x7f1d940c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d94153290 0x7f1d941e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 --2- 172.21.15.102:0/491891035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d940cd8e0 0x7f1d941f1ea0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.065+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 >> 172.21.15.102:0/491891035 conn(0x7f1d94065fb0 msgr2=0x7f1d94157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.066+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 shutdown_connections 2023-03-03T15:42:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:42:35.066+0000 7f1d98dff700 1 -- 172.21.15.102:0/491891035 wait complete. 2023-03-03T15:42:35.068 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:42:35.094 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-03-03T15:42:35.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:35.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:06.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:06.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:36.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:36.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:50.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:43:50.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.344+0000 7f8c87dff700 1 Processor -- start 2023-03-03T15:43:50.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.344+0000 7f8c87dff700 1 -- start start 2023-03-03T15:43:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c87dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8c800cd8e0 0x7f8c800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c87dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:50.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c87dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c80153660 con 0x7f8c80153290 2023-03-03T15:43:50.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c87dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8006c360 con 0x7f8c800cd8e0 2023-03-03T15:43:50.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c87dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8015cea0 con 0x7f8c800cd390 2023-03-03T15:43:50.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c85b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8c80153290 0x7f8c80158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57590/0 (socket says 172.21.15.102:57590) 2023-03-03T15:43:50.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c85b9b700 1 -- 172.21.15.102:0/1711117174 learned_addr learned my addr 172.21.15.102:0/1711117174 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:43:50.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.345+0000 7f8c8539a700 1 --2- 172.21.15.102:0/1711117174 >> v2:172.21.15.102:3300/0 conn(0x7f8c800cd8e0 0x7f8c800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:50.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.346+0000 7f8c8639c700 1 --2- 172.21.15.102:0/1711117174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:50.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.346+0000 7f8c8539a700 1 -- 172.21.15.102:0/1711117174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 msgr2=0x7f8c800ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:43:50.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.346+0000 7f8c84b99700 1 -- 172.21.15.102:0/1711117174 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79810976 0 0) 0x7f8c80153660 con 0x7f8c80153290 2023-03-03T15:43:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.346+0000 7f8c8539a700 1 --2- 172.21.15.102:0/1711117174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.346+0000 7f8c8539a700 1 -- 172.21.15.102:0/1711117174 >> v1:172.21.15.172:6789/0 conn(0x7f8c80153290 legacy=0x7f8c80158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.346+0000 7f8c8539a700 1 -- 172.21.15.102:0/1711117174 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8006c7f0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.346+0000 7f8c8539a700 1 --2- 172.21.15.102:0/1711117174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c800cd8e0 0x7f8c800cde20 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009840 tx=0x7f8c70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947d94aeaf64eb72 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:43:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c84b99700 1 -- 172.21.15.102:0/1711117174 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70011830 con 0x7f8c800cd8e0 2023-03-03T15:43:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c84b99700 1 -- 172.21.15.102:0/1711117174 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c70011990 con 0x7f8c800cd8e0 2023-03-03T15:43:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c84b99700 1 -- 172.21.15.102:0/1711117174 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70011d20 con 0x7f8c800cd8e0 2023-03-03T15:43:50.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 -- 172.21.15.102:0/1711117174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c800cd8e0 msgr2=0x7f8c800cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:50.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1711117174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c800cd8e0 0x7f8c800cde20 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009840 tx=0x7f8c70009b50 comp rx=0 tx=0).stop 2023-03-03T15:43:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 -- 172.21.15.102:0/1711117174 shutdown_connections 2023-03-03T15:43:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1711117174 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1711117174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c800cd8e0 0x7f8c800cde20 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 -- 172.21.15.102:0/1711117174 >> 172.21.15.102:0/1711117174 conn(0x7f8c80065fb0 msgr2=0x7f8c800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:43:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 -- 172.21.15.102:0/1711117174 shutdown_connections 2023-03-03T15:43:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.347+0000 7f8c87dff700 1 -- 172.21.15.102:0/1711117174 wait complete. 2023-03-03T15:43:50.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.349+0000 7f8c87dff700 1 Processor -- start 2023-03-03T15:43:50.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.350+0000 7f8c87dff700 1 -- start start 2023-03-03T15:43:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.350+0000 7f8c87dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.350+0000 7f8c87dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c800cd8e0 0x7f8c8015ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c87dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c80153290 0x7f8c800c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:50.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c87dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c801e3ed0 con 0x7f8c80153290 2023-03-03T15:43:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c87dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c801ef270 con 0x7f8c800cd8e0 2023-03-03T15:43:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c87dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c801ef550 con 0x7f8c800cd390 2023-03-03T15:43:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c85b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c85b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51134/0 (socket says 172.21.15.102:51134) 2023-03-03T15:43:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c85b9b700 1 -- 172.21.15.102:0/1612450015 learned_addr learned my addr 172.21.15.102:0/1612450015 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:43:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c8539a700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c800cd8e0 0x7f8c8015ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:50.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c8639c700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c80153290 0x7f8c800c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c8539a700 1 -- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 msgr2=0x7f8c800b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c8539a700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c8539a700 1 -- 172.21.15.102:0/1612450015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c80153290 msgr2=0x7f8c800c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c8539a700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c80153290 0x7f8c800c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.351+0000 7f8c8539a700 1 -- 172.21.15.102:0/1612450015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c700094a0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.352+0000 7f8c8539a700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c800cd8e0 0x7f8c8015ae70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8c70016000 tx=0x7f8c70004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.352+0000 7f8c767fc700 1 -- 172.21.15.102:0/1612450015 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c700036a0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.352+0000 7f8c767fc700 1 -- 172.21.15.102:0/1612450015 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7001e070 con 0x7f8c800cd8e0 2023-03-03T15:43:50.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.352+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c800ccde0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.352+0000 7f8c767fc700 1 -- 172.21.15.102:0/1612450015 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70003a40 con 0x7f8c800cd8e0 2023-03-03T15:43:50.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.352+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8015d160 con 0x7f8c800cd8e0 2023-03-03T15:43:50.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.353+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c800cd0c0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.355+0000 7f8c767fc700 1 -- 172.21.15.102:0/1612450015 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c7000f050 con 0x7f8c800cd8e0 2023-03-03T15:43:50.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.356+0000 7f8c767fc700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c64068320 0x7f8c6406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:50.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.356+0000 7f8c767fc700 1 -- 172.21.15.102:0/1612450015 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8c70087830 con 0x7f8c800cd8e0 2023-03-03T15:43:50.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.356+0000 7f8c85b9b700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c64068320 0x7f8c6406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:50.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.356+0000 7f8c85b9b700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c64068320 0x7f8c6406a7d0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c000f60 tx=0x7f8c7c007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:43:50.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.359+0000 7f8c767fc700 1 -- 172.21.15.102:0/1612450015 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c70053fc0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.591+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c8004e6e0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.593+0000 7f8c767fc700 1 -- 172.21.15.102:0/1612450015 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8c700575e0 con 0x7f8c800cd8e0 2023-03-03T15:43:50.595 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:43:50.595 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:43:50.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c64068320 msgr2=0x7f8c6406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:50.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c64068320 0x7f8c6406a7d0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c000f60 tx=0x7f8c7c007ea0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c800cd8e0 msgr2=0x7f8c8015ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c800cd8e0 0x7f8c8015ae70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8c70016000 tx=0x7f8c70004750 comp rx=0 tx=0).stop 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 shutdown_connections 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c64068320 0x7f8c6406a7d0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c800cd390 0x7f8c800b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c80153290 0x7f8c800c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 --2- 172.21.15.102:0/1612450015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c800cd8e0 0x7f8c8015ae70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 >> 172.21.15.102:0/1612450015 conn(0x7f8c80065fb0 msgr2=0x7f8c80158940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:43:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 shutdown_connections 2023-03-03T15:43:50.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:43:50.595+0000 7f8c87dff700 1 -- 172.21.15.102:0/1612450015 wait complete. 2023-03-03T15:43:50.599 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:43:50.624 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T15:44:06.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:06.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:36.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:36.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:00.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:45:00.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.876+0000 7ff7c5fff700 1 Processor -- start 2023-03-03T15:45:00.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.876+0000 7ff7c5fff700 1 -- start start 2023-03-03T15:45:00.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.877+0000 7ff7c5fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c00cd8e0 0x7ff7c00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:00.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.877+0000 7ff7c5fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff7c00cd390 0x7ff7c00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:00.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.877+0000 7ff7c5fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c0153660 con 0x7ff7c0153290 2023-03-03T15:45:00.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.877+0000 7ff7c5fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c006c360 con 0x7ff7c00cd390 2023-03-03T15:45:00.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.877+0000 7ff7c5fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c015cea0 con 0x7ff7c00cd8e0 2023-03-03T15:45:00.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7bf7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff7c0153290 0x7ff7c0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:52684/0 (socket says 172.21.15.102:52684) 2023-03-03T15:45:00.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7bf7fe700 1 -- 172.21.15.102:0/942701423 learned_addr learned my addr 172.21.15.102:0/942701423 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:45:00.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7beffd700 1 --2- 172.21.15.102:0/942701423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c00cd8e0 0x7ff7c00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:00.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7bffff700 1 --2- 172.21.15.102:0/942701423 >> v2:172.21.15.172:3300/0 conn(0x7ff7c00cd390 0x7ff7c00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:00.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7be7fc700 1 -- 172.21.15.102:0/942701423 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754413483 0 0) 0x7ff7c0153660 con 0x7ff7c0153290 2023-03-03T15:45:00.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7be7fc700 1 -- 172.21.15.102:0/942701423 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a80035b0 con 0x7ff7c0153290 2023-03-03T15:45:00.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7beffd700 1 -- 172.21.15.102:0/942701423 >> v2:172.21.15.172:3300/0 conn(0x7ff7c00cd390 msgr2=0x7ff7c00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:45:00.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7beffd700 1 --2- 172.21.15.102:0/942701423 >> v2:172.21.15.172:3300/0 conn(0x7ff7c00cd390 0x7ff7c00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:00.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7beffd700 1 -- 172.21.15.102:0/942701423 >> v1:172.21.15.172:6789/0 conn(0x7ff7c0153290 legacy=0x7ff7c0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:00.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7beffd700 1 -- 172.21.15.102:0/942701423 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c006c7f0 con 0x7ff7c00cd8e0 2023-03-03T15:45:00.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.878+0000 7ff7bffff700 1 --2- 172.21.15.102:0/942701423 >> v2:172.21.15.172:3300/0 conn(0x7ff7c00cd390 0x7ff7c00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:45:00.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7beffd700 1 --2- 172.21.15.102:0/942701423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c00cd8e0 0x7ff7c00cde20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009840 tx=0x7ff7b4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ba2a35bf6b469b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:00.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7be7fc700 1 -- 172.21.15.102:0/942701423 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4011830 con 0x7ff7c00cd8e0 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7be7fc700 1 -- 172.21.15.102:0/942701423 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b4011990 con 0x7ff7c00cd8e0 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7be7fc700 1 -- 172.21.15.102:0/942701423 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4011d20 con 0x7ff7c00cd8e0 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7c5fff700 1 -- 172.21.15.102:0/942701423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c00cd8e0 msgr2=0x7ff7c00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/942701423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c00cd8e0 0x7ff7c00cde20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009840 tx=0x7ff7b4009b50 comp rx=0 tx=0).stop 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7c5fff700 1 -- 172.21.15.102:0/942701423 shutdown_connections 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/942701423 >> v2:172.21.15.172:3300/0 conn(0x7ff7c00cd390 0x7ff7c00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/942701423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c00cd8e0 0x7ff7c00cde20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.879+0000 7ff7c5fff700 1 -- 172.21.15.102:0/942701423 >> 172.21.15.102:0/942701423 conn(0x7ff7c0065fb0 msgr2=0x7ff7c00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:00.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.880+0000 7ff7c5fff700 1 -- 172.21.15.102:0/942701423 shutdown_connections 2023-03-03T15:45:00.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.880+0000 7ff7c5fff700 1 -- 172.21.15.102:0/942701423 wait complete. 2023-03-03T15:45:00.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.881+0000 7ff7c5fff700 1 Processor -- start 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.882+0000 7ff7c5fff700 1 -- start start 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7c5fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7c00cd390 0x7ff7c01efed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7c5fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c00cd8e0 0x7ff7c00b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7c5fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c0153290 0x7ff7c0155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7c5fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b44b0 con 0x7ff7c00cd8e0 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7c5fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b2cc0 con 0x7ff7c00cd390 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7c5fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b2ed0 con 0x7ff7c0153290 2023-03-03T15:45:00.886 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bf7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7c00cd390 0x7ff7c01efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:00.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bf7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7c00cd390 0x7ff7c01efed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:36188/0 (socket says 172.21.15.102:36188) 2023-03-03T15:45:00.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bf7fe700 1 -- 172.21.15.102:0/1988568514 learned_addr learned my addr 172.21.15.102:0/1988568514 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:45:00.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7beffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c00cd8e0 0x7ff7c00b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:00.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bffff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c0153290 0x7ff7c0155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:00.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bf7fe700 1 -- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c0153290 msgr2=0x7ff7c0155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:00.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bf7fe700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c0153290 0x7ff7c0155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:00.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bf7fe700 1 -- 172.21.15.102:0/1988568514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c00cd8e0 msgr2=0x7ff7c00b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.883+0000 7ff7bf7fe700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c00cd8e0 0x7ff7c00b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7bf7fe700 1 -- 172.21.15.102:0/1988568514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8009a70 con 0x7ff7c00cd390 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7bffff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c0153290 0x7ff7c0155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7bf7fe700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7c00cd390 0x7ff7c01efed0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c730 tx=0x7ff7b000ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7a7fff700 1 -- 172.21.15.102:0/1988568514 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0005d70 con 0x7ff7c00cd390 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7a7fff700 1 -- 172.21.15.102:0/1988568514 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b00070d0 con 0x7ff7c00cd390 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7a7fff700 1 -- 172.21.15.102:0/1988568514 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0007460 con 0x7ff7c00cd390 2023-03-03T15:45:00.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b40094a0 con 0x7ff7c00cd390 2023-03-03T15:45:00.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.884+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c00ccde0 con 0x7ff7c00cd390 2023-03-03T15:45:00.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.886+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c015d160 con 0x7ff7c00cd390 2023-03-03T15:45:00.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.887+0000 7ff7a7fff700 1 -- 172.21.15.102:0/1988568514 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff7b0019030 con 0x7ff7c00cd390 2023-03-03T15:45:00.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.888+0000 7ff7a7fff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7ac0682f0 0x7ff7ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:00.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.888+0000 7ff7a7fff700 1 -- 172.21.15.102:0/1988568514 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff7b00851b0 con 0x7ff7c00cd390 2023-03-03T15:45:00.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.888+0000 7ff7beffd700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7ac0682f0 0x7ff7ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:00.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.889+0000 7ff7beffd700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7ac0682f0 0x7ff7ac06a7a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4000c00 tx=0x7ff7b401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:00.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:00.892+0000 7ff7a7fff700 1 -- 172.21.15.102:0/1988568514 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7b0051830 con 0x7ff7c00cd390 2023-03-03T15:45:01.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.127+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7c00cd0c0 con 0x7ff7c00cd390 2023-03-03T15:45:01.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.128+0000 7ff7a7fff700 1 -- 172.21.15.102:0/1988568514 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff7b0017090 con 0x7ff7c00cd390 2023-03-03T15:45:01.130 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:45:01.130 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:45:01.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7ac0682f0 msgr2=0x7ff7ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:01.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7ac0682f0 0x7ff7ac06a7a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4000c00 tx=0x7ff7b401c040 comp rx=0 tx=0).stop 2023-03-03T15:45:01.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7c00cd390 msgr2=0x7ff7c01efed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:01.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7c00cd390 0x7ff7c01efed0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c730 tx=0x7ff7b000ca40 comp rx=0 tx=0).stop 2023-03-03T15:45:01.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 shutdown_connections 2023-03-03T15:45:01.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff7ac0682f0 0x7ff7ac06a7a0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff7c0153290 0x7ff7c0155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7c00cd390 0x7ff7c01efed0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 --2- 172.21.15.102:0/1988568514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c00cd8e0 0x7ff7c00b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 >> 172.21.15.102:0/1988568514 conn(0x7ff7c0065fb0 msgr2=0x7ff7c00cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:01.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 shutdown_connections 2023-03-03T15:45:01.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:01.131+0000 7ff7c5fff700 1 -- 172.21.15.102:0/1988568514 wait complete. 2023-03-03T15:45:01.134 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:45:01.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T15:45:06.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:06.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:36.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:45:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.422+0000 7f236c5ff700 1 Processor -- start 2023-03-03T15:45:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.422+0000 7f236c5ff700 1 -- start start 2023-03-03T15:45:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.423+0000 7f236c5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2364153290 0x7f2364158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.423+0000 7f236c5ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364153660 con 0x7f23640cd8e0 2023-03-03T15:45:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.423+0000 7f236c5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236406c360 con 0x7f23641593c0 2023-03-03T15:45:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.423+0000 7f236c5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23640cec40 con 0x7f2364153290 2023-03-03T15:45:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f236a39b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2364153290 0x7f2364158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f2369b9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f23641593c0 0x7f23640cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46926/0 (socket says 172.21.15.102:46926) 2023-03-03T15:45:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f2369b9a700 1 -- 172.21.15.102:0/3702415760 learned_addr learned my addr 172.21.15.102:0/3702415760 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:45:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f2369399700 1 -- 172.21.15.102:0/3702415760 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524381646 0 0) 0x7f236406c360 con 0x7f23641593c0 2023-03-03T15:45:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f236a39b700 1 -- 172.21.15.102:0/3702415760 >> v1:172.21.15.172:6789/0 conn(0x7f23641593c0 legacy=0x7f23640cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f236a39b700 1 -- 172.21.15.102:0/3702415760 >> v1:172.21.15.102:6789/0 conn(0x7f23640cd8e0 legacy=0x7f23640ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f236a39b700 1 -- 172.21.15.102:0/3702415760 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236415cea0 con 0x7f2364153290 2023-03-03T15:45:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.424+0000 7f236a39b700 1 --2- 172.21.15.102:0/3702415760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2364153290 0x7f2364158e80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2360009840 tx=0x7f2360009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54dd26d632ce5b27 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f2369399700 1 -- 172.21.15.102:0/3702415760 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360011830 con 0x7f2364153290 2023-03-03T15:45:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f2369399700 1 -- 172.21.15.102:0/3702415760 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2360011990 con 0x7f2364153290 2023-03-03T15:45:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f2369399700 1 -- 172.21.15.102:0/3702415760 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360011d20 con 0x7f2364153290 2023-03-03T15:45:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f236c5ff700 1 -- 172.21.15.102:0/3702415760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2364153290 msgr2=0x7f2364158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3702415760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2364153290 0x7f2364158e80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2360009840 tx=0x7f2360009b50 comp rx=0 tx=0).stop 2023-03-03T15:45:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f236c5ff700 1 -- 172.21.15.102:0/3702415760 shutdown_connections 2023-03-03T15:45:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3702415760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2364153290 0x7f2364158e80 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f236c5ff700 1 -- 172.21.15.102:0/3702415760 >> 172.21.15.102:0/3702415760 conn(0x7f2364065fb0 msgr2=0x7f23640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.425+0000 7f236c5ff700 1 -- 172.21.15.102:0/3702415760 shutdown_connections 2023-03-03T15:45:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.426+0000 7f236c5ff700 1 -- 172.21.15.102:0/3702415760 wait complete. 2023-03-03T15:45:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.427+0000 7f236c5ff700 1 Processor -- start 2023-03-03T15:45:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.428+0000 7f236c5ff700 1 -- start start 2023-03-03T15:45:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.428+0000 7f236c5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23640cd8e0 0x7f23641e55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236c5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2364153290 0x7f23641e5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236c5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23641593c0 0x7f23640c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236c5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640c7730 con 0x7f23640cd8e0 2023-03-03T15:45:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236c5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23641e6030 con 0x7f23641593c0 2023-03-03T15:45:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236c5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23641ef220 con 0x7f2364153290 2023-03-03T15:45:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23640cd8e0 0x7f23641e55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23640cd8e0 0x7f23641e55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43626/0 (socket says 172.21.15.102:43626) 2023-03-03T15:45:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 -- 172.21.15.102:0/3556864943 learned_addr learned my addr 172.21.15.102:0/3556864943 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:45:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f2369b9a700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2364153290 0x7f23641e5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236ab9c700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23641593c0 0x7f23640c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 -- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2364153290 msgr2=0x7f23641e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2364153290 0x7f23641e5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 -- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23641593c0 msgr2=0x7f23640c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23641593c0 0x7f23640c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.429+0000 7f236a39b700 1 -- 172.21.15.102:0/3556864943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23600094a0 con 0x7f23640cd8e0 2023-03-03T15:45:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.430+0000 7f236a39b700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23640cd8e0 0x7f23641e55b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f23600090b0 tx=0x7f2360004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.430+0000 7f235affd700 1 -- 172.21.15.102:0/3556864943 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236001a760 con 0x7f23640cd8e0 2023-03-03T15:45:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.430+0000 7f235affd700 1 -- 172.21.15.102:0/3556864943 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236001a8c0 con 0x7f23640cd8e0 2023-03-03T15:45:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.430+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236415d160 con 0x7f23640cd8e0 2023-03-03T15:45:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.430+0000 7f235affd700 1 -- 172.21.15.102:0/3556864943 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236001ac30 con 0x7f23640cd8e0 2023-03-03T15:45:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.430+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236406c7f0 con 0x7f23640cd8e0 2023-03-03T15:45:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.431+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23640ccde0 con 0x7f23640cd8e0 2023-03-03T15:45:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.432+0000 7f235affd700 1 -- 172.21.15.102:0/3556864943 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f236002b030 con 0x7f23640cd8e0 2023-03-03T15:45:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.434+0000 7f235affd700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23480682f0 0x7f234806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.434+0000 7f235affd700 1 -- 172.21.15.102:0/3556864943 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2360086ff0 con 0x7f23640cd8e0 2023-03-03T15:45:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.434+0000 7f2369b9a700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23480682f0 0x7f234806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.435+0000 7f2369b9a700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23480682f0 0x7f234806a7a0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f23640b4900 tx=0x7f2354007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:36.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.438+0000 7f235affd700 1 -- 172.21.15.102:0/3556864943 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23600b85e0 con 0x7f23640cd8e0 2023-03-03T15:45:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.673+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23640cc750 con 0x7f23640cd8e0 2023-03-03T15:45:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.674+0000 7f235affd700 1 -- 172.21.15.102:0/3556864943 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2360029090 con 0x7f23640cd8e0 2023-03-03T15:45:36.675 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:45:36.675 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:45:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23480682f0 msgr2=0x7f234806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23480682f0 0x7f234806a7a0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f23640b4900 tx=0x7f2354007a60 comp rx=0 tx=0).stop 2023-03-03T15:45:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23640cd8e0 msgr2=0x7f23641e55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23640cd8e0 0x7f23641e55b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f23600090b0 tx=0x7f2360004750 comp rx=0 tx=0).stop 2023-03-03T15:45:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 shutdown_connections 2023-03-03T15:45:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23641593c0 0x7f23640c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23640cd8e0 0x7f23641e55b0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f23480682f0 0x7f234806a7a0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 --2- 172.21.15.102:0/3556864943 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2364153290 0x7f23641e5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 >> 172.21.15.102:0/3556864943 conn(0x7f2364065fb0 msgr2=0x7f236415baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 shutdown_connections 2023-03-03T15:45:36.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:45:36.677+0000 7f236c5ff700 1 -- 172.21.15.102:0/3556864943 wait complete. 2023-03-03T15:45:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:45:36.705 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-03-03T15:45:36.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:36.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:07.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:07.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:22.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:46:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.959+0000 7f973e9ff700 1 Processor -- start 2023-03-03T15:46:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.959+0000 7f973e9ff700 1 -- start start 2023-03-03T15:46:22.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.960+0000 7f973e9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9738153290 0x7f9738158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f973e9ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9738153660 con 0x7f97381593c0 2023-03-03T15:46:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f973e9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973806c360 con 0x7f97380cd8e0 2023-03-03T15:46:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f973e9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97380cec40 con 0x7f9738153290 2023-03-03T15:46:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f9737fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9738153290 0x7f9738158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:22.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f9737fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9738153290 0x7f9738158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48644/0 (socket says 172.21.15.102:48644) 2023-03-03T15:46:22.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f9737fff700 1 -- 172.21.15.102:0/282763095 learned_addr learned my addr 172.21.15.102:0/282763095 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:46:22.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f97377fe700 1 --1- 172.21.15.102:0/282763095 >> v1:172.21.15.102:6789/0 conn(0x7f97381593c0 0x7f97380cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43762/0 (socket says 172.21.15.102:43762) 2023-03-03T15:46:22.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f9736ffd700 1 -- 172.21.15.102:0/282763095 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493458102 0 0) 0x7f973806c360 con 0x7f97380cd8e0 2023-03-03T15:46:22.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.961+0000 7f9737fff700 1 -- 172.21.15.102:0/282763095 >> v1:172.21.15.172:6789/0 conn(0x7f97380cd8e0 legacy=0x7f97380ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:22.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.962+0000 7f9737fff700 1 -- 172.21.15.102:0/282763095 >> v1:172.21.15.102:6789/0 conn(0x7f97381593c0 legacy=0x7f97380cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:22.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.962+0000 7f9737fff700 1 -- 172.21.15.102:0/282763095 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973815cea0 con 0x7f9738153290 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.962+0000 7f9737fff700 1 --2- 172.21.15.102:0/282763095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9738153290 0x7f9738158e80 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f97280098f0 tx=0x7f9728009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09813d41e1e9dbc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.962+0000 7f9736ffd700 1 -- 172.21.15.102:0/282763095 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800b8a0 con 0x7f9738153290 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.962+0000 7f9736ffd700 1 -- 172.21.15.102:0/282763095 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800ba00 con 0x7f9738153290 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.962+0000 7f9736ffd700 1 -- 172.21.15.102:0/282763095 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bd90 con 0x7f9738153290 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.963+0000 7f973e9ff700 1 -- 172.21.15.102:0/282763095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9738153290 msgr2=0x7f9738158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.963+0000 7f973e9ff700 1 --2- 172.21.15.102:0/282763095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9738153290 0x7f9738158e80 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f97280098f0 tx=0x7f9728009c00 comp rx=0 tx=0).stop 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.963+0000 7f973e9ff700 1 -- 172.21.15.102:0/282763095 shutdown_connections 2023-03-03T15:46:22.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.963+0000 7f973e9ff700 1 --2- 172.21.15.102:0/282763095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9738153290 0x7f9738158e80 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:22.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.963+0000 7f973e9ff700 1 -- 172.21.15.102:0/282763095 >> 172.21.15.102:0/282763095 conn(0x7f9738065fb0 msgr2=0x7f97380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:46:22.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.963+0000 7f973e9ff700 1 -- 172.21.15.102:0/282763095 shutdown_connections 2023-03-03T15:46:22.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.963+0000 7f973e9ff700 1 -- 172.21.15.102:0/282763095 wait complete. 2023-03-03T15:46:22.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.965+0000 7f973e9ff700 1 Processor -- start 2023-03-03T15:46:22.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.965+0000 7f973e9ff700 1 -- start start 2023-03-03T15:46:22.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f973e9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97380cd8e0 0x7f973815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:22.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f973e9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9738153290 0x7f97381549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:22.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f973e9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97381593c0 0x7f9738152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:22.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f973e9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97381f2170 con 0x7f97381593c0 2023-03-03T15:46:22.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f973e9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97381f1880 con 0x7f9738153290 2023-03-03T15:46:22.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f973e9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97381eee50 con 0x7f97380cd8e0 2023-03-03T15:46:22.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f97377fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9738153290 0x7f97381549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:22.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f973cf9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97381593c0 0x7f9738152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:22.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f97377fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9738153290 0x7f97381549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47976/0 (socket says 172.21.15.102:47976) 2023-03-03T15:46:22.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f97377fe700 1 -- 172.21.15.102:0/4179618590 learned_addr learned my addr 172.21.15.102:0/4179618590 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:46:22.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.966+0000 7f9737fff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97380cd8e0 0x7f973815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:22.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f97377fe700 1 -- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97380cd8e0 msgr2=0x7f973815c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:22.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f97377fe700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97380cd8e0 0x7f973815c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:22.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f97377fe700 1 -- 172.21.15.102:0/4179618590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97381593c0 msgr2=0x7f9738152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:22.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f97377fe700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97381593c0 0x7f9738152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:22.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f97377fe700 1 -- 172.21.15.102:0/4179618590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9728009550 con 0x7f9738153290 2023-03-03T15:46:22.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f97377fe700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9738153290 0x7f97381549a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c520 tx=0x7f972c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:46:22.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f9734ff9700 1 -- 172.21.15.102:0/4179618590 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c005cb0 con 0x7f9738153290 2023-03-03T15:46:22.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f9734ff9700 1 -- 172.21.15.102:0/4179618590 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c005e10 con 0x7f9738153290 2023-03-03T15:46:22.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973815d160 con 0x7f9738153290 2023-03-03T15:46:22.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.967+0000 7f9734ff9700 1 -- 172.21.15.102:0/4179618590 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c0072d0 con 0x7f9738153290 2023-03-03T15:46:22.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.968+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973806c7f0 con 0x7f9738153290 2023-03-03T15:46:22.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.969+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97380ccde0 con 0x7f9738153290 2023-03-03T15:46:22.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.970+0000 7f9734ff9700 1 -- 172.21.15.102:0/4179618590 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f972c019030 con 0x7f9738153290 2023-03-03T15:46:22.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.971+0000 7f9734ff9700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9724068320 0x7f972406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:22.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.971+0000 7f9734ff9700 1 -- 172.21.15.102:0/4179618590 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f972c084fb0 con 0x7f9738153290 2023-03-03T15:46:22.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.972+0000 7f9737fff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9724068320 0x7f972406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:22.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.972+0000 7f9737fff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9724068320 0x7f972406a7d0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f9728014040 tx=0x7f972801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:46:22.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:22.976+0000 7f9734ff9700 1 -- 172.21.15.102:0/4179618590 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f972c04b5e0 con 0x7f9738153290 2023-03-03T15:46:23.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.211+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97380c9490 con 0x7f9738153290 2023-03-03T15:46:23.212 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.212+0000 7f9734ff9700 1 -- 172.21.15.102:0/4179618590 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f972c017070 con 0x7f9738153290 2023-03-03T15:46:23.213 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:46:23.213 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:46:23.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9724068320 msgr2=0x7f972406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:23.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9724068320 0x7f972406a7d0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f9728014040 tx=0x7f972801c040 comp rx=0 tx=0).stop 2023-03-03T15:46:23.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9738153290 msgr2=0x7f97381549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9738153290 0x7f97381549a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c520 tx=0x7f972c00c830 comp rx=0 tx=0).stop 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 shutdown_connections 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9724068320 0x7f972406a7d0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f97380cd8e0 0x7f973815c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97381593c0 0x7f9738152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 --2- 172.21.15.102:0/4179618590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9738153290 0x7f97381549a0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 >> 172.21.15.102:0/4179618590 conn(0x7f9738065fb0 msgr2=0x7f97380cb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:46:23.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 shutdown_connections 2023-03-03T15:46:23.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:46:23.214+0000 7f973e9ff700 1 -- 172.21.15.102:0/4179618590 wait complete. 2023-03-03T15:46:23.217 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:46:23.242 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 41 secs before thrashing 2023-03-03T15:46:37.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:37.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:04.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:47:04.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.490+0000 7f1fa2bff700 1 Processor -- start 2023-03-03T15:47:04.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.491+0000 7f1fa2bff700 1 -- start start 2023-03-03T15:47:04.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa2bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f9c153290 0x7f1f9c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa2bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c1593c0 0x7f1f9c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa2bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9c153660 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa2bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c06c360 con 0x7f1f9c153290 2023-03-03T15:47:04.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa2bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c15cea0 con 0x7f1f9c1593c0 2023-03-03T15:47:04.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa099b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f9c153290 0x7f1f9c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1f9bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c1593c0 0x7f1f9c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.496 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa099b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f9c153290 0x7f1f9c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:35764/0 (socket says 172.21.15.102:35764) 2023-03-03T15:47:04.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.492+0000 7f1fa099b700 1 -- 172.21.15.102:0/471177254 learned_addr learned my addr 172.21.15.102:0/471177254 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:47:04.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1fa099b700 1 -- 172.21.15.102:0/471177254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c1593c0 msgr2=0x7f1f9c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:04.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1fa099b700 1 --2- 172.21.15.102:0/471177254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c1593c0 0x7f1f9c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1fa099b700 1 -- 172.21.15.102:0/471177254 >> v1:172.21.15.102:6789/0 conn(0x7f1f9c0cd8e0 legacy=0x7f1f9c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1fa099b700 1 -- 172.21.15.102:0/471177254 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c06c7f0 con 0x7f1f9c153290 2023-03-03T15:47:04.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1fa099b700 1 --2- 172.21.15.102:0/471177254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c153290 0x7f1f9c158e80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1f900098f0 tx=0x7f1f90009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b5cce9bc6a00f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1f9b7fe700 1 -- 172.21.15.102:0/471177254 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90011830 con 0x7f1f9c153290 2023-03-03T15:47:04.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1f9b7fe700 1 -- 172.21.15.102:0/471177254 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f90011990 con 0x7f1f9c153290 2023-03-03T15:47:04.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.493+0000 7f1f9b7fe700 1 -- 172.21.15.102:0/471177254 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90011d20 con 0x7f1f9c153290 2023-03-03T15:47:04.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 -- 172.21.15.102:0/471177254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c153290 msgr2=0x7f1f9c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/471177254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c153290 0x7f1f9c158e80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1f900098f0 tx=0x7f1f90009c00 comp rx=0 tx=0).stop 2023-03-03T15:47:04.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 -- 172.21.15.102:0/471177254 shutdown_connections 2023-03-03T15:47:04.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/471177254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c153290 0x7f1f9c158e80 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/471177254 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c1593c0 0x7f1f9c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 -- 172.21.15.102:0/471177254 >> 172.21.15.102:0/471177254 conn(0x7f1f9c065fb0 msgr2=0x7f1f9c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:04.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 -- 172.21.15.102:0/471177254 shutdown_connections 2023-03-03T15:47:04.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.494+0000 7f1fa2bff700 1 -- 172.21.15.102:0/471177254 wait complete. 2023-03-03T15:47:04.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.496+0000 7f1fa2bff700 1 Processor -- start 2023-03-03T15:47:04.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.496+0000 7f1fa2bff700 1 -- start start 2023-03-03T15:47:04.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.497+0000 7f1fa2bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f9c0cd8e0 0x7f1f9c151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.497+0000 7f1fa2bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c153290 0x7f1f9c1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.497+0000 7f1fa2bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c1593c0 0x7f1f9c1ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.497+0000 7f1fa2bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c1f3090 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.497+0000 7f1fa2bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c1ef630 con 0x7f1f9c1593c0 2023-03-03T15:47:04.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.497+0000 7f1fa2bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c1f2b30 con 0x7f1f9c153290 2023-03-03T15:47:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f9c0cd8e0 0x7f1f9c151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa119c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c1593c0 0x7f1f9c1ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f9c0cd8e0 0x7f1f9c151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40064/0 (socket says 172.21.15.102:40064) 2023-03-03T15:47:04.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 -- 172.21.15.102:0/1044301198 learned_addr learned my addr 172.21.15.102:0/1044301198 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:47:04.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1f9bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c153290 0x7f1f9c1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 -- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c153290 msgr2=0x7f1f9c1eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c153290 0x7f1f9c1eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 -- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c1593c0 msgr2=0x7f1f9c1ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c1593c0 0x7f1f9c1ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.498+0000 7f1fa099b700 1 -- 172.21.15.102:0/1044301198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f90009550 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.499+0000 7f1fa099b700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f9c0cd8e0 0x7f1f9c151130 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f1f90000c00 tx=0x7f1f90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.499+0000 7f1f997fa700 1 -- 172.21.15.102:0/1044301198 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90004480 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.499+0000 7f1f997fa700 1 -- 172.21.15.102:0/1044301198 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9001f070 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.499+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c0ccde0 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.499+0000 7f1f997fa700 1 -- 172.21.15.102:0/1044301198 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f900038c0 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.499+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c15d160 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.500+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c0cd0c0 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.501+0000 7f1f997fa700 1 -- 172.21.15.102:0/1044301198 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f9002a020 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.504+0000 7f1f997fa700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f800682c0 0x7f1f8006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.504+0000 7f1f997fa700 1 -- 172.21.15.102:0/1044301198 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1f90087330 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.504+0000 7f1f9bfff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f800682c0 0x7f1f8006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.505+0000 7f1f9bfff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f800682c0 0x7f1f8006a770 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0b4d70 tx=0x7f1f88009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.508+0000 7f1f997fa700 1 -- 172.21.15.102:0/1044301198 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f900539c0 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.743+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f9c06d430 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.744+0000 7f1f997fa700 1 -- 172.21.15.102:0/1044301198 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1f90056fe0 con 0x7f1f9c0cd8e0 2023-03-03T15:47:04.747 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:47:04.747 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:47:04.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f800682c0 msgr2=0x7f1f8006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f800682c0 0x7f1f8006a770 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0b4d70 tx=0x7f1f88009210 comp rx=0 tx=0).stop 2023-03-03T15:47:04.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f9c0cd8e0 msgr2=0x7f1f9c151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f9c0cd8e0 0x7f1f9c151130 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f1f90000c00 tx=0x7f1f90004750 comp rx=0 tx=0).stop 2023-03-03T15:47:04.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 shutdown_connections 2023-03-03T15:47:04.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f9c1593c0 0x7f1f9c1ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f9c0cd8e0 0x7f1f9c151130 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1f800682c0 0x7f1f8006a770 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.746+0000 7f1fa2bff700 1 --2- 172.21.15.102:0/1044301198 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f9c153290 0x7f1f9c1eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.747+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 >> 172.21.15.102:0/1044301198 conn(0x7f1f9c065fb0 msgr2=0x7f1f9c0cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:04.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.747+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 shutdown_connections 2023-03-03T15:47:04.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:04.747+0000 7f1fa2bff700 1 -- 172.21.15.102:0/1044301198 wait complete. 2023-03-03T15:47:04.753 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:47:04.776 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T15:47:07.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:07.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:37.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:37.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:44.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:47:45.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.026+0000 7f4c1c1ff700 1 Processor -- start 2023-03-03T15:47:45.029 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.027+0000 7f4c1c1ff700 1 -- start start 2023-03-03T15:47:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1c1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c14153290 0x7f4c14158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1c1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c141593c0 0x7f4c140cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1c1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c14153660 con 0x7f4c140cd8e0 2023-03-03T15:47:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1c1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1406c360 con 0x7f4c141593c0 2023-03-03T15:47:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1c1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1415cea0 con 0x7f4c14153290 2023-03-03T15:47:45.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1979a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c141593c0 0x7f4c140cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:45.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1979a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c141593c0 0x7f4c140cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47004/0 (socket says 172.21.15.102:47004) 2023-03-03T15:47:45.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c1979a700 1 -- 172.21.15.102:0/325433832 learned_addr learned my addr 172.21.15.102:0/325433832 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:47:45.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.028+0000 7f4c19f9b700 1 --2- 172.21.15.102:0/325433832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c14153290 0x7f4c14158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:45.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.029+0000 7f4c1979a700 1 -- 172.21.15.102:0/325433832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c14153290 msgr2=0x7f4c14158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:45.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.029+0000 7f4c1979a700 1 --2- 172.21.15.102:0/325433832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c14153290 0x7f4c14158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.029+0000 7f4c1979a700 1 -- 172.21.15.102:0/325433832 >> v1:172.21.15.102:6789/0 conn(0x7f4c140cd8e0 legacy=0x7f4c140ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:45.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.029+0000 7f4c1979a700 1 -- 172.21.15.102:0/325433832 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1406c7f0 con 0x7f4c141593c0 2023-03-03T15:47:45.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.029+0000 7f4c18f99700 1 -- 172.21.15.102:0/325433832 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017026156 0 0) 0x7f4c14153660 con 0x7f4c140cd8e0 2023-03-03T15:47:45.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.029+0000 7f4c1979a700 1 --2- 172.21.15.102:0/325433832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c141593c0 0x7f4c140cde20 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4c040098f0 tx=0x7f4c04009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7085b92af4a8106 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:45.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c18f99700 1 -- 172.21.15.102:0/325433832 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04011830 con 0x7f4c141593c0 2023-03-03T15:47:45.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c18f99700 1 -- 172.21.15.102:0/325433832 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c04011990 con 0x7f4c141593c0 2023-03-03T15:47:45.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c18f99700 1 -- 172.21.15.102:0/325433832 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04011d20 con 0x7f4c141593c0 2023-03-03T15:47:45.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/325433832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c141593c0 msgr2=0x7f4c140cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:45.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/325433832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c141593c0 0x7f4c140cde20 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4c040098f0 tx=0x7f4c04009c00 comp rx=0 tx=0).stop 2023-03-03T15:47:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/325433832 shutdown_connections 2023-03-03T15:47:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/325433832 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c14153290 0x7f4c14158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/325433832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c141593c0 0x7f4c140cde20 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/325433832 >> 172.21.15.102:0/325433832 conn(0x7f4c14065fb0 msgr2=0x7f4c140663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/325433832 shutdown_connections 2023-03-03T15:47:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.030+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/325433832 wait complete. 2023-03-03T15:47:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.032+0000 7f4c1c1ff700 1 Processor -- start 2023-03-03T15:47:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.033+0000 7f4c1c1ff700 1 -- start start 2023-03-03T15:47:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.033+0000 7f4c1c1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c140cd8e0 0x7f4c140c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.033+0000 7f4c1c1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c14153290 0x7f4c141f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1c1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c141593c0 0x7f4c141ec860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1c1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c141f2b20 con 0x7f4c14153290 2023-03-03T15:47:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1c1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c141f0a50 con 0x7f4c140cd8e0 2023-03-03T15:47:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1c1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c141f0d30 con 0x7f4c141593c0 2023-03-03T15:47:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c14153290 0x7f4c141f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c14153290 0x7f4c141f1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47018/0 (socket says 172.21.15.102:47018) 2023-03-03T15:47:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 -- 172.21.15.102:0/3706382815 learned_addr learned my addr 172.21.15.102:0/3706382815 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:47:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c19f9b700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c140cd8e0 0x7f4c140c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1a79c700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c141593c0 0x7f4c141ec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 -- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c141593c0 msgr2=0x7f4c141ec860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c141593c0 0x7f4c141ec860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 -- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c140cd8e0 msgr2=0x7f4c140c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c140cd8e0 0x7f4c140c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.034+0000 7f4c1979a700 1 -- 172.21.15.102:0/3706382815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04009550 con 0x7f4c14153290 2023-03-03T15:47:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.035+0000 7f4c1979a700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c14153290 0x7f4c141f1810 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009c40 tx=0x7f4c04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.035+0000 7f4c0a7fc700 1 -- 172.21.15.102:0/3706382815 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040045d0 con 0x7f4c14153290 2023-03-03T15:47:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.035+0000 7f4c0a7fc700 1 -- 172.21.15.102:0/3706382815 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0401f070 con 0x7f4c14153290 2023-03-03T15:47:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.035+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c140ccde0 con 0x7f4c14153290 2023-03-03T15:47:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.035+0000 7f4c0a7fc700 1 -- 172.21.15.102:0/3706382815 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040038a0 con 0x7f4c14153290 2023-03-03T15:47:45.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.035+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1415d160 con 0x7f4c14153290 2023-03-03T15:47:45.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.036+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c140cd0c0 con 0x7f4c14153290 2023-03-03T15:47:45.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.037+0000 7f4c0a7fc700 1 -- 172.21.15.102:0/3706382815 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c0400f050 con 0x7f4c14153290 2023-03-03T15:47:45.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.039+0000 7f4c0a7fc700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bf80682f0 0x7f4bf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:45.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.039+0000 7f4c0a7fc700 1 -- 172.21.15.102:0/3706382815 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4c04021070 con 0x7f4c14153290 2023-03-03T15:47:45.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.039+0000 7f4c19f9b700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bf80682f0 0x7f4bf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:45.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.040+0000 7f4c19f9b700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bf80682f0 0x7f4bf806a7a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009b40 tx=0x7f4c10009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:45.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.043+0000 7f4c0a7fc700 1 -- 172.21.15.102:0/3706382815 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c04053de0 con 0x7f4c14153290 2023-03-03T15:47:45.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.279+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c1404e6e0 con 0x7f4c14153290 2023-03-03T15:47:45.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.279+0000 7f4c0a7fc700 1 -- 172.21.15.102:0/3706382815 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4c0402b090 con 0x7f4c14153290 2023-03-03T15:47:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:47:45.282 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:47:45.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bf80682f0 msgr2=0x7f4bf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:45.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bf80682f0 0x7f4bf806a7a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009b40 tx=0x7f4c10009210 comp rx=0 tx=0).stop 2023-03-03T15:47:45.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c14153290 msgr2=0x7f4c141f1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:45.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c14153290 0x7f4c141f1810 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009c40 tx=0x7f4c04004750 comp rx=0 tx=0).stop 2023-03-03T15:47:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 shutdown_connections 2023-03-03T15:47:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4bf80682f0 0x7f4bf806a7a0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4c141593c0 0x7f4c141ec860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c140cd8e0 0x7f4c140c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 --2- 172.21.15.102:0/3706382815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c14153290 0x7f4c141f1810 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:45.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 >> 172.21.15.102:0/3706382815 conn(0x7f4c14065fb0 msgr2=0x7f4c140cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:45.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 shutdown_connections 2023-03-03T15:47:45.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:47:45.282+0000 7f4c1c1ff700 1 -- 172.21.15.102:0/3706382815 wait complete. 2023-03-03T15:47:45.286 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:47:45.311 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T15:48:07.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:07.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:38.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:38.024 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:08.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:08.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:38.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:38.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:40.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:49:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.567+0000 7fddcdbff700 1 Processor -- start 2023-03-03T15:49:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.568+0000 7fddcdbff700 1 -- start start 2023-03-03T15:49:40.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddcdbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc8153290 0x7fddc8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:40.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddcdbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fddc80cd8e0 0x7fddc80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:40.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddcdbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc8153660 con 0x7fddc81593c0 2023-03-03T15:49:40.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddcdbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc806c360 con 0x7fddc80cd8e0 2023-03-03T15:49:40.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddcdbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc815cea0 con 0x7fddc8153290 2023-03-03T15:49:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddc6ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fddc81593c0 0x7fddc80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55782/0 (socket says 172.21.15.102:55782) 2023-03-03T15:49:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddc6ffd700 1 -- 172.21.15.102:0/1915658275 learned_addr learned my addr 172.21.15.102:0/1915658275 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:49:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.569+0000 7fddc7fff700 1 --2- 172.21.15.102:0/1915658275 >> v2:172.21.15.172:3300/0 conn(0x7fddc80cd8e0 0x7fddc80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc77fe700 1 --2- 172.21.15.102:0/1915658275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc8153290 0x7fddc8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc67fc700 1 -- 172.21.15.102:0/1915658275 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063478205 0 0) 0x7fddc8153660 con 0x7fddc81593c0 2023-03-03T15:49:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc67fc700 1 -- 172.21.15.102:0/1915658275 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb00035b0 con 0x7fddc81593c0 2023-03-03T15:49:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc77fe700 1 -- 172.21.15.102:0/1915658275 >> v2:172.21.15.172:3300/0 conn(0x7fddc80cd8e0 msgr2=0x7fddc80ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:49:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc77fe700 1 --2- 172.21.15.102:0/1915658275 >> v2:172.21.15.172:3300/0 conn(0x7fddc80cd8e0 0x7fddc80ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc77fe700 1 -- 172.21.15.102:0/1915658275 >> v1:172.21.15.102:6789/0 conn(0x7fddc81593c0 legacy=0x7fddc80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc77fe700 1 -- 172.21.15.102:0/1915658275 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc806c7f0 con 0x7fddc8153290 2023-03-03T15:49:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc7fff700 1 --2- 172.21.15.102:0/1915658275 >> v2:172.21.15.172:3300/0 conn(0x7fddc80cd8e0 0x7fddc80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:49:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.570+0000 7fddc77fe700 1 --2- 172.21.15.102:0/1915658275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc8153290 0x7fddc8158e80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009840 tx=0x7fddb8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3b3692f80b31fdf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddc67fc700 1 -- 172.21.15.102:0/1915658275 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8011830 con 0x7fddc8153290 2023-03-03T15:49:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddc67fc700 1 -- 172.21.15.102:0/1915658275 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb8011990 con 0x7fddc8153290 2023-03-03T15:49:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddc67fc700 1 -- 172.21.15.102:0/1915658275 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8011d20 con 0x7fddc8153290 2023-03-03T15:49:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddcdbff700 1 -- 172.21.15.102:0/1915658275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc8153290 msgr2=0x7fddc8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddcdbff700 1 --2- 172.21.15.102:0/1915658275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc8153290 0x7fddc8158e80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009840 tx=0x7fddb8009b50 comp rx=0 tx=0).stop 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddcdbff700 1 -- 172.21.15.102:0/1915658275 shutdown_connections 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddcdbff700 1 --2- 172.21.15.102:0/1915658275 >> v2:172.21.15.172:3300/0 conn(0x7fddc80cd8e0 0x7fddc80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddcdbff700 1 --2- 172.21.15.102:0/1915658275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc8153290 0x7fddc8158e80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.571+0000 7fddcdbff700 1 -- 172.21.15.102:0/1915658275 >> 172.21.15.102:0/1915658275 conn(0x7fddc8065fb0 msgr2=0x7fddc80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.572+0000 7fddcdbff700 1 -- 172.21.15.102:0/1915658275 shutdown_connections 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.572+0000 7fddcdbff700 1 -- 172.21.15.102:0/1915658275 wait complete. 2023-03-03T15:49:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.574+0000 7fddcdbff700 1 Processor -- start 2023-03-03T15:49:40.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.574+0000 7fddcdbff700 1 -- start start 2023-03-03T15:49:40.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddcdbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc80cd8e0 0x7fddc814e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:40.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddcdbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddc8153290 0x7fddc81ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:40.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddcdbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc81593c0 0x7fddc81eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddcdbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80c6b50 con 0x7fddc8153290 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddcdbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc814cf60 con 0x7fddc80cd8e0 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddcdbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80c6580 con 0x7fddc81593c0 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddc8153290 0x7fddc81ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddc77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc80cd8e0 0x7fddc814e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddc6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddc8153290 0x7fddc81ee730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45560/0 (socket says 172.21.15.102:45560) 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddc6ffd700 1 -- 172.21.15.102:0/2736081818 learned_addr learned my addr 172.21.15.102:0/2736081818 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:49:40.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.575+0000 7fddc7fff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc81593c0 0x7fddc81eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddc6ffd700 1 -- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc81593c0 msgr2=0x7fddc81eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddc6ffd700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc81593c0 0x7fddc81eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddc6ffd700 1 -- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc80cd8e0 msgr2=0x7fddc814e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddc6ffd700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc80cd8e0 0x7fddc814e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddc6ffd700 1 -- 172.21.15.102:0/2736081818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0009a70 con 0x7fddc8153290 2023-03-03T15:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddc6ffd700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddc8153290 0x7fddc81ee730 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00c700 tx=0x7fddbc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddaffff700 1 -- 172.21.15.102:0/2736081818 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc005d70 con 0x7fddc8153290 2023-03-03T15:49:40.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddaffff700 1 -- 172.21.15.102:0/2736081818 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc0070d0 con 0x7fddc8153290 2023-03-03T15:49:40.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddaffff700 1 -- 172.21.15.102:0/2736081818 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc007460 con 0x7fddc8153290 2023-03-03T15:49:40.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.576+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb80094a0 con 0x7fddc8153290 2023-03-03T15:49:40.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.577+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc80ccde0 con 0x7fddc8153290 2023-03-03T15:49:40.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.578+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc815d160 con 0x7fddc8153290 2023-03-03T15:49:40.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.579+0000 7fddaffff700 1 -- 172.21.15.102:0/2736081818 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fddbc019030 con 0x7fddc8153290 2023-03-03T15:49:40.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.580+0000 7fddaffff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddb40682f0 0x7fddb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:40.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.580+0000 7fddaffff700 1 -- 172.21.15.102:0/2736081818 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fddbc0850d0 con 0x7fddc8153290 2023-03-03T15:49:40.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.580+0000 7fddc77fe700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddb40682f0 0x7fddb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:40.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.581+0000 7fddc77fe700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddb40682f0 0x7fddb406a7a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fddb8000c00 tx=0x7fddb8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:40.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.584+0000 7fddaffff700 1 -- 172.21.15.102:0/2736081818 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fddbc0517e0 con 0x7fddc8153290 2023-03-03T15:49:40.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.823+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddc80cd0c0 con 0x7fddc8153290 2023-03-03T15:49:40.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.823+0000 7fddaffff700 1 -- 172.21.15.102:0/2736081818 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fddbc054e00 con 0x7fddc8153290 2023-03-03T15:49:40.824 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:49:40.825 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:49:40.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddb40682f0 msgr2=0x7fddb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:40.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddb40682f0 0x7fddb406a7a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fddb8000c00 tx=0x7fddb8008030 comp rx=0 tx=0).stop 2023-03-03T15:49:40.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddc8153290 msgr2=0x7fddc81ee730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:40.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddc8153290 0x7fddc81ee730 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00c700 tx=0x7fddbc00ca10 comp rx=0 tx=0).stop 2023-03-03T15:49:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 shutdown_connections 2023-03-03T15:49:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fddb40682f0 0x7fddb406a7a0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fddc81593c0 0x7fddc81eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc80cd8e0 0x7fddc814e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 --2- 172.21.15.102:0/2736081818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddc8153290 0x7fddc81ee730 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 >> 172.21.15.102:0/2736081818 conn(0x7fddc8065fb0 msgr2=0x7fddc80cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:49:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 shutdown_connections 2023-03-03T15:49:40.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:49:40.826+0000 7fddcdbff700 1 -- 172.21.15.102:0/2736081818 wait complete. 2023-03-03T15:49:40.829 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:49:40.855 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T15:50:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T15:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:50:08.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:08.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:38.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:38.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:59.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:51:00.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.105+0000 7f0286fff700 1 Processor -- start 2023-03-03T15:51:00.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.106+0000 7f0286fff700 1 -- start start 2023-03-03T15:51:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f0286fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0280153290 0x7f0280158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f0286fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f02801593c0 0x7f02800cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f0286fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0280153660 con 0x7f02800cd8e0 2023-03-03T15:51:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f0286fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028006c360 con 0x7f02801593c0 2023-03-03T15:51:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f0286fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028015cea0 con 0x7f0280153290 2023-03-03T15:51:00.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f0284d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0280153290 0x7f0280158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:00.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f027ffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f02801593c0 0x7f02800cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f028559c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f02800cd8e0 0x7f02800ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:46710/0 (socket says 172.21.15.102:46710) 2023-03-03T15:51:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.107+0000 7f028559c700 1 -- 172.21.15.102:0/2106668422 learned_addr learned my addr 172.21.15.102:0/2106668422 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:51:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.108+0000 7f027f7fe700 1 -- 172.21.15.102:0/2106668422 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480712624 0 0) 0x7f0280153660 con 0x7f02800cd8e0 2023-03-03T15:51:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.108+0000 7f0284d9b700 1 -- 172.21.15.102:0/2106668422 >> v2:172.21.15.172:3300/0 conn(0x7f02801593c0 msgr2=0x7f02800cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:51:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.108+0000 7f0284d9b700 1 --2- 172.21.15.102:0/2106668422 >> v2:172.21.15.172:3300/0 conn(0x7f02801593c0 0x7f02800cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.108+0000 7f0284d9b700 1 -- 172.21.15.102:0/2106668422 >> v1:172.21.15.172:6789/0 conn(0x7f02800cd8e0 legacy=0x7f02800ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:00.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.108+0000 7f0284d9b700 1 -- 172.21.15.102:0/2106668422 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028006c7f0 con 0x7f0280153290 2023-03-03T15:51:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.108+0000 7f0284d9b700 1 --2- 172.21.15.102:0/2106668422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0280153290 0x7f0280158e80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0274009840 tx=0x7f0274009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8567e6243acf46ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:51:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f027f7fe700 1 -- 172.21.15.102:0/2106668422 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274011830 con 0x7f0280153290 2023-03-03T15:51:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f027f7fe700 1 -- 172.21.15.102:0/2106668422 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0274011990 con 0x7f0280153290 2023-03-03T15:51:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f027f7fe700 1 -- 172.21.15.102:0/2106668422 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274011d20 con 0x7f0280153290 2023-03-03T15:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 -- 172.21.15.102:0/2106668422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0280153290 msgr2=0x7f0280158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 --2- 172.21.15.102:0/2106668422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0280153290 0x7f0280158e80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0274009840 tx=0x7f0274009b50 comp rx=0 tx=0).stop 2023-03-03T15:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 -- 172.21.15.102:0/2106668422 shutdown_connections 2023-03-03T15:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 --2- 172.21.15.102:0/2106668422 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0280153290 0x7f0280158e80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 --2- 172.21.15.102:0/2106668422 >> v2:172.21.15.172:3300/0 conn(0x7f02801593c0 0x7f02800cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 -- 172.21.15.102:0/2106668422 >> 172.21.15.102:0/2106668422 conn(0x7f0280065fb0 msgr2=0x7f02800663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:51:00.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 -- 172.21.15.102:0/2106668422 shutdown_connections 2023-03-03T15:51:00.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.109+0000 7f0286fff700 1 -- 172.21.15.102:0/2106668422 wait complete. 2023-03-03T15:51:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.111+0000 7f0286fff700 1 Processor -- start 2023-03-03T15:51:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.112+0000 7f0286fff700 1 -- start start 2023-03-03T15:51:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0286fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02800cd8e0 0x7f02800b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0286fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0280153290 0x7f02800c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:00.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0286fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02801593c0 0x7f02801e2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0286fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02800b2c90 con 0x7f02801593c0 2023-03-03T15:51:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0286fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02801e4920 con 0x7f0280153290 2023-03-03T15:51:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0286fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02801e4ea0 con 0x7f02800cd8e0 2023-03-03T15:51:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f027ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0280153290 0x7f02800c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0284d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02800cd8e0 0x7f02800b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f027ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0280153290 0x7f02800c6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59094/0 (socket says 172.21.15.102:59094) 2023-03-03T15:51:00.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0284d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02800cd8e0 0x7f02800b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55918/0 (socket says 172.21.15.102:55918) 2023-03-03T15:51:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f027ffff700 1 -- 172.21.15.102:0/921984301 learned_addr learned my addr 172.21.15.102:0/921984301 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:51:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f028559c700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02801593c0 0x7f02801e2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0284d9b700 1 -- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0280153290 msgr2=0x7f02800c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0284d9b700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0280153290 0x7f02800c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0284d9b700 1 -- 172.21.15.102:0/921984301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02801593c0 msgr2=0x7f02801e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0284d9b700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02801593c0 0x7f02801e2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f0284d9b700 1 -- 172.21.15.102:0/921984301 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02740094a0 con 0x7f02800cd8e0 2023-03-03T15:51:00.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.113+0000 7f027ffff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0280153290 0x7f02800c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:51:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.114+0000 7f0284d9b700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02800cd8e0 0x7f02800b1ee0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f028006d0e0 tx=0x7f0274004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:51:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.114+0000 7f027d7fa700 1 -- 172.21.15.102:0/921984301 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02740036a0 con 0x7f02800cd8e0 2023-03-03T15:51:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.114+0000 7f027d7fa700 1 -- 172.21.15.102:0/921984301 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027401e070 con 0x7f02800cd8e0 2023-03-03T15:51:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.114+0000 7f027d7fa700 1 -- 172.21.15.102:0/921984301 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274003a60 con 0x7f02800cd8e0 2023-03-03T15:51:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.114+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02800ccde0 con 0x7f02800cd8e0 2023-03-03T15:51:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.114+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028015d160 con 0x7f02800cd8e0 2023-03-03T15:51:00.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.115+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02800cd0c0 con 0x7f02800cd8e0 2023-03-03T15:51:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.116+0000 7f027d7fa700 1 -- 172.21.15.102:0/921984301 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f027400f050 con 0x7f02800cd8e0 2023-03-03T15:51:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.118+0000 7f027d7fa700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02640682f0 0x7f026406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.118+0000 7f027d7fa700 1 -- 172.21.15.102:0/921984301 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0274086db0 con 0x7f02800cd8e0 2023-03-03T15:51:00.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.118+0000 7f027ffff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02640682f0 0x7f026406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:00.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.119+0000 7f027ffff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02640682f0 0x7f026406a7a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f02801e48d0 tx=0x7f026c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:51:00.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.122+0000 7f027d7fa700 1 -- 172.21.15.102:0/921984301 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f027408b070 con 0x7f02800cd8e0 2023-03-03T15:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.354+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f028004e6e0 con 0x7f02800cd8e0 2023-03-03T15:51:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.356+0000 7f027d7fa700 1 -- 172.21.15.102:0/921984301 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f02740534c0 con 0x7f02800cd8e0 2023-03-03T15:51:00.357 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:51:00.358 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:51:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02640682f0 msgr2=0x7f026406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02640682f0 0x7f026406a7a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f02801e48d0 tx=0x7f026c009210 comp rx=0 tx=0).stop 2023-03-03T15:51:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02800cd8e0 msgr2=0x7f02800b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:00.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02800cd8e0 0x7f02800b1ee0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f028006d0e0 tx=0x7f0274004750 comp rx=0 tx=0).stop 2023-03-03T15:51:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 shutdown_connections 2023-03-03T15:51:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f02640682f0 0x7f026406a7a0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f02800cd8e0 0x7f02800b1ee0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02801593c0 0x7f02801e2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 --2- 172.21.15.102:0/921984301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0280153290 0x7f02800c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 >> 172.21.15.102:0/921984301 conn(0x7f0280065fb0 msgr2=0x7f02800cf640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:51:00.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 shutdown_connections 2023-03-03T15:51:00.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:51:00.358+0000 7f0286fff700 1 -- 172.21.15.102:0/921984301 wait complete. 2023-03-03T15:51:00.362 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:51:00.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T15:51:08.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:08.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:39.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:39.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:09.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:09.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:37.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:52:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.644+0000 7f9ad359e700 1 Processor -- start 2023-03-03T15:52:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.644+0000 7f9ad359e700 1 -- start start 2023-03-03T15:52:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ad40ced00 0x7f9ad40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ad40b3970 0x7f9ad40c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad4158b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad359e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad40c4930 con 0x7f9ad40b3970 2023-03-03T15:52:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad40c3d20 con 0x7f9ad40ced00 2023-03-03T15:52:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40cf570 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad1d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ad40b3970 0x7f9ad40c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad4158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ad40ced00 0x7f9ad40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad1d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ad40b3970 0x7f9ad40c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55554/0 (socket says 172.21.15.102:55554) 2023-03-03T15:52:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.646+0000 7f9ad1d9b700 1 -- 172.21.15.102:0/1116981409 learned_addr learned my addr 172.21.15.102:0/1116981409 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:52:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad1d9b700 1 -- 172.21.15.102:0/1116981409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 msgr2=0x7f9ad4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad1d9b700 1 --2- 172.21.15.102:0/1116981409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad4158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad1d9b700 1 -- 172.21.15.102:0/1116981409 >> v2:172.21.15.172:3300/0 conn(0x7f9ad40ced00 msgr2=0x7f9ad40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad1d9b700 1 --2- 172.21.15.102:0/1116981409 >> v2:172.21.15.172:3300/0 conn(0x7f9ad40ced00 0x7f9ad40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad1d9b700 1 -- 172.21.15.102:0/1116981409 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad415cba0 con 0x7f9ad40b3970 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad1d9b700 1 --2- 172.21.15.102:0/1116981409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c41b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40b1b40 tx=0x7f9ac8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e6a9d32797be90 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad159a700 1 -- 172.21.15.102:0/1116981409 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8011830 con 0x7f9ad40b3970 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.647+0000 7f9ad159a700 1 -- 172.21.15.102:0/1116981409 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac8011990 con 0x7f9ad40b3970 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad159a700 1 -- 172.21.15.102:0/1116981409 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8011d20 con 0x7f9ad40b3970 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 -- 172.21.15.102:0/1116981409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40b3970 msgr2=0x7f9ad40c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 --2- 172.21.15.102:0/1116981409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c41b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40b1b40 tx=0x7f9ac8009bd0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 -- 172.21.15.102:0/1116981409 shutdown_connections 2023-03-03T15:52:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 --2- 172.21.15.102:0/1116981409 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad4158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 --2- 172.21.15.102:0/1116981409 >> v2:172.21.15.172:3300/0 conn(0x7f9ad40ced00 0x7f9ad40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 --2- 172.21.15.102:0/1116981409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c41b0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 -- 172.21.15.102:0/1116981409 >> 172.21.15.102:0/1116981409 conn(0x7f9ad408b1c0 msgr2=0x7f9ad4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 -- 172.21.15.102:0/1116981409 shutdown_connections 2023-03-03T15:52:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.648+0000 7f9ad359e700 1 -- 172.21.15.102:0/1116981409 wait complete. 2023-03-03T15:52:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.650+0000 7f9ad359e700 1 Processor -- start 2023-03-03T15:52:37.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.651+0000 7f9ad359e700 1 -- start start 2023-03-03T15:52:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40ced00 0x7f9ad41f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad414dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad406b8e0 con 0x7f9ad40ced00 2023-03-03T15:52:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40c6a80 con 0x7f9ad40b3970 2023-03-03T15:52:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad41f1aa0 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:58710/0 (socket says 172.21.15.102:58710) 2023-03-03T15:52:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad2d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad414dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad259c700 1 -- 172.21.15.102:0/2106499049 learned_addr learned my addr 172.21.15.102:0/2106499049 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:52:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad1d9b700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40ced00 0x7f9ad41f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad2d9d700 1 -- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ad40b3970 msgr2=0x7f9ad40c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad2d9d700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad2d9d700 1 -- 172.21.15.102:0/2106499049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40ced00 msgr2=0x7f9ad41f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad2d9d700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40ced00 0x7f9ad41f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad2d9d700 1 -- 172.21.15.102:0/2106499049 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac8009550 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.652+0000 7f9ad259c700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.653+0000 7f9ad2d9d700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad414dbb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00efc0 tx=0x7f9acc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.653+0000 7f9ac2ffd700 1 -- 172.21.15.102:0/2106499049 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00cc90 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.653+0000 7f9ac2ffd700 1 -- 172.21.15.102:0/2106499049 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc00f040 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.653+0000 7f9ac2ffd700 1 -- 172.21.15.102:0/2106499049 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc0146c0 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.653+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad406bf20 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.654+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad415ce80 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.655+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad406c200 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.656+0000 7f9ac2ffd700 1 -- 172.21.15.102:0/2106499049 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9acc024070 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.657+0000 7f9ac2ffd700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ab4068320 0x7f9ab406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.657+0000 7f9ac2ffd700 1 -- 172.21.15.102:0/2106499049 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9acc0868b0 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.657+0000 7f9ad259c700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ab4068320 0x7f9ab406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:37.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.658+0000 7f9ad259c700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ab4068320 0x7f9ab406a7d0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4011310 tx=0x7f9ac4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.661+0000 7f9ac2ffd700 1 -- 172.21.15.102:0/2106499049 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9acc052fc0 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.896+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ad404e6e0 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.898+0000 7f9ac2ffd700 1 -- 172.21.15.102:0/2106499049 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9acc009e60 con 0x7f9ad40cf0d0 2023-03-03T15:52:37.899 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:52:37.899 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:52:37.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ab4068320 msgr2=0x7f9ab406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:37.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ab4068320 0x7f9ab406a7d0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4011310 tx=0x7f9ac4009210 comp rx=0 tx=0).stop 2023-03-03T15:52:37.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 msgr2=0x7f9ad414dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:37.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad414dbb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00efc0 tx=0x7f9acc00c3a0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 shutdown_connections 2023-03-03T15:52:37.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9ab4068320 0x7f9ab406a7d0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9ad40cf0d0 0x7f9ad414dbb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ad40b3970 0x7f9ad40c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 --2- 172.21.15.102:0/2106499049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad40ced00 0x7f9ad41f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:37.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 >> 172.21.15.102:0/2106499049 conn(0x7f9ad408b1c0 msgr2=0x7f9ad4065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:37.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 shutdown_connections 2023-03-03T15:52:37.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:52:37.900+0000 7f9ad359e700 1 -- 172.21.15.102:0/2106499049 wait complete. 2023-03-03T15:52:37.904 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:52:37.928 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T15:52:39.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:39.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:09.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:09.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:39.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:39.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:47.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:53:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.183+0000 7f964a59d700 1 Processor -- start 2023-03-03T15:53:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.183+0000 7f964a59d700 1 -- start start 2023-03-03T15:53:48.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f964a59d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f96440ced00 0x7f96440c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:48.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f964a59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440c41b0 0x7f96440cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:48.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f964a59d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96440c3d20 con 0x7f96440b3970 2023-03-03T15:53:48.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f964a59d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96440c4930 con 0x7f96440ced00 2023-03-03T15:53:48.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f964a59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96440cf610 con 0x7f96440c41b0 2023-03-03T15:53:48.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f964959b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f96440ced00 0x7f96440c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:48.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f9648d9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f96440b3970 0x7f9644158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:44886/0 (socket says 172.21.15.102:44886) 2023-03-03T15:53:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f9649d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440c41b0 0x7f96440cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.185+0000 7f9648d9a700 1 -- 172.21.15.102:0/2884260208 learned_addr learned my addr 172.21.15.102:0/2884260208 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:53:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f963bfff700 1 -- 172.21.15.102:0/2884260208 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624568706 0 0) 0x7f96440c3d20 con 0x7f96440b3970 2023-03-03T15:53:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f963bfff700 1 -- 172.21.15.102:0/2884260208 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96300035b0 con 0x7f96440b3970 2023-03-03T15:53:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f964959b700 1 -- 172.21.15.102:0/2884260208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440c41b0 msgr2=0x7f96440cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:53:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f964959b700 1 --2- 172.21.15.102:0/2884260208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440c41b0 0x7f96440cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f964959b700 1 -- 172.21.15.102:0/2884260208 >> v1:172.21.15.172:6789/0 conn(0x7f96440b3970 legacy=0x7f9644158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:48.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f964959b700 1 -- 172.21.15.102:0/2884260208 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964415cba0 con 0x7f96440ced00 2023-03-03T15:53:48.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f9649d9c700 1 --2- 172.21.15.102:0/2884260208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440c41b0 0x7f96440cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:53:48.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.186+0000 7f964959b700 1 --2- 172.21.15.102:0/2884260208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440ced00 0x7f96440c5a90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f9640009870 tx=0x7f9640009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99351693f3e8ab7a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:48.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f963bfff700 1 -- 172.21.15.102:0/2884260208 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640011830 con 0x7f96440ced00 2023-03-03T15:53:48.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f963bfff700 1 -- 172.21.15.102:0/2884260208 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9640011990 con 0x7f96440ced00 2023-03-03T15:53:48.189 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f963bfff700 1 -- 172.21.15.102:0/2884260208 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640011d20 con 0x7f96440ced00 2023-03-03T15:53:48.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 -- 172.21.15.102:0/2884260208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440ced00 msgr2=0x7f96440c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:48.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 --2- 172.21.15.102:0/2884260208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440ced00 0x7f96440c5a90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f9640009870 tx=0x7f9640009b80 comp rx=0 tx=0).stop 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 -- 172.21.15.102:0/2884260208 shutdown_connections 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 --2- 172.21.15.102:0/2884260208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440c41b0 0x7f96440cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 --2- 172.21.15.102:0/2884260208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440ced00 0x7f96440c5a90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 -- 172.21.15.102:0/2884260208 >> 172.21.15.102:0/2884260208 conn(0x7f964408b1c0 msgr2=0x7f9644071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 -- 172.21.15.102:0/2884260208 shutdown_connections 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.187+0000 7f964a59d700 1 -- 172.21.15.102:0/2884260208 wait complete. 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.189+0000 7f964a59d700 1 Processor -- start 2023-03-03T15:53:48.191 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.190+0000 7f964a59d700 1 -- start start 2023-03-03T15:53:48.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.190+0000 7f964a59d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96440b3970 0x7f96441f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:48.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.190+0000 7f964a59d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440c41b0 0x7f96441e4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:48.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f964a59d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440ced00 0x7f96441e5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:48.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f964a59d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96441f2810 con 0x7f96440c41b0 2023-03-03T15:53:48.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f964a59d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96440cfbb0 con 0x7f96440b3970 2023-03-03T15:53:48.193 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f964a59d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9644151e80 con 0x7f96440ced00 2023-03-03T15:53:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440c41b0 0x7f96441e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440c41b0 0x7f96441e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54534/0 (socket says 172.21.15.102:54534) 2023-03-03T15:53:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 -- 172.21.15.102:0/5368751 learned_addr learned my addr 172.21.15.102:0/5368751 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:53:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9649d9c700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440ced00 0x7f96441e5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f964959b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96440b3970 0x7f96441f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 -- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440ced00 msgr2=0x7f96441e5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:48.194 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440ced00 0x7f96441e5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 -- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96440b3970 msgr2=0x7f96441f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96440b3970 0x7f96441f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.191+0000 7f9648d9a700 1 -- 172.21.15.102:0/5368751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c009a70 con 0x7f96440c41b0 2023-03-03T15:53:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.192+0000 7f9648d9a700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440c41b0 0x7f96441e4510 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f9634005910 tx=0x7f963400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.192+0000 7f9639ffb700 1 -- 172.21.15.102:0/5368751 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634005d70 con 0x7f96440c41b0 2023-03-03T15:53:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.192+0000 7f9639ffb700 1 -- 172.21.15.102:0/5368751 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963400ce70 con 0x7f96440c41b0 2023-03-03T15:53:48.195 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.192+0000 7f9639ffb700 1 -- 172.21.15.102:0/5368751 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96340072d0 con 0x7f96440c41b0 2023-03-03T15:53:48.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.192+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96400094a0 con 0x7f96440c41b0 2023-03-03T15:53:48.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.192+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964406bf20 con 0x7f96440c41b0 2023-03-03T15:53:48.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.193+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964415ce80 con 0x7f96440c41b0 2023-03-03T15:53:48.196 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.194+0000 7f9639ffb700 1 -- 172.21.15.102:0/5368751 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9634019030 con 0x7f96440c41b0 2023-03-03T15:53:48.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.196+0000 7f9639ffb700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96280682f0 0x7f962806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:48.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.196+0000 7f9639ffb700 1 -- 172.21.15.102:0/5368751 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9634085060 con 0x7f96440c41b0 2023-03-03T15:53:48.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.197+0000 7f964959b700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96280682f0 0x7f962806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:48.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.197+0000 7f964959b700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96280682f0 0x7f962806a7a0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f96400090b0 tx=0x7f9640008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:48.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.201+0000 7f9639ffb700 1 -- 172.21.15.102:0/5368751 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9634051770 con 0x7f96440c41b0 2023-03-03T15:53:48.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.433+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f964406c200 con 0x7f96440c41b0 2023-03-03T15:53:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.434+0000 7f9639ffb700 1 -- 172.21.15.102:0/5368751 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9634017090 con 0x7f96440c41b0 2023-03-03T15:53:48.435 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:53:48.435 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:53:48.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96280682f0 msgr2=0x7f962806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:48.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96280682f0 0x7f962806a7a0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f96400090b0 tx=0x7f9640008030 comp rx=0 tx=0).stop 2023-03-03T15:53:48.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440c41b0 msgr2=0x7f96441e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:48.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440c41b0 0x7f96441e4510 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f9634005910 tx=0x7f963400c7b0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 shutdown_connections 2023-03-03T15:53:48.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f96280682f0 0x7f962806a7a0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f96440ced00 0x7f96441e5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96440b3970 0x7f96441f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 --2- 172.21.15.102:0/5368751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96440c41b0 0x7f96441e4510 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 >> 172.21.15.102:0/5368751 conn(0x7f964408b1c0 msgr2=0x7f96440ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:53:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 shutdown_connections 2023-03-03T15:53:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:53:48.437+0000 7f964a59d700 1 -- 172.21.15.102:0/5368751 wait complete. 2023-03-03T15:53:48.440 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:53:48.465 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T15:54:10.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:10.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:40.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:40.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:07.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:55:07.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.722+0000 7f076afff700 1 Processor -- start 2023-03-03T15:55:07.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.722+0000 7f076afff700 1 -- start start 2023-03-03T15:55:07.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.723+0000 7f076afff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0764153290 0x7f0764158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:07.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.723+0000 7f076afff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07640cd8e0 0x7f07640ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:07.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076afff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0764153660 con 0x7f07641593c0 2023-03-03T15:55:07.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076afff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076406c360 con 0x7f07640cd8e0 2023-03-03T15:55:07.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076afff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076415cea0 con 0x7f0764153290 2023-03-03T15:55:07.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076959c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07640cd8e0 0x7f07640ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:07.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076959c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07640cd8e0 0x7f07640ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37156/0 (socket says 172.21.15.102:37156) 2023-03-03T15:55:07.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076959c700 1 -- 172.21.15.102:0/2493492920 learned_addr learned my addr 172.21.15.102:0/2493492920 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:55:07.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f0768d9b700 1 --2- 172.21.15.102:0/2493492920 >> v2:172.21.15.172:3300/0 conn(0x7f0764153290 0x7f0764158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:07.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076959c700 1 -- 172.21.15.102:0/2493492920 >> v2:172.21.15.172:3300/0 conn(0x7f0764153290 msgr2=0x7f0764158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:55:07.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076959c700 1 --2- 172.21.15.102:0/2493492920 >> v2:172.21.15.172:3300/0 conn(0x7f0764153290 0x7f0764158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076959c700 1 -- 172.21.15.102:0/2493492920 >> v1:172.21.15.172:6789/0 conn(0x7f07641593c0 legacy=0x7f07640cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:07.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.724+0000 7f076959c700 1 -- 172.21.15.102:0/2493492920 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076406c7f0 con 0x7f07640cd8e0 2023-03-03T15:55:07.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.725+0000 7f076959c700 1 --2- 172.21.15.102:0/2493492920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07640ce530 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f076408cc70 tx=0x7f075c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67dd2263ae868e0b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:07.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.725+0000 7f07637fe700 1 -- 172.21.15.102:0/2493492920 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00e070 con 0x7f07640cd8e0 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.725+0000 7f07637fe700 1 -- 172.21.15.102:0/2493492920 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c00b910 con 0x7f07640cd8e0 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.725+0000 7f07637fe700 1 -- 172.21.15.102:0/2493492920 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00bcc0 con 0x7f07640cd8e0 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 -- 172.21.15.102:0/2493492920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 msgr2=0x7f07640ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 --2- 172.21.15.102:0/2493492920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07640ce530 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f076408cc70 tx=0x7f075c00b330 comp rx=0 tx=0).stop 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 -- 172.21.15.102:0/2493492920 shutdown_connections 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 --2- 172.21.15.102:0/2493492920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07640cd8e0 0x7f07640ce530 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 --2- 172.21.15.102:0/2493492920 >> v2:172.21.15.172:3300/0 conn(0x7f0764153290 0x7f0764158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 -- 172.21.15.102:0/2493492920 >> 172.21.15.102:0/2493492920 conn(0x7f0764065fb0 msgr2=0x7f07640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:07.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 -- 172.21.15.102:0/2493492920 shutdown_connections 2023-03-03T15:55:07.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.726+0000 7f076afff700 1 -- 172.21.15.102:0/2493492920 wait complete. 2023-03-03T15:55:07.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.728+0000 7f076afff700 1 Processor -- start 2023-03-03T15:55:07.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.728+0000 7f076afff700 1 -- start start 2023-03-03T15:55:07.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.729+0000 7f076afff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd8e0 0x7f07640c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:07.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.729+0000 7f076afff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0764153290 0x7f07641501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:07.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.729+0000 7f076afff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07641593c0 0x7f07641f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:07.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.729+0000 7f076afff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641567f0 con 0x7f0764153290 2023-03-03T15:55:07.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.729+0000 7f076afff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07640cf920 con 0x7f07641593c0 2023-03-03T15:55:07.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.729+0000 7f076afff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07640cf3e0 con 0x7f07640cd8e0 2023-03-03T15:55:07.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.729+0000 7f0763fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0764153290 0x7f07641501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:07.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0764153290 0x7f07641501d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37164/0 (socket says 172.21.15.102:37164) 2023-03-03T15:55:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 -- 172.21.15.102:0/3643041878 learned_addr learned my addr 172.21.15.102:0/3643041878 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:55:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0768d9b700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd8e0 0x7f07640c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f076959c700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07641593c0 0x7f07641f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 -- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd8e0 msgr2=0x7f07640c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd8e0 0x7f07640c87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 -- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07641593c0 msgr2=0x7f07641f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07641593c0 0x7f07641f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 -- 172.21.15.102:0/3643041878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c009b30 con 0x7f0764153290 2023-03-03T15:55:07.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f076959c700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07641593c0 0x7f07641f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:55:07.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.730+0000 7f0763fff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0764153290 0x7f07641501d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f075000c4e0 tx=0x7f075000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:07.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.731+0000 7f07617fa700 1 -- 172.21.15.102:0/3643041878 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07500052d0 con 0x7f0764153290 2023-03-03T15:55:07.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.731+0000 7f07617fa700 1 -- 172.21.15.102:0/3643041878 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0750005430 con 0x7f0764153290 2023-03-03T15:55:07.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.731+0000 7f07617fa700 1 -- 172.21.15.102:0/3643041878 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07500038c0 con 0x7f0764153290 2023-03-03T15:55:07.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.731+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07640ccde0 con 0x7f0764153290 2023-03-03T15:55:07.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.731+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076415d160 con 0x7f0764153290 2023-03-03T15:55:07.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.732+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07640cd0c0 con 0x7f0764153290 2023-03-03T15:55:07.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.733+0000 7f07617fa700 1 -- 172.21.15.102:0/3643041878 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0750019030 con 0x7f0764153290 2023-03-03T15:55:07.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.734+0000 7f07617fa700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0748068320 0x7f074806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:07.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.735+0000 7f07617fa700 1 -- 172.21.15.102:0/3643041878 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0750084fe0 con 0x7f0764153290 2023-03-03T15:55:07.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.735+0000 7f0768d9b700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0748068320 0x7f074806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:07.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.736+0000 7f0768d9b700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0748068320 0x7f074806a7d0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0758009be0 tx=0x7f0758009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:07.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.739+0000 7f07617fa700 1 -- 172.21.15.102:0/3643041878 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0750051750 con 0x7f0764153290 2023-03-03T15:55:07.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.971+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f076404e6e0 con 0x7f0764153290 2023-03-03T15:55:07.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.972+0000 7f07617fa700 1 -- 172.21.15.102:0/3643041878 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f0750017090 con 0x7f0764153290 2023-03-03T15:55:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:55:07.973 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:55:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.974+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0748068320 msgr2=0x7f074806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.974+0000 7f076afff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0748068320 0x7f074806a7d0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0758009be0 tx=0x7f0758009210 comp rx=0 tx=0).stop 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0764153290 msgr2=0x7f07641501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0764153290 0x7f07641501d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f075000c4e0 tx=0x7f075000c7f0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 shutdown_connections 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f0748068320 0x7f074806a7d0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07640cd8e0 0x7f07640c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07641593c0 0x7f07641f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 --2- 172.21.15.102:0/3643041878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0764153290 0x7f07641501d0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 >> 172.21.15.102:0/3643041878 conn(0x7f0764065fb0 msgr2=0x7f076415ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 shutdown_connections 2023-03-03T15:55:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:55:07.975+0000 7f076afff700 1 -- 172.21.15.102:0/3643041878 wait complete. 2023-03-03T15:55:07.977 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:55:08.002 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-03-03T15:55:10.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:10.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:40.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:40.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:10.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:10.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:23.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:56:23.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.251+0000 7fc9f73ff700 1 Processor -- start 2023-03-03T15:56:23.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.252+0000 7fc9f73ff700 1 -- start start 2023-03-03T15:56:23.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.253+0000 7fc9f73ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9f00c9cb0 0x7fc9f00cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:23.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.253+0000 7fc9f73ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9f00ca190 0x7fc9f00cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:23.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.253+0000 7fc9f73ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f00cf940 con 0x7fc9f0153260 2023-03-03T15:56:23.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.253+0000 7fc9f73ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f00cfe70 con 0x7fc9f00ca190 2023-03-03T15:56:23.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.253+0000 7fc9f73ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f0158d70 con 0x7fc9f00c9cb0 2023-03-03T15:56:23.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.253+0000 7fc9f599c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9f00ca190 0x7fc9f00cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:23.256 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.253+0000 7fc9f499a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9f00c9cb0 0x7fc9f00cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:23.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f519b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc9f0153260 0x7fc9f00ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40188/0 (socket says 172.21.15.102:40188) 2023-03-03T15:56:23.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f499a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9f00c9cb0 0x7fc9f00cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47148/0 (socket says 172.21.15.102:47148) 2023-03-03T15:56:23.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f519b700 1 -- 172.21.15.102:0/3854841940 learned_addr learned my addr 172.21.15.102:0/3854841940 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:56:23.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f499a700 1 -- 172.21.15.102:0/3854841940 >> v2:172.21.15.102:3300/0 conn(0x7fc9f00ca190 msgr2=0x7fc9f00cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:56:23.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f499a700 1 --2- 172.21.15.102:0/3854841940 >> v2:172.21.15.102:3300/0 conn(0x7fc9f00ca190 0x7fc9f00cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f499a700 1 -- 172.21.15.102:0/3854841940 >> v1:172.21.15.172:6789/0 conn(0x7fc9f0153260 legacy=0x7fc9f00ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:23.257 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f499a700 1 -- 172.21.15.102:0/3854841940 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f00cfb90 con 0x7fc9f00c9cb0 2023-03-03T15:56:23.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f599c700 1 --2- 172.21.15.102:0/3854841940 >> v2:172.21.15.102:3300/0 conn(0x7fc9f00ca190 0x7fc9f00cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:56:23.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9e7fff700 1 -- 172.21.15.102:0/3854841940 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18542388 0 0) 0x7fc9f00cf940 con 0x7fc9f0153260 2023-03-03T15:56:23.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.254+0000 7fc9f499a700 1 --2- 172.21.15.102:0/3854841940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f00c9cb0 0x7fc9f00cb880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc9f008cd80 tx=0x7fc9e0009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a8c8b6dd7fa5727 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:56:23.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9e7fff700 1 -- 172.21.15.102:0/3854841940 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0011830 con 0x7fc9f00c9cb0 2023-03-03T15:56:23.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9e7fff700 1 -- 172.21.15.102:0/3854841940 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e0011990 con 0x7fc9f00c9cb0 2023-03-03T15:56:23.258 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9e7fff700 1 -- 172.21.15.102:0/3854841940 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0011d20 con 0x7fc9f00c9cb0 2023-03-03T15:56:23.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 -- 172.21.15.102:0/3854841940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f00c9cb0 msgr2=0x7fc9f00cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:23.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/3854841940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f00c9cb0 0x7fc9f00cb880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc9f008cd80 tx=0x7fc9e0009b20 comp rx=0 tx=0).stop 2023-03-03T15:56:23.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 -- 172.21.15.102:0/3854841940 shutdown_connections 2023-03-03T15:56:23.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/3854841940 >> v2:172.21.15.102:3300/0 conn(0x7fc9f00ca190 0x7fc9f00cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.259 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/3854841940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f00c9cb0 0x7fc9f00cb880 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 -- 172.21.15.102:0/3854841940 >> 172.21.15.102:0/3854841940 conn(0x7fc9f0066c00 msgr2=0x7fc9f00ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:56:23.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 -- 172.21.15.102:0/3854841940 shutdown_connections 2023-03-03T15:56:23.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.255+0000 7fc9f73ff700 1 -- 172.21.15.102:0/3854841940 wait complete. 2023-03-03T15:56:23.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.257+0000 7fc9f73ff700 1 Processor -- start 2023-03-03T15:56:23.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.258+0000 7fc9f73ff700 1 -- start start 2023-03-03T15:56:23.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f73ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9f00ca190 0x7fc9f014e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f73ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f0153260 0x7fc9f01e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f73ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9f00c5920 0x7fc9f01f12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f73ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f0154e10 con 0x7fc9f00c5920 2023-03-03T15:56:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f73ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f006c740 con 0x7fc9f0153260 2023-03-03T15:56:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f73ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f0155230 con 0x7fc9f00ca190 2023-03-03T15:56:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f599c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9f00c5920 0x7fc9f01f12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f519b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9f00ca190 0x7fc9f014e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f499a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f0153260 0x7fc9f01e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f599c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9f00c5920 0x7fc9f01f12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51200/0 (socket says 172.21.15.102:51200) 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f599c700 1 -- 172.21.15.102:0/1183252781 learned_addr learned my addr 172.21.15.102:0/1183252781 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f599c700 1 -- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9f00ca190 msgr2=0x7fc9f014e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f599c700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9f00ca190 0x7fc9f014e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.259+0000 7fc9f599c700 1 -- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f0153260 msgr2=0x7fc9f01e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9f599c700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f0153260 0x7fc9f01e20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9f599c700 1 -- 172.21.15.102:0/1183252781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8009a70 con 0x7fc9f00c5920 2023-03-03T15:56:23.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9f499a700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f0153260 0x7fc9f01e20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:56:23.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9f599c700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9f00c5920 0x7fc9f01f12f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800efc0 tx=0x7fc9e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:56:23.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9e5ffb700 1 -- 172.21.15.102:0/1183252781 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800ca90 con 0x7fc9f00c5920 2023-03-03T15:56:23.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9e5ffb700 1 -- 172.21.15.102:0/1183252781 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e800f040 con 0x7fc9f00c5920 2023-03-03T15:56:23.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e00094a0 con 0x7fc9f00c5920 2023-03-03T15:56:23.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9e5ffb700 1 -- 172.21.15.102:0/1183252781 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800ce10 con 0x7fc9f00c5920 2023-03-03T15:56:23.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.260+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f0159260 con 0x7fc9f00c5920 2023-03-03T15:56:23.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.261+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f0158f80 con 0x7fc9f00c5920 2023-03-03T15:56:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.263+0000 7fc9e5ffb700 1 -- 172.21.15.102:0/1183252781 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc9e8014480 con 0x7fc9f00c5920 2023-03-03T15:56:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.264+0000 7fc9e5ffb700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9d4068320 0x7fc9d406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.264+0000 7fc9e5ffb700 1 -- 172.21.15.102:0/1183252781 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc9e8086570 con 0x7fc9f00c5920 2023-03-03T15:56:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.264+0000 7fc9f519b700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9d4068320 0x7fc9d406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.265+0000 7fc9f519b700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9d4068320 0x7fc9d406a7d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec007fd0 tx=0x7fc9ec007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:56:23.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.268+0000 7fc9e5ffb700 1 -- 172.21.15.102:0/1183252781 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc9e8052b90 con 0x7fc9f00c5920 2023-03-03T15:56:23.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.499+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9f0159540 con 0x7fc9f00c5920 2023-03-03T15:56:23.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.500+0000 7fc9e5ffb700 1 -- 172.21.15.102:0/1183252781 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc9e8009d70 con 0x7fc9f00c5920 2023-03-03T15:56:23.502 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:56:23.502 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:56:23.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9d4068320 msgr2=0x7fc9d406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:23.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9d4068320 0x7fc9d406a7d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec007fd0 tx=0x7fc9ec007de0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9f00c5920 msgr2=0x7fc9f01f12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:23.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9f00c5920 0x7fc9f01f12f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800efc0 tx=0x7fc9e800c3a0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 shutdown_connections 2023-03-03T15:56:23.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9d4068320 0x7fc9d406a7d0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9f00ca190 0x7fc9f014e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9f00c5920 0x7fc9f01f12f0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 --2- 172.21.15.102:0/1183252781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9f0153260 0x7fc9f01e20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:23.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 >> 172.21.15.102:0/1183252781 conn(0x7fc9f00c9cb0 msgr2=0x7fc9f0156be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:56:23.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 shutdown_connections 2023-03-03T15:56:23.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:56:23.503+0000 7fc9f73ff700 1 -- 172.21.15.102:0/1183252781 wait complete. 2023-03-03T15:56:23.506 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:56:23.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-03-03T15:56:40.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:40.996 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:11.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:11.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:41.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:41.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:07.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:58:07.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.781+0000 7fe7c1d9d700 1 Processor -- start 2023-03-03T15:58:07.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.781+0000 7fe7c1d9d700 1 -- start start 2023-03-03T15:58:07.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.783+0000 7fe7c1d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc0cab50 0x7fe7bc1646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:07.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.783+0000 7fe7c1d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc0c9ca0 con 0x7fe7bc1564b0 2023-03-03T15:58:07.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.783+0000 7fe7c1d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc0b46f0 con 0x7fe7bc0c9610 2023-03-03T15:58:07.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.783+0000 7fe7c1d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0b4920 con 0x7fe7bc0cab50 2023-03-03T15:58:07.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.783+0000 7fe7c0d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe7bc1564b0 0x7fe7bc0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49252/0 (socket says 172.21.15.102:49252) 2023-03-03T15:58:07.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.783+0000 7fe7c0d9b700 1 -- 172.21.15.102:0/2142848304 learned_addr learned my addr 172.21.15.102:0/2142848304 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:58:07.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.783+0000 7fe7c159c700 1 --2- 172.21.15.102:0/2142848304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc0cab50 0x7fe7bc1646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:07.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7bb7fe700 1 -- 172.21.15.102:0/2142848304 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134909167 0 0) 0x7fe7bc0c9ca0 con 0x7fe7bc1564b0 2023-03-03T15:58:07.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7bb7fe700 1 -- 172.21.15.102:0/2142848304 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ac0035b0 con 0x7fe7bc1564b0 2023-03-03T15:58:07.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7bb7fe700 1 -- 172.21.15.102:0/2142848304 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642473614 0 0) 0x7fe7bc0b46f0 con 0x7fe7bc0c9610 2023-03-03T15:58:07.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7bb7fe700 1 -- 172.21.15.102:0/2142848304 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc0c9ca0 con 0x7fe7bc0c9610 2023-03-03T15:58:07.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7c159c700 1 -- 172.21.15.102:0/2142848304 >> v1:172.21.15.172:6789/0 conn(0x7fe7bc0c9610 legacy=0x7fe7bc0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:07.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7c159c700 1 -- 172.21.15.102:0/2142848304 >> v1:172.21.15.102:6789/0 conn(0x7fe7bc1564b0 legacy=0x7fe7bc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:07.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7c159c700 1 -- 172.21.15.102:0/2142848304 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc0b4c00 con 0x7fe7bc0cab50 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7c159c700 1 --2- 172.21.15.102:0/2142848304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc0cab50 0x7fe7bc1646b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009dc0 tx=0x7fe7b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1ce6b2fe68803fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.784+0000 7fe7bb7fe700 1 -- 172.21.15.102:0/2142848304 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b400e070 con 0x7fe7bc0cab50 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7bb7fe700 1 -- 172.21.15.102:0/2142848304 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b400bb20 con 0x7fe7bc0cab50 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7bb7fe700 1 -- 172.21.15.102:0/2142848304 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4012430 con 0x7fe7bc0cab50 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/2142848304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc0cab50 msgr2=0x7fe7bc1646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/2142848304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc0cab50 0x7fe7bc1646b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009dc0 tx=0x7fe7b400b330 comp rx=0 tx=0).stop 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/2142848304 shutdown_connections 2023-03-03T15:58:07.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/2142848304 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc0cab50 0x7fe7bc1646b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:07.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/2142848304 >> 172.21.15.102:0/2142848304 conn(0x7fe7bc065fb0 msgr2=0x7fe7bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:07.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/2142848304 shutdown_connections 2023-03-03T15:58:07.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.785+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/2142848304 wait complete. 2023-03-03T15:58:07.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.787+0000 7fe7c1d9d700 1 Processor -- start 2023-03-03T15:58:07.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.787+0000 7fe7c1d9d700 1 -- start start 2023-03-03T15:58:07.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.788+0000 7fe7c1d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7bc0c9610 0x7fe7bc158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:07.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.788+0000 7fe7c1d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7bc0cab50 0x7fe7bc0c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.788+0000 7fe7c1d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc1564b0 0x7fe7bc0c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c1d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0c3920 con 0x7fe7bc0cab50 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c1d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0c35a0 con 0x7fe7bc0c9610 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c1d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0c2bd0 con 0x7fe7bc1564b0 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7bc0c9610 0x7fe7bc158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7bbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7bc0cab50 0x7fe7bc0c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7bc0c9610 0x7fe7bc158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:60186/0 (socket says 172.21.15.102:60186) 2023-03-03T15:58:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 -- 172.21.15.102:0/870171674 learned_addr learned my addr 172.21.15.102:0/870171674 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:58:07.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c159c700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc1564b0 0x7fe7bc0c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:07.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 -- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc1564b0 msgr2=0x7fe7bc0c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:07.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc1564b0 0x7fe7bc0c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:07.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 -- 172.21.15.102:0/870171674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7bc0cab50 msgr2=0x7fe7bc0c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:07.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7bc0cab50 0x7fe7bc0c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:07.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.789+0000 7fe7c0d9b700 1 -- 172.21.15.102:0/870171674 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4009a70 con 0x7fe7bc0c9610 2023-03-03T15:58:07.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.790+0000 7fe7c0d9b700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7bc0c9610 0x7fe7bc158c30 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000c730 tx=0x7fe7b000ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:07.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.790+0000 7fe7b97fa700 1 -- 172.21.15.102:0/870171674 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0005d70 con 0x7fe7bc0c9610 2023-03-03T15:58:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.790+0000 7fe7b97fa700 1 -- 172.21.15.102:0/870171674 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b00070d0 con 0x7fe7bc0c9610 2023-03-03T15:58:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.790+0000 7fe7b97fa700 1 -- 172.21.15.102:0/870171674 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0007440 con 0x7fe7bc0c9610 2023-03-03T15:58:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.790+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc0b2210 con 0x7fe7bc0c9610 2023-03-03T15:58:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.790+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc0b1f30 con 0x7fe7bc0c9610 2023-03-03T15:58:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.791+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc156880 con 0x7fe7bc0c9610 2023-03-03T15:58:07.794 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.792+0000 7fe7b97fa700 1 -- 172.21.15.102:0/870171674 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe7b0019030 con 0x7fe7bc0c9610 2023-03-03T15:58:07.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.794+0000 7fe7b97fa700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7a0068320 0x7fe7a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:07.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.794+0000 7fe7b97fa700 1 -- 172.21.15.102:0/870171674 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe7b0084f30 con 0x7fe7bc0c9610 2023-03-03T15:58:07.795 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.794+0000 7fe7bbfff700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7a0068320 0x7fe7a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:07.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.794+0000 7fe7bbfff700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7a0068320 0x7fe7a006a7d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0c7cc0 tx=0x7fe7a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:07.799 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:07.797+0000 7fe7b97fa700 1 -- 172.21.15.102:0/870171674 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7b004b5e0 con 0x7fe7bc0c9610 2023-03-03T15:58:08.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.033+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7bc0c2eb0 con 0x7fe7bc0c9610 2023-03-03T15:58:08.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.034+0000 7fe7b97fa700 1 -- 172.21.15.102:0/870171674 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe7b0017090 con 0x7fe7bc0c9610 2023-03-03T15:58:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:58:08.036 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:58:08.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7a0068320 msgr2=0x7fe7a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7a0068320 0x7fe7a006a7d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0c7cc0 tx=0x7fe7a8009040 comp rx=0 tx=0).stop 2023-03-03T15:58:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7bc0c9610 msgr2=0x7fe7bc158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7bc0c9610 0x7fe7bc158c30 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000c730 tx=0x7fe7b000ca40 comp rx=0 tx=0).stop 2023-03-03T15:58:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 shutdown_connections 2023-03-03T15:58:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe7a0068320 0x7fe7a006a7d0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe7bc1564b0 0x7fe7bc0c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe7bc0c9610 0x7fe7bc158c30 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:08.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 --2- 172.21.15.102:0/870171674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7bc0cab50 0x7fe7bc0c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:08.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 >> 172.21.15.102:0/870171674 conn(0x7fe7bc065fb0 msgr2=0x7fe7bc155670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:08.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 shutdown_connections 2023-03-03T15:58:08.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:08.037+0000 7fe7c1d9d700 1 -- 172.21.15.102:0/870171674 wait complete. 2023-03-03T15:58:08.040 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:58:08.065 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 24 secs before thrashing 2023-03-03T15:58:11.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:11.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:32.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:58:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.330+0000 7f1d6999d700 1 Processor -- start 2023-03-03T15:58:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.330+0000 7f1d6999d700 1 -- start start 2023-03-03T15:58:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d6999d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d640c33d0 0x7f1d6406fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d6999d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1d640c37a0 0x7f1d641634d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d6999d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6406b410 con 0x7f1d6406cfe0 2023-03-03T15:58:32.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d6999d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6406d430 con 0x7f1d640c33d0 2023-03-03T15:58:32.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d6999d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d640c8140 con 0x7f1d640c37a0 2023-03-03T15:58:32.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d6899b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d640c33d0 0x7f1d6406fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d63fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1d6406cfe0 0x7f1d6415c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42688/0 (socket says 172.21.15.102:42688) 2023-03-03T15:58:32.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d63fff700 1 -- 172.21.15.102:0/2494414851 learned_addr learned my addr 172.21.15.102:0/2494414851 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T15:58:32.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.332+0000 7f1d6919c700 1 --2- 172.21.15.102:0/2494414851 >> v2:172.21.15.172:3300/0 conn(0x7f1d640c37a0 0x7f1d641634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.333+0000 7f1d6919c700 1 -- 172.21.15.102:0/2494414851 >> v2:172.21.15.102:3300/0 conn(0x7f1d640c33d0 msgr2=0x7f1d6406fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:32.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.333+0000 7f1d6919c700 1 --2- 172.21.15.102:0/2494414851 >> v2:172.21.15.102:3300/0 conn(0x7f1d640c33d0 0x7f1d6406fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.333+0000 7f1d6919c700 1 -- 172.21.15.102:0/2494414851 >> v1:172.21.15.102:6789/0 conn(0x7f1d6406cfe0 legacy=0x7f1d6415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.333+0000 7f1d6919c700 1 -- 172.21.15.102:0/2494414851 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6406efb0 con 0x7f1d640c37a0 2023-03-03T15:58:32.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.333+0000 7f1d6919c700 1 --2- 172.21.15.102:0/2494414851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 0x7f1d641634d0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1d6406cc70 tx=0x7f1d5c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=673e55f2be01a1be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:32.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.333+0000 7f1d637fe700 1 -- 172.21.15.102:0/2494414851 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c00e070 con 0x7f1d640c37a0 2023-03-03T15:58:32.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d637fe700 1 -- 172.21.15.102:0/2494414851 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c00bb20 con 0x7f1d640c37a0 2023-03-03T15:58:32.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d637fe700 1 -- 172.21.15.102:0/2494414851 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c012430 con 0x7f1d640c37a0 2023-03-03T15:58:32.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 -- 172.21.15.102:0/2494414851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 msgr2=0x7f1d641634d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2494414851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 0x7f1d641634d0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1d6406cc70 tx=0x7f1d5c00b330 comp rx=0 tx=0).stop 2023-03-03T15:58:32.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 -- 172.21.15.102:0/2494414851 shutdown_connections 2023-03-03T15:58:32.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2494414851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 0x7f1d641634d0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2494414851 >> v2:172.21.15.102:3300/0 conn(0x7f1d640c33d0 0x7f1d6406fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 -- 172.21.15.102:0/2494414851 >> 172.21.15.102:0/2494414851 conn(0x7f1d64065fb0 msgr2=0x7f1d64066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:32.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 -- 172.21.15.102:0/2494414851 shutdown_connections 2023-03-03T15:58:32.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.334+0000 7f1d6999d700 1 -- 172.21.15.102:0/2494414851 wait complete. 2023-03-03T15:58:32.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.336+0000 7f1d6999d700 1 Processor -- start 2023-03-03T15:58:32.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.336+0000 7f1d6999d700 1 -- start start 2023-03-03T15:58:32.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.337+0000 7f1d6999d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d6406cfe0 0x7f1d641f1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.337+0000 7f1d6999d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d640c33d0 0x7f1d641f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.337+0000 7f1d6999d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 0x7f1d641f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.337+0000 7f1d6999d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d641f0a40 con 0x7f1d640c33d0 2023-03-03T15:58:32.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d6999d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d640af540 con 0x7f1d640c37a0 2023-03-03T15:58:32.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d6999d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64151170 con 0x7f1d6406cfe0 2023-03-03T15:58:32.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d640c33d0 0x7f1d641f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d6899b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d6406cfe0 0x7f1d641f1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d640c33d0 0x7f1d641f23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35944/0 (socket says 172.21.15.102:35944) 2023-03-03T15:58:32.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 -- 172.21.15.102:0/2181193311 learned_addr learned my addr 172.21.15.102:0/2181193311 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:58:32.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d6919c700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 0x7f1d641f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 -- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d6406cfe0 msgr2=0x7f1d641f1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d6406cfe0 0x7f1d641f1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 -- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 msgr2=0x7f1d641f0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 0x7f1d641f0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d63fff700 1 -- 172.21.15.102:0/2181193311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c009a70 con 0x7f1d640c33d0 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.338+0000 7f1d6899b700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d6406cfe0 0x7f1d641f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.339+0000 7f1d63fff700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d640c33d0 0x7f1d641f23d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000c4b0 tx=0x7f1d5000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.339+0000 7f1d617fa700 1 -- 172.21.15.102:0/2181193311 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d50005d70 con 0x7f1d640c33d0 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.339+0000 7f1d617fa700 1 -- 172.21.15.102:0/2181193311 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5000ce80 con 0x7f1d640c33d0 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.339+0000 7f1d617fa700 1 -- 172.21.15.102:0/2181193311 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d500072d0 con 0x7f1d640c33d0 2023-03-03T15:58:32.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.339+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6406b660 con 0x7f1d640c33d0 2023-03-03T15:58:32.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.339+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6406f290 con 0x7f1d640c33d0 2023-03-03T15:58:32.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.340+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d640c7c20 con 0x7f1d640c33d0 2023-03-03T15:58:32.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.341+0000 7f1d617fa700 1 -- 172.21.15.102:0/2181193311 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1d50019030 con 0x7f1d640c33d0 2023-03-03T15:58:32.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.342+0000 7f1d617fa700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d48068320 0x7f1d4806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.343+0000 7f1d617fa700 1 -- 172.21.15.102:0/2181193311 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1d50084e70 con 0x7f1d640c33d0 2023-03-03T15:58:32.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.343+0000 7f1d6899b700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d48068320 0x7f1d4806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.343+0000 7f1d6899b700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d48068320 0x7f1d4806a7d0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d58009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:32.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.347+0000 7f1d617fa700 1 -- 172.21.15.102:0/2181193311 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1d5004b5e0 con 0x7f1d640c33d0 2023-03-03T15:58:32.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.577+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d641f0b80 con 0x7f1d640c33d0 2023-03-03T15:58:32.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.578+0000 7f1d617fa700 1 -- 172.21.15.102:0/2181193311 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1d641f0b80 con 0x7f1d640c33d0 2023-03-03T15:58:32.578 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:58:32.579 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:58:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d48068320 msgr2=0x7f1d4806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d48068320 0x7f1d4806a7d0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d58009210 comp rx=0 tx=0).stop 2023-03-03T15:58:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d640c33d0 msgr2=0x7f1d641f23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d640c33d0 0x7f1d641f23d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000c4b0 tx=0x7f1d5000c7c0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 shutdown_connections 2023-03-03T15:58:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1d48068320 0x7f1d4806a7d0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1d6406cfe0 0x7f1d641f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d640c37a0 0x7f1d641f0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 --2- 172.21.15.102:0/2181193311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d640c33d0 0x7f1d641f23d0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 >> 172.21.15.102:0/2181193311 conn(0x7f1d64065fb0 msgr2=0x7f1d640afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 shutdown_connections 2023-03-03T15:58:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:58:32.581+0000 7f1d6999d700 1 -- 172.21.15.102:0/2181193311 wait complete. 2023-03-03T15:58:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:58:32.609 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-03-03T15:58:41.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:41.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:12.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:12.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:42.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:42.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:57.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:59:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.865+0000 7fcee69ff700 1 Processor -- start 2023-03-03T15:59:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.865+0000 7fcee69ff700 1 -- start start 2023-03-03T15:59:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcee69ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcee00cd8e0 0x7fcee00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcee69ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:57.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcee69ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee0153660 con 0x7fcee0153290 2023-03-03T15:59:57.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcee69ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee006c360 con 0x7fcee00cd8e0 2023-03-03T15:59:57.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcee69ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee015cea0 con 0x7fcee00cd390 2023-03-03T15:59:57.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcedf7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcee00cd8e0 0x7fcee00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:57.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcedf7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcee00cd8e0 0x7fcee00cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57598/0 (socket says 172.21.15.102:57598) 2023-03-03T15:59:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcedf7fe700 1 -- 172.21.15.102:0/3587853777 learned_addr learned my addr 172.21.15.102:0/3587853777 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:59:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcee4f9c700 1 --2- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcedf7fe700 1 -- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 msgr2=0x7fcee00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:59:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcedf7fe700 1 --2- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.867+0000 7fcedf7fe700 1 -- 172.21.15.102:0/3587853777 >> v1:172.21.15.102:6789/0 conn(0x7fcee0153290 legacy=0x7fcee0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.868+0000 7fcedf7fe700 1 -- 172.21.15.102:0/3587853777 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee006c7f0 con 0x7fcee00cd8e0 2023-03-03T15:59:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.868+0000 7fcedeffd700 1 -- 172.21.15.102:0/3587853777 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270580763 0 0) 0x7fcee0153660 con 0x7fcee0153290 2023-03-03T15:59:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.868+0000 7fcedf7fe700 1 --2- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 0x7fcee00cde20 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fced40098f0 tx=0x7fced4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=621d53565e20c5be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:59:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.868+0000 7fcedeffd700 1 -- 172.21.15.102:0/3587853777 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4011830 con 0x7fcee00cd8e0 2023-03-03T15:59:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.868+0000 7fcedeffd700 1 -- 172.21.15.102:0/3587853777 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced4011990 con 0x7fcee00cd8e0 2023-03-03T15:59:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.868+0000 7fcedeffd700 1 -- 172.21.15.102:0/3587853777 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4011d20 con 0x7fcee00cd8e0 2023-03-03T15:59:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.868+0000 7fcee69ff700 1 -- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 msgr2=0x7fcee00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.869+0000 7fcee69ff700 1 --2- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 0x7fcee00cde20 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fced40098f0 tx=0x7fced4009c00 comp rx=0 tx=0).stop 2023-03-03T15:59:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.869+0000 7fcee69ff700 1 -- 172.21.15.102:0/3587853777 shutdown_connections 2023-03-03T15:59:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.869+0000 7fcee69ff700 1 --2- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.869+0000 7fcee69ff700 1 --2- 172.21.15.102:0/3587853777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 0x7fcee00cde20 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.869+0000 7fcee69ff700 1 -- 172.21.15.102:0/3587853777 >> 172.21.15.102:0/3587853777 conn(0x7fcee0065fb0 msgr2=0x7fcee00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:59:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.869+0000 7fcee69ff700 1 -- 172.21.15.102:0/3587853777 shutdown_connections 2023-03-03T15:59:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.869+0000 7fcee69ff700 1 -- 172.21.15.102:0/3587853777 wait complete. 2023-03-03T15:59:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.871+0000 7fcee69ff700 1 Processor -- start 2023-03-03T15:59:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.871+0000 7fcee69ff700 1 -- start start 2023-03-03T15:59:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee69ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee01f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee69ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 0x7fcee0158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee69ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee0153290 0x7fcee014e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee69ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c6880 con 0x7fcee0153290 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee69ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c62c0 con 0x7fcee00cd8e0 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee69ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c65a0 con 0x7fcee00cd390 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee4f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee0153290 0x7fcee014e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee4f9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee0153290 0x7fcee014e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37548/0 (socket says 172.21.15.102:37548) 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcee4f9c700 1 -- 172.21.15.102:0/1770969769 learned_addr learned my addr 172.21.15.102:0/1770969769 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcedffff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee01f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.872+0000 7fcedf7fe700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 0x7fcee0158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee4f9c700 1 -- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 msgr2=0x7fcee01f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee4f9c700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee01f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee4f9c700 1 -- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 msgr2=0x7fcee0158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee4f9c700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 0x7fcee0158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee4f9c700 1 -- 172.21.15.102:0/1770969769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4009550 con 0x7fcee0153290 2023-03-03T15:59:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee4f9c700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee0153290 0x7fcee014e0b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fced800efc0 tx=0x7fced800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:59:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcedcff9700 1 -- 172.21.15.102:0/1770969769 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800cad0 con 0x7fcee0153290 2023-03-03T15:59:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcedcff9700 1 -- 172.21.15.102:0/1770969769 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced800f040 con 0x7fcee0153290 2023-03-03T15:59:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee00ccde0 con 0x7fcee0153290 2023-03-03T15:59:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcedcff9700 1 -- 172.21.15.102:0/1770969769 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800ce70 con 0x7fcee0153290 2023-03-03T15:59:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.873+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee015d160 con 0x7fcee0153290 2023-03-03T15:59:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.875+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee00cd0c0 con 0x7fcee0153290 2023-03-03T15:59:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.876+0000 7fcedcff9700 1 -- 172.21.15.102:0/1770969769 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fced8014480 con 0x7fcee0153290 2023-03-03T15:59:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.877+0000 7fcedcff9700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcecc0682f0 0x7fcecc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.877+0000 7fcedcff9700 1 -- 172.21.15.102:0/1770969769 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fced8086670 con 0x7fcee0153290 2023-03-03T15:59:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.878+0000 7fcedffff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcecc0682f0 0x7fcecc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:57.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.878+0000 7fcedffff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcecc0682f0 0x7fcecc06a7a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fced0007fd0 tx=0x7fced0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:59:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:57.881+0000 7fcedcff9700 1 -- 172.21.15.102:0/1770969769 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fced8052d20 con 0x7fcee0153290 2023-03-03T15:59:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.114+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcee004e6e0 con 0x7fcee0153290 2023-03-03T15:59:58.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.115+0000 7fcedcff9700 1 -- 172.21.15.102:0/1770969769 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fced8056340 con 0x7fcee0153290 2023-03-03T15:59:58.116 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T15:59:58.116 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T15:59:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcecc0682f0 msgr2=0x7fcecc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcecc0682f0 0x7fcecc06a7a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fced0007fd0 tx=0x7fced0009040 comp rx=0 tx=0).stop 2023-03-03T15:59:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee0153290 msgr2=0x7fcee014e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee0153290 0x7fcee014e0b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fced800efc0 tx=0x7fced800c0c0 comp rx=0 tx=0).stop 2023-03-03T15:59:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 shutdown_connections 2023-03-03T15:59:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcecc0682f0 0x7fcecc06a7a0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcee00cd390 0x7fcee01f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee0153290 0x7fcee014e0b0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 --2- 172.21.15.102:0/1770969769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcee00cd8e0 0x7fcee0158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.117+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 >> 172.21.15.102:0/1770969769 conn(0x7fcee0065fb0 msgr2=0x7fcee01575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:59:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.118+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 shutdown_connections 2023-03-03T15:59:58.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T15:59:58.118+0000 7fcee69ff700 1 -- 172.21.15.102:0/1770969769 wait complete. 2023-03-03T15:59:58.120 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T15:59:58.145 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T16:00:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T15:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:00:12.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:12.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:41.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:00:41.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.398+0000 7f7aadbff700 1 Processor -- start 2023-03-03T16:00:41.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.398+0000 7f7aadbff700 1 -- start start 2023-03-03T16:00:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.399+0000 7f7aadbff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7aa8153290 0x7f7aa8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aadbff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7aa80cd8e0 0x7f7aa80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aadbff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa8153660 con 0x7f7aa81593c0 2023-03-03T16:00:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aadbff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa806c360 con 0x7f7aa80cd8e0 2023-03-03T16:00:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aadbff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa815cea0 con 0x7f7aa8153290 2023-03-03T16:00:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aa7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7aa80cd8e0 0x7f7aa80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aa7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7aa80cd8e0 0x7f7aa80ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37894/0 (socket says 172.21.15.102:37894) 2023-03-03T16:00:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aa7fff700 1 -- 172.21.15.102:0/299280219 learned_addr learned my addr 172.21.15.102:0/299280219 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:00:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aa6ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7aa81593c0 0x7f7aa80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49102/0 (socket says 172.21.15.102:49102) 2023-03-03T16:00:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.400+0000 7f7aa77fe700 1 --2- 172.21.15.102:0/299280219 >> v2:172.21.15.172:3300/0 conn(0x7f7aa8153290 0x7f7aa8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa7fff700 1 -- 172.21.15.102:0/299280219 >> v2:172.21.15.172:3300/0 conn(0x7f7aa8153290 msgr2=0x7f7aa8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa7fff700 1 --2- 172.21.15.102:0/299280219 >> v2:172.21.15.172:3300/0 conn(0x7f7aa8153290 0x7f7aa8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa7fff700 1 -- 172.21.15.102:0/299280219 >> v1:172.21.15.102:6789/0 conn(0x7f7aa81593c0 legacy=0x7f7aa80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa7fff700 1 -- 172.21.15.102:0/299280219 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa806c7f0 con 0x7f7aa80cd8e0 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa67fc700 1 -- 172.21.15.102:0/299280219 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790307634 0 0) 0x7f7aa8153660 con 0x7f7aa81593c0 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa7fff700 1 --2- 172.21.15.102:0/299280219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa80ce530 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7aa808cc70 tx=0x7f7aa000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6137865f8543af server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa67fc700 1 -- 172.21.15.102:0/299280219 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000e070 con 0x7f7aa80cd8e0 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa67fc700 1 -- 172.21.15.102:0/299280219 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa000b910 con 0x7f7aa80cd8e0 2023-03-03T16:00:41.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.401+0000 7f7aa67fc700 1 -- 172.21.15.102:0/299280219 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000bcc0 con 0x7f7aa80cd8e0 2023-03-03T16:00:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 -- 172.21.15.102:0/299280219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa80cd8e0 msgr2=0x7f7aa80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 --2- 172.21.15.102:0/299280219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa80ce530 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7aa808cc70 tx=0x7f7aa000b330 comp rx=0 tx=0).stop 2023-03-03T16:00:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 -- 172.21.15.102:0/299280219 shutdown_connections 2023-03-03T16:00:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 --2- 172.21.15.102:0/299280219 >> v2:172.21.15.172:3300/0 conn(0x7f7aa8153290 0x7f7aa8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 --2- 172.21.15.102:0/299280219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa80ce530 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 -- 172.21.15.102:0/299280219 >> 172.21.15.102:0/299280219 conn(0x7f7aa8065fb0 msgr2=0x7f7aa80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:00:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 -- 172.21.15.102:0/299280219 shutdown_connections 2023-03-03T16:00:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.402+0000 7f7aadbff700 1 -- 172.21.15.102:0/299280219 wait complete. 2023-03-03T16:00:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.404+0000 7f7aadbff700 1 Processor -- start 2023-03-03T16:00:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.404+0000 7f7aadbff700 1 -- start start 2023-03-03T16:00:41.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.405+0000 7f7aadbff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa8151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:41.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.405+0000 7f7aadbff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa8153290 0x7f7aa81ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:41.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.405+0000 7f7aadbff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa81593c0 0x7f7aa814bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:41.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.405+0000 7f7aadbff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80c5c90 con 0x7f7aa81593c0 2023-03-03T16:00:41.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.405+0000 7f7aadbff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa81e5df0 con 0x7f7aa80cd8e0 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.405+0000 7f7aadbff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8155b60 con 0x7f7aa8153290 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa81593c0 0x7f7aa814bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa81593c0 0x7f7aa814bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37898/0 (socket says 172.21.15.102:37898) 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 -- 172.21.15.102:0/3753891373 learned_addr learned my addr 172.21.15.102:0/3753891373 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa77fe700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa8151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa6ffd700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa8153290 0x7f7aa81ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 -- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa8153290 msgr2=0x7f7aa81ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa8153290 0x7f7aa81ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 -- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa80cd8e0 msgr2=0x7f7aa8151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa8151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 -- 172.21.15.102:0/3753891373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa0009b30 con 0x7f7aa81593c0 2023-03-03T16:00:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa77fe700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa8151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:00:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.406+0000 7f7aa7fff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa81593c0 0x7f7aa814bf90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8074540 tx=0x7f7aa0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.407+0000 7f7a8ffff700 1 -- 172.21.15.102:0/3753891373 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0004840 con 0x7f7aa81593c0 2023-03-03T16:00:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.407+0000 7f7a8ffff700 1 -- 172.21.15.102:0/3753891373 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa00099b0 con 0x7f7aa81593c0 2023-03-03T16:00:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.407+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa80ccde0 con 0x7f7aa81593c0 2023-03-03T16:00:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.407+0000 7f7a8ffff700 1 -- 172.21.15.102:0/3753891373 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa001b670 con 0x7f7aa81593c0 2023-03-03T16:00:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.407+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa815d160 con 0x7f7aa81593c0 2023-03-03T16:00:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.408+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa80cd0c0 con 0x7f7aa81593c0 2023-03-03T16:00:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.409+0000 7f7a8ffff700 1 -- 172.21.15.102:0/3753891373 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7aa0021030 con 0x7f7aa81593c0 2023-03-03T16:00:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.410+0000 7f7a8ffff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7a940682f0 0x7f7a9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.410+0000 7f7a8ffff700 1 -- 172.21.15.102:0/3753891373 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7aa0087130 con 0x7f7aa81593c0 2023-03-03T16:00:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.411+0000 7f7aa77fe700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7a940682f0 0x7f7a9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.411+0000 7f7aa77fe700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7a940682f0 0x7f7a9406a7a0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009fd0 tx=0x7f7a98009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:41.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.415+0000 7f7a8ffff700 1 -- 172.21.15.102:0/3753891373 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7aa0053840 con 0x7f7aa81593c0 2023-03-03T16:00:41.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.651+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aa81559c0 con 0x7f7aa81593c0 2023-03-03T16:00:41.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.651+0000 7f7a8ffff700 1 -- 172.21.15.102:0/3753891373 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7aa81559c0 con 0x7f7aa81593c0 2023-03-03T16:00:41.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:00:41.652 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:00:41.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7a940682f0 msgr2=0x7f7a9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:41.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7a940682f0 0x7f7a9406a7a0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009fd0 tx=0x7f7a98009210 comp rx=0 tx=0).stop 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa81593c0 msgr2=0x7f7aa814bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa81593c0 0x7f7aa814bf90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8074540 tx=0x7f7aa0004ad0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 shutdown_connections 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aa81593c0 0x7f7aa814bf90 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7aa80cd8e0 0x7f7aa8151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7a940682f0 0x7f7a9406a7a0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 --2- 172.21.15.102:0/3753891373 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7aa8153290 0x7f7aa81ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:41.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 >> 172.21.15.102:0/3753891373 conn(0x7f7aa8065fb0 msgr2=0x7f7aa8157840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:00:41.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 shutdown_connections 2023-03-03T16:00:41.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:00:41.654+0000 7f7aadbff700 1 -- 172.21.15.102:0/3753891373 wait complete. 2023-03-03T16:00:41.656 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:00:41.681 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2023-03-03T16:00:42.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:42.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:12.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:12.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:42.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:42.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:00.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:02:00.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.930+0000 7f645b3ff700 1 Processor -- start 2023-03-03T16:02:00.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.930+0000 7f645b3ff700 1 -- start start 2023-03-03T16:02:00.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.931+0000 7f645b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:00.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.931+0000 7f645b3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64540cd8e0 0x7f64540ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.931+0000 7f645b3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6454153660 con 0x7f64541593c0 2023-03-03T16:02:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.931+0000 7f645b3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645406c360 con 0x7f64540cd8e0 2023-03-03T16:02:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.931+0000 7f645b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645415cea0 con 0x7f6454153290 2023-03-03T16:02:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645899a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f64541593c0 0x7f64540cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57678/0 (socket says 172.21.15.102:57678) 2023-03-03T16:02:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645899a700 1 -- 172.21.15.102:0/3183019786 learned_addr learned my addr 172.21.15.102:0/3183019786 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:02:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645999c700 1 --2- 172.21.15.102:0/3183019786 >> v2:172.21.15.102:3300/0 conn(0x7f64540cd8e0 0x7f64540ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645919b700 1 --2- 172.21.15.102:0/3183019786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f644bfff700 1 -- 172.21.15.102:0/3183019786 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268449439 0 0) 0x7f6454153660 con 0x7f64541593c0 2023-03-03T16:02:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f644bfff700 1 -- 172.21.15.102:0/3183019786 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64400035b0 con 0x7f64541593c0 2023-03-03T16:02:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645999c700 1 -- 172.21.15.102:0/3183019786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 msgr2=0x7f6454158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:02:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645999c700 1 --2- 172.21.15.102:0/3183019786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645999c700 1 -- 172.21.15.102:0/3183019786 >> v1:172.21.15.172:6789/0 conn(0x7f64541593c0 legacy=0x7f64540cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.932+0000 7f645999c700 1 -- 172.21.15.102:0/3183019786 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645406c7f0 con 0x7f64540cd8e0 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.933+0000 7f645999c700 1 --2- 172.21.15.102:0/3183019786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540ce530 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f644c009d90 tx=0x7f644c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518393cc64abb8d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.933+0000 7f644bfff700 1 -- 172.21.15.102:0/3183019786 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c00e070 con 0x7f64540cd8e0 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.933+0000 7f644bfff700 1 -- 172.21.15.102:0/3183019786 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c00bb20 con 0x7f64540cd8e0 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.933+0000 7f644bfff700 1 -- 172.21.15.102:0/3183019786 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c012430 con 0x7f64540cd8e0 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.933+0000 7f645b3ff700 1 -- 172.21.15.102:0/3183019786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 msgr2=0x7f64540ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.933+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3183019786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540ce530 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f644c009d90 tx=0x7f644c00b330 comp rx=0 tx=0).stop 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.934+0000 7f645b3ff700 1 -- 172.21.15.102:0/3183019786 shutdown_connections 2023-03-03T16:02:00.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.934+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3183019786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540ce530 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:00.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.934+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3183019786 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:00.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.934+0000 7f645b3ff700 1 -- 172.21.15.102:0/3183019786 >> 172.21.15.102:0/3183019786 conn(0x7f6454065fb0 msgr2=0x7f64540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:00.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.934+0000 7f645b3ff700 1 -- 172.21.15.102:0/3183019786 shutdown_connections 2023-03-03T16:02:00.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.934+0000 7f645b3ff700 1 -- 172.21.15.102:0/3183019786 wait complete. 2023-03-03T16:02:00.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.936+0000 7f645b3ff700 1 Processor -- start 2023-03-03T16:02:00.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.936+0000 7f645b3ff700 1 -- start start 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645b3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645b3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645b3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64541593c0 0x7f64540c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645b3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64540c3d20 con 0x7f64540cd8e0 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645b3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64540c8380 con 0x7f64541593c0 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645b3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64540c8660 con 0x7f6454153290 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:00.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645919b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540c7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43888/0 (socket says 172.21.15.102:43888) 2023-03-03T16:02:00.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.937+0000 7f645919b700 1 -- 172.21.15.102:0/3574397330 learned_addr learned my addr 172.21.15.102:0/3574397330 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:02:00.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645899a700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:00.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645999c700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64541593c0 0x7f64540c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:00.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645919b700 1 -- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 msgr2=0x7f6454153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:00.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645919b700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:00.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645919b700 1 -- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64541593c0 msgr2=0x7f64540c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:00.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645919b700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64541593c0 0x7f64540c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:00.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645919b700 1 -- 172.21.15.102:0/3574397330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c009a70 con 0x7f64540cd8e0 2023-03-03T16:02:00.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f645919b700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540c7cd0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f645000e9b0 tx=0x7f645000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:00.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.938+0000 7f6449ffb700 1 -- 172.21.15.102:0/3574397330 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450009710 con 0x7f64540cd8e0 2023-03-03T16:02:00.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.939+0000 7f6449ffb700 1 -- 172.21.15.102:0/3574397330 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645000ee50 con 0x7f64540cd8e0 2023-03-03T16:02:00.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.939+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64540ccde0 con 0x7f64540cd8e0 2023-03-03T16:02:00.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.939+0000 7f6449ffb700 1 -- 172.21.15.102:0/3574397330 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000ca50 con 0x7f64540cd8e0 2023-03-03T16:02:00.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.939+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645415d160 con 0x7f64540cd8e0 2023-03-03T16:02:00.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.940+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64540cd0c0 con 0x7f64540cd8e0 2023-03-03T16:02:00.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.941+0000 7f6449ffb700 1 -- 172.21.15.102:0/3574397330 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6450019030 con 0x7f64540cd8e0 2023-03-03T16:02:00.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.942+0000 7f6449ffb700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64380682f0 0x7f643806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:00.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.942+0000 7f6449ffb700 1 -- 172.21.15.102:0/3574397330 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6450085d10 con 0x7f64540cd8e0 2023-03-03T16:02:00.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.943+0000 7f645899a700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64380682f0 0x7f643806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:00.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.943+0000 7f645899a700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64380682f0 0x7f643806a7a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f64541581d0 tx=0x7f6444007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:00.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:00.946+0000 7f6449ffb700 1 -- 172.21.15.102:0/3574397330 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f645008b020 con 0x7f64540cd8e0 2023-03-03T16:02:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.179+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f645404e6e0 con 0x7f64540cd8e0 2023-03-03T16:02:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.180+0000 7f6449ffb700 1 -- 172.21.15.102:0/3574397330 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6450021090 con 0x7f64540cd8e0 2023-03-03T16:02:01.182 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:02:01.182 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:02:01.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.182+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64380682f0 msgr2=0x7f643806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:01.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.182+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64380682f0 0x7f643806a7a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f64541581d0 tx=0x7f6444007dc0 comp rx=0 tx=0).stop 2023-03-03T16:02:01.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 msgr2=0x7f64540c7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:01.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540c7cd0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f645000e9b0 tx=0x7f645000ecc0 comp rx=0 tx=0).stop 2023-03-03T16:02:01.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 shutdown_connections 2023-03-03T16:02:01.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64541593c0 0x7f64540c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:01.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540cd8e0 0x7f64540c7cd0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:01.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f64380682f0 0x7f643806a7a0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:01.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 --2- 172.21.15.102:0/3574397330 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6454153290 0x7f6454153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:01.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 >> 172.21.15.102:0/3574397330 conn(0x7f6454065fb0 msgr2=0x7f645415be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:01.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 shutdown_connections 2023-03-03T16:02:01.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:01.183+0000 7f645b3ff700 1 -- 172.21.15.102:0/3574397330 wait complete. 2023-03-03T16:02:01.186 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:02:01.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-03-03T16:02:13.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:13.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:43.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:43.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:49.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:02:49.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.479+0000 7fc95a39d700 1 Processor -- start 2023-03-03T16:02:49.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.479+0000 7fc95a39d700 1 -- start start 2023-03-03T16:02:49.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc95a39d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9540ced00 0x7fc9540c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:49.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc95a39d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9540b3970 0x7fc9540c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:49.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc95a39d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9540c3d20 con 0x7fc9540cf0d0 2023-03-03T16:02:49.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc95a39d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9540c4930 con 0x7fc9540b3970 2023-03-03T16:02:49.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc95a39d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9540cf570 con 0x7fc9540ced00 2023-03-03T16:02:49.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc958b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9540b3970 0x7fc9540c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:49.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc95939b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc9540ced00 0x7fc9540c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:49.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc958b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9540b3970 0x7fc9540c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34158/0 (socket says 172.21.15.102:34158) 2023-03-03T16:02:49.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.481+0000 7fc958b9a700 1 -- 172.21.15.102:0/3653075930 learned_addr learned my addr 172.21.15.102:0/3653075930 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:02:49.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc95939b700 1 -- 172.21.15.102:0/3653075930 >> v2:172.21.15.102:3300/0 conn(0x7fc9540b3970 msgr2=0x7fc9540c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:02:49.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc95939b700 1 --2- 172.21.15.102:0/3653075930 >> v2:172.21.15.102:3300/0 conn(0x7fc9540b3970 0x7fc9540c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc95939b700 1 -- 172.21.15.102:0/3653075930 >> v1:172.21.15.172:6789/0 conn(0x7fc9540cf0d0 legacy=0x7fc954158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:49.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc95939b700 1 -- 172.21.15.102:0/3653075930 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95415cba0 con 0x7fc9540ced00 2023-03-03T16:02:49.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc95939b700 1 --2- 172.21.15.102:0/3653075930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540ced00 0x7fc9540c5a90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc9540b1b40 tx=0x7fc950009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74b6c21d426988 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:49.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc94bfff700 1 -- 172.21.15.102:0/3653075930 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950011830 con 0x7fc9540ced00 2023-03-03T16:02:49.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc94bfff700 1 -- 172.21.15.102:0/3653075930 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc950011990 con 0x7fc9540ced00 2023-03-03T16:02:49.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.482+0000 7fc94bfff700 1 -- 172.21.15.102:0/3653075930 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950011d20 con 0x7fc9540ced00 2023-03-03T16:02:49.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 -- 172.21.15.102:0/3653075930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540ced00 msgr2=0x7fc9540c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:49.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 --2- 172.21.15.102:0/3653075930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540ced00 0x7fc9540c5a90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc9540b1b40 tx=0x7fc950009b20 comp rx=0 tx=0).stop 2023-03-03T16:02:49.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 -- 172.21.15.102:0/3653075930 shutdown_connections 2023-03-03T16:02:49.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 --2- 172.21.15.102:0/3653075930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540ced00 0x7fc9540c5a90 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 --2- 172.21.15.102:0/3653075930 >> v2:172.21.15.102:3300/0 conn(0x7fc9540b3970 0x7fc9540c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 -- 172.21.15.102:0/3653075930 >> 172.21.15.102:0/3653075930 conn(0x7fc95408b1c0 msgr2=0x7fc954071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:49.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 -- 172.21.15.102:0/3653075930 shutdown_connections 2023-03-03T16:02:49.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.483+0000 7fc95a39d700 1 -- 172.21.15.102:0/3653075930 wait complete. 2023-03-03T16:02:49.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.485+0000 7fc95a39d700 1 Processor -- start 2023-03-03T16:02:49.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.485+0000 7fc95a39d700 1 -- start start 2023-03-03T16:02:49.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95a39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540b3970 0x7fc95414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95a39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9540ced00 0x7fc9541f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95a39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9540cf0d0 0x7fc954150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95a39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc954159cc0 con 0x7fc9540cf0d0 2023-03-03T16:02:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95a39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc954159eb0 con 0x7fc9540b3970 2023-03-03T16:02:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95a39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95415a450 con 0x7fc9540ced00 2023-03-03T16:02:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95939b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540b3970 0x7fc95414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:49.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95939b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540b3970 0x7fc95414cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:41308/0 (socket says 172.21.15.102:41308) 2023-03-03T16:02:49.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc95939b700 1 -- 172.21.15.102:0/1672701019 learned_addr learned my addr 172.21.15.102:0/1672701019 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:02:49.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.486+0000 7fc958b9a700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9540ced00 0x7fc9541f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:49.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc959b9c700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9540cf0d0 0x7fc954150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:49.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc95939b700 1 -- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9540ced00 msgr2=0x7fc9541f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:49.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc95939b700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9540ced00 0x7fc9541f1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc95939b700 1 -- 172.21.15.102:0/1672701019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9540cf0d0 msgr2=0x7fc954150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:49.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc95939b700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9540cf0d0 0x7fc954150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc95939b700 1 -- 172.21.15.102:0/1672701019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500094a0 con 0x7fc9540b3970 2023-03-03T16:02:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc95939b700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540b3970 0x7fc95414cc40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc9500090b0 tx=0x7fc950004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc949ffb700 1 -- 172.21.15.102:0/1672701019 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9500036a0 con 0x7fc9540b3970 2023-03-03T16:02:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc949ffb700 1 -- 172.21.15.102:0/1672701019 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95001e070 con 0x7fc9540b3970 2023-03-03T16:02:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.487+0000 7fc949ffb700 1 -- 172.21.15.102:0/1672701019 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950003a60 con 0x7fc9540b3970 2023-03-03T16:02:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.488+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95406bf20 con 0x7fc9540b3970 2023-03-03T16:02:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.488+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95415ce80 con 0x7fc9540b3970 2023-03-03T16:02:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.489+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95406c200 con 0x7fc9540b3970 2023-03-03T16:02:49.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.490+0000 7fc949ffb700 1 -- 172.21.15.102:0/1672701019 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc95000f050 con 0x7fc9540b3970 2023-03-03T16:02:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.491+0000 7fc949ffb700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9380682f0 0x7fc93806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.491+0000 7fc949ffb700 1 -- 172.21.15.102:0/1672701019 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc9500875a0 con 0x7fc9540b3970 2023-03-03T16:02:49.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.491+0000 7fc958b9a700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9380682f0 0x7fc93806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:49.494 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.492+0000 7fc958b9a700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9380682f0 0x7fc93806a7a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc95414b940 tx=0x7fc9440092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:49.495 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.494+0000 7fc949ffb700 1 -- 172.21.15.102:0/1672701019 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc950053d30 con 0x7fc9540b3970 2023-03-03T16:02:49.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.727+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9540c55f0 con 0x7fc9540b3970 2023-03-03T16:02:49.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.727+0000 7fc949ffb700 1 -- 172.21.15.102:0/1672701019 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc950057350 con 0x7fc9540b3970 2023-03-03T16:02:49.729 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:02:49.729 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:02:49.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9380682f0 msgr2=0x7fc93806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:49.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9380682f0 0x7fc93806a7a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc95414b940 tx=0x7fc9440092c0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540b3970 msgr2=0x7fc95414cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:49.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540b3970 0x7fc95414cc40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc9500090b0 tx=0x7fc950004750 comp rx=0 tx=0).stop 2023-03-03T16:02:49.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 shutdown_connections 2023-03-03T16:02:49.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9540cf0d0 0x7fc954150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc9540b3970 0x7fc95414cc40 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc9380682f0 0x7fc93806a7a0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 --2- 172.21.15.102:0/1672701019 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc9540ced00 0x7fc9541f1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:49.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.730+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 >> 172.21.15.102:0/1672701019 conn(0x7fc95408b1c0 msgr2=0x7fc954065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:49.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.731+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 shutdown_connections 2023-03-03T16:02:49.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:02:49.731+0000 7fc95a39d700 1 -- 172.21.15.102:0/1672701019 wait complete. 2023-03-03T16:02:49.734 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:02:49.760 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 86 secs before thrashing 2023-03-03T16:03:13.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:13.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:43.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:43.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:13.973 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:13.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:15.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:04:16.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.007+0000 7f07dbdff700 1 Processor -- start 2023-03-03T16:04:16.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.008+0000 7f07dbdff700 1 -- start start 2023-03-03T16:04:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07dbdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07d40cd8e0 0x7f07d40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07dbdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd390 0x7f07d40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07dbdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d4153660 con 0x7f07d4153290 2023-03-03T16:04:16.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07dbdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d406c360 con 0x7f07d40cd8e0 2023-03-03T16:04:16.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07dbdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d415cea0 con 0x7f07d40cd390 2023-03-03T16:04:16.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07d939a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07d40cd8e0 0x7f07d40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:16.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07da39c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd390 0x7f07d40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:16.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07d9b9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f07d4153290 0x7f07d4158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47784/0 (socket says 172.21.15.102:47784) 2023-03-03T16:04:16.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.009+0000 7f07d9b9b700 1 -- 172.21.15.102:0/3102496716 learned_addr learned my addr 172.21.15.102:0/3102496716 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:04:16.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07da39c700 1 -- 172.21.15.102:0/3102496716 >> v2:172.21.15.102:3300/0 conn(0x7f07d40cd8e0 msgr2=0x7f07d40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:04:16.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07da39c700 1 --2- 172.21.15.102:0/3102496716 >> v2:172.21.15.102:3300/0 conn(0x7f07d40cd8e0 0x7f07d40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07da39c700 1 -- 172.21.15.102:0/3102496716 >> v1:172.21.15.172:6789/0 conn(0x7f07d4153290 legacy=0x7f07d4158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07d8b99700 1 -- 172.21.15.102:0/3102496716 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073116318 0 0) 0x7f07d4153660 con 0x7f07d4153290 2023-03-03T16:04:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07da39c700 1 -- 172.21.15.102:0/3102496716 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d406c7f0 con 0x7f07d40cd390 2023-03-03T16:04:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07da39c700 1 --2- 172.21.15.102:0/3102496716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd390 0x7f07d40ce360 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009d90 tx=0x7f07cc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3268748185d0a8f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07d8b99700 1 -- 172.21.15.102:0/3102496716 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc00e070 con 0x7f07d40cd390 2023-03-03T16:04:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07d8b99700 1 -- 172.21.15.102:0/3102496716 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc00bb20 con 0x7f07d40cd390 2023-03-03T16:04:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.010+0000 7f07d8b99700 1 -- 172.21.15.102:0/3102496716 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc012430 con 0x7f07d40cd390 2023-03-03T16:04:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 -- 172.21.15.102:0/3102496716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd390 msgr2=0x7f07d40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 --2- 172.21.15.102:0/3102496716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd390 0x7f07d40ce360 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009d90 tx=0x7f07cc00b330 comp rx=0 tx=0).stop 2023-03-03T16:04:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 -- 172.21.15.102:0/3102496716 shutdown_connections 2023-03-03T16:04:16.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 --2- 172.21.15.102:0/3102496716 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd390 0x7f07d40ce360 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 --2- 172.21.15.102:0/3102496716 >> v2:172.21.15.102:3300/0 conn(0x7f07d40cd8e0 0x7f07d40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 -- 172.21.15.102:0/3102496716 >> 172.21.15.102:0/3102496716 conn(0x7f07d4065fb0 msgr2=0x7f07d40663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:16.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 -- 172.21.15.102:0/3102496716 shutdown_connections 2023-03-03T16:04:16.014 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.011+0000 7f07dbdff700 1 -- 172.21.15.102:0/3102496716 wait complete. 2023-03-03T16:04:16.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.013+0000 7f07dbdff700 1 Processor -- start 2023-03-03T16:04:16.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.014+0000 7f07dbdff700 1 -- start start 2023-03-03T16:04:16.015 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07dbdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d40cd390 0x7f07d41564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07dbdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd8e0 0x7f07d40c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07dbdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d4153290 0x7f07d40c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07dbdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4156a00 con 0x7f07d40cd390 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07dbdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d40c7e10 con 0x7f07d4153290 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07dbdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d40c86f0 con 0x7f07d40cd8e0 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07d9b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d40cd390 0x7f07d41564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07d9b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d40cd390 0x7f07d41564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51040/0 (socket says 172.21.15.102:51040) 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07d9b9b700 1 -- 172.21.15.102:0/1013910332 learned_addr learned my addr 172.21.15.102:0/1013910332 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07d939a700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd8e0 0x7f07d40c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.015+0000 7f07da39c700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d4153290 0x7f07d40c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07d9b9b700 1 -- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd8e0 msgr2=0x7f07d40c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07d9b9b700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd8e0 0x7f07d40c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07d9b9b700 1 -- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d4153290 msgr2=0x7f07d40c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07d9b9b700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d4153290 0x7f07d40c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07d9b9b700 1 -- 172.21.15.102:0/1013910332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc009a70 con 0x7f07d40cd390 2023-03-03T16:04:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07d9b9b700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d40cd390 0x7f07d41564c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f07d00011e0 tx=0x7f07d000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07ca7fc700 1 -- 172.21.15.102:0/1013910332 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0004c80 con 0x7f07d40cd390 2023-03-03T16:04:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07ca7fc700 1 -- 172.21.15.102:0/1013910332 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0004e00 con 0x7f07d40cd390 2023-03-03T16:04:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d40ccde0 con 0x7f07d40cd390 2023-03-03T16:04:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.016+0000 7f07ca7fc700 1 -- 172.21.15.102:0/1013910332 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0007310 con 0x7f07d40cd390 2023-03-03T16:04:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.017+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d415d160 con 0x7f07d40cd390 2023-03-03T16:04:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.018+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d40cd0c0 con 0x7f07d40cd390 2023-03-03T16:04:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.019+0000 7f07ca7fc700 1 -- 172.21.15.102:0/1013910332 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f07d001a020 con 0x7f07d40cd390 2023-03-03T16:04:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.020+0000 7f07ca7fc700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b8068320 0x7f07b806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.020+0000 7f07ca7fc700 1 -- 172.21.15.102:0/1013910332 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f07d0085880 con 0x7f07d40cd390 2023-03-03T16:04:16.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.020+0000 7f07d939a700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b8068320 0x7f07b806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:16.021 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.021+0000 7f07d939a700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b8068320 0x7f07b806a7d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f07d40c8f60 tx=0x7f07c4009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:16.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.024+0000 7f07ca7fc700 1 -- 172.21.15.102:0/1013910332 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07d0052040 con 0x7f07d40cd390 2023-03-03T16:04:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.260+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07d40c6d80 con 0x7f07d40cd390 2023-03-03T16:04:16.261 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.260+0000 7f07ca7fc700 1 -- 172.21.15.102:0/1013910332 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f07d0055660 con 0x7f07d40cd390 2023-03-03T16:04:16.261 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:04:16.261 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:04:16.263 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b8068320 msgr2=0x7f07b806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b8068320 0x7f07b806a7d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f07d40c8f60 tx=0x7f07c4009210 comp rx=0 tx=0).stop 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d40cd390 msgr2=0x7f07d41564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d40cd390 0x7f07d41564c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f07d00011e0 tx=0x7f07d000ca30 comp rx=0 tx=0).stop 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 shutdown_connections 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07d4153290 0x7f07d40c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07d40cd390 0x7f07d41564c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f07b8068320 0x7f07b806a7d0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.264 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 --2- 172.21.15.102:0/1013910332 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07d40cd8e0 0x7f07d40c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:16.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 >> 172.21.15.102:0/1013910332 conn(0x7f07d4065fb0 msgr2=0x7f07d4150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:16.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 shutdown_connections 2023-03-03T16:04:16.265 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:16.263+0000 7f07dbdff700 1 -- 172.21.15.102:0/1013910332 wait complete. 2023-03-03T16:04:16.265 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:04:16.290 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2023-03-03T16:04:44.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:44.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:46.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:04:46.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.553+0000 7f70f11ff700 1 Processor -- start 2023-03-03T16:04:46.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.554+0000 7f70f11ff700 1 -- start start 2023-03-03T16:04:46.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70f11ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70ec153290 0x7f70ec158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:46.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70f11ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec0cd8e0 0x7f70ec0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:46.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70f11ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec153660 con 0x7f70ec1593c0 2023-03-03T16:04:46.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70f11ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec06c360 con 0x7f70ec153290 2023-03-03T16:04:46.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70f11ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec15cea0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70ead9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70ec153290 0x7f70ec158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:46.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70e3fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f70ec1593c0 0x7f70ec0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60700/0 (socket says 172.21.15.102:60700) 2023-03-03T16:04:46.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70ead9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70ec153290 0x7f70ec158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35784/0 (socket says 172.21.15.102:35784) 2023-03-03T16:04:46.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70e3fff700 1 -- 172.21.15.102:0/2776199618 learned_addr learned my addr 172.21.15.102:0/2776199618 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:04:46.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.555+0000 7f70eb59e700 1 --2- 172.21.15.102:0/2776199618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec0cd8e0 0x7f70ec0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.556+0000 7f70ead9d700 1 -- 172.21.15.102:0/2776199618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec0cd8e0 msgr2=0x7f70ec0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.556+0000 7f70ead9d700 1 --2- 172.21.15.102:0/2776199618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec0cd8e0 0x7f70ec0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.556+0000 7f70ead9d700 1 -- 172.21.15.102:0/2776199618 >> v1:172.21.15.102:6789/0 conn(0x7f70ec1593c0 legacy=0x7f70ec0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.556+0000 7f70ead9d700 1 -- 172.21.15.102:0/2776199618 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec06c7f0 con 0x7f70ec153290 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.556+0000 7f70ea59c700 1 -- 172.21.15.102:0/2776199618 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378777279 0 0) 0x7f70ec153660 con 0x7f70ec1593c0 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.556+0000 7f70ead9d700 1 --2- 172.21.15.102:0/2776199618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec153290 0x7f70ec158e80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009840 tx=0x7f70dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98218359202219f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.556+0000 7f70ea59c700 1 -- 172.21.15.102:0/2776199618 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc011830 con 0x7f70ec153290 2023-03-03T16:04:46.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70ea59c700 1 -- 172.21.15.102:0/2776199618 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc011990 con 0x7f70ec153290 2023-03-03T16:04:46.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70ea59c700 1 -- 172.21.15.102:0/2776199618 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc011d20 con 0x7f70ec153290 2023-03-03T16:04:46.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 -- 172.21.15.102:0/2776199618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec153290 msgr2=0x7f70ec158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:46.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 --2- 172.21.15.102:0/2776199618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec153290 0x7f70ec158e80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009840 tx=0x7f70dc009b50 comp rx=0 tx=0).stop 2023-03-03T16:04:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 -- 172.21.15.102:0/2776199618 shutdown_connections 2023-03-03T16:04:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 --2- 172.21.15.102:0/2776199618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec0cd8e0 0x7f70ec0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 --2- 172.21.15.102:0/2776199618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec153290 0x7f70ec158e80 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 -- 172.21.15.102:0/2776199618 >> 172.21.15.102:0/2776199618 conn(0x7f70ec065fb0 msgr2=0x7f70ec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 -- 172.21.15.102:0/2776199618 shutdown_connections 2023-03-03T16:04:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.557+0000 7f70f11ff700 1 -- 172.21.15.102:0/2776199618 wait complete. 2023-03-03T16:04:46.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.559+0000 7f70f11ff700 1 Processor -- start 2023-03-03T16:04:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.559+0000 7f70f11ff700 1 -- start start 2023-03-03T16:04:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.560+0000 7f70f11ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec0cd8e0 0x7f70ec0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.560+0000 7f70f11ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70ec153290 0x7f70ec14e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.560+0000 7f70f11ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec1593c0 0x7f70ec14e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.560+0000 7f70f11ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0b1f90 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.560+0000 7f70f11ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0b2860 con 0x7f70ec153290 2023-03-03T16:04:46.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.560+0000 7f70f11ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0b2180 con 0x7f70ec1593c0 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec0cd8e0 0x7f70ec0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec0cd8e0 0x7f70ec0c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35790/0 (socket says 172.21.15.102:35790) 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 -- 172.21.15.102:0/1223640913 learned_addr learned my addr 172.21.15.102:0/1223640913 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70e3fff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70ec153290 0x7f70ec14e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70eb59e700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec1593c0 0x7f70ec14e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 -- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec1593c0 msgr2=0x7f70ec14e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec1593c0 0x7f70ec14e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 -- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70ec153290 msgr2=0x7f70ec14e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:46.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70ec153290 0x7f70ec14e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.561+0000 7f70ead9d700 1 -- 172.21.15.102:0/1223640913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc0094a0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.564 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.562+0000 7f70ead9d700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec0cd8e0 0x7f70ec0c4750 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009fd0 tx=0x7f70dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.562+0000 7f70e37fe700 1 -- 172.21.15.102:0/1223640913 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0036a0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.562+0000 7f70e37fe700 1 -- 172.21.15.102:0/1223640913 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc01e070 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.562+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec0ccde0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.562+0000 7f70e37fe700 1 -- 172.21.15.102:0/1223640913 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc003a60 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.562+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec15d160 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.563+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ec0cd0c0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.564+0000 7f70e37fe700 1 -- 172.21.15.102:0/1223640913 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f70dc00f050 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.567+0000 7f70e37fe700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70cc068320 0x7f70cc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:46.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.567+0000 7f70e37fe700 1 -- 172.21.15.102:0/1223640913 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f70dc086d20 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.567+0000 7f70e3fff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70cc068320 0x7f70cc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:46.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.568+0000 7f70e3fff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70cc068320 0x7f70cc06a7d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f70ec15bdd0 tx=0x7f70d4007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:46.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.571+0000 7f70e37fe700 1 -- 172.21.15.102:0/1223640913 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70dc0533b0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.805+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70ec04e6e0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.806+0000 7f70e37fe700 1 -- 172.21.15.102:0/1223640913 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f70dc0569d0 con 0x7f70ec0cd8e0 2023-03-03T16:04:46.807 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:04:46.807 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:04:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70cc068320 msgr2=0x7f70cc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70cc068320 0x7f70cc06a7d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f70ec15bdd0 tx=0x7f70d4007de0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec0cd8e0 msgr2=0x7f70ec0c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec0cd8e0 0x7f70ec0c4750 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009fd0 tx=0x7f70dc004750 comp rx=0 tx=0).stop 2023-03-03T16:04:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 shutdown_connections 2023-03-03T16:04:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70cc068320 0x7f70cc06a7d0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70ec1593c0 0x7f70ec14e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ec0cd8e0 0x7f70ec0c4750 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 --2- 172.21.15.102:0/1223640913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70ec153290 0x7f70ec14e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 >> 172.21.15.102:0/1223640913 conn(0x7f70ec065fb0 msgr2=0x7f70ec150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 shutdown_connections 2023-03-03T16:04:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:04:46.809+0000 7f70f11ff700 1 -- 172.21.15.102:0/1223640913 wait complete. 2023-03-03T16:04:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:04:46.836 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T16:05:14.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:14.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:29.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:05:30.088 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.088+0000 7fe8e0f9d700 1 Processor -- start 2023-03-03T16:05:30.089 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.088+0000 7fe8e0f9d700 1 -- start start 2023-03-03T16:05:30.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8e0f9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe8dc0ced00 0x7fe8dc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:30.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8e0f9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe8dc0c41b0 0x7fe8dc0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:30.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8e0f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc0c3d20 con 0x7fe8dc0b3970 2023-03-03T16:05:30.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8e0f9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8dc0c4930 con 0x7fe8dc0ced00 2023-03-03T16:05:30.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8e0f9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8dc0cf610 con 0x7fe8dc0c41b0 2023-03-03T16:05:30.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8db7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe8dc0ced00 0x7fe8dc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:30.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8dbfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe8dc0c41b0 0x7fe8dc0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:30.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8db7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe8dc0ced00 0x7fe8dc0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35262/0 (socket says 172.21.15.102:35262) 2023-03-03T16:05:30.092 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8db7fe700 1 -- 172.21.15.102:0/324865589 learned_addr learned my addr 172.21.15.102:0/324865589 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.090+0000 7fe8d3fff700 1 --1- 172.21.15.102:0/324865589 >> v1:172.21.15.172:6789/0 conn(0x7fe8dc0b3970 0x7fe8dc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36312/0 (socket says 172.21.15.102:36312) 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8dbfff700 1 -- 172.21.15.102:0/324865589 >> v2:172.21.15.102:3300/0 conn(0x7fe8dc0ced00 msgr2=0x7fe8dc0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8dbfff700 1 --2- 172.21.15.102:0/324865589 >> v2:172.21.15.102:3300/0 conn(0x7fe8dc0ced00 0x7fe8dc0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8dbfff700 1 -- 172.21.15.102:0/324865589 >> v1:172.21.15.172:6789/0 conn(0x7fe8dc0b3970 legacy=0x7fe8dc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8dbfff700 1 -- 172.21.15.102:0/324865589 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc15cba0 con 0x7fe8dc0c41b0 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8daffd700 1 -- 172.21.15.102:0/324865589 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606892285 0 0) 0x7fe8dc0c3d20 con 0x7fe8dc0b3970 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8db7fe700 1 --2- 172.21.15.102:0/324865589 >> v2:172.21.15.102:3300/0 conn(0x7fe8dc0ced00 0x7fe8dc0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:05:30.093 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8dbfff700 1 --2- 172.21.15.102:0/324865589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0c41b0 0x7fe8dc0cf0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc0ca080 tx=0x7fe8d400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72ddb3a9270727eb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:30.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8daffd700 1 -- 172.21.15.102:0/324865589 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d400e070 con 0x7fe8dc0c41b0 2023-03-03T16:05:30.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8daffd700 1 -- 172.21.15.102:0/324865589 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d400bb20 con 0x7fe8dc0c41b0 2023-03-03T16:05:30.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.091+0000 7fe8daffd700 1 -- 172.21.15.102:0/324865589 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4012430 con 0x7fe8dc0c41b0 2023-03-03T16:05:30.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/324865589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0c41b0 msgr2=0x7fe8dc0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:30.094 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/324865589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0c41b0 0x7fe8dc0cf0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc0ca080 tx=0x7fe8d400b330 comp rx=0 tx=0).stop 2023-03-03T16:05:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/324865589 shutdown_connections 2023-03-03T16:05:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/324865589 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0c41b0 0x7fe8dc0cf0d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/324865589 >> v2:172.21.15.102:3300/0 conn(0x7fe8dc0ced00 0x7fe8dc0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/324865589 >> 172.21.15.102:0/324865589 conn(0x7fe8dc08b1c0 msgr2=0x7fe8dc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:30.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/324865589 shutdown_connections 2023-03-03T16:05:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.092+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/324865589 wait complete. 2023-03-03T16:05:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.094+0000 7fe8e0f9d700 1 Processor -- start 2023-03-03T16:05:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.094+0000 7fe8e0f9d700 1 -- start start 2023-03-03T16:05:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.095+0000 7fe8e0f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0b3970 0x7fe8dc15bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.095+0000 7fe8e0f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8dc0c41b0 0x7fe8dc15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.095+0000 7fe8e0f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8dc0ced00 0x7fe8dc14dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.095+0000 7fe8e0f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0c8e90 con 0x7fe8dc0ced00 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.095+0000 7fe8e0f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0cd4b0 con 0x7fe8dc0b3970 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.095+0000 7fe8e0f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0c6b20 con 0x7fe8dc0c41b0 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8dc0ced00 0x7fe8dc14dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8dc0ced00 0x7fe8dc14dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35276/0 (socket says 172.21.15.102:35276) 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 -- 172.21.15.102:0/679190492 learned_addr learned my addr 172.21.15.102:0/679190492 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:05:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8db7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0b3970 0x7fe8dc15bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8d3fff700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8dc0c41b0 0x7fe8dc15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 -- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8dc0c41b0 msgr2=0x7fe8dc15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8dc0c41b0 0x7fe8dc15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 -- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0b3970 msgr2=0x7fe8dc15bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0b3970 0x7fe8dc15bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.096+0000 7fe8dbfff700 1 -- 172.21.15.102:0/679190492 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc009550 con 0x7fe8dc0ced00 2023-03-03T16:05:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.097+0000 7fe8dbfff700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8dc0ced00 0x7fe8dc14dc40 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400b680 tx=0x7fe8d4004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.097+0000 7fe8d8ff9700 1 -- 172.21.15.102:0/679190492 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d401b810 con 0x7fe8dc0ced00 2023-03-03T16:05:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.097+0000 7fe8d8ff9700 1 -- 172.21.15.102:0/679190492 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d4020070 con 0x7fe8dc0ced00 2023-03-03T16:05:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.097+0000 7fe8d8ff9700 1 -- 172.21.15.102:0/679190492 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d401bbb0 con 0x7fe8dc0ced00 2023-03-03T16:05:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.097+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d4009a70 con 0x7fe8dc0ced00 2023-03-03T16:05:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.097+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc06bf20 con 0x7fe8dc0ced00 2023-03-03T16:05:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.098+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc15ce80 con 0x7fe8dc0ced00 2023-03-03T16:05:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.099+0000 7fe8d8ff9700 1 -- 172.21.15.102:0/679190492 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8d401bd10 con 0x7fe8dc0ced00 2023-03-03T16:05:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.101+0000 7fe8d8ff9700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.101+0000 7fe8d8ff9700 1 -- 172.21.15.102:0/679190492 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe8d4022070 con 0x7fe8dc0ced00 2023-03-03T16:05:30.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.101+0000 7fe8db7fe700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:30.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.102+0000 7fe8db7fe700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006d30 tx=0x7fe8cc0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.105+0000 7fe8d8ff9700 1 -- 172.21.15.102:0/679190492 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8d4054ac0 con 0x7fe8dc0ced00 2023-03-03T16:05:30.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.340+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8dc06c200 con 0x7fe8dc0ced00 2023-03-03T16:05:30.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.341+0000 7fe8d8ff9700 1 -- 172.21.15.102:0/679190492 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe8d40580e0 con 0x7fe8dc0ced00 2023-03-03T16:05:30.342 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:05:30.342 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:05:30.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8bc0682f0 msgr2=0x7fe8bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:30.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006d30 tx=0x7fe8cc0092c0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8dc0ced00 msgr2=0x7fe8dc14dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:30.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8dc0ced00 0x7fe8dc14dc40 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400b680 tx=0x7fe8d4004b20 comp rx=0 tx=0).stop 2023-03-03T16:05:30.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 shutdown_connections 2023-03-03T16:05:30.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8dc0ced00 0x7fe8dc14dc40 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe8dc0b3970 0x7fe8dc15bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 --2- 172.21.15.102:0/679190492 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe8dc0c41b0 0x7fe8dc15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:30.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 >> 172.21.15.102:0/679190492 conn(0x7fe8dc08b1c0 msgr2=0x7fe8dc066930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:30.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 shutdown_connections 2023-03-03T16:05:30.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:05:30.344+0000 7fe8e0f9d700 1 -- 172.21.15.102:0/679190492 wait complete. 2023-03-03T16:05:30.346 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:05:30.371 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-03-03T16:05:44.534 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:44.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:14.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:14.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:44.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:44.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:07.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:07:07.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.636+0000 7f94a4dff700 1 Processor -- start 2023-03-03T16:07:07.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.637+0000 7f94a4dff700 1 -- start start 2023-03-03T16:07:07.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f94a4dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a0153290 0x7f94a0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:07.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f94a4dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94a01593c0 0x7f94a00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:07.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f94a4dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a0153660 con 0x7f94a00cd8e0 2023-03-03T16:07:07.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f94a4dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a006c360 con 0x7f94a01593c0 2023-03-03T16:07:07.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f94a4dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a015cea0 con 0x7f94a0153290 2023-03-03T16:07:07.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f949e59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a0153290 0x7f94a0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:07.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f949dd9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94a01593c0 0x7f94a00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:07.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f949ed9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f94a00cd8e0 0x7f94a00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51702/0 (socket says 172.21.15.102:51702) 2023-03-03T16:07:07.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.638+0000 7f949ed9d700 1 -- 172.21.15.102:0/3719664049 learned_addr learned my addr 172.21.15.102:0/3719664049 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:07:07.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949d59a700 1 -- 172.21.15.102:0/3719664049 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272339927 0 0) 0x7f94a0153660 con 0x7f94a00cd8e0 2023-03-03T16:07:07.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949d59a700 1 -- 172.21.15.102:0/3719664049 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94880035b0 con 0x7f94a00cd8e0 2023-03-03T16:07:07.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949e59c700 1 -- 172.21.15.102:0/3719664049 >> v2:172.21.15.172:3300/0 conn(0x7f94a01593c0 msgr2=0x7f94a00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:07:07.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949e59c700 1 --2- 172.21.15.102:0/3719664049 >> v2:172.21.15.172:3300/0 conn(0x7f94a01593c0 0x7f94a00cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949e59c700 1 -- 172.21.15.102:0/3719664049 >> v1:172.21.15.172:6789/0 conn(0x7f94a00cd8e0 legacy=0x7f94a00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:07.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949e59c700 1 -- 172.21.15.102:0/3719664049 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a006c7f0 con 0x7f94a0153290 2023-03-03T16:07:07.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949dd9b700 1 --2- 172.21.15.102:0/3719664049 >> v2:172.21.15.172:3300/0 conn(0x7f94a01593c0 0x7f94a00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:07:07.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949e59c700 1 --2- 172.21.15.102:0/3719664049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a0153290 0x7f94a0158e80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9490009840 tx=0x7f9490009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8406b8419a69ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:07:07.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949d59a700 1 -- 172.21.15.102:0/3719664049 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490011830 con 0x7f94a0153290 2023-03-03T16:07:07.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949d59a700 1 -- 172.21.15.102:0/3719664049 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9490011990 con 0x7f94a0153290 2023-03-03T16:07:07.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.639+0000 7f949d59a700 1 -- 172.21.15.102:0/3719664049 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490011d20 con 0x7f94a0153290 2023-03-03T16:07:07.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 -- 172.21.15.102:0/3719664049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a0153290 msgr2=0x7f94a0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:07.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 --2- 172.21.15.102:0/3719664049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a0153290 0x7f94a0158e80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9490009840 tx=0x7f9490009b50 comp rx=0 tx=0).stop 2023-03-03T16:07:07.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 -- 172.21.15.102:0/3719664049 shutdown_connections 2023-03-03T16:07:07.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 --2- 172.21.15.102:0/3719664049 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a0153290 0x7f94a0158e80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 --2- 172.21.15.102:0/3719664049 >> v2:172.21.15.172:3300/0 conn(0x7f94a01593c0 0x7f94a00cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 -- 172.21.15.102:0/3719664049 >> 172.21.15.102:0/3719664049 conn(0x7f94a0065fb0 msgr2=0x7f94a00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:07:07.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 -- 172.21.15.102:0/3719664049 shutdown_connections 2023-03-03T16:07:07.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.640+0000 7f94a4dff700 1 -- 172.21.15.102:0/3719664049 wait complete. 2023-03-03T16:07:07.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.643+0000 7f94a4dff700 1 Processor -- start 2023-03-03T16:07:07.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.643+0000 7f94a4dff700 1 -- start start 2023-03-03T16:07:07.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.644+0000 7f94a4dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a00cd8e0 0x7f94a014ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:07.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.644+0000 7f94a4dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94a0153290 0x7f94a01f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:07.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.644+0000 7f94a4dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a01593c0 0x7f94a014dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:07.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.644+0000 7f94a4dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a014d800 con 0x7f94a01593c0 2023-03-03T16:07:07.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.644+0000 7f94a4dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a01f2240 con 0x7f94a0153290 2023-03-03T16:07:07.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.644+0000 7f94a4dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a015bb50 con 0x7f94a00cd8e0 2023-03-03T16:07:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94a0153290 0x7f94a01f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94a0153290 0x7f94a01f1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59390/0 (socket says 172.21.15.102:59390) 2023-03-03T16:07:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 -- 172.21.15.102:0/1627197728 learned_addr learned my addr 172.21.15.102:0/1627197728 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:07:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949e59c700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a00cd8e0 0x7f94a014ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949ed9d700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a01593c0 0x7f94a014dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 -- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a00cd8e0 msgr2=0x7f94a014ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a00cd8e0 0x7f94a014ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 -- 172.21.15.102:0/1627197728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a01593c0 msgr2=0x7f94a014dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a01593c0 0x7f94a014dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 -- 172.21.15.102:0/1627197728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940094a0 con 0x7f94a0153290 2023-03-03T16:07:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.645+0000 7f949dd9b700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94a0153290 0x7f94a01f1910 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f949400ebe0 tx=0x7f949400eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:07:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.646+0000 7f948effd700 1 -- 172.21.15.102:0/1627197728 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949400c850 con 0x7f94a0153290 2023-03-03T16:07:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.646+0000 7f948effd700 1 -- 172.21.15.102:0/1627197728 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949400c9b0 con 0x7f94a0153290 2023-03-03T16:07:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.646+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94900094a0 con 0x7f94a0153290 2023-03-03T16:07:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.646+0000 7f948effd700 1 -- 172.21.15.102:0/1627197728 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949400cd40 con 0x7f94a0153290 2023-03-03T16:07:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.646+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a00ccde0 con 0x7f94a0153290 2023-03-03T16:07:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.647+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a015d160 con 0x7f94a0153290 2023-03-03T16:07:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.648+0000 7f948effd700 1 -- 172.21.15.102:0/1627197728 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9494019030 con 0x7f94a0153290 2023-03-03T16:07:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.649+0000 7f948effd700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9480068320 0x7f948006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.649+0000 7f948effd700 1 -- 172.21.15.102:0/1627197728 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9494085d80 con 0x7f94a0153290 2023-03-03T16:07:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.650+0000 7f949e59c700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9480068320 0x7f948006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.650+0000 7f949e59c700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9480068320 0x7f948006a7d0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9490000c00 tx=0x7f9490008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:07:07.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.654+0000 7f948effd700 1 -- 172.21.15.102:0/1627197728 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9494052400 con 0x7f94a0153290 2023-03-03T16:07:07.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.886+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94a00cd0c0 con 0x7f94a0153290 2023-03-03T16:07:07.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.888+0000 7f948effd700 1 -- 172.21.15.102:0/1627197728 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9494017090 con 0x7f94a0153290 2023-03-03T16:07:07.888 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:07:07.888 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:07:07.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9480068320 msgr2=0x7f948006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:07.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9480068320 0x7f948006a7d0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9490000c00 tx=0x7f9490008030 comp rx=0 tx=0).stop 2023-03-03T16:07:07.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94a0153290 msgr2=0x7f94a01f1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:07.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94a0153290 0x7f94a01f1910 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f949400ebe0 tx=0x7f949400eef0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 shutdown_connections 2023-03-03T16:07:07.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9480068320 0x7f948006a7d0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f94a00cd8e0 0x7f94a014ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a01593c0 0x7f94a014dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 --2- 172.21.15.102:0/1627197728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94a0153290 0x7f94a01f1910 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:07.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 >> 172.21.15.102:0/1627197728 conn(0x7f94a0065fb0 msgr2=0x7f94a00cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:07:07.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.890+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 shutdown_connections 2023-03-03T16:07:07.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:07:07.891+0000 7f94a4dff700 1 -- 172.21.15.102:0/1627197728 wait complete. 2023-03-03T16:07:07.893 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:07:07.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-03-03T16:07:15.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:15.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:45.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:45.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:15.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:15.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:17.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:08:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.167+0000 7ff203fff700 1 Processor -- start 2023-03-03T16:08:18.168 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.167+0000 7ff203fff700 1 -- start start 2023-03-03T16:08:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.168+0000 7ff203fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff1fc0ced80 0x7ff1fc0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.168+0000 7ff203fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.168+0000 7ff203fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff1fc0cf9a0 0x7ff1fc15c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.168+0000 7ff203fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc0ca750 con 0x7ff1fc0ced80 2023-03-03T16:08:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.168+0000 7ff203fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc15a230 con 0x7ff1fc0cf9a0 2023-03-03T16:08:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.168+0000 7ff203fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc0cf690 con 0x7ff1fc1597f0 2023-03-03T16:08:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff1fc0ced80 0x7ff1fc0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff1fc0ced80 0x7ff1fc0cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50718/0 (socket says 172.21.15.102:50718) 2023-03-03T16:08:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 -- 172.21.15.102:0/257836814 learned_addr learned my addr 172.21.15.102:0/257836814 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:08:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff20259c700 1 --2- 172.21.15.102:0/257836814 >> v2:172.21.15.172:3300/0 conn(0x7ff1fc0cf9a0 0x7ff1fc15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff20159a700 1 --2- 172.21.15.102:0/257836814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:18.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 -- 172.21.15.102:0/257836814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 msgr2=0x7ff1fc0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 --2- 172.21.15.102:0/257836814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 -- 172.21.15.102:0/257836814 >> v2:172.21.15.172:3300/0 conn(0x7ff1fc0cf9a0 msgr2=0x7ff1fc15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 --2- 172.21.15.102:0/257836814 >> v2:172.21.15.172:3300/0 conn(0x7ff1fc0cf9a0 0x7ff1fc15c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff201d9b700 1 -- 172.21.15.102:0/257836814 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0cb9f0 con 0x7ff1fc0ced80 2023-03-03T16:08:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.169+0000 7ff20259c700 1 --2- 172.21.15.102:0/257836814 >> v2:172.21.15.172:3300/0 conn(0x7ff1fc0cf9a0 0x7ff1fc15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:08:18.173 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff201d9b700 1 --2- 172.21.15.102:0/257836814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc0cf150 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc1569f0 tx=0x7ff1f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd715b4fc48124a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff200d99700 1 -- 172.21.15.102:0/257836814 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8011830 con 0x7ff1fc0ced80 2023-03-03T16:08:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff200d99700 1 -- 172.21.15.102:0/257836814 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f8011990 con 0x7ff1fc0ced80 2023-03-03T16:08:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff200d99700 1 -- 172.21.15.102:0/257836814 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8011d20 con 0x7ff1fc0ced80 2023-03-03T16:08:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff203fff700 1 -- 172.21.15.102:0/257836814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 msgr2=0x7ff1fc0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff203fff700 1 --2- 172.21.15.102:0/257836814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc0cf150 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc1569f0 tx=0x7ff1f8009c00 comp rx=0 tx=0).stop 2023-03-03T16:08:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff203fff700 1 -- 172.21.15.102:0/257836814 shutdown_connections 2023-03-03T16:08:18.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff203fff700 1 --2- 172.21.15.102:0/257836814 >> v2:172.21.15.172:3300/0 conn(0x7ff1fc0cf9a0 0x7ff1fc15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff203fff700 1 --2- 172.21.15.102:0/257836814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc0cf150 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff203fff700 1 --2- 172.21.15.102:0/257836814 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.170+0000 7ff203fff700 1 -- 172.21.15.102:0/257836814 >> 172.21.15.102:0/257836814 conn(0x7ff1fc065fb0 msgr2=0x7ff1fc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.171+0000 7ff203fff700 1 -- 172.21.15.102:0/257836814 shutdown_connections 2023-03-03T16:08:18.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.171+0000 7ff203fff700 1 -- 172.21.15.102:0/257836814 wait complete. 2023-03-03T16:08:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.173+0000 7ff203fff700 1 Processor -- start 2023-03-03T16:08:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.173+0000 7ff203fff700 1 -- start start 2023-03-03T16:08:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff203fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff203fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1fc0cf9a0 0x7ff1fc15a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff203fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc1ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff203fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc15b970 con 0x7ff1fc0ced80 2023-03-03T16:08:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff203fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc150900 con 0x7ff1fc0cf9a0 2023-03-03T16:08:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff203fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc15b3b0 con 0x7ff1fc1597f0 2023-03-03T16:08:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff201d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff201d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc14fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50722/0 (socket says 172.21.15.102:50722) 2023-03-03T16:08:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff201d9b700 1 -- 172.21.15.102:0/650752371 learned_addr learned my addr 172.21.15.102:0/650752371 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:08:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff20159a700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1fc0cf9a0 0x7ff1fc15a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.174+0000 7ff20259c700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc1ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.175+0000 7ff201d9b700 1 -- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 msgr2=0x7ff1fc1ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.175+0000 7ff201d9b700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc1ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.175+0000 7ff201d9b700 1 -- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1fc0cf9a0 msgr2=0x7ff1fc15a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.175+0000 7ff201d9b700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1fc0cf9a0 0x7ff1fc15a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.175+0000 7ff201d9b700 1 -- 172.21.15.102:0/650752371 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009a70 con 0x7ff1fc0ced80 2023-03-03T16:08:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.175+0000 7ff201d9b700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc14fa10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8000c00 tx=0x7ff1f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:18.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.175+0000 7ff1f27fc700 1 -- 172.21.15.102:0/650752371 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8004450 con 0x7ff1fc0ced80 2023-03-03T16:08:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.176+0000 7ff1f27fc700 1 -- 172.21.15.102:0/650752371 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f801e070 con 0x7ff1fc0ced80 2023-03-03T16:08:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.176+0000 7ff1f27fc700 1 -- 172.21.15.102:0/650752371 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80038c0 con 0x7ff1fc0ced80 2023-03-03T16:08:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.176+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f8009550 con 0x7ff1fc0ced80 2023-03-03T16:08:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.176+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc0cade0 con 0x7ff1fc0ced80 2023-03-03T16:08:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.177+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc159c80 con 0x7ff1fc0ced80 2023-03-03T16:08:18.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.178+0000 7ff1f27fc700 1 -- 172.21.15.102:0/650752371 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff1f800f050 con 0x7ff1fc0ced80 2023-03-03T16:08:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.179+0000 7ff1f27fc700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1e00682f0 0x7ff1e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.179+0000 7ff1f27fc700 1 -- 172.21.15.102:0/650752371 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ff1f8086ac0 con 0x7ff1fc0ced80 2023-03-03T16:08:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.180+0000 7ff20159a700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1e00682f0 0x7ff1e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:18.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.180+0000 7ff20159a700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1e00682f0 0x7ff1e006a7a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc1f17a0 tx=0x7ff1ec009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:18.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.184+0000 7ff1f27fc700 1 -- 172.21.15.102:0/650752371 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff1f80531d0 con 0x7ff1fc0ced80 2023-03-03T16:08:18.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.417+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1fc0cb0c0 con 0x7ff1fc0ced80 2023-03-03T16:08:18.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.418+0000 7ff1f27fc700 1 -- 172.21.15.102:0/650752371 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7ff1f8015090 con 0x7ff1fc0ced80 2023-03-03T16:08:18.420 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:08:18.420 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:08:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.420+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1e00682f0 msgr2=0x7ff1e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.420+0000 7ff203fff700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1e00682f0 0x7ff1e006a7a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc1f17a0 tx=0x7ff1ec009210 comp rx=0 tx=0).stop 2023-03-03T16:08:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.420+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 msgr2=0x7ff1fc14fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.420+0000 7ff203fff700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc14fa10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8000c00 tx=0x7ff1f8004750 comp rx=0 tx=0).stop 2023-03-03T16:08:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 shutdown_connections 2023-03-03T16:08:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7ff1e00682f0 0x7ff1e006a7a0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff1fc1597f0 0x7ff1fc1ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1fc0ced80 0x7ff1fc14fa10 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 --2- 172.21.15.102:0/650752371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1fc0cf9a0 0x7ff1fc15a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 >> 172.21.15.102:0/650752371 conn(0x7ff1fc065fb0 msgr2=0x7ff1fc1583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 shutdown_connections 2023-03-03T16:08:18.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:08:18.421+0000 7ff203fff700 1 -- 172.21.15.102:0/650752371 wait complete. 2023-03-03T16:08:18.424 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:08:18.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-03-03T16:08:45.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:45.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:15.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:15.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:45.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:45.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:54.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:09:54.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.715+0000 7fabee5ff700 1 Processor -- start 2023-03-03T16:09:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.716+0000 7fabee5ff700 1 -- start start 2023-03-03T16:09:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabee5ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fabe80cd8e0 0x7fabe80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabee5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabee5ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe8153660 con 0x7fabe8153290 2023-03-03T16:09:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabee5ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabe806c360 con 0x7fabe80cd8e0 2023-03-03T16:09:54.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabee5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe815cea0 con 0x7fabe80cd390 2023-03-03T16:09:54.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabe7fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fabe8153290 0x7fabe8158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:39442/0 (socket says 172.21.15.102:39442) 2023-03-03T16:09:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabe7fff700 1 -- 172.21.15.102:0/3155697904 learned_addr learned my addr 172.21.15.102:0/3155697904 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:09:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabecb9c700 1 --2- 172.21.15.102:0/3155697904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.718+0000 7fabdffff700 1 --2- 172.21.15.102:0/3155697904 >> v2:172.21.15.102:3300/0 conn(0x7fabe80cd8e0 0x7fabe80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabe77fe700 1 -- 172.21.15.102:0/3155697904 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351717163 0 0) 0x7fabe8153660 con 0x7fabe8153290 2023-03-03T16:09:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabe77fe700 1 -- 172.21.15.102:0/3155697904 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd40035b0 con 0x7fabe8153290 2023-03-03T16:09:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabecb9c700 1 -- 172.21.15.102:0/3155697904 >> v2:172.21.15.102:3300/0 conn(0x7fabe80cd8e0 msgr2=0x7fabe80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:09:54.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabecb9c700 1 --2- 172.21.15.102:0/3155697904 >> v2:172.21.15.102:3300/0 conn(0x7fabe80cd8e0 0x7fabe80cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabecb9c700 1 -- 172.21.15.102:0/3155697904 >> v1:172.21.15.172:6789/0 conn(0x7fabe8153290 legacy=0x7fabe8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabecb9c700 1 -- 172.21.15.102:0/3155697904 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe806c7f0 con 0x7fabe80cd390 2023-03-03T16:09:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabecb9c700 1 --2- 172.21.15.102:0/3155697904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe80ce360 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009840 tx=0x7fabe0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10994da43f0bf1bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabe77fe700 1 -- 172.21.15.102:0/3155697904 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0011830 con 0x7fabe80cd390 2023-03-03T16:09:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabe77fe700 1 -- 172.21.15.102:0/3155697904 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe0011990 con 0x7fabe80cd390 2023-03-03T16:09:54.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.719+0000 7fabe77fe700 1 -- 172.21.15.102:0/3155697904 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0011d20 con 0x7fabe80cd390 2023-03-03T16:09:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 -- 172.21.15.102:0/3155697904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 msgr2=0x7fabe80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3155697904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe80ce360 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009840 tx=0x7fabe0009b50 comp rx=0 tx=0).stop 2023-03-03T16:09:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 -- 172.21.15.102:0/3155697904 shutdown_connections 2023-03-03T16:09:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3155697904 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe80ce360 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3155697904 >> v2:172.21.15.102:3300/0 conn(0x7fabe80cd8e0 0x7fabe80cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 -- 172.21.15.102:0/3155697904 >> 172.21.15.102:0/3155697904 conn(0x7fabe8065fb0 msgr2=0x7fabe80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:09:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 -- 172.21.15.102:0/3155697904 shutdown_connections 2023-03-03T16:09:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.720+0000 7fabee5ff700 1 -- 172.21.15.102:0/3155697904 wait complete. 2023-03-03T16:09:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.722+0000 7fabee5ff700 1 Processor -- start 2023-03-03T16:09:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.722+0000 7fabee5ff700 1 -- start start 2023-03-03T16:09:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.723+0000 7fabee5ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe8155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.723+0000 7fabee5ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabe80cd8e0 0x7fabe814f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.723+0000 7fabee5ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe8153290 0x7fabe814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.723+0000 7fabee5ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80c8a50 con 0x7fabe8153290 2023-03-03T16:09:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.723+0000 7fabee5ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8156a60 con 0x7fabe80cd8e0 2023-03-03T16:09:54.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.723+0000 7fabee5ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80c56a0 con 0x7fabe80cd390 2023-03-03T16:09:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe8155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabdffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabe80cd8e0 0x7fabe814f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe8155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:33814/0 (socket says 172.21.15.102:33814) 2023-03-03T16:09:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 -- 172.21.15.102:0/3120779015 learned_addr learned my addr 172.21.15.102:0/3120779015 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:09:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabecb9c700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe8153290 0x7fabe814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 -- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabe80cd8e0 msgr2=0x7fabe814f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:54.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabe80cd8e0 0x7fabe814f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 -- 172.21.15.102:0/3120779015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe8153290 msgr2=0x7fabe814fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe8153290 0x7fabe814fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 -- 172.21.15.102:0/3120779015 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00094a0 con 0x7fabe80cd390 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.724+0000 7fabe7fff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe8155120 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fabd800cbf0 tx=0x7fabd800cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.725+0000 7fabe57fa700 1 -- 172.21.15.102:0/3120779015 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd800aa60 con 0x7fabe80cd390 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.725+0000 7fabe57fa700 1 -- 172.21.15.102:0/3120779015 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd800abc0 con 0x7fabe80cd390 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.725+0000 7fabe57fa700 1 -- 172.21.15.102:0/3120779015 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8012400 con 0x7fabe80cd390 2023-03-03T16:09:54.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.725+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe80ccde0 con 0x7fabe80cd390 2023-03-03T16:09:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.725+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe815d160 con 0x7fabe80cd390 2023-03-03T16:09:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.726+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe80cd0c0 con 0x7fabe80cd390 2023-03-03T16:09:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.727+0000 7fabe57fa700 1 -- 172.21.15.102:0/3120779015 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fabd8019030 con 0x7fabe80cd390 2023-03-03T16:09:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.728+0000 7fabe57fa700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabc8068320 0x7fabc806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.729+0000 7fabe57fa700 1 -- 172.21.15.102:0/3120779015 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fabd80859c0 con 0x7fabe80cd390 2023-03-03T16:09:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.729+0000 7fabdffff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabc8068320 0x7fabc806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:54.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.729+0000 7fabdffff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabc8068320 0x7fabc806a7d0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fabe8150520 tx=0x7fabd0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:54.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.733+0000 7fabe57fa700 1 -- 172.21.15.102:0/3120779015 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fabd80520d0 con 0x7fabe80cd390 2023-03-03T16:09:54.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.973+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabe804e6e0 con 0x7fabe80cd390 2023-03-03T16:09:54.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.973+0000 7fabe57fa700 1 -- 172.21.15.102:0/3120779015 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fabd80556f0 con 0x7fabe80cd390 2023-03-03T16:09:54.975 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:09:54.975 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:09:54.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabc8068320 msgr2=0x7fabc806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:54.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabc8068320 0x7fabc806a7d0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fabe8150520 tx=0x7fabd0009210 comp rx=0 tx=0).stop 2023-03-03T16:09:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 msgr2=0x7fabe8155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe8155120 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fabd800cbf0 tx=0x7fabd800cf00 comp rx=0 tx=0).stop 2023-03-03T16:09:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 shutdown_connections 2023-03-03T16:09:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fabc8068320 0x7fabc806a7d0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fabe80cd390 0x7fabe8155120 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe8153290 0x7fabe814fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 --2- 172.21.15.102:0/3120779015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabe80cd8e0 0x7fabe814f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 >> 172.21.15.102:0/3120779015 conn(0x7fabe8065fb0 msgr2=0x7fabe8158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:09:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 shutdown_connections 2023-03-03T16:09:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:09:54.976+0000 7fabee5ff700 1 -- 172.21.15.102:0/3120779015 wait complete. 2023-03-03T16:09:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:09:55.004 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-03-03T16:10:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T16:09:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:10:00.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:10:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.278+0000 7f2fa01ff700 1 Processor -- start 2023-03-03T16:10:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.279+0000 7f2fa01ff700 1 -- start start 2023-03-03T16:10:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.280+0000 7f2fa01ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f980c5f30 0x7f2f9806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.280+0000 7f2fa01ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.280+0000 7f2fa01ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f980c5390 con 0x7f2f98150c80 2023-03-03T16:10:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.280+0000 7f2fa01ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f98157ad0 con 0x7f2f980c5f30 2023-03-03T16:10:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.280+0000 7f2fa01ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f980c4a20 con 0x7f2f980c2020 2023-03-03T16:10:00.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9e79c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:00.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9d79a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f980c5f30 0x7f2f9806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:00.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9df9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2f98150c80 0x7f2f980c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36046/0 (socket says 172.21.15.102:36046) 2023-03-03T16:10:00.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9df9b700 1 -- 172.21.15.102:0/523542896 learned_addr learned my addr 172.21.15.102:0/523542896 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:10:00.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9e79c700 1 -- 172.21.15.102:0/523542896 >> v2:172.21.15.172:3300/0 conn(0x7f2f980c5f30 msgr2=0x7f2f9806d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:10:00.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9e79c700 1 --2- 172.21.15.102:0/523542896 >> v2:172.21.15.172:3300/0 conn(0x7f2f980c5f30 0x7f2f9806d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9e79c700 1 -- 172.21.15.102:0/523542896 >> v1:172.21.15.172:6789/0 conn(0x7f2f98150c80 legacy=0x7f2f980c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9e79c700 1 -- 172.21.15.102:0/523542896 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9806f010 con 0x7f2f980c2020 2023-03-03T16:10:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9d79a700 1 --2- 172.21.15.102:0/523542896 >> v2:172.21.15.172:3300/0 conn(0x7f2f980c5f30 0x7f2f9806d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:10:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9cf99700 1 -- 172.21.15.102:0/523542896 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550256396 0 0) 0x7f2f980c5390 con 0x7f2f98150c80 2023-03-03T16:10:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.281+0000 7f2f9e79c700 1 --2- 172.21.15.102:0/523542896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c2720 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009d90 tx=0x7f2f9000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c19572436ec5388 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2f9cf99700 1 -- 172.21.15.102:0/523542896 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9000e070 con 0x7f2f980c2020 2023-03-03T16:10:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2f9cf99700 1 -- 172.21.15.102:0/523542896 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9000bb20 con 0x7f2f980c2020 2023-03-03T16:10:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2f9cf99700 1 -- 172.21.15.102:0/523542896 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f90012430 con 0x7f2f980c2020 2023-03-03T16:10:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2fa01ff700 1 -- 172.21.15.102:0/523542896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 msgr2=0x7f2f980c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/523542896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c2720 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009d90 tx=0x7f2f9000b330 comp rx=0 tx=0).stop 2023-03-03T16:10:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2fa01ff700 1 -- 172.21.15.102:0/523542896 shutdown_connections 2023-03-03T16:10:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/523542896 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c2720 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/523542896 >> v2:172.21.15.172:3300/0 conn(0x7f2f980c5f30 0x7f2f9806d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.282+0000 7f2fa01ff700 1 -- 172.21.15.102:0/523542896 >> 172.21.15.102:0/523542896 conn(0x7f2f9806af90 msgr2=0x7f2f9806c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.283+0000 7f2fa01ff700 1 -- 172.21.15.102:0/523542896 shutdown_connections 2023-03-03T16:10:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.283+0000 7f2fa01ff700 1 -- 172.21.15.102:0/523542896 wait complete. 2023-03-03T16:10:00.288 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.285+0000 7f2fa01ff700 1 Processor -- start 2023-03-03T16:10:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.285+0000 7f2fa01ff700 1 -- start start 2023-03-03T16:10:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2fa01ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2fa01ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980c5f30 0x7f2f981e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2fa01ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f98150c80 0x7f2f98071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2fa01ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f98070f10 con 0x7f2f98150c80 2023-03-03T16:10:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2fa01ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f980706f0 con 0x7f2f980c5f30 2023-03-03T16:10:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2fa01ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f980c8e00 con 0x7f2f980c2020 2023-03-03T16:10:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2f9e79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f98150c80 0x7f2f98071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2f9df9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2f9d79a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980c5f30 0x7f2f981e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2f9d79a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980c5f30 0x7f2f981e4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:57812/0 (socket says 172.21.15.102:57812) 2023-03-03T16:10:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2f9e79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f98150c80 0x7f2f98071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53112/0 (socket says 172.21.15.102:53112) 2023-03-03T16:10:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2f9d79a700 1 -- 172.21.15.102:0/2225923208 learned_addr learned my addr 172.21.15.102:0/2225923208 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:10:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.286+0000 7f2f9df9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45216/0 (socket says 172.21.15.102:45216) 2023-03-03T16:10:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f9df9b700 1 -- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980c5f30 msgr2=0x7f2f981e4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f9df9b700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980c5f30 0x7f2f981e4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f9df9b700 1 -- 172.21.15.102:0/2225923208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f98150c80 msgr2=0x7f2f98071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f9df9b700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f98150c80 0x7f2f98071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f9df9b700 1 -- 172.21.15.102:0/2225923208 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f880094a0 con 0x7f2f980c2020 2023-03-03T16:10:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f9df9b700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c88c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400c410 tx=0x7f2f9400c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f8e7fc700 1 -- 172.21.15.102:0/2225923208 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94004c80 con 0x7f2f980c2020 2023-03-03T16:10:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f8e7fc700 1 -- 172.21.15.102:0/2225923208 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f94004de0 con 0x7f2f980c2020 2023-03-03T16:10:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2f8e7fc700 1 -- 172.21.15.102:0/2225923208 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94007310 con 0x7f2f980c2020 2023-03-03T16:10:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.287+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f90009a70 con 0x7f2f980c2020 2023-03-03T16:10:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.288+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f981577f0 con 0x7f2f980c2020 2023-03-03T16:10:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.289+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f98157510 con 0x7f2f980c2020 2023-03-03T16:10:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.290+0000 7f2f8e7fc700 1 -- 172.21.15.102:0/2225923208 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f9401a020 con 0x7f2f980c2020 2023-03-03T16:10:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.291+0000 7f2f8e7fc700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f7c0682f0 0x7f2f7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.291+0000 7f2f8e7fc700 1 -- 172.21.15.102:0/2225923208 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2f94085740 con 0x7f2f980c2020 2023-03-03T16:10:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.292+0000 7f2f9d79a700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f7c0682f0 0x7f2f7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.292+0000 7f2f9d79a700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f7c0682f0 0x7f2f7c06a7a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f2f98150270 tx=0x7f2f88009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:00.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.296+0000 7f2f8e7fc700 1 -- 172.21.15.102:0/2225923208 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f94051e50 con 0x7f2f980c2020 2023-03-03T16:10:00.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.530+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f980c4cc0 con 0x7f2f980c2020 2023-03-03T16:10:00.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.531+0000 7f2f8e7fc700 1 -- 172.21.15.102:0/2225923208 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2f94055470 con 0x7f2f980c2020 2023-03-03T16:10:00.532 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:10:00.533 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:10:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f7c0682f0 msgr2=0x7f2f7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f7c0682f0 0x7f2f7c06a7a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f2f98150270 tx=0x7f2f88009210 comp rx=0 tx=0).stop 2023-03-03T16:10:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 msgr2=0x7f2f980c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c88c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400c410 tx=0x7f2f9400c720 comp rx=0 tx=0).stop 2023-03-03T16:10:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 shutdown_connections 2023-03-03T16:10:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2f7c0682f0 0x7f2f7c06a7a0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2f980c2020 0x7f2f980c88c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f98150c80 0x7f2f98071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 --2- 172.21.15.102:0/2225923208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980c5f30 0x7f2f981e4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 >> 172.21.15.102:0/2225923208 conn(0x7f2f9806af90 msgr2=0x7f2f9814bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 shutdown_connections 2023-03-03T16:10:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:00.534+0000 7f2fa01ff700 1 -- 172.21.15.102:0/2225923208 wait complete. 2023-03-03T16:10:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:10:00.562 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-03-03T16:10:16.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:16.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:19.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:10:19.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.815+0000 7f8c69bff700 1 Processor -- start 2023-03-03T16:10:19.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.815+0000 7f8c69bff700 1 -- start start 2023-03-03T16:10:19.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.816+0000 7f8c69bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8c640cd8e0 0x7f8c640cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:19.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.816+0000 7f8c69bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c640ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:19.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.816+0000 7f8c69bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c64153660 con 0x7f8c64153290 2023-03-03T16:10:19.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.816+0000 7f8c69bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6406c360 con 0x7f8c640cd8e0 2023-03-03T16:10:19.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.816+0000 7f8c69bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6415cea0 con 0x7f8c640cd390 2023-03-03T16:10:19.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c637fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8c64153290 0x7f8c64158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:48564/0 (socket says 172.21.15.102:48564) 2023-03-03T16:10:19.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c62ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8c640cd8e0 0x7f8c640cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:19.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c637fe700 1 -- 172.21.15.102:0/2136556204 learned_addr learned my addr 172.21.15.102:0/2136556204 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:10:19.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c63fff700 1 --2- 172.21.15.102:0/2136556204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c640ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:19.819 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c627fc700 1 -- 172.21.15.102:0/2136556204 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264620651 0 0) 0x7f8c64153660 con 0x7f8c64153290 2023-03-03T16:10:19.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c627fc700 1 -- 172.21.15.102:0/2136556204 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4c0035b0 con 0x7f8c64153290 2023-03-03T16:10:19.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c63fff700 1 -- 172.21.15.102:0/2136556204 >> v2:172.21.15.172:3300/0 conn(0x7f8c640cd8e0 msgr2=0x7f8c640cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:10:19.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c63fff700 1 --2- 172.21.15.102:0/2136556204 >> v2:172.21.15.172:3300/0 conn(0x7f8c640cd8e0 0x7f8c640cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:19.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c63fff700 1 -- 172.21.15.102:0/2136556204 >> v1:172.21.15.172:6789/0 conn(0x7f8c64153290 legacy=0x7f8c64158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:19.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c63fff700 1 -- 172.21.15.102:0/2136556204 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6406c7f0 con 0x7f8c640cd390 2023-03-03T16:10:19.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.817+0000 7f8c62ffd700 1 --2- 172.21.15.102:0/2136556204 >> v2:172.21.15.172:3300/0 conn(0x7f8c640cd8e0 0x7f8c640cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:10:19.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.818+0000 7f8c63fff700 1 --2- 172.21.15.102:0/2136556204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c640ce360 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009d90 tx=0x7f8c5c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76da465a8d1a67d1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.818+0000 7f8c627fc700 1 -- 172.21.15.102:0/2136556204 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00e070 con 0x7f8c640cd390 2023-03-03T16:10:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.818+0000 7f8c627fc700 1 -- 172.21.15.102:0/2136556204 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c00bb20 con 0x7f8c640cd390 2023-03-03T16:10:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.818+0000 7f8c627fc700 1 -- 172.21.15.102:0/2136556204 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00be90 con 0x7f8c640cd390 2023-03-03T16:10:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.818+0000 7f8c69bff700 1 -- 172.21.15.102:0/2136556204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 msgr2=0x7f8c640ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.818+0000 7f8c69bff700 1 --2- 172.21.15.102:0/2136556204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c640ce360 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009d90 tx=0x7f8c5c00b330 comp rx=0 tx=0).stop 2023-03-03T16:10:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.819+0000 7f8c69bff700 1 -- 172.21.15.102:0/2136556204 shutdown_connections 2023-03-03T16:10:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.819+0000 7f8c69bff700 1 --2- 172.21.15.102:0/2136556204 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c640ce360 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.819+0000 7f8c69bff700 1 --2- 172.21.15.102:0/2136556204 >> v2:172.21.15.172:3300/0 conn(0x7f8c640cd8e0 0x7f8c640cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.819+0000 7f8c69bff700 1 -- 172.21.15.102:0/2136556204 >> 172.21.15.102:0/2136556204 conn(0x7f8c64065fb0 msgr2=0x7f8c640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.819+0000 7f8c69bff700 1 -- 172.21.15.102:0/2136556204 shutdown_connections 2023-03-03T16:10:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.819+0000 7f8c69bff700 1 -- 172.21.15.102:0/2136556204 wait complete. 2023-03-03T16:10:19.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.821+0000 7f8c69bff700 1 Processor -- start 2023-03-03T16:10:19.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.821+0000 7f8c69bff700 1 -- start start 2023-03-03T16:10:19.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.822+0000 7f8c69bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c64150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:19.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.822+0000 7f8c69bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c640cd8e0 0x7f8c640b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:19.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.822+0000 7f8c69bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c64153290 0x7f8c640b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:19.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.822+0000 7f8c69bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c640b2460 con 0x7f8c640cd8e0 2023-03-03T16:10:19.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.822+0000 7f8c69bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c640b4c10 con 0x7f8c64153290 2023-03-03T16:10:19.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.822+0000 7f8c69bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c640b46c0 con 0x7f8c640cd390 2023-03-03T16:10:19.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.822+0000 7f8c637fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c64150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:19.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c64150470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:45346/0 (socket says 172.21.15.102:45346) 2023-03-03T16:10:19.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 -- 172.21.15.102:0/714559364 learned_addr learned my addr 172.21.15.102:0/714559364 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:10:19.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c63fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c64153290 0x7f8c640b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c62ffd700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c640cd8e0 0x7f8c640b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 -- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c64153290 msgr2=0x7f8c640b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c64153290 0x7f8c640b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 -- 172.21.15.102:0/714559364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c640cd8e0 msgr2=0x7f8c640b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c640cd8e0 0x7f8c640b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 -- 172.21.15.102:0/714559364 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c580094a0 con 0x7f8c640cd390 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.823+0000 7f8c637fe700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c64150470 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400c4d0 tx=0x7f8c5400c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:19.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.824+0000 7f8c4bfff700 1 -- 172.21.15.102:0/714559364 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c54005d70 con 0x7f8c640cd390 2023-03-03T16:10:19.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.824+0000 7f8c4bfff700 1 -- 172.21.15.102:0/714559364 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c540070d0 con 0x7f8c640cd390 2023-03-03T16:10:19.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.824+0000 7f8c4bfff700 1 -- 172.21.15.102:0/714559364 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c54007440 con 0x7f8c640cd390 2023-03-03T16:10:19.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.824+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c009a70 con 0x7f8c640cd390 2023-03-03T16:10:19.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.824+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c640ccde0 con 0x7f8c640cd390 2023-03-03T16:10:19.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.825+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6415d160 con 0x7f8c640cd390 2023-03-03T16:10:19.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.826+0000 7f8c4bfff700 1 -- 172.21.15.102:0/714559364 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c54019030 con 0x7f8c640cd390 2023-03-03T16:10:19.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.827+0000 7f8c4bfff700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c500682f0 0x7f8c5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:19.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.828+0000 7f8c4bfff700 1 -- 172.21.15.102:0/714559364 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8c54084f40 con 0x7f8c640cd390 2023-03-03T16:10:19.828 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.828+0000 7f8c62ffd700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c500682f0 0x7f8c5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:19.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.828+0000 7f8c62ffd700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c500682f0 0x7f8c5006a7a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f8c6415a510 tx=0x7f8c58009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:19.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:19.832+0000 7f8c4bfff700 1 -- 172.21.15.102:0/714559364 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c5404b5e0 con 0x7f8c640cd390 2023-03-03T16:10:20.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.066+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c640cd0c0 con 0x7f8c640cd390 2023-03-03T16:10:20.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.067+0000 7f8c4bfff700 1 -- 172.21.15.102:0/714559364 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8c54017020 con 0x7f8c640cd390 2023-03-03T16:10:20.067 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:10:20.068 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:10:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c500682f0 msgr2=0x7f8c5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c500682f0 0x7f8c5006a7a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f8c6415a510 tx=0x7f8c58009210 comp rx=0 tx=0).stop 2023-03-03T16:10:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 msgr2=0x7f8c64150470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c64150470 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400c4d0 tx=0x7f8c5400c7e0 comp rx=0 tx=0).stop 2023-03-03T16:10:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 shutdown_connections 2023-03-03T16:10:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8c500682f0 0x7f8c5006a7a0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8c640cd390 0x7f8c64150470 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c64153290 0x7f8c640b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 --2- 172.21.15.102:0/714559364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c640cd8e0 0x7f8c640b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 >> 172.21.15.102:0/714559364 conn(0x7f8c64065fb0 msgr2=0x7f8c64158920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 shutdown_connections 2023-03-03T16:10:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:10:20.070+0000 7f8c69bff700 1 -- 172.21.15.102:0/714559364 wait complete. 2023-03-03T16:10:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:10:20.098 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 82 secs before thrashing 2023-03-03T16:10:46.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:46.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:16.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:16.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:42.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:11:42.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.350+0000 7f66b4dff700 1 Processor -- start 2023-03-03T16:11:42.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.350+0000 7f66b4dff700 1 -- start start 2023-03-03T16:11:42.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.351+0000 7f66b4dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66b0153290 0x7f66b0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:42.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.351+0000 7f66b4dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66b00cd8e0 0x7f66b00ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:42.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.351+0000 7f66b4dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b0153660 con 0x7f66b01593c0 2023-03-03T16:11:42.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.351+0000 7f66b4dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b006c360 con 0x7f66b00cd8e0 2023-03-03T16:11:42.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.351+0000 7f66b4dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b015cea0 con 0x7f66b0153290 2023-03-03T16:11:42.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66ae59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f66b0153290 0x7f66b0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66add9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f66b01593c0 0x7f66b00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51196/0 (socket says 172.21.15.102:51196) 2023-03-03T16:11:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66add9b700 1 -- 172.21.15.102:0/3434512142 learned_addr learned my addr 172.21.15.102:0/3434512142 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:11:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66aed9d700 1 --2- 172.21.15.102:0/3434512142 >> v2:172.21.15.102:3300/0 conn(0x7f66b00cd8e0 0x7f66b00ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66ae59c700 1 -- 172.21.15.102:0/3434512142 >> v2:172.21.15.102:3300/0 conn(0x7f66b00cd8e0 msgr2=0x7f66b00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:11:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66ae59c700 1 --2- 172.21.15.102:0/3434512142 >> v2:172.21.15.102:3300/0 conn(0x7f66b00cd8e0 0x7f66b00ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66ae59c700 1 -- 172.21.15.102:0/3434512142 >> v1:172.21.15.172:6789/0 conn(0x7f66b01593c0 legacy=0x7f66b00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66ae59c700 1 -- 172.21.15.102:0/3434512142 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b006c7f0 con 0x7f66b0153290 2023-03-03T16:11:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66ad59a700 1 -- 172.21.15.102:0/3434512142 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855048816 0 0) 0x7f66b0153660 con 0x7f66b01593c0 2023-03-03T16:11:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.352+0000 7f66ae59c700 1 --2- 172.21.15.102:0/3434512142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 0x7f66b0158e80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f66b008cc70 tx=0x7f66a0009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42dc48320ca1cce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:11:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66ad59a700 1 -- 172.21.15.102:0/3434512142 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0011830 con 0x7f66b0153290 2023-03-03T16:11:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66ad59a700 1 -- 172.21.15.102:0/3434512142 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a0011990 con 0x7f66b0153290 2023-03-03T16:11:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66ad59a700 1 -- 172.21.15.102:0/3434512142 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0011d20 con 0x7f66b0153290 2023-03-03T16:11:42.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66b4dff700 1 -- 172.21.15.102:0/3434512142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 msgr2=0x7f66b0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:42.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66b4dff700 1 --2- 172.21.15.102:0/3434512142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 0x7f66b0158e80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f66b008cc70 tx=0x7f66a0009b20 comp rx=0 tx=0).stop 2023-03-03T16:11:42.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66b4dff700 1 -- 172.21.15.102:0/3434512142 shutdown_connections 2023-03-03T16:11:42.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66b4dff700 1 --2- 172.21.15.102:0/3434512142 >> v2:172.21.15.102:3300/0 conn(0x7f66b00cd8e0 0x7f66b00ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66b4dff700 1 --2- 172.21.15.102:0/3434512142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 0x7f66b0158e80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66b4dff700 1 -- 172.21.15.102:0/3434512142 >> 172.21.15.102:0/3434512142 conn(0x7f66b0065fb0 msgr2=0x7f66b00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:11:42.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.353+0000 7f66b4dff700 1 -- 172.21.15.102:0/3434512142 shutdown_connections 2023-03-03T16:11:42.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.354+0000 7f66b4dff700 1 -- 172.21.15.102:0/3434512142 wait complete. 2023-03-03T16:11:42.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.355+0000 7f66b4dff700 1 Processor -- start 2023-03-03T16:11:42.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.356+0000 7f66b4dff700 1 -- start start 2023-03-03T16:11:42.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66b4dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b00cd8e0 0x7f66b01ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:42.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66b4dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 0x7f66b00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:42.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66b4dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66b01593c0 0x7f66b00c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:42.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66b4dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00c8970 con 0x7f66b00cd8e0 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66b4dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00c78d0 con 0x7f66b0153290 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66b4dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00c6eb0 con 0x7f66b01593c0 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b00cd8e0 0x7f66b01ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66add9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 0x7f66b00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66aed9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66b01593c0 0x7f66b00c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b00cd8e0 0x7f66b01ef0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54538/0 (socket says 172.21.15.102:54538) 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 -- 172.21.15.102:0/798536964 learned_addr learned my addr 172.21.15.102:0/798536964 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:11:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 -- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66b01593c0 msgr2=0x7f66b00c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:42.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66b01593c0 0x7f66b00c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 -- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 msgr2=0x7f66b00cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:42.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 0x7f66b00cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.357+0000 7f66ae59c700 1 -- 172.21.15.102:0/798536964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00094a0 con 0x7f66b00cd8e0 2023-03-03T16:11:42.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.358+0000 7f66ae59c700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b00cd8e0 0x7f66b01ef0f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f66b008cd80 tx=0x7f66a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:11:42.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.358+0000 7f669effd700 1 -- 172.21.15.102:0/798536964 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a00036a0 con 0x7f66b00cd8e0 2023-03-03T16:11:42.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.358+0000 7f669effd700 1 -- 172.21.15.102:0/798536964 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a001f070 con 0x7f66b00cd8e0 2023-03-03T16:11:42.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.358+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b00ccde0 con 0x7f66b00cd8e0 2023-03-03T16:11:42.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.358+0000 7f669effd700 1 -- 172.21.15.102:0/798536964 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0003a20 con 0x7f66b00cd8e0 2023-03-03T16:11:42.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.358+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b015d160 con 0x7f66b00cd8e0 2023-03-03T16:11:42.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.359+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b00cd0c0 con 0x7f66b00cd8e0 2023-03-03T16:11:42.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.360+0000 7f669effd700 1 -- 172.21.15.102:0/798536964 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f66a000f050 con 0x7f66b00cd8e0 2023-03-03T16:11:42.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.362+0000 7f669effd700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66900682f0 0x7f669006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:42.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.362+0000 7f669effd700 1 -- 172.21.15.102:0/798536964 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f66a0021070 con 0x7f66b00cd8e0 2023-03-03T16:11:42.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.362+0000 7f66add9b700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66900682f0 0x7f669006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:42.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.363+0000 7f66add9b700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66900682f0 0x7f669006a7a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f66b00cff60 tx=0x7f66a4007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:11:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.366+0000 7f669effd700 1 -- 172.21.15.102:0/798536964 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66a0053ce0 con 0x7f66b00cd8e0 2023-03-03T16:11:42.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.599+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66b00c7bb0 con 0x7f66b00cd8e0 2023-03-03T16:11:42.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.599+0000 7f669effd700 1 -- 172.21.15.102:0/798536964 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f66a0057300 con 0x7f66b00cd8e0 2023-03-03T16:11:42.600 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:11:42.600 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:11:42.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66900682f0 msgr2=0x7f669006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:42.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66900682f0 0x7f669006a7a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f66b00cff60 tx=0x7f66a4007de0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b00cd8e0 msgr2=0x7f66b01ef0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:42.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b00cd8e0 0x7f66b01ef0f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f66b008cd80 tx=0x7f66a0004750 comp rx=0 tx=0).stop 2023-03-03T16:11:42.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 shutdown_connections 2023-03-03T16:11:42.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f66900682f0 0x7f669006a7a0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f66b01593c0 0x7f66b00c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b00cd8e0 0x7f66b01ef0f0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 --2- 172.21.15.102:0/798536964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66b0153290 0x7f66b00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:42.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 >> 172.21.15.102:0/798536964 conn(0x7f66b0065fb0 msgr2=0x7f66b0150f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:11:42.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 shutdown_connections 2023-03-03T16:11:42.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:11:42.602+0000 7f66b4dff700 1 -- 172.21.15.102:0/798536964 wait complete. 2023-03-03T16:11:42.604 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:11:42.630 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 86 secs before thrashing 2023-03-03T16:11:46.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:46.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:16.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:16.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:46.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:46.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:08.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:13:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.887+0000 7f5dcfdff700 1 Processor -- start 2023-03-03T16:13:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.888+0000 7f5dcfdff700 1 -- start start 2023-03-03T16:13:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcfdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5dc80cea00 0x7f5dc806cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcfdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc815cd00 con 0x7f5dc8152eb0 2023-03-03T16:13:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcfdff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc815cfa0 con 0x7f5dc80cd040 2023-03-03T16:13:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcfdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc80ce5b0 con 0x7f5dc80cea00 2023-03-03T16:13:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcd39a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5dc80cea00 0x7f5dc806cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcd39a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5dc80cea00 0x7f5dc806cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43704/0 (socket says 172.21.15.102:43704) 2023-03-03T16:13:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcd39a700 1 -- 172.21.15.102:0/2171035205 learned_addr learned my addr 172.21.15.102:0/2171035205 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:13:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dcdb9b700 1 --1- 172.21.15.102:0/2171035205 >> v1:172.21.15.102:6789/0 conn(0x7f5dc8152eb0 0x7f5dc806b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39770/0 (socket says 172.21.15.102:39770) 2023-03-03T16:13:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.889+0000 7f5dce39c700 1 --1- 172.21.15.102:0/2171035205 >> v1:172.21.15.172:6789/0 conn(0x7f5dc80cd040 0x7f5dc815c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33970/0 (socket says 172.21.15.102:33970) 2023-03-03T16:13:08.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.890+0000 7f5dcd39a700 1 -- 172.21.15.102:0/2171035205 >> v1:172.21.15.172:6789/0 conn(0x7f5dc80cd040 legacy=0x7f5dc815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.890+0000 7f5dcd39a700 1 -- 172.21.15.102:0/2171035205 >> v1:172.21.15.102:6789/0 conn(0x7f5dc8152eb0 legacy=0x7f5dc806b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.890+0000 7f5dcd39a700 1 -- 172.21.15.102:0/2171035205 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc80cf750 con 0x7f5dc80cea00 2023-03-03T16:13:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.890+0000 7f5dccb99700 1 -- 172.21.15.102:0/2171035205 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169000473 0 0) 0x7f5dc815cfa0 con 0x7f5dc80cd040 2023-03-03T16:13:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.890+0000 7f5dcd39a700 1 --2- 172.21.15.102:0/2171035205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc806cfd0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5db80098f0 tx=0x7f5db8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b560d500fba793 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dccb99700 1 -- 172.21.15.102:0/2171035205 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db800b8a0 con 0x7f5dc80cea00 2023-03-03T16:13:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dccb99700 1 -- 172.21.15.102:0/2171035205 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db800ba00 con 0x7f5dc80cea00 2023-03-03T16:13:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dccb99700 1 -- 172.21.15.102:0/2171035205 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db800bd90 con 0x7f5dc80cea00 2023-03-03T16:13:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dcfdff700 1 -- 172.21.15.102:0/2171035205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 msgr2=0x7f5dc806cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/2171035205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc806cfd0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5db80098f0 tx=0x7f5db8009c00 comp rx=0 tx=0).stop 2023-03-03T16:13:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dcfdff700 1 -- 172.21.15.102:0/2171035205 shutdown_connections 2023-03-03T16:13:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/2171035205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc806cfd0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dcfdff700 1 -- 172.21.15.102:0/2171035205 >> 172.21.15.102:0/2171035205 conn(0x7f5dc8066c40 msgr2=0x7f5dc80cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:13:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dcfdff700 1 -- 172.21.15.102:0/2171035205 shutdown_connections 2023-03-03T16:13:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.891+0000 7f5dcfdff700 1 -- 172.21.15.102:0/2171035205 wait complete. 2023-03-03T16:13:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.893+0000 7f5dcfdff700 1 Processor -- start 2023-03-03T16:13:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.894+0000 7f5dcfdff700 1 -- start start 2023-03-03T16:13:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.894+0000 7f5dcfdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dc80cd040 0x7f5dc81e1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.894+0000 7f5dcfdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc80c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.894+0000 7f5dcfdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dc8152eb0 0x7f5dc80c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcfdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc81ef590 con 0x7f5dc80cea00 2023-03-03T16:13:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcfdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc81ee700 con 0x7f5dc80cd040 2023-03-03T16:13:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcfdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc81e2530 con 0x7f5dc8152eb0 2023-03-03T16:13:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcd39a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc80c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcdb9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dc80cd040 0x7f5dc81e1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dce39c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dc8152eb0 0x7f5dc80c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcdb9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dc80cd040 0x7f5dc81e1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34984/0 (socket says 172.21.15.102:34984) 2023-03-03T16:13:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcdb9b700 1 -- 172.21.15.102:0/556469860 learned_addr learned my addr 172.21.15.102:0/556469860 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:13:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcd39a700 1 -- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dc8152eb0 msgr2=0x7f5dc80c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcd39a700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dc8152eb0 0x7f5dc80c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcd39a700 1 -- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dc80cd040 msgr2=0x7f5dc81e1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcd39a700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dc80cd040 0x7f5dc81e1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcd39a700 1 -- 172.21.15.102:0/556469860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db8009550 con 0x7f5dc80cea00 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dce39c700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dc8152eb0 0x7f5dc80c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.895+0000 7f5dcdb9b700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dc80cd040 0x7f5dc81e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.896+0000 7f5dcd39a700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc80c8500 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5db8016000 tx=0x7f5db8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.896+0000 7f5dbe7fc700 1 -- 172.21.15.102:0/556469860 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db801a900 con 0x7f5dc80cea00 2023-03-03T16:13:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.896+0000 7f5dbe7fc700 1 -- 172.21.15.102:0/556469860 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db801f070 con 0x7f5dc80cea00 2023-03-03T16:13:08.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.896+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc8153280 con 0x7f5dc80cea00 2023-03-03T16:13:08.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.896+0000 7f5dbe7fc700 1 -- 172.21.15.102:0/556469860 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db801ac60 con 0x7f5dc80cea00 2023-03-03T16:13:08.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.896+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc80cfa30 con 0x7f5dc80cea00 2023-03-03T16:13:08.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.897+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc80cfd10 con 0x7f5dc80cea00 2023-03-03T16:13:08.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.898+0000 7f5dbe7fc700 1 -- 172.21.15.102:0/556469860 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5db800f050 con 0x7f5dc80cea00 2023-03-03T16:13:08.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.900+0000 7f5dbe7fc700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dac0682f0 0x7f5dac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:08.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.901+0000 7f5dbe7fc700 1 -- 172.21.15.102:0/556469860 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5db8021070 con 0x7f5dc80cea00 2023-03-03T16:13:08.902 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.901+0000 7f5dcdb9b700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dac0682f0 0x7f5dac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:08.903 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.901+0000 7f5dcdb9b700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dac0682f0 0x7f5dac06a7a0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000f60 tx=0x7f5dc4007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:08.906 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:08.905+0000 7f5dbe7fc700 1 -- 172.21.15.102:0/556469860 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5db8053d80 con 0x7f5dc80cea00 2023-03-03T16:13:09.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.141+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc81ef6d0 con 0x7f5dc80cea00 2023-03-03T16:13:09.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.142+0000 7f5dbe7fc700 1 -- 172.21.15.102:0/556469860 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5db80573a0 con 0x7f5dc80cea00 2023-03-03T16:13:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:13:09.144 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:13:09.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dac0682f0 msgr2=0x7f5dac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:09.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dac0682f0 0x7f5dac06a7a0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000f60 tx=0x7f5dc4007f90 comp rx=0 tx=0).stop 2023-03-03T16:13:09.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 msgr2=0x7f5dc80c8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:09.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc80c8500 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5db8016000 tx=0x7f5db8004750 comp rx=0 tx=0).stop 2023-03-03T16:13:09.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 shutdown_connections 2023-03-03T16:13:09.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f5dac0682f0 0x7f5dac06a7a0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:09.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f5dc8152eb0 0x7f5dc80c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:09.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dc80cd040 0x7f5dc81e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 --2- 172.21.15.102:0/556469860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dc80cea00 0x7f5dc80c8500 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 >> 172.21.15.102:0/556469860 conn(0x7f5dc8066c40 msgr2=0x7f5dc815aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:13:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 shutdown_connections 2023-03-03T16:13:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:09.145+0000 7f5dcfdff700 1 -- 172.21.15.102:0/556469860 wait complete. 2023-03-03T16:13:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:13:09.173 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 11 secs before thrashing 2023-03-03T16:13:16.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:16.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:20.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:13:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.422+0000 7f1f1619d700 1 Processor -- start 2023-03-03T16:13:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.423+0000 7f1f1619d700 1 -- start start 2023-03-03T16:13:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.424+0000 7f1f1619d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f100b3970 0x7f1f100c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.424+0000 7f1f1619d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f100c3d20 con 0x7f1f100c41b0 2023-03-03T16:13:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.424+0000 7f1f1619d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1015cba0 con 0x7f1f100ced00 2023-03-03T16:13:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.424+0000 7f1f1619d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f100c4930 con 0x7f1f100b3970 2023-03-03T16:13:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.424+0000 7f1f1499a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1f100b3970 0x7f1f100c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.424+0000 7f1f1519b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1f100ced00 0x7f1f10158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34330/0 (socket says 172.21.15.102:34330) 2023-03-03T16:13:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.424+0000 7f1f1519b700 1 -- 172.21.15.102:0/1759497416 learned_addr learned my addr 172.21.15.102:0/1759497416 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:13:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.425+0000 7f1f1499a700 1 -- 172.21.15.102:0/1759497416 >> v1:172.21.15.172:6789/0 conn(0x7f1f100ced00 legacy=0x7f1f10158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.425+0000 7f1f1499a700 1 -- 172.21.15.102:0/1759497416 >> v1:172.21.15.102:6789/0 conn(0x7f1f100c41b0 legacy=0x7f1f10163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.425+0000 7f1f1499a700 1 -- 172.21.15.102:0/1759497416 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1015ce80 con 0x7f1f100b3970 2023-03-03T16:13:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.425+0000 7f1f1499a700 1 --2- 172.21.15.102:0/1759497416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f100c5a90 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009840 tx=0x7f1f00009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f6eccbc28d2a06 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.425+0000 7f1f07fff700 1 -- 172.21.15.102:0/1759497416 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00011830 con 0x7f1f100b3970 2023-03-03T16:13:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.425+0000 7f1f07fff700 1 -- 172.21.15.102:0/1759497416 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00011990 con 0x7f1f100b3970 2023-03-03T16:13:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.425+0000 7f1f07fff700 1 -- 172.21.15.102:0/1759497416 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00011d20 con 0x7f1f100b3970 2023-03-03T16:13:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.426+0000 7f1f1619d700 1 -- 172.21.15.102:0/1759497416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 msgr2=0x7f1f100c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.426+0000 7f1f1619d700 1 --2- 172.21.15.102:0/1759497416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f100c5a90 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009840 tx=0x7f1f00009b50 comp rx=0 tx=0).stop 2023-03-03T16:13:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.426+0000 7f1f1619d700 1 -- 172.21.15.102:0/1759497416 shutdown_connections 2023-03-03T16:13:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.426+0000 7f1f1619d700 1 --2- 172.21.15.102:0/1759497416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f100c5a90 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.426+0000 7f1f1619d700 1 -- 172.21.15.102:0/1759497416 >> 172.21.15.102:0/1759497416 conn(0x7f1f1008b1c0 msgr2=0x7f1f10071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:13:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.426+0000 7f1f1619d700 1 -- 172.21.15.102:0/1759497416 shutdown_connections 2023-03-03T16:13:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.426+0000 7f1f1619d700 1 -- 172.21.15.102:0/1759497416 wait complete. 2023-03-03T16:13:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.428+0000 7f1f1619d700 1 Processor -- start 2023-03-03T16:13:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.428+0000 7f1f1619d700 1 -- start start 2023-03-03T16:13:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.429+0000 7f1f1619d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f101512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.429+0000 7f1f1619d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f100c41b0 0x7f1f10159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.429+0000 7f1f1619d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f100ced00 0x7f1f10151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.429+0000 7f1f1619d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1014d850 con 0x7f1f100b3970 2023-03-03T16:13:20.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.429+0000 7f1f1619d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101f2900 con 0x7f1f100c41b0 2023-03-03T16:13:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.429+0000 7f1f1619d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1015a200 con 0x7f1f100ced00 2023-03-03T16:13:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f101512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f101512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52884/0 (socket says 172.21.15.102:52884) 2023-03-03T16:13:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 -- 172.21.15.102:0/204309935 learned_addr learned my addr 172.21.15.102:0/204309935 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:13:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1499a700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f100c41b0 0x7f1f10159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1599c700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f100ced00 0x7f1f10151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:20.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 -- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f100ced00 msgr2=0x7f1f10151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:20.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f100ced00 0x7f1f10151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:20.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 -- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f100c41b0 msgr2=0x7f1f10159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:20.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f100c41b0 0x7f1f10159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:20.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 -- 172.21.15.102:0/204309935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000094a0 con 0x7f1f100b3970 2023-03-03T16:13:20.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.430+0000 7f1f1519b700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f101512e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00c520 tx=0x7f1f0c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:20.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.431+0000 7f1f05ffb700 1 -- 172.21.15.102:0/204309935 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c005d70 con 0x7f1f100b3970 2023-03-03T16:13:20.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.431+0000 7f1f05ffb700 1 -- 172.21.15.102:0/204309935 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c0070d0 con 0x7f1f100b3970 2023-03-03T16:13:20.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.431+0000 7f1f05ffb700 1 -- 172.21.15.102:0/204309935 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c007440 con 0x7f1f100b3970 2023-03-03T16:13:20.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.431+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1006c200 con 0x7f1f100b3970 2023-03-03T16:13:20.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.431+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1006bf20 con 0x7f1f100b3970 2023-03-03T16:13:20.434 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.432+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1015c890 con 0x7f1f100b3970 2023-03-03T16:13:20.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.433+0000 7f1f05ffb700 1 -- 172.21.15.102:0/204309935 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f0c019030 con 0x7f1f100b3970 2023-03-03T16:13:20.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.434+0000 7f1f05ffb700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ef4068320 0x7f1ef406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:20.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.435+0000 7f1f05ffb700 1 -- 172.21.15.102:0/204309935 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1f0c084ee0 con 0x7f1f100b3970 2023-03-03T16:13:20.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.435+0000 7f1f1499a700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ef4068320 0x7f1ef406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:20.435 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.435+0000 7f1f1499a700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ef4068320 0x7f1ef406a7d0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f1f000090b0 tx=0x7f1f0001c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:20.438 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.438+0000 7f1f05ffb700 1 -- 172.21.15.102:0/204309935 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f0c04b5e0 con 0x7f1f100b3970 2023-03-03T16:13:20.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.674+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f101f2be0 con 0x7f1f100b3970 2023-03-03T16:13:20.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.674+0000 7f1f05ffb700 1 -- 172.21.15.102:0/204309935 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1f0c017090 con 0x7f1f100b3970 2023-03-03T16:13:20.675 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:13:20.675 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:13:20.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ef4068320 msgr2=0x7f1ef406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:20.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ef4068320 0x7f1ef406a7d0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f1f000090b0 tx=0x7f1f0001c040 comp rx=0 tx=0).stop 2023-03-03T16:13:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 msgr2=0x7f1f101512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f101512e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00c520 tx=0x7f1f0c00c830 comp rx=0 tx=0).stop 2023-03-03T16:13:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 shutdown_connections 2023-03-03T16:13:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1ef4068320 0x7f1ef406a7d0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f100ced00 0x7f1f10151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f100b3970 0x7f1f101512e0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 --2- 172.21.15.102:0/204309935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f100c41b0 0x7f1f10159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 >> 172.21.15.102:0/204309935 conn(0x7f1f1008b1c0 msgr2=0x7f1f100cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:13:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 shutdown_connections 2023-03-03T16:13:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:13:20.677+0000 7f1f1619d700 1 -- 172.21.15.102:0/204309935 wait complete. 2023-03-03T16:13:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:13:20.705 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-03-03T16:13:47.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:47.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:17.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:17.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:47.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:47.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:13.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:15:13.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.962+0000 7f1074fff700 1 Processor -- start 2023-03-03T16:15:13.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.963+0000 7f1074fff700 1 -- start start 2023-03-03T16:15:13.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f1074fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f107014f4d0 0x7f1070155150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:13.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f1074fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f10700706f0 0x7f1070070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:13.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f1074fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10700c7500 con 0x7f10700715a0 2023-03-03T16:15:13.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f1074fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1070071a00 con 0x7f107014f4d0 2023-03-03T16:15:13.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f1074fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10700728f0 con 0x7f10700706f0 2023-03-03T16:15:13.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f106e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f107014f4d0 0x7f1070155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:13.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f106dd9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f10700715a0 0x7f10700c7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44498/0 (socket says 172.21.15.102:44498) 2023-03-03T16:15:13.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f106dd9b700 1 -- 172.21.15.102:0/1938857124 learned_addr learned my addr 172.21.15.102:0/1938857124 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:15:13.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f106e59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f107014f4d0 0x7f1070155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58050/0 (socket says 172.21.15.102:58050) 2023-03-03T16:15:13.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.965+0000 7f106ed9d700 1 --2- 172.21.15.102:0/1938857124 >> v2:172.21.15.172:3300/0 conn(0x7f10700706f0 0x7f1070070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:13.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.966+0000 7f106e59c700 1 -- 172.21.15.102:0/1938857124 >> v2:172.21.15.172:3300/0 conn(0x7f10700706f0 msgr2=0x7f1070070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.966+0000 7f106e59c700 1 --2- 172.21.15.102:0/1938857124 >> v2:172.21.15.172:3300/0 conn(0x7f10700706f0 0x7f1070070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.966+0000 7f106e59c700 1 -- 172.21.15.102:0/1938857124 >> v1:172.21.15.102:6789/0 conn(0x7f10700715a0 legacy=0x7f10700c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.966+0000 7f106e59c700 1 -- 172.21.15.102:0/1938857124 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700c71d0 con 0x7f107014f4d0 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.966+0000 7f106d59a700 1 -- 172.21.15.102:0/1938857124 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931484668 0 0) 0x7f10700c7500 con 0x7f10700715a0 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.966+0000 7f106e59c700 1 --2- 172.21.15.102:0/1938857124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f107014f4d0 0x7f1070155150 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f10700c88c0 tx=0x7f1060009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30b005f42616ce4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f106d59a700 1 -- 172.21.15.102:0/1938857124 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060011830 con 0x7f107014f4d0 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f106d59a700 1 -- 172.21.15.102:0/1938857124 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1060011990 con 0x7f107014f4d0 2023-03-03T16:15:13.968 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f106d59a700 1 -- 172.21.15.102:0/1938857124 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060011d20 con 0x7f107014f4d0 2023-03-03T16:15:13.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 -- 172.21.15.102:0/1938857124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f107014f4d0 msgr2=0x7f1070155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:13.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 --2- 172.21.15.102:0/1938857124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f107014f4d0 0x7f1070155150 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f10700c88c0 tx=0x7f1060009b20 comp rx=0 tx=0).stop 2023-03-03T16:15:13.969 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 -- 172.21.15.102:0/1938857124 shutdown_connections 2023-03-03T16:15:13.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 --2- 172.21.15.102:0/1938857124 >> v2:172.21.15.172:3300/0 conn(0x7f10700706f0 0x7f1070070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:13.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 --2- 172.21.15.102:0/1938857124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f107014f4d0 0x7f1070155150 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:13.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 -- 172.21.15.102:0/1938857124 >> 172.21.15.102:0/1938857124 conn(0x7f107006af90 msgr2=0x7f107006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:15:13.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 -- 172.21.15.102:0/1938857124 shutdown_connections 2023-03-03T16:15:13.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.967+0000 7f1074fff700 1 -- 172.21.15.102:0/1938857124 wait complete. 2023-03-03T16:15:13.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.970+0000 7f1074fff700 1 Processor -- start 2023-03-03T16:15:13.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.971+0000 7f1074fff700 1 -- start start 2023-03-03T16:15:13.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f1074fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10700706f0 0x7f107014a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:13.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f1074fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10700715a0 0x7f10700c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:13.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f1074fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f107014f4d0 0x7f107013d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:13.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f1074fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1070154b10 con 0x7f10700706f0 2023-03-03T16:15:13.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f1074fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1070146a90 con 0x7f107014f4d0 2023-03-03T16:15:13.973 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f1074fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1070155980 con 0x7f10700715a0 2023-03-03T16:15:13.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f106e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10700706f0 0x7f107014a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:13.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f106e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10700706f0 0x7f107014a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58062/0 (socket says 172.21.15.102:58062) 2023-03-03T16:15:13.974 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.972+0000 7f106e59c700 1 -- 172.21.15.102:0/2918311836 learned_addr learned my addr 172.21.15.102:0/2918311836 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:15:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106ed9d700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f107014f4d0 0x7f107013d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106dd9b700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10700715a0 0x7f10700c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106e59c700 1 -- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10700715a0 msgr2=0x7f10700c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106e59c700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10700715a0 0x7f10700c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106e59c700 1 -- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f107014f4d0 msgr2=0x7f107013d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106e59c700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f107014f4d0 0x7f107013d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106e59c700 1 -- 172.21.15.102:0/2918311836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600094a0 con 0x7f10700706f0 2023-03-03T16:15:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106dd9b700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10700715a0 0x7f10700c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:15:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f106e59c700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10700706f0 0x7f107014a7c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f10600090b0 tx=0x7f1060004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:15:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.973+0000 7f105effd700 1 -- 172.21.15.102:0/2918311836 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600036a0 con 0x7f10700706f0 2023-03-03T16:15:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.974+0000 7f105effd700 1 -- 172.21.15.102:0/2918311836 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106001e070 con 0x7f10700706f0 2023-03-03T16:15:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.974+0000 7f105effd700 1 -- 172.21.15.102:0/2918311836 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060003a20 con 0x7f10700706f0 2023-03-03T16:15:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.974+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10700c5390 con 0x7f10700706f0 2023-03-03T16:15:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.974+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107014f8a0 con 0x7f10700706f0 2023-03-03T16:15:13.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.975+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1070071c50 con 0x7f10700706f0 2023-03-03T16:15:13.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.976+0000 7f105effd700 1 -- 172.21.15.102:0/2918311836 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f106000f050 con 0x7f10700706f0 2023-03-03T16:15:13.977 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.977+0000 7f105effd700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1050068320 0x7f105006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:13.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.977+0000 7f105effd700 1 -- 172.21.15.102:0/2918311836 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1060086e30 con 0x7f10700706f0 2023-03-03T16:15:13.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.977+0000 7f106dd9b700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1050068320 0x7f105006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:13.978 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.978+0000 7f106dd9b700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1050068320 0x7f105006a7d0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f107014cb80 tx=0x7f1064007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:15:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:13.982+0000 7f105effd700 1 -- 172.21.15.102:0/2918311836 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1060053540 con 0x7f10700706f0 2023-03-03T16:15:14.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.214+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f107014ad00 con 0x7f10700706f0 2023-03-03T16:15:14.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.215+0000 7f105effd700 1 -- 172.21.15.102:0/2918311836 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1060020090 con 0x7f10700706f0 2023-03-03T16:15:14.215 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:15:14.216 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:15:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1050068320 msgr2=0x7f105006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1050068320 0x7f105006a7d0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f107014cb80 tx=0x7f1064007ca0 comp rx=0 tx=0).stop 2023-03-03T16:15:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10700706f0 msgr2=0x7f107014a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10700706f0 0x7f107014a7c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f10600090b0 tx=0x7f1060004750 comp rx=0 tx=0).stop 2023-03-03T16:15:14.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 shutdown_connections 2023-03-03T16:15:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f107014f4d0 0x7f107013d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10700706f0 0x7f107014a7c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1050068320 0x7f105006a7d0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 --2- 172.21.15.102:0/2918311836 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f10700715a0 0x7f10700c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 >> 172.21.15.102:0/2918311836 conn(0x7f107006af90 msgr2=0x7f10700c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:15:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 shutdown_connections 2023-03-03T16:15:14.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:15:14.218+0000 7f1074fff700 1 -- 172.21.15.102:0/2918311836 wait complete. 2023-03-03T16:15:14.220 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:15:14.245 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 119 secs before thrashing 2023-03-03T16:15:17.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:17.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:47.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:47.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:17.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:17.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:48.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:48.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:13.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:17:13.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.513+0000 7f49345ff700 1 Processor -- start 2023-03-03T16:17:13.515 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.514+0000 7f49345ff700 1 -- start start 2023-03-03T16:17:13.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f49345ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c153290 0x7f492c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:13.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f49345ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c153660 con 0x7f492c0cd8e0 2023-03-03T16:17:13.516 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f49345ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c06c360 con 0x7f492c1593c0 2023-03-03T16:17:13.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f49345ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0cec40 con 0x7f492c153290 2023-03-03T16:17:13.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f4932b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f492c0cd8e0 0x7f492c0ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36116/0 (socket says 172.21.15.102:36116) 2023-03-03T16:17:13.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f4932b9c700 1 -- 172.21.15.102:0/3138838523 learned_addr learned my addr 172.21.15.102:0/3138838523 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:17:13.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f493239b700 1 --2- 172.21.15.102:0/3138838523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c153290 0x7f492c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:13.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.515+0000 7f4931b9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f492c1593c0 0x7f492c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57768/0 (socket says 172.21.15.102:57768) 2023-03-03T16:17:13.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f4931399700 1 -- 172.21.15.102:0/3138838523 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896984833 0 0) 0x7f492c153660 con 0x7f492c0cd8e0 2023-03-03T16:17:13.518 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f4931399700 1 -- 172.21.15.102:0/3138838523 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49180035b0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f4931399700 1 -- 172.21.15.102:0/3138838523 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291779094 0 0) 0x7f492c06c360 con 0x7f492c1593c0 2023-03-03T16:17:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f4931399700 1 -- 172.21.15.102:0/3138838523 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492c153660 con 0x7f492c1593c0 2023-03-03T16:17:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f493239b700 1 -- 172.21.15.102:0/3138838523 >> v1:172.21.15.172:6789/0 conn(0x7f492c1593c0 legacy=0x7f492c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f493239b700 1 -- 172.21.15.102:0/3138838523 >> v1:172.21.15.102:6789/0 conn(0x7f492c0cd8e0 legacy=0x7f492c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f493239b700 1 -- 172.21.15.102:0/3138838523 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c15cea0 con 0x7f492c153290 2023-03-03T16:17:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.516+0000 7f493239b700 1 --2- 172.21.15.102:0/3138838523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c153290 0x7f492c158e80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4928009870 tx=0x7f4928009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef4bade5e5e0be14 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f4931399700 1 -- 172.21.15.102:0/3138838523 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928011830 con 0x7f492c153290 2023-03-03T16:17:13.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f4931399700 1 -- 172.21.15.102:0/3138838523 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4928011990 con 0x7f492c153290 2023-03-03T16:17:13.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f4931399700 1 -- 172.21.15.102:0/3138838523 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928011d20 con 0x7f492c153290 2023-03-03T16:17:13.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f49345ff700 1 -- 172.21.15.102:0/3138838523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c153290 msgr2=0x7f492c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:13.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f49345ff700 1 --2- 172.21.15.102:0/3138838523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c153290 0x7f492c158e80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4928009870 tx=0x7f4928009b80 comp rx=0 tx=0).stop 2023-03-03T16:17:13.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f49345ff700 1 -- 172.21.15.102:0/3138838523 shutdown_connections 2023-03-03T16:17:13.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f49345ff700 1 --2- 172.21.15.102:0/3138838523 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c153290 0x7f492c158e80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:13.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.517+0000 7f49345ff700 1 -- 172.21.15.102:0/3138838523 >> 172.21.15.102:0/3138838523 conn(0x7f492c065fb0 msgr2=0x7f492c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:17:13.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.518+0000 7f49345ff700 1 -- 172.21.15.102:0/3138838523 shutdown_connections 2023-03-03T16:17:13.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.518+0000 7f49345ff700 1 -- 172.21.15.102:0/3138838523 wait complete. 2023-03-03T16:17:13.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.519+0000 7f49345ff700 1 Processor -- start 2023-03-03T16:17:13.522 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.520+0000 7f49345ff700 1 -- start start 2023-03-03T16:17:13.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f49345ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f492c0cd8e0 0x7f492c0b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:13.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f49345ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f492c153290 0x7f492c0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:13.523 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f49345ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c1593c0 0x7f492c1f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f49345ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0b3b10 con 0x7f492c0cd8e0 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f49345ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c1f23a0 con 0x7f492c153290 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f49345ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0b3d60 con 0x7f492c1593c0 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f492c0cd8e0 0x7f492c0b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f492c0cd8e0 0x7f492c0b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47224/0 (socket says 172.21.15.102:47224) 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 -- 172.21.15.102:0/3977639902 learned_addr learned my addr 172.21.15.102:0/3977639902 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f4932b9c700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c1593c0 0x7f492c1f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f4931b9a700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f492c153290 0x7f492c0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 -- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c1593c0 msgr2=0x7f492c1f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c1593c0 0x7f492c1f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 -- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f492c153290 msgr2=0x7f492c0c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f492c153290 0x7f492c0c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.521+0000 7f493239b700 1 -- 172.21.15.102:0/3977639902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280094a0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.522+0000 7f493239b700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f492c0cd8e0 0x7f492c0b1ee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4928014040 tx=0x7f4928004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.522+0000 7f4922ffd700 1 -- 172.21.15.102:0/3977639902 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492801a6e0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.522+0000 7f4922ffd700 1 -- 172.21.15.102:0/3977639902 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492801a840 con 0x7f492c0cd8e0 2023-03-03T16:17:13.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.522+0000 7f4922ffd700 1 -- 172.21.15.102:0/3977639902 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492801abb0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.522+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c15d160 con 0x7f492c0cd8e0 2023-03-03T16:17:13.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.522+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c06c7f0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.523+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c0ccde0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.524+0000 7f4922ffd700 1 -- 172.21.15.102:0/3977639902 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f492800f050 con 0x7f492c0cd8e0 2023-03-03T16:17:13.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.526+0000 7f4922ffd700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49100682f0 0x7f491006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:13.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.526+0000 7f4922ffd700 1 -- 172.21.15.102:0/3977639902 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4928086a30 con 0x7f492c0cd8e0 2023-03-03T16:17:13.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.526+0000 7f4931b9a700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49100682f0 0x7f491006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:13.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.527+0000 7f4931b9a700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49100682f0 0x7f491006a7a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f491c007fd0 tx=0x7f491c007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:13.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.530+0000 7f4922ffd700 1 -- 172.21.15.102:0/3977639902 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49280531f0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.765 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.764+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f492c1f26f0 con 0x7f492c0cd8e0 2023-03-03T16:17:13.766 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.765+0000 7f4922ffd700 1 -- 172.21.15.102:0/3977639902 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4928029090 con 0x7f492c0cd8e0 2023-03-03T16:17:13.766 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:17:13.767 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:17:13.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49100682f0 msgr2=0x7f491006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:13.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49100682f0 0x7f491006a7a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f491c007fd0 tx=0x7f491c007d80 comp rx=0 tx=0).stop 2023-03-03T16:17:13.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f492c0cd8e0 msgr2=0x7f492c0b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:13.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f492c0cd8e0 0x7f492c0b1ee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4928014040 tx=0x7f4928004750 comp rx=0 tx=0).stop 2023-03-03T16:17:13.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 shutdown_connections 2023-03-03T16:17:13.769 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f49100682f0 0x7f491006a7a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:13.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f492c1593c0 0x7f492c1f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:13.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f492c0cd8e0 0x7f492c0b1ee0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:13.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 --2- 172.21.15.102:0/3977639902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f492c153290 0x7f492c0c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:13.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 >> 172.21.15.102:0/3977639902 conn(0x7f492c065fb0 msgr2=0x7f492c0cc120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:17:13.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 shutdown_connections 2023-03-03T16:17:13.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:17:13.767+0000 7f49345ff700 1 -- 172.21.15.102:0/3977639902 wait complete. 2023-03-03T16:17:13.771 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:17:13.795 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2023-03-03T16:17:18.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:18.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:48.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:48.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:18.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:18.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:47.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:18:48.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.051+0000 7f80d13ff700 1 Processor -- start 2023-03-03T16:18:48.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.051+0000 7f80d13ff700 1 -- start start 2023-03-03T16:18:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.052+0000 7f80d13ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f80cc153290 0x7f80cc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.052+0000 7f80d13ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.052+0000 7f80d13ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc153660 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.052+0000 7f80d13ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80cc06c360 con 0x7f80cc153290 2023-03-03T16:18:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.052+0000 7f80d13ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc15cea0 con 0x7f80cc1593c0 2023-03-03T16:18:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80ca7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80caffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f80cc153290 0x7f80cc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80cb7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f80cc0cd8e0 0x7f80cc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60534/0 (socket says 172.21.15.102:60534) 2023-03-03T16:18:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80cb7fe700 1 -- 172.21.15.102:0/3580081769 learned_addr learned my addr 172.21.15.102:0/3580081769 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:18:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80caffd700 1 -- 172.21.15.102:0/3580081769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 msgr2=0x7f80cc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:18:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80caffd700 1 --2- 172.21.15.102:0/3580081769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80caffd700 1 -- 172.21.15.102:0/3580081769 >> v1:172.21.15.172:6789/0 conn(0x7f80cc0cd8e0 legacy=0x7f80cc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80caffd700 1 -- 172.21.15.102:0/3580081769 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc06c7f0 con 0x7f80cc153290 2023-03-03T16:18:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.053+0000 7f80caffd700 1 --2- 172.21.15.102:0/3580081769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc153290 0x7f80cc158e80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009840 tx=0x7f80bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e79e92fb19365fa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:18:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80c9ffb700 1 -- 172.21.15.102:0/3580081769 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc011830 con 0x7f80cc153290 2023-03-03T16:18:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80c9ffb700 1 -- 172.21.15.102:0/3580081769 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc011990 con 0x7f80cc153290 2023-03-03T16:18:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80c9ffb700 1 -- 172.21.15.102:0/3580081769 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc011d20 con 0x7f80cc153290 2023-03-03T16:18:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80d13ff700 1 -- 172.21.15.102:0/3580081769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc153290 msgr2=0x7f80cc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3580081769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc153290 0x7f80cc158e80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009840 tx=0x7f80bc009b50 comp rx=0 tx=0).stop 2023-03-03T16:18:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80d13ff700 1 -- 172.21.15.102:0/3580081769 shutdown_connections 2023-03-03T16:18:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3580081769 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc153290 0x7f80cc158e80 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3580081769 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80d13ff700 1 -- 172.21.15.102:0/3580081769 >> 172.21.15.102:0/3580081769 conn(0x7f80cc065fb0 msgr2=0x7f80cc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:18:48.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.054+0000 7f80d13ff700 1 -- 172.21.15.102:0/3580081769 shutdown_connections 2023-03-03T16:18:48.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.055+0000 7f80d13ff700 1 -- 172.21.15.102:0/3580081769 wait complete. 2023-03-03T16:18:48.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.056+0000 7f80d13ff700 1 Processor -- start 2023-03-03T16:18:48.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.057+0000 7f80d13ff700 1 -- start start 2023-03-03T16:18:48.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80d13ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc0cd8e0 0x7f80cc151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:48.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80d13ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80cc153290 0x7f80cc1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:48.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80d13ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc1ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:48.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80d13ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc1f3090 con 0x7f80cc153290 2023-03-03T16:18:48.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80d13ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc1ef630 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80d13ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc1f2b30 con 0x7f80cc1593c0 2023-03-03T16:18:48.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc0cd8e0 0x7f80cc151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:48.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80ca7fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80cc153290 0x7f80cc1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:48.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc0cd8e0 0x7f80cc151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:48992/0 (socket says 172.21.15.102:48992) 2023-03-03T16:18:48.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 -- 172.21.15.102:0/3231891903 learned_addr learned my addr 172.21.15.102:0/3231891903 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:18:48.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80cb7fe700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc1ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:48.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 -- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 msgr2=0x7f80cc1ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:48.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc1ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 -- 172.21.15.102:0/3231891903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80cc153290 msgr2=0x7f80cc1eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:48.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80cc153290 0x7f80cc1eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.058+0000 7f80caffd700 1 -- 172.21.15.102:0/3231891903 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc0094a0 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.059+0000 7f80caffd700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc0cd8e0 0x7f80cc151130 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0090b0 tx=0x7f80bc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:18:48.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.059+0000 7f80b37fe700 1 -- 172.21.15.102:0/3231891903 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc0036a0 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.059+0000 7f80b37fe700 1 -- 172.21.15.102:0/3231891903 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc01f070 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.059+0000 7f80b37fe700 1 -- 172.21.15.102:0/3231891903 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc003a40 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.059+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc0ccde0 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.059+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc15d160 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.060+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc0cd0c0 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.061+0000 7f80b37fe700 1 -- 172.21.15.102:0/3231891903 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80bc00f050 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.063+0000 7f80b37fe700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b80682c0 0x7f80b806a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:48.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.063+0000 7f80b37fe700 1 -- 172.21.15.102:0/3231891903 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f80bc021070 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.063+0000 7f80ca7fc700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b80682c0 0x7f80b806a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:48.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.064+0000 7f80ca7fc700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b80682c0 0x7f80b806a770 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0b4d70 tx=0x7f80c0009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:18:48.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.067+0000 7f80b37fe700 1 -- 172.21.15.102:0/3231891903 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80bc053d60 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.302+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80cc06d430 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.304+0000 7f80b37fe700 1 -- 172.21.15.102:0/3231891903 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f80bc057380 con 0x7f80cc0cd8e0 2023-03-03T16:18:48.304 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:18:48.304 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:18:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b80682c0 msgr2=0x7f80b806a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b80682c0 0x7f80b806a770 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0b4d70 tx=0x7f80c0009210 comp rx=0 tx=0).stop 2023-03-03T16:18:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc0cd8e0 msgr2=0x7f80cc151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc0cd8e0 0x7f80cc151130 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0090b0 tx=0x7f80bc004750 comp rx=0 tx=0).stop 2023-03-03T16:18:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 shutdown_connections 2023-03-03T16:18:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f80b80682c0 0x7f80b806a770 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f80cc1593c0 0x7f80cc1ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80cc0cd8e0 0x7f80cc151130 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 --2- 172.21.15.102:0/3231891903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80cc153290 0x7f80cc1eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 >> 172.21.15.102:0/3231891903 conn(0x7f80cc065fb0 msgr2=0x7f80cc0cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:18:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.305+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 shutdown_connections 2023-03-03T16:18:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:18:48.306+0000 7f80d13ff700 1 -- 172.21.15.102:0/3231891903 wait complete. 2023-03-03T16:18:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:18:48.333 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T16:18:48.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:48.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:19.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:19.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:49.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:49.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:00.000 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T16:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:20:19.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:19.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:20.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:20:20.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.585+0000 7fb3d0d9d700 1 Processor -- start 2023-03-03T16:20:20.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.585+0000 7fb3d0d9d700 1 -- start start 2023-03-03T16:20:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.587+0000 7fb3d0d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3cc0ced00 0x7fb3cc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3d0d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3cc0c3d20 con 0x7fb3cc0c41b0 2023-03-03T16:20:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3d0d9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3cc15cba0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3d0d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3cc0c4930 con 0x7fb3cc0ced00 2023-03-03T16:20:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3cb7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb3cc0ced00 0x7fb3cc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3caffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb3cc0b3970 0x7fb3cc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50340/0 (socket says 172.21.15.102:50340) 2023-03-03T16:20:20.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3caffd700 1 -- 172.21.15.102:0/2032329579 learned_addr learned my addr 172.21.15.102:0/2032329579 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:20:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3ca7fc700 1 -- 172.21.15.102:0/2032329579 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807995427 0 0) 0x7fb3cc15cba0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3ca7fc700 1 -- 172.21.15.102:0/2032329579 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b40035b0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3ca7fc700 1 -- 172.21.15.102:0/2032329579 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185973856 0 0) 0x7fb3cc0c3d20 con 0x7fb3cc0c41b0 2023-03-03T16:20:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3ca7fc700 1 -- 172.21.15.102:0/2032329579 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3cc15cba0 con 0x7fb3cc0c41b0 2023-03-03T16:20:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3cb7fe700 1 -- 172.21.15.102:0/2032329579 >> v1:172.21.15.172:6789/0 conn(0x7fb3cc0b3970 legacy=0x7fb3cc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3cb7fe700 1 -- 172.21.15.102:0/2032329579 >> v1:172.21.15.102:6789/0 conn(0x7fb3cc0c41b0 legacy=0x7fb3cc163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.588+0000 7fb3cb7fe700 1 -- 172.21.15.102:0/2032329579 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc15ce80 con 0x7fb3cc0ced00 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.589+0000 7fb3cb7fe700 1 --2- 172.21.15.102:0/2032329579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 0x7fb3cc0c5a90 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009840 tx=0x7fb3bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a061438ca1b11dd0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.589+0000 7fb3ca7fc700 1 -- 172.21.15.102:0/2032329579 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc011830 con 0x7fb3cc0ced00 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.589+0000 7fb3ca7fc700 1 -- 172.21.15.102:0/2032329579 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc011990 con 0x7fb3cc0ced00 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.589+0000 7fb3ca7fc700 1 -- 172.21.15.102:0/2032329579 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc011d20 con 0x7fb3cc0ced00 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.590+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/2032329579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 msgr2=0x7fb3cc0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.590+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/2032329579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 0x7fb3cc0c5a90 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009840 tx=0x7fb3bc009b50 comp rx=0 tx=0).stop 2023-03-03T16:20:20.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.590+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/2032329579 shutdown_connections 2023-03-03T16:20:20.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.590+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/2032329579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 0x7fb3cc0c5a90 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:20.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.590+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/2032329579 >> 172.21.15.102:0/2032329579 conn(0x7fb3cc08b1c0 msgr2=0x7fb3cc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:20.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.590+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/2032329579 shutdown_connections 2023-03-03T16:20:20.593 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.590+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/2032329579 wait complete. 2023-03-03T16:20:20.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.592+0000 7fb3d0d9d700 1 Processor -- start 2023-03-03T16:20:20.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.592+0000 7fb3d0d9d700 1 -- start start 2023-03-03T16:20:20.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3d0d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3cc0b3970 0x7fb3cc14d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:20.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3d0d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc0c41b0 0x7fb3cc1ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:20.594 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3d0d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 0x7fb3cc1e47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3d0d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc1ee3f0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3d0d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc0c7800 con 0x7fb3cc0ced00 2023-03-03T16:20:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3d0d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc14dc10 con 0x7fb3cc0c41b0 2023-03-03T16:20:20.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3cb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3cc0b3970 0x7fb3cc14d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:20.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3caffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc0c41b0 0x7fb3cc1ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:20.596 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3cb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3cc0b3970 0x7fb3cc14d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52138/0 (socket says 172.21.15.102:52138) 2023-03-03T16:20:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3cb7fe700 1 -- 172.21.15.102:0/3941879308 learned_addr learned my addr 172.21.15.102:0/3941879308 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:20:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.593+0000 7fb3cbfff700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 0x7fb3cc1e47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3cb7fe700 1 -- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc0c41b0 msgr2=0x7fb3cc1ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3cb7fe700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc0c41b0 0x7fb3cc1ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3cb7fe700 1 -- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 msgr2=0x7fb3cc1e47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3cb7fe700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 0x7fb3cc1e47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3cb7fe700 1 -- 172.21.15.102:0/3941879308 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0094a0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3caffd700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc0c41b0 0x7fb3cc1ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:20:20.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3cb7fe700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3cc0b3970 0x7fb3cc14d6d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0090b0 tx=0x7fb3bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:20.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3b3fff700 1 -- 172.21.15.102:0/3941879308 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc01a6e0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3b3fff700 1 -- 172.21.15.102:0/3941879308 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc01a840 con 0x7fb3cc0b3970 2023-03-03T16:20:20.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3b3fff700 1 -- 172.21.15.102:0/3941879308 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc01abb0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.594+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3cc06c200 con 0x7fb3cc0b3970 2023-03-03T16:20:20.598 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.595+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3cc06bf20 con 0x7fb3cc0b3970 2023-03-03T16:20:20.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.596+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3cc15c890 con 0x7fb3cc0b3970 2023-03-03T16:20:20.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.597+0000 7fb3b3fff700 1 -- 172.21.15.102:0/3941879308 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb3bc02b030 con 0x7fb3cc0b3970 2023-03-03T16:20:20.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.598+0000 7fb3b3fff700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3b80682f0 0x7fb3b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:20.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.598+0000 7fb3b3fff700 1 -- 172.21.15.102:0/3941879308 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb3bc086e20 con 0x7fb3cc0b3970 2023-03-03T16:20:20.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.599+0000 7fb3caffd700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3b80682f0 0x7fb3b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:20.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.599+0000 7fb3caffd700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3b80682f0 0x7fb3b806a7a0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0c9380 tx=0x7fb3c0007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.602+0000 7fb3b3fff700 1 -- 172.21.15.102:0/3941879308 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb3bc0535e0 con 0x7fb3cc0b3970 2023-03-03T16:20:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.837+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3cc1ee660 con 0x7fb3cc0b3970 2023-03-03T16:20:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.838+0000 7fb3b3fff700 1 -- 172.21.15.102:0/3941879308 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb3bc056c00 con 0x7fb3cc0b3970 2023-03-03T16:20:20.840 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:20:20.840 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:20:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.840+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3b80682f0 msgr2=0x7fb3b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.840+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3b80682f0 0x7fb3b806a7a0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0c9380 tx=0x7fb3c0007e00 comp rx=0 tx=0).stop 2023-03-03T16:20:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3cc0b3970 msgr2=0x7fb3cc14d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3cc0b3970 0x7fb3cc14d6d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0090b0 tx=0x7fb3bc004750 comp rx=0 tx=0).stop 2023-03-03T16:20:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 shutdown_connections 2023-03-03T16:20:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3cc0ced00 0x7fb3cc1e47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3cc0b3970 0x7fb3cc14d6d0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb3b80682f0 0x7fb3b806a7a0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 --2- 172.21.15.102:0/3941879308 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb3cc0c41b0 0x7fb3cc1ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 >> 172.21.15.102:0/3941879308 conn(0x7fb3cc08b1c0 msgr2=0x7fb3cc151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 shutdown_connections 2023-03-03T16:20:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:20:20.841+0000 7fb3d0d9d700 1 -- 172.21.15.102:0/3941879308 wait complete. 2023-03-03T16:20:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:20:20.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2023-03-03T16:20:49.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:49.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:04.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:21:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.133+0000 7f924339d700 1 Processor -- start 2023-03-03T16:21:05.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.133+0000 7f924339d700 1 -- start start 2023-03-03T16:21:05.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f924339d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f923c0ced00 0x7f923c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:05.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f924339d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0b3970 0x7f923c0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:05.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f924339d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f923c0c3d20 con 0x7f923c0cf0d0 2023-03-03T16:21:05.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f924339d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f923c0c4930 con 0x7f923c0ced00 2023-03-03T16:21:05.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f924339d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923c15cba0 con 0x7f923c0b3970 2023-03-03T16:21:05.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f924239b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f923c0ced00 0x7f923c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:05.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f9241b9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0b3970 0x7f923c0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:05.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f9242b9c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f923c0cf0d0 0x7f923c158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40458/0 (socket says 172.21.15.102:40458) 2023-03-03T16:21:05.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.135+0000 7f9242b9c700 1 -- 172.21.15.102:0/140696955 learned_addr learned my addr 172.21.15.102:0/140696955 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:21:05.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f9241b9a700 1 -- 172.21.15.102:0/140696955 >> v2:172.21.15.172:3300/0 conn(0x7f923c0ced00 msgr2=0x7f923c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:21:05.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f9241b9a700 1 --2- 172.21.15.102:0/140696955 >> v2:172.21.15.172:3300/0 conn(0x7f923c0ced00 0x7f923c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f9241b9a700 1 -- 172.21.15.102:0/140696955 >> v1:172.21.15.102:6789/0 conn(0x7f923c0cf0d0 legacy=0x7f923c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f9241b9a700 1 -- 172.21.15.102:0/140696955 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c15ce80 con 0x7f923c0b3970 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f924239b700 1 --2- 172.21.15.102:0/140696955 >> v2:172.21.15.172:3300/0 conn(0x7f923c0ced00 0x7f923c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f9241399700 1 -- 172.21.15.102:0/140696955 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916731536 0 0) 0x7f923c0c3d20 con 0x7f923c0cf0d0 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f9241b9a700 1 --2- 172.21.15.102:0/140696955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0b3970 0x7f923c0c41b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f922c009840 tx=0x7f922c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54addefff360d375 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.136+0000 7f9241399700 1 -- 172.21.15.102:0/140696955 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c011830 con 0x7f923c0b3970 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f9241399700 1 -- 172.21.15.102:0/140696955 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c011990 con 0x7f923c0b3970 2023-03-03T16:21:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f9241399700 1 -- 172.21.15.102:0/140696955 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c011d20 con 0x7f923c0b3970 2023-03-03T16:21:05.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 -- 172.21.15.102:0/140696955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0b3970 msgr2=0x7f923c0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:05.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 --2- 172.21.15.102:0/140696955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0b3970 0x7f923c0c41b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f922c009840 tx=0x7f922c009b50 comp rx=0 tx=0).stop 2023-03-03T16:21:05.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 -- 172.21.15.102:0/140696955 shutdown_connections 2023-03-03T16:21:05.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 --2- 172.21.15.102:0/140696955 >> v2:172.21.15.172:3300/0 conn(0x7f923c0ced00 0x7f923c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 --2- 172.21.15.102:0/140696955 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0b3970 0x7f923c0c41b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 -- 172.21.15.102:0/140696955 >> 172.21.15.102:0/140696955 conn(0x7f923c08b1c0 msgr2=0x7f923c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:05.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 -- 172.21.15.102:0/140696955 shutdown_connections 2023-03-03T16:21:05.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.137+0000 7f924339d700 1 -- 172.21.15.102:0/140696955 wait complete. 2023-03-03T16:21:05.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.140+0000 7f924339d700 1 Processor -- start 2023-03-03T16:21:05.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.140+0000 7f924339d700 1 -- start start 2023-03-03T16:21:05.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f924339d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f923c0b3970 0x7f923c159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:05.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f924339d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0ced00 0x7f923c1e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:05.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f924339d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f923c0cf0d0 0x7f923c14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:05.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f924339d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c15a200 con 0x7f923c0cf0d0 2023-03-03T16:21:05.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f924339d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c0cd400 con 0x7f923c0b3970 2023-03-03T16:21:05.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f924339d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923c15a420 con 0x7f923c0ced00 2023-03-03T16:21:05.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f9242b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f923c0cf0d0 0x7f923c14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:05.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f924239b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f923c0b3970 0x7f923c159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:05.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f9242b9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f923c0cf0d0 0x7f923c14e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46370/0 (socket says 172.21.15.102:46370) 2023-03-03T16:21:05.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f9242b9c700 1 -- 172.21.15.102:0/1221731305 learned_addr learned my addr 172.21.15.102:0/1221731305 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:21:05.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.142+0000 7f9241b9a700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0ced00 0x7f923c1e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:05.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9241b9a700 1 -- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f923c0b3970 msgr2=0x7f923c159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:05.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9241b9a700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f923c0b3970 0x7f923c159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9241b9a700 1 -- 172.21.15.102:0/1221731305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f923c0cf0d0 msgr2=0x7f923c14e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9241b9a700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f923c0cf0d0 0x7f923c14e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9241b9a700 1 -- 172.21.15.102:0/1221731305 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92380094a0 con 0x7f923c0ced00 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9241b9a700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0ced00 0x7f923c1e4480 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f922c016000 tx=0x7f922c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9232ffd700 1 -- 172.21.15.102:0/1221731305 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c0036a0 con 0x7f923c0ced00 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9232ffd700 1 -- 172.21.15.102:0/1221731305 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c01e070 con 0x7f923c0ced00 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f9232ffd700 1 -- 172.21.15.102:0/1221731305 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c003a60 con 0x7f923c0ced00 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.143+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c0094a0 con 0x7f923c0ced00 2023-03-03T16:21:05.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.144+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923c06c200 con 0x7f923c0ced00 2023-03-03T16:21:05.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.145+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923c06bf20 con 0x7f923c0ced00 2023-03-03T16:21:05.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.146+0000 7f9232ffd700 1 -- 172.21.15.102:0/1221731305 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f922c00f050 con 0x7f923c0ced00 2023-03-03T16:21:05.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.147+0000 7f9232ffd700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92200682f0 0x7f922006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:05.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.147+0000 7f9232ffd700 1 -- 172.21.15.102:0/1221731305 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f922c086da0 con 0x7f923c0ced00 2023-03-03T16:21:05.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.147+0000 7f924239b700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92200682f0 0x7f922006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:05.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.148+0000 7f924239b700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92200682f0 0x7f922006a7a0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f9238009d70 tx=0x7f9238009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:05.153 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.152+0000 7f9232ffd700 1 -- 172.21.15.102:0/1221731305 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f922c053430 con 0x7f923c0ced00 2023-03-03T16:21:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.387+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f923c0cf540 con 0x7f923c0ced00 2023-03-03T16:21:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.388+0000 7f9232ffd700 1 -- 172.21.15.102:0/1221731305 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f922c020090 con 0x7f923c0ced00 2023-03-03T16:21:05.390 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:21:05.390 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:21:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92200682f0 msgr2=0x7f922006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92200682f0 0x7f922006a7a0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f9238009d70 tx=0x7f9238009210 comp rx=0 tx=0).stop 2023-03-03T16:21:05.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0ced00 msgr2=0x7f923c1e4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0ced00 0x7f923c1e4480 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f922c016000 tx=0x7f922c004750 comp rx=0 tx=0).stop 2023-03-03T16:21:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 shutdown_connections 2023-03-03T16:21:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f923c0cf0d0 0x7f923c14e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f923c0b3970 0x7f923c159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f92200682f0 0x7f922006a7a0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 --2- 172.21.15.102:0/1221731305 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f923c0ced00 0x7f923c1e4480 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:05.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 >> 172.21.15.102:0/1221731305 conn(0x7f923c08b1c0 msgr2=0x7f923c0669c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 shutdown_connections 2023-03-03T16:21:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:05.391+0000 7f924339d700 1 -- 172.21.15.102:0/1221731305 wait complete. 2023-03-03T16:21:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:21:05.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 16 secs before thrashing 2023-03-03T16:21:19.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:19.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:21.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:21:21.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.672+0000 7feda23ff700 1 Processor -- start 2023-03-03T16:21:21.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.672+0000 7feda23ff700 1 -- start start 2023-03-03T16:21:21.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.673+0000 7feda23ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c071ef0 0x7fed9c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.673+0000 7feda23ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed9c0722c0 0x7fed9c15cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.673+0000 7feda23ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c159a80 con 0x7fed9c06a390 2023-03-03T16:21:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.673+0000 7feda23ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9c0cc710 con 0x7fed9c0722c0 2023-03-03T16:21:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.673+0000 7feda23ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c15d1d0 con 0x7fed9c071ef0 2023-03-03T16:21:21.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7fed9b7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed9c0722c0 0x7fed9c15cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:21.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7feda099c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fed9c06a390 0x7fed9c06f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:55852/0 (socket says 172.21.15.102:55852) 2023-03-03T16:21:21.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7feda099c700 1 -- 172.21.15.102:0/2559240117 learned_addr learned my addr 172.21.15.102:0/2559240117 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:21:21.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7fed9bfff700 1 --2- 172.21.15.102:0/2559240117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c071ef0 0x7fed9c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:21.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7fed9b7fe700 1 -- 172.21.15.102:0/2559240117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c071ef0 msgr2=0x7fed9c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:21:21.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7fed9affd700 1 -- 172.21.15.102:0/2559240117 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458335595 0 0) 0x7fed9c159a80 con 0x7fed9c06a390 2023-03-03T16:21:21.676 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7fed9b7fe700 1 --2- 172.21.15.102:0/2559240117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c071ef0 0x7fed9c0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7fed9b7fe700 1 -- 172.21.15.102:0/2559240117 >> v1:172.21.15.172:6789/0 conn(0x7fed9c06a390 legacy=0x7fed9c06f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.674+0000 7fed9b7fe700 1 -- 172.21.15.102:0/2559240117 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c0cc9a0 con 0x7fed9c0722c0 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7fed9b7fe700 1 --2- 172.21.15.102:0/2559240117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c0722c0 0x7fed9c15cc90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fed90009840 tx=0x7fed90009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f385c723599686b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7fed9affd700 1 -- 172.21.15.102:0/2559240117 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed90011830 con 0x7fed9c0722c0 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7fed9affd700 1 -- 172.21.15.102:0/2559240117 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed90011990 con 0x7fed9c0722c0 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7fed9affd700 1 -- 172.21.15.102:0/2559240117 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed90011d20 con 0x7fed9c0722c0 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7feda23ff700 1 -- 172.21.15.102:0/2559240117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c0722c0 msgr2=0x7fed9c15cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:21.677 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7feda23ff700 1 --2- 172.21.15.102:0/2559240117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c0722c0 0x7fed9c15cc90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fed90009840 tx=0x7fed90009b50 comp rx=0 tx=0).stop 2023-03-03T16:21:21.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7feda23ff700 1 -- 172.21.15.102:0/2559240117 shutdown_connections 2023-03-03T16:21:21.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7feda23ff700 1 --2- 172.21.15.102:0/2559240117 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c071ef0 0x7fed9c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7feda23ff700 1 --2- 172.21.15.102:0/2559240117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c0722c0 0x7fed9c15cc90 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.675+0000 7feda23ff700 1 -- 172.21.15.102:0/2559240117 >> 172.21.15.102:0/2559240117 conn(0x7fed9c06c0a0 msgr2=0x7fed9c0cbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:21.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.676+0000 7feda23ff700 1 -- 172.21.15.102:0/2559240117 shutdown_connections 2023-03-03T16:21:21.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.676+0000 7feda23ff700 1 -- 172.21.15.102:0/2559240117 wait complete. 2023-03-03T16:21:21.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.678+0000 7feda23ff700 1 Processor -- start 2023-03-03T16:21:21.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.678+0000 7feda23ff700 1 -- start start 2023-03-03T16:21:21.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda23ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c06a390 0x7fed9c148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:21.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda23ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c071ef0 0x7fed9c1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:21.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda23ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed9c0722c0 0x7fed9c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:21.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda23ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c147630 con 0x7fed9c071ef0 2023-03-03T16:21:21.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda23ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c146640 con 0x7fed9c0722c0 2023-03-03T16:21:21.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda23ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c146360 con 0x7fed9c06a390 2023-03-03T16:21:21.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c071ef0 0x7fed9c1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:21.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c06a390 0x7fed9c148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:21.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c071ef0 0x7fed9c1470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43270/0 (socket says 172.21.15.102:43270) 2023-03-03T16:21:21.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 -- 172.21.15.102:0/1817114121 learned_addr learned my addr 172.21.15.102:0/1817114121 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:21:21.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda099c700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed9c0722c0 0x7fed9c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:21.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 -- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c06a390 msgr2=0x7fed9c148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:21.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c06a390 0x7fed9c148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 -- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed9c0722c0 msgr2=0x7fed9c15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:21.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed9c0722c0 0x7fed9c15bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7fed9b7fe700 1 -- 172.21.15.102:0/1817114121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900094a0 con 0x7fed9c071ef0 2023-03-03T16:21:21.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.679+0000 7feda099c700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed9c0722c0 0x7fed9c15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:21:21.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.680+0000 7fed9b7fe700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c071ef0 0x7fed9c1470f0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fed90014040 tx=0x7fed90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:21.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.680+0000 7fed98ff9700 1 -- 172.21.15.102:0/1817114121 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed900036a0 con 0x7fed9c071ef0 2023-03-03T16:21:21.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.680+0000 7fed98ff9700 1 -- 172.21.15.102:0/1817114121 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9001e070 con 0x7fed9c071ef0 2023-03-03T16:21:21.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.680+0000 7fed98ff9700 1 -- 172.21.15.102:0/1817114121 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed90003a20 con 0x7fed9c071ef0 2023-03-03T16:21:21.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.680+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c159150 con 0x7fed9c071ef0 2023-03-03T16:21:21.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.681+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c158e70 con 0x7fed9c071ef0 2023-03-03T16:21:21.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.682+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c159430 con 0x7fed9c071ef0 2023-03-03T16:21:21.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.682+0000 7fed98ff9700 1 -- 172.21.15.102:0/1817114121 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed9000f050 con 0x7fed9c071ef0 2023-03-03T16:21:21.684 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.684+0000 7fed98ff9700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed880682f0 0x7fed8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.684+0000 7fed98ff9700 1 -- 172.21.15.102:0/1817114121 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fed90086dd0 con 0x7fed9c071ef0 2023-03-03T16:21:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.684+0000 7fed9bfff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed880682f0 0x7fed8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.685+0000 7fed9bfff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed880682f0 0x7fed8806a7a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009be0 tx=0x7fed8c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:21.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.688+0000 7fed98ff9700 1 -- 172.21.15.102:0/1817114121 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed900534e0 con 0x7fed9c071ef0 2023-03-03T16:21:21.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.925+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed9c04e6e0 con 0x7fed9c071ef0 2023-03-03T16:21:21.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.926+0000 7fed98ff9700 1 -- 172.21.15.102:0/1817114121 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fed90020020 con 0x7fed9c071ef0 2023-03-03T16:21:21.927 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:21:21.927 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:21:21.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed880682f0 msgr2=0x7fed8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed880682f0 0x7fed8806a7a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009be0 tx=0x7fed8c009210 comp rx=0 tx=0).stop 2023-03-03T16:21:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c071ef0 msgr2=0x7fed9c1470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c071ef0 0x7fed9c1470f0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fed90014040 tx=0x7fed90004750 comp rx=0 tx=0).stop 2023-03-03T16:21:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 shutdown_connections 2023-03-03T16:21:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fed880682f0 0x7fed8806a7a0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fed9c06a390 0x7fed9c148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed9c0722c0 0x7fed9c15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 --2- 172.21.15.102:0/1817114121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed9c071ef0 0x7fed9c1470f0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:21.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 >> 172.21.15.102:0/1817114121 conn(0x7fed9c06c0a0 msgr2=0x7fed9c14c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:21.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 shutdown_connections 2023-03-03T16:21:21.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:21:21.929+0000 7feda23ff700 1 -- 172.21.15.102:0/1817114121 wait complete. 2023-03-03T16:21:21.931 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:21:21.956 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-03-03T16:21:50.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:50.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:20.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:20.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:42.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:22:43.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.211+0000 7fe3c31ff700 1 Processor -- start 2023-03-03T16:22:43.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.212+0000 7fe3c31ff700 1 -- start start 2023-03-03T16:22:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3c31ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3c31ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe3bc0cd8e0 0x7fe3bc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3c31ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc153660 con 0x7fe3bc1593c0 2023-03-03T16:22:43.214 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3c31ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc06c360 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.215 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3c31ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc15cea0 con 0x7fe3bc153290 2023-03-03T16:22:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3c179c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe3bc0cd8e0 0x7fe3bc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3bbfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe3bc1593c0 0x7fe3bc0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37442/0 (socket says 172.21.15.102:37442) 2023-03-03T16:22:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3c0f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.213+0000 7fe3bbfff700 1 -- 172.21.15.102:0/3966868612 learned_addr learned my addr 172.21.15.102:0/3966868612 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:22:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3c179c700 1 -- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 msgr2=0x7fe3bc158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:22:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3c179c700 1 --2- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3c179c700 1 -- 172.21.15.102:0/3966868612 >> v1:172.21.15.102:6789/0 conn(0x7fe3bc1593c0 legacy=0x7fe3bc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3c179c700 1 -- 172.21.15.102:0/3966868612 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc06c7f0 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3c0f9b700 1 --2- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:22:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3bb7fe700 1 -- 172.21.15.102:0/3966868612 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361461053 0 0) 0x7fe3bc153660 con 0x7fe3bc1593c0 2023-03-03T16:22:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3c179c700 1 --2- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0ce530 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009d90 tx=0x7fe3b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f76e20df60abaf0b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.214+0000 7fe3bb7fe700 1 -- 172.21.15.102:0/3966868612 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400e070 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3bb7fe700 1 -- 172.21.15.102:0/3966868612 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b400bb20 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3bb7fe700 1 -- 172.21.15.102:0/3966868612 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4012430 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 -- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 msgr2=0x7fe3bc0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:43.218 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0ce530 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009d90 tx=0x7fe3b400b330 comp rx=0 tx=0).stop 2023-03-03T16:22:43.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 -- 172.21.15.102:0/3966868612 shutdown_connections 2023-03-03T16:22:43.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0ce530 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/3966868612 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 -- 172.21.15.102:0/3966868612 >> 172.21.15.102:0/3966868612 conn(0x7fe3bc065fb0 msgr2=0x7fe3bc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:43.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 -- 172.21.15.102:0/3966868612 shutdown_connections 2023-03-03T16:22:43.219 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.215+0000 7fe3c31ff700 1 -- 172.21.15.102:0/3966868612 wait complete. 2023-03-03T16:22:43.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.217+0000 7fe3c31ff700 1 Processor -- start 2023-03-03T16:22:43.220 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.217+0000 7fe3c31ff700 1 -- start start 2023-03-03T16:22:43.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.218+0000 7fe3c31ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:43.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.218+0000 7fe3c31ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc1ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:43.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.218+0000 7fe3c31ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3bc1593c0 0x7fe3bc14f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:43.221 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.218+0000 7fe3c31ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0c6b50 con 0x7fe3bc1593c0 2023-03-03T16:22:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.218+0000 7fe3c31ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc156e90 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.218+0000 7fe3c31ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0c6da0 con 0x7fe3bc153290 2023-03-03T16:22:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c179c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3bc1593c0 0x7fe3bc14f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c179c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3bc1593c0 0x7fe3bc14f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34994/0 (socket says 172.21.15.102:34994) 2023-03-03T16:22:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c179c700 1 -- 172.21.15.102:0/1354593335 learned_addr learned my addr 172.21.15.102:0/1354593335 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:22:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c0f9b700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:43.222 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3bbfff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc1ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:43.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c0f9b700 1 -- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 msgr2=0x7fe3bc1ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:43.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c0f9b700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc1ef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c0f9b700 1 -- 172.21.15.102:0/1354593335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3bc1593c0 msgr2=0x7fe3bc14f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:43.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c0f9b700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3bc1593c0 0x7fe3bc14f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c0f9b700 1 -- 172.21.15.102:0/1354593335 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00094a0 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c179c700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3bc1593c0 0x7fe3bc14f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:22:43.223 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.219+0000 7fe3c0f9b700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0c5850 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9b0 tx=0x7fe3b000ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:43.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.220+0000 7fe3b97fa700 1 -- 172.21.15.102:0/1354593335 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000ee50 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.220+0000 7fe3b97fa700 1 -- 172.21.15.102:0/1354593335 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b000c850 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.220+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b4009a70 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.220+0000 7fe3b97fa700 1 -- 172.21.15.102:0/1354593335 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000cc00 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.224 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.220+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc0ccde0 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.221+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc15d160 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.222+0000 7fe3b97fa700 1 -- 172.21.15.102:0/1354593335 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe3b0019030 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.224+0000 7fe3b97fa700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe3a0068320 0x7fe3a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:43.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.224+0000 7fe3b97fa700 1 -- 172.21.15.102:0/1354593335 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe3b0085c00 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.225 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.224+0000 7fe3bbfff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe3a0068320 0x7fe3a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:43.226 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.225+0000 7fe3bbfff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe3a0068320 0x7fe3a006a7d0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0c2d40 tx=0x7fe3a8007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.228+0000 7fe3b97fa700 1 -- 172.21.15.102:0/1354593335 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe3b0052310 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.460+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3bc0cd0c0 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.461+0000 7fe3b97fa700 1 -- 172.21.15.102:0/1354593335 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe3b0055930 con 0x7fe3bc0cd8e0 2023-03-03T16:22:43.462 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:22:43.462 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:22:43.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe3a0068320 msgr2=0x7fe3a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe3a0068320 0x7fe3a006a7d0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0c2d40 tx=0x7fe3a8007de0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 msgr2=0x7fe3bc0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0c5850 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9b0 tx=0x7fe3b000ecc0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 shutdown_connections 2023-03-03T16:22:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3bc1593c0 0x7fe3bc14f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3bc0cd8e0 0x7fe3bc0c5850 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe3a0068320 0x7fe3a006a7d0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 --2- 172.21.15.102:0/1354593335 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe3bc153290 0x7fe3bc1ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 >> 172.21.15.102:0/1354593335 conn(0x7fe3bc065fb0 msgr2=0x7fe3bc0cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.463+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 shutdown_connections 2023-03-03T16:22:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:43.464+0000 7fe3c31ff700 1 -- 172.21.15.102:0/1354593335 wait complete. 2023-03-03T16:22:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:22:43.491 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2023-03-03T16:22:49.492 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:22:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.742+0000 7f302af9d700 1 Processor -- start 2023-03-03T16:22:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.742+0000 7f302af9d700 1 -- start start 2023-03-03T16:22:49.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.743+0000 7f302af9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30240cab50 0x7f30241646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:49.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.743+0000 7f302af9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240c9ca0 con 0x7f30241564b0 2023-03-03T16:22:49.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.743+0000 7f302af9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240b46f0 con 0x7f30240c9610 2023-03-03T16:22:49.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.743+0000 7f302af9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30240b4920 con 0x7f30240cab50 2023-03-03T16:22:49.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.743+0000 7f3029f9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f30241564b0 0x7f30240cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49326/0 (socket says 172.21.15.102:49326) 2023-03-03T16:22:49.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.744+0000 7f3029f9b700 1 -- 172.21.15.102:0/2864789461 learned_addr learned my addr 172.21.15.102:0/2864789461 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:22:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.744+0000 7f302a79c700 1 --2- 172.21.15.102:0/2864789461 >> v2:172.21.15.172:3300/0 conn(0x7f30240cab50 0x7f30241646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.744+0000 7f302a79c700 1 -- 172.21.15.102:0/2864789461 >> v1:172.21.15.172:6789/0 conn(0x7f30240c9610 legacy=0x7f30240ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.744+0000 7f302a79c700 1 -- 172.21.15.102:0/2864789461 >> v1:172.21.15.102:6789/0 conn(0x7f30241564b0 legacy=0x7f30240cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.744+0000 7f3028f99700 1 -- 172.21.15.102:0/2864789461 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327230165 0 0) 0x7f30240b46f0 con 0x7f30240c9610 2023-03-03T16:22:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.744+0000 7f302a79c700 1 -- 172.21.15.102:0/2864789461 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240b4c00 con 0x7f30240cab50 2023-03-03T16:22:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.744+0000 7f302a79c700 1 --2- 172.21.15.102:0/2864789461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30240cab50 0x7f30241646b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f301c009d90 tx=0x7f301c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14152eba25090604 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:49.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f3028f99700 1 -- 172.21.15.102:0/2864789461 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00e070 con 0x7f30240cab50 2023-03-03T16:22:49.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f3028f99700 1 -- 172.21.15.102:0/2864789461 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c00bad0 con 0x7f30240cab50 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f3028f99700 1 -- 172.21.15.102:0/2864789461 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00be80 con 0x7f30240cab50 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f302af9d700 1 -- 172.21.15.102:0/2864789461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30240cab50 msgr2=0x7f30241646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f302af9d700 1 --2- 172.21.15.102:0/2864789461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30240cab50 0x7f30241646b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f301c009d90 tx=0x7f301c00b330 comp rx=0 tx=0).stop 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f302af9d700 1 -- 172.21.15.102:0/2864789461 shutdown_connections 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f302af9d700 1 --2- 172.21.15.102:0/2864789461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30240cab50 0x7f30241646b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f302af9d700 1 -- 172.21.15.102:0/2864789461 >> 172.21.15.102:0/2864789461 conn(0x7f3024065fb0 msgr2=0x7f30240663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f302af9d700 1 -- 172.21.15.102:0/2864789461 shutdown_connections 2023-03-03T16:22:49.747 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.745+0000 7f302af9d700 1 -- 172.21.15.102:0/2864789461 wait complete. 2023-03-03T16:22:49.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.747+0000 7f302af9d700 1 Processor -- start 2023-03-03T16:22:49.748 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.748+0000 7f302af9d700 1 -- start start 2023-03-03T16:22:49.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.748+0000 7f302af9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30240c9610 0x7f3024154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:49.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.748+0000 7f302af9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30240cab50 0x7f30241f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:49.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302af9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30241564b0 0x7f30241f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:49.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302af9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3024155a70 con 0x7f30240c9610 2023-03-03T16:22:49.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302af9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240c6ac0 con 0x7f30241564b0 2023-03-03T16:22:49.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302af9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30240c6da0 con 0x7f30240cab50 2023-03-03T16:22:49.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f3029f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30240c9610 0x7f3024154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:49.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302979a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30240cab50 0x7f30241f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:49.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30241564b0 0x7f30241f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:49.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30241564b0 0x7f30241f4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:59750/0 (socket says 172.21.15.102:59750) 2023-03-03T16:22:49.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 -- 172.21.15.102:0/833614568 learned_addr learned my addr 172.21.15.102:0/833614568 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:22:49.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f3029f9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30240c9610 0x7f3024154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55042/0 (socket says 172.21.15.102:55042) 2023-03-03T16:22:49.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 -- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30240cab50 msgr2=0x7f30241f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:49.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30240cab50 0x7f30241f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 -- 172.21.15.102:0/833614568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30240c9610 msgr2=0x7f3024154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:49.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30240c9610 0x7f3024154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f302a79c700 1 -- 172.21.15.102:0/833614568 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c009a70 con 0x7f30241564b0 2023-03-03T16:22:49.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.749+0000 7f3029f9b700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30240c9610 0x7f3024154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:22:49.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.750+0000 7f302a79c700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30241564b0 0x7f30241f4040 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f302406c8c0 tx=0x7f301c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:49.752 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.750+0000 7f301a7fc700 1 -- 172.21.15.102:0/833614568 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c012630 con 0x7f30241564b0 2023-03-03T16:22:49.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.750+0000 7f301a7fc700 1 -- 172.21.15.102:0/833614568 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c012790 con 0x7f30241564b0 2023-03-03T16:22:49.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.750+0000 7f301a7fc700 1 -- 172.21.15.102:0/833614568 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c012b40 con 0x7f30241564b0 2023-03-03T16:22:49.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.750+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30240b2210 con 0x7f30241564b0 2023-03-03T16:22:49.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.750+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30240b1f30 con 0x7f30241564b0 2023-03-03T16:22:49.753 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.751+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3024156880 con 0x7f30241564b0 2023-03-03T16:22:49.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.753+0000 7f301a7fc700 1 -- 172.21.15.102:0/833614568 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f301c02c070 con 0x7f30241564b0 2023-03-03T16:22:49.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.754+0000 7f301a7fc700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3008068320 0x7f300806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:49.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.754+0000 7f301a7fc700 1 -- 172.21.15.102:0/833614568 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f301c087e40 con 0x7f30241564b0 2023-03-03T16:22:49.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.754+0000 7f3029f9b700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3008068320 0x7f300806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:49.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.755+0000 7f3029f9b700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3008068320 0x7f300806a7d0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3020007c20 tx=0x7f3020007b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:49.758 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.758+0000 7f301a7fc700 1 -- 172.21.15.102:0/833614568 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f301c054550 con 0x7f30241564b0 2023-03-03T16:22:49.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.991+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30240677f0 con 0x7f30241564b0 2023-03-03T16:22:49.992 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.992+0000 7f301a7fc700 1 -- 172.21.15.102:0/833614568 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f301c02a090 con 0x7f30241564b0 2023-03-03T16:22:49.993 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:22:49.993 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:22:49.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3008068320 msgr2=0x7f300806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:49.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3008068320 0x7f300806a7d0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3020007c20 tx=0x7f3020007b70 comp rx=0 tx=0).stop 2023-03-03T16:22:49.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30241564b0 msgr2=0x7f30241f4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:49.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30241564b0 0x7f30241f4040 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f302406c8c0 tx=0x7f301c004ad0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 shutdown_connections 2023-03-03T16:22:49.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30241564b0 0x7f30241f4040 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30240c9610 0x7f3024154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3008068320 0x7f300806a7d0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 --2- 172.21.15.102:0/833614568 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f30240cab50 0x7f30241f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:49.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 >> 172.21.15.102:0/833614568 conn(0x7f3024065fb0 msgr2=0x7f3024155150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:49.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 shutdown_connections 2023-03-03T16:22:49.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:22:49.995+0000 7f302af9d700 1 -- 172.21.15.102:0/833614568 wait complete. 2023-03-03T16:22:49.997 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:22:50.022 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T16:22:50.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:50.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:20.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:20.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:50.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:50.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:20.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:20.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:41.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:24:41.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.290+0000 7f8f5359e700 1 Processor -- start 2023-03-03T16:24:41.290 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.290+0000 7f8f5359e700 1 -- start start 2023-03-03T16:24:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.292+0000 7f8f5359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f540c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.292+0000 7f8f5359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540c3d20 con 0x7f8f540ced00 2023-03-03T16:24:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.292+0000 7f8f5359e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5415cba0 con 0x7f8f540b3970 2023-03-03T16:24:41.292 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.292+0000 7f8f5359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540c4930 con 0x7f8f540c5a90 2023-03-03T16:24:41.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.292+0000 7f8f5259c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8f540ced00 0x7f8f54158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33734/0 (socket says 172.21.15.102:33734) 2023-03-03T16:24:41.293 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.292+0000 7f8f5259c700 1 -- 172.21.15.102:0/3513904411 learned_addr learned my addr 172.21.15.102:0/3513904411 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.292+0000 7f8f52d9d700 1 --2- 172.21.15.102:0/3513904411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f540c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.293+0000 7f8f51d9b700 1 -- 172.21.15.102:0/3513904411 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20021849 0 0) 0x7f8f540c3d20 con 0x7f8f540ced00 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.293+0000 7f8f51d9b700 1 -- 172.21.15.102:0/3513904411 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f400035b0 con 0x7f8f540ced00 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.293+0000 7f8f52d9d700 1 -- 172.21.15.102:0/3513904411 >> v1:172.21.15.172:6789/0 conn(0x7f8f540b3970 legacy=0x7f8f54161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.293+0000 7f8f51d9b700 1 -- 172.21.15.102:0/3513904411 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362881943 0 0) 0x7f8f5415cba0 con 0x7f8f540b3970 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.293+0000 7f8f52d9d700 1 -- 172.21.15.102:0/3513904411 >> v1:172.21.15.102:6789/0 conn(0x7f8f540ced00 legacy=0x7f8f54158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.293+0000 7f8f52d9d700 1 -- 172.21.15.102:0/3513904411 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5415ce80 con 0x7f8f540c5a90 2023-03-03T16:24:41.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.293+0000 7f8f52d9d700 1 --2- 172.21.15.102:0/3513904411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f540c41b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009dc0 tx=0x7f8f4c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dabbedf2e5199219 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:24:41.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f51d9b700 1 -- 172.21.15.102:0/3513904411 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c00e070 con 0x7f8f540c5a90 2023-03-03T16:24:41.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f51d9b700 1 -- 172.21.15.102:0/3513904411 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c00bb20 con 0x7f8f540c5a90 2023-03-03T16:24:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f51d9b700 1 -- 172.21.15.102:0/3513904411 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c00bed0 con 0x7f8f540c5a90 2023-03-03T16:24:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f5359e700 1 -- 172.21.15.102:0/3513904411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 msgr2=0x7f8f540c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f5359e700 1 --2- 172.21.15.102:0/3513904411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f540c41b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009dc0 tx=0x7f8f4c00b330 comp rx=0 tx=0).stop 2023-03-03T16:24:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f5359e700 1 -- 172.21.15.102:0/3513904411 shutdown_connections 2023-03-03T16:24:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f5359e700 1 --2- 172.21.15.102:0/3513904411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f540c41b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f5359e700 1 -- 172.21.15.102:0/3513904411 >> 172.21.15.102:0/3513904411 conn(0x7f8f5408b1c0 msgr2=0x7f8f54071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:24:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f5359e700 1 -- 172.21.15.102:0/3513904411 shutdown_connections 2023-03-03T16:24:41.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.294+0000 7f8f5359e700 1 -- 172.21.15.102:0/3513904411 wait complete. 2023-03-03T16:24:41.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.296+0000 7f8f5359e700 1 Processor -- start 2023-03-03T16:24:41.297 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.296+0000 7f8f5359e700 1 -- start start 2023-03-03T16:24:41.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.297+0000 7f8f5359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f540b3970 0x7f8f5415bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:41.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.297+0000 7f8f5359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f541e3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:41.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.297+0000 7f8f5359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f540ced00 0x7f8f54159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:41.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.297+0000 7f8f5359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f541e2910 con 0x7f8f540b3970 2023-03-03T16:24:41.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.297+0000 7f8f5359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f541e2ed0 con 0x7f8f540ced00 2023-03-03T16:24:41.298 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.297+0000 7f8f5359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f541e4350 con 0x7f8f540c5a90 2023-03-03T16:24:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f5259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f540b3970 0x7f8f5415bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f540ced00 0x7f8f54159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f4bfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f541e3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f540ced00 0x7f8f54159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45062/0 (socket says 172.21.15.102:45062) 2023-03-03T16:24:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 -- 172.21.15.102:0/2796289196 learned_addr learned my addr 172.21.15.102:0/2796289196 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:24:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 -- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 msgr2=0x7f8f541e3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f541e3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 -- 172.21.15.102:0/2796289196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f540b3970 msgr2=0x7f8f5415bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f540b3970 0x7f8f5415bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 -- 172.21.15.102:0/2796289196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c009a70 con 0x7f8f540ced00 2023-03-03T16:24:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f4bfff700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f541e3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:24:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.298+0000 7f8f52d9d700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f540ced00 0x7f8f54159cc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009f30 tx=0x7f8f4c004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:24:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.299+0000 7f8f4affd700 1 -- 172.21.15.102:0/2796289196 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c01b810 con 0x7f8f540ced00 2023-03-03T16:24:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.299+0000 7f8f4affd700 1 -- 172.21.15.102:0/2796289196 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c020070 con 0x7f8f540ced00 2023-03-03T16:24:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.299+0000 7f8f4affd700 1 -- 172.21.15.102:0/2796289196 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c01bbf0 con 0x7f8f540ced00 2023-03-03T16:24:41.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.299+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5406c200 con 0x7f8f540ced00 2023-03-03T16:24:41.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.299+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5406bf20 con 0x7f8f540ced00 2023-03-03T16:24:41.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.300+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5415c890 con 0x7f8f540ced00 2023-03-03T16:24:41.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.301+0000 7f8f4affd700 1 -- 172.21.15.102:0/2796289196 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8f4c01bd50 con 0x7f8f540ced00 2023-03-03T16:24:41.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.302+0000 7f8f4affd700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f340682f0 0x7f8f3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:41.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.302+0000 7f8f4affd700 1 -- 172.21.15.102:0/2796289196 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8f4c022070 con 0x7f8f540ced00 2023-03-03T16:24:41.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.303+0000 7f8f5259c700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f340682f0 0x7f8f3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:41.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.303+0000 7f8f5259c700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f340682f0 0x7f8f3406a7a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f8f44007fd0 tx=0x7f8f44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:24:41.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.307+0000 7f8f4affd700 1 -- 172.21.15.102:0/2796289196 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8f4c054ac0 con 0x7f8f540ced00 2023-03-03T16:24:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.540+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f541e3060 con 0x7f8f540ced00 2023-03-03T16:24:41.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.541+0000 7f8f4affd700 1 -- 172.21.15.102:0/2796289196 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8f4c0580e0 con 0x7f8f540ced00 2023-03-03T16:24:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:24:41.542 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:24:41.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f340682f0 msgr2=0x7f8f3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:41.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f340682f0 0x7f8f3406a7a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f8f44007fd0 tx=0x7f8f44009040 comp rx=0 tx=0).stop 2023-03-03T16:24:41.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f540ced00 msgr2=0x7f8f54159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:41.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f540ced00 0x7f8f54159cc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009f30 tx=0x7f8f4c004b20 comp rx=0 tx=0).stop 2023-03-03T16:24:41.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 shutdown_connections 2023-03-03T16:24:41.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f540ced00 0x7f8f54159cc0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:41.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f540b3970 0x7f8f5415bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:41.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f340682f0 0x7f8f3406a7a0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:41.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 --2- 172.21.15.102:0/2796289196 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f540c5a90 0x7f8f541e3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:41.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 >> 172.21.15.102:0/2796289196 conn(0x7f8f5408b1c0 msgr2=0x7f8f54071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:24:41.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 shutdown_connections 2023-03-03T16:24:41.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:24:41.544+0000 7f8f5359e700 1 -- 172.21.15.102:0/2796289196 wait complete. 2023-03-03T16:24:41.546 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:24:41.571 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-03-03T16:24:50.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:50.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:21.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:21.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:51.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:51.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:56.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:25:56.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.819+0000 7f9c59fff700 1 Processor -- start 2023-03-03T16:25:56.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.819+0000 7f9c59fff700 1 -- start start 2023-03-03T16:25:56.821 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.821+0000 7f9c59fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c54153290 0x7f9c54158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:56.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.821+0000 7f9c59fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54153660 con 0x7f9c540cd8e0 2023-03-03T16:25:56.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.821+0000 7f9c59fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5406c360 con 0x7f9c541593c0 2023-03-03T16:25:56.822 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.821+0000 7f9c59fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540cec40 con 0x7f9c54153290 2023-03-03T16:25:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.821+0000 7f9c52ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9c541593c0 0x7f9c540cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:36446/0 (socket says 172.21.15.102:36446) 2023-03-03T16:25:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.821+0000 7f9c52ffd700 1 -- 172.21.15.102:0/3858317692 learned_addr learned my addr 172.21.15.102:0/3858317692 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:25:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.821+0000 7f9c537fe700 1 --2- 172.21.15.102:0/3858317692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c54153290 0x7f9c54158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:56.824 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c527fc700 1 -- 172.21.15.102:0/3858317692 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413930666 0 0) 0x7f9c5406c360 con 0x7f9c541593c0 2023-03-03T16:25:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c527fc700 1 -- 172.21.15.102:0/3858317692 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c3c0035b0 con 0x7f9c541593c0 2023-03-03T16:25:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c527fc700 1 -- 172.21.15.102:0/3858317692 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394026200 0 0) 0x7f9c54153660 con 0x7f9c540cd8e0 2023-03-03T16:25:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c527fc700 1 -- 172.21.15.102:0/3858317692 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5406c360 con 0x7f9c540cd8e0 2023-03-03T16:25:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c537fe700 1 -- 172.21.15.102:0/3858317692 >> v1:172.21.15.172:6789/0 conn(0x7f9c541593c0 legacy=0x7f9c540cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c537fe700 1 -- 172.21.15.102:0/3858317692 >> v1:172.21.15.102:6789/0 conn(0x7f9c540cd8e0 legacy=0x7f9c540ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c537fe700 1 -- 172.21.15.102:0/3858317692 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5415cea0 con 0x7f9c54153290 2023-03-03T16:25:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c537fe700 1 --2- 172.21.15.102:0/3858317692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c54153290 0x7f9c54158e80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009870 tx=0x7f9c44009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c433f800cef14b63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c527fc700 1 -- 172.21.15.102:0/3858317692 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44011830 con 0x7f9c54153290 2023-03-03T16:25:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c527fc700 1 -- 172.21.15.102:0/3858317692 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c44011990 con 0x7f9c54153290 2023-03-03T16:25:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.822+0000 7f9c527fc700 1 -- 172.21.15.102:0/3858317692 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44011d20 con 0x7f9c54153290 2023-03-03T16:25:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.823+0000 7f9c59fff700 1 -- 172.21.15.102:0/3858317692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c54153290 msgr2=0x7f9c54158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.823+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3858317692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c54153290 0x7f9c54158e80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009870 tx=0x7f9c44009b80 comp rx=0 tx=0).stop 2023-03-03T16:25:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.823+0000 7f9c59fff700 1 -- 172.21.15.102:0/3858317692 shutdown_connections 2023-03-03T16:25:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.823+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3858317692 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c54153290 0x7f9c54158e80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.823+0000 7f9c59fff700 1 -- 172.21.15.102:0/3858317692 >> 172.21.15.102:0/3858317692 conn(0x7f9c54065fb0 msgr2=0x7f9c540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:25:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.823+0000 7f9c59fff700 1 -- 172.21.15.102:0/3858317692 shutdown_connections 2023-03-03T16:25:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.823+0000 7f9c59fff700 1 -- 172.21.15.102:0/3858317692 wait complete. 2023-03-03T16:25:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.825+0000 7f9c59fff700 1 Processor -- start 2023-03-03T16:25:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.825+0000 7f9c59fff700 1 -- start start 2023-03-03T16:25:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.826+0000 7f9c59fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c540cd8e0 0x7f9c541517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.826+0000 7f9c59fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c54153290 0x7f9c541ef420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.826+0000 7f9c59fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c541593c0 0x7f9c540c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.826+0000 7f9c59fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541f3090 con 0x7f9c541593c0 2023-03-03T16:25:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.826+0000 7f9c59fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5415a490 con 0x7f9c54153290 2023-03-03T16:25:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.826+0000 7f9c59fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541eef90 con 0x7f9c540cd8e0 2023-03-03T16:25:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c537fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c540cd8e0 0x7f9c541517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c54153290 0x7f9c541ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c53fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c541593c0 0x7f9c540c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c537fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c540cd8e0 0x7f9c541517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:51566/0 (socket says 172.21.15.102:51566) 2023-03-03T16:25:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c537fe700 1 -- 172.21.15.102:0/3232434740 learned_addr learned my addr 172.21.15.102:0/3232434740 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:25:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c54153290 0x7f9c541ef420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46500/0 (socket says 172.21.15.102:46500) 2023-03-03T16:25:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 -- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c540cd8e0 msgr2=0x7f9c541517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c540cd8e0 0x7f9c541517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 -- 172.21.15.102:0/3232434740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c541593c0 msgr2=0x7f9c540c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c541593c0 0x7f9c540c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 -- 172.21.15.102:0/3232434740 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c440094a0 con 0x7f9c54153290 2023-03-03T16:25:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c53fff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c541593c0 0x7f9c540c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:25:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c537fe700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c540cd8e0 0x7f9c541517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:25:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.827+0000 7f9c52ffd700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c54153290 0x7f9c541ef420 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800c500 tx=0x7f9c4800c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.828+0000 7f9c3bfff700 1 -- 172.21.15.102:0/3232434740 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c48005d70 con 0x7f9c54153290 2023-03-03T16:25:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.828+0000 7f9c3bfff700 1 -- 172.21.15.102:0/3232434740 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c480070d0 con 0x7f9c54153290 2023-03-03T16:25:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.828+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5415d160 con 0x7f9c54153290 2023-03-03T16:25:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.828+0000 7f9c3bfff700 1 -- 172.21.15.102:0/3232434740 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c48007480 con 0x7f9c54153290 2023-03-03T16:25:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.828+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5406c7f0 con 0x7f9c54153290 2023-03-03T16:25:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.829+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c540ccde0 con 0x7f9c54153290 2023-03-03T16:25:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.830+0000 7f9c3bfff700 1 -- 172.21.15.102:0/3232434740 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c48019030 con 0x7f9c54153290 2023-03-03T16:25:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.831+0000 7f9c3bfff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c40068320 0x7f9c4006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.832+0000 7f9c3bfff700 1 -- 172.21.15.102:0/3232434740 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9c48085080 con 0x7f9c54153290 2023-03-03T16:25:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.832+0000 7f9c537fe700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c40068320 0x7f9c4006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:56.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.832+0000 7f9c537fe700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c40068320 0x7f9c4006a7d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9c44000c00 tx=0x7f9c4401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:56.836+0000 7f9c3bfff700 1 -- 172.21.15.102:0/3232434740 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c48051790 con 0x7f9c54153290 2023-03-03T16:25:57.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.068+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c5406d110 con 0x7f9c54153290 2023-03-03T16:25:57.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.069+0000 7f9c3bfff700 1 -- 172.21.15.102:0/3232434740 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9c48017090 con 0x7f9c54153290 2023-03-03T16:25:57.071 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:25:57.071 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:25:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c40068320 msgr2=0x7f9c4006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c40068320 0x7f9c4006a7d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9c44000c00 tx=0x7f9c4401c040 comp rx=0 tx=0).stop 2023-03-03T16:25:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c54153290 msgr2=0x7f9c541ef420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c54153290 0x7f9c541ef420 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800c500 tx=0x7f9c4800c810 comp rx=0 tx=0).stop 2023-03-03T16:25:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 shutdown_connections 2023-03-03T16:25:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9c40068320 0x7f9c4006a7d0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9c540cd8e0 0x7f9c541517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c541593c0 0x7f9c540c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:57.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 --2- 172.21.15.102:0/3232434740 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c54153290 0x7f9c541ef420 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 >> 172.21.15.102:0/3232434740 conn(0x7f9c54065fb0 msgr2=0x7f9c540cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:25:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 shutdown_connections 2023-03-03T16:25:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:25:57.072+0000 7f9c59fff700 1 -- 172.21.15.102:0/3232434740 wait complete. 2023-03-03T16:25:57.075 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:25:57.101 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 61 secs before thrashing 2023-03-03T16:26:21.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:21.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:51.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:51.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:58.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:26:58.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.354+0000 7f253d39d700 1 Processor -- start 2023-03-03T16:26:58.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.354+0000 7f253d39d700 1 -- start start 2023-03-03T16:26:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f253d39d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25380c9610 0x7f25380ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f253d39d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f25380ca980 0x7f25381636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f253d39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25380c9ca0 con 0x7f25381564b0 2023-03-03T16:26:58.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f253d39d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25380b4920 con 0x7f25380ca980 2023-03-03T16:26:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f253d39d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25380b4c00 con 0x7f25380c9610 2023-03-03T16:26:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f2537fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f25381564b0 0x7f25380cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45390/0 (socket says 172.21.15.102:45390) 2023-03-03T16:26:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f2537fff700 1 -- 172.21.15.102:0/368718703 learned_addr learned my addr 172.21.15.102:0/368718703 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:26:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f253cb9c700 1 --2- 172.21.15.102:0/368718703 >> v2:172.21.15.102:3300/0 conn(0x7f25380ca980 0x7f25381636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:58.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.356+0000 7f25377fe700 1 --2- 172.21.15.102:0/368718703 >> v2:172.21.15.172:3300/0 conn(0x7f25380c9610 0x7f25380ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:58.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f253cb9c700 1 -- 172.21.15.102:0/368718703 >> v2:172.21.15.172:3300/0 conn(0x7f25380c9610 msgr2=0x7f25380ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f253cb9c700 1 --2- 172.21.15.102:0/368718703 >> v2:172.21.15.172:3300/0 conn(0x7f25380c9610 0x7f25380ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f253cb9c700 1 -- 172.21.15.102:0/368718703 >> v1:172.21.15.102:6789/0 conn(0x7f25381564b0 legacy=0x7f25380cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f253cb9c700 1 -- 172.21.15.102:0/368718703 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25380b46f0 con 0x7f25380ca980 2023-03-03T16:26:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f25377fe700 1 --2- 172.21.15.102:0/368718703 >> v2:172.21.15.172:3300/0 conn(0x7f25380c9610 0x7f25380ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:26:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f2536ffd700 1 -- 172.21.15.102:0/368718703 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429285089 0 0) 0x7f25380c9ca0 con 0x7f25381564b0 2023-03-03T16:26:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f253cb9c700 1 --2- 172.21.15.102:0/368718703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25381636a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f253815d410 tx=0x7f253000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf170fac09e9c10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:58.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f2536ffd700 1 -- 172.21.15.102:0/368718703 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000e070 con 0x7f25380ca980 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f2536ffd700 1 -- 172.21.15.102:0/368718703 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253000bb20 con 0x7f25380ca980 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.357+0000 7f2536ffd700 1 -- 172.21.15.102:0/368718703 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530012430 con 0x7f25380ca980 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 -- 172.21.15.102:0/368718703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 msgr2=0x7f25381636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 --2- 172.21.15.102:0/368718703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25381636a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f253815d410 tx=0x7f253000b330 comp rx=0 tx=0).stop 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 -- 172.21.15.102:0/368718703 shutdown_connections 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 --2- 172.21.15.102:0/368718703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25381636a0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 --2- 172.21.15.102:0/368718703 >> v2:172.21.15.172:3300/0 conn(0x7f25380c9610 0x7f25380ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.360 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 -- 172.21.15.102:0/368718703 >> 172.21.15.102:0/368718703 conn(0x7f2538065fb0 msgr2=0x7f25380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 -- 172.21.15.102:0/368718703 shutdown_connections 2023-03-03T16:26:58.361 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.358+0000 7f253d39d700 1 -- 172.21.15.102:0/368718703 wait complete. 2023-03-03T16:26:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.360+0000 7f253d39d700 1 Processor -- start 2023-03-03T16:26:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.360+0000 7f253d39d700 1 -- start start 2023-03-03T16:26:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.361+0000 7f253d39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25380c9610 0x7f25380c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:58.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.361+0000 7f253d39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25380c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.361+0000 7f253d39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25381564b0 0x7f25380c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.361+0000 7f253d39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25380c3e30 con 0x7f25380ca980 2023-03-03T16:26:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.361+0000 7f253d39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538155410 con 0x7f25381564b0 2023-03-03T16:26:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.361+0000 7f253d39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380c7390 con 0x7f25380c9610 2023-03-03T16:26:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f25377fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25380c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f2537fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25380c9610 0x7f25380c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:58.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f25377fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25380c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51940/0 (socket says 172.21.15.102:51940) 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f25377fe700 1 -- 172.21.15.102:0/1291975886 learned_addr learned my addr 172.21.15.102:0/1291975886 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f253cb9c700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25381564b0 0x7f25380c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f2537fff700 1 -- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25381564b0 msgr2=0x7f25380c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f2537fff700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25381564b0 0x7f25380c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f2537fff700 1 -- 172.21.15.102:0/1291975886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 msgr2=0x7f25380c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f2537fff700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25380c4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f2537fff700 1 -- 172.21.15.102:0/1291975886 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c0094a0 con 0x7f25380c9610 2023-03-03T16:26:58.364 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f25377fe700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25380c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:26:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.362+0000 7f2537fff700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25380c9610 0x7f25380c49c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f252800c410 tx=0x7f252800c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.363+0000 7f2534ff9700 1 -- 172.21.15.102:0/1291975886 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528004c80 con 0x7f25380c9610 2023-03-03T16:26:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.363+0000 7f2534ff9700 1 -- 172.21.15.102:0/1291975886 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528004de0 con 0x7f25380c9610 2023-03-03T16:26:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.363+0000 7f2534ff9700 1 -- 172.21.15.102:0/1291975886 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528007310 con 0x7f25380c9610 2023-03-03T16:26:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.363+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2530009a70 con 0x7f25380c9610 2023-03-03T16:26:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.363+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25380b2210 con 0x7f25380c9610 2023-03-03T16:26:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.364+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25380b1f30 con 0x7f25380c9610 2023-03-03T16:26:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.365+0000 7f2534ff9700 1 -- 172.21.15.102:0/1291975886 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f252801a020 con 0x7f25380c9610 2023-03-03T16:26:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.366+0000 7f2534ff9700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2524068320 0x7f252406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.366+0000 7f2534ff9700 1 -- 172.21.15.102:0/1291975886 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f25280857a0 con 0x7f25380c9610 2023-03-03T16:26:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.367+0000 7f25377fe700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2524068320 0x7f252406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.367+0000 7f25377fe700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2524068320 0x7f252406a7d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f252c009fd0 tx=0x7f252c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:58.370 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.370+0000 7f2534ff9700 1 -- 172.21.15.102:0/1291975886 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2528051e20 con 0x7f25380c9610 2023-03-03T16:26:58.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.604+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2538156880 con 0x7f25380c9610 2023-03-03T16:26:58.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.605+0000 7f2534ff9700 1 -- 172.21.15.102:0/1291975886 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2528055440 con 0x7f25380c9610 2023-03-03T16:26:58.605 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:26:58.605 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:26:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2524068320 msgr2=0x7f252406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2524068320 0x7f252406a7d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f252c009fd0 tx=0x7f252c009210 comp rx=0 tx=0).stop 2023-03-03T16:26:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25380c9610 msgr2=0x7f25380c49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:58.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25380c9610 0x7f25380c49c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f252800c410 tx=0x7f252800c720 comp rx=0 tx=0).stop 2023-03-03T16:26:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 shutdown_connections 2023-03-03T16:26:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2524068320 0x7f252406a7d0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f25380c9610 0x7f25380c49c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25381564b0 0x7f25380c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 --2- 172.21.15.102:0/1291975886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25380ca980 0x7f25380c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 >> 172.21.15.102:0/1291975886 conn(0x7f2538065fb0 msgr2=0x7f2538066380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:58.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 shutdown_connections 2023-03-03T16:26:58.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:26:58.608+0000 7f253d39d700 1 -- 172.21.15.102:0/1291975886 wait complete. 2023-03-03T16:26:58.610 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:26:58.635 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-03-03T16:27:08.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:27:08.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.885+0000 7f79a539d700 1 Processor -- start 2023-03-03T16:27:08.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.885+0000 7f79a539d700 1 -- start start 2023-03-03T16:27:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f79a539d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79a00ced00 0x7f79a00c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f79a539d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f79a00c41b0 0x7f79a00cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:08.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f79a539d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a00c3d20 con 0x7f79a00b3970 2023-03-03T16:27:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f79a539d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a00c4930 con 0x7f79a00ced00 2023-03-03T16:27:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f79a539d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a00cf610 con 0x7f79a00c41b0 2023-03-03T16:27:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f799ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79a00ced00 0x7f79a00c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:08.888 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f799f7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f79a00b3970 0x7f79a0158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:47096/0 (socket says 172.21.15.102:47096) 2023-03-03T16:27:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f799f7fe700 1 -- 172.21.15.102:0/4065936905 learned_addr learned my addr 172.21.15.102:0/4065936905 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:27:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f79a4b9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f79a00c41b0 0x7f79a00cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.887+0000 7f799effd700 1 -- 172.21.15.102:0/4065936905 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199526903 0 0) 0x7f79a00c3d20 con 0x7f79a00b3970 2023-03-03T16:27:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f799effd700 1 -- 172.21.15.102:0/4065936905 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79880035b0 con 0x7f79a00b3970 2023-03-03T16:27:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f79a4b9c700 1 -- 172.21.15.102:0/4065936905 >> v2:172.21.15.102:3300/0 conn(0x7f79a00ced00 msgr2=0x7f79a00c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:27:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f79a4b9c700 1 --2- 172.21.15.102:0/4065936905 >> v2:172.21.15.102:3300/0 conn(0x7f79a00ced00 0x7f79a00c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f79a4b9c700 1 -- 172.21.15.102:0/4065936905 >> v1:172.21.15.172:6789/0 conn(0x7f79a00b3970 legacy=0x7f79a0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f79a4b9c700 1 -- 172.21.15.102:0/4065936905 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a015cba0 con 0x7f79a00c41b0 2023-03-03T16:27:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f799ffff700 1 --2- 172.21.15.102:0/4065936905 >> v2:172.21.15.102:3300/0 conn(0x7f79a00ced00 0x7f79a00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:27:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f79a4b9c700 1 --2- 172.21.15.102:0/4065936905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00c41b0 0x7f79a00cf0d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f79a00ca080 tx=0x7f799800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf96c83a2a0ef430 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:27:08.891 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f799effd700 1 -- 172.21.15.102:0/4065936905 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799800e070 con 0x7f79a00c41b0 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f799effd700 1 -- 172.21.15.102:0/4065936905 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799800bb20 con 0x7f79a00c41b0 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.888+0000 7f799effd700 1 -- 172.21.15.102:0/4065936905 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998012430 con 0x7f79a00c41b0 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 -- 172.21.15.102:0/4065936905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00c41b0 msgr2=0x7f79a00cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 --2- 172.21.15.102:0/4065936905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00c41b0 0x7f79a00cf0d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f79a00ca080 tx=0x7f799800b330 comp rx=0 tx=0).stop 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 -- 172.21.15.102:0/4065936905 shutdown_connections 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 --2- 172.21.15.102:0/4065936905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00c41b0 0x7f79a00cf0d0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 --2- 172.21.15.102:0/4065936905 >> v2:172.21.15.102:3300/0 conn(0x7f79a00ced00 0x7f79a00c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:08.892 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 -- 172.21.15.102:0/4065936905 >> 172.21.15.102:0/4065936905 conn(0x7f79a008b1c0 msgr2=0x7f79a0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:27:08.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 -- 172.21.15.102:0/4065936905 shutdown_connections 2023-03-03T16:27:08.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.889+0000 7f79a539d700 1 -- 172.21.15.102:0/4065936905 wait complete. 2023-03-03T16:27:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.891+0000 7f79a539d700 1 Processor -- start 2023-03-03T16:27:08.894 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.891+0000 7f79a539d700 1 -- start start 2023-03-03T16:27:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.892+0000 7f79a539d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a00b3970 0x7f79a014de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.892+0000 7f79a539d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79a00c41b0 0x7f79a0148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.892+0000 7f79a539d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00ced00 0x7f79a0148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.892+0000 7f79a539d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a00cfc50 con 0x7f79a00b3970 2023-03-03T16:27:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.892+0000 7f79a539d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a01e2030 con 0x7f79a00ced00 2023-03-03T16:27:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.892+0000 7f79a539d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a01eeeb0 con 0x7f79a00c41b0 2023-03-03T16:27:08.895 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a00b3970 0x7f79a014de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f79a4b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00ced00 0x7f79a0148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a00b3970 0x7f79a014de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41908/0 (socket says 172.21.15.102:41908) 2023-03-03T16:27:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 -- 172.21.15.102:0/3995146882 learned_addr learned my addr 172.21.15.102:0/3995146882 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:27:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799f7fe700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79a00c41b0 0x7f79a0148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 -- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79a00c41b0 msgr2=0x7f79a0148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79a00c41b0 0x7f79a0148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 -- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00ced00 msgr2=0x7f79a0148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00ced00 0x7f79a0148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 -- 172.21.15.102:0/3995146882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79900094a0 con 0x7f79a00b3970 2023-03-03T16:27:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f79a4b9c700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00ced00 0x7f79a0148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:27:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.893+0000 7f799ffff700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a00b3970 0x7f79a014de80 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f799000ebe0 tx=0x7f799000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:27:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.894+0000 7f799cff9700 1 -- 172.21.15.102:0/3995146882 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799000c7f0 con 0x7f79a00b3970 2023-03-03T16:27:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.894+0000 7f799cff9700 1 -- 172.21.15.102:0/3995146882 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799000c950 con 0x7f79a00b3970 2023-03-03T16:27:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.894+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7998009a70 con 0x7f79a00b3970 2023-03-03T16:27:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.894+0000 7f799cff9700 1 -- 172.21.15.102:0/3995146882 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799000cce0 con 0x7f79a00b3970 2023-03-03T16:27:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.894+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a006bf20 con 0x7f79a00b3970 2023-03-03T16:27:08.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.895+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a015ce80 con 0x7f79a00b3970 2023-03-03T16:27:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.896+0000 7f799cff9700 1 -- 172.21.15.102:0/3995146882 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7990023070 con 0x7f79a00b3970 2023-03-03T16:27:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.897+0000 7f799cff9700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f798c0682f0 0x7f798c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.898+0000 7f799cff9700 1 -- 172.21.15.102:0/3995146882 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f79900860d0 con 0x7f79a00b3970 2023-03-03T16:27:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.898+0000 7f799f7fe700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f798c0682f0 0x7f798c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.899+0000 7f799f7fe700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f798c0682f0 0x7f798c06a7a0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f79a00ca640 tx=0x7f7994007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:27:08.901 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:08.901+0000 7f799cff9700 1 -- 172.21.15.102:0/3995146882 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f79900526f0 con 0x7f79a00b3970 2023-03-03T16:27:09.132 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.132+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79a006c200 con 0x7f79a00b3970 2023-03-03T16:27:09.133 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.133+0000 7f799cff9700 1 -- 172.21.15.102:0/3995146882 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7990055d10 con 0x7f79a00b3970 2023-03-03T16:27:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:27:09.134 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:27:09.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.135+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f798c0682f0 msgr2=0x7f798c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:09.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.135+0000 7f79a539d700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f798c0682f0 0x7f798c06a7a0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f79a00ca640 tx=0x7f7994007dc0 comp rx=0 tx=0).stop 2023-03-03T16:27:09.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.135+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a00b3970 msgr2=0x7f79a014de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:09.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.135+0000 7f79a539d700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a00b3970 0x7f79a014de80 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f799000ebe0 tx=0x7f799000eef0 comp rx=0 tx=0).stop 2023-03-03T16:27:09.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 shutdown_connections 2023-03-03T16:27:09.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79a00ced00 0x7f79a0148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:09.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a00b3970 0x7f79a014de80 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:09.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f798c0682f0 0x7f798c06a7a0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:09.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 --2- 172.21.15.102:0/3995146882 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f79a00c41b0 0x7f79a0148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:09.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 >> 172.21.15.102:0/3995146882 conn(0x7f79a008b1c0 msgr2=0x7f79a00b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:27:09.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 shutdown_connections 2023-03-03T16:27:09.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:27:09.136+0000 7f79a539d700 1 -- 172.21.15.102:0/3995146882 wait complete. 2023-03-03T16:27:09.138 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:27:09.164 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2023-03-03T16:27:21.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:21.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:51.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:51.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:21.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:21.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:52.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:52.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:05.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:29:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.415+0000 7fdbe39ff700 1 Processor -- start 2023-03-03T16:29:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.415+0000 7fdbe39ff700 1 -- start start 2023-03-03T16:29:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.416+0000 7fdbe39ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.416+0000 7fdbe39ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdbdc0cd8e0 0x7fdbdc0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.416+0000 7fdbe39ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc153660 con 0x7fdbdc1593c0 2023-03-03T16:29:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe39ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc06c360 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe39ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc15cea0 con 0x7fdbdc153290 2023-03-03T16:29:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe179b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe0f9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdbdc1593c0 0x7fdbdc0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39028/0 (socket says 172.21.15.102:39028) 2023-03-03T16:29:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe0f9a700 1 -- 172.21.15.102:0/3392535022 learned_addr learned my addr 172.21.15.102:0/3392535022 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:29:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe1f9c700 1 --2- 172.21.15.102:0/3392535022 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc0cd8e0 0x7fdbdc0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbd3fff700 1 -- 172.21.15.102:0/3392535022 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183374295 0 0) 0x7fdbdc153660 con 0x7fdbdc1593c0 2023-03-03T16:29:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe179b700 1 -- 172.21.15.102:0/3392535022 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc0cd8e0 msgr2=0x7fdbdc0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:29:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe179b700 1 --2- 172.21.15.102:0/3392535022 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc0cd8e0 0x7fdbdc0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe179b700 1 -- 172.21.15.102:0/3392535022 >> v1:172.21.15.102:6789/0 conn(0x7fdbdc1593c0 legacy=0x7fdbdc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.417+0000 7fdbe179b700 1 -- 172.21.15.102:0/3392535022 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc06c7f0 con 0x7fdbdc153290 2023-03-03T16:29:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.418+0000 7fdbe179b700 1 --2- 172.21.15.102:0/3392535022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc158e80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009840 tx=0x7fdbd8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c22dc1c78b69440 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:29:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.418+0000 7fdbd3fff700 1 -- 172.21.15.102:0/3392535022 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8011830 con 0x7fdbdc153290 2023-03-03T16:29:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.418+0000 7fdbd3fff700 1 -- 172.21.15.102:0/3392535022 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd8011990 con 0x7fdbdc153290 2023-03-03T16:29:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.418+0000 7fdbd3fff700 1 -- 172.21.15.102:0/3392535022 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8011d20 con 0x7fdbdc153290 2023-03-03T16:29:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.418+0000 7fdbe39ff700 1 -- 172.21.15.102:0/3392535022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 msgr2=0x7fdbdc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.418+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/3392535022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc158e80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009840 tx=0x7fdbd8009b50 comp rx=0 tx=0).stop 2023-03-03T16:29:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.419+0000 7fdbe39ff700 1 -- 172.21.15.102:0/3392535022 shutdown_connections 2023-03-03T16:29:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.419+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/3392535022 >> v2:172.21.15.172:3300/0 conn(0x7fdbdc0cd8e0 0x7fdbdc0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.419+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/3392535022 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc158e80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.419+0000 7fdbe39ff700 1 -- 172.21.15.102:0/3392535022 >> 172.21.15.102:0/3392535022 conn(0x7fdbdc065fb0 msgr2=0x7fdbdc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:29:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.419+0000 7fdbe39ff700 1 -- 172.21.15.102:0/3392535022 shutdown_connections 2023-03-03T16:29:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.419+0000 7fdbe39ff700 1 -- 172.21.15.102:0/3392535022 wait complete. 2023-03-03T16:29:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.421+0000 7fdbe39ff700 1 Processor -- start 2023-03-03T16:29:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.421+0000 7fdbe39ff700 1 -- start start 2023-03-03T16:29:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe39ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbdc0cd8e0 0x7fdbdc14fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe39ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc0c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:05.424 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe39ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc1593c0 0x7fdbdc0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe39ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0c6140 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe39ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc150080 con 0x7fdbdc1593c0 2023-03-03T16:29:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe39ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0c63e0 con 0x7fdbdc153290 2023-03-03T16:29:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe179b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbdc0cd8e0 0x7fdbdc14fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe1f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc1593c0 0x7fdbdc0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe0f9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc0c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:05.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe179b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbdc0cd8e0 0x7fdbdc14fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43756/0 (socket says 172.21.15.102:43756) 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe179b700 1 -- 172.21.15.102:0/1081181084 learned_addr learned my addr 172.21.15.102:0/1081181084 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe1f9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc1593c0 0x7fdbdc0c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47760/0 (socket says 172.21.15.102:47760) 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.422+0000 7fdbe179b700 1 -- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 msgr2=0x7fdbdc0c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbe179b700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc0c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbe179b700 1 -- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc1593c0 msgr2=0x7fdbdc0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbe179b700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc1593c0 0x7fdbdc0c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbe179b700 1 -- 172.21.15.102:0/1081181084 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd80094a0 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbe179b700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbdc0cd8e0 0x7fdbdc14fb40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009b90 tx=0x7fdbd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:29:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbd1ffb700 1 -- 172.21.15.102:0/1081181084 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80036a0 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbd1ffb700 1 -- 172.21.15.102:0/1081181084 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd801f070 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbd1ffb700 1 -- 172.21.15.102:0/1081181084 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8003a40 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc0ccde0 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.423+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc15d160 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.424+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc0cd0c0 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.426+0000 7fdbd1ffb700 1 -- 172.21.15.102:0/1081181084 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdbd800f050 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.427+0000 7fdbd1ffb700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbc0068320 0x7fdbc006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:05.428 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.428+0000 7fdbd1ffb700 1 -- 172.21.15.102:0/1081181084 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdbd8021070 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.428+0000 7fdbe0f9a700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbc0068320 0x7fdbc006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:05.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.428+0000 7fdbe0f9a700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbc0068320 0x7fdbc006a7d0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc08c0c0 tx=0x7fdbcc007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:29:05.433 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.432+0000 7fdbd1ffb700 1 -- 172.21.15.102:0/1081181084 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdbd8053d30 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.664+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbdc0b3700 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.665+0000 7fdbd1ffb700 1 -- 172.21.15.102:0/1081181084 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdbdc0b3700 con 0x7fdbdc0cd8e0 2023-03-03T16:29:05.666 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:29:05.666 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:29:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbc0068320 msgr2=0x7fdbc006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbc0068320 0x7fdbc006a7d0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc08c0c0 tx=0x7fdbcc007de0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbdc0cd8e0 msgr2=0x7fdbdc14fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbdc0cd8e0 0x7fdbdc14fb40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009b90 tx=0x7fdbd8004750 comp rx=0 tx=0).stop 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 shutdown_connections 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdbdc1593c0 0x7fdbdc0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbdc0cd8e0 0x7fdbdc14fb40 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdbc0068320 0x7fdbc006a7d0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 --2- 172.21.15.102:0/1081181084 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdbdc153290 0x7fdbdc0c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 >> 172.21.15.102:0/1081181084 conn(0x7fdbdc065fb0 msgr2=0x7fdbdc156b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 shutdown_connections 2023-03-03T16:29:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:29:05.668+0000 7fdbe39ff700 1 -- 172.21.15.102:0/1081181084 wait complete. 2023-03-03T16:29:05.671 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:29:05.696 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T16:29:22.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:22.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:52.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:52.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:59.999 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T16:29:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:30:14.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:30:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.948+0000 7fe1b3bff700 1 Processor -- start 2023-03-03T16:30:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.948+0000 7fe1b3bff700 1 -- start start 2023-03-03T16:30:14.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.949+0000 7fe1b3bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1ac153290 0x7fe1ac158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:14.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b3bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:14.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b3bff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ac153660 con 0x7fe1ac1593c0 2023-03-03T16:30:14.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b3bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1ac06c360 con 0x7fe1ac153290 2023-03-03T16:30:14.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b3bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac15cea0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b199b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1ac153290 0x7fe1ac158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:14.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b219c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:14.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b119a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe1ac1593c0 0x7fe1ac0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58916/0 (socket says 172.21.15.102:58916) 2023-03-03T16:30:14.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b119a700 1 -- 172.21.15.102:0/1938028027 learned_addr learned my addr 172.21.15.102:0/1938028027 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:30:14.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b219c700 1 -- 172.21.15.102:0/1938028027 >> v2:172.21.15.172:3300/0 conn(0x7fe1ac153290 msgr2=0x7fe1ac158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:30:14.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b219c700 1 --2- 172.21.15.102:0/1938028027 >> v2:172.21.15.172:3300/0 conn(0x7fe1ac153290 0x7fe1ac158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:14.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b0999700 1 -- 172.21.15.102:0/1938028027 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178132566 0 0) 0x7fe1ac153660 con 0x7fe1ac1593c0 2023-03-03T16:30:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b219c700 1 -- 172.21.15.102:0/1938028027 >> v1:172.21.15.102:6789/0 conn(0x7fe1ac1593c0 legacy=0x7fe1ac0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.950+0000 7fe1b219c700 1 -- 172.21.15.102:0/1938028027 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ac06c7f0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.951+0000 7fe1b199b700 1 --2- 172.21.15.102:0/1938028027 >> v2:172.21.15.172:3300/0 conn(0x7fe1ac153290 0x7fe1ac158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:30:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.951+0000 7fe1b219c700 1 --2- 172.21.15.102:0/1938028027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac0ce530 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009d90 tx=0x7fe1a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a884a0ff14bbc0c2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.951+0000 7fe1b0999700 1 -- 172.21.15.102:0/1938028027 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a400e070 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.951+0000 7fe1b0999700 1 -- 172.21.15.102:0/1938028027 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a400bb20 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.951+0000 7fe1b0999700 1 -- 172.21.15.102:0/1938028027 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4012430 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 -- 172.21.15.102:0/1938028027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 msgr2=0x7fe1ac0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/1938028027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac0ce530 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009d90 tx=0x7fe1a400b330 comp rx=0 tx=0).stop 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 -- 172.21.15.102:0/1938028027 shutdown_connections 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/1938028027 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac0ce530 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/1938028027 >> v2:172.21.15.172:3300/0 conn(0x7fe1ac153290 0x7fe1ac158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 -- 172.21.15.102:0/1938028027 >> 172.21.15.102:0/1938028027 conn(0x7fe1ac065fb0 msgr2=0x7fe1ac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 -- 172.21.15.102:0/1938028027 shutdown_connections 2023-03-03T16:30:14.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.952+0000 7fe1b3bff700 1 -- 172.21.15.102:0/1938028027 wait complete. 2023-03-03T16:30:14.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.954+0000 7fe1b3bff700 1 Processor -- start 2023-03-03T16:30:14.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.954+0000 7fe1b3bff700 1 -- start start 2023-03-03T16:30:14.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.955+0000 7fe1b3bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:14.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.955+0000 7fe1b3bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1ac153290 0x7fe1ac0b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:14.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.955+0000 7fe1b3bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1ac1593c0 0x7fe1ac0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:14.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.955+0000 7fe1b3bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac14bf00 con 0x7fe1ac1593c0 2023-03-03T16:30:14.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.955+0000 7fe1b3bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac14c1a0 con 0x7fe1ac153290 2023-03-03T16:30:14.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.955+0000 7fe1b3bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac0b40d0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.955+0000 7fe1b199b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:14.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b119a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1ac153290 0x7fe1ac0b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:14.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac14cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:60720/0 (socket says 172.21.15.102:60720) 2023-03-03T16:30:14.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 -- 172.21.15.102:0/447291503 learned_addr learned my addr 172.21.15.102:0/447291503 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:30:14.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b219c700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1ac1593c0 0x7fe1ac0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:14.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 -- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1ac153290 msgr2=0x7fe1ac0b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:14.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1ac153290 0x7fe1ac0b2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:14.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 -- 172.21.15.102:0/447291503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1ac1593c0 msgr2=0x7fe1ac0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:14.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1ac1593c0 0x7fe1ac0b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:14.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 -- 172.21.15.102:0/447291503 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80094a0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b119a700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1ac153290 0x7fe1ac0b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:30:14.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.956+0000 7fe1b199b700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac14cb00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800ebe0 tx=0x7fe1a800eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:14.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.957+0000 7fe1a27fc700 1 -- 172.21.15.102:0/447291503 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8009760 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.957+0000 7fe1a27fc700 1 -- 172.21.15.102:0/447291503 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a800c850 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.957+0000 7fe1a27fc700 1 -- 172.21.15.102:0/447291503 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800cbe0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.957+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a4009a70 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.957+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ac0ccde0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.958+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ac15d160 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.959+0000 7fe1a27fc700 1 -- 172.21.15.102:0/447291503 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe1a8019030 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.960+0000 7fe1a27fc700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1900682f0 0x7fe19006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:14.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.960+0000 7fe1a27fc700 1 -- 172.21.15.102:0/447291503 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe1a8085ca0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:14.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.961+0000 7fe1b119a700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1900682f0 0x7fe19006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:14.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.961+0000 7fe1b119a700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1900682f0 0x7fe19006a7a0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac1519c0 tx=0x7fe19c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:14.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:14.964+0000 7fe1a27fc700 1 -- 172.21.15.102:0/447291503 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1a80523b0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:15.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.201+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1ac0cd0c0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.202+0000 7fe1a27fc700 1 -- 172.21.15.102:0/447291503 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe1a80559d0 con 0x7fe1ac0cd8e0 2023-03-03T16:30:15.202 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:30:15.202 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:30:15.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1900682f0 msgr2=0x7fe19006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:15.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1900682f0 0x7fe19006a7a0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac1519c0 tx=0x7fe19c007de0 comp rx=0 tx=0).stop 2023-03-03T16:30:15.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 msgr2=0x7fe1ac14cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:15.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac14cb00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800ebe0 tx=0x7fe1a800eef0 comp rx=0 tx=0).stop 2023-03-03T16:30:15.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 shutdown_connections 2023-03-03T16:30:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe1900682f0 0x7fe19006a7a0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe1ac0cd8e0 0x7fe1ac14cb00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1ac1593c0 0x7fe1ac0b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 --2- 172.21.15.102:0/447291503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1ac153290 0x7fe1ac0b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 >> 172.21.15.102:0/447291503 conn(0x7fe1ac065fb0 msgr2=0x7fe1ac1524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 shutdown_connections 2023-03-03T16:30:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:15.205+0000 7fe1b3bff700 1 -- 172.21.15.102:0/447291503 wait complete. 2023-03-03T16:30:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:30:15.232 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2023-03-03T16:30:17.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:30:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.501+0000 7f6c3cf9d700 1 Processor -- start 2023-03-03T16:30:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.501+0000 7f6c3cf9d700 1 -- start start 2023-03-03T16:30:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c3cf9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c381564b0 0x7f6c380cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c3cf9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6c380c9610 0x7f6c380cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c3cf9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c380c9ca0 con 0x7f6c380ca440 2023-03-03T16:30:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c3cf9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c380b4920 con 0x7f6c380c9610 2023-03-03T16:30:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c3cf9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c380b4c00 con 0x7f6c381564b0 2023-03-03T16:30:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c2ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6c380c9610 0x7f6c380cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c377fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c381564b0 0x7f6c380cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c37fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6c380ca440 0x7f6c380ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38592/0 (socket says 172.21.15.102:38592) 2023-03-03T16:30:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c37fff700 1 -- 172.21.15.102:0/3229209388 learned_addr learned my addr 172.21.15.102:0/3229209388 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:30:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.503+0000 7f6c377fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6c381564b0 0x7f6c380cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39930/0 (socket says 172.21.15.102:39930) 2023-03-03T16:30:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c377fe700 1 -- 172.21.15.102:0/3229209388 >> v2:172.21.15.102:3300/0 conn(0x7f6c380c9610 msgr2=0x7f6c380cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:30:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c377fe700 1 --2- 172.21.15.102:0/3229209388 >> v2:172.21.15.102:3300/0 conn(0x7f6c380c9610 0x7f6c380cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c377fe700 1 -- 172.21.15.102:0/3229209388 >> v1:172.21.15.102:6789/0 conn(0x7f6c380ca440 legacy=0x7f6c380ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c377fe700 1 -- 172.21.15.102:0/3229209388 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c380b46f0 con 0x7f6c381564b0 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c36ffd700 1 -- 172.21.15.102:0/3229209388 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658951251 0 0) 0x7f6c380c9ca0 con 0x7f6c380ca440 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c377fe700 1 --2- 172.21.15.102:0/3229209388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c381564b0 0x7f6c380cb660 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6c3808c600 tx=0x7f6c28009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd1a9d87e8518430 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c36ffd700 1 -- 172.21.15.102:0/3229209388 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28011830 con 0x7f6c381564b0 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c36ffd700 1 -- 172.21.15.102:0/3229209388 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c28011990 con 0x7f6c381564b0 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.504+0000 7f6c36ffd700 1 -- 172.21.15.102:0/3229209388 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28011d20 con 0x7f6c381564b0 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3229209388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c381564b0 msgr2=0x7f6c380cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3229209388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c381564b0 0x7f6c380cb660 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6c3808c600 tx=0x7f6c28009b20 comp rx=0 tx=0).stop 2023-03-03T16:30:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3229209388 shutdown_connections 2023-03-03T16:30:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3229209388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c381564b0 0x7f6c380cb660 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3229209388 >> v2:172.21.15.102:3300/0 conn(0x7f6c380c9610 0x7f6c380cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3229209388 >> 172.21.15.102:0/3229209388 conn(0x7f6c38065fb0 msgr2=0x7f6c380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3229209388 shutdown_connections 2023-03-03T16:30:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.505+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3229209388 wait complete. 2023-03-03T16:30:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.507+0000 7f6c3cf9d700 1 Processor -- start 2023-03-03T16:30:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.507+0000 7f6c3cf9d700 1 -- start start 2023-03-03T16:30:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c3cf9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380c9610 0x7f6c381e3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c3cf9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c380ca440 0x7f6c381f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c3cf9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c381564b0 0x7f6c381f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c3cf9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380c72a0 con 0x7f6c381564b0 2023-03-03T16:30:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c3cf9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c38154d10 con 0x7f6c380ca440 2023-03-03T16:30:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c3cf9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380c69e0 con 0x7f6c380c9610 2023-03-03T16:30:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c37fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c381564b0 0x7f6c381f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.508+0000 7f6c37fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c381564b0 0x7f6c381f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53828/0 (socket says 172.21.15.102:53828) 2023-03-03T16:30:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c37fff700 1 -- 172.21.15.102:0/3791873753 learned_addr learned my addr 172.21.15.102:0/3791873753 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:30:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c2ffff700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c380ca440 0x7f6c381f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c377fe700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380c9610 0x7f6c381e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c37fff700 1 -- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380c9610 msgr2=0x7f6c381e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c37fff700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380c9610 0x7f6c381e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c37fff700 1 -- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c380ca440 msgr2=0x7f6c381f1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c37fff700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c380ca440 0x7f6c381f1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c37fff700 1 -- 172.21.15.102:0/3791873753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280094a0 con 0x7f6c381564b0 2023-03-03T16:30:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c37fff700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c381564b0 0x7f6c381f2530 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000cca0 tx=0x7f6c3000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.509+0000 7f6c34ff9700 1 -- 172.21.15.102:0/3791873753 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3000aaf0 con 0x7f6c381564b0 2023-03-03T16:30:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.510+0000 7f6c34ff9700 1 -- 172.21.15.102:0/3791873753 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3000ac70 con 0x7f6c381564b0 2023-03-03T16:30:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.510+0000 7f6c34ff9700 1 -- 172.21.15.102:0/3791873753 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c300070d0 con 0x7f6c381564b0 2023-03-03T16:30:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.510+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c380b2210 con 0x7f6c381564b0 2023-03-03T16:30:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.510+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c380b1f30 con 0x7f6c381564b0 2023-03-03T16:30:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.511+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c38156880 con 0x7f6c381564b0 2023-03-03T16:30:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.512+0000 7f6c34ff9700 1 -- 172.21.15.102:0/3791873753 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c30010070 con 0x7f6c381564b0 2023-03-03T16:30:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.513+0000 7f6c34ff9700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.513+0000 7f6c34ff9700 1 -- 172.21.15.102:0/3791873753 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6c300857d0 con 0x7f6c381564b0 2023-03-03T16:30:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.514+0000 7f6c377fe700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.514+0000 7f6c377fe700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f6c280090b0 tx=0x7f6c2801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:17.517 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.517+0000 7f6c34ff9700 1 -- 172.21.15.102:0/3791873753 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c30051e50 con 0x7f6c381564b0 2023-03-03T16:30:17.750 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.751+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c380c7450 con 0x7f6c381564b0 2023-03-03T16:30:17.751 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.751+0000 7f6c34ff9700 1 -- 172.21.15.102:0/3791873753 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6c30018070 con 0x7f6c381564b0 2023-03-03T16:30:17.751 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:30:17.752 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:30:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 msgr2=0x7f6c1806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f6c280090b0 tx=0x7f6c2801c040 comp rx=0 tx=0).stop 2023-03-03T16:30:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c381564b0 msgr2=0x7f6c381f2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c381564b0 0x7f6c381f2530 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000cca0 tx=0x7f6c3000a060 comp rx=0 tx=0).stop 2023-03-03T16:30:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 shutdown_connections 2023-03-03T16:30:17.754 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6c18068320 0x7f6c1806a7d0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6c380c9610 0x7f6c381e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c381564b0 0x7f6c381f2530 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 --2- 172.21.15.102:0/3791873753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c380ca440 0x7f6c381f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 >> 172.21.15.102:0/3791873753 conn(0x7f6c38065fb0 msgr2=0x7f6c38155140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 shutdown_connections 2023-03-03T16:30:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:17.754+0000 7f6c3cf9d700 1 -- 172.21.15.102:0/3791873753 wait complete. 2023-03-03T16:30:17.756 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:30:17.781 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T16:30:22.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:22.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:52.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:52.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:56.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:30:57.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.032+0000 7fb20759d700 1 Processor -- start 2023-03-03T16:30:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.033+0000 7fb20759d700 1 -- start start 2023-03-03T16:30:57.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.034+0000 7fb20759d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb200150ac0 0x7fb20006efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:57.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.034+0000 7fb20759d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2001569f0 con 0x7fb2000c7a30 2023-03-03T16:30:57.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.034+0000 7fb20759d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2000c28a0 con 0x7fb2000c7660 2023-03-03T16:30:57.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.034+0000 7fb20759d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20006e1e0 con 0x7fb200150ac0 2023-03-03T16:30:57.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb205d9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb2000c7a30 0x7fb2000c1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36118/0 (socket says 172.21.15.102:36118) 2023-03-03T16:30:57.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb205d9a700 1 -- 172.21.15.102:0/2133288461 learned_addr learned my addr 172.21.15.102:0/2133288461 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:30:57.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb206d9c700 1 --2- 172.21.15.102:0/2133288461 >> v2:172.21.15.172:3300/0 conn(0x7fb200150ac0 0x7fb20006efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:57.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb205599700 1 -- 172.21.15.102:0/2133288461 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523136685 0 0) 0x7fb2001569f0 con 0x7fb2000c7a30 2023-03-03T16:30:57.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb205599700 1 -- 172.21.15.102:0/2133288461 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ec0035b0 con 0x7fb2000c7a30 2023-03-03T16:30:57.036 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb206d9c700 1 -- 172.21.15.102:0/2133288461 >> v1:172.21.15.172:6789/0 conn(0x7fb2000c7660 legacy=0x7fb2001503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb205599700 1 -- 172.21.15.102:0/2133288461 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966352616 0 0) 0x7fb2000c28a0 con 0x7fb2000c7660 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb206d9c700 1 -- 172.21.15.102:0/2133288461 >> v1:172.21.15.102:6789/0 conn(0x7fb2000c7a30 legacy=0x7fb2000c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.035+0000 7fb206d9c700 1 -- 172.21.15.102:0/2133288461 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20006e4c0 con 0x7fb200150ac0 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.036+0000 7fb206d9c700 1 --2- 172.21.15.102:0/2133288461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb20006efa0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009d90 tx=0x7fb1f800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7337023b740cd620 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.036+0000 7fb205599700 1 -- 172.21.15.102:0/2133288461 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800e070 con 0x7fb200150ac0 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.036+0000 7fb205599700 1 -- 172.21.15.102:0/2133288461 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f800bb20 con 0x7fb200150ac0 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.036+0000 7fb205599700 1 -- 172.21.15.102:0/2133288461 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800bed0 con 0x7fb200150ac0 2023-03-03T16:30:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.036+0000 7fb20759d700 1 -- 172.21.15.102:0/2133288461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 msgr2=0x7fb20006efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.036+0000 7fb20759d700 1 --2- 172.21.15.102:0/2133288461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb20006efa0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009d90 tx=0x7fb1f800b330 comp rx=0 tx=0).stop 2023-03-03T16:30:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.037+0000 7fb20759d700 1 -- 172.21.15.102:0/2133288461 shutdown_connections 2023-03-03T16:30:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.037+0000 7fb20759d700 1 --2- 172.21.15.102:0/2133288461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb20006efa0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.037+0000 7fb20759d700 1 -- 172.21.15.102:0/2133288461 >> 172.21.15.102:0/2133288461 conn(0x7fb200070300 msgr2=0x7fb200070700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.037+0000 7fb20759d700 1 -- 172.21.15.102:0/2133288461 shutdown_connections 2023-03-03T16:30:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.037+0000 7fb20759d700 1 -- 172.21.15.102:0/2133288461 wait complete. 2023-03-03T16:30:57.039 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.039+0000 7fb20759d700 1 Processor -- start 2023-03-03T16:30:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.039+0000 7fb20759d700 1 -- start start 2023-03-03T16:30:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb20759d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2000c7660 0x7fb2001545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb20759d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2000c7a30 0x7fb200157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb20759d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb200157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb20759d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20015a990 con 0x7fb2000c7a30 2023-03-03T16:30:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb20759d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20006bf30 con 0x7fb200150ac0 2023-03-03T16:30:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb20759d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb200158440 con 0x7fb2000c7660 2023-03-03T16:30:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb205d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2000c7a30 0x7fb200157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb205d9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2000c7a30 0x7fb200157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33594/0 (socket says 172.21.15.102:33594) 2023-03-03T16:30:57.041 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb205d9a700 1 -- 172.21.15.102:0/3421423446 learned_addr learned my addr 172.21.15.102:0/3421423446 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:30:57.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb206d9c700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb200157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:57.042 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.040+0000 7fb20659b700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2000c7660 0x7fb2001545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb206d9c700 1 -- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2000c7660 msgr2=0x7fb2001545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb206d9c700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2000c7660 0x7fb2001545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb206d9c700 1 -- 172.21.15.102:0/3421423446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2000c7a30 msgr2=0x7fb200157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb206d9c700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2000c7a30 0x7fb200157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb206d9c700 1 -- 172.21.15.102:0/3421423446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8009a70 con 0x7fb200150ac0 2023-03-03T16:30:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb205d9a700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2000c7a30 0x7fb200157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb206d9c700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb200157dd0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb2000c3d90 tx=0x7fb1f8004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb1f6ffd700 1 -- 172.21.15.102:0/3421423446 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f801c810 con 0x7fb200150ac0 2023-03-03T16:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.041+0000 7fb1f6ffd700 1 -- 172.21.15.102:0/3421423446 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f8020070 con 0x7fb200150ac0 2023-03-03T16:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.042+0000 7fb1f6ffd700 1 -- 172.21.15.102:0/3421423446 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f801cbd0 con 0x7fb200150ac0 2023-03-03T16:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.042+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20006ea80 con 0x7fb200150ac0 2023-03-03T16:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.042+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20006e7a0 con 0x7fb200150ac0 2023-03-03T16:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.043+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2000c88c0 con 0x7fb200150ac0 2023-03-03T16:30:57.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.044+0000 7fb1f6ffd700 1 -- 172.21.15.102:0/3421423446 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb1f802e070 con 0x7fb200150ac0 2023-03-03T16:30:57.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.045+0000 7fb1f6ffd700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1e40682f0 0x7fb1e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:57.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.045+0000 7fb1f6ffd700 1 -- 172.21.15.102:0/3421423446 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb1f8022070 con 0x7fb200150ac0 2023-03-03T16:30:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.046+0000 7fb20659b700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1e40682f0 0x7fb1e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:57.046 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.046+0000 7fb20659b700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1e40682f0 0x7fb1e406a7a0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc007fd0 tx=0x7fb1fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:57.050 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.050+0000 7fb1f6ffd700 1 -- 172.21.15.102:0/3421423446 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb1f80543f0 con 0x7fb200150ac0 2023-03-03T16:30:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.287+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2000677f0 con 0x7fb200150ac0 2023-03-03T16:30:57.289 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.290+0000 7fb1f6ffd700 1 -- 172.21.15.102:0/3421423446 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb1f802c090 con 0x7fb200150ac0 2023-03-03T16:30:57.290 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:30:57.290 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:30:57.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1e40682f0 msgr2=0x7fb1e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:57.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1e40682f0 0x7fb1e406a7a0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc007fd0 tx=0x7fb1fc009040 comp rx=0 tx=0).stop 2023-03-03T16:30:57.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 msgr2=0x7fb200157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:57.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb200157dd0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb2000c3d90 tx=0x7fb1f8004ad0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 shutdown_connections 2023-03-03T16:30:57.294 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb1e40682f0 0x7fb1e406a7a0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb2000c7660 0x7fb2001545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb200150ac0 0x7fb200157dd0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 --2- 172.21.15.102:0/3421423446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2000c7a30 0x7fb200157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:57.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 >> 172.21.15.102:0/3421423446 conn(0x7fb200070300 msgr2=0x7fb200155370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:57.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 shutdown_connections 2023-03-03T16:30:57.295 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:30:57.294+0000 7fb20759d700 1 -- 172.21.15.102:0/3421423446 wait complete. 2023-03-03T16:30:57.296 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:30:57.324 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T16:31:23.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:23.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:36.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:31:36.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.574+0000 7fbfdbdff700 1 Processor -- start 2023-03-03T16:31:36.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.574+0000 7fbfdbdff700 1 -- start start 2023-03-03T16:31:36.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfdbdff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbfd414fbe0 0x7fbfd40c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:36.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfdbdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd40c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:36.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfdbdff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd40c3c80 con 0x7fbfd40716c0 2023-03-03T16:31:36.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfdbdff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd40c41a0 con 0x7fbfd414fbe0 2023-03-03T16:31:36.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfdbdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4071cd0 con 0x7fbfd414ffb0 2023-03-03T16:31:36.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfd9b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbfd414fbe0 0x7fbfd40c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:36.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfda39c700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbfd40716c0 0x7fbfd40706f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58570/0 (socket says 172.21.15.102:58570) 2023-03-03T16:31:36.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfda39c700 1 -- 172.21.15.102:0/1616127023 learned_addr learned my addr 172.21.15.102:0/1616127023 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:31:36.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.576+0000 7fbfd939a700 1 --2- 172.21.15.102:0/1616127023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd40c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:36.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd939a700 1 -- 172.21.15.102:0/1616127023 >> v2:172.21.15.102:3300/0 conn(0x7fbfd414fbe0 msgr2=0x7fbfd40c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:31:36.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd939a700 1 --2- 172.21.15.102:0/1616127023 >> v2:172.21.15.102:3300/0 conn(0x7fbfd414fbe0 0x7fbfd40c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd939a700 1 -- 172.21.15.102:0/1616127023 >> v1:172.21.15.102:6789/0 conn(0x7fbfd40716c0 legacy=0x7fbfd40706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:36.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd939a700 1 -- 172.21.15.102:0/1616127023 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40c3ec0 con 0x7fbfd414ffb0 2023-03-03T16:31:36.578 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd8b99700 1 -- 172.21.15.102:0/1616127023 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276906118 0 0) 0x7fbfcc001490 con 0x7fbfd40716c0 2023-03-03T16:31:36.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd939a700 1 --2- 172.21.15.102:0/1616127023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd40c45a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009840 tx=0x7fbfc4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e304a30cfecd55f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:36.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd8b99700 1 -- 172.21.15.102:0/1616127023 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4011830 con 0x7fbfd414ffb0 2023-03-03T16:31:36.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.577+0000 7fbfd8b99700 1 -- 172.21.15.102:0/1616127023 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc4011990 con 0x7fbfd414ffb0 2023-03-03T16:31:36.579 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfd8b99700 1 -- 172.21.15.102:0/1616127023 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4011d20 con 0x7fbfd414ffb0 2023-03-03T16:31:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 -- 172.21.15.102:0/1616127023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 msgr2=0x7fbfd40c45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/1616127023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd40c45a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009840 tx=0x7fbfc4009b50 comp rx=0 tx=0).stop 2023-03-03T16:31:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 -- 172.21.15.102:0/1616127023 shutdown_connections 2023-03-03T16:31:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/1616127023 >> v2:172.21.15.102:3300/0 conn(0x7fbfd414fbe0 0x7fbfd40c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/1616127023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd40c45a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 -- 172.21.15.102:0/1616127023 >> 172.21.15.102:0/1616127023 conn(0x7fbfd406af90 msgr2=0x7fbfd406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:31:36.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 -- 172.21.15.102:0/1616127023 shutdown_connections 2023-03-03T16:31:36.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.578+0000 7fbfdbdff700 1 -- 172.21.15.102:0/1616127023 wait complete. 2023-03-03T16:31:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.580+0000 7fbfdbdff700 1 Processor -- start 2023-03-03T16:31:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.581+0000 7fbfdbdff700 1 -- start start 2023-03-03T16:31:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfdbdff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd40716c0 0x7fbfd40c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:36.582 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfdbdff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfd414fbe0 0x7fbfd40ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfdbdff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd413d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfdbdff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd41469e0 con 0x7fbfd40716c0 2023-03-03T16:31:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfdbdff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4146c50 con 0x7fbfd414fbe0 2023-03-03T16:31:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfdbdff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd41418f0 con 0x7fbfd414ffb0 2023-03-03T16:31:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd40716c0 0x7fbfd40c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd40716c0 0x7fbfd40c87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38426/0 (socket says 172.21.15.102:38426) 2023-03-03T16:31:36.583 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 -- 172.21.15.102:0/3763597282 learned_addr learned my addr 172.21.15.102:0/3763597282 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:31:36.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfda39c700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd413d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:36.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd939a700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfd414fbe0 0x7fbfd40ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:36.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 -- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 msgr2=0x7fbfd413d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:36.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd413d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 -- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfd414fbe0 msgr2=0x7fbfd40ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:36.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfd414fbe0 0x7fbfd40ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.582+0000 7fbfd9b9b700 1 -- 172.21.15.102:0/3763597282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc40094a0 con 0x7fbfd40716c0 2023-03-03T16:31:36.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.583+0000 7fbfd9b9b700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd40716c0 0x7fbfd40c87f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000e9b0 tx=0x7fbfd000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:36.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.583+0000 7fbfca7fc700 1 -- 172.21.15.102:0/3763597282 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0009710 con 0x7fbfd40716c0 2023-03-03T16:31:36.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.583+0000 7fbfca7fc700 1 -- 172.21.15.102:0/3763597282 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd000ee50 con 0x7fbfd40716c0 2023-03-03T16:31:36.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.583+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd4072250 con 0x7fbfd40716c0 2023-03-03T16:31:36.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.583+0000 7fbfca7fc700 1 -- 172.21.15.102:0/3763597282 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000ca50 con 0x7fbfd40716c0 2023-03-03T16:31:36.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.583+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd4071f70 con 0x7fbfd40716c0 2023-03-03T16:31:36.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.584+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd4072530 con 0x7fbfd40716c0 2023-03-03T16:31:36.586 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.585+0000 7fbfca7fc700 1 -- 172.21.15.102:0/3763597282 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbfd0019030 con 0x7fbfd40716c0 2023-03-03T16:31:36.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.587+0000 7fbfca7fc700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfb80682f0 0x7fbfb806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:36.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.587+0000 7fbfca7fc700 1 -- 172.21.15.102:0/3763597282 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbfd0085c60 con 0x7fbfd40716c0 2023-03-03T16:31:36.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.587+0000 7fbfd939a700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfb80682f0 0x7fbfb806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:36.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.588+0000 7fbfd939a700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfb80682f0 0x7fbfb806a7a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40090b0 tx=0x7fbfc401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:36.591 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.591+0000 7fbfca7fc700 1 -- 172.21.15.102:0/3763597282 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbfd0052370 con 0x7fbfd40716c0 2023-03-03T16:31:36.829 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.829+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfd413ef40 con 0x7fbfd40716c0 2023-03-03T16:31:36.830 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.830+0000 7fbfca7fc700 1 -- 172.21.15.102:0/3763597282 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbfd0055990 con 0x7fbfd40716c0 2023-03-03T16:31:36.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:31:36.830 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:31:36.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfb80682f0 msgr2=0x7fbfb806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:36.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfb80682f0 0x7fbfb806a7a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40090b0 tx=0x7fbfc401c040 comp rx=0 tx=0).stop 2023-03-03T16:31:36.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd40716c0 msgr2=0x7fbfd40c87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:36.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd40716c0 0x7fbfd40c87f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000e9b0 tx=0x7fbfd000ecc0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 shutdown_connections 2023-03-03T16:31:36.833 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfb80682f0 0x7fbfb806a7a0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfd414ffb0 0x7fbfd413d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd40716c0 0x7fbfd40c87f0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 --2- 172.21.15.102:0/3763597282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfd414fbe0 0x7fbfd40ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 >> 172.21.15.102:0/3763597282 conn(0x7fbfd406af90 msgr2=0x7fbfd406cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:31:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 shutdown_connections 2023-03-03T16:31:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:31:36.833+0000 7fbfdbdff700 1 -- 172.21.15.102:0/3763597282 wait complete. 2023-03-03T16:31:36.835 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:31:36.860 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2023-03-03T16:31:53.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:53.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:23.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:23.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:53.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:53.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:23.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:23.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:33.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:33:34.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.108+0000 7f35a3fff700 1 Processor -- start 2023-03-03T16:33:34.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.108+0000 7f35a3fff700 1 -- start start 2023-03-03T16:33:34.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.110+0000 7f35a3fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f35a40b3970 0x7f35a40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:33:34.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.110+0000 7f35a3fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40c3d20 con 0x7f35a40c41b0 2023-03-03T16:33:34.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.110+0000 7f35a3fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a415cba0 con 0x7f35a40ced00 2023-03-03T16:33:34.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.110+0000 7f35a3fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40c4930 con 0x7f35a40b3970 2023-03-03T16:33:34.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.110+0000 7f35a2ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f35a40ced00 0x7f35a4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:39426/0 (socket says 172.21.15.102:39426) 2023-03-03T16:33:34.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.110+0000 7f35a2ffd700 1 -- 172.21.15.102:0/781804225 learned_addr learned my addr 172.21.15.102:0/781804225 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.110+0000 7f35a27fc700 1 --2- 172.21.15.102:0/781804225 >> v2:172.21.15.102:3300/0 conn(0x7f35a40b3970 0x7f35a40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a1ffb700 1 -- 172.21.15.102:0/781804225 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932463711 0 0) 0x7f35a415cba0 con 0x7f35a40ced00 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a1ffb700 1 -- 172.21.15.102:0/781804225 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c0035b0 con 0x7f35a40ced00 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a1ffb700 1 -- 172.21.15.102:0/781804225 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100863166 0 0) 0x7f35a40c3d20 con 0x7f35a40c41b0 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a1ffb700 1 -- 172.21.15.102:0/781804225 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a415cba0 con 0x7f35a40c41b0 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a27fc700 1 -- 172.21.15.102:0/781804225 >> v1:172.21.15.172:6789/0 conn(0x7f35a40ced00 legacy=0x7f35a4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a27fc700 1 -- 172.21.15.102:0/781804225 >> v1:172.21.15.102:6789/0 conn(0x7f35a40c41b0 legacy=0x7f35a4163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:33:34.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a27fc700 1 -- 172.21.15.102:0/781804225 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a415ce80 con 0x7f35a40b3970 2023-03-03T16:33:34.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.111+0000 7f35a27fc700 1 --2- 172.21.15.102:0/781804225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40c5a90 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f3598009840 tx=0x7f3598009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3324e77aab219f9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:33:34.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a1ffb700 1 -- 172.21.15.102:0/781804225 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598011830 con 0x7f35a40b3970 2023-03-03T16:33:34.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a1ffb700 1 -- 172.21.15.102:0/781804225 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3598011990 con 0x7f35a40b3970 2023-03-03T16:33:34.113 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a1ffb700 1 -- 172.21.15.102:0/781804225 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598011d20 con 0x7f35a40b3970 2023-03-03T16:33:34.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a3fff700 1 -- 172.21.15.102:0/781804225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 msgr2=0x7f35a40c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:33:34.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a3fff700 1 --2- 172.21.15.102:0/781804225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40c5a90 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f3598009840 tx=0x7f3598009b50 comp rx=0 tx=0).stop 2023-03-03T16:33:34.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a3fff700 1 -- 172.21.15.102:0/781804225 shutdown_connections 2023-03-03T16:33:34.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a3fff700 1 --2- 172.21.15.102:0/781804225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40c5a90 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:33:34.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a3fff700 1 -- 172.21.15.102:0/781804225 >> 172.21.15.102:0/781804225 conn(0x7f35a408b1c0 msgr2=0x7f35a4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:33:34.114 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a3fff700 1 -- 172.21.15.102:0/781804225 shutdown_connections 2023-03-03T16:33:34.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.112+0000 7f35a3fff700 1 -- 172.21.15.102:0/781804225 wait complete. 2023-03-03T16:33:34.115 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.114+0000 7f35a3fff700 1 Processor -- start 2023-03-03T16:33:34.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.115+0000 7f35a3fff700 1 -- start start 2023-03-03T16:33:34.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:33:34.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35a40c41b0 0x7f35a40c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:33:34.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35a40ced00 0x7f35a41e5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:33:34.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a415bd40 con 0x7f35a40b3970 2023-03-03T16:33:34.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40c6fb0 con 0x7f35a40c41b0 2023-03-03T16:33:34.116 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a415b0f0 con 0x7f35a40ced00 2023-03-03T16:33:34.117 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:33:34.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41228/0 (socket says 172.21.15.102:41228) 2023-03-03T16:33:34.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a2ffd700 1 -- 172.21.15.102:0/4274530594 learned_addr learned my addr 172.21.15.102:0/4274530594 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:33:34.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a27fc700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35a40c41b0 0x7f35a40c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:33:34.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.116+0000 7f35a37fe700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35a40ced00 0x7f35a41e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:33:34.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a2ffd700 1 -- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35a40ced00 msgr2=0x7f35a41e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:33:34.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a2ffd700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35a40ced00 0x7f35a41e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:33:34.118 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a2ffd700 1 -- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35a40c41b0 msgr2=0x7f35a40c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:33:34.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a2ffd700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35a40c41b0 0x7f35a40c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:33:34.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a2ffd700 1 -- 172.21.15.102:0/4274530594 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35980094a0 con 0x7f35a40b3970 2023-03-03T16:33:34.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a2ffd700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40b2100 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f359400c730 tx=0x7f359400ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:33:34.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f358b7fe700 1 -- 172.21.15.102:0/4274530594 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594005d70 con 0x7f35a40b3970 2023-03-03T16:33:34.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f358b7fe700 1 -- 172.21.15.102:0/4274530594 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35940070d0 con 0x7f35a40b3970 2023-03-03T16:33:34.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f358b7fe700 1 -- 172.21.15.102:0/4274530594 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594007440 con 0x7f35a40b3970 2023-03-03T16:33:34.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a406c200 con 0x7f35a40b3970 2023-03-03T16:33:34.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.117+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a406bf20 con 0x7f35a40b3970 2023-03-03T16:33:34.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.118+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a415c890 con 0x7f35a40b3970 2023-03-03T16:33:34.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.119+0000 7f358b7fe700 1 -- 172.21.15.102:0/4274530594 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3594019030 con 0x7f35a40b3970 2023-03-03T16:33:34.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.121+0000 7f358b7fe700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35900682f0 0x7f359006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:33:34.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.121+0000 7f358b7fe700 1 -- 172.21.15.102:0/4274530594 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3594084fb0 con 0x7f35a40b3970 2023-03-03T16:33:34.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.121+0000 7f35a27fc700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35900682f0 0x7f359006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:33:34.121 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.122+0000 7f35a27fc700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35900682f0 0x7f359006a7a0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f3598009fd0 tx=0x7f359801c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:33:34.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.125+0000 7f358b7fe700 1 -- 172.21.15.102:0/4274530594 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f359404b5e0 con 0x7f35a40b3970 2023-03-03T16:33:34.362 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.362+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a415b570 con 0x7f35a40b3970 2023-03-03T16:33:34.363 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.363+0000 7f358b7fe700 1 -- 172.21.15.102:0/4274530594 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3594017090 con 0x7f35a40b3970 2023-03-03T16:33:34.363 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:33:34.363 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:33:34.365 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.365+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35900682f0 msgr2=0x7f359006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:33:34.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.365+0000 7f35a3fff700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35900682f0 0x7f359006a7a0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f3598009fd0 tx=0x7f359801c040 comp rx=0 tx=0).stop 2023-03-03T16:33:34.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.365+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 msgr2=0x7f35a40b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:33:34.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.365+0000 7f35a3fff700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40b2100 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f359400c730 tx=0x7f359400ca40 comp rx=0 tx=0).stop 2023-03-03T16:33:34.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 shutdown_connections 2023-03-03T16:33:34.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f35900682f0 0x7f359006a7a0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:33:34.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35a40ced00 0x7f35a41e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:33:34.366 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35a40b3970 0x7f35a40b2100 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:33:34.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 --2- 172.21.15.102:0/4274530594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35a40c41b0 0x7f35a40c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:33:34.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 >> 172.21.15.102:0/4274530594 conn(0x7f35a408b1c0 msgr2=0x7f35a40cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:33:34.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 shutdown_connections 2023-03-03T16:33:34.367 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:33:34.366+0000 7f35a3fff700 1 -- 172.21.15.102:0/4274530594 wait complete. 2023-03-03T16:33:34.367 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:33:34.393 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2023-03-03T16:33:53.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:53.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:23.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:23.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:54.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:54.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:55.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:34:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.643+0000 7f34e79ff700 1 Processor -- start 2023-03-03T16:34:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.644+0000 7f34e79ff700 1 -- start start 2023-03-03T16:34:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e79ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e0153290 0x7f34e0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e79ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f34e00cd8e0 0x7f34e00ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e79ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e0153660 con 0x7f34e01593c0 2023-03-03T16:34:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e79ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e006c360 con 0x7f34e00cd8e0 2023-03-03T16:34:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e79ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e015cea0 con 0x7f34e0153290 2023-03-03T16:34:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e4f9a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f34e01593c0 0x7f34e00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50560/0 (socket says 172.21.15.102:50560) 2023-03-03T16:34:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e4f9a700 1 -- 172.21.15.102:0/706425298 learned_addr learned my addr 172.21.15.102:0/706425298 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:34:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e5f9c700 1 --2- 172.21.15.102:0/706425298 >> v2:172.21.15.172:3300/0 conn(0x7f34e00cd8e0 0x7f34e00ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.645+0000 7f34e579b700 1 --2- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e0153290 0x7f34e0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34d7fff700 1 -- 172.21.15.102:0/706425298 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195054257 0 0) 0x7f34e0153660 con 0x7f34e01593c0 2023-03-03T16:34:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34d7fff700 1 -- 172.21.15.102:0/706425298 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34cc0035b0 con 0x7f34e01593c0 2023-03-03T16:34:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34e5f9c700 1 -- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e0153290 msgr2=0x7f34e0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:34:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34e5f9c700 1 --2- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e0153290 0x7f34e0158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34e5f9c700 1 -- 172.21.15.102:0/706425298 >> v1:172.21.15.102:6789/0 conn(0x7f34e01593c0 legacy=0x7f34e00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34e5f9c700 1 -- 172.21.15.102:0/706425298 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e006c7f0 con 0x7f34e00cd8e0 2023-03-03T16:34:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34e579b700 1 --2- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e0153290 0x7f34e0158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:34:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34e5f9c700 1 --2- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 0x7f34e00ce530 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009d90 tx=0x7f34d800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d8caab8a60674da server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.646+0000 7f34d7fff700 1 -- 172.21.15.102:0/706425298 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d800e070 con 0x7f34e00cd8e0 2023-03-03T16:34:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34d7fff700 1 -- 172.21.15.102:0/706425298 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d800bb20 con 0x7f34e00cd8e0 2023-03-03T16:34:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34d7fff700 1 -- 172.21.15.102:0/706425298 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8012430 con 0x7f34e00cd8e0 2023-03-03T16:34:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 -- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 msgr2=0x7f34e00ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 --2- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 0x7f34e00ce530 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009d90 tx=0x7f34d800b330 comp rx=0 tx=0).stop 2023-03-03T16:34:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 -- 172.21.15.102:0/706425298 shutdown_connections 2023-03-03T16:34:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 --2- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 0x7f34e00ce530 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 --2- 172.21.15.102:0/706425298 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e0153290 0x7f34e0158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 -- 172.21.15.102:0/706425298 >> 172.21.15.102:0/706425298 conn(0x7f34e0065fb0 msgr2=0x7f34e00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:34:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 -- 172.21.15.102:0/706425298 shutdown_connections 2023-03-03T16:34:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.647+0000 7f34e79ff700 1 -- 172.21.15.102:0/706425298 wait complete. 2023-03-03T16:34:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.649+0000 7f34e79ff700 1 Processor -- start 2023-03-03T16:34:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.649+0000 7f34e79ff700 1 -- start start 2023-03-03T16:34:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.650+0000 7f34e79ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 0x7f34e00c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.650+0000 7f34e79ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34e0153290 0x7f34e0153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.650+0000 7f34e79ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e01593c0 0x7f34e00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.650+0000 7f34e79ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e00c3d20 con 0x7f34e0153290 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.650+0000 7f34e79ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e00c8380 con 0x7f34e00cd8e0 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.650+0000 7f34e79ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e00c8660 con 0x7f34e01593c0 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34e0153290 0x7f34e0153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34e0153290 0x7f34e0153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56000/0 (socket says 172.21.15.102:56000) 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 -- 172.21.15.102:0/1273648461 learned_addr learned my addr 172.21.15.102:0/1273648461 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e579b700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 0x7f34e00c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e5f9c700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e01593c0 0x7f34e00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 -- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e01593c0 msgr2=0x7f34e00c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e01593c0 0x7f34e00c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 -- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 msgr2=0x7f34e00c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 0x7f34e00c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 -- 172.21.15.102:0/1273648461 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc0094a0 con 0x7f34e0153290 2023-03-03T16:34:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.651+0000 7f34e4f9a700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34e0153290 0x7f34e0153a60 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f34d000c700 tx=0x7f34d000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.652+0000 7f34d5ffb700 1 -- 172.21.15.102:0/1273648461 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0005d70 con 0x7f34e0153290 2023-03-03T16:34:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.652+0000 7f34d5ffb700 1 -- 172.21.15.102:0/1273648461 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d00070d0 con 0x7f34e0153290 2023-03-03T16:34:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.652+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d8009a70 con 0x7f34e0153290 2023-03-03T16:34:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.652+0000 7f34d5ffb700 1 -- 172.21.15.102:0/1273648461 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0007460 con 0x7f34e0153290 2023-03-03T16:34:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.652+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e00ccde0 con 0x7f34e0153290 2023-03-03T16:34:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.653+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e015d160 con 0x7f34e0153290 2023-03-03T16:34:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.654+0000 7f34d5ffb700 1 -- 172.21.15.102:0/1273648461 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f34d0019030 con 0x7f34e0153290 2023-03-03T16:34:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.656+0000 7f34d5ffb700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f34c40682f0 0x7f34c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.656+0000 7f34d5ffb700 1 -- 172.21.15.102:0/1273648461 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f34d0085140 con 0x7f34e0153290 2023-03-03T16:34:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.656+0000 7f34e579b700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f34c40682f0 0x7f34c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:55.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.657+0000 7f34e579b700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f34c40682f0 0x7f34c406a7a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009d70 tx=0x7f34dc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:55.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.660+0000 7f34d5ffb700 1 -- 172.21.15.102:0/1273648461 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f34d0051760 con 0x7f34e0153290 2023-03-03T16:34:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.893+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34e00cd0c0 con 0x7f34e0153290 2023-03-03T16:34:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.894+0000 7f34d5ffb700 1 -- 172.21.15.102:0/1273648461 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f34d0017020 con 0x7f34e0153290 2023-03-03T16:34:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:34:55.894 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:34:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f34c40682f0 msgr2=0x7f34c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f34c40682f0 0x7f34c406a7a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009d70 tx=0x7f34dc009210 comp rx=0 tx=0).stop 2023-03-03T16:34:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34e0153290 msgr2=0x7f34e0153a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34e0153290 0x7f34e0153a60 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f34d000c700 tx=0x7f34d000ca10 comp rx=0 tx=0).stop 2023-03-03T16:34:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 shutdown_connections 2023-03-03T16:34:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f34c40682f0 0x7f34c406a7a0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f34e01593c0 0x7f34e00c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34e00cd8e0 0x7f34e00c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 --2- 172.21.15.102:0/1273648461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34e0153290 0x7f34e0153a60 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 >> 172.21.15.102:0/1273648461 conn(0x7f34e0065fb0 msgr2=0x7f34e015be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:34:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 shutdown_connections 2023-03-03T16:34:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:34:55.897+0000 7f34e79ff700 1 -- 172.21.15.102:0/1273648461 wait complete. 2023-03-03T16:34:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:34:55.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 20 secs before thrashing 2023-03-03T16:35:15.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:35:16.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.173+0000 7f4fe239d700 1 Processor -- start 2023-03-03T16:35:16.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.173+0000 7f4fe239d700 1 -- start start 2023-03-03T16:35:16.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe239d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4fdc0b3970 0x7f4fdc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe239d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4fdc0c41b0 0x7f4fdc0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe239d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fdc0c3d20 con 0x7f4fdc0ced00 2023-03-03T16:35:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe239d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc0c4930 con 0x7f4fdc0b3970 2023-03-03T16:35:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe239d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc0cf610 con 0x7f4fdc0c41b0 2023-03-03T16:35:16.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe0b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4fdc0b3970 0x7f4fdc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:16.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe0b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4fdc0b3970 0x7f4fdc0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50114/0 (socket says 172.21.15.102:50114) 2023-03-03T16:35:16.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe0b9a700 1 -- 172.21.15.102:0/4287682822 learned_addr learned my addr 172.21.15.102:0/4287682822 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:35:16.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe139b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4fdc0ced00 0x7f4fdc158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59094/0 (socket says 172.21.15.102:59094) 2023-03-03T16:35:16.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.175+0000 7f4fe1b9c700 1 --2- 172.21.15.102:0/4287682822 >> v2:172.21.15.172:3300/0 conn(0x7f4fdc0c41b0 0x7f4fdc0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:16.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fe0b9a700 1 -- 172.21.15.102:0/4287682822 >> v2:172.21.15.172:3300/0 conn(0x7f4fdc0c41b0 msgr2=0x7f4fdc0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:35:16.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fe0b9a700 1 --2- 172.21.15.102:0/4287682822 >> v2:172.21.15.172:3300/0 conn(0x7f4fdc0c41b0 0x7f4fdc0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fe0b9a700 1 -- 172.21.15.102:0/4287682822 >> v1:172.21.15.102:6789/0 conn(0x7f4fdc0ced00 legacy=0x7f4fdc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fe0b9a700 1 -- 172.21.15.102:0/4287682822 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc15cba0 con 0x7f4fdc0b3970 2023-03-03T16:35:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fd3fff700 1 -- 172.21.15.102:0/4287682822 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671718042 0 0) 0x7f4fdc0c3d20 con 0x7f4fdc0ced00 2023-03-03T16:35:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fe0b9a700 1 --2- 172.21.15.102:0/4287682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc0c5a90 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc08ce30 tx=0x7f4fcc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffa270dc04cdd18 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fd3fff700 1 -- 172.21.15.102:0/4287682822 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc011830 con 0x7f4fdc0b3970 2023-03-03T16:35:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fd3fff700 1 -- 172.21.15.102:0/4287682822 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc011990 con 0x7f4fdc0b3970 2023-03-03T16:35:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.176+0000 7f4fd3fff700 1 -- 172.21.15.102:0/4287682822 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc011d20 con 0x7f4fdc0b3970 2023-03-03T16:35:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 -- 172.21.15.102:0/4287682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0b3970 msgr2=0x7f4fdc0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 --2- 172.21.15.102:0/4287682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc0c5a90 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc08ce30 tx=0x7f4fcc009c00 comp rx=0 tx=0).stop 2023-03-03T16:35:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 -- 172.21.15.102:0/4287682822 shutdown_connections 2023-03-03T16:35:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 --2- 172.21.15.102:0/4287682822 >> v2:172.21.15.172:3300/0 conn(0x7f4fdc0c41b0 0x7f4fdc0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 --2- 172.21.15.102:0/4287682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc0c5a90 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 -- 172.21.15.102:0/4287682822 >> 172.21.15.102:0/4287682822 conn(0x7f4fdc08b1c0 msgr2=0x7f4fdc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 -- 172.21.15.102:0/4287682822 shutdown_connections 2023-03-03T16:35:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.177+0000 7f4fe239d700 1 -- 172.21.15.102:0/4287682822 wait complete. 2023-03-03T16:35:16.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.179+0000 7f4fe239d700 1 Processor -- start 2023-03-03T16:35:16.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.179+0000 7f4fe239d700 1 -- start start 2023-03-03T16:35:16.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.180+0000 7f4fe239d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc1e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.180+0000 7f4fe239d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0c41b0 0x7f4fdc151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.180+0000 7f4fe239d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fdc0ced00 0x7f4fdc0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.180+0000 7f4fe239d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc1e4e80 con 0x7f4fdc0c41b0 2023-03-03T16:35:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.180+0000 7f4fe239d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc1e3cc0 con 0x7f4fdc0b3970 2023-03-03T16:35:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.180+0000 7f4fe239d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc1f2710 con 0x7f4fdc0ced00 2023-03-03T16:35:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe139b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc1e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe139b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc1e4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:58278/0 (socket says 172.21.15.102:58278) 2023-03-03T16:35:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe139b700 1 -- 172.21.15.102:0/3079658850 learned_addr learned my addr 172.21.15.102:0/3079658850 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:35:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe0b9a700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0c41b0 0x7f4fdc151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe1b9c700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fdc0ced00 0x7f4fdc0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe1b9c700 1 -- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fdc0b3970 msgr2=0x7f4fdc1e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe1b9c700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc1e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe1b9c700 1 -- 172.21.15.102:0/3079658850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0c41b0 msgr2=0x7f4fdc151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe1b9c700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0c41b0 0x7f4fdc151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe1b9c700 1 -- 172.21.15.102:0/3079658850 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc009550 con 0x7f4fdc0ced00 2023-03-03T16:35:16.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe0b9a700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0c41b0 0x7f4fdc151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:35:16.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe139b700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc1e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:35:16.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.181+0000 7f4fe1b9c700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fdc0ced00 0x7f4fdc0ccea0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400ec00 tx=0x7f4fd400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:16.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.182+0000 7f4fd1ffb700 1 -- 172.21.15.102:0/3079658850 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400c9e0 con 0x7f4fdc0ced00 2023-03-03T16:35:16.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.182+0000 7f4fd1ffb700 1 -- 172.21.15.102:0/3079658850 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd400f040 con 0x7f4fdc0ced00 2023-03-03T16:35:16.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.182+0000 7f4fd1ffb700 1 -- 172.21.15.102:0/3079658850 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400cd80 con 0x7f4fdc0ced00 2023-03-03T16:35:16.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.182+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc06bf20 con 0x7f4fdc0ced00 2023-03-03T16:35:16.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.182+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc15ce80 con 0x7f4fdc0ced00 2023-03-03T16:35:16.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.183+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc06c200 con 0x7f4fdc0ced00 2023-03-03T16:35:16.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.184+0000 7f4fd1ffb700 1 -- 172.21.15.102:0/3079658850 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4fd4024070 con 0x7f4fdc0ced00 2023-03-03T16:35:16.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.185+0000 7f4fd1ffb700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fc00682f0 0x7f4fc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:16.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.185+0000 7f4fd1ffb700 1 -- 172.21.15.102:0/3079658850 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4fd4086790 con 0x7f4fdc0ced00 2023-03-03T16:35:16.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.186+0000 7f4fe139b700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fc00682f0 0x7f4fc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:16.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.186+0000 7f4fe139b700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fc00682f0 0x7f4fc006a7a0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8000f60 tx=0x7f4fd8007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:16.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.190+0000 7f4fd1ffb700 1 -- 172.21.15.102:0/3079658850 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4fd4052ea0 con 0x7f4fdc0ced00 2023-03-03T16:35:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.424+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fdc04e6e0 con 0x7f4fdc0ced00 2023-03-03T16:35:16.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.425+0000 7f4fd1ffb700 1 -- 172.21.15.102:0/3079658850 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4fd40564c0 con 0x7f4fdc0ced00 2023-03-03T16:35:16.426 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:35:16.426 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:35:16.429 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fc00682f0 msgr2=0x7f4fc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fc00682f0 0x7f4fc006a7a0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8000f60 tx=0x7f4fd8007ea0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fdc0ced00 msgr2=0x7f4fdc0ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fdc0ced00 0x7f4fdc0ccea0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400ec00 tx=0x7f4fd400ef10 comp rx=0 tx=0).stop 2023-03-03T16:35:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 shutdown_connections 2023-03-03T16:35:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4fc00682f0 0x7f4fc006a7a0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4fdc0ced00 0x7f4fdc0ccea0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fdc0b3970 0x7f4fdc1e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 --2- 172.21.15.102:0/3079658850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fdc0c41b0 0x7f4fdc151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 >> 172.21.15.102:0/3079658850 conn(0x7f4fdc08b1c0 msgr2=0x7f4fdc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 shutdown_connections 2023-03-03T16:35:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:16.428+0000 7f4fe239d700 1 -- 172.21.15.102:0/3079658850 wait complete. 2023-03-03T16:35:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:35:16.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T16:35:24.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:24.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:54.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:54.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:55.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:35:55.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.707+0000 7fa1ff1ff700 1 Processor -- start 2023-03-03T16:35:55.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.708+0000 7fa1ff1ff700 1 -- start start 2023-03-03T16:35:55.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1ff1ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa1f8153290 0x7fa1f8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:55.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1ff1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa1f80cd8e0 0x7fa1f80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:55.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1ff1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8153660 con 0x7fa1f81593c0 2023-03-03T16:35:55.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1ff1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f806c360 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1ff1ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f815cea0 con 0x7fa1f8153290 2023-03-03T16:35:55.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1fd79c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa1f80cd8e0 0x7fa1f80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:55.710 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1fcf9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa1f8153290 0x7fa1f8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:55.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1f7fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa1f81593c0 0x7fa1f80cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48584/0 (socket says 172.21.15.102:48584) 2023-03-03T16:35:55.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.709+0000 7fa1f7fff700 1 -- 172.21.15.102:0/1657529553 learned_addr learned my addr 172.21.15.102:0/1657529553 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:35:55.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.710+0000 7fa1fd79c700 1 -- 172.21.15.102:0/1657529553 >> v2:172.21.15.172:3300/0 conn(0x7fa1f8153290 msgr2=0x7fa1f8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:35:55.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.710+0000 7fa1fd79c700 1 --2- 172.21.15.102:0/1657529553 >> v2:172.21.15.172:3300/0 conn(0x7fa1f8153290 0x7fa1f8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.710+0000 7fa1fd79c700 1 -- 172.21.15.102:0/1657529553 >> v1:172.21.15.102:6789/0 conn(0x7fa1f81593c0 legacy=0x7fa1f80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:55.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.710+0000 7fa1fd79c700 1 -- 172.21.15.102:0/1657529553 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f806c7f0 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.710+0000 7fa1f77fe700 1 -- 172.21.15.102:0/1657529553 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920799058 0 0) 0x7fa1f8153660 con 0x7fa1f81593c0 2023-03-03T16:35:55.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.710+0000 7fa1fd79c700 1 --2- 172.21.15.102:0/1657529553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f80cd8e0 0x7fa1f80ce530 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa1f808cc70 tx=0x7fa1f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c631baa53ce5be7b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:55.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1f77fe700 1 -- 172.21.15.102:0/1657529553 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000e070 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1f77fe700 1 -- 172.21.15.102:0/1657529553 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f000bb20 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1f77fe700 1 -- 172.21.15.102:0/1657529553 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0012430 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/1657529553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f80cd8e0 msgr2=0x7fa1f80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:55.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/1657529553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f80cd8e0 0x7fa1f80ce530 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa1f808cc70 tx=0x7fa1f000b330 comp rx=0 tx=0).stop 2023-03-03T16:35:55.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/1657529553 shutdown_connections 2023-03-03T16:35:55.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/1657529553 >> v2:172.21.15.172:3300/0 conn(0x7fa1f8153290 0x7fa1f8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/1657529553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f80cd8e0 0x7fa1f80ce530 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/1657529553 >> 172.21.15.102:0/1657529553 conn(0x7fa1f8065fb0 msgr2=0x7fa1f80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/1657529553 shutdown_connections 2023-03-03T16:35:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.711+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/1657529553 wait complete. 2023-03-03T16:35:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.714+0000 7fa1ff1ff700 1 Processor -- start 2023-03-03T16:35:55.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.714+0000 7fa1ff1ff700 1 -- start start 2023-03-03T16:35:55.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1ff1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1f80cd8e0 0x7fa1f80b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:55.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1ff1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1f8153290 0x7fa1f814c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:55.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1ff1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f81593c0 0x7fa1f81480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:55.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1ff1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81e4f70 con 0x7fa1f81593c0 2023-03-03T16:35:55.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1ff1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81e2660 con 0x7fa1f8153290 2023-03-03T16:35:55.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1ff1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f814cb20 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1fcf9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1f80cd8e0 0x7fa1f80b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1fd79c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f81593c0 0x7fa1f81480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1f7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1f8153290 0x7fa1f814c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1fcf9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1f80cd8e0 0x7fa1f80b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:59798/0 (socket says 172.21.15.102:59798) 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.715+0000 7fa1fcf9b700 1 -- 172.21.15.102:0/2977059125 learned_addr learned my addr 172.21.15.102:0/2977059125 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1fcf9b700 1 -- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1f8153290 msgr2=0x7fa1f814c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1fcf9b700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1f8153290 0x7fa1f814c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1fcf9b700 1 -- 172.21.15.102:0/2977059125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f81593c0 msgr2=0x7fa1f81480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:55.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1fcf9b700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f81593c0 0x7fa1f81480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1fcf9b700 1 -- 172.21.15.102:0/2977059125 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0009a70 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1fcf9b700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1f80cd8e0 0x7fa1f80b1c70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00e9b0 tx=0x7fa1ec00ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1f57fa700 1 -- 172.21.15.102:0/2977059125 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec009710 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1f57fa700 1 -- 172.21.15.102:0/2977059125 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec00ee50 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1f57fa700 1 -- 172.21.15.102:0/2977059125 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec00ca50 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.716+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f80ccde0 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.717+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f815d160 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.718+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f80cd0c0 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.719+0000 7fa1f57fa700 1 -- 172.21.15.102:0/2977059125 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa1ec019030 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.720+0000 7fa1f57fa700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:55.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.720+0000 7fa1f57fa700 1 -- 172.21.15.102:0/2977059125 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa1ec085d70 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.720+0000 7fa1f7fff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:55.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.721+0000 7fa1f7fff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8159060 tx=0x7fa1e4007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:55.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.724+0000 7fa1f57fa700 1 -- 172.21.15.102:0/2977059125 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1ec052390 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.955+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1f815cbb0 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.956+0000 7fa1f57fa700 1 -- 172.21.15.102:0/2977059125 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fa1ec021090 con 0x7fa1f80cd8e0 2023-03-03T16:35:55.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:35:55.956 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:35:55.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1dc0682f0 msgr2=0x7fa1dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:55.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8159060 tx=0x7fa1e4007de0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1f80cd8e0 msgr2=0x7fa1f80b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:55.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1f80cd8e0 0x7fa1f80b1c70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00e9b0 tx=0x7fa1ec00ecc0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 shutdown_connections 2023-03-03T16:35:55.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fa1f80cd8e0 0x7fa1f80b1c70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1f81593c0 0x7fa1f81480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 --2- 172.21.15.102:0/2977059125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1f8153290 0x7fa1f814c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:55.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 >> 172.21.15.102:0/2977059125 conn(0x7fa1f8065fb0 msgr2=0x7fa1f815ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:55.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 shutdown_connections 2023-03-03T16:35:55.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:35:55.959+0000 7fa1ff1ff700 1 -- 172.21.15.102:0/2977059125 wait complete. 2023-03-03T16:35:55.961 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:35:55.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 100 secs before thrashing 2023-03-03T16:36:24.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:24.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:54.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:54.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:24.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:24.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:35.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:37:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.236+0000 7f514559d700 1 Processor -- start 2023-03-03T16:37:36.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.236+0000 7f514559d700 1 -- start start 2023-03-03T16:37:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f514559d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f51400c5a90 0x7f51400c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f514559d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51400c3d20 con 0x7f51400ced00 2023-03-03T16:37:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f514559d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514015cba0 con 0x7f51400b3970 2023-03-03T16:37:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f514559d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51400c4930 con 0x7f51400c5a90 2023-03-03T16:37:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f5144d9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f51400c5a90 0x7f51400c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f513ffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f51400ced00 0x7f5140158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46736/0 (socket says 172.21.15.102:46736) 2023-03-03T16:37:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f513ffff700 1 -- 172.21.15.102:0/2454847945 learned_addr learned my addr 172.21.15.102:0/2454847945 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:37:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.238+0000 7f5137fff700 1 --1- 172.21.15.102:0/2454847945 >> v1:172.21.15.172:6789/0 conn(0x7f51400b3970 0x7f5140161470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:49766/0 (socket says 172.21.15.102:49766) 2023-03-03T16:37:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.239+0000 7f5144d9c700 1 -- 172.21.15.102:0/2454847945 >> v1:172.21.15.172:6789/0 conn(0x7f51400b3970 legacy=0x7f5140161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.239+0000 7f5144d9c700 1 -- 172.21.15.102:0/2454847945 >> v1:172.21.15.102:6789/0 conn(0x7f51400ced00 legacy=0x7f5140158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.239+0000 7f5144d9c700 1 -- 172.21.15.102:0/2454847945 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514015ce80 con 0x7f51400c5a90 2023-03-03T16:37:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.239+0000 7f513f7fe700 1 -- 172.21.15.102:0/2454847945 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303584482 0 0) 0x7f514015cba0 con 0x7f51400b3970 2023-03-03T16:37:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.239+0000 7f5144d9c700 1 --2- 172.21.15.102:0/2454847945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f51400c41b0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f5138009d90 tx=0x7f513800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919f4f3d9d9bc131 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:36.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.239+0000 7f513f7fe700 1 -- 172.21.15.102:0/2454847945 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513800e070 con 0x7f51400c5a90 2023-03-03T16:37:36.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f513f7fe700 1 -- 172.21.15.102:0/2454847945 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513800bbd0 con 0x7f51400c5a90 2023-03-03T16:37:36.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f513f7fe700 1 -- 172.21.15.102:0/2454847945 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138012430 con 0x7f51400c5a90 2023-03-03T16:37:36.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f514559d700 1 -- 172.21.15.102:0/2454847945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 msgr2=0x7f51400c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f514559d700 1 --2- 172.21.15.102:0/2454847945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f51400c41b0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f5138009d90 tx=0x7f513800b330 comp rx=0 tx=0).stop 2023-03-03T16:37:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f514559d700 1 -- 172.21.15.102:0/2454847945 shutdown_connections 2023-03-03T16:37:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f514559d700 1 --2- 172.21.15.102:0/2454847945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f51400c41b0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f514559d700 1 -- 172.21.15.102:0/2454847945 >> 172.21.15.102:0/2454847945 conn(0x7f514008b1c0 msgr2=0x7f5140071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:37:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f514559d700 1 -- 172.21.15.102:0/2454847945 shutdown_connections 2023-03-03T16:37:36.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.240+0000 7f514559d700 1 -- 172.21.15.102:0/2454847945 wait complete. 2023-03-03T16:37:36.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.242+0000 7f514559d700 1 Processor -- start 2023-03-03T16:37:36.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.243+0000 7f514559d700 1 -- start start 2023-03-03T16:37:36.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f514559d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51400b3970 0x7f514014e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:36.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f514559d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f5140159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:36.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f514559d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51400ced00 0x7f51401ef010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:36.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f514559d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514014e460 con 0x7f51400c5a90 2023-03-03T16:37:36.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f514559d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514015b150 con 0x7f51400b3970 2023-03-03T16:37:36.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f514559d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51401ef550 con 0x7f51400ced00 2023-03-03T16:37:36.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f513ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51400b3970 0x7f514014e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:36.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5137fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f5140159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f513ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51400b3970 0x7f514014e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55710/0 (socket says 172.21.15.102:55710) 2023-03-03T16:37:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f513ffff700 1 -- 172.21.15.102:0/3095883179 learned_addr learned my addr 172.21.15.102:0/3095883179 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:37:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5144d9c700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51400ced00 0x7f51401ef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5137fff700 1 -- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51400ced00 msgr2=0x7f51401ef010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5137fff700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51400ced00 0x7f51401ef010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5137fff700 1 -- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51400b3970 msgr2=0x7f514014e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:36.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5137fff700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51400b3970 0x7f514014e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5137fff700 1 -- 172.21.15.102:0/3095883179 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5138009a70 con 0x7f51400c5a90 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f513ffff700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51400b3970 0x7f514014e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.244+0000 7f5144d9c700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51400ced00 0x7f51401ef010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.245+0000 7f5137fff700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f5140159cc0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f512800c410 tx=0x7f512800c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.245+0000 7f513d7fa700 1 -- 172.21.15.102:0/3095883179 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128004a70 con 0x7f51400c5a90 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.245+0000 7f513d7fa700 1 -- 172.21.15.102:0/3095883179 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5128004bd0 con 0x7f51400c5a90 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.245+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514006c200 con 0x7f51400c5a90 2023-03-03T16:37:36.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.245+0000 7f513d7fa700 1 -- 172.21.15.102:0/3095883179 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51280070d0 con 0x7f51400c5a90 2023-03-03T16:37:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.245+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514006bf20 con 0x7f51400c5a90 2023-03-03T16:37:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.246+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514015c890 con 0x7f51400c5a90 2023-03-03T16:37:36.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.247+0000 7f513d7fa700 1 -- 172.21.15.102:0/3095883179 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f512801a020 con 0x7f51400c5a90 2023-03-03T16:37:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.249+0000 7f513d7fa700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51200682f0 0x7f512006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.249+0000 7f513d7fa700 1 -- 172.21.15.102:0/3095883179 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5128085710 con 0x7f51400c5a90 2023-03-03T16:37:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.249+0000 7f513ffff700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51200682f0 0x7f512006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:36.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.250+0000 7f513ffff700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51200682f0 0x7f512006a7a0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f5130005910 tx=0x7f5130009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:36.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.253+0000 7f513d7fa700 1 -- 172.21.15.102:0/3095883179 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5128051e20 con 0x7f51400c5a90 2023-03-03T16:37:36.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.486+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51401523b0 con 0x7f51400c5a90 2023-03-03T16:37:36.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.486+0000 7f513d7fa700 1 -- 172.21.15.102:0/3095883179 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5128055440 con 0x7f51400c5a90 2023-03-03T16:37:36.486 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:37:36.486 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:37:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51200682f0 msgr2=0x7f512006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51200682f0 0x7f512006a7a0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f5130005910 tx=0x7f5130009040 comp rx=0 tx=0).stop 2023-03-03T16:37:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 msgr2=0x7f5140159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f5140159cc0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f512800c410 tx=0x7f512800c720 comp rx=0 tx=0).stop 2023-03-03T16:37:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 shutdown_connections 2023-03-03T16:37:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f51200682f0 0x7f512006a7a0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f51400ced00 0x7f51401ef010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51400b3970 0x7f514014e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 --2- 172.21.15.102:0/3095883179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51400c5a90 0x7f5140159cc0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 >> 172.21.15.102:0/3095883179 conn(0x7f514008b1c0 msgr2=0x7f51400b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:37:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 shutdown_connections 2023-03-03T16:37:36.491 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:37:36.489+0000 7f514559d700 1 -- 172.21.15.102:0/3095883179 wait complete. 2023-03-03T16:37:36.491 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:37:36.517 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-03-03T16:37:54.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:54.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:09.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:38:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.770+0000 7fcc43fff700 1 Processor -- start 2023-03-03T16:38:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.770+0000 7fcc43fff700 1 -- start start 2023-03-03T16:38:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc43fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcc441564b0 0x7fcc440cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc43fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc440c9610 0x7fcc440cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc43fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440ca440 0x7fcc440ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc43fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc440c9ca0 con 0x7fcc441564b0 2023-03-03T16:38:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc43fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc440b46f0 con 0x7fcc440c9610 2023-03-03T16:38:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc43fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440b4920 con 0x7fcc440ca440 2023-03-03T16:38:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc427fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc440c9610 0x7fcc440cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc42ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcc441564b0 0x7fcc440cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc437fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440ca440 0x7fcc440ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc427fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc440c9610 0x7fcc440cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:49102/0 (socket says 172.21.15.102:49102) 2023-03-03T16:38:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.772+0000 7fcc427fc700 1 -- 172.21.15.102:0/2663195761 learned_addr learned my addr 172.21.15.102:0/2663195761 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:38:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc427fc700 1 -- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440ca440 msgr2=0x7fcc440ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:38:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc427fc700 1 --2- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440ca440 0x7fcc440ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc427fc700 1 -- 172.21.15.102:0/2663195761 >> v2:172.21.15.102:3300/0 conn(0x7fcc441564b0 msgr2=0x7fcc440cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:38:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc427fc700 1 --2- 172.21.15.102:0/2663195761 >> v2:172.21.15.102:3300/0 conn(0x7fcc441564b0 0x7fcc440cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc427fc700 1 -- 172.21.15.102:0/2663195761 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc440b4c00 con 0x7fcc440c9610 2023-03-03T16:38:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc437fe700 1 --2- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440ca440 0x7fcc440ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:38:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc42ffd700 1 --2- 172.21.15.102:0/2663195761 >> v2:172.21.15.102:3300/0 conn(0x7fcc441564b0 0x7fcc440cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:38:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc427fc700 1 --2- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440c9610 0x7fcc440cbba0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcc4415d410 tx=0x7fcc38009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1c1df2e1741d0d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2663195761 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38011830 con 0x7fcc440c9610 2023-03-03T16:38:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2663195761 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc38011990 con 0x7fcc440c9610 2023-03-03T16:38:09.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.773+0000 7fcc41ffb700 1 -- 172.21.15.102:0/2663195761 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38011d20 con 0x7fcc440c9610 2023-03-03T16:38:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 -- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440c9610 msgr2=0x7fcc440cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 --2- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440c9610 0x7fcc440cbba0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcc4415d410 tx=0x7fcc38009bd0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 -- 172.21.15.102:0/2663195761 shutdown_connections 2023-03-03T16:38:09.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 --2- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440ca440 0x7fcc440ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 --2- 172.21.15.102:0/2663195761 >> v2:172.21.15.102:3300/0 conn(0x7fcc441564b0 0x7fcc440cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 --2- 172.21.15.102:0/2663195761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440c9610 0x7fcc440cbba0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 -- 172.21.15.102:0/2663195761 >> 172.21.15.102:0/2663195761 conn(0x7fcc44065fb0 msgr2=0x7fcc440663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 -- 172.21.15.102:0/2663195761 shutdown_connections 2023-03-03T16:38:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.774+0000 7fcc43fff700 1 -- 172.21.15.102:0/2663195761 wait complete. 2023-03-03T16:38:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.776+0000 7fcc43fff700 1 Processor -- start 2023-03-03T16:38:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.776+0000 7fcc43fff700 1 -- start start 2023-03-03T16:38:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.777+0000 7fcc43fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440c9610 0x7fcc441ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.777+0000 7fcc43fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440ca440 0x7fcc440c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.777+0000 7fcc43fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc441564b0 0x7fcc441efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.777+0000 7fcc43fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc441ef7d0 con 0x7fcc441564b0 2023-03-03T16:38:09.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.777+0000 7fcc43fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4414b150 con 0x7fcc440ca440 2023-03-03T16:38:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.777+0000 7fcc43fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440c5b20 con 0x7fcc440c9610 2023-03-03T16:38:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440c9610 0x7fcc441ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc427fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440ca440 0x7fcc440c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc437fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc441564b0 0x7fcc441efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440c9610 0x7fcc441ef290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:52436/0 (socket says 172.21.15.102:52436) 2023-03-03T16:38:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 -- 172.21.15.102:0/3227235172 learned_addr learned my addr 172.21.15.102:0/3227235172 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:38:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 -- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440ca440 msgr2=0x7fcc440c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440ca440 0x7fcc440c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 -- 172.21.15.102:0/3227235172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc441564b0 msgr2=0x7fcc441efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc441564b0 0x7fcc441efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 -- 172.21.15.102:0/3227235172 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c009a70 con 0x7fcc440c9610 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.778+0000 7fcc42ffd700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440c9610 0x7fcc441ef290 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400ebe0 tx=0x7fcc3400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.779+0000 7fcc2b7fe700 1 -- 172.21.15.102:0/3227235172 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34009760 con 0x7fcc440c9610 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.779+0000 7fcc2b7fe700 1 -- 172.21.15.102:0/3227235172 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3400c850 con 0x7fcc440c9610 2023-03-03T16:38:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.779+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc340094a0 con 0x7fcc440c9610 2023-03-03T16:38:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.779+0000 7fcc2b7fe700 1 -- 172.21.15.102:0/3227235172 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400cbe0 con 0x7fcc440c9610 2023-03-03T16:38:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.779+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc38009550 con 0x7fcc440c9610 2023-03-03T16:38:09.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.780+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc440b2210 con 0x7fcc440c9610 2023-03-03T16:38:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.781+0000 7fcc2b7fe700 1 -- 172.21.15.102:0/3227235172 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcc34023070 con 0x7fcc440c9610 2023-03-03T16:38:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.782+0000 7fcc2b7fe700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc30068320 0x7fcc3006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.783+0000 7fcc2b7fe700 1 -- 172.21.15.102:0/3227235172 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcc34086060 con 0x7fcc440c9610 2023-03-03T16:38:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.783+0000 7fcc427fc700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc30068320 0x7fcc3006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:09.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.783+0000 7fcc427fc700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc30068320 0x7fcc3006a7d0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcc440c9a30 tx=0x7fcc38008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:09.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:09.787+0000 7fcc2b7fe700 1 -- 172.21.15.102:0/3227235172 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc34052680 con 0x7fcc440c9610 2023-03-03T16:38:10.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.020+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc44156880 con 0x7fcc440c9610 2023-03-03T16:38:10.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.021+0000 7fcc2b7fe700 1 -- 172.21.15.102:0/3227235172 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcc34055ca0 con 0x7fcc440c9610 2023-03-03T16:38:10.021 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:38:10.021 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:38:10.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc30068320 msgr2=0x7fcc3006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:10.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc30068320 0x7fcc3006a7d0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcc440c9a30 tx=0x7fcc38008030 comp rx=0 tx=0).stop 2023-03-03T16:38:10.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440c9610 msgr2=0x7fcc441ef290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:10.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440c9610 0x7fcc441ef290 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400ebe0 tx=0x7fcc3400eef0 comp rx=0 tx=0).stop 2023-03-03T16:38:10.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 shutdown_connections 2023-03-03T16:38:10.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc30068320 0x7fcc3006a7d0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:10.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc440c9610 0x7fcc441ef290 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:10.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc441564b0 0x7fcc441efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:10.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 --2- 172.21.15.102:0/3227235172 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc440ca440 0x7fcc440c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:10.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 >> 172.21.15.102:0/3227235172 conn(0x7fcc44065fb0 msgr2=0x7fcc441550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:10.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 shutdown_connections 2023-03-03T16:38:10.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:10.024+0000 7fcc43fff700 1 -- 172.21.15.102:0/3227235172 wait complete. 2023-03-03T16:38:10.025 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:38:10.051 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2023-03-03T16:38:25.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:25.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:48.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:38:48.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.300+0000 7fab4f19d700 1 Processor -- start 2023-03-03T16:38:48.300 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.300+0000 7fab4f19d700 1 -- start start 2023-03-03T16:38:48.301 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.302+0000 7fab4f19d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fab480ced00 0x7fab480c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:48.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.302+0000 7fab4f19d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480c3d20 con 0x7fab480b3970 2023-03-03T16:38:48.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.302+0000 7fab4f19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4815cba0 con 0x7fab480c41b0 2023-03-03T16:38:48.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.302+0000 7fab4e19b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fab480ced00 0x7fab480c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:48.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.302+0000 7fab4d99a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fab480b3970 0x7fab48158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36058/0 (socket says 172.21.15.102:36058) 2023-03-03T16:38:48.302 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.302+0000 7fab4f19d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480c4930 con 0x7fab480ced00 2023-03-03T16:38:48.303 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.302+0000 7fab4d99a700 1 -- 172.21.15.102:0/2148310099 learned_addr learned my addr 172.21.15.102:0/2148310099 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:38:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4e19b700 1 -- 172.21.15.102:0/2148310099 >> v1:172.21.15.172:6789/0 conn(0x7fab480c41b0 legacy=0x7fab48163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4e19b700 1 -- 172.21.15.102:0/2148310099 >> v1:172.21.15.102:6789/0 conn(0x7fab480b3970 legacy=0x7fab48158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4e19b700 1 -- 172.21.15.102:0/2148310099 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4815ce80 con 0x7fab480ced00 2023-03-03T16:38:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4d199700 1 -- 172.21.15.102:0/2148310099 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583841399 0 0) 0x7fab480c3d20 con 0x7fab480b3970 2023-03-03T16:38:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4e19b700 1 --2- 172.21.15.102:0/2148310099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 0x7fab480c5a90 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fab44009840 tx=0x7fab44009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bf03bdb0214c9cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4d199700 1 -- 172.21.15.102:0/2148310099 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44011830 con 0x7fab480ced00 2023-03-03T16:38:48.304 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4d199700 1 -- 172.21.15.102:0/2148310099 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab44011990 con 0x7fab480ced00 2023-03-03T16:38:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.303+0000 7fab4d199700 1 -- 172.21.15.102:0/2148310099 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44011d20 con 0x7fab480ced00 2023-03-03T16:38:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.304+0000 7fab4f19d700 1 -- 172.21.15.102:0/2148310099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 msgr2=0x7fab480c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.304+0000 7fab4f19d700 1 --2- 172.21.15.102:0/2148310099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 0x7fab480c5a90 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fab44009840 tx=0x7fab44009b50 comp rx=0 tx=0).stop 2023-03-03T16:38:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.304+0000 7fab4f19d700 1 -- 172.21.15.102:0/2148310099 shutdown_connections 2023-03-03T16:38:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.304+0000 7fab4f19d700 1 --2- 172.21.15.102:0/2148310099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 0x7fab480c5a90 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.304+0000 7fab4f19d700 1 -- 172.21.15.102:0/2148310099 >> 172.21.15.102:0/2148310099 conn(0x7fab4808b1c0 msgr2=0x7fab48071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.304+0000 7fab4f19d700 1 -- 172.21.15.102:0/2148310099 shutdown_connections 2023-03-03T16:38:48.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.304+0000 7fab4f19d700 1 -- 172.21.15.102:0/2148310099 wait complete. 2023-03-03T16:38:48.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.306+0000 7fab4f19d700 1 Processor -- start 2023-03-03T16:38:48.306 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.306+0000 7fab4f19d700 1 -- start start 2023-03-03T16:38:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.307+0000 7fab4f19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab480b3970 0x7fab48159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.307+0000 7fab4f19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab480c41b0 0x7fab481f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.307+0000 7fab4f19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 0x7fab481f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.307+0000 7fab4f19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481e4cb0 con 0x7fab480c41b0 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.307+0000 7fab4f19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480c79a0 con 0x7fab480ced00 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.307+0000 7fab4f19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480c6cf0 con 0x7fab480b3970 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.307+0000 7fab4d99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab480c41b0 0x7fab481f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab480c41b0 0x7fab481f1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41276/0 (socket says 172.21.15.102:41276) 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 -- 172.21.15.102:0/4238457360 learned_addr learned my addr 172.21.15.102:0/4238457360 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:38:48.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4e19b700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab480b3970 0x7fab48159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4e99c700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 0x7fab481f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 -- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab480b3970 msgr2=0x7fab48159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab480b3970 0x7fab48159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 -- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 msgr2=0x7fab481f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 0x7fab481f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:48.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 -- 172.21.15.102:0/4238457360 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab440094a0 con 0x7fab480c41b0 2023-03-03T16:38:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab4d99a700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab480c41b0 0x7fab481f1840 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fab3800c480 tx=0x7fab3800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab3effd700 1 -- 172.21.15.102:0/4238457360 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38004c80 con 0x7fab480c41b0 2023-03-03T16:38:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.308+0000 7fab3effd700 1 -- 172.21.15.102:0/4238457360 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab38004e00 con 0x7fab480c41b0 2023-03-03T16:38:48.310 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.309+0000 7fab3effd700 1 -- 172.21.15.102:0/4238457360 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38007310 con 0x7fab480c41b0 2023-03-03T16:38:48.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.309+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4806c200 con 0x7fab480c41b0 2023-03-03T16:38:48.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.309+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4806bf20 con 0x7fab480c41b0 2023-03-03T16:38:48.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.310+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4815c890 con 0x7fab480c41b0 2023-03-03T16:38:48.311 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.311+0000 7fab3effd700 1 -- 172.21.15.102:0/4238457360 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab3801a020 con 0x7fab480c41b0 2023-03-03T16:38:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.312+0000 7fab3effd700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c0682f0 0x7fab2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.312+0000 7fab3effd700 1 -- 172.21.15.102:0/4238457360 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fab380858d0 con 0x7fab480c41b0 2023-03-03T16:38:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.312+0000 7fab4e19b700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c0682f0 0x7fab2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:48.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.313+0000 7fab4e19b700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c0682f0 0x7fab2c06a7a0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fab440090b0 tx=0x7fab4401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:48.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.316+0000 7fab3effd700 1 -- 172.21.15.102:0/4238457360 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab38051ef0 con 0x7fab480c41b0 2023-03-03T16:38:48.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.553+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab481e4a30 con 0x7fab480c41b0 2023-03-03T16:38:48.553 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.554+0000 7fab3effd700 1 -- 172.21.15.102:0/4238457360 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fab38055510 con 0x7fab480c41b0 2023-03-03T16:38:48.554 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:38:48.554 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:38:48.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c0682f0 msgr2=0x7fab2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:48.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c0682f0 0x7fab2c06a7a0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fab440090b0 tx=0x7fab4401c040 comp rx=0 tx=0).stop 2023-03-03T16:38:48.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab480c41b0 msgr2=0x7fab481f1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:48.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab480c41b0 0x7fab481f1840 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fab3800c480 tx=0x7fab3800c790 comp rx=0 tx=0).stop 2023-03-03T16:38:48.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 shutdown_connections 2023-03-03T16:38:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fab2c0682f0 0x7fab2c06a7a0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fab480b3970 0x7fab48159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fab480ced00 0x7fab481f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 --2- 172.21.15.102:0/4238457360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab480c41b0 0x7fab481f1840 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 >> 172.21.15.102:0/4238457360 conn(0x7fab4808b1c0 msgr2=0x7fab48151f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 shutdown_connections 2023-03-03T16:38:48.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:38:48.556+0000 7fab4f19d700 1 -- 172.21.15.102:0/4238457360 wait complete. 2023-03-03T16:38:48.558 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:38:48.583 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-03-03T16:38:55.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:55.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:25.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:25.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:55.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:55.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:59.999 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T16:39:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:40:25.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:25.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:36.585 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:40:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.834+0000 7f7e3b19d700 1 Processor -- start 2023-03-03T16:40:36.834 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.835+0000 7f7e3b19d700 1 -- start start 2023-03-03T16:40:36.835 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3b19d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7e341564b0 0x7f7e340cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:36.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3b19d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340c9ca0 con 0x7f7e340c9610 2023-03-03T16:40:36.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3b19d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340b46f0 con 0x7f7e340cab50 2023-03-03T16:40:36.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3b19d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e340b4920 con 0x7f7e341564b0 2023-03-03T16:40:36.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3a19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7e341564b0 0x7f7e340cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:36.836 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3a19b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7e341564b0 0x7f7e340cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38094/0 (socket says 172.21.15.102:38094) 2023-03-03T16:40:36.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3a19b700 1 -- 172.21.15.102:0/940839204 learned_addr learned my addr 172.21.15.102:0/940839204 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:40:36.837 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.836+0000 7f7e3999a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7e340c9610 0x7f7e340ca440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37162/0 (socket says 172.21.15.102:37162) 2023-03-03T16:40:36.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e3a19b700 1 -- 172.21.15.102:0/940839204 >> v1:172.21.15.172:6789/0 conn(0x7f7e340cab50 legacy=0x7f7e341636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:36.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e3a19b700 1 -- 172.21.15.102:0/940839204 >> v1:172.21.15.102:6789/0 conn(0x7f7e340c9610 legacy=0x7f7e340ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:36.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e3a19b700 1 -- 172.21.15.102:0/940839204 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340b4c00 con 0x7f7e341564b0 2023-03-03T16:40:36.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e39199700 1 -- 172.21.15.102:0/940839204 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943114058 0 0) 0x7f7e340b46f0 con 0x7f7e340cab50 2023-03-03T16:40:36.838 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e3a19b700 1 --2- 172.21.15.102:0/940839204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e340cb660 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7e300098f0 tx=0x7f7e30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52dfb6b66f2b8fdc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e39199700 1 -- 172.21.15.102:0/940839204 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000b8a0 con 0x7f7e341564b0 2023-03-03T16:40:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e39199700 1 -- 172.21.15.102:0/940839204 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3000ba00 con 0x7f7e341564b0 2023-03-03T16:40:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.837+0000 7f7e39199700 1 -- 172.21.15.102:0/940839204 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000bd90 con 0x7f7e341564b0 2023-03-03T16:40:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.838+0000 7f7e3b19d700 1 -- 172.21.15.102:0/940839204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 msgr2=0x7f7e340cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.838+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/940839204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e340cb660 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7e300098f0 tx=0x7f7e30009c00 comp rx=0 tx=0).stop 2023-03-03T16:40:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.838+0000 7f7e3b19d700 1 -- 172.21.15.102:0/940839204 shutdown_connections 2023-03-03T16:40:36.839 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.838+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/940839204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e340cb660 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:36.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.838+0000 7f7e3b19d700 1 -- 172.21.15.102:0/940839204 >> 172.21.15.102:0/940839204 conn(0x7f7e34065fb0 msgr2=0x7f7e340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:36.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.838+0000 7f7e3b19d700 1 -- 172.21.15.102:0/940839204 shutdown_connections 2023-03-03T16:40:36.840 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.838+0000 7f7e3b19d700 1 -- 172.21.15.102:0/940839204 wait complete. 2023-03-03T16:40:36.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.840+0000 7f7e3b19d700 1 Processor -- start 2023-03-03T16:40:36.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.840+0000 7f7e3b19d700 1 -- start start 2023-03-03T16:40:36.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.841+0000 7f7e3b19d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e340c9610 0x7f7e34159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:36.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.841+0000 7f7e3b19d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e340cab50 0x7f7e3415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:36.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.841+0000 7f7e3b19d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e341545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:36.841 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.841+0000 7f7e3b19d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3415af90 con 0x7f7e341564b0 2023-03-03T16:40:36.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.841+0000 7f7e3b19d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e341ee520 con 0x7f7e340cab50 2023-03-03T16:40:36.842 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.841+0000 7f7e3b19d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3415b550 con 0x7f7e340c9610 2023-03-03T16:40:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e340cab50 0x7f7e3415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e340c9610 0x7f7e34159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e340cab50 0x7f7e3415aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52604/0 (socket says 172.21.15.102:52604) 2023-03-03T16:40:36.843 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3999a700 1 -- 172.21.15.102:0/3351105392 learned_addr learned my addr 172.21.15.102:0/3351105392 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:40:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a99c700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e341545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a99c700 1 -- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e340c9610 msgr2=0x7f7e34159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a99c700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e340c9610 0x7f7e34159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a99c700 1 -- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e340cab50 msgr2=0x7f7e3415aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a99c700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e340cab50 0x7f7e3415aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a99c700 1 -- 172.21.15.102:0/3351105392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30009550 con 0x7f7e341564b0 2023-03-03T16:40:36.844 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3999a700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e340cab50 0x7f7e3415aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:40:36.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a19b700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e340c9610 0x7f7e34159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:40:36.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.842+0000 7f7e3a99c700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e341545a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00c730 tx=0x7f7e2c00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:36.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.843+0000 7f7e2affd700 1 -- 172.21.15.102:0/3351105392 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c003e30 con 0x7f7e341564b0 2023-03-03T16:40:36.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.843+0000 7f7e2affd700 1 -- 172.21.15.102:0/3351105392 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c005070 con 0x7f7e341564b0 2023-03-03T16:40:36.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.843+0000 7f7e2affd700 1 -- 172.21.15.102:0/3351105392 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c005420 con 0x7f7e341564b0 2023-03-03T16:40:36.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.843+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e340b2210 con 0x7f7e341564b0 2023-03-03T16:40:36.845 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.843+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e340b1f30 con 0x7f7e341564b0 2023-03-03T16:40:36.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.844+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e34156880 con 0x7f7e341564b0 2023-03-03T16:40:36.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.845+0000 7f7e2affd700 1 -- 172.21.15.102:0/3351105392 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e2c010070 con 0x7f7e341564b0 2023-03-03T16:40:36.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.847+0000 7f7e2affd700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e18068320 0x7f7e1806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:36.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.847+0000 7f7e2affd700 1 -- 172.21.15.102:0/3351105392 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7e2c085700 con 0x7f7e341564b0 2023-03-03T16:40:36.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.847+0000 7f7e3a19b700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e18068320 0x7f7e1806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:36.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.848+0000 7f7e3a19b700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e18068320 0x7f7e1806a7d0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f7e30000c00 tx=0x7f7e30008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:36.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:36.851+0000 7f7e2affd700 1 -- 172.21.15.102:0/3351105392 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e2c051e10 con 0x7f7e341564b0 2023-03-03T16:40:37.086 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.087+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e340677f0 con 0x7f7e341564b0 2023-03-03T16:40:37.087 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.087+0000 7f7e2affd700 1 -- 172.21.15.102:0/3351105392 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7e2c056e60 con 0x7f7e341564b0 2023-03-03T16:40:37.087 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:40:37.087 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:40:37.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e18068320 msgr2=0x7f7e1806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:37.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e18068320 0x7f7e1806a7d0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f7e30000c00 tx=0x7f7e30008030 comp rx=0 tx=0).stop 2023-03-03T16:40:37.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 msgr2=0x7f7e341545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:37.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e341545a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00c730 tx=0x7f7e2c00ca40 comp rx=0 tx=0).stop 2023-03-03T16:40:37.090 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 shutdown_connections 2023-03-03T16:40:37.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7e18068320 0x7f7e1806a7d0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:37.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7e340c9610 0x7f7e34159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:37.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e341564b0 0x7f7e341545a0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:37.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 --2- 172.21.15.102:0/3351105392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e340cab50 0x7f7e3415aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:37.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 >> 172.21.15.102:0/3351105392 conn(0x7f7e34065fb0 msgr2=0x7f7e341554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:37.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 shutdown_connections 2023-03-03T16:40:37.091 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:37.090+0000 7f7e3b19d700 1 -- 172.21.15.102:0/3351105392 wait complete. 2023-03-03T16:40:37.092 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:40:37.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before thrashing 2023-03-03T16:40:38.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:40:38.376 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.377+0000 7f20595ff700 1 Processor -- start 2023-03-03T16:40:38.377 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.377+0000 7f20595ff700 1 -- start start 2023-03-03T16:40:38.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.378+0000 7f20595ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2054153290 0x7f2054158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:38.378 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.378+0000 7f20595ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f20541593c0 0x7f20540cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.378+0000 7f20595ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2054153660 con 0x7f20540cd8e0 2023-03-03T16:40:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.378+0000 7f20595ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205406c360 con 0x7f2054153290 2023-03-03T16:40:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.378+0000 7f20595ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205415cea0 con 0x7f20541593c0 2023-03-03T16:40:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f204bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f20541593c0 0x7f20540cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f2052ffd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2054153290 0x7f2054158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:38.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f204bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f20541593c0 0x7f20540cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52606/0 (socket says 172.21.15.102:52606) 2023-03-03T16:40:38.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f204bfff700 1 -- 172.21.15.102:0/3613731397 learned_addr learned my addr 172.21.15.102:0/3613731397 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:40:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f204bfff700 1 -- 172.21.15.102:0/3613731397 >> v2:172.21.15.102:3300/0 conn(0x7f2054153290 msgr2=0x7f2054158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:40:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f204bfff700 1 --2- 172.21.15.102:0/3613731397 >> v2:172.21.15.102:3300/0 conn(0x7f2054153290 0x7f2054158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f204bfff700 1 -- 172.21.15.102:0/3613731397 >> v1:172.21.15.102:6789/0 conn(0x7f20540cd8e0 legacy=0x7f20540ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f204bfff700 1 -- 172.21.15.102:0/3613731397 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205406c7f0 con 0x7f20541593c0 2023-03-03T16:40:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.379+0000 7f2052ffd700 1 --2- 172.21.15.102:0/3613731397 >> v2:172.21.15.102:3300/0 conn(0x7f2054153290 0x7f2054158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:40:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.380+0000 7f204bfff700 1 --2- 172.21.15.102:0/3613731397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20541593c0 0x7f20540cde20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2054159fb0 tx=0x7f203c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64de0c3c4873ed0d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.380+0000 7f20527fc700 1 -- 172.21.15.102:0/3613731397 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c011830 con 0x7f20541593c0 2023-03-03T16:40:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.380+0000 7f20527fc700 1 -- 172.21.15.102:0/3613731397 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c011990 con 0x7f20541593c0 2023-03-03T16:40:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.380+0000 7f20527fc700 1 -- 172.21.15.102:0/3613731397 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c011d20 con 0x7f20541593c0 2023-03-03T16:40:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 -- 172.21.15.102:0/3613731397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20541593c0 msgr2=0x7f20540cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 --2- 172.21.15.102:0/3613731397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20541593c0 0x7f20540cde20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2054159fb0 tx=0x7f203c009bd0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 -- 172.21.15.102:0/3613731397 shutdown_connections 2023-03-03T16:40:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 --2- 172.21.15.102:0/3613731397 >> v2:172.21.15.102:3300/0 conn(0x7f2054153290 0x7f2054158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 --2- 172.21.15.102:0/3613731397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20541593c0 0x7f20540cde20 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 -- 172.21.15.102:0/3613731397 >> 172.21.15.102:0/3613731397 conn(0x7f2054065fb0 msgr2=0x7f20540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:38.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 -- 172.21.15.102:0/3613731397 shutdown_connections 2023-03-03T16:40:38.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.381+0000 7f20595ff700 1 -- 172.21.15.102:0/3613731397 wait complete. 2023-03-03T16:40:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.383+0000 7f20595ff700 1 Processor -- start 2023-03-03T16:40:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.383+0000 7f20595ff700 1 -- start start 2023-03-03T16:40:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.384+0000 7f20595ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20540cd8e0 0x7f20540c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.384+0000 7f20595ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2054153290 0x7f20541509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.384+0000 7f20595ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20541593c0 0x7f2054150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.384+0000 7f20595ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20540b1c70 con 0x7f2054153290 2023-03-03T16:40:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.384+0000 7f20595ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20540c8e60 con 0x7f20540cd8e0 2023-03-03T16:40:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.384+0000 7f20595ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2054150300 con 0x7f20541593c0 2023-03-03T16:40:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.384+0000 7f204bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2054153290 0x7f20541509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f2052ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20540cd8e0 0x7f20540c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2054153290 0x7f20541509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38116/0 (socket says 172.21.15.102:38116) 2023-03-03T16:40:38.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 -- 172.21.15.102:0/3294392908 learned_addr learned my addr 172.21.15.102:0/3294392908 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:40:38.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f20537fe700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20541593c0 0x7f2054150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 -- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20541593c0 msgr2=0x7f2054150f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20541593c0 0x7f2054150f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 -- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20540cd8e0 msgr2=0x7f20540c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20540cd8e0 0x7f20540c7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 -- 172.21.15.102:0/3294392908 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440094a0 con 0x7f2054153290 2023-03-03T16:40:38.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.385+0000 7f204bfff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2054153290 0x7f20541509e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f205408cd80 tx=0x7f203c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.386+0000 7f204b7fe700 1 -- 172.21.15.102:0/3294392908 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c004380 con 0x7f2054153290 2023-03-03T16:40:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.386+0000 7f204b7fe700 1 -- 172.21.15.102:0/3294392908 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c01f070 con 0x7f2054153290 2023-03-03T16:40:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.386+0000 7f204b7fe700 1 -- 172.21.15.102:0/3294392908 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0036a0 con 0x7f2054153290 2023-03-03T16:40:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.386+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c009550 con 0x7f2054153290 2023-03-03T16:40:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.386+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20540ccde0 con 0x7f2054153290 2023-03-03T16:40:38.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.387+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205415d160 con 0x7f2054153290 2023-03-03T16:40:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.388+0000 7f204b7fe700 1 -- 172.21.15.102:0/3294392908 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f203c02a020 con 0x7f2054153290 2023-03-03T16:40:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.389+0000 7f204b7fe700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20340682f0 0x7f203406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.389+0000 7f204b7fe700 1 -- 172.21.15.102:0/3294392908 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f203c087260 con 0x7f2054153290 2023-03-03T16:40:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.390+0000 7f2052ffd700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20340682f0 0x7f203406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:38.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.390+0000 7f2052ffd700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20340682f0 0x7f203406a7a0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f2054152570 tx=0x7f2044009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:38.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.394+0000 7f204b7fe700 1 -- 172.21.15.102:0/3294392908 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f203c053970 con 0x7f2054153290 2023-03-03T16:40:38.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.632+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20540cd0c0 con 0x7f2054153290 2023-03-03T16:40:38.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.633+0000 7f204b7fe700 1 -- 172.21.15.102:0/3294392908 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f203c056f90 con 0x7f2054153290 2023-03-03T16:40:38.633 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:40:38.633 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:40:38.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20340682f0 msgr2=0x7f203406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20340682f0 0x7f203406a7a0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f2054152570 tx=0x7f2044009210 comp rx=0 tx=0).stop 2023-03-03T16:40:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2054153290 msgr2=0x7f20541509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2054153290 0x7f20541509e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f205408cd80 tx=0x7f203c004750 comp rx=0 tx=0).stop 2023-03-03T16:40:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 shutdown_connections 2023-03-03T16:40:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20340682f0 0x7f203406a7a0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20541593c0 0x7f2054150f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.636 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20540cd8e0 0x7f20540c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 --2- 172.21.15.102:0/3294392908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2054153290 0x7f20541509e0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 >> 172.21.15.102:0/3294392908 conn(0x7f2054065fb0 msgr2=0x7f20540cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 shutdown_connections 2023-03-03T16:40:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:40:38.636+0000 7f20595ff700 1 -- 172.21.15.102:0/3294392908 wait complete. 2023-03-03T16:40:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:40:38.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-03-03T16:40:55.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:55.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:25.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:25.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:47.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:41:47.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.925+0000 7f7ecd3ff700 1 Processor -- start 2023-03-03T16:41:47.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.926+0000 7f7ecd3ff700 1 -- start start 2023-03-03T16:41:47.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ecd3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec8153290 0x7f7ec8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:47.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ecd3ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ec81593c0 0x7f7ec80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:47.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ecd3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8153660 con 0x7f7ec80cd8e0 2023-03-03T16:41:47.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ecd3ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec806c360 con 0x7f7ec81593c0 2023-03-03T16:41:47.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ecd3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec815cea0 con 0x7f7ec8153290 2023-03-03T16:41:47.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ec67fc700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ec81593c0 0x7f7ec80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:47.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ec77fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7ec80cd8e0 0x7f7ec80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34974/0 (socket says 172.21.15.102:34974) 2023-03-03T16:41:47.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ec77fe700 1 -- 172.21.15.102:0/2257740993 learned_addr learned my addr 172.21.15.102:0/2257740993 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:41:47.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.927+0000 7f7ec6ffd700 1 --2- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec8153290 0x7f7ec8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:47.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.928+0000 7f7ec67fc700 1 -- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec8153290 msgr2=0x7f7ec8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:41:47.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.928+0000 7f7ec67fc700 1 --2- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec8153290 0x7f7ec8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:47.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.928+0000 7f7ec67fc700 1 -- 172.21.15.102:0/2257740993 >> v1:172.21.15.172:6789/0 conn(0x7f7ec80cd8e0 legacy=0x7f7ec80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:47.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.928+0000 7f7ec67fc700 1 -- 172.21.15.102:0/2257740993 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec806c7f0 con 0x7f7ec81593c0 2023-03-03T16:41:47.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.928+0000 7f7ec5ffb700 1 -- 172.21.15.102:0/2257740993 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505309002 0 0) 0x7f7ec8153660 con 0x7f7ec80cd8e0 2023-03-03T16:41:47.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.928+0000 7f7ec67fc700 1 --2- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec81593c0 0x7f7ec80cde20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009840 tx=0x7f7ebc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f36ac0b0fd7a11fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:41:47.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ec5ffb700 1 -- 172.21.15.102:0/2257740993 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc011830 con 0x7f7ec81593c0 2023-03-03T16:41:47.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ec5ffb700 1 -- 172.21.15.102:0/2257740993 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc011990 con 0x7f7ec81593c0 2023-03-03T16:41:47.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ec5ffb700 1 -- 172.21.15.102:0/2257740993 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc011d20 con 0x7f7ec81593c0 2023-03-03T16:41:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec81593c0 msgr2=0x7f7ec80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec81593c0 0x7f7ec80cde20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009840 tx=0x7f7ebc009b50 comp rx=0 tx=0).stop 2023-03-03T16:41:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2257740993 shutdown_connections 2023-03-03T16:41:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec8153290 0x7f7ec8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2257740993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec81593c0 0x7f7ec80cde20 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2257740993 >> 172.21.15.102:0/2257740993 conn(0x7f7ec8065fb0 msgr2=0x7f7ec80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:41:47.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2257740993 shutdown_connections 2023-03-03T16:41:47.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.929+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2257740993 wait complete. 2023-03-03T16:41:47.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.932+0000 7f7ecd3ff700 1 Processor -- start 2023-03-03T16:41:47.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.933+0000 7f7ecd3ff700 1 -- start start 2023-03-03T16:41:47.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ecd3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ec80cd8e0 0x7f7ec80cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ecd3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec8153290 0x7f7ec8154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ecd3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec81593c0 0x7f7ec814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ecd3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8153f50 con 0x7f7ec80cd8e0 2023-03-03T16:41:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ecd3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81501a0 con 0x7f7ec8153290 2023-03-03T16:41:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ecd3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80b2740 con 0x7f7ec81593c0 2023-03-03T16:41:47.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ec6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ec80cd8e0 0x7f7ec80cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:47.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ec6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ec80cd8e0 0x7f7ec80cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60510/0 (socket says 172.21.15.102:60510) 2023-03-03T16:41:47.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ec6ffd700 1 -- 172.21.15.102:0/2489925693 learned_addr learned my addr 172.21.15.102:0/2489925693 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:41:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ec67fc700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec8153290 0x7f7ec8154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.934+0000 7f7ec77fe700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec81593c0 0x7f7ec814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.935+0000 7f7ec67fc700 1 -- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec81593c0 msgr2=0x7f7ec814fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.935+0000 7f7ec67fc700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec81593c0 0x7f7ec814fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.935+0000 7f7ec67fc700 1 -- 172.21.15.102:0/2489925693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ec80cd8e0 msgr2=0x7f7ec80cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.935+0000 7f7ec67fc700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ec80cd8e0 0x7f7ec80cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:47.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.935+0000 7f7ec67fc700 1 -- 172.21.15.102:0/2489925693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0094a0 con 0x7f7ec8153290 2023-03-03T16:41:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.935+0000 7f7ec67fc700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec8153290 0x7f7ec8154370 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc016000 tx=0x7f7ebc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:41:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.935+0000 7f7eaf7fe700 1 -- 172.21.15.102:0/2489925693 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0036a0 con 0x7f7ec8153290 2023-03-03T16:41:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.936+0000 7f7eaf7fe700 1 -- 172.21.15.102:0/2489925693 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc01f070 con 0x7f7ec8153290 2023-03-03T16:41:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.936+0000 7f7eaf7fe700 1 -- 172.21.15.102:0/2489925693 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc003a60 con 0x7f7ec8153290 2023-03-03T16:41:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.936+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec80ccde0 con 0x7f7ec8153290 2023-03-03T16:41:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.936+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec815d160 con 0x7f7ec8153290 2023-03-03T16:41:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.937+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec80cd0c0 con 0x7f7ec8153290 2023-03-03T16:41:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.938+0000 7f7eaf7fe700 1 -- 172.21.15.102:0/2489925693 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ebc00f050 con 0x7f7ec8153290 2023-03-03T16:41:47.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.939+0000 7f7eaf7fe700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7eb40682f0 0x7f7eb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:47.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.939+0000 7f7eaf7fe700 1 -- 172.21.15.102:0/2489925693 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7ebc021070 con 0x7f7ec8153290 2023-03-03T16:41:47.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.940+0000 7f7ec6ffd700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7eb40682f0 0x7f7eb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:47.941 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.940+0000 7f7ec6ffd700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7eb40682f0 0x7f7eb406a7a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009ca0 tx=0x7f7eb8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:41:47.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:47.944+0000 7f7eaf7fe700 1 -- 172.21.15.102:0/2489925693 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ebc053cb0 con 0x7f7ec8153290 2023-03-03T16:41:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.175+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ec804e6e0 con 0x7f7ec8153290 2023-03-03T16:41:48.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.176+0000 7f7eaf7fe700 1 -- 172.21.15.102:0/2489925693 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f7ebc0572d0 con 0x7f7ec8153290 2023-03-03T16:41:48.177 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:41:48.178 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:41:48.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7eb40682f0 msgr2=0x7f7eb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:48.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7eb40682f0 0x7f7eb406a7a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009ca0 tx=0x7f7eb8009210 comp rx=0 tx=0).stop 2023-03-03T16:41:48.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec8153290 msgr2=0x7f7ec8154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:48.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec8153290 0x7f7ec8154370 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc016000 tx=0x7f7ebc004750 comp rx=0 tx=0).stop 2023-03-03T16:41:48.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 shutdown_connections 2023-03-03T16:41:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7eb40682f0 0x7f7eb406a7a0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7ec81593c0 0x7f7ec814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ec80cd8e0 0x7f7ec80cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 --2- 172.21.15.102:0/2489925693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ec8153290 0x7f7ec8154370 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 >> 172.21.15.102:0/2489925693 conn(0x7f7ec8065fb0 msgr2=0x7f7ec81e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:41:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 shutdown_connections 2023-03-03T16:41:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:41:48.179+0000 7f7ecd3ff700 1 -- 172.21.15.102:0/2489925693 wait complete. 2023-03-03T16:41:48.182 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:41:48.207 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T16:41:56.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:56.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:26.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:26.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:31.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:42:31.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.466+0000 7f9f25fff700 1 Processor -- start 2023-03-03T16:42:31.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.466+0000 7f9f25fff700 1 -- start start 2023-03-03T16:42:31.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f25fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f20153290 0x7f9f20158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:31.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f25fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f201593c0 0x7f9f200cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:31.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f25fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f20153660 con 0x7f9f200cd8e0 2023-03-03T16:42:31.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f25fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2006c360 con 0x7f9f201593c0 2023-03-03T16:42:31.469 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f25fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2015cea0 con 0x7f9f20153290 2023-03-03T16:42:31.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f1effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f201593c0 0x7f9f200cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:31.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f1effd700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f201593c0 0x7f9f200cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45724/0 (socket says 172.21.15.102:45724) 2023-03-03T16:42:31.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f1effd700 1 -- 172.21.15.102:0/1044198403 learned_addr learned my addr 172.21.15.102:0/1044198403 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:42:31.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.468+0000 7f9f1f7fe700 1 --2- 172.21.15.102:0/1044198403 >> v2:172.21.15.172:3300/0 conn(0x7f9f20153290 0x7f9f20158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:31.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1effd700 1 -- 172.21.15.102:0/1044198403 >> v2:172.21.15.172:3300/0 conn(0x7f9f20153290 msgr2=0x7f9f20158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1effd700 1 --2- 172.21.15.102:0/1044198403 >> v2:172.21.15.172:3300/0 conn(0x7f9f20153290 0x7f9f20158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1effd700 1 -- 172.21.15.102:0/1044198403 >> v1:172.21.15.172:6789/0 conn(0x7f9f200cd8e0 legacy=0x7f9f200ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1effd700 1 -- 172.21.15.102:0/1044198403 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2006c7f0 con 0x7f9f201593c0 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1e7fc700 1 -- 172.21.15.102:0/1044198403 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027710816 0 0) 0x7f9f20153660 con 0x7f9f200cd8e0 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1effd700 1 --2- 172.21.15.102:0/1044198403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f200cde20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9f2008cc70 tx=0x7f9f14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f13290e898e55ee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1e7fc700 1 -- 172.21.15.102:0/1044198403 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14011830 con 0x7f9f201593c0 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1e7fc700 1 -- 172.21.15.102:0/1044198403 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14011990 con 0x7f9f201593c0 2023-03-03T16:42:31.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.469+0000 7f9f1e7fc700 1 -- 172.21.15.102:0/1044198403 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14011d20 con 0x7f9f201593c0 2023-03-03T16:42:31.472 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 -- 172.21.15.102:0/1044198403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 msgr2=0x7f9f200cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:31.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 --2- 172.21.15.102:0/1044198403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f200cde20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9f2008cc70 tx=0x7f9f14009bd0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 -- 172.21.15.102:0/1044198403 shutdown_connections 2023-03-03T16:42:31.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 --2- 172.21.15.102:0/1044198403 >> v2:172.21.15.172:3300/0 conn(0x7f9f20153290 0x7f9f20158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 --2- 172.21.15.102:0/1044198403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f200cde20 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 -- 172.21.15.102:0/1044198403 >> 172.21.15.102:0/1044198403 conn(0x7f9f20065fb0 msgr2=0x7f9f200663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:31.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 -- 172.21.15.102:0/1044198403 shutdown_connections 2023-03-03T16:42:31.473 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.470+0000 7f9f25fff700 1 -- 172.21.15.102:0/1044198403 wait complete. 2023-03-03T16:42:31.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.472+0000 7f9f25fff700 1 Processor -- start 2023-03-03T16:42:31.474 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.472+0000 7f9f25fff700 1 -- start start 2023-03-03T16:42:31.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.473+0000 7f9f25fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f200cd8e0 0x7f9f200b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:31.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.473+0000 7f9f25fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f20153290 0x7f9f201eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:31.475 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.473+0000 7f9f25fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f201ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:31.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.473+0000 7f9f25fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200b4100 con 0x7f9f201593c0 2023-03-03T16:42:31.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.473+0000 7f9f25fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201ee9d0 con 0x7f9f200cd8e0 2023-03-03T16:42:31.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.473+0000 7f9f25fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200b4bf0 con 0x7f9f20153290 2023-03-03T16:42:31.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f201ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:31.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f200cd8e0 0x7f9f200b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:31.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f201ef3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45728/0 (socket says 172.21.15.102:45728) 2023-03-03T16:42:31.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 -- 172.21.15.102:0/2626680687 learned_addr learned my addr 172.21.15.102:0/2626680687 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:42:31.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1effd700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f20153290 0x7f9f201eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:31.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 -- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f20153290 msgr2=0x7f9f201eee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:31.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f20153290 0x7f9f201eee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 -- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f200cd8e0 msgr2=0x7f9f200b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:31.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f200cd8e0 0x7f9f200b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 -- 172.21.15.102:0/2626680687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f14009550 con 0x7f9f201593c0 2023-03-03T16:42:31.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f1ffff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f201ef3a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800c800 tx=0x7f9f1800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:31.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.474+0000 7f9f07fff700 1 -- 172.21.15.102:0/2626680687 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18003e30 con 0x7f9f201593c0 2023-03-03T16:42:31.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.475+0000 7f9f07fff700 1 -- 172.21.15.102:0/2626680687 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18005070 con 0x7f9f201593c0 2023-03-03T16:42:31.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.475+0000 7f9f07fff700 1 -- 172.21.15.102:0/2626680687 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18005420 con 0x7f9f201593c0 2023-03-03T16:42:31.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.475+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f200ccde0 con 0x7f9f201593c0 2023-03-03T16:42:31.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.475+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2015d160 con 0x7f9f201593c0 2023-03-03T16:42:31.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.476+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f200cd0c0 con 0x7f9f201593c0 2023-03-03T16:42:31.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.477+0000 7f9f07fff700 1 -- 172.21.15.102:0/2626680687 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9f18019030 con 0x7f9f201593c0 2023-03-03T16:42:31.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.478+0000 7f9f07fff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f0c0682f0 0x7f9f0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:31.479 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.478+0000 7f9f07fff700 1 -- 172.21.15.102:0/2626680687 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9f180851f0 con 0x7f9f201593c0 2023-03-03T16:42:31.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.479+0000 7f9f1f7fe700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f0c0682f0 0x7f9f0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:31.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.479+0000 7f9f1f7fe700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f0c0682f0 0x7f9f0c06a7a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009be0 tx=0x7f9f10009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:31.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.483+0000 7f9f07fff700 1 -- 172.21.15.102:0/2626680687 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9f18051900 con 0x7f9f201593c0 2023-03-03T16:42:31.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.716+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f2004e6e0 con 0x7f9f201593c0 2023-03-03T16:42:31.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.717+0000 7f9f07fff700 1 -- 172.21.15.102:0/2626680687 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9f18017090 con 0x7f9f201593c0 2023-03-03T16:42:31.718 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:42:31.718 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:42:31.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f0c0682f0 msgr2=0x7f9f0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:31.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f0c0682f0 0x7f9f0c06a7a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009be0 tx=0x7f9f10009210 comp rx=0 tx=0).stop 2023-03-03T16:42:31.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 msgr2=0x7f9f201ef3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:31.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f201ef3a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800c800 tx=0x7f9f1800cb10 comp rx=0 tx=0).stop 2023-03-03T16:42:31.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 shutdown_connections 2023-03-03T16:42:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f201593c0 0x7f9f201ef3a0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f200cd8e0 0x7f9f200b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f0c0682f0 0x7f9f0c06a7a0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 --2- 172.21.15.102:0/2626680687 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f20153290 0x7f9f201eee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 >> 172.21.15.102:0/2626680687 conn(0x7f9f20065fb0 msgr2=0x7f9f2015b800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 shutdown_connections 2023-03-03T16:42:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:42:31.720+0000 7f9f25fff700 1 -- 172.21.15.102:0/2626680687 wait complete. 2023-03-03T16:42:31.723 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:42:31.748 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-03-03T16:42:56.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:56.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:26.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:26.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:48.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:43:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.000+0000 7fdcb0fff700 1 Processor -- start 2023-03-03T16:43:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.000+0000 7fdcb0fff700 1 -- start start 2023-03-03T16:43:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.001+0000 7fdcb0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac153290 0x7fdcac158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.001+0000 7fdcb0fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcac1593c0 0x7fdcac0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.001+0000 7fdcb0fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac153660 con 0x7fdcac0cd8e0 2023-03-03T16:43:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdcb0fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcac06c360 con 0x7fdcac1593c0 2023-03-03T16:43:49.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdcb0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac15cea0 con 0x7fdcac153290 2023-03-03T16:43:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdcaa59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac153290 0x7fdcac158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdca3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcac1593c0 0x7fdcac0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdcaad9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdcac0cd8e0 0x7fdcac0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51408/0 (socket says 172.21.15.102:51408) 2023-03-03T16:43:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdcaad9d700 1 -- 172.21.15.102:0/2687818710 learned_addr learned my addr 172.21.15.102:0/2687818710 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:43:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdca9d9b700 1 -- 172.21.15.102:0/2687818710 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041863543 0 0) 0x7fdcac153660 con 0x7fdcac0cd8e0 2023-03-03T16:43:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdca9d9b700 1 -- 172.21.15.102:0/2687818710 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc980035b0 con 0x7fdcac0cd8e0 2023-03-03T16:43:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdca3fff700 1 -- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac153290 msgr2=0x7fdcac158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.002+0000 7fdca3fff700 1 --2- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac153290 0x7fdcac158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.003+0000 7fdca3fff700 1 -- 172.21.15.102:0/2687818710 >> v1:172.21.15.172:6789/0 conn(0x7fdcac0cd8e0 legacy=0x7fdcac0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.003+0000 7fdca3fff700 1 -- 172.21.15.102:0/2687818710 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac06c7f0 con 0x7fdcac1593c0 2023-03-03T16:43:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.003+0000 7fdcaa59c700 1 --2- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac153290 0x7fdcac158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:43:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.003+0000 7fdca3fff700 1 --2- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 0x7fdcac0cde20 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009840 tx=0x7fdc94009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c38659de44e8e74 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.003+0000 7fdca9d9b700 1 -- 172.21.15.102:0/2687818710 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94011830 con 0x7fdcac1593c0 2023-03-03T16:43:49.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.003+0000 7fdca9d9b700 1 -- 172.21.15.102:0/2687818710 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc94011990 con 0x7fdcac1593c0 2023-03-03T16:43:49.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.003+0000 7fdca9d9b700 1 -- 172.21.15.102:0/2687818710 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94011d20 con 0x7fdcac1593c0 2023-03-03T16:43:49.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 -- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 msgr2=0x7fdcac0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:49.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 0x7fdcac0cde20 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009840 tx=0x7fdc94009b50 comp rx=0 tx=0).stop 2023-03-03T16:43:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 -- 172.21.15.102:0/2687818710 shutdown_connections 2023-03-03T16:43:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac153290 0x7fdcac158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/2687818710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 0x7fdcac0cde20 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 -- 172.21.15.102:0/2687818710 >> 172.21.15.102:0/2687818710 conn(0x7fdcac065fb0 msgr2=0x7fdcac0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:43:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 -- 172.21.15.102:0/2687818710 shutdown_connections 2023-03-03T16:43:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.004+0000 7fdcb0fff700 1 -- 172.21.15.102:0/2687818710 wait complete. 2023-03-03T16:43:49.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.006+0000 7fdcb0fff700 1 Processor -- start 2023-03-03T16:43:49.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.006+0000 7fdcb0fff700 1 -- start start 2023-03-03T16:43:49.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdcb0fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac0cd8e0 0x7fdcac0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:49.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdcb0fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcac153290 0x7fdcac1eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:49.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdcb0fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 0x7fdcac1ef410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdcb0fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac14f730 con 0x7fdcac153290 2023-03-03T16:43:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdcb0fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac0c71a0 con 0x7fdcac1593c0 2023-03-03T16:43:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdcb0fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac151cc0 con 0x7fdcac0cd8e0 2023-03-03T16:43:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdca3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcac153290 0x7fdcac1eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.007+0000 7fdcaa59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac0cd8e0 0x7fdcac0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdcaad9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 0x7fdcac1ef410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdcaa59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac0cd8e0 0x7fdcac0c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:36600/0 (socket says 172.21.15.102:36600) 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdcaa59c700 1 -- 172.21.15.102:0/1799068703 learned_addr learned my addr 172.21.15.102:0/1799068703 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdca3fff700 1 -- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac0cd8e0 msgr2=0x7fdcac0c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdca3fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac0cd8e0 0x7fdcac0c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdca3fff700 1 -- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 msgr2=0x7fdcac1ef410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdca3fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 0x7fdcac1ef410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdca3fff700 1 -- 172.21.15.102:0/1799068703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c0094a0 con 0x7fdcac153290 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.008+0000 7fdca3fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcac153290 0x7fdcac1eeed0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fdc94016000 tx=0x7fdc94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:49.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.009+0000 7fdca2ffd700 1 -- 172.21.15.102:0/1799068703 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc940036a0 con 0x7fdcac153290 2023-03-03T16:43:49.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.009+0000 7fdca2ffd700 1 -- 172.21.15.102:0/1799068703 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9401f070 con 0x7fdcac153290 2023-03-03T16:43:49.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.009+0000 7fdca2ffd700 1 -- 172.21.15.102:0/1799068703 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94003a80 con 0x7fdcac153290 2023-03-03T16:43:49.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.009+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc940094a0 con 0x7fdcac153290 2023-03-03T16:43:49.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.009+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac0ccde0 con 0x7fdcac153290 2023-03-03T16:43:49.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.010+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac15d160 con 0x7fdcac153290 2023-03-03T16:43:49.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.011+0000 7fdca2ffd700 1 -- 172.21.15.102:0/1799068703 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc9400f050 con 0x7fdcac153290 2023-03-03T16:43:49.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.012+0000 7fdca2ffd700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc8c0682c0 0x7fdc8c06a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:49.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.012+0000 7fdca2ffd700 1 -- 172.21.15.102:0/1799068703 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdc94021070 con 0x7fdcac153290 2023-03-03T16:43:49.012 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.013+0000 7fdcaa59c700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc8c0682c0 0x7fdc8c06a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:49.013 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.013+0000 7fdcaa59c700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc8c0682c0 0x7fdc8c06a770 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009e70 tx=0x7fdc9c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:49.016 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.016+0000 7fdca2ffd700 1 -- 172.21.15.102:0/1799068703 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc94053c40 con 0x7fdcac153290 2023-03-03T16:43:49.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.248+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcac0cd0c0 con 0x7fdcac153290 2023-03-03T16:43:49.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.248+0000 7fdca2ffd700 1 -- 172.21.15.102:0/1799068703 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fdc9402b090 con 0x7fdcac153290 2023-03-03T16:43:49.248 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:43:49.248 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:43:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc8c0682c0 msgr2=0x7fdc8c06a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc8c0682c0 0x7fdc8c06a770 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009e70 tx=0x7fdc9c009210 comp rx=0 tx=0).stop 2023-03-03T16:43:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcac153290 msgr2=0x7fdcac1eeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcac153290 0x7fdcac1eeed0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fdc94016000 tx=0x7fdc94004750 comp rx=0 tx=0).stop 2023-03-03T16:43:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 shutdown_connections 2023-03-03T16:43:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fdc8c0682c0 0x7fdc8c06a770 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdcac0cd8e0 0x7fdcac0c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac1593c0 0x7fdcac1ef410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 --2- 172.21.15.102:0/1799068703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcac153290 0x7fdcac1eeed0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:49.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.251+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 >> 172.21.15.102:0/1799068703 conn(0x7fdcac065fb0 msgr2=0x7fdcac0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:43:49.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.252+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 shutdown_connections 2023-03-03T16:43:49.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:43:49.252+0000 7fdcb0fff700 1 -- 172.21.15.102:0/1799068703 wait complete. 2023-03-03T16:43:49.253 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:43:49.278 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 68 secs before thrashing 2023-03-03T16:43:56.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:56.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:26.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:26.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:56.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:56.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:57.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:44:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.526+0000 7f8d7af9d700 1 Processor -- start 2023-03-03T16:44:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.526+0000 7f8d7af9d700 1 -- start start 2023-03-03T16:44:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7af9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d740b0040 0x7f8d74070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:57.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7af9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8d740c7900 0x7f8d740c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7af9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d740c22d0 con 0x7f8d7406f3b0 2023-03-03T16:44:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7af9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7406f800 con 0x7f8d740c7900 2023-03-03T16:44:57.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7af9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7406b660 con 0x7f8d740b0040 2023-03-03T16:44:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7a79c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8d740c7900 0x7f8d740c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7979a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8d7406f3b0 0x7f8d74158df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:45402/0 (socket says 172.21.15.102:45402) 2023-03-03T16:44:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d79f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d740b0040 0x7f8d74070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:57.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.528+0000 7f8d7979a700 1 -- 172.21.15.102:0/1198181672 learned_addr learned my addr 172.21.15.102:0/1198181672 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:44:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.529+0000 7f8d7a79c700 1 -- 172.21.15.102:0/1198181672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d740b0040 msgr2=0x7f8d74070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:44:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.529+0000 7f8d7a79c700 1 --2- 172.21.15.102:0/1198181672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d740b0040 0x7f8d74070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.529+0000 7f8d7a79c700 1 -- 172.21.15.102:0/1198181672 >> v1:172.21.15.172:6789/0 conn(0x7f8d7406f3b0 legacy=0x7f8d74158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.529+0000 7f8d7a79c700 1 -- 172.21.15.102:0/1198181672 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d740c5b00 con 0x7f8d740c7900 2023-03-03T16:44:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.529+0000 7f8d78f99700 1 -- 172.21.15.102:0/1198181672 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898666842 0 0) 0x7f8d740c22d0 con 0x7f8d7406f3b0 2023-03-03T16:44:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.529+0000 7f8d7a79c700 1 --2- 172.21.15.102:0/1198181672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740c7900 0x7f8d740c7d70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009d90 tx=0x7f8d6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7eaae1ee1166fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:57.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.529+0000 7f8d78f99700 1 -- 172.21.15.102:0/1198181672 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c00e070 con 0x7f8d740c7900 2023-03-03T16:44:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d78f99700 1 -- 172.21.15.102:0/1198181672 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c00bb20 con 0x7f8d740c7900 2023-03-03T16:44:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d78f99700 1 -- 172.21.15.102:0/1198181672 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c012430 con 0x7f8d740c7900 2023-03-03T16:44:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1198181672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740c7900 msgr2=0x7f8d740c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1198181672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740c7900 0x7f8d740c7d70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009d90 tx=0x7f8d6c00b330 comp rx=0 tx=0).stop 2023-03-03T16:44:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1198181672 shutdown_connections 2023-03-03T16:44:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1198181672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740c7900 0x7f8d740c7d70 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1198181672 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d740b0040 0x7f8d74070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1198181672 >> 172.21.15.102:0/1198181672 conn(0x7f8d7406cf70 msgr2=0x7f8d7406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1198181672 shutdown_connections 2023-03-03T16:44:57.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.530+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1198181672 wait complete. 2023-03-03T16:44:57.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.532+0000 7f8d7af9d700 1 Processor -- start 2023-03-03T16:44:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.533+0000 7f8d7af9d700 1 -- start start 2023-03-03T16:44:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.533+0000 7f8d7af9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d7406f3b0 0x7f8d741512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.533+0000 7f8d7af9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740b0040 0x7f8d740acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7af9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d740c7900 0x7f8d740ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7af9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d74153020 con 0x7f8d740b0040 2023-03-03T16:44:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7af9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d74071170 con 0x7f8d740c7900 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7af9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d74154f80 con 0x7f8d7406f3b0 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740b0040 0x7f8d740acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740b0040 0x7f8d740acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37010/0 (socket says 172.21.15.102:37010) 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d79f9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d7406f3b0 0x7f8d741512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 -- 172.21.15.102:0/1688161999 learned_addr learned my addr 172.21.15.102:0/1688161999 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7a79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d740c7900 0x7f8d740ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 -- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d7406f3b0 msgr2=0x7f8d741512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d7406f3b0 0x7f8d741512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 -- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d740c7900 msgr2=0x7f8d740ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d740c7900 0x7f8d740ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d7979a700 1 -- 172.21.15.102:0/1688161999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c009a70 con 0x7f8d740b0040 2023-03-03T16:44:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.534+0000 7f8d79f9b700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d7406f3b0 0x7f8d741512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:44:57.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.535+0000 7f8d7979a700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740b0040 0x7f8d740acca0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400c550 tx=0x7f8d6400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:57.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.535+0000 7f8d6a7fc700 1 -- 172.21.15.102:0/1688161999 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d64004c80 con 0x7f8d740b0040 2023-03-03T16:44:57.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.535+0000 7f8d6a7fc700 1 -- 172.21.15.102:0/1688161999 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d64004de0 con 0x7f8d740b0040 2023-03-03T16:44:57.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.535+0000 7f8d6a7fc700 1 -- 172.21.15.102:0/1688161999 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d640072d0 con 0x7f8d740b0040 2023-03-03T16:44:57.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.535+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d740c6270 con 0x7f8d740b0040 2023-03-03T16:44:57.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.535+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7415d120 con 0x7f8d740b0040 2023-03-03T16:44:57.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.536+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7406b8b0 con 0x7f8d740b0040 2023-03-03T16:44:57.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.537+0000 7f8d6a7fc700 1 -- 172.21.15.102:0/1688161999 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d6401a020 con 0x7f8d740b0040 2023-03-03T16:44:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.539+0000 7f8d6a7fc700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8d580682f0 0x7f8d5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.540+0000 7f8d6a7fc700 1 -- 172.21.15.102:0/1688161999 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8d64085930 con 0x7f8d740b0040 2023-03-03T16:44:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.540+0000 7f8d79f9b700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8d580682f0 0x7f8d5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:57.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.540+0000 7f8d79f9b700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8d580682f0 0x7f8d5806a7a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009fd0 tx=0x7f8d70009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:57.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.544+0000 7f8d6a7fc700 1 -- 172.21.15.102:0/1688161999 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8d64052060 con 0x7f8d740b0040 2023-03-03T16:44:57.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.777+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d7404e6e0 con 0x7f8d740b0040 2023-03-03T16:44:57.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.777+0000 7f8d6a7fc700 1 -- 172.21.15.102:0/1688161999 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8d64055680 con 0x7f8d740b0040 2023-03-03T16:44:57.778 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:44:57.778 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:44:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8d580682f0 msgr2=0x7f8d5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8d580682f0 0x7f8d5806a7a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009fd0 tx=0x7f8d70009210 comp rx=0 tx=0).stop 2023-03-03T16:44:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740b0040 msgr2=0x7f8d740acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740b0040 0x7f8d740acca0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400c550 tx=0x7f8d6400c860 comp rx=0 tx=0).stop 2023-03-03T16:44:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 shutdown_connections 2023-03-03T16:44:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8d580682f0 0x7f8d5806a7a0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8d7406f3b0 0x7f8d741512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d740c7900 0x7f8d740ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 --2- 172.21.15.102:0/1688161999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d740b0040 0x7f8d740acca0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 >> 172.21.15.102:0/1688161999 conn(0x7f8d7406cf70 msgr2=0x7f8d7415aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 shutdown_connections 2023-03-03T16:44:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:44:57.780+0000 7f8d7af9d700 1 -- 172.21.15.102:0/1688161999 wait complete. 2023-03-03T16:44:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:44:57.807 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2023-03-03T16:45:11.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:45:12.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.058+0000 7f7438d9d700 1 Processor -- start 2023-03-03T16:45:12.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.058+0000 7f7438d9d700 1 -- start start 2023-03-03T16:45:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7438d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f74341564b0 0x7f74340cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7438d9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f74340c9610 0x7f74340cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:12.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7438d9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74340c9ca0 con 0x7f74340ca440 2023-03-03T16:45:12.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7438d9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74340b4920 con 0x7f74341564b0 2023-03-03T16:45:12.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7438d9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74340b4c00 con 0x7f74340c9610 2023-03-03T16:45:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f74337fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f74341564b0 0x7f74340cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7433fff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f74340ca440 0x7f74340ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36512/0 (socket says 172.21.15.102:36512) 2023-03-03T16:45:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7433fff700 1 -- 172.21.15.102:0/1492664559 learned_addr learned my addr 172.21.15.102:0/1492664559 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:45:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.060+0000 7f7432ffd700 1 --2- 172.21.15.102:0/1492664559 >> v2:172.21.15.172:3300/0 conn(0x7f74340c9610 0x7f74340cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:12.061 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74337fe700 1 -- 172.21.15.102:0/1492664559 >> v2:172.21.15.172:3300/0 conn(0x7f74340c9610 msgr2=0x7f74340cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:45:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74337fe700 1 --2- 172.21.15.102:0/1492664559 >> v2:172.21.15.172:3300/0 conn(0x7f74340c9610 0x7f74340cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74337fe700 1 -- 172.21.15.102:0/1492664559 >> v1:172.21.15.102:6789/0 conn(0x7f74340ca440 legacy=0x7f74340ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74337fe700 1 -- 172.21.15.102:0/1492664559 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340b46f0 con 0x7f74341564b0 2023-03-03T16:45:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74337fe700 1 --2- 172.21.15.102:0/1492664559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74341564b0 0x7f74340cb660 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f743415d410 tx=0x7f7424009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45d89e3609bba67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74327fc700 1 -- 172.21.15.102:0/1492664559 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7424011830 con 0x7f74341564b0 2023-03-03T16:45:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74327fc700 1 -- 172.21.15.102:0/1492664559 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7424011990 con 0x7f74341564b0 2023-03-03T16:45:12.062 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.061+0000 7f74327fc700 1 -- 172.21.15.102:0/1492664559 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7424011d20 con 0x7f74341564b0 2023-03-03T16:45:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 -- 172.21.15.102:0/1492664559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74341564b0 msgr2=0x7f74340cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1492664559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74341564b0 0x7f74340cb660 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f743415d410 tx=0x7f7424009b20 comp rx=0 tx=0).stop 2023-03-03T16:45:12.063 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 -- 172.21.15.102:0/1492664559 shutdown_connections 2023-03-03T16:45:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1492664559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74341564b0 0x7f74340cb660 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1492664559 >> v2:172.21.15.172:3300/0 conn(0x7f74340c9610 0x7f74340cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 -- 172.21.15.102:0/1492664559 >> 172.21.15.102:0/1492664559 conn(0x7f7434065fb0 msgr2=0x7f74340663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 -- 172.21.15.102:0/1492664559 shutdown_connections 2023-03-03T16:45:12.064 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.062+0000 7f7438d9d700 1 -- 172.21.15.102:0/1492664559 wait complete. 2023-03-03T16:45:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.064+0000 7f7438d9d700 1 Processor -- start 2023-03-03T16:45:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.064+0000 7f7438d9d700 1 -- start start 2023-03-03T16:45:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.065+0000 7f7438d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74340c9610 0x7f7434158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.065+0000 7f7438d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74340ca440 0x7f74341f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f7438d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74341564b0 0x7f74341f08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:12.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f7438d9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74340c8840 con 0x7f74340c9610 2023-03-03T16:45:12.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f7438d9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74341f0410 con 0x7f74340ca440 2023-03-03T16:45:12.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f7438d9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74340cf700 con 0x7f74341564b0 2023-03-03T16:45:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74340c9610 0x7f7434158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74340c9610 0x7f7434158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52672/0 (socket says 172.21.15.102:52672) 2023-03-03T16:45:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 -- 172.21.15.102:0/1595719793 learned_addr learned my addr 172.21.15.102:0/1595719793 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:45:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f7432ffd700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74340ca440 0x7f74341f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:12.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f7433fff700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74341564b0 0x7f74341f08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 -- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74341564b0 msgr2=0x7f74341f08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74341564b0 0x7f74341f08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 -- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74340ca440 msgr2=0x7f74341f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74340ca440 0x7f74341f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.066+0000 7f74337fe700 1 -- 172.21.15.102:0/1595719793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74240094a0 con 0x7f74340c9610 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.067+0000 7f74337fe700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74340c9610 0x7f7434158500 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f7424009b60 tx=0x7f7424004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.067+0000 7f741bfff700 1 -- 172.21.15.102:0/1595719793 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74240036a0 con 0x7f74340c9610 2023-03-03T16:45:12.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.067+0000 7f741bfff700 1 -- 172.21.15.102:0/1595719793 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742401e070 con 0x7f74340c9610 2023-03-03T16:45:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.067+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74340b2210 con 0x7f74340c9610 2023-03-03T16:45:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.067+0000 7f741bfff700 1 -- 172.21.15.102:0/1595719793 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7424003a20 con 0x7f74340c9610 2023-03-03T16:45:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.067+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74340b1f30 con 0x7f74340c9610 2023-03-03T16:45:12.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.069+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7434156880 con 0x7f74340c9610 2023-03-03T16:45:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.069+0000 7f741bfff700 1 -- 172.21.15.102:0/1595719793 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f742400f050 con 0x7f74340c9610 2023-03-03T16:45:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.071+0000 7f741bfff700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7420068320 0x7f742006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.071+0000 7f741bfff700 1 -- 172.21.15.102:0/1595719793 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f7424086d10 con 0x7f74340c9610 2023-03-03T16:45:12.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.071+0000 7f7432ffd700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7420068320 0x7f742006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:12.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.072+0000 7f7432ffd700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7420068320 0x7f742006a7d0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f74341525a0 tx=0x7f7428009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:12.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.075+0000 7f741bfff700 1 -- 172.21.15.102:0/1595719793 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f742408c020 con 0x7f74340c9610 2023-03-03T16:45:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.309+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74341f27f0 con 0x7f74340c9610 2023-03-03T16:45:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.310+0000 7f741bfff700 1 -- 172.21.15.102:0/1595719793 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f74341f27f0 con 0x7f74340c9610 2023-03-03T16:45:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:45:12.310 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:45:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.312+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7420068320 msgr2=0x7f742006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.312+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7420068320 0x7f742006a7d0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f74341525a0 tx=0x7f7428009210 comp rx=0 tx=0).stop 2023-03-03T16:45:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.312+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74340c9610 msgr2=0x7f7434158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.312+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74340c9610 0x7f7434158500 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f7424009b60 tx=0x7f7424004750 comp rx=0 tx=0).stop 2023-03-03T16:45:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 shutdown_connections 2023-03-03T16:45:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7420068320 0x7f742006a7d0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f74341564b0 0x7f74341f08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74340c9610 0x7f7434158500 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 --2- 172.21.15.102:0/1595719793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74340ca440 0x7f74341f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 >> 172.21.15.102:0/1595719793 conn(0x7f7434065fb0 msgr2=0x7f74340cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 shutdown_connections 2023-03-03T16:45:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:12.313+0000 7f7438d9d700 1 -- 172.21.15.102:0/1595719793 wait complete. 2023-03-03T16:45:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:45:12.339 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-03-03T16:45:20.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:45:20.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.600+0000 7f90553ff700 1 Processor -- start 2023-03-03T16:45:20.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.600+0000 7f90553ff700 1 -- start start 2023-03-03T16:45:20.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.601+0000 7f90553ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9050153290 0x7f9050158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:20.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.601+0000 7f90553ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90500cd8e0 0x7f90500ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:20.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.601+0000 7f90553ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9050153660 con 0x7f90501593c0 2023-03-03T16:45:20.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.601+0000 7f90553ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905006c360 con 0x7f90500cd8e0 2023-03-03T16:45:20.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.601+0000 7f90553ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905015cea0 con 0x7f9050153290 2023-03-03T16:45:20.602 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9050153290 0x7f9050158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904e7fc700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f90501593c0 0x7f90500cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51914/0 (socket says 172.21.15.102:51914) 2023-03-03T16:45:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904e7fc700 1 -- 172.21.15.102:0/868547691 learned_addr learned my addr 172.21.15.102:0/868547691 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:45:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904f7fe700 1 --2- 172.21.15.102:0/868547691 >> v2:172.21.15.102:3300/0 conn(0x7f90500cd8e0 0x7f90500ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904dffb700 1 -- 172.21.15.102:0/868547691 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954452860 0 0) 0x7f9050153660 con 0x7f90501593c0 2023-03-03T16:45:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904dffb700 1 -- 172.21.15.102:0/868547691 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90380035b0 con 0x7f90501593c0 2023-03-03T16:45:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904effd700 1 -- 172.21.15.102:0/868547691 >> v2:172.21.15.102:3300/0 conn(0x7f90500cd8e0 msgr2=0x7f90500ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:45:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904effd700 1 --2- 172.21.15.102:0/868547691 >> v2:172.21.15.102:3300/0 conn(0x7f90500cd8e0 0x7f90500ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904effd700 1 -- 172.21.15.102:0/868547691 >> v1:172.21.15.102:6789/0 conn(0x7f90501593c0 legacy=0x7f90500cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.602+0000 7f904effd700 1 -- 172.21.15.102:0/868547691 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905006c7f0 con 0x7f9050153290 2023-03-03T16:45:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.603+0000 7f904effd700 1 --2- 172.21.15.102:0/868547691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9050153290 0x7f9050158e80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9040009840 tx=0x7f9040009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb655278e492b41 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.603+0000 7f904dffb700 1 -- 172.21.15.102:0/868547691 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040011830 con 0x7f9050153290 2023-03-03T16:45:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.603+0000 7f904dffb700 1 -- 172.21.15.102:0/868547691 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9040011990 con 0x7f9050153290 2023-03-03T16:45:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.603+0000 7f904dffb700 1 -- 172.21.15.102:0/868547691 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040011d20 con 0x7f9050153290 2023-03-03T16:45:20.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 -- 172.21.15.102:0/868547691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9050153290 msgr2=0x7f9050158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:20.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 --2- 172.21.15.102:0/868547691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9050153290 0x7f9050158e80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9040009840 tx=0x7f9040009b50 comp rx=0 tx=0).stop 2023-03-03T16:45:20.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 -- 172.21.15.102:0/868547691 shutdown_connections 2023-03-03T16:45:20.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 --2- 172.21.15.102:0/868547691 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9050153290 0x7f9050158e80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 --2- 172.21.15.102:0/868547691 >> v2:172.21.15.102:3300/0 conn(0x7f90500cd8e0 0x7f90500ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 -- 172.21.15.102:0/868547691 >> 172.21.15.102:0/868547691 conn(0x7f9050065fb0 msgr2=0x7f90500663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:20.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 -- 172.21.15.102:0/868547691 shutdown_connections 2023-03-03T16:45:20.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.604+0000 7f90553ff700 1 -- 172.21.15.102:0/868547691 wait complete. 2023-03-03T16:45:20.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.607+0000 7f90553ff700 1 Processor -- start 2023-03-03T16:45:20.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.607+0000 7f90553ff700 1 -- start start 2023-03-03T16:45:20.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.608+0000 7f90553ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90500cd8e0 0x7f90501f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:20.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.608+0000 7f90553ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9050153290 0x7f9050155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:20.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.608+0000 7f90553ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90501593c0 0x7f90501561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:20.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.608+0000 7f90553ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9050151570 con 0x7f90501593c0 2023-03-03T16:45:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.608+0000 7f90553ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90500c8ec0 con 0x7f9050153290 2023-03-03T16:45:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.608+0000 7f90553ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90501f23b0 con 0x7f90500cd8e0 2023-03-03T16:45:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90500cd8e0 0x7f90501f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904e7fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9050153290 0x7f9050155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90500cd8e0 0x7f90501f1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:43018/0 (socket says 172.21.15.102:43018) 2023-03-03T16:45:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 -- 172.21.15.102:0/834504676 learned_addr learned my addr 172.21.15.102:0/834504676 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:45:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904f7fe700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90501593c0 0x7f90501561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:20.610 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 -- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9050153290 msgr2=0x7f9050155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:20.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9050153290 0x7f9050155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 -- 172.21.15.102:0/834504676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90501593c0 msgr2=0x7f90501561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:20.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90501593c0 0x7f90501561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 -- 172.21.15.102:0/834504676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90400094a0 con 0x7f90500cd8e0 2023-03-03T16:45:20.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.609+0000 7f904effd700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90500cd8e0 0x7f90501f1980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f90400090b0 tx=0x7f9040004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:20.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.610+0000 7f90377fe700 1 -- 172.21.15.102:0/834504676 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90400036a0 con 0x7f90500cd8e0 2023-03-03T16:45:20.611 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.610+0000 7f90377fe700 1 -- 172.21.15.102:0/834504676 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904001f070 con 0x7f90500cd8e0 2023-03-03T16:45:20.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.610+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90500ccde0 con 0x7f90500cd8e0 2023-03-03T16:45:20.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.610+0000 7f90377fe700 1 -- 172.21.15.102:0/834504676 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040003a40 con 0x7f90500cd8e0 2023-03-03T16:45:20.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.610+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905015d160 con 0x7f90500cd8e0 2023-03-03T16:45:20.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.611+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90500cd0c0 con 0x7f90500cd8e0 2023-03-03T16:45:20.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.613+0000 7f90377fe700 1 -- 172.21.15.102:0/834504676 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f904000f050 con 0x7f90500cd8e0 2023-03-03T16:45:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.614+0000 7f90377fe700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f903c0682f0 0x7f903c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.614+0000 7f90377fe700 1 -- 172.21.15.102:0/834504676 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9040021070 con 0x7f90500cd8e0 2023-03-03T16:45:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.614+0000 7f904e7fc700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f903c0682f0 0x7f903c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:20.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.615+0000 7f904e7fc700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f903c0682f0 0x7f903c06a7a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f9050157080 tx=0x7f9044007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:20.618 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.619+0000 7f90377fe700 1 -- 172.21.15.102:0/834504676 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9040053ce0 con 0x7f90500cd8e0 2023-03-03T16:45:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.855+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90501f2550 con 0x7f90500cd8e0 2023-03-03T16:45:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.856+0000 7f90377fe700 1 -- 172.21.15.102:0/834504676 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9040057300 con 0x7f90500cd8e0 2023-03-03T16:45:20.856 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:45:20.856 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:45:20.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f903c0682f0 msgr2=0x7f903c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:20.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f903c0682f0 0x7f903c06a7a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f9050157080 tx=0x7f9044007dc0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90500cd8e0 msgr2=0x7f90501f1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90500cd8e0 0x7f90501f1980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f90400090b0 tx=0x7f9040004750 comp rx=0 tx=0).stop 2023-03-03T16:45:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 shutdown_connections 2023-03-03T16:45:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f903c0682f0 0x7f903c06a7a0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f90500cd8e0 0x7f90501f1980 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90501593c0 0x7f90501561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 --2- 172.21.15.102:0/834504676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9050153290 0x7f9050155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 >> 172.21.15.102:0/834504676 conn(0x7f9050065fb0 msgr2=0x7f90500cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 shutdown_connections 2023-03-03T16:45:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:45:20.859+0000 7f90553ff700 1 -- 172.21.15.102:0/834504676 wait complete. 2023-03-03T16:45:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:45:20.885 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-03-03T16:45:27.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:27.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:57.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:57.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:27.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:27.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:57.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:57.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:11.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:47:12.135 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.135+0000 7f444db9d700 1 Processor -- start 2023-03-03T16:47:12.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.135+0000 7f444db9d700 1 -- start start 2023-03-03T16:47:12.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444db9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f444814f2f0 0x7f4448158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:12.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444db9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f44480c6dc0 0x7f44480c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:12.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444db9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44480c5530 con 0x7f44480cb2d0 2023-03-03T16:47:12.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444db9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44481594e0 con 0x7f44480c6dc0 2023-03-03T16:47:12.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444db9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4448070610 con 0x7f444814f2f0 2023-03-03T16:47:12.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444d39c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f44480cb2d0 0x7f444806e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56804/0 (socket says 172.21.15.102:56804) 2023-03-03T16:47:12.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444d39c700 1 -- 172.21.15.102:0/3772455344 learned_addr learned my addr 172.21.15.102:0/3772455344 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:47:12.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f4447fff700 1 --2- 172.21.15.102:0/3772455344 >> v2:172.21.15.172:3300/0 conn(0x7f44480c6dc0 0x7f44480c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:12.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.137+0000 7f444cb9b700 1 --2- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f444814f2f0 0x7f4448158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:12.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f44477fe700 1 -- 172.21.15.102:0/3772455344 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708759676 0 0) 0x7f44480c5530 con 0x7f44480cb2d0 2023-03-03T16:47:12.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f44477fe700 1 -- 172.21.15.102:0/3772455344 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44380034d0 con 0x7f44480cb2d0 2023-03-03T16:47:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f4447fff700 1 -- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f444814f2f0 msgr2=0x7f4448158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:47:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f4447fff700 1 --2- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f444814f2f0 0x7f4448158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f4447fff700 1 -- 172.21.15.102:0/3772455344 >> v1:172.21.15.172:6789/0 conn(0x7f44480cb2d0 legacy=0x7f444806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f4447fff700 1 -- 172.21.15.102:0/3772455344 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44480708f0 con 0x7f44480c6dc0 2023-03-03T16:47:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f444cb9b700 1 --2- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f444814f2f0 0x7f4448158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:47:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.138+0000 7f4447fff700 1 --2- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c59c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4434009840 tx=0x7f4434009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afa96868f768aaba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f44477fe700 1 -- 172.21.15.102:0/3772455344 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434011830 con 0x7f44480c6dc0 2023-03-03T16:47:12.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f44477fe700 1 -- 172.21.15.102:0/3772455344 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4434011990 con 0x7f44480c6dc0 2023-03-03T16:47:12.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f44477fe700 1 -- 172.21.15.102:0/3772455344 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434011d20 con 0x7f44480c6dc0 2023-03-03T16:47:12.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 -- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 msgr2=0x7f44480c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 --2- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c59c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4434009840 tx=0x7f4434009b50 comp rx=0 tx=0).stop 2023-03-03T16:47:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 -- 172.21.15.102:0/3772455344 shutdown_connections 2023-03-03T16:47:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 --2- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f444814f2f0 0x7f4448158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 --2- 172.21.15.102:0/3772455344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c59c0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 -- 172.21.15.102:0/3772455344 >> 172.21.15.102:0/3772455344 conn(0x7f44480ca7c0 msgr2=0x7f44480cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 -- 172.21.15.102:0/3772455344 shutdown_connections 2023-03-03T16:47:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.139+0000 7f444db9d700 1 -- 172.21.15.102:0/3772455344 wait complete. 2023-03-03T16:47:12.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.141+0000 7f444db9d700 1 Processor -- start 2023-03-03T16:47:12.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.142+0000 7f444db9d700 1 -- start start 2023-03-03T16:47:12.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444db9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444db9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44480cb2d0 0x7f44480c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444db9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f444814f2f0 0x7f44480c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444db9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44480bf9b0 con 0x7f444814f2f0 2023-03-03T16:47:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444db9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44480c28f0 con 0x7f44480c6dc0 2023-03-03T16:47:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444db9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44480c3470 con 0x7f44480cb2d0 2023-03-03T16:47:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444d39c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f444814f2f0 0x7f44480c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f4447fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44480cb2d0 0x7f44480c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:37956/0 (socket says 172.21.15.102:37956) 2023-03-03T16:47:12.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 -- 172.21.15.102:0/331322043 learned_addr learned my addr 172.21.15.102:0/331322043 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:47:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 -- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44480cb2d0 msgr2=0x7f44480c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44480cb2d0 0x7f44480c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 -- 172.21.15.102:0/331322043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f444814f2f0 msgr2=0x7f44480c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f444814f2f0 0x7f44480c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444cb9b700 1 -- 172.21.15.102:0/331322043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0094a0 con 0x7f44480c6dc0 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.143+0000 7f444d39c700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f444814f2f0 0x7f44480c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.144+0000 7f444cb9b700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c4ac0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f443c00ec40 tx=0x7f443c00ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.144+0000 7f44457fa700 1 -- 172.21.15.102:0/331322043 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c00c850 con 0x7f44480c6dc0 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.144+0000 7f44457fa700 1 -- 172.21.15.102:0/331322043 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c00c9b0 con 0x7f44480c6dc0 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.144+0000 7f44457fa700 1 -- 172.21.15.102:0/331322043 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c00cd20 con 0x7f44480c6dc0 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.144+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44340094a0 con 0x7f44480c6dc0 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.144+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44480c51c0 con 0x7f44480c6dc0 2023-03-03T16:47:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.145+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4448070bd0 con 0x7f44480c6dc0 2023-03-03T16:47:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.147+0000 7f44457fa700 1 -- 172.21.15.102:0/331322043 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f443c019030 con 0x7f44480c6dc0 2023-03-03T16:47:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.148+0000 7f44457fa700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f442c0682f0 0x7f442c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.148+0000 7f44457fa700 1 -- 172.21.15.102:0/331322043 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f443c085de0 con 0x7f44480c6dc0 2023-03-03T16:47:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.148+0000 7f4447fff700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f442c0682f0 0x7f442c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.149+0000 7f4447fff700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f442c0682f0 0x7f442c06a7a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4434000c00 tx=0x7f443401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:12.151 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.151+0000 7f44457fa700 1 -- 172.21.15.102:0/331322043 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f443c052480 con 0x7f44480c6dc0 2023-03-03T16:47:12.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.387+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f444814f6c0 con 0x7f44480c6dc0 2023-03-03T16:47:12.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.388+0000 7f44457fa700 1 -- 172.21.15.102:0/331322043 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f443c017090 con 0x7f44480c6dc0 2023-03-03T16:47:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:47:12.389 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:47:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.390+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f442c0682f0 msgr2=0x7f442c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f442c0682f0 0x7f442c06a7a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4434000c00 tx=0x7f443401c040 comp rx=0 tx=0).stop 2023-03-03T16:47:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 msgr2=0x7f44480c4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c4ac0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f443c00ec40 tx=0x7f443c00ef50 comp rx=0 tx=0).stop 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 shutdown_connections 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f444814f2f0 0x7f44480c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f44480c6dc0 0x7f44480c4ac0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f442c0682f0 0x7f442c06a7a0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 --2- 172.21.15.102:0/331322043 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f44480cb2d0 0x7f44480c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 >> 172.21.15.102:0/331322043 conn(0x7f44480ca7c0 msgr2=0x7f444814df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 shutdown_connections 2023-03-03T16:47:12.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:47:12.391+0000 7f444db9d700 1 -- 172.21.15.102:0/331322043 wait complete. 2023-03-03T16:47:12.393 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:47:12.419 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-03-03T16:47:27.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:27.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:57.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:57.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:27.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:27.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:50.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:48:50.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.686+0000 7f6bc2bff700 1 Processor -- start 2023-03-03T16:48:50.687 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.687+0000 7f6bc2bff700 1 -- start start 2023-03-03T16:48:50.688 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.688+0000 7f6bc2bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc153290 0x7f6bbc158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.688+0000 7f6bc2bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6bbc1593c0 0x7f6bbc0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.688+0000 7f6bc2bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bbc153660 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.688+0000 7f6bc2bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc06c360 con 0x7f6bbc1593c0 2023-03-03T16:48:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.688+0000 7f6bc2bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc15cea0 con 0x7f6bbc153290 2023-03-03T16:48:50.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc119c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6bbc0cd8e0 0x7f6bbc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:51298/0 (socket says 172.21.15.102:51298) 2023-03-03T16:48:50.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc119c700 1 -- 172.21.15.102:0/679304587 learned_addr learned my addr 172.21.15.102:0/679304587 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:48:50.690 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bbbfff700 1 --2- 172.21.15.102:0/679304587 >> v2:172.21.15.172:3300/0 conn(0x7f6bbc1593c0 0x7f6bbc0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:50.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc099b700 1 --2- 172.21.15.102:0/679304587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc153290 0x7f6bbc158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:50.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bbb7fe700 1 -- 172.21.15.102:0/679304587 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049359623 0 0) 0x7f6bbc153660 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bbb7fe700 1 -- 172.21.15.102:0/679304587 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bac0035b0 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc099b700 1 -- 172.21.15.102:0/679304587 >> v2:172.21.15.172:3300/0 conn(0x7f6bbc1593c0 msgr2=0x7f6bbc0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:48:50.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc099b700 1 --2- 172.21.15.102:0/679304587 >> v2:172.21.15.172:3300/0 conn(0x7f6bbc1593c0 0x7f6bbc0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc099b700 1 -- 172.21.15.102:0/679304587 >> v1:172.21.15.172:6789/0 conn(0x7f6bbc0cd8e0 legacy=0x7f6bbc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:50.691 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc099b700 1 -- 172.21.15.102:0/679304587 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc06c7f0 con 0x7f6bbc153290 2023-03-03T16:48:50.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bbbfff700 1 --2- 172.21.15.102:0/679304587 >> v2:172.21.15.172:3300/0 conn(0x7f6bbc1593c0 0x7f6bbc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:48:50.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.689+0000 7f6bc099b700 1 --2- 172.21.15.102:0/679304587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc153290 0x7f6bbc158e80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009840 tx=0x7f6bb0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deca7487002a390a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:50.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bbb7fe700 1 -- 172.21.15.102:0/679304587 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0011830 con 0x7f6bbc153290 2023-03-03T16:48:50.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bbb7fe700 1 -- 172.21.15.102:0/679304587 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb0011990 con 0x7f6bbc153290 2023-03-03T16:48:50.692 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bbb7fe700 1 -- 172.21.15.102:0/679304587 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0011d20 con 0x7f6bbc153290 2023-03-03T16:48:50.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bc2bff700 1 -- 172.21.15.102:0/679304587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc153290 msgr2=0x7f6bbc158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:50.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/679304587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc153290 0x7f6bbc158e80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009840 tx=0x7f6bb0009b50 comp rx=0 tx=0).stop 2023-03-03T16:48:50.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bc2bff700 1 -- 172.21.15.102:0/679304587 shutdown_connections 2023-03-03T16:48:50.693 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/679304587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc153290 0x7f6bbc158e80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/679304587 >> v2:172.21.15.172:3300/0 conn(0x7f6bbc1593c0 0x7f6bbc0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.690+0000 7f6bc2bff700 1 -- 172.21.15.102:0/679304587 >> 172.21.15.102:0/679304587 conn(0x7f6bbc065fb0 msgr2=0x7f6bbc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:50.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.691+0000 7f6bc2bff700 1 -- 172.21.15.102:0/679304587 shutdown_connections 2023-03-03T16:48:50.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.691+0000 7f6bc2bff700 1 -- 172.21.15.102:0/679304587 wait complete. 2023-03-03T16:48:50.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.693+0000 7f6bc2bff700 1 Processor -- start 2023-03-03T16:48:50.694 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.693+0000 7f6bc2bff700 1 -- start start 2023-03-03T16:48:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc2bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc0cd8e0 0x7f6bbc14ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc2bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bbc153290 0x7f6bbc1f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc2bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bbc1593c0 0x7f6bbc14dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc2bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc14d800 con 0x7f6bbc1593c0 2023-03-03T16:48:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc2bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc1f2240 con 0x7f6bbc153290 2023-03-03T16:48:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc2bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc15bb50 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.696 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc0cd8e0 0x7f6bbc14ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bbbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bbc153290 0x7f6bbc1f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc119c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bbc1593c0 0x7f6bbc14dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc099b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc0cd8e0 0x7f6bbc14ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:33070/0 (socket says 172.21.15.102:33070) 2023-03-03T16:48:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.694+0000 7f6bc099b700 1 -- 172.21.15.102:0/2280376417 learned_addr learned my addr 172.21.15.102:0/2280376417 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:48:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc099b700 1 -- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bbc153290 msgr2=0x7f6bbc1f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc099b700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bbc153290 0x7f6bbc1f1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc099b700 1 -- 172.21.15.102:0/2280376417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bbc1593c0 msgr2=0x7f6bbc14dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc099b700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bbc1593c0 0x7f6bbc14dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc099b700 1 -- 172.21.15.102:0/2280376417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80094a0 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bbbfff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bbc153290 0x7f6bbc1f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc099b700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc0cd8e0 0x7f6bbc14ca20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00090b0 tx=0x7f6bb0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bb97fa700 1 -- 172.21.15.102:0/2280376417 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb00036a0 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bb97fa700 1 -- 172.21.15.102:0/2280376417 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb001f070 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bb97fa700 1 -- 172.21.15.102:0/2280376417 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0003a40 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.698 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb00094a0 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.695+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc0ccde0 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.696+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc15d160 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.698+0000 7f6bb97fa700 1 -- 172.21.15.102:0/2280376417 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6bb000f050 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.699+0000 7f6bb97fa700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ba0068320 0x7f6ba006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.699+0000 7f6bb97fa700 1 -- 172.21.15.102:0/2280376417 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6bb0021070 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.700+0000 7f6bbbfff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ba0068320 0x7f6ba006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:50.700 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.700+0000 7f6bbbfff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ba0068320 0x7f6ba006a7d0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc1f2760 tx=0x7f6ba8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:50.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.703+0000 7f6bb97fa700 1 -- 172.21.15.102:0/2280376417 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6bb0053c80 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.937+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bbc0cd0c0 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.939+0000 7f6bb97fa700 1 -- 172.21.15.102:0/2280376417 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6bb00572a0 con 0x7f6bbc0cd8e0 2023-03-03T16:48:50.939 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:48:50.940 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:48:50.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.941+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ba0068320 msgr2=0x7f6ba006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:50.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.941+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ba0068320 0x7f6ba006a7d0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc1f2760 tx=0x7f6ba8009210 comp rx=0 tx=0).stop 2023-03-03T16:48:50.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.941+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc0cd8e0 msgr2=0x7f6bbc14ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:50.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.941+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc0cd8e0 0x7f6bbc14ca20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00090b0 tx=0x7f6bb0004750 comp rx=0 tx=0).stop 2023-03-03T16:48:50.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 shutdown_connections 2023-03-03T16:48:50.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6ba0068320 0x7f6ba006a7d0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bbc0cd8e0 0x7f6bbc14ca20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bbc1593c0 0x7f6bbc14dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 --2- 172.21.15.102:0/2280376417 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bbc153290 0x7f6bbc1f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:50.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 >> 172.21.15.102:0/2280376417 conn(0x7f6bbc065fb0 msgr2=0x7f6bbc0cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:50.943 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 shutdown_connections 2023-03-03T16:48:50.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:48:50.942+0000 7f6bc2bff700 1 -- 172.21.15.102:0/2280376417 wait complete. 2023-03-03T16:48:50.944 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:48:50.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2023-03-03T16:48:58.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:58.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:28.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:28.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:58.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:58.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:59.999 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T16:49:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:50:22.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:50:23.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.227+0000 7fe5f05ff700 1 Processor -- start 2023-03-03T16:50:23.227 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.227+0000 7fe5f05ff700 1 -- start start 2023-03-03T16:50:23.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.228+0000 7fe5f05ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5e8071960 0x7fe5e8071d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:23.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.228+0000 7fe5f05ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe5e80c7410 0x7fe5e80c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:23.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5f05ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8159890 con 0x7fe5e806a9a0 2023-03-03T16:50:23.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5f05ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e8072270 con 0x7fe5e8071960 2023-03-03T16:50:23.229 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5f05ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e80c84f0 con 0x7fe5e80c7410 2023-03-03T16:50:23.230 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5ee39b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5e8071960 0x7fe5e8071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:23.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5ee39b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5e8071960 0x7fe5e8071d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51832/0 (socket says 172.21.15.102:51832) 2023-03-03T16:50:23.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5ee39b700 1 -- 172.21.15.102:0/615205933 learned_addr learned my addr 172.21.15.102:0/615205933 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:50:23.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5edb9a700 1 --2- 172.21.15.102:0/615205933 >> v2:172.21.15.172:3300/0 conn(0x7fe5e80c7410 0x7fe5e80c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:23.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5ee39b700 1 -- 172.21.15.102:0/615205933 >> v2:172.21.15.172:3300/0 conn(0x7fe5e80c7410 msgr2=0x7fe5e80c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:50:23.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5ee39b700 1 --2- 172.21.15.102:0/615205933 >> v2:172.21.15.172:3300/0 conn(0x7fe5e80c7410 0x7fe5e80c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5ee39b700 1 -- 172.21.15.102:0/615205933 >> v1:172.21.15.102:6789/0 conn(0x7fe5e806a9a0 legacy=0x7fe5e815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:23.231 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.230+0000 7fe5ee39b700 1 -- 172.21.15.102:0/615205933 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8158f60 con 0x7fe5e8071960 2023-03-03T16:50:23.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.229+0000 7fe5ed399700 1 -- 172.21.15.102:0/615205933 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124745636 0 0) 0x7fe5e8159890 con 0x7fe5e806a9a0 2023-03-03T16:50:23.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.230+0000 7fe5ee39b700 1 --2- 172.21.15.102:0/615205933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e8071960 0x7fe5e8071d30 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80c58c0 tx=0x7fe5e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea9d4fe451941f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:23.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.230+0000 7fe5ed399700 1 -- 172.21.15.102:0/615205933 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4011830 con 0x7fe5e8071960 2023-03-03T16:50:23.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.230+0000 7fe5ed399700 1 -- 172.21.15.102:0/615205933 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e4011990 con 0x7fe5e8071960 2023-03-03T16:50:23.232 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.230+0000 7fe5ed399700 1 -- 172.21.15.102:0/615205933 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4011d20 con 0x7fe5e8071960 2023-03-03T16:50:23.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 -- 172.21.15.102:0/615205933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e8071960 msgr2=0x7fe5e8071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:23.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/615205933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e8071960 0x7fe5e8071d30 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80c58c0 tx=0x7fe5e4009bd0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 -- 172.21.15.102:0/615205933 shutdown_connections 2023-03-03T16:50:23.233 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/615205933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e8071960 0x7fe5e8071d30 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/615205933 >> v2:172.21.15.172:3300/0 conn(0x7fe5e80c7410 0x7fe5e80c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 -- 172.21.15.102:0/615205933 >> 172.21.15.102:0/615205933 conn(0x7fe5e806ea70 msgr2=0x7fe5e806ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:23.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 -- 172.21.15.102:0/615205933 shutdown_connections 2023-03-03T16:50:23.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.231+0000 7fe5f05ff700 1 -- 172.21.15.102:0/615205933 wait complete. 2023-03-03T16:50:23.234 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.233+0000 7fe5f05ff700 1 Processor -- start 2023-03-03T16:50:23.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.233+0000 7fe5f05ff700 1 -- start start 2023-03-03T16:50:23.235 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5f05ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e806a9a0 0x7fe5e80b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5f05ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e8071960 0x7fe5e814d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5f05ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e80c7410 0x7fe5e814df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5f05ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e814ed80 con 0x7fe5e806a9a0 2023-03-03T16:50:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5f05ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e814c9e0 con 0x7fe5e8071960 2023-03-03T16:50:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5f05ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e814d560 con 0x7fe5e80c7410 2023-03-03T16:50:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5ee39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e806a9a0 0x7fe5e80b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5ee39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e806a9a0 0x7fe5e80b0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51848/0 (socket says 172.21.15.102:51848) 2023-03-03T16:50:23.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.234+0000 7fe5ee39b700 1 -- 172.21.15.102:0/902615567 learned_addr learned my addr 172.21.15.102:0/902615567 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:50:23.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5edb9a700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e8071960 0x7fe5e814d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:23.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5eeb9c700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e80c7410 0x7fe5e814df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:23.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5ee39b700 1 -- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e80c7410 msgr2=0x7fe5e814df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:23.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5ee39b700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e80c7410 0x7fe5e814df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5ee39b700 1 -- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e8071960 msgr2=0x7fe5e814d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:23.237 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5ee39b700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e8071960 0x7fe5e814d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5ee39b700 1 -- 172.21.15.102:0/902615567 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4009550 con 0x7fe5e806a9a0 2023-03-03T16:50:23.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5ee39b700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e806a9a0 0x7fe5e80b0ad0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009c10 tx=0x7fe5e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:23.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.235+0000 7fe5deffd700 1 -- 172.21.15.102:0/902615567 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4004510 con 0x7fe5e806a9a0 2023-03-03T16:50:23.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.236+0000 7fe5deffd700 1 -- 172.21.15.102:0/902615567 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e40036a0 con 0x7fe5e806a9a0 2023-03-03T16:50:23.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.236+0000 7fe5deffd700 1 -- 172.21.15.102:0/902615567 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4003a10 con 0x7fe5e806a9a0 2023-03-03T16:50:23.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.236+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e8159520 con 0x7fe5e806a9a0 2023-03-03T16:50:23.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.236+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e8159240 con 0x7fe5e806a9a0 2023-03-03T16:50:23.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.237+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e8158c10 con 0x7fe5e806a9a0 2023-03-03T16:50:23.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.238+0000 7fe5deffd700 1 -- 172.21.15.102:0/902615567 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5e402d070 con 0x7fe5e806a9a0 2023-03-03T16:50:23.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.239+0000 7fe5deffd700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5cc068320 0x7fe5cc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:23.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.239+0000 7fe5deffd700 1 -- 172.21.15.102:0/902615567 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fe5e4021030 con 0x7fe5e806a9a0 2023-03-03T16:50:23.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.240+0000 7fe5edb9a700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5cc068320 0x7fe5cc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:23.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.240+0000 7fe5edb9a700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5cc068320 0x7fe5cc06a7d0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80ae160 tx=0x7fe5d8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:23.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.244+0000 7fe5deffd700 1 -- 172.21.15.102:0/902615567 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5e4053a20 con 0x7fe5e806a9a0 2023-03-03T16:50:23.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.478+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5e80c2470 con 0x7fe5e806a9a0 2023-03-03T16:50:23.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.478+0000 7fe5deffd700 1 -- 172.21.15.102:0/902615567 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fe5e4015090 con 0x7fe5e806a9a0 2023-03-03T16:50:23.479 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:50:23.479 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:50:23.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5cc068320 msgr2=0x7fe5cc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:23.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5cc068320 0x7fe5cc06a7d0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80ae160 tx=0x7fe5d8009210 comp rx=0 tx=0).stop 2023-03-03T16:50:23.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e806a9a0 msgr2=0x7fe5e80b0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:23.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e806a9a0 0x7fe5e80b0ad0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009c10 tx=0x7fe5e4004750 comp rx=0 tx=0).stop 2023-03-03T16:50:23.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 shutdown_connections 2023-03-03T16:50:23.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fe5cc068320 0x7fe5cc06a7d0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe5e80c7410 0x7fe5e814df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5e806a9a0 0x7fe5e80b0ad0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 --2- 172.21.15.102:0/902615567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe5e8071960 0x7fe5e814d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:23.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 >> 172.21.15.102:0/902615567 conn(0x7fe5e806ea70 msgr2=0x7fe5e8156a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:23.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.481+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 shutdown_connections 2023-03-03T16:50:23.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:50:23.482+0000 7fe5f05ff700 1 -- 172.21.15.102:0/902615567 wait complete. 2023-03-03T16:50:23.484 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:50:23.510 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T16:50:28.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:28.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:59.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:59.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:03.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:51:03.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.777+0000 7feeb3fff700 1 Processor -- start 2023-03-03T16:51:03.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.777+0000 7feeb3fff700 1 -- start start 2023-03-03T16:51:03.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeb3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40ced00 0x7feeb40c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:03.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeb3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feeb40c41b0 0x7feeb40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:03.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeb3fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb40c3d20 con 0x7feeb40b3970 2023-03-03T16:51:03.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeb3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb40c4930 con 0x7feeb40c41b0 2023-03-03T16:51:03.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeb3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb40cf610 con 0x7feeb40ced00 2023-03-03T16:51:03.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeabfff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7feeb40b3970 0x7feeb4158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59172/0 (socket says 172.21.15.102:59172) 2023-03-03T16:51:03.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeabfff700 1 -- 172.21.15.102:0/3725393994 learned_addr learned my addr 172.21.15.102:0/3725393994 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:51:03.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeb37fe700 1 --2- 172.21.15.102:0/3725393994 >> v2:172.21.15.172:3300/0 conn(0x7feeb40c41b0 0x7feeb40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:03.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.779+0000 7feeb2ffd700 1 --2- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40ced00 0x7feeb40c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:03.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb37fe700 1 -- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40ced00 msgr2=0x7feeb40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:51:03.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb37fe700 1 --2- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40ced00 0x7feeb40c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:03.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb37fe700 1 -- 172.21.15.102:0/3725393994 >> v1:172.21.15.102:6789/0 conn(0x7feeb40b3970 legacy=0x7feeb4158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:03.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb27fc700 1 -- 172.21.15.102:0/3725393994 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531572075 0 0) 0x7feeb40c3d20 con 0x7feeb40b3970 2023-03-03T16:51:03.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb37fe700 1 -- 172.21.15.102:0/3725393994 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb415cba0 con 0x7feeb40c41b0 2023-03-03T16:51:03.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb37fe700 1 --2- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 0x7feeb40cf0d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7feeac009dc0 tx=0x7feeac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f74d76ee2047501 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb27fc700 1 -- 172.21.15.102:0/3725393994 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac00e070 con 0x7feeb40c41b0 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb27fc700 1 -- 172.21.15.102:0/3725393994 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac00bae0 con 0x7feeb40c41b0 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.780+0000 7feeb27fc700 1 -- 172.21.15.102:0/3725393994 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac00be90 con 0x7feeb40c41b0 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 -- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 msgr2=0x7feeb40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 0x7feeb40cf0d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7feeac009dc0 tx=0x7feeac00b330 comp rx=0 tx=0).stop 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 -- 172.21.15.102:0/3725393994 shutdown_connections 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 0x7feeb40cf0d0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3725393994 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40ced00 0x7feeb40c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:03.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 -- 172.21.15.102:0/3725393994 >> 172.21.15.102:0/3725393994 conn(0x7feeb408b1c0 msgr2=0x7feeb4071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:51:03.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 -- 172.21.15.102:0/3725393994 shutdown_connections 2023-03-03T16:51:03.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.781+0000 7feeb3fff700 1 -- 172.21.15.102:0/3725393994 wait complete. 2023-03-03T16:51:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.783+0000 7feeb3fff700 1 Processor -- start 2023-03-03T16:51:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.783+0000 7feeb3fff700 1 -- start start 2023-03-03T16:51:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.784+0000 7feeb3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40b3970 0x7feeb4159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.784+0000 7feeb3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 0x7feeb415aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.784+0000 7feeb3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb40ced00 0x7feeb415b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.784+0000 7feeb3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb4159a30 con 0x7feeb40ced00 2023-03-03T16:51:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.784+0000 7feeb3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb41480c0 con 0x7feeb40c41b0 2023-03-03T16:51:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.784+0000 7feeb3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb4148360 con 0x7feeb40b3970 2023-03-03T16:51:03.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40b3970 0x7feeb4159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:03.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40b3970 0x7feeb4159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:52604/0 (socket says 172.21.15.102:52604) 2023-03-03T16:51:03.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 -- 172.21.15.102:0/3877779618 learned_addr learned my addr 172.21.15.102:0/3877779618 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:51:03.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeabfff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 0x7feeb415aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:03.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb37fe700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb40ced00 0x7feeb415b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:03.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 -- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 msgr2=0x7feeb415aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:03.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 0x7feeb415aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:03.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 -- 172.21.15.102:0/3877779618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb40ced00 msgr2=0x7feeb415b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:03.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb40ced00 0x7feeb415b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:03.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 -- 172.21.15.102:0/3877779618 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac009a70 con 0x7feeb40b3970 2023-03-03T16:51:03.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.785+0000 7feeb2ffd700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40b3970 0x7feeb4159f30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9b0 tx=0x7feea400ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:03.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.786+0000 7feeab7fe700 1 -- 172.21.15.102:0/3877779618 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4009710 con 0x7feeb40b3970 2023-03-03T16:51:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.786+0000 7feeab7fe700 1 -- 172.21.15.102:0/3877779618 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea400ee50 con 0x7feeb40b3970 2023-03-03T16:51:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.786+0000 7feeab7fe700 1 -- 172.21.15.102:0/3877779618 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea400ca50 con 0x7feeb40b3970 2023-03-03T16:51:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.786+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb406bf20 con 0x7feeb40b3970 2023-03-03T16:51:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.786+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb415ce80 con 0x7feeb40b3970 2023-03-03T16:51:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.787+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb406c200 con 0x7feeb40b3970 2023-03-03T16:51:03.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.788+0000 7feeab7fe700 1 -- 172.21.15.102:0/3877779618 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feea4023070 con 0x7feeb40b3970 2023-03-03T16:51:03.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.789+0000 7feeab7fe700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee94068320 0x7fee9406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:03.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.790+0000 7feeab7fe700 1 -- 172.21.15.102:0/3877779618 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7feea4085f80 con 0x7feeb40b3970 2023-03-03T16:51:03.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.790+0000 7feeabfff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee94068320 0x7fee9406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:03.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.790+0000 7feeabfff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee94068320 0x7fee9406a7d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7feeb4159c50 tx=0x7fee9c007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:03.793 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:03.794+0000 7feeab7fe700 1 -- 172.21.15.102:0/3877779618 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feea4052630 con 0x7feeb40b3970 2023-03-03T16:51:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.027+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feeb404e6e0 con 0x7feeb40b3970 2023-03-03T16:51:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.028+0000 7feeab7fe700 1 -- 172.21.15.102:0/3877779618 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7feea4055c50 con 0x7feeb40b3970 2023-03-03T16:51:04.028 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:51:04.028 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:51:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee94068320 msgr2=0x7fee9406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee94068320 0x7fee9406a7d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7feeb4159c50 tx=0x7fee9c007e70 comp rx=0 tx=0).stop 2023-03-03T16:51:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40b3970 msgr2=0x7feeb4159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40b3970 0x7feeb4159f30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9b0 tx=0x7feea400ecc0 comp rx=0 tx=0).stop 2023-03-03T16:51:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 shutdown_connections 2023-03-03T16:51:04.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fee94068320 0x7fee9406a7d0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7feeb40b3970 0x7feeb4159f30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeb40ced00 0x7feeb415b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 --2- 172.21.15.102:0/3877779618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feeb40c41b0 0x7feeb415aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 >> 172.21.15.102:0/3877779618 conn(0x7feeb408b1c0 msgr2=0x7feeb40ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:51:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 shutdown_connections 2023-03-03T16:51:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:51:04.031+0000 7feeb3fff700 1 -- 172.21.15.102:0/3877779618 wait complete. 2023-03-03T16:51:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:51:04.058 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2023-03-03T16:51:29.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:29.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:59.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:59.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:06.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:52:06.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.325+0000 7f1c7f59e700 1 Processor -- start 2023-03-03T16:52:06.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.326+0000 7f1c7f59e700 1 -- start start 2023-03-03T16:52:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.327+0000 7f1c7f59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c800b3970 0x7f1c800c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.327+0000 7f1c7f59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c800c3d20 con 0x7f1c800c41b0 2023-03-03T16:52:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.327+0000 7f1c7f59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8015cba0 con 0x7f1c800ced00 2023-03-03T16:52:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.327+0000 7f1c7f59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c800c4930 con 0x7f1c800b3970 2023-03-03T16:52:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c77fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c800b3970 0x7f1c800c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:06.328 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c7e59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1c800ced00 0x7f1c80158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:38590/0 (socket says 172.21.15.102:38590) 2023-03-03T16:52:06.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c7e59c700 1 -- 172.21.15.102:0/3719902643 learned_addr learned my addr 172.21.15.102:0/3719902643 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:52:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c7dd9b700 1 -- 172.21.15.102:0/3719902643 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568037473 0 0) 0x7f1c8015cba0 con 0x7f1c800ced00 2023-03-03T16:52:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c7dd9b700 1 -- 172.21.15.102:0/3719902643 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c0035b0 con 0x7f1c800ced00 2023-03-03T16:52:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c7dd9b700 1 -- 172.21.15.102:0/3719902643 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111170950 0 0) 0x7f1c800c3d20 con 0x7f1c800c41b0 2023-03-03T16:52:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c7dd9b700 1 -- 172.21.15.102:0/3719902643 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c8015cba0 con 0x7f1c800c41b0 2023-03-03T16:52:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c77fff700 1 -- 172.21.15.102:0/3719902643 >> v1:172.21.15.172:6789/0 conn(0x7f1c800ced00 legacy=0x7f1c80158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c77fff700 1 -- 172.21.15.102:0/3719902643 >> v1:172.21.15.102:6789/0 conn(0x7f1c800c41b0 legacy=0x7f1c80163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.328+0000 7f1c77fff700 1 -- 172.21.15.102:0/3719902643 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8015ce80 con 0x7f1c800b3970 2023-03-03T16:52:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.329+0000 7f1c77fff700 1 --2- 172.21.15.102:0/3719902643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3970 0x7f1c800c5a90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009840 tx=0x7f1c68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c55fb5e207c09446 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.329+0000 7f1c7dd9b700 1 -- 172.21.15.102:0/3719902643 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68011830 con 0x7f1c800b3970 2023-03-03T16:52:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.329+0000 7f1c7dd9b700 1 -- 172.21.15.102:0/3719902643 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c68011990 con 0x7f1c800b3970 2023-03-03T16:52:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.329+0000 7f1c7dd9b700 1 -- 172.21.15.102:0/3719902643 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68011d20 con 0x7f1c800b3970 2023-03-03T16:52:06.331 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.329+0000 7f1c7f59e700 1 -- 172.21.15.102:0/3719902643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3970 msgr2=0x7f1c800c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.329+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/3719902643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3970 0x7f1c800c5a90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009840 tx=0x7f1c68009b50 comp rx=0 tx=0).stop 2023-03-03T16:52:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.330+0000 7f1c7f59e700 1 -- 172.21.15.102:0/3719902643 shutdown_connections 2023-03-03T16:52:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.330+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/3719902643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800b3970 0x7f1c800c5a90 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.330+0000 7f1c7f59e700 1 -- 172.21.15.102:0/3719902643 >> 172.21.15.102:0/3719902643 conn(0x7f1c8008b1c0 msgr2=0x7f1c80071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.330+0000 7f1c7f59e700 1 -- 172.21.15.102:0/3719902643 shutdown_connections 2023-03-03T16:52:06.332 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.330+0000 7f1c7f59e700 1 -- 172.21.15.102:0/3719902643 wait complete. 2023-03-03T16:52:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.332+0000 7f1c7f59e700 1 Processor -- start 2023-03-03T16:52:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.332+0000 7f1c7f59e700 1 -- start start 2023-03-03T16:52:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7f59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c800b3970 0x7f1c800b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7f59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800c41b0 0x7f1c800c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800ced00 0x7f1c801e5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7f59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8015bd40 con 0x7f1c800b3970 2023-03-03T16:52:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800c6fb0 con 0x7f1c800ced00 2023-03-03T16:52:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7f59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8015b0f0 con 0x7f1c800c41b0 2023-03-03T16:52:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c800b3970 0x7f1c800b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c77fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800c41b0 0x7f1c800c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c800b3970 0x7f1c800b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43100/0 (socket says 172.21.15.102:43100) 2023-03-03T16:52:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7e59c700 1 -- 172.21.15.102:0/1640872569 learned_addr learned my addr 172.21.15.102:0/1640872569 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.333+0000 7f1c7ed9d700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800ced00 0x7f1c801e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c77fff700 1 -- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800ced00 msgr2=0x7f1c801e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c77fff700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800ced00 0x7f1c801e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c77fff700 1 -- 172.21.15.102:0/1640872569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c800b3970 msgr2=0x7f1c800b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c77fff700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c800b3970 0x7f1c800b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c77fff700 1 -- 172.21.15.102:0/1640872569 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c680094a0 con 0x7f1c800c41b0 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c7e59c700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c800b3970 0x7f1c800b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:52:06.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c77fff700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800c41b0 0x7f1c800c6270 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1c8008ce30 tx=0x7f1c68004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:06.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c76ffd700 1 -- 172.21.15.102:0/1640872569 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6801a8f0 con 0x7f1c800c41b0 2023-03-03T16:52:06.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c76ffd700 1 -- 172.21.15.102:0/1640872569 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6801aa70 con 0x7f1c800c41b0 2023-03-03T16:52:06.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.334+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8006c200 con 0x7f1c800c41b0 2023-03-03T16:52:06.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.335+0000 7f1c76ffd700 1 -- 172.21.15.102:0/1640872569 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6801ae20 con 0x7f1c800c41b0 2023-03-03T16:52:06.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.335+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8006bf20 con 0x7f1c800c41b0 2023-03-03T16:52:06.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.336+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8015c890 con 0x7f1c800c41b0 2023-03-03T16:52:06.338 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.337+0000 7f1c76ffd700 1 -- 172.21.15.102:0/1640872569 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1c6800f050 con 0x7f1c800c41b0 2023-03-03T16:52:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.338+0000 7f1c76ffd700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c600682f0 0x7f1c6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.338+0000 7f1c76ffd700 1 -- 172.21.15.102:0/1640872569 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1c68086e70 con 0x7f1c800c41b0 2023-03-03T16:52:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.338+0000 7f1c7e59c700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c600682f0 0x7f1c6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.339+0000 7f1c7e59c700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c600682f0 0x7f1c6006a7a0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f1c70000f60 tx=0x7f1c70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.342+0000 7f1c76ffd700 1 -- 172.21.15.102:0/1640872569 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1c68053580 con 0x7f1c800c41b0 2023-03-03T16:52:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.576+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c8015b5a0 con 0x7f1c800c41b0 2023-03-03T16:52:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.577+0000 7f1c76ffd700 1 -- 172.21.15.102:0/1640872569 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1c8015b5a0 con 0x7f1c800c41b0 2023-03-03T16:52:06.577 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:52:06.578 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:52:06.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c600682f0 msgr2=0x7f1c6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:06.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c600682f0 0x7f1c6006a7a0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f1c70000f60 tx=0x7f1c70009040 comp rx=0 tx=0).stop 2023-03-03T16:52:06.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800c41b0 msgr2=0x7f1c800c6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:06.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800c41b0 0x7f1c800c6270 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1c8008ce30 tx=0x7f1c68004750 comp rx=0 tx=0).stop 2023-03-03T16:52:06.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 shutdown_connections 2023-03-03T16:52:06.580 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c800ced00 0x7f1c801e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:06.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c800b3970 0x7f1c800b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:06.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1c600682f0 0x7f1c6006a7a0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:06.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 --2- 172.21.15.102:0/1640872569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1c800c41b0 0x7f1c800c6270 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:06.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 >> 172.21.15.102:0/1640872569 conn(0x7f1c8008b1c0 msgr2=0x7f1c800cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:06.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 shutdown_connections 2023-03-03T16:52:06.581 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:06.579+0000 7f1c7f59e700 1 -- 172.21.15.102:0/1640872569 wait complete. 2023-03-03T16:52:06.582 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:52:06.607 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2023-03-03T16:52:15.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:52:15.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.868+0000 7f8f3339d700 1 Processor -- start 2023-03-03T16:52:15.868 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.868+0000 7f8f3339d700 1 -- start start 2023-03-03T16:52:15.869 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f3339d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c1564b0 0x7f8f2c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f3339d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8f2c0c9610 0x7f8f2c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f3339d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2c0c9ca0 con 0x7f8f2c0ca440 2023-03-03T16:52:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f3339d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f2c0b4920 con 0x7f8f2c0c9610 2023-03-03T16:52:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f3339d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c0b4c00 con 0x7f8f2c1564b0 2023-03-03T16:52:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f31b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8f2c0c9610 0x7f8f2c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f31b9a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8f2c0c9610 0x7f8f2c0cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41382/0 (socket says 172.21.15.102:41382) 2023-03-03T16:52:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f31b9a700 1 -- 172.21.15.102:0/592052130 learned_addr learned my addr 172.21.15.102:0/592052130 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:52:15.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.869+0000 7f8f3239b700 1 --2- 172.21.15.102:0/592052130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c1564b0 0x7f8f2c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:15.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31b9a700 1 -- 172.21.15.102:0/592052130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c1564b0 msgr2=0x7f8f2c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:52:15.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31b9a700 1 --2- 172.21.15.102:0/592052130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c1564b0 0x7f8f2c0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:15.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31b9a700 1 -- 172.21.15.102:0/592052130 >> v1:172.21.15.172:6789/0 conn(0x7f8f2c0ca440 legacy=0x7f8f2c0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:15.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31b9a700 1 -- 172.21.15.102:0/592052130 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c0b46f0 con 0x7f8f2c0c9610 2023-03-03T16:52:15.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31b9a700 1 --2- 172.21.15.102:0/592052130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0c9610 0x7f8f2c0cbba0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0098f0 tx=0x7f8f1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67aea726d806b739 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:15.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31399700 1 -- 172.21.15.102:0/592052130 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c011830 con 0x7f8f2c0c9610 2023-03-03T16:52:15.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31399700 1 -- 172.21.15.102:0/592052130 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c011990 con 0x7f8f2c0c9610 2023-03-03T16:52:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.870+0000 7f8f31399700 1 -- 172.21.15.102:0/592052130 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c011d20 con 0x7f8f2c0c9610 2023-03-03T16:52:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 -- 172.21.15.102:0/592052130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0c9610 msgr2=0x7f8f2c0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 --2- 172.21.15.102:0/592052130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0c9610 0x7f8f2c0cbba0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0098f0 tx=0x7f8f1c009c00 comp rx=0 tx=0).stop 2023-03-03T16:52:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 -- 172.21.15.102:0/592052130 shutdown_connections 2023-03-03T16:52:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 --2- 172.21.15.102:0/592052130 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c1564b0 0x7f8f2c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 --2- 172.21.15.102:0/592052130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0c9610 0x7f8f2c0cbba0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 -- 172.21.15.102:0/592052130 >> 172.21.15.102:0/592052130 conn(0x7f8f2c065fb0 msgr2=0x7f8f2c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:15.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 -- 172.21.15.102:0/592052130 shutdown_connections 2023-03-03T16:52:15.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.871+0000 7f8f3339d700 1 -- 172.21.15.102:0/592052130 wait complete. 2023-03-03T16:52:15.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.873+0000 7f8f3339d700 1 Processor -- start 2023-03-03T16:52:15.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.873+0000 7f8f3339d700 1 -- start start 2023-03-03T16:52:15.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.874+0000 7f8f3339d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c0c9610 0x7f8f2c1e4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:15.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.874+0000 7f8f3339d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0ca440 0x7f8f2c159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:15.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.874+0000 7f8f3339d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f2c1564b0 0x7f8f2c14d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:15.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.874+0000 7f8f3339d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c0c5f30 con 0x7f8f2c0ca440 2023-03-03T16:52:15.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.874+0000 7f8f3339d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c158910 con 0x7f8f2c1564b0 2023-03-03T16:52:15.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.874+0000 7f8f3339d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2c157be0 con 0x7f8f2c0c9610 2023-03-03T16:52:15.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c0c9610 0x7f8f2c1e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:15.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f31b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0ca440 0x7f8f2c159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:15.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c0c9610 0x7f8f2c1e4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:42872/0 (socket says 172.21.15.102:42872) 2023-03-03T16:52:15.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 -- 172.21.15.102:0/1335813238 learned_addr learned my addr 172.21.15.102:0/1335813238 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:52:15.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f32b9c700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f2c1564b0 0x7f8f2c14d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:15.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 -- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f2c1564b0 msgr2=0x7f8f2c14d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:15.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f2c1564b0 0x7f8f2c14d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:15.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 -- 172.21.15.102:0/1335813238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0ca440 msgr2=0x7f8f2c159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:15.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0ca440 0x7f8f2c159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:15.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 -- 172.21.15.102:0/1335813238 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c009550 con 0x7f8f2c0c9610 2023-03-03T16:52:15.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f3239b700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c0c9610 0x7f8f2c1e4280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e8a0 tx=0x7f8f2800ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:15.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.875+0000 7f8f22ffd700 1 -- 172.21.15.102:0/1335813238 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28009590 con 0x7f8f2c0c9610 2023-03-03T16:52:15.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.876+0000 7f8f22ffd700 1 -- 172.21.15.102:0/1335813238 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f28013680 con 0x7f8f2c0c9610 2023-03-03T16:52:15.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.876+0000 7f8f22ffd700 1 -- 172.21.15.102:0/1335813238 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280139f0 con 0x7f8f2c0c9610 2023-03-03T16:52:15.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.876+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f2c0b2210 con 0x7f8f2c0c9610 2023-03-03T16:52:15.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.876+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2c0b1f30 con 0x7f8f2c0c9610 2023-03-03T16:52:15.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.877+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2c156880 con 0x7f8f2c0c9610 2023-03-03T16:52:15.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.878+0000 7f8f22ffd700 1 -- 172.21.15.102:0/1335813238 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8f28023070 con 0x7f8f2c0c9610 2023-03-03T16:52:15.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.879+0000 7f8f22ffd700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f100682f0 0x7f8f1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:15.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.879+0000 7f8f22ffd700 1 -- 172.21.15.102:0/1335813238 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8f28085e20 con 0x7f8f2c0c9610 2023-03-03T16:52:15.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.879+0000 7f8f31b9a700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f100682f0 0x7f8f1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:15.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.880+0000 7f8f31b9a700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f100682f0 0x7f8f1006a7a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009c40 tx=0x7f8f1c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:15.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:15.884+0000 7f8f22ffd700 1 -- 172.21.15.102:0/1335813238 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8f280524d0 con 0x7f8f2c0c9610 2023-03-03T16:52:16.119 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.119+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f2c1591b0 con 0x7f8f2c0c9610 2023-03-03T16:52:16.120 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.120+0000 7f8f22ffd700 1 -- 172.21.15.102:0/1335813238 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8f28055af0 con 0x7f8f2c0c9610 2023-03-03T16:52:16.120 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:52:16.120 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:52:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f100682f0 msgr2=0x7f8f1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f100682f0 0x7f8f1006a7a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009c40 tx=0x7f8f1c008030 comp rx=0 tx=0).stop 2023-03-03T16:52:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c0c9610 msgr2=0x7f8f2c1e4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c0c9610 0x7f8f2c1e4280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e8a0 tx=0x7f8f2800ebb0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 shutdown_connections 2023-03-03T16:52:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8f100682f0 0x7f8f1006a7a0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8f2c0c9610 0x7f8f2c1e4280 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f2c1564b0 0x7f8f2c14d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 --2- 172.21.15.102:0/1335813238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f2c0ca440 0x7f8f2c159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 >> 172.21.15.102:0/1335813238 conn(0x7f8f2c065fb0 msgr2=0x7f8f2c0ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:16.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 shutdown_connections 2023-03-03T16:52:16.124 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.122+0000 7f8f3339d700 1 -- 172.21.15.102:0/1335813238 wait complete. 2023-03-03T16:52:16.125 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:52:16.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:52:16.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.395+0000 7f97901ff700 1 Processor -- start 2023-03-03T16:52:16.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.396+0000 7f97901ff700 1 -- start start 2023-03-03T16:52:16.396 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.397+0000 7f97901ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9788153290 0x7f9788158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:16.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.397+0000 7f97901ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9788153660 con 0x7f97881593c0 2023-03-03T16:52:16.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.397+0000 7f97901ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978806c360 con 0x7f97880cd8e0 2023-03-03T16:52:16.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.397+0000 7f97901ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97880cec40 con 0x7f9788153290 2023-03-03T16:52:16.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.397+0000 7f978d79a700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f97881593c0 0x7f97880cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55592/0 (socket says 172.21.15.102:55592) 2023-03-03T16:52:16.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.397+0000 7f978d79a700 1 -- 172.21.15.102:0/2356602699 learned_addr learned my addr 172.21.15.102:0/2356602699 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:52:16.397 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.397+0000 7f978df9b700 1 --2- 172.21.15.102:0/2356602699 >> v2:172.21.15.172:3300/0 conn(0x7f9788153290 0x7f9788158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:16.398 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.398+0000 7f978df9b700 1 -- 172.21.15.102:0/2356602699 >> v1:172.21.15.172:6789/0 conn(0x7f97880cd8e0 legacy=0x7f97880ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.398+0000 7f978df9b700 1 -- 172.21.15.102:0/2356602699 >> v1:172.21.15.102:6789/0 conn(0x7f97881593c0 legacy=0x7f97880cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.398+0000 7f978df9b700 1 -- 172.21.15.102:0/2356602699 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978815cea0 con 0x7f9788153290 2023-03-03T16:52:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.398+0000 7f978cf99700 1 -- 172.21.15.102:0/2356602699 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315089130 0 0) 0x7f978806c360 con 0x7f97880cd8e0 2023-03-03T16:52:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.398+0000 7f978df9b700 1 --2- 172.21.15.102:0/2356602699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9788153290 0x7f9788158e80 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9784009840 tx=0x7f9784009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcc8aa52012ca4a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.398+0000 7f978cf99700 1 -- 172.21.15.102:0/2356602699 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784011830 con 0x7f9788153290 2023-03-03T16:52:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f978cf99700 1 -- 172.21.15.102:0/2356602699 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784011990 con 0x7f9788153290 2023-03-03T16:52:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f978cf99700 1 -- 172.21.15.102:0/2356602699 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784011d20 con 0x7f9788153290 2023-03-03T16:52:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f97901ff700 1 -- 172.21.15.102:0/2356602699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9788153290 msgr2=0x7f9788158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f97901ff700 1 --2- 172.21.15.102:0/2356602699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9788153290 0x7f9788158e80 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9784009840 tx=0x7f9784009b50 comp rx=0 tx=0).stop 2023-03-03T16:52:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f97901ff700 1 -- 172.21.15.102:0/2356602699 shutdown_connections 2023-03-03T16:52:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f97901ff700 1 --2- 172.21.15.102:0/2356602699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9788153290 0x7f9788158e80 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f97901ff700 1 -- 172.21.15.102:0/2356602699 >> 172.21.15.102:0/2356602699 conn(0x7f9788065fb0 msgr2=0x7f97880663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f97901ff700 1 -- 172.21.15.102:0/2356602699 shutdown_connections 2023-03-03T16:52:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.399+0000 7f97901ff700 1 -- 172.21.15.102:0/2356602699 wait complete. 2023-03-03T16:52:16.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.401+0000 7f97901ff700 1 Processor -- start 2023-03-03T16:52:16.401 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.401+0000 7f97901ff700 1 -- start start 2023-03-03T16:52:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.402+0000 7f97901ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97880cd8e0 0x7f978815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.402+0000 7f97901ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9788153290 0x7f97881549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.402+0000 7f97901ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97881593c0 0x7f9788152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:16.402 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.402+0000 7f97901ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97881f2170 con 0x7f97880cd8e0 2023-03-03T16:52:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.402+0000 7f97901ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97881f1880 con 0x7f97881593c0 2023-03-03T16:52:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f97901ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97881eee50 con 0x7f9788153290 2023-03-03T16:52:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978e79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97881593c0 0x7f9788152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978d79a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9788153290 0x7f97881549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978df9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97880cd8e0 0x7f978815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978e79c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97881593c0 0x7f9788152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56574/0 (socket says 172.21.15.102:56574) 2023-03-03T16:52:16.403 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978e79c700 1 -- 172.21.15.102:0/918453714 learned_addr learned my addr 172.21.15.102:0/918453714 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:52:16.404 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978df9b700 1 -- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9788153290 msgr2=0x7f97881549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978df9b700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9788153290 0x7f97881549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978df9b700 1 -- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97881593c0 msgr2=0x7f9788152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978df9b700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97881593c0 0x7f9788152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978df9b700 1 -- 172.21.15.102:0/918453714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97840094a0 con 0x7f97880cd8e0 2023-03-03T16:52:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978e79c700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97881593c0 0x7f9788152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:52:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.403+0000 7f978d79a700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9788153290 0x7f97881549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:52:16.405 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.404+0000 7f978df9b700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97880cd8e0 0x7f978815c1e0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9784016000 tx=0x7f9784004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.404+0000 7f977e7fc700 1 -- 172.21.15.102:0/918453714 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978401a7e0 con 0x7f97880cd8e0 2023-03-03T16:52:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.404+0000 7f977e7fc700 1 -- 172.21.15.102:0/918453714 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978401a940 con 0x7f97880cd8e0 2023-03-03T16:52:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.404+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978815d160 con 0x7f97880cd8e0 2023-03-03T16:52:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.404+0000 7f977e7fc700 1 -- 172.21.15.102:0/918453714 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978401acb0 con 0x7f97880cd8e0 2023-03-03T16:52:16.406 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.404+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978806c7f0 con 0x7f97880cd8e0 2023-03-03T16:52:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.405+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97880ccde0 con 0x7f97880cd8e0 2023-03-03T16:52:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.406+0000 7f977e7fc700 1 -- 172.21.15.102:0/918453714 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f978402b030 con 0x7f97880cd8e0 2023-03-03T16:52:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.408+0000 7f977e7fc700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f976c068320 0x7f976c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:16.407 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.408+0000 7f977e7fc700 1 -- 172.21.15.102:0/918453714 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9784086ff0 con 0x7f97880cd8e0 2023-03-03T16:52:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.408+0000 7f978d79a700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f976c068320 0x7f976c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.408+0000 7f978d79a700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f976c068320 0x7f976c06a7d0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9788158050 tx=0x7f9778009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.412+0000 7f977e7fc700 1 -- 172.21.15.102:0/918453714 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97840b85e0 con 0x7f97880cd8e0 2023-03-03T16:52:16.647 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.648+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97880c9490 con 0x7f97880cd8e0 2023-03-03T16:52:16.648 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.649+0000 7f977e7fc700 1 -- 172.21.15.102:0/918453714 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9784029090 con 0x7f97880cd8e0 2023-03-03T16:52:16.648 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:52:16.649 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:52:16.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.651+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f976c068320 msgr2=0x7f976c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.651+0000 7f97901ff700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f976c068320 0x7f976c06a7d0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9788158050 tx=0x7f9778009040 comp rx=0 tx=0).stop 2023-03-03T16:52:16.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.651+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97880cd8e0 msgr2=0x7f978815c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:16.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.651+0000 7f97901ff700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97880cd8e0 0x7f978815c1e0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f9784016000 tx=0x7f9784004750 comp rx=0 tx=0).stop 2023-03-03T16:52:16.651 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 shutdown_connections 2023-03-03T16:52:16.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97881593c0 0x7f9788152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97880cd8e0 0x7f978815c1e0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f976c068320 0x7f976c06a7d0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 --2- 172.21.15.102:0/918453714 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9788153290 0x7f97881549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:16.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 >> 172.21.15.102:0/918453714 conn(0x7f9788065fb0 msgr2=0x7f97880cb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:16.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 shutdown_connections 2023-03-03T16:52:16.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:52:16.652+0000 7f97901ff700 1 -- 172.21.15.102:0/918453714 wait complete. 2023-03-03T16:52:16.653 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:52:16.677 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-03-03T16:52:29.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:29.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:59.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:59.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:30.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:30.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:00.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:00.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:06.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:54:06.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.927+0000 7fcdde39d700 1 Processor -- start 2023-03-03T16:54:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.927+0000 7fcdde39d700 1 -- start start 2023-03-03T16:54:06.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.929+0000 7fcdde39d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcdd80c5a90 0x7fcdd80c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:06.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.929+0000 7fcdde39d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd80c3d20 con 0x7fcdd80ced00 2023-03-03T16:54:06.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.929+0000 7fcdde39d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd815cba0 con 0x7fcdd80b3970 2023-03-03T16:54:06.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.929+0000 7fcdde39d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80c4930 con 0x7fcdd80c5a90 2023-03-03T16:54:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.929+0000 7fcdddb9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcdd80c5a90 0x7fcdd80c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.929+0000 7fcddd39b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcdd80ced00 0x7fcdd8158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34134/0 (socket says 172.21.15.102:34134) 2023-03-03T16:54:06.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.929+0000 7fcddd39b700 1 -- 172.21.15.102:0/4092513595 learned_addr learned my addr 172.21.15.102:0/4092513595 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:54:06.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdddb9c700 1 -- 172.21.15.102:0/4092513595 >> v1:172.21.15.172:6789/0 conn(0x7fcdd80b3970 legacy=0x7fcdd8161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:06.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdddb9c700 1 -- 172.21.15.102:0/4092513595 >> v1:172.21.15.102:6789/0 conn(0x7fcdd80ced00 legacy=0x7fcdd8158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:06.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdddb9c700 1 -- 172.21.15.102:0/4092513595 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd815ce80 con 0x7fcdd80c5a90 2023-03-03T16:54:06.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdcffff700 1 -- 172.21.15.102:0/4092513595 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207827544 0 0) 0x7fcdd815cba0 con 0x7fcdd80b3970 2023-03-03T16:54:06.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdddb9c700 1 --2- 172.21.15.102:0/4092513595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80c5a90 0x7fcdd80c41b0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009d90 tx=0x7fcdd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4caa1faf3790a31 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:06.930 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdcffff700 1 -- 172.21.15.102:0/4092513595 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd000e070 con 0x7fcdd80c5a90 2023-03-03T16:54:06.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdcffff700 1 -- 172.21.15.102:0/4092513595 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd000bad0 con 0x7fcdd80c5a90 2023-03-03T16:54:06.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.930+0000 7fcdcffff700 1 -- 172.21.15.102:0/4092513595 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd000be80 con 0x7fcdd80c5a90 2023-03-03T16:54:06.931 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.931+0000 7fcdde39d700 1 -- 172.21.15.102:0/4092513595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80c5a90 msgr2=0x7fcdd80c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.931+0000 7fcdde39d700 1 --2- 172.21.15.102:0/4092513595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80c5a90 0x7fcdd80c41b0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009d90 tx=0x7fcdd000b330 comp rx=0 tx=0).stop 2023-03-03T16:54:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.931+0000 7fcdde39d700 1 -- 172.21.15.102:0/4092513595 shutdown_connections 2023-03-03T16:54:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.931+0000 7fcdde39d700 1 --2- 172.21.15.102:0/4092513595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80c5a90 0x7fcdd80c41b0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.931+0000 7fcdde39d700 1 -- 172.21.15.102:0/4092513595 >> 172.21.15.102:0/4092513595 conn(0x7fcdd808b1c0 msgr2=0x7fcdd8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.931+0000 7fcdde39d700 1 -- 172.21.15.102:0/4092513595 shutdown_connections 2023-03-03T16:54:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.931+0000 7fcdde39d700 1 -- 172.21.15.102:0/4092513595 wait complete. 2023-03-03T16:54:06.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.933+0000 7fcdde39d700 1 Processor -- start 2023-03-03T16:54:06.933 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.933+0000 7fcdde39d700 1 -- start start 2023-03-03T16:54:06.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.934+0000 7fcdde39d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80b3970 0x7fcdd81e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:06.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.934+0000 7fcdde39d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd80c5a90 0x7fcdd81efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:06.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.934+0000 7fcdde39d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd80ced00 0x7fcdd81f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:06.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.934+0000 7fcdde39d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd814dc90 con 0x7fcdd80b3970 2023-03-03T16:54:06.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.934+0000 7fcdde39d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd814cec0 con 0x7fcdd80ced00 2023-03-03T16:54:06.934 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.934+0000 7fcdde39d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd81ef850 con 0x7fcdd80c5a90 2023-03-03T16:54:06.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.934+0000 7fcddd39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80b3970 0x7fcdd81e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:06.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcdddb9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd80ced00 0x7fcdd81f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:06.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddcb9a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd80c5a90 0x7fcdd81efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:06.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80b3970 0x7fcdd81e3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35126/0 (socket says 172.21.15.102:35126) 2023-03-03T16:54:06.935 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 -- 172.21.15.102:0/1058035444 learned_addr learned my addr 172.21.15.102:0/1058035444 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:54:06.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 -- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd80c5a90 msgr2=0x7fcdd81efce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:06.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd80c5a90 0x7fcdd81efce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:06.936 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 -- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd80ced00 msgr2=0x7fcdd81f0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:06.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd80ced00 0x7fcdd81f0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:06.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 -- 172.21.15.102:0/1058035444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0009a70 con 0x7fcdd80b3970 2023-03-03T16:54:06.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddcb9a700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd80c5a90 0x7fcdd81efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:54:06.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.935+0000 7fcddd39b700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80b3970 0x7fcdd81e3b30 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400c7b0 tx=0x7fcdd400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:06.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.936+0000 7fcdcdffb700 1 -- 172.21.15.102:0/1058035444 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4004c80 con 0x7fcdd80b3970 2023-03-03T16:54:06.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.936+0000 7fcdcdffb700 1 -- 172.21.15.102:0/1058035444 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd4004de0 con 0x7fcdd80b3970 2023-03-03T16:54:06.937 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.936+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd806c200 con 0x7fcdd80b3970 2023-03-03T16:54:06.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.936+0000 7fcdcdffb700 1 -- 172.21.15.102:0/1058035444 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4007310 con 0x7fcdd80b3970 2023-03-03T16:54:06.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.936+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd806bf20 con 0x7fcdd80b3970 2023-03-03T16:54:06.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.937+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd815c890 con 0x7fcdd80b3970 2023-03-03T16:54:06.938 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.938+0000 7fcdcdffb700 1 -- 172.21.15.102:0/1058035444 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcdd401a020 con 0x7fcdd80b3970 2023-03-03T16:54:06.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.940+0000 7fcdcdffb700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcdbc0682f0 0x7fcdbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:06.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.940+0000 7fcdcdffb700 1 -- 172.21.15.102:0/1058035444 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcdd40859f0 con 0x7fcdd80b3970 2023-03-03T16:54:06.939 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.940+0000 7fcddcb9a700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcdbc0682f0 0x7fcdbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.941+0000 7fcddcb9a700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcdbc0682f0 0x7fcdbc06a7a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80cf960 tx=0x7fcdc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:06.943+0000 7fcdcdffb700 1 -- 172.21.15.102:0/1058035444 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcdd4052100 con 0x7fcdd80b3970 2023-03-03T16:54:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.175+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd80c9ef0 con 0x7fcdd80b3970 2023-03-03T16:54:07.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.176+0000 7fcdcdffb700 1 -- 172.21.15.102:0/1058035444 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcdd4055720 con 0x7fcdd80b3970 2023-03-03T16:54:07.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:54:07.175 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:54:07.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcdbc0682f0 msgr2=0x7fcdbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:07.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcdbc0682f0 0x7fcdbc06a7a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80cf960 tx=0x7fcdc8009040 comp rx=0 tx=0).stop 2023-03-03T16:54:07.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80b3970 msgr2=0x7fcdd81e3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:07.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80b3970 0x7fcdd81e3b30 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400c7b0 tx=0x7fcdd400cac0 comp rx=0 tx=0).stop 2023-03-03T16:54:07.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 shutdown_connections 2023-03-03T16:54:07.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcdd80ced00 0x7fcdd81f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:07.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdd80b3970 0x7fcdd81e3b30 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:07.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcdbc0682f0 0x7fcdbc06a7a0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:07.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 --2- 172.21.15.102:0/1058035444 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcdd80c5a90 0x7fcdd81efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:07.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 >> 172.21.15.102:0/1058035444 conn(0x7fcdd808b1c0 msgr2=0x7fcdd80cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:07.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.178+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 shutdown_connections 2023-03-03T16:54:07.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:54:07.179+0000 7fcdde39d700 1 -- 172.21.15.102:0/1058035444 wait complete. 2023-03-03T16:54:07.180 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:54:07.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T16:54:30.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:30.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:00.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:00.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:14.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:14.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.455+0000 7f70375ff700 1 Processor -- start 2023-03-03T16:55:14.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.455+0000 7f70375ff700 1 -- start start 2023-03-03T16:55:14.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.456+0000 7f70375ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7030153290 0x7f7030158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:14.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.456+0000 7f70375ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70300cd8e0 0x7f70300ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:14.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.456+0000 7f70375ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7030153660 con 0x7f70301593c0 2023-03-03T16:55:14.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.456+0000 7f70375ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f703006c360 con 0x7f70300cd8e0 2023-03-03T16:55:14.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.456+0000 7f70375ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703015cea0 con 0x7f7030153290 2023-03-03T16:55:14.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f703539b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7030153290 0x7f7030158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:14.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f7034b9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f70301593c0 0x7f70300cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34438/0 (socket says 172.21.15.102:34438) 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f7034b9a700 1 -- 172.21.15.102:0/726901030 learned_addr learned my addr 172.21.15.102:0/726901030 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f7035b9c700 1 --2- 172.21.15.102:0/726901030 >> v2:172.21.15.172:3300/0 conn(0x7f70300cd8e0 0x7f70300ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f7027fff700 1 -- 172.21.15.102:0/726901030 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989444203 0 0) 0x7f7030153660 con 0x7f70301593c0 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f7027fff700 1 -- 172.21.15.102:0/726901030 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c0035b0 con 0x7f70301593c0 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f703539b700 1 -- 172.21.15.102:0/726901030 >> v2:172.21.15.172:3300/0 conn(0x7f70300cd8e0 msgr2=0x7f70300ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f703539b700 1 --2- 172.21.15.102:0/726901030 >> v2:172.21.15.172:3300/0 conn(0x7f70300cd8e0 0x7f70300ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f703539b700 1 -- 172.21.15.102:0/726901030 >> v1:172.21.15.172:6789/0 conn(0x7f70301593c0 legacy=0x7f70300cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f703539b700 1 -- 172.21.15.102:0/726901030 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703006c7f0 con 0x7f7030153290 2023-03-03T16:55:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f7035b9c700 1 --2- 172.21.15.102:0/726901030 >> v2:172.21.15.172:3300/0 conn(0x7f70300cd8e0 0x7f70300ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:55:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.457+0000 7f703539b700 1 --2- 172.21.15.102:0/726901030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7030153290 0x7f7030158e80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f702c009840 tx=0x7f702c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe1683acf7f16e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f7027fff700 1 -- 172.21.15.102:0/726901030 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c011830 con 0x7f7030153290 2023-03-03T16:55:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f7027fff700 1 -- 172.21.15.102:0/726901030 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c011990 con 0x7f7030153290 2023-03-03T16:55:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f7027fff700 1 -- 172.21.15.102:0/726901030 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c011d20 con 0x7f7030153290 2023-03-03T16:55:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f70375ff700 1 -- 172.21.15.102:0/726901030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7030153290 msgr2=0x7f7030158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f70375ff700 1 --2- 172.21.15.102:0/726901030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7030153290 0x7f7030158e80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f702c009840 tx=0x7f702c009b50 comp rx=0 tx=0).stop 2023-03-03T16:55:14.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f70375ff700 1 -- 172.21.15.102:0/726901030 shutdown_connections 2023-03-03T16:55:14.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f70375ff700 1 --2- 172.21.15.102:0/726901030 >> v2:172.21.15.172:3300/0 conn(0x7f70300cd8e0 0x7f70300ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f70375ff700 1 --2- 172.21.15.102:0/726901030 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f7030153290 0x7f7030158e80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.458+0000 7f70375ff700 1 -- 172.21.15.102:0/726901030 >> 172.21.15.102:0/726901030 conn(0x7f7030065fb0 msgr2=0x7f70300663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:14.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.459+0000 7f70375ff700 1 -- 172.21.15.102:0/726901030 shutdown_connections 2023-03-03T16:55:14.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.459+0000 7f70375ff700 1 -- 172.21.15.102:0/726901030 wait complete. 2023-03-03T16:55:14.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.460+0000 7f70375ff700 1 Processor -- start 2023-03-03T16:55:14.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.461+0000 7f70375ff700 1 -- start start 2023-03-03T16:55:14.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f70375ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70300cd8e0 0x7f703014e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f70375ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7030153290 0x7f70301ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f70375ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70301593c0 0x7f70301eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f70375ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70300c6b50 con 0x7f7030153290 2023-03-03T16:55:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f70375ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703014cf60 con 0x7f70301593c0 2023-03-03T16:55:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f70375ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70300c6580 con 0x7f70300cd8e0 2023-03-03T16:55:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f703539b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70300cd8e0 0x7f703014e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f7035b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70301593c0 0x7f70301eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:14.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f703539b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70300cd8e0 0x7f703014e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:38264/0 (socket says 172.21.15.102:38264) 2023-03-03T16:55:14.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f7034b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7030153290 0x7f70301ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:14.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f703539b700 1 -- 172.21.15.102:0/855937602 learned_addr learned my addr 172.21.15.102:0/855937602 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:55:14.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f7035b9c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70301593c0 0x7f70301eec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:52688/0 (socket says 172.21.15.102:52688) 2023-03-03T16:55:14.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f703539b700 1 -- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70301593c0 msgr2=0x7f70301eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:14.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f703539b700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70301593c0 0x7f70301eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.465 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.462+0000 7f703539b700 1 -- 172.21.15.102:0/855937602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7030153290 msgr2=0x7f70301ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f703539b700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7030153290 0x7f70301ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f703539b700 1 -- 172.21.15.102:0/855937602 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7028009a70 con 0x7f70300cd8e0 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f703539b700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70300cd8e0 0x7f703014e9b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f702c000c00 tx=0x7f702c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f7025ffb700 1 -- 172.21.15.102:0/855937602 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c004550 con 0x7f70300cd8e0 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f7025ffb700 1 -- 172.21.15.102:0/855937602 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c01f070 con 0x7f70300cd8e0 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f7025ffb700 1 -- 172.21.15.102:0/855937602 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c0038c0 con 0x7f70300cd8e0 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c0094a0 con 0x7f70300cd8e0 2023-03-03T16:55:14.466 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.463+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70300ccde0 con 0x7f70300cd8e0 2023-03-03T16:55:14.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.464+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703015d160 con 0x7f70300cd8e0 2023-03-03T16:55:14.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.466+0000 7f7025ffb700 1 -- 172.21.15.102:0/855937602 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f702c00f050 con 0x7f70300cd8e0 2023-03-03T16:55:14.467 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.467+0000 7f7025ffb700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70140682f0 0x7f701406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:14.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.467+0000 7f7025ffb700 1 -- 172.21.15.102:0/855937602 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f702c021070 con 0x7f70300cd8e0 2023-03-03T16:55:14.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.467+0000 7f7034b9a700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70140682f0 0x7f701406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:14.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.468+0000 7f7034b9a700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70140682f0 0x7f701406a7a0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f7030073c90 tx=0x7f7020007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:14.471 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.471+0000 7f7025ffb700 1 -- 172.21.15.102:0/855937602 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f702c053ce0 con 0x7f70300cd8e0 2023-03-03T16:55:14.708 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.707+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70300cd0c0 con 0x7f70300cd8e0 2023-03-03T16:55:14.709 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.709+0000 7f7025ffb700 1 -- 172.21.15.102:0/855937602 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f702c057300 con 0x7f70300cd8e0 2023-03-03T16:55:14.709 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:55:14.709 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:55:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70140682f0 msgr2=0x7f701406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70140682f0 0x7f701406a7a0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f7030073c90 tx=0x7f7020007dc0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70300cd8e0 msgr2=0x7f703014e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70300cd8e0 0x7f703014e9b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f702c000c00 tx=0x7f702c004750 comp rx=0 tx=0).stop 2023-03-03T16:55:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 shutdown_connections 2023-03-03T16:55:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f70140682f0 0x7f701406a7a0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f70300cd8e0 0x7f703014e9b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70301593c0 0x7f70301eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 --2- 172.21.15.102:0/855937602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7030153290 0x7f70301ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 >> 172.21.15.102:0/855937602 conn(0x7f7030065fb0 msgr2=0x7f70300cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 shutdown_connections 2023-03-03T16:55:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:14.711+0000 7f70375ff700 1 -- 172.21.15.102:0/855937602 wait complete. 2023-03-03T16:55:14.714 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:55:14.739 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 37 secs before thrashing 2023-03-03T16:55:31.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:31.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:51.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:51.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.995+0000 7f76471ff700 1 Processor -- start 2023-03-03T16:55:51.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.995+0000 7f76471ff700 1 -- start start 2023-03-03T16:55:51.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.996+0000 7f76471ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7640153290 0x7f7640158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:51.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.996+0000 7f76471ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7640153660 con 0x7f76400cd8e0 2023-03-03T16:55:51.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.996+0000 7f76471ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764006c360 con 0x7f76401593c0 2023-03-03T16:55:51.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.996+0000 7f76471ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76400cec40 con 0x7f7640153290 2023-03-03T16:55:51.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.997+0000 7f7644f9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7640153290 0x7f7640158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:51.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.997+0000 7f763ffff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f76401593c0 0x7f76400cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42752/0 (socket says 172.21.15.102:42752) 2023-03-03T16:55:51.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.997+0000 7f763ffff700 1 -- 172.21.15.102:0/1762909937 learned_addr learned my addr 172.21.15.102:0/1762909937 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:55:51.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.997+0000 7f763f7fe700 1 -- 172.21.15.102:0/1762909937 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640773181 0 0) 0x7f764006c360 con 0x7f76401593c0 2023-03-03T16:55:51.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.997+0000 7f7644f9b700 1 -- 172.21.15.102:0/1762909937 >> v1:172.21.15.172:6789/0 conn(0x7f76401593c0 legacy=0x7f76400cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:51.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.997+0000 7f7644f9b700 1 -- 172.21.15.102:0/1762909937 >> v1:172.21.15.102:6789/0 conn(0x7f76400cd8e0 legacy=0x7f76400ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.997+0000 7f7644f9b700 1 -- 172.21.15.102:0/1762909937 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764015cea0 con 0x7f7640153290 2023-03-03T16:55:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.998+0000 7f7644f9b700 1 --2- 172.21.15.102:0/1762909937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f7640158e80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f7634009840 tx=0x7f7634009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17483af0b10e0bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.998+0000 7f763f7fe700 1 -- 172.21.15.102:0/1762909937 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634004230 con 0x7f7640153290 2023-03-03T16:55:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.998+0000 7f763f7fe700 1 -- 172.21.15.102:0/1762909937 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7634004390 con 0x7f7640153290 2023-03-03T16:55:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.998+0000 7f763f7fe700 1 -- 172.21.15.102:0/1762909937 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340036a0 con 0x7f7640153290 2023-03-03T16:55:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.998+0000 7f76471ff700 1 -- 172.21.15.102:0/1762909937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 msgr2=0x7f7640158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.998+0000 7f76471ff700 1 --2- 172.21.15.102:0/1762909937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f7640158e80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f7634009840 tx=0x7f7634009b50 comp rx=0 tx=0).stop 2023-03-03T16:55:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.999+0000 7f76471ff700 1 -- 172.21.15.102:0/1762909937 shutdown_connections 2023-03-03T16:55:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.999+0000 7f76471ff700 1 --2- 172.21.15.102:0/1762909937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f7640158e80 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.999+0000 7f76471ff700 1 -- 172.21.15.102:0/1762909937 >> 172.21.15.102:0/1762909937 conn(0x7f7640065fb0 msgr2=0x7f76400663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.999+0000 7f76471ff700 1 -- 172.21.15.102:0/1762909937 shutdown_connections 2023-03-03T16:55:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:51.999+0000 7f76471ff700 1 -- 172.21.15.102:0/1762909937 wait complete. 2023-03-03T16:55:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.001+0000 7f76471ff700 1 Processor -- start 2023-03-03T16:55:52.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.001+0000 7f76471ff700 1 -- start start 2023-03-03T16:55:52.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f76471ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76400cd8e0 0x7f76400cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:52.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f76471ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f764014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:52.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f76471ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76401593c0 0x7f764014e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:52.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f76471ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76400c6b10 con 0x7f7640153290 2023-03-03T16:55:52.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f76471ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76401ef8e0 con 0x7f76400cd8e0 2023-03-03T16:55:52.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f76471ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76401ef380 con 0x7f76401593c0 2023-03-03T16:55:52.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f763ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f764014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:52.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f763ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f764014e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47426/0 (socket says 172.21.15.102:47426) 2023-03-03T16:55:52.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f763ffff700 1 -- 172.21.15.102:0/1365911669 learned_addr learned my addr 172.21.15.102:0/1365911669 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:55:52.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f7644f9b700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76400cd8e0 0x7f76400cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:52.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.002+0000 7f764579c700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76401593c0 0x7f764014e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:52.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763ffff700 1 -- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76401593c0 msgr2=0x7f764014e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:52.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763ffff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76401593c0 0x7f764014e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:52.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763ffff700 1 -- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76400cd8e0 msgr2=0x7f76400cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:52.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763ffff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76400cd8e0 0x7f76400cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:52.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763ffff700 1 -- 172.21.15.102:0/1365911669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76340094a0 con 0x7f7640153290 2023-03-03T16:55:52.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f764579c700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76401593c0 0x7f764014e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:55:52.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763ffff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f764014e140 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f762c00c580 tx=0x7f762c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:52.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763d7fa700 1 -- 172.21.15.102:0/1365911669 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c004c80 con 0x7f7640153290 2023-03-03T16:55:52.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763d7fa700 1 -- 172.21.15.102:0/1365911669 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c004e00 con 0x7f7640153290 2023-03-03T16:55:52.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f763d7fa700 1 -- 172.21.15.102:0/1365911669 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c007310 con 0x7f7640153290 2023-03-03T16:55:52.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764015d160 con 0x7f7640153290 2023-03-03T16:55:52.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.003+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764006c7f0 con 0x7f7640153290 2023-03-03T16:55:52.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.004+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76400ccde0 con 0x7f7640153290 2023-03-03T16:55:52.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.006+0000 7f763d7fa700 1 -- 172.21.15.102:0/1365911669 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f762c01a020 con 0x7f7640153290 2023-03-03T16:55:52.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.007+0000 7f763d7fa700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7624068320 0x7f762406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:52.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.007+0000 7f763d7fa700 1 -- 172.21.15.102:0/1365911669 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f762c085930 con 0x7f7640153290 2023-03-03T16:55:52.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.007+0000 7f7644f9b700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7624068320 0x7f762406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:52.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.008+0000 7f7644f9b700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7624068320 0x7f762406a7d0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f76340090b0 tx=0x7f7634008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:52.011 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.011+0000 7f763d7fa700 1 -- 172.21.15.102:0/1365911669 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f762c0520f0 con 0x7f7640153290 2023-03-03T16:55:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.246+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7640158810 con 0x7f7640153290 2023-03-03T16:55:52.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.247+0000 7f763d7fa700 1 -- 172.21.15.102:0/1365911669 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f762c055710 con 0x7f7640153290 2023-03-03T16:55:52.249 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:55:52.249 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:55:52.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7624068320 msgr2=0x7f762406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:52.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7624068320 0x7f762406a7d0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f76340090b0 tx=0x7f7634008030 comp rx=0 tx=0).stop 2023-03-03T16:55:52.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 msgr2=0x7f764014e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:52.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f764014e140 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f762c00c580 tx=0x7f762c00c890 comp rx=0 tx=0).stop 2023-03-03T16:55:52.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 shutdown_connections 2023-03-03T16:55:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f7624068320 0x7f762406a7d0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f76401593c0 0x7f764014e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76400cd8e0 0x7f76400cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 --2- 172.21.15.102:0/1365911669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7640153290 0x7f764014e140 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 >> 172.21.15.102:0/1365911669 conn(0x7f7640065fb0 msgr2=0x7f764015b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 shutdown_connections 2023-03-03T16:55:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:55:52.249+0000 7f76471ff700 1 -- 172.21.15.102:0/1365911669 wait complete. 2023-03-03T16:55:52.252 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:55:52.276 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T16:56:01.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:01.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:31.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:31.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:01.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:01.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:19.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:57:19.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.528+0000 7f24a5fff700 1 Processor -- start 2023-03-03T16:57:19.528 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.528+0000 7f24a5fff700 1 -- start start 2023-03-03T16:57:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f24a5fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24a0153290 0x7f24a0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f24a5fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a00cd8e0 0x7f24a00ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f24a5fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a0153660 con 0x7f24a01593c0 2023-03-03T16:57:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f24a5fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a006c360 con 0x7f24a0153290 2023-03-03T16:57:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f24a5fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a015cea0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f249f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24a0153290 0x7f24a0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f249effd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f24a01593c0 0x7f24a00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57304/0 (socket says 172.21.15.102:57304) 2023-03-03T16:57:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f249f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24a0153290 0x7f24a0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49364/0 (socket says 172.21.15.102:49364) 2023-03-03T16:57:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f249effd700 1 -- 172.21.15.102:0/3411456118 learned_addr learned my addr 172.21.15.102:0/3411456118 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:57:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.530+0000 7f249ffff700 1 --2- 172.21.15.102:0/3411456118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a00cd8e0 0x7f24a00ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249f7fe700 1 -- 172.21.15.102:0/3411456118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a00cd8e0 msgr2=0x7f24a00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:57:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249f7fe700 1 --2- 172.21.15.102:0/3411456118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a00cd8e0 0x7f24a00ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249f7fe700 1 -- 172.21.15.102:0/3411456118 >> v1:172.21.15.102:6789/0 conn(0x7f24a01593c0 legacy=0x7f24a00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249f7fe700 1 -- 172.21.15.102:0/3411456118 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a006c7f0 con 0x7f24a0153290 2023-03-03T16:57:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249e7fc700 1 -- 172.21.15.102:0/3411456118 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650750076 0 0) 0x7f24a0153660 con 0x7f24a01593c0 2023-03-03T16:57:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249f7fe700 1 --2- 172.21.15.102:0/3411456118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a0158e80 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f2490009840 tx=0x7f2490009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6f0b942b08d2d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:57:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249e7fc700 1 -- 172.21.15.102:0/3411456118 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490011830 con 0x7f24a0153290 2023-03-03T16:57:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249e7fc700 1 -- 172.21.15.102:0/3411456118 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2490011990 con 0x7f24a0153290 2023-03-03T16:57:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.531+0000 7f249e7fc700 1 -- 172.21.15.102:0/3411456118 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490011d20 con 0x7f24a0153290 2023-03-03T16:57:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 -- 172.21.15.102:0/3411456118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 msgr2=0x7f24a0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 --2- 172.21.15.102:0/3411456118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a0158e80 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f2490009840 tx=0x7f2490009b50 comp rx=0 tx=0).stop 2023-03-03T16:57:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 -- 172.21.15.102:0/3411456118 shutdown_connections 2023-03-03T16:57:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 --2- 172.21.15.102:0/3411456118 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a00cd8e0 0x7f24a00ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 --2- 172.21.15.102:0/3411456118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a0158e80 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 -- 172.21.15.102:0/3411456118 >> 172.21.15.102:0/3411456118 conn(0x7f24a0065fb0 msgr2=0x7f24a00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:57:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 -- 172.21.15.102:0/3411456118 shutdown_connections 2023-03-03T16:57:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.532+0000 7f24a5fff700 1 -- 172.21.15.102:0/3411456118 wait complete. 2023-03-03T16:57:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.534+0000 7f24a5fff700 1 Processor -- start 2023-03-03T16:57:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.534+0000 7f24a5fff700 1 -- start start 2023-03-03T16:57:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f24a5fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24a00cd8e0 0x7f24a00b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f24a5fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a00b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f24a5fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a01593c0 0x7f24a00caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f24a5fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a0157350 con 0x7f24a0153290 2023-03-03T16:57:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f24a5fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a00c6cc0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f24a5fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a01e44c0 con 0x7f24a01593c0 2023-03-03T16:57:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f249effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a00b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f249effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a00b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49378/0 (socket says 172.21.15.102:49378) 2023-03-03T16:57:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f249ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a01593c0 0x7f24a00caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f249effd700 1 -- 172.21.15.102:0/1790480501 learned_addr learned my addr 172.21.15.102:0/1790480501 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.535+0000 7f249f7fe700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24a00cd8e0 0x7f24a00b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249f7fe700 1 -- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a01593c0 msgr2=0x7f24a00caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249f7fe700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a01593c0 0x7f24a00caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249f7fe700 1 -- 172.21.15.102:0/1790480501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 msgr2=0x7f24a00b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249f7fe700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a00b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249f7fe700 1 -- 172.21.15.102:0/1790480501 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24900094a0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249ffff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a01593c0 0x7f24a00caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:57:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249effd700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a00b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:57:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f249f7fe700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24a00cd8e0 0x7f24a00b1ee0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f24a0159fb0 tx=0x7f2490004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:57:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.536+0000 7f2487fff700 1 -- 172.21.15.102:0/1790480501 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24900036a0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.537+0000 7f2487fff700 1 -- 172.21.15.102:0/1790480501 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249001e070 con 0x7f24a00cd8e0 2023-03-03T16:57:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.537+0000 7f2487fff700 1 -- 172.21.15.102:0/1790480501 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490003a40 con 0x7f24a00cd8e0 2023-03-03T16:57:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.537+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a00ccde0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.537+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a015d160 con 0x7f24a00cd8e0 2023-03-03T16:57:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.538+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a00cd0c0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.539+0000 7f2487fff700 1 -- 172.21.15.102:0/1790480501 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f249000f050 con 0x7f24a00cd8e0 2023-03-03T16:57:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.540+0000 7f2487fff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f248c0682f0 0x7f248c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.540+0000 7f2487fff700 1 -- 172.21.15.102:0/1790480501 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2490086c30 con 0x7f24a00cd8e0 2023-03-03T16:57:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.541+0000 7f249effd700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f248c0682f0 0x7f248c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.541+0000 7f249effd700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f248c0682f0 0x7f248c06a7a0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f24a006b290 tx=0x7f2494007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:57:19.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.545+0000 7f2487fff700 1 -- 172.21.15.102:0/1790480501 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24900532c0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.782+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a004e6e0 con 0x7f24a00cd8e0 2023-03-03T16:57:19.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.783+0000 7f2487fff700 1 -- 172.21.15.102:0/1790480501 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2490020090 con 0x7f24a00cd8e0 2023-03-03T16:57:19.782 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:57:19.783 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:57:19.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f248c0682f0 msgr2=0x7f248c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:19.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f248c0682f0 0x7f248c06a7a0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f24a006b290 tx=0x7f2494007ea0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24a00cd8e0 msgr2=0x7f24a00b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:19.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24a00cd8e0 0x7f24a00b1ee0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f24a0159fb0 tx=0x7f2490004750 comp rx=0 tx=0).stop 2023-03-03T16:57:19.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 shutdown_connections 2023-03-03T16:57:19.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f248c0682f0 0x7f248c06a7a0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f24a01593c0 0x7f24a00caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24a00cd8e0 0x7f24a00b1ee0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 --2- 172.21.15.102:0/1790480501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a0153290 0x7f24a00b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:19.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 >> 172.21.15.102:0/1790480501 conn(0x7f24a0065fb0 msgr2=0x7f24a015b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:57:19.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 shutdown_connections 2023-03-03T16:57:19.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:57:19.785+0000 7f24a5fff700 1 -- 172.21.15.102:0/1790480501 wait complete. 2023-03-03T16:57:19.787 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:57:19.813 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 45 secs before thrashing 2023-03-03T16:57:31.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:31.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:01.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:02.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:04.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:58:05.065 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.066+0000 7f8e30bff700 1 Processor -- start 2023-03-03T16:58:05.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.066+0000 7f8e30bff700 1 -- start start 2023-03-03T16:58:05.066 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.067+0000 7f8e30bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8e2c153290 0x7f8e2c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:05.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.067+0000 7f8e30bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c0cd8e0 0x7f8e2c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:05.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.067+0000 7f8e30bff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2c153660 con 0x7f8e2c1593c0 2023-03-03T16:58:05.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.067+0000 7f8e30bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2c06c360 con 0x7f8e2c153290 2023-03-03T16:58:05.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.067+0000 7f8e30bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c15cea0 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c0cd8e0 0x7f8e2c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:05.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2a59c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8e2c153290 0x7f8e2c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:05.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e29d9b700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8e2c1593c0 0x7f8e2c0cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:33174/0 (socket says 172.21.15.102:33174) 2023-03-03T16:58:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e29d9b700 1 -- 172.21.15.102:0/484405813 learned_addr learned my addr 172.21.15.102:0/484405813 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:58:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2959a700 1 -- 172.21.15.102:0/484405813 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611559647 0 0) 0x7f8e2c153660 con 0x7f8e2c1593c0 2023-03-03T16:58:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2ad9d700 1 -- 172.21.15.102:0/484405813 >> v2:172.21.15.172:3300/0 conn(0x7f8e2c153290 msgr2=0x7f8e2c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:58:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2ad9d700 1 --2- 172.21.15.102:0/484405813 >> v2:172.21.15.172:3300/0 conn(0x7f8e2c153290 0x7f8e2c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2ad9d700 1 -- 172.21.15.102:0/484405813 >> v1:172.21.15.172:6789/0 conn(0x7f8e2c1593c0 legacy=0x7f8e2c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2ad9d700 1 -- 172.21.15.102:0/484405813 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c06c7f0 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.068+0000 7f8e2ad9d700 1 --2- 172.21.15.102:0/484405813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c0cd8e0 0x7f8e2c0ce530 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009d90 tx=0x7f8e2400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc1beebecb41e06b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e2959a700 1 -- 172.21.15.102:0/484405813 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2400e070 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e2959a700 1 -- 172.21.15.102:0/484405813 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2400bb20 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e2959a700 1 -- 172.21.15.102:0/484405813 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24012430 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 -- 172.21.15.102:0/484405813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c0cd8e0 msgr2=0x7f8e2c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:05.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 --2- 172.21.15.102:0/484405813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c0cd8e0 0x7f8e2c0ce530 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009d90 tx=0x7f8e2400b330 comp rx=0 tx=0).stop 2023-03-03T16:58:05.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 -- 172.21.15.102:0/484405813 shutdown_connections 2023-03-03T16:58:05.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 --2- 172.21.15.102:0/484405813 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c0cd8e0 0x7f8e2c0ce530 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 --2- 172.21.15.102:0/484405813 >> v2:172.21.15.172:3300/0 conn(0x7f8e2c153290 0x7f8e2c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 -- 172.21.15.102:0/484405813 >> 172.21.15.102:0/484405813 conn(0x7f8e2c065fb0 msgr2=0x7f8e2c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:58:05.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 -- 172.21.15.102:0/484405813 shutdown_connections 2023-03-03T16:58:05.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.069+0000 7f8e30bff700 1 -- 172.21.15.102:0/484405813 wait complete. 2023-03-03T16:58:05.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.071+0000 7f8e30bff700 1 Processor -- start 2023-03-03T16:58:05.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.071+0000 7f8e30bff700 1 -- start start 2023-03-03T16:58:05.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.072+0000 7f8e30bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e2c0cd8e0 0x7f8e2c14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:05.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.072+0000 7f8e30bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c153290 0x7f8e2c0b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:05.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.072+0000 7f8e30bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e2c1593c0 0x7f8e2c0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:05.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.072+0000 7f8e30bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c14bf00 con 0x7f8e2c1593c0 2023-03-03T16:58:05.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.072+0000 7f8e30bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c14c1a0 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.072+0000 7f8e30bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c0b40d0 con 0x7f8e2c153290 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e2c1593c0 0x7f8e2c0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e29d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c153290 0x7f8e2c0b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e2c0cd8e0 0x7f8e2c14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e2c1593c0 0x7f8e2c0b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51044/0 (socket says 172.21.15.102:51044) 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2ad9d700 1 -- 172.21.15.102:0/4018758139 learned_addr learned my addr 172.21.15.102:0/4018758139 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2a59c700 1 -- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c153290 msgr2=0x7f8e2c0b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2a59c700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c153290 0x7f8e2c0b2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2a59c700 1 -- 172.21.15.102:0/4018758139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e2c1593c0 msgr2=0x7f8e2c0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:05.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2a59c700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e2c1593c0 0x7f8e2c0b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2a59c700 1 -- 172.21.15.102:0/4018758139 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24009a70 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.073+0000 7f8e2ad9d700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e2c1593c0 0x7f8e2c0b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:58:05.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.074+0000 7f8e2a59c700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e2c0cd8e0 0x7f8e2c14cb00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e870 tx=0x7f8e1c00eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:05.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.074+0000 7f8e1affd700 1 -- 172.21.15.102:0/4018758139 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c013680 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.074+0000 7f8e1affd700 1 -- 172.21.15.102:0/4018758139 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c0137e0 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.074+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2c0ccde0 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.074+0000 7f8e1affd700 1 -- 172.21.15.102:0/4018758139 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c013b90 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.074+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2c15d160 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.075+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2c0cd0c0 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.076+0000 7f8e1affd700 1 -- 172.21.15.102:0/4018758139 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e1c019030 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.078+0000 7f8e1affd700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e0c0682f0 0x7f8e0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:05.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.078+0000 7f8e1affd700 1 -- 172.21.15.102:0/4018758139 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8e1c085a50 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.078+0000 7f8e29d9b700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e0c0682f0 0x7f8e0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:05.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.079+0000 7f8e29d9b700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e0c0682f0 0x7f8e0c06a7a0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c1519c0 tx=0x7f8e20007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:05.081 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.082+0000 7f8e1affd700 1 -- 172.21.15.102:0/4018758139 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e1c052160 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.316+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e2c04e6e0 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.317+0000 7f8e1affd700 1 -- 172.21.15.102:0/4018758139 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8e1c055780 con 0x7f8e2c0cd8e0 2023-03-03T16:58:05.316 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:58:05.316 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:58:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.319+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e0c0682f0 msgr2=0x7f8e0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.319+0000 7f8e30bff700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e0c0682f0 0x7f8e0c06a7a0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c1519c0 tx=0x7f8e20007de0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.319+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e2c0cd8e0 msgr2=0x7f8e2c14cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:05.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.319+0000 7f8e30bff700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e2c0cd8e0 0x7f8e2c14cb00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e870 tx=0x7f8e1c00eb80 comp rx=0 tx=0).stop 2023-03-03T16:58:05.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.319+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 shutdown_connections 2023-03-03T16:58:05.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.319+0000 7f8e30bff700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e2c1593c0 0x7f8e2c0b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.319+0000 7f8e30bff700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e2c0cd8e0 0x7f8e2c14cb00 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.320+0000 7f8e30bff700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f8e0c0682f0 0x7f8e0c06a7a0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.320+0000 7f8e30bff700 1 --2- 172.21.15.102:0/4018758139 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8e2c153290 0x7f8e2c0b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:05.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.320+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 >> 172.21.15.102:0/4018758139 conn(0x7f8e2c065fb0 msgr2=0x7f8e2c1524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:58:05.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.320+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 shutdown_connections 2023-03-03T16:58:05.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:58:05.320+0000 7f8e30bff700 1 -- 172.21.15.102:0/4018758139 wait complete. 2023-03-03T16:58:05.321 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:58:05.346 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 72 secs before thrashing 2023-03-03T16:58:32.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:32.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:02.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:02.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:17.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:59:17.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.599+0000 7fbfc7fff700 1 Processor -- start 2023-03-03T16:59:17.599 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.599+0000 7fbfc7fff700 1 -- start start 2023-03-03T16:59:17.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbfc00cd8e0 0x7fbfc00cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:17.600 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc00cd390 0x7fbfc00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc7fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0153660 con 0x7fbfc0153290 2023-03-03T16:59:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc7fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc006c360 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc015cea0 con 0x7fbfc00cd390 2023-03-03T16:59:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc5d9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbfc0153290 0x7fbfc0158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42798/0 (socket says 172.21.15.102:42798) 2023-03-03T16:59:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc5d9b700 1 -- 172.21.15.102:0/2551787944 learned_addr learned my addr 172.21.15.102:0/2551787944 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T16:59:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc559a700 1 --2- 172.21.15.102:0/2551787944 >> v2:172.21.15.102:3300/0 conn(0x7fbfc00cd8e0 0x7fbfc00cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.601+0000 7fbfc659c700 1 --2- 172.21.15.102:0/2551787944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc00cd390 0x7fbfc00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc4d99700 1 -- 172.21.15.102:0/2551787944 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747411136 0 0) 0x7fbfc0153660 con 0x7fbfc0153290 2023-03-03T16:59:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc4d99700 1 -- 172.21.15.102:0/2551787944 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac0035b0 con 0x7fbfc0153290 2023-03-03T16:59:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc559a700 1 -- 172.21.15.102:0/2551787944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc00cd390 msgr2=0x7fbfc00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:59:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc559a700 1 --2- 172.21.15.102:0/2551787944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc00cd390 0x7fbfc00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc559a700 1 -- 172.21.15.102:0/2551787944 >> v1:172.21.15.102:6789/0 conn(0x7fbfc0153290 legacy=0x7fbfc0158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc559a700 1 -- 172.21.15.102:0/2551787944 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc006c7f0 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc559a700 1 --2- 172.21.15.102:0/2551787944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc00cde20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009840 tx=0x7fbfb0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a995e15ab7a72ecd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc4d99700 1 -- 172.21.15.102:0/2551787944 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0011830 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc4d99700 1 -- 172.21.15.102:0/2551787944 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0011990 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.602+0000 7fbfc4d99700 1 -- 172.21.15.102:0/2551787944 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0011d20 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 -- 172.21.15.102:0/2551787944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 msgr2=0x7fbfc00cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/2551787944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc00cde20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009840 tx=0x7fbfb0009b50 comp rx=0 tx=0).stop 2023-03-03T16:59:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 -- 172.21.15.102:0/2551787944 shutdown_connections 2023-03-03T16:59:17.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/2551787944 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc00cd390 0x7fbfc00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/2551787944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc00cde20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 -- 172.21.15.102:0/2551787944 >> 172.21.15.102:0/2551787944 conn(0x7fbfc0065fb0 msgr2=0x7fbfc00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:59:17.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 -- 172.21.15.102:0/2551787944 shutdown_connections 2023-03-03T16:59:17.605 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.603+0000 7fbfc7fff700 1 -- 172.21.15.102:0/2551787944 wait complete. 2023-03-03T16:59:17.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.605+0000 7fbfc7fff700 1 Processor -- start 2023-03-03T16:59:17.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.605+0000 7fbfc7fff700 1 -- start start 2023-03-03T16:59:17.606 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.606+0000 7fbfc7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfc00cd390 0x7fbfc0153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:17.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.606+0000 7fbfc7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc014f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:17.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.606+0000 7fbfc7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc0153290 0x7fbfc014fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:17.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.606+0000 7fbfc7fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc0155360 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.606+0000 7fbfc7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc0155b60 con 0x7fbfc00cd390 2023-03-03T16:59:17.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.606+0000 7fbfc7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc00b4b10 con 0x7fbfc0153290 2023-03-03T16:59:17.607 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc559a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc014f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:17.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc5d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfc00cd390 0x7fbfc0153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:17.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc5d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfc00cd390 0x7fbfc0153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:34516/0 (socket says 172.21.15.102:34516) 2023-03-03T16:59:17.608 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc5d9b700 1 -- 172.21.15.102:0/3386796717 learned_addr learned my addr 172.21.15.102:0/3386796717 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T16:59:17.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc559a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc014f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39734/0 (socket says 172.21.15.102:39734) 2023-03-03T16:59:17.609 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc659c700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc0153290 0x7fbfc014fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:17.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc559a700 1 -- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc0153290 msgr2=0x7fbfc014fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:17.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc559a700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc0153290 0x7fbfc014fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.612 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc559a700 1 -- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfc00cd390 msgr2=0x7fbfc0153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:17.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc559a700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfc00cd390 0x7fbfc0153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.607+0000 7fbfc559a700 1 -- 172.21.15.102:0/3386796717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb00094a0 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.613 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.608+0000 7fbfc559a700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc014f610 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0016000 tx=0x7fbfb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:17.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.608+0000 7fbfb67fc700 1 -- 172.21.15.102:0/3386796717 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0004590 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.608+0000 7fbfb67fc700 1 -- 172.21.15.102:0/3386796717 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb001e070 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.608+0000 7fbfb67fc700 1 -- 172.21.15.102:0/3386796717 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00038c0 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.608+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc00ccde0 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.614 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.608+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc015d160 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.609+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc00cd0c0 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.610+0000 7fbfb67fc700 1 -- 172.21.15.102:0/3386796717 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbfb000f050 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.612+0000 7fbfb67fc700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfa40682f0 0x7fbfa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:17.615 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.612+0000 7fbfb67fc700 1 -- 172.21.15.102:0/3386796717 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbfb0086d60 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.612+0000 7fbfc5d9b700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfa40682f0 0x7fbfa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:17.616 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.613+0000 7fbfc5d9b700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfa40682f0 0x7fbfa406a7a0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0011e0 tx=0x7fbfbc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.616+0000 7fbfb67fc700 1 -- 172.21.15.102:0/3386796717 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbfb00533f0 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.854+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfc00ca380 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.854+0000 7fbfb67fc700 1 -- 172.21.15.102:0/3386796717 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbfb0020090 con 0x7fbfc00cd8e0 2023-03-03T16:59:17.854 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T16:59:17.855 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T16:59:17.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfa40682f0 msgr2=0x7fbfa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:17.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfa40682f0 0x7fbfa406a7a0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0011e0 tx=0x7fbfbc009040 comp rx=0 tx=0).stop 2023-03-03T16:59:17.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 msgr2=0x7fbfc014f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:17.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc014f610 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0016000 tx=0x7fbfb0004750 comp rx=0 tx=0).stop 2023-03-03T16:59:17.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 shutdown_connections 2023-03-03T16:59:17.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbfa40682f0 0x7fbfa406a7a0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbfc0153290 0x7fbfc014fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfc00cd390 0x7fbfc0153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 --2- 172.21.15.102:0/3386796717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfc00cd8e0 0x7fbfc014f610 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:17.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 >> 172.21.15.102:0/3386796717 conn(0x7fbfc0065fb0 msgr2=0x7fbfc0158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:59:17.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 shutdown_connections 2023-03-03T16:59:17.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T16:59:17.857+0000 7fbfc7fff700 1 -- 172.21.15.102:0/3386796717 wait complete. 2023-03-03T16:59:17.859 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T16:59:17.884 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 90 secs before thrashing 2023-03-03T16:59:32.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:32.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:59.999 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T16:59:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:00:02.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:02.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:32.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:32.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:47.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:00:48.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.133+0000 7f2e411ff700 1 Processor -- start 2023-03-03T17:00:48.134 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.134+0000 7f2e411ff700 1 -- start start 2023-03-03T17:00:48.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e411ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e3c153290 0x7f2e3c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:48.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e411ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c1593c0 0x7f2e3c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:48.136 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e411ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c153660 con 0x7f2e3c0cd8e0 2023-03-03T17:00:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e411ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3c06c360 con 0x7f2e3c153290 2023-03-03T17:00:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e411ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c15cea0 con 0x7f2e3c1593c0 2023-03-03T17:00:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e3ad9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e3c153290 0x7f2e3c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e3ad9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e3c153290 0x7f2e3c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58062/0 (socket says 172.21.15.102:58062) 2023-03-03T17:00:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e3ad9d700 1 -- 172.21.15.102:0/3124880619 learned_addr learned my addr 172.21.15.102:0/3124880619 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:00:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.135+0000 7f2e3a59c700 1 --2- 172.21.15.102:0/3124880619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c1593c0 0x7f2e3c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e3ad9d700 1 -- 172.21.15.102:0/3124880619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c1593c0 msgr2=0x7f2e3c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:00:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e3ad9d700 1 --2- 172.21.15.102:0/3124880619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c1593c0 0x7f2e3c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e3ad9d700 1 -- 172.21.15.102:0/3124880619 >> v1:172.21.15.172:6789/0 conn(0x7f2e3c0cd8e0 legacy=0x7f2e3c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e3ad9d700 1 -- 172.21.15.102:0/3124880619 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c06c7f0 con 0x7f2e3c153290 2023-03-03T17:00:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e39d9b700 1 -- 172.21.15.102:0/3124880619 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521713815 0 0) 0x7f2e3c153660 con 0x7f2e3c0cd8e0 2023-03-03T17:00:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e3ad9d700 1 --2- 172.21.15.102:0/3124880619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c158e80 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0098f0 tx=0x7f2e2c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a882a81b94872a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e39d9b700 1 -- 172.21.15.102:0/3124880619 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c011830 con 0x7f2e3c153290 2023-03-03T17:00:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.136+0000 7f2e39d9b700 1 -- 172.21.15.102:0/3124880619 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2c011990 con 0x7f2e3c153290 2023-03-03T17:00:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e39d9b700 1 -- 172.21.15.102:0/3124880619 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c011d20 con 0x7f2e3c153290 2023-03-03T17:00:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 -- 172.21.15.102:0/3124880619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 msgr2=0x7f2e3c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3124880619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c158e80 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0098f0 tx=0x7f2e2c009c00 comp rx=0 tx=0).stop 2023-03-03T17:00:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 -- 172.21.15.102:0/3124880619 shutdown_connections 2023-03-03T17:00:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3124880619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c158e80 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3124880619 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c1593c0 0x7f2e3c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 -- 172.21.15.102:0/3124880619 >> 172.21.15.102:0/3124880619 conn(0x7f2e3c065fb0 msgr2=0x7f2e3c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:00:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 -- 172.21.15.102:0/3124880619 shutdown_connections 2023-03-03T17:00:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.137+0000 7f2e411ff700 1 -- 172.21.15.102:0/3124880619 wait complete. 2023-03-03T17:00:48.142 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.139+0000 7f2e411ff700 1 Processor -- start 2023-03-03T17:00:48.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.139+0000 7f2e411ff700 1 -- start start 2023-03-03T17:00:48.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.140+0000 7f2e411ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c0cd8e0 0x7f2e3c14f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:48.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.140+0000 7f2e411ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c0b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:48.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.140+0000 7f2e411ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e3c1593c0 0x7f2e3c0b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:48.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.140+0000 7f2e411ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c1e58f0 con 0x7f2e3c153290 2023-03-03T17:00:48.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.140+0000 7f2e411ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c14fef0 con 0x7f2e3c1593c0 2023-03-03T17:00:48.143 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.140+0000 7f2e411ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c1e5610 con 0x7f2e3c0cd8e0 2023-03-03T17:00:48.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c0b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:48.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c0cd8e0 0x7f2e3c14f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:48.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c0b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58076/0 (socket says 172.21.15.102:58076) 2023-03-03T17:00:48.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 -- 172.21.15.102:0/3497931588 learned_addr learned my addr 172.21.15.102:0/3497931588 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:00:48.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c0cd8e0 0x7f2e3c14f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:57888/0 (socket says 172.21.15.102:57888) 2023-03-03T17:00:48.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3b59e700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e3c1593c0 0x7f2e3c0b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:48.144 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 -- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c0cd8e0 msgr2=0x7f2e3c14f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:48.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c0cd8e0 0x7f2e3c14f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 -- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e3c1593c0 msgr2=0x7f2e3c0b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:48.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e3c1593c0 0x7f2e3c0b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3a59c700 1 -- 172.21.15.102:0/3497931588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c009550 con 0x7f2e3c153290 2023-03-03T17:00:48.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.141+0000 7f2e3ad9d700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c0cd8e0 0x7f2e3c14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:00:48.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.142+0000 7f2e3a59c700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c0b3860 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000e9b0 tx=0x7f2e3000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:48.145 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.142+0000 7f2e237fe700 1 -- 172.21.15.102:0/3497931588 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e30009710 con 0x7f2e3c153290 2023-03-03T17:00:48.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.142+0000 7f2e237fe700 1 -- 172.21.15.102:0/3497931588 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3000ee50 con 0x7f2e3c153290 2023-03-03T17:00:48.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.142+0000 7f2e237fe700 1 -- 172.21.15.102:0/3497931588 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000ca50 con 0x7f2e3c153290 2023-03-03T17:00:48.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.142+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c0ccde0 con 0x7f2e3c153290 2023-03-03T17:00:48.146 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.142+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c15d160 con 0x7f2e3c153290 2023-03-03T17:00:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.143+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c0cd0c0 con 0x7f2e3c153290 2023-03-03T17:00:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.144+0000 7f2e237fe700 1 -- 172.21.15.102:0/3497931588 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e30019030 con 0x7f2e3c153290 2023-03-03T17:00:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.146+0000 7f2e237fe700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e280682f0 0x7f2e2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.146+0000 7f2e237fe700 1 -- 172.21.15.102:0/3497931588 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2e30085c80 con 0x7f2e3c153290 2023-03-03T17:00:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.146+0000 7f2e3ad9d700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e280682f0 0x7f2e2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.147+0000 7f2e3ad9d700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e280682f0 0x7f2e2806a7a0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c014040 tx=0x7f2e2c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:48.150 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.149+0000 7f2e237fe700 1 -- 172.21.15.102:0/3497931588 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e30052390 con 0x7f2e3c153290 2023-03-03T17:00:48.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.382+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e3c0c6ef0 con 0x7f2e3c153290 2023-03-03T17:00:48.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.383+0000 7f2e237fe700 1 -- 172.21.15.102:0/3497931588 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2e300559b0 con 0x7f2e3c153290 2023-03-03T17:00:48.384 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:00:48.384 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:00:48.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e280682f0 msgr2=0x7f2e2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:48.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e280682f0 0x7f2e2806a7a0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c014040 tx=0x7f2e2c008030 comp rx=0 tx=0).stop 2023-03-03T17:00:48.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 msgr2=0x7f2e3c0b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:48.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c0b3860 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000e9b0 tx=0x7f2e3000ecc0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 shutdown_connections 2023-03-03T17:00:48.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2e280682f0 0x7f2e2806a7a0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2e3c0cd8e0 0x7f2e3c14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e3c1593c0 0x7f2e3c0b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 --2- 172.21.15.102:0/3497931588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e3c153290 0x7f2e3c0b3860 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:48.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.385+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 >> 172.21.15.102:0/3497931588 conn(0x7f2e3c065fb0 msgr2=0x7f2e3c0cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:00:48.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.386+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 shutdown_connections 2023-03-03T17:00:48.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:00:48.386+0000 7f2e411ff700 1 -- 172.21.15.102:0/3497931588 wait complete. 2023-03-03T17:00:48.388 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:00:48.413 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 74 secs before thrashing 2023-03-03T17:01:03.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:03.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:33.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:33.393 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:02.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:02:02.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.668+0000 7f4ec09ff700 1 Processor -- start 2023-03-03T17:02:02.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.669+0000 7f4ec09ff700 1 -- start start 2023-03-03T17:02:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4ec09ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ebc1597f0 0x7f4ebc0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:02:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4ec09ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ebc0cf9a0 0x7f4ebc163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:02:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4ec09ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0cb9f0 con 0x7f4ebc0ced80 2023-03-03T17:02:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4ec09ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ebc0ca750 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4ec09ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ebc15a230 con 0x7f4ebc1597f0 2023-03-03T17:02:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4eb9d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ebc1597f0 0x7f4ebc0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:02:02.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4eba59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4ebc0ced80 0x7f4ebc0cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41912/0 (socket says 172.21.15.102:41912) 2023-03-03T17:02:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4eba59c700 1 -- 172.21.15.102:0/1995358490 learned_addr learned my addr 172.21.15.102:0/1995358490 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:02:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.671+0000 7f4ebad9d700 1 --2- 172.21.15.102:0/1995358490 >> v2:172.21.15.102:3300/0 conn(0x7f4ebc0cf9a0 0x7f4ebc163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:02:02.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb9d9b700 1 -- 172.21.15.102:0/1995358490 >> v2:172.21.15.102:3300/0 conn(0x7f4ebc0cf9a0 msgr2=0x7f4ebc163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:02:02.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb9d9b700 1 --2- 172.21.15.102:0/1995358490 >> v2:172.21.15.102:3300/0 conn(0x7f4ebc0cf9a0 0x7f4ebc163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb9d9b700 1 -- 172.21.15.102:0/1995358490 >> v1:172.21.15.172:6789/0 conn(0x7f4ebc0ced80 legacy=0x7f4ebc0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:02:02.673 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb9d9b700 1 -- 172.21.15.102:0/1995358490 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc159c80 con 0x7f4ebc1597f0 2023-03-03T17:02:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb959a700 1 -- 172.21.15.102:0/1995358490 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878653560 0 0) 0x7f4ebc0cb9f0 con 0x7f4ebc0ced80 2023-03-03T17:02:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb9d9b700 1 --2- 172.21.15.102:0/1995358490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 0x7f4ebc0cf150 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0ced50 tx=0x7f4eb0009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8baca0a3b61c7493 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:02:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb959a700 1 -- 172.21.15.102:0/1995358490 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0011830 con 0x7f4ebc1597f0 2023-03-03T17:02:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb959a700 1 -- 172.21.15.102:0/1995358490 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0011990 con 0x7f4ebc1597f0 2023-03-03T17:02:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.672+0000 7f4eb959a700 1 -- 172.21.15.102:0/1995358490 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0011d20 con 0x7f4ebc1597f0 2023-03-03T17:02:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 -- 172.21.15.102:0/1995358490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 msgr2=0x7f4ebc0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:02:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/1995358490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 0x7f4ebc0cf150 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0ced50 tx=0x7f4eb0009b20 comp rx=0 tx=0).stop 2023-03-03T17:02:02.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 -- 172.21.15.102:0/1995358490 shutdown_connections 2023-03-03T17:02:02.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/1995358490 >> v2:172.21.15.102:3300/0 conn(0x7f4ebc0cf9a0 0x7f4ebc163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/1995358490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 0x7f4ebc0cf150 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 -- 172.21.15.102:0/1995358490 >> 172.21.15.102:0/1995358490 conn(0x7f4ebc065fb0 msgr2=0x7f4ebc0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:02:02.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 -- 172.21.15.102:0/1995358490 shutdown_connections 2023-03-03T17:02:02.675 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.673+0000 7f4ec09ff700 1 -- 172.21.15.102:0/1995358490 wait complete. 2023-03-03T17:02:02.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.675+0000 7f4ec09ff700 1 Processor -- start 2023-03-03T17:02:02.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.675+0000 7f4ec09ff700 1 -- start start 2023-03-03T17:02:02.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.676+0000 7f4ec09ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ebc0ced80 0x7f4ebc152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:02:02.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.676+0000 7f4ec09ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ebc0cf9a0 0x7f4ebc1e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:02:02.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.676+0000 7f4ec09ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 0x7f4ebc14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:02:02.678 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.676+0000 7f4ec09ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc14e540 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.676+0000 7f4ec09ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc14ef80 con 0x7f4ebc1597f0 2023-03-03T17:02:02.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.676+0000 7f4ec09ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc1517d0 con 0x7f4ebc0ced80 2023-03-03T17:02:02.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.676+0000 7f4eb9d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ebc0cf9a0 0x7f4ebc1e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:02:02.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4ebad9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 0x7f4ebc14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:02:02.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ebc0cf9a0 0x7f4ebc1e5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36798/0 (socket says 172.21.15.102:36798) 2023-03-03T17:02:02.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 -- 172.21.15.102:0/472090659 learned_addr learned my addr 172.21.15.102:0/472090659 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:02:02.679 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eba59c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ebc0ced80 0x7f4ebc152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:02:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 -- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ebc0ced80 msgr2=0x7f4ebc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:02:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ebc0ced80 0x7f4ebc152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 -- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 msgr2=0x7f4ebc14f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:02:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 0x7f4ebc14f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 -- 172.21.15.102:0/472090659 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb00094a0 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.677+0000 7f4eb9d9b700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ebc0cf9a0 0x7f4ebc1e5a50 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0014040 tx=0x7f4eb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:02:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.678+0000 7f4eaaffd700 1 -- 172.21.15.102:0/472090659 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00036a0 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.678+0000 7f4eaaffd700 1 -- 172.21.15.102:0/472090659 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb001e070 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.678+0000 7f4eaaffd700 1 -- 172.21.15.102:0/472090659 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0003a20 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.678+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc0cb0c0 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.678+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc0cade0 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.679+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc0cb3a0 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.680+0000 7f4eaaffd700 1 -- 172.21.15.102:0/472090659 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4eb000f050 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.682+0000 7f4eaaffd700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:02:02.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.683+0000 7f4eaaffd700 1 -- 172.21.15.102:0/472090659 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4eb0086c70 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.682 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.683+0000 7f4eba59c700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:02:02.683 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.684+0000 7f4eba59c700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f4eac000f60 tx=0x7f4eac007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:02:02.686 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.687+0000 7f4eaaffd700 1 -- 172.21.15.102:0/472090659 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4eb0053380 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.920+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ebc14f130 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.921+0000 7f4eaaffd700 1 -- 172.21.15.102:0/472090659 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4ebc14f130 con 0x7f4ebc0cf9a0 2023-03-03T17:02:02.921 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:02:02.922 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:02:02.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4e9c0682f0 msgr2=0x7f4e9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:02:02.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f4eac000f60 tx=0x7f4eac007e70 comp rx=0 tx=0).stop 2023-03-03T17:02:02.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ebc0cf9a0 msgr2=0x7f4ebc1e5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:02:02.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ebc0cf9a0 0x7f4ebc1e5a50 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0014040 tx=0x7f4eb0004750 comp rx=0 tx=0).stop 2023-03-03T17:02:02.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 shutdown_connections 2023-03-03T17:02:02.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ebc0ced80 0x7f4ebc152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ebc1597f0 0x7f4ebc14f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 --2- 172.21.15.102:0/472090659 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ebc0cf9a0 0x7f4ebc1e5a50 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:02:02.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 >> 172.21.15.102:0/472090659 conn(0x7f4ebc065fb0 msgr2=0x7f4ebc0cbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:02:02.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 shutdown_connections 2023-03-03T17:02:02.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:02:02.923+0000 7f4ec09ff700 1 -- 172.21.15.102:0/472090659 wait complete. 2023-03-03T17:02:02.926 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:02:02.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-03T17:02:03.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:03.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:33.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:33.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:03.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:03.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:13.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:03:14.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.198+0000 7f4ec75ff700 1 Processor -- start 2023-03-03T17:03:14.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.199+0000 7f4ec75ff700 1 -- start start 2023-03-03T17:03:14.199 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec75ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ec0153290 0x7f4ec0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:14.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec75ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ec00cd8e0 0x7f4ec00ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:14.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec75ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec0153660 con 0x7f4ec01593c0 2023-03-03T17:03:14.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec75ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec006c360 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec75ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec015cea0 con 0x7f4ec0153290 2023-03-03T17:03:14.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec4b9a700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4ec01593c0 0x7f4ec00cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:60586/0 (socket says 172.21.15.102:60586) 2023-03-03T17:03:14.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec5b9c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ec00cd8e0 0x7f4ec00ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:14.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec4b9a700 1 -- 172.21.15.102:0/794839778 learned_addr learned my addr 172.21.15.102:0/794839778 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:03:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.200+0000 7f4ec539b700 1 --2- 172.21.15.102:0/794839778 >> v2:172.21.15.172:3300/0 conn(0x7f4ec0153290 0x7f4ec0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.201+0000 7f4eb7fff700 1 -- 172.21.15.102:0/794839778 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004335993 0 0) 0x7f4ec0153660 con 0x7f4ec01593c0 2023-03-03T17:03:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.201+0000 7f4eb7fff700 1 -- 172.21.15.102:0/794839778 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eac0035b0 con 0x7f4ec01593c0 2023-03-03T17:03:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.201+0000 7f4ec539b700 1 -- 172.21.15.102:0/794839778 >> v2:172.21.15.102:3300/0 conn(0x7f4ec00cd8e0 msgr2=0x7f4ec00ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.201+0000 7f4ec539b700 1 --2- 172.21.15.102:0/794839778 >> v2:172.21.15.102:3300/0 conn(0x7f4ec00cd8e0 0x7f4ec00ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.201+0000 7f4ec539b700 1 -- 172.21.15.102:0/794839778 >> v1:172.21.15.172:6789/0 conn(0x7f4ec01593c0 legacy=0x7f4ec00cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.201+0000 7f4ec539b700 1 -- 172.21.15.102:0/794839778 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec006c7f0 con 0x7f4ec0153290 2023-03-03T17:03:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.201+0000 7f4ec539b700 1 --2- 172.21.15.102:0/794839778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec0153290 0x7f4ec0158e80 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4ec008cc70 tx=0x7f4ebc009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74e02fad813550c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4eb7fff700 1 -- 172.21.15.102:0/794839778 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc011830 con 0x7f4ec0153290 2023-03-03T17:03:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4eb7fff700 1 -- 172.21.15.102:0/794839778 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc011990 con 0x7f4ec0153290 2023-03-03T17:03:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4eb7fff700 1 -- 172.21.15.102:0/794839778 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc011d20 con 0x7f4ec0153290 2023-03-03T17:03:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 -- 172.21.15.102:0/794839778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec0153290 msgr2=0x7f4ec0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/794839778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec0153290 0x7f4ec0158e80 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4ec008cc70 tx=0x7f4ebc009b20 comp rx=0 tx=0).stop 2023-03-03T17:03:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 -- 172.21.15.102:0/794839778 shutdown_connections 2023-03-03T17:03:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/794839778 >> v2:172.21.15.102:3300/0 conn(0x7f4ec00cd8e0 0x7f4ec00ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/794839778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec0153290 0x7f4ec0158e80 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 -- 172.21.15.102:0/794839778 >> 172.21.15.102:0/794839778 conn(0x7f4ec0065fb0 msgr2=0x7f4ec00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:14.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 -- 172.21.15.102:0/794839778 shutdown_connections 2023-03-03T17:03:14.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.202+0000 7f4ec75ff700 1 -- 172.21.15.102:0/794839778 wait complete. 2023-03-03T17:03:14.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.205+0000 7f4ec75ff700 1 Processor -- start 2023-03-03T17:03:14.205 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.205+0000 7f4ec75ff700 1 -- start start 2023-03-03T17:03:14.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec75ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec00cd8e0 0x7f4ec00b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:14.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec75ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec0153290 0x7f4ec00b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:14.206 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec75ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec01593c0 0x7f4ec01e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec75ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00b20f0 con 0x7f4ec0153290 2023-03-03T17:03:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec75ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00b3d80 con 0x7f4ec01593c0 2023-03-03T17:03:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec75ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00c7490 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec4b9a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec0153290 0x7f4ec00b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec539b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec00cd8e0 0x7f4ec00b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec539b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec00cd8e0 0x7f4ec00b3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54942/0 (socket says 172.21.15.102:54942) 2023-03-03T17:03:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec539b700 1 -- 172.21.15.102:0/4231926138 learned_addr learned my addr 172.21.15.102:0/4231926138 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:03:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.206+0000 7f4ec5b9c700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec01593c0 0x7f4ec01e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4ec539b700 1 -- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec01593c0 msgr2=0x7f4ec01e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4ec539b700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec01593c0 0x7f4ec01e5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4ec539b700 1 -- 172.21.15.102:0/4231926138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec0153290 msgr2=0x7f4ec00b2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4ec539b700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec0153290 0x7f4ec00b2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4ec539b700 1 -- 172.21.15.102:0/4231926138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0094a0 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4ec539b700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec00cd8e0 0x7f4ec00b3840 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009b60 tx=0x7f4ebc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:14.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4eb5ffb700 1 -- 172.21.15.102:0/4231926138 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc004590 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4eb5ffb700 1 -- 172.21.15.102:0/4231926138 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc01f070 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.209 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4eb5ffb700 1 -- 172.21.15.102:0/4231926138 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0038e0 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.207+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec00ccde0 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.208+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec015d160 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.208+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec00cd0c0 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.210+0000 7f4eb5ffb700 1 -- 172.21.15.102:0/4231926138 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ebc048070 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.210 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.211+0000 7f4eb5ffb700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea4068320 0x7f4ea406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:14.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.211+0000 7f4eb5ffb700 1 -- 172.21.15.102:0/4231926138 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f4ebc021070 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.211+0000 7f4ec4b9a700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea4068320 0x7f4ea406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:14.211 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.212+0000 7f4ec4b9a700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea4068320 0x7f4ea406a7d0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00c6740 tx=0x7f4eb0007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:14.213 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.214+0000 7f4eb5ffb700 1 -- 172.21.15.102:0/4231926138 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ebc054740 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.449+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ec004e6e0 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.450+0000 7f4eb5ffb700 1 -- 172.21.15.102:0/4231926138 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f4ebc02b090 con 0x7f4ec00cd8e0 2023-03-03T17:03:14.450 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:03:14.450 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:03:14.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea4068320 msgr2=0x7f4ea406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:14.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea4068320 0x7f4ea406a7d0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00c6740 tx=0x7f4eb0007dc0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec00cd8e0 msgr2=0x7f4ec00b3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:14.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec00cd8e0 0x7f4ec00b3840 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009b60 tx=0x7f4ebc004750 comp rx=0 tx=0).stop 2023-03-03T17:03:14.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 shutdown_connections 2023-03-03T17:03:14.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4ea4068320 0x7f4ea406a7d0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4ec00cd8e0 0x7f4ec00b3840 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ec01593c0 0x7f4ec01e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 --2- 172.21.15.102:0/4231926138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec0153290 0x7f4ec00b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:14.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 >> 172.21.15.102:0/4231926138 conn(0x7f4ec0065fb0 msgr2=0x7f4ec00cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:14.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 shutdown_connections 2023-03-03T17:03:14.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:14.453+0000 7f4ec75ff700 1 -- 172.21.15.102:0/4231926138 wait complete. 2023-03-03T17:03:14.455 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:03:14.481 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-03-03T17:03:17.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:03:17.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.731+0000 7f2048f9d700 1 Processor -- start 2023-03-03T17:03:17.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.731+0000 7f2048f9d700 1 -- start start 2023-03-03T17:03:17.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f2048f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440cb2d0 0x7f204406e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:17.733 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f2048f9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440c5530 con 0x7f20440c6dc0 2023-03-03T17:03:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f2048f9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440c60d0 con 0x7f204414f2f0 2023-03-03T17:03:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f2048f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2044070610 con 0x7f20440cb2d0 2023-03-03T17:03:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f20437fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f204414f2f0 0x7f2044158fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34832/0 (socket says 172.21.15.102:34832) 2023-03-03T17:03:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f20437fe700 1 -- 172.21.15.102:0/3215773799 learned_addr learned my addr 172.21.15.102:0/3215773799 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:03:17.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f2043fff700 1 --2- 172.21.15.102:0/3215773799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440cb2d0 0x7f204406e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:17.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f20427fc700 1 -- 172.21.15.102:0/3215773799 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489203373 0 0) 0x7f20440c60d0 con 0x7f204414f2f0 2023-03-03T17:03:17.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.734+0000 7f2043fff700 1 -- 172.21.15.102:0/3215773799 >> v1:172.21.15.172:6789/0 conn(0x7f204414f2f0 legacy=0x7f2044158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.735+0000 7f2043fff700 1 -- 172.21.15.102:0/3215773799 >> v1:172.21.15.102:6789/0 conn(0x7f20440c6dc0 legacy=0x7f20440c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.735+0000 7f2043fff700 1 -- 172.21.15.102:0/3215773799 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440708f0 con 0x7f20440cb2d0 2023-03-03T17:03:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.735+0000 7f2043fff700 1 --2- 172.21.15.102:0/3215773799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440cb2d0 0x7f204406e430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f203c00aa10 tx=0x7f203c00ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55f0c42f0c28b30 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.735+0000 7f20427fc700 1 -- 172.21.15.102:0/3215773799 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c00cb70 con 0x7f20440cb2d0 2023-03-03T17:03:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.735+0000 7f20427fc700 1 -- 172.21.15.102:0/3215773799 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c00ccd0 con 0x7f20440cb2d0 2023-03-03T17:03:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.735+0000 7f20427fc700 1 -- 172.21.15.102:0/3215773799 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c018620 con 0x7f20440cb2d0 2023-03-03T17:03:17.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.736+0000 7f2048f9d700 1 -- 172.21.15.102:0/3215773799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440cb2d0 msgr2=0x7f204406e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.736+0000 7f2048f9d700 1 --2- 172.21.15.102:0/3215773799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440cb2d0 0x7f204406e430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f203c00aa10 tx=0x7f203c00ad20 comp rx=0 tx=0).stop 2023-03-03T17:03:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.736+0000 7f2048f9d700 1 -- 172.21.15.102:0/3215773799 shutdown_connections 2023-03-03T17:03:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.736+0000 7f2048f9d700 1 --2- 172.21.15.102:0/3215773799 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440cb2d0 0x7f204406e430 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.736+0000 7f2048f9d700 1 -- 172.21.15.102:0/3215773799 >> 172.21.15.102:0/3215773799 conn(0x7f20440ca7c0 msgr2=0x7f20440cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.736+0000 7f2048f9d700 1 -- 172.21.15.102:0/3215773799 shutdown_connections 2023-03-03T17:03:17.737 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.736+0000 7f2048f9d700 1 -- 172.21.15.102:0/3215773799 wait complete. 2023-03-03T17:03:17.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.739+0000 7f2048f9d700 1 Processor -- start 2023-03-03T17:03:17.738 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.739+0000 7f2048f9d700 1 -- start start 2023-03-03T17:03:17.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2048f9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440c6dc0 0x7f2044142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:17.739 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2048f9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20440cb2d0 0x7f204406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2048f9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f204414f2f0 0x7f2044141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2048f9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20441ee940 con 0x7f20440cb2d0 2023-03-03T17:03:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2048f9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20441ef760 con 0x7f204414f2f0 2023-03-03T17:03:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2048f9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20441eebe0 con 0x7f20440c6dc0 2023-03-03T17:03:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f20437fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440c6dc0 0x7f2044142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2042ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20440cb2d0 0x7f204406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f2043fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f204414f2f0 0x7f2044141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:17.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f20437fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440c6dc0 0x7f2044142280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:41242/0 (socket says 172.21.15.102:41242) 2023-03-03T17:03:17.741 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.740+0000 7f20437fe700 1 -- 172.21.15.102:0/403904048 learned_addr learned my addr 172.21.15.102:0/403904048 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:03:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f20437fe700 1 -- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f204414f2f0 msgr2=0x7f2044141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f20437fe700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f204414f2f0 0x7f2044141060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f20437fe700 1 -- 172.21.15.102:0/403904048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20440cb2d0 msgr2=0x7f204406e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f20437fe700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20440cb2d0 0x7f204406e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f20437fe700 1 -- 172.21.15.102:0/403904048 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c00a640 con 0x7f20440c6dc0 2023-03-03T17:03:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f20437fe700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440c6dc0 0x7f2044142280 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f203400cb90 tx=0x7f203400cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f202bfff700 1 -- 172.21.15.102:0/403904048 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203400aaa0 con 0x7f20440c6dc0 2023-03-03T17:03:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f202bfff700 1 -- 172.21.15.102:0/403904048 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203400ac00 con 0x7f20440c6dc0 2023-03-03T17:03:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20440c51c0 con 0x7f20440c6dc0 2023-03-03T17:03:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f202bfff700 1 -- 172.21.15.102:0/403904048 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20340070d0 con 0x7f20440c6dc0 2023-03-03T17:03:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.741+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2044070bd0 con 0x7f20440c6dc0 2023-03-03T17:03:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.742+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204414f6c0 con 0x7f20440c6dc0 2023-03-03T17:03:17.744 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.745+0000 7f202bfff700 1 -- 172.21.15.102:0/403904048 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2034010070 con 0x7f20440c6dc0 2023-03-03T17:03:17.745 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.746+0000 7f202bfff700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20300682f0 0x7f203006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:17.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.746+0000 7f202bfff700 1 -- 172.21.15.102:0/403904048 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2034085d60 con 0x7f20440c6dc0 2023-03-03T17:03:17.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.747+0000 7f2042ffd700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20300682f0 0x7f203006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:17.746 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.747+0000 7f2042ffd700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20300682f0 0x7f203006a7a0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f20441ec460 tx=0x7f2038009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:17.749 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.750+0000 7f202bfff700 1 -- 172.21.15.102:0/403904048 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2034052570 con 0x7f20440c6dc0 2023-03-03T17:03:17.985 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.986+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f204404e6e0 con 0x7f20440c6dc0 2023-03-03T17:03:17.986 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.987+0000 7f202bfff700 1 -- 172.21.15.102:0/403904048 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2034018030 con 0x7f20440c6dc0 2023-03-03T17:03:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:03:17.987 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:03:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20300682f0 msgr2=0x7f203006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20300682f0 0x7f203006a7a0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f20441ec460 tx=0x7f2038009040 comp rx=0 tx=0).stop 2023-03-03T17:03:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440c6dc0 msgr2=0x7f2044142280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440c6dc0 0x7f2044142280 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f203400cb90 tx=0x7f203400cea0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 shutdown_connections 2023-03-03T17:03:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f20300682f0 0x7f203006a7a0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20440c6dc0 0x7f2044142280 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f204414f2f0 0x7f2044141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 --2- 172.21.15.102:0/403904048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20440cb2d0 0x7f204406e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.989+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 >> 172.21.15.102:0/403904048 conn(0x7f20440ca7c0 msgr2=0x7f204414d380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.990+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 shutdown_connections 2023-03-03T17:03:17.991 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:03:17.990+0000 7f2048f9d700 1 -- 172.21.15.102:0/403904048 wait complete. 2023-03-03T17:03:17.991 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:03:18.016 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-03-03T17:03:34.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:34.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:04.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:04.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:34.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:34.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:04.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:04.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:13.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:05:13.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.268+0000 7f31067ff700 1 Processor -- start 2023-03-03T17:05:13.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.268+0000 7f31067ff700 1 -- start start 2023-03-03T17:05:13.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f31067ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f31000cd8e0 0x7f31000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:13.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f31067ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:13.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f31067ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3100153660 con 0x7f3100153290 2023-03-03T17:05:13.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f31067ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310006c360 con 0x7f31000cd8e0 2023-03-03T17:05:13.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f31067ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310015cea0 con 0x7f31000cd390 2023-03-03T17:05:13.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f30fffff700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3100153290 0x7f3100158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35606/0 (socket says 172.21.15.102:35606) 2023-03-03T17:05:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f3104d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f30ff7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f31000cd8e0 0x7f31000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.270+0000 7f30fffff700 1 -- 172.21.15.102:0/533401946 learned_addr learned my addr 172.21.15.102:0/533401946 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:05:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30ff7fe700 1 -- 172.21.15.102:0/533401946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 msgr2=0x7f31000ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:05:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30ff7fe700 1 --2- 172.21.15.102:0/533401946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30ff7fe700 1 -- 172.21.15.102:0/533401946 >> v1:172.21.15.102:6789/0 conn(0x7f3100153290 legacy=0x7f3100158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30ff7fe700 1 -- 172.21.15.102:0/533401946 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310006c7f0 con 0x7f31000cd8e0 2023-03-03T17:05:13.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30ff7fe700 1 --2- 172.21.15.102:0/533401946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31000cd8e0 0x7f31000cde20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009840 tx=0x7f30f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ad4322ab726736 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:13.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30feffd700 1 -- 172.21.15.102:0/533401946 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4011830 con 0x7f31000cd8e0 2023-03-03T17:05:13.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30feffd700 1 -- 172.21.15.102:0/533401946 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f4011990 con 0x7f31000cd8e0 2023-03-03T17:05:13.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.271+0000 7f30feffd700 1 -- 172.21.15.102:0/533401946 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4011d20 con 0x7f31000cd8e0 2023-03-03T17:05:13.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 -- 172.21.15.102:0/533401946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31000cd8e0 msgr2=0x7f31000cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 --2- 172.21.15.102:0/533401946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31000cd8e0 0x7f31000cde20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009840 tx=0x7f30f4009b50 comp rx=0 tx=0).stop 2023-03-03T17:05:13.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 -- 172.21.15.102:0/533401946 shutdown_connections 2023-03-03T17:05:13.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 --2- 172.21.15.102:0/533401946 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 --2- 172.21.15.102:0/533401946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31000cd8e0 0x7f31000cde20 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 -- 172.21.15.102:0/533401946 >> 172.21.15.102:0/533401946 conn(0x7f3100065fb0 msgr2=0x7f31000663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:13.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 -- 172.21.15.102:0/533401946 shutdown_connections 2023-03-03T17:05:13.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.272+0000 7f31067ff700 1 -- 172.21.15.102:0/533401946 wait complete. 2023-03-03T17:05:13.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.274+0000 7f31067ff700 1 Processor -- start 2023-03-03T17:05:13.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.274+0000 7f31067ff700 1 -- start start 2023-03-03T17:05:13.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.275+0000 7f31067ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:13.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.275+0000 7f31067ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31000cd8e0 0x7f31000c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:13.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.275+0000 7f31067ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3100153290 0x7f31001f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.275+0000 7f31067ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31000b4410 con 0x7f3100153290 2023-03-03T17:05:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.275+0000 7f31067ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31000b2ac0 con 0x7f31000cd8e0 2023-03-03T17:05:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.275+0000 7f31067ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31000b36c0 con 0x7f31000cd390 2023-03-03T17:05:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f3104d9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3100153290 0x7f31001f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30ff7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31000cd8e0 0x7f31000c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:43568/0 (socket says 172.21.15.102:43568) 2023-03-03T17:05:13.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 -- 172.21.15.102:0/1478141352 learned_addr learned my addr 172.21.15.102:0/1478141352 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:05:13.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 -- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31000cd8e0 msgr2=0x7f31000c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31000cd8e0 0x7f31000c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 -- 172.21.15.102:0/1478141352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3100153290 msgr2=0x7f31001f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3100153290 0x7f31001f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 -- 172.21.15.102:0/1478141352 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f40094a0 con 0x7f31000cd390 2023-03-03T17:05:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.276+0000 7f30fffff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000b4830 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c550 tx=0x7f30f000c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.277+0000 7f30fcff9700 1 -- 172.21.15.102:0/1478141352 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0005d70 con 0x7f31000cd390 2023-03-03T17:05:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.277+0000 7f30fcff9700 1 -- 172.21.15.102:0/1478141352 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f00070d0 con 0x7f31000cd390 2023-03-03T17:05:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.277+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31000ccde0 con 0x7f31000cd390 2023-03-03T17:05:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.277+0000 7f30fcff9700 1 -- 172.21.15.102:0/1478141352 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0007440 con 0x7f31000cd390 2023-03-03T17:05:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.277+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310015d160 con 0x7f31000cd390 2023-03-03T17:05:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.278+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31000cd0c0 con 0x7f31000cd390 2023-03-03T17:05:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.279+0000 7f30fcff9700 1 -- 172.21.15.102:0/1478141352 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30f0019030 con 0x7f31000cd390 2023-03-03T17:05:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.280+0000 7f30fcff9700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ec0682f0 0x7f30ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.280+0000 7f30fcff9700 1 -- 172.21.15.102:0/1478141352 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f30f0085300 con 0x7f31000cd390 2023-03-03T17:05:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.281+0000 7f30ff7fe700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ec0682f0 0x7f30ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:13.282 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.281+0000 7f30ff7fe700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ec0682f0 0x7f30ec06a7a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f30f4000c00 tx=0x7f30f401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:13.284 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.284+0000 7f30fcff9700 1 -- 172.21.15.102:0/1478141352 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30f0051b40 con 0x7f31000cd390 2023-03-03T17:05:13.520 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.520+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31000b4680 con 0x7f31000cd390 2023-03-03T17:05:13.521 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.521+0000 7f30fcff9700 1 -- 172.21.15.102:0/1478141352 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f30f0055160 con 0x7f31000cd390 2023-03-03T17:05:13.521 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:05:13.522 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:05:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ec0682f0 msgr2=0x7f30ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ec0682f0 0x7f30ec06a7a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f30f4000c00 tx=0x7f30f401c040 comp rx=0 tx=0).stop 2023-03-03T17:05:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 msgr2=0x7f31000b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.524 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000b4830 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c550 tx=0x7f30f000c860 comp rx=0 tx=0).stop 2023-03-03T17:05:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 shutdown_connections 2023-03-03T17:05:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f30ec0682f0 0x7f30ec06a7a0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f31000cd390 0x7f31000b4830 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3100153290 0x7f31001f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 --2- 172.21.15.102:0/1478141352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31000cd8e0 0x7f31000c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.525 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.523+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 >> 172.21.15.102:0/1478141352 conn(0x7f3100065fb0 msgr2=0x7f31000cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:13.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.524+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 shutdown_connections 2023-03-03T17:05:13.526 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:05:13.524+0000 7f31067ff700 1 -- 172.21.15.102:0/1478141352 wait complete. 2023-03-03T17:05:13.526 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:05:13.551 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-03-03T17:05:34.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:34.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:04.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:04.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:35.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:35.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:38.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:06:38.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.804+0000 7f870db9d700 1 Processor -- start 2023-03-03T17:06:38.804 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.804+0000 7f870db9d700 1 -- start start 2023-03-03T17:06:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f870db9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f87080b3970 0x7f87080c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f870db9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f87080c41b0 0x7f87080cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f870db9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87080c3d20 con 0x7f87080ced00 2023-03-03T17:06:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f870db9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87080c4930 con 0x7f87080b3970 2023-03-03T17:06:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f870db9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87080cf610 con 0x7f87080c41b0 2023-03-03T17:06:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f8707fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f87080b3970 0x7f87080c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f870d39c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f87080c41b0 0x7f87080cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f8707fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f87080b3970 0x7f87080c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48796/0 (socket says 172.21.15.102:48796) 2023-03-03T17:06:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.806+0000 7f8707fff700 1 -- 172.21.15.102:0/537505264 learned_addr learned my addr 172.21.15.102:0/537505264 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:06:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f8707fff700 1 -- 172.21.15.102:0/537505264 >> v2:172.21.15.172:3300/0 conn(0x7f87080c41b0 msgr2=0x7f87080cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:06:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f8707fff700 1 --2- 172.21.15.102:0/537505264 >> v2:172.21.15.172:3300/0 conn(0x7f87080c41b0 0x7f87080cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f8707fff700 1 -- 172.21.15.102:0/537505264 >> v1:172.21.15.102:6789/0 conn(0x7f87080ced00 legacy=0x7f8708158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f8707fff700 1 -- 172.21.15.102:0/537505264 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870815cba0 con 0x7f87080b3970 2023-03-03T17:06:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f8707fff700 1 --2- 172.21.15.102:0/537505264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080b3970 0x7f87080c5a90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f870808ce30 tx=0x7f86f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df637f8a7d0f95d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:06:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f87077fe700 1 -- 172.21.15.102:0/537505264 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f4011830 con 0x7f87080b3970 2023-03-03T17:06:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f87077fe700 1 -- 172.21.15.102:0/537505264 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f4011990 con 0x7f87080b3970 2023-03-03T17:06:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.807+0000 7f87077fe700 1 -- 172.21.15.102:0/537505264 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f4011d20 con 0x7f87080b3970 2023-03-03T17:06:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 -- 172.21.15.102:0/537505264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080b3970 msgr2=0x7f87080c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 --2- 172.21.15.102:0/537505264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080b3970 0x7f87080c5a90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f870808ce30 tx=0x7f86f4009c00 comp rx=0 tx=0).stop 2023-03-03T17:06:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 -- 172.21.15.102:0/537505264 shutdown_connections 2023-03-03T17:06:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 --2- 172.21.15.102:0/537505264 >> v2:172.21.15.172:3300/0 conn(0x7f87080c41b0 0x7f87080cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 --2- 172.21.15.102:0/537505264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080b3970 0x7f87080c5a90 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 -- 172.21.15.102:0/537505264 >> 172.21.15.102:0/537505264 conn(0x7f870808b1c0 msgr2=0x7f8708071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:06:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 -- 172.21.15.102:0/537505264 shutdown_connections 2023-03-03T17:06:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.808+0000 7f870db9d700 1 -- 172.21.15.102:0/537505264 wait complete. 2023-03-03T17:06:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.810+0000 7f870db9d700 1 Processor -- start 2023-03-03T17:06:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.810+0000 7f870db9d700 1 -- start start 2023-03-03T17:06:38.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.811+0000 7f870db9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87080b3970 0x7f87080c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:38.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.811+0000 7f870db9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87080c41b0 0x7f87081f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:38.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.811+0000 7f870db9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080ced00 0x7f8708150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:38.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.811+0000 7f870db9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87080c7770 con 0x7f87080ced00 2023-03-03T17:06:38.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.811+0000 7f870db9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87080c89b0 con 0x7f87080c41b0 2023-03-03T17:06:38.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.811+0000 7f870db9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87081f2eb0 con 0x7f87080b3970 2023-03-03T17:06:38.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f870d39c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080ced00 0x7f8708150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:38.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f8707fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87080c41b0 0x7f87081f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:38.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f8707fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87080c41b0 0x7f87081f2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47920/0 (socket says 172.21.15.102:47920) 2023-03-03T17:06:38.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f8707fff700 1 -- 172.21.15.102:0/575753947 learned_addr learned my addr 172.21.15.102:0/575753947 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:06:38.812 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f870cb9b700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87080b3970 0x7f87080c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:38.813 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f870d39c700 1 -- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87080b3970 msgr2=0x7f87080c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f870d39c700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87080b3970 0x7f87080c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f870d39c700 1 -- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87080c41b0 msgr2=0x7f87081f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f870d39c700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87080c41b0 0x7f87081f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f870d39c700 1 -- 172.21.15.102:0/575753947 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f4009550 con 0x7f87080ced00 2023-03-03T17:06:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.812+0000 7f8707fff700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87080c41b0 0x7f87081f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:06:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.813+0000 7f870d39c700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080ced00 0x7f8708150a00 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f870000eba0 tx=0x7f870000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:06:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.813+0000 7f87057fa700 1 -- 172.21.15.102:0/575753947 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700009a80 con 0x7f87080ced00 2023-03-03T17:06:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.813+0000 7f87057fa700 1 -- 172.21.15.102:0/575753947 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870000f040 con 0x7f87080ced00 2023-03-03T17:06:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.813+0000 7f87057fa700 1 -- 172.21.15.102:0/575753947 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000cc40 con 0x7f87080ced00 2023-03-03T17:06:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.813+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870806bf20 con 0x7f87080ced00 2023-03-03T17:06:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.813+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870815ce80 con 0x7f87080ced00 2023-03-03T17:06:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.814+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870806c200 con 0x7f87080ced00 2023-03-03T17:06:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.815+0000 7f87057fa700 1 -- 172.21.15.102:0/575753947 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8700024070 con 0x7f87080ced00 2023-03-03T17:06:38.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.817+0000 7f87057fa700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86ec0682f0 0x7f86ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:38.816 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.817+0000 7f87057fa700 1 -- 172.21.15.102:0/575753947 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8700086790 con 0x7f87080ced00 2023-03-03T17:06:38.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.817+0000 7f870cb9b700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86ec0682f0 0x7f86ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:38.817 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.818+0000 7f870cb9b700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86ec0682f0 0x7f86ec06a7a0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f86fc007fd0 tx=0x7f86fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:06:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:38.821+0000 7f87057fa700 1 -- 172.21.15.102:0/575753947 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8700052e10 con 0x7f87080ced00 2023-03-03T17:06:39.055 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.055+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f870804e6e0 con 0x7f87080ced00 2023-03-03T17:06:39.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.056+0000 7f87057fa700 1 -- 172.21.15.102:0/575753947 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8700057e60 con 0x7f87080ced00 2023-03-03T17:06:39.056 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:06:39.056 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:06:39.058 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86ec0682f0 msgr2=0x7f86ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86ec0682f0 0x7f86ec06a7a0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f86fc007fd0 tx=0x7f86fc009040 comp rx=0 tx=0).stop 2023-03-03T17:06:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080ced00 msgr2=0x7f8708150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080ced00 0x7f8708150a00 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f870000eba0 tx=0x7f870000eeb0 comp rx=0 tx=0).stop 2023-03-03T17:06:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 shutdown_connections 2023-03-03T17:06:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f86ec0682f0 0x7f86ec06a7a0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f87080b3970 0x7f87080c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87080ced00 0x7f8708150a00 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:39.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 --2- 172.21.15.102:0/575753947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87080c41b0 0x7f87081f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:39.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 >> 172.21.15.102:0/575753947 conn(0x7f870808b1c0 msgr2=0x7f87080ca990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:06:39.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 shutdown_connections 2023-03-03T17:06:39.060 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:06:39.059+0000 7f870db9d700 1 -- 172.21.15.102:0/575753947 wait complete. 2023-03-03T17:06:39.060 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:06:39.085 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-03-03T17:07:05.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:05.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:22.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:07:22.336 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.337+0000 7fc2be7ff700 1 Processor -- start 2023-03-03T17:07:22.337 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.337+0000 7fc2be7ff700 1 -- start start 2023-03-03T17:07:22.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2be7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b8153290 0x7fc2b8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:22.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2be7ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2b81593c0 0x7fc2b80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:22.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2be7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc2b80cd8e0 0x7fc2b80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:22.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2be7ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b8153660 con 0x7fc2b81593c0 2023-03-03T17:07:22.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2be7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b806c360 con 0x7fc2b80cd8e0 2023-03-03T17:07:22.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2be7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b815cea0 con 0x7fc2b8153290 2023-03-03T17:07:22.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2b81593c0 0x7fc2b80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:22.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2b81593c0 0x7fc2b80cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45432/0 (socket says 172.21.15.102:45432) 2023-03-03T17:07:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b8153290 0x7fc2b8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 -- 172.21.15.102:0/1397400232 learned_addr learned my addr 172.21.15.102:0/1397400232 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:07:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2bcd9c700 1 --2- 172.21.15.102:0/1397400232 >> v2:172.21.15.172:3300/0 conn(0x7fc2b80cd8e0 0x7fc2b80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:22.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 -- 172.21.15.102:0/1397400232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b8153290 msgr2=0x7fc2b8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:07:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 --2- 172.21.15.102:0/1397400232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b8153290 0x7fc2b8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 -- 172.21.15.102:0/1397400232 >> v2:172.21.15.172:3300/0 conn(0x7fc2b80cd8e0 msgr2=0x7fc2b80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:07:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 --2- 172.21.15.102:0/1397400232 >> v2:172.21.15.172:3300/0 conn(0x7fc2b80cd8e0 0x7fc2b80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.339+0000 7fc2b77fe700 1 -- 172.21.15.102:0/1397400232 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b806c7f0 con 0x7fc2b81593c0 2023-03-03T17:07:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.340+0000 7fc2b77fe700 1 --2- 172.21.15.102:0/1397400232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b80cde20 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc2b806d0e0 tx=0x7fc2ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97377e121b900ddb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.340+0000 7fc2b6ffd700 1 -- 172.21.15.102:0/1397400232 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac011830 con 0x7fc2b81593c0 2023-03-03T17:07:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.340+0000 7fc2b6ffd700 1 -- 172.21.15.102:0/1397400232 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac011990 con 0x7fc2b81593c0 2023-03-03T17:07:22.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.340+0000 7fc2b6ffd700 1 -- 172.21.15.102:0/1397400232 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac011d20 con 0x7fc2b81593c0 2023-03-03T17:07:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 -- 172.21.15.102:0/1397400232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 msgr2=0x7fc2b80cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/1397400232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b80cde20 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc2b806d0e0 tx=0x7fc2ac009bd0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 -- 172.21.15.102:0/1397400232 shutdown_connections 2023-03-03T17:07:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/1397400232 >> v2:172.21.15.172:3300/0 conn(0x7fc2b80cd8e0 0x7fc2b80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/1397400232 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b8153290 0x7fc2b8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/1397400232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b80cde20 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 -- 172.21.15.102:0/1397400232 >> 172.21.15.102:0/1397400232 conn(0x7fc2b8065fb0 msgr2=0x7fc2b80663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 -- 172.21.15.102:0/1397400232 shutdown_connections 2023-03-03T17:07:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.341+0000 7fc2be7ff700 1 -- 172.21.15.102:0/1397400232 wait complete. 2023-03-03T17:07:22.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.343+0000 7fc2be7ff700 1 Processor -- start 2023-03-03T17:07:22.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.343+0000 7fc2be7ff700 1 -- start start 2023-03-03T17:07:22.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2be7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b80cd8e0 0x7fc2b80c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2be7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b8153290 0x7fc2b8155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2be7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b8155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2be7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80c7060 con 0x7fc2b81593c0 2023-03-03T17:07:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2be7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b81e50c0 con 0x7fc2b8153290 2023-03-03T17:07:22.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2be7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b8153970 con 0x7fc2b80cd8e0 2023-03-03T17:07:22.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2bcd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b8155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:22.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2bcd9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b8155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45436/0 (socket says 172.21.15.102:45436) 2023-03-03T17:07:22.352 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2bcd9c700 1 -- 172.21.15.102:0/2628768322 learned_addr learned my addr 172.21.15.102:0/2628768322 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:07:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2b7fff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b80cd8e0 0x7fc2b80c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.344+0000 7fc2b77fe700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b8153290 0x7fc2b8155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2bcd9c700 1 -- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b80cd8e0 msgr2=0x7fc2b80c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:22.353 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2bcd9c700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b80cd8e0 0x7fc2b80c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2bcd9c700 1 -- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b8153290 msgr2=0x7fc2b8155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2bcd9c700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b8153290 0x7fc2b8155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2bcd9c700 1 -- 172.21.15.102:0/2628768322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac009550 con 0x7fc2b81593c0 2023-03-03T17:07:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2bcd9c700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b8155a40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000efc0 tx=0x7fc2b000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2b4ff9700 1 -- 172.21.15.102:0/2628768322 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b000cad0 con 0x7fc2b81593c0 2023-03-03T17:07:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2b4ff9700 1 -- 172.21.15.102:0/2628768322 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b000f040 con 0x7fc2b81593c0 2023-03-03T17:07:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2b4ff9700 1 -- 172.21.15.102:0/2628768322 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b000ce70 con 0x7fc2b81593c0 2023-03-03T17:07:22.355 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.345+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b80ccde0 con 0x7fc2b81593c0 2023-03-03T17:07:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.346+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b815d160 con 0x7fc2b81593c0 2023-03-03T17:07:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.347+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b80cd0c0 con 0x7fc2b81593c0 2023-03-03T17:07:22.356 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.348+0000 7fc2b4ff9700 1 -- 172.21.15.102:0/2628768322 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc2b0014480 con 0x7fc2b81593c0 2023-03-03T17:07:22.357 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.349+0000 7fc2b4ff9700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2a40682f0 0x7fc2a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:22.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.349+0000 7fc2b4ff9700 1 -- 172.21.15.102:0/2628768322 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fc2b0086750 con 0x7fc2b81593c0 2023-03-03T17:07:22.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.350+0000 7fc2b7fff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2a40682f0 0x7fc2a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:22.358 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.350+0000 7fc2b7fff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2a40682f0 0x7fc2a406a7a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009ac0 tx=0x7fc2a8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:22.359 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.353+0000 7fc2b4ff9700 1 -- 172.21.15.102:0/2628768322 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2b0052d70 con 0x7fc2b81593c0 2023-03-03T17:07:22.584 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.584+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2b804e6e0 con 0x7fc2b81593c0 2023-03-03T17:07:22.585 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.585+0000 7fc2b4ff9700 1 -- 172.21.15.102:0/2628768322 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fc2b0018090 con 0x7fc2b81593c0 2023-03-03T17:07:22.585 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:07:22.586 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:07:22.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2a40682f0 msgr2=0x7fc2a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:22.587 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2a40682f0 0x7fc2a406a7a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009ac0 tx=0x7fc2a8009210 comp rx=0 tx=0).stop 2023-03-03T17:07:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 msgr2=0x7fc2b8155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b8155a40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000efc0 tx=0x7fc2b000c0c0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 shutdown_connections 2023-03-03T17:07:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fc2a40682f0 0x7fc2a406a7a0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc2b80cd8e0 0x7fc2b80c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b81593c0 0x7fc2b8155a40 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 --2- 172.21.15.102:0/2628768322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc2b8153290 0x7fc2b8155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:22.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.587+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 >> 172.21.15.102:0/2628768322 conn(0x7fc2b8065fb0 msgr2=0x7fc2b81e52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:22.589 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.588+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 shutdown_connections 2023-03-03T17:07:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:22.588+0000 7fc2be7ff700 1 -- 172.21.15.102:0/2628768322 wait complete. 2023-03-03T17:07:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:07:22.616 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-03-03T17:07:35.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:35.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:57.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:07:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.872+0000 7f435a7ff700 1 Processor -- start 2023-03-03T17:07:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.872+0000 7f435a7ff700 1 -- start start 2023-03-03T17:07:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.873+0000 7f435a7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4354153290 0x7f4354158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.873+0000 7f435a7ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f43540cd8e0 0x7f43540ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.873+0000 7f435a7ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4354153660 con 0x7f43541593c0 2023-03-03T17:07:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.873+0000 7f435a7ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435406c360 con 0x7f43540cd8e0 2023-03-03T17:07:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.873+0000 7f435a7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435415cea0 con 0x7f4354153290 2023-03-03T17:07:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4353fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4354153290 0x7f4354158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4358d9c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f43540cd8e0 0x7f43540ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f434bfff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f43541593c0 0x7f43540cde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:34170/0 (socket says 172.21.15.102:34170) 2023-03-03T17:07:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4353fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4354153290 0x7f4354158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54638/0 (socket says 172.21.15.102:54638) 2023-03-03T17:07:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f434bfff700 1 -- 172.21.15.102:0/3158066763 learned_addr learned my addr 172.21.15.102:0/3158066763 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:07:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4358d9c700 1 -- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4354153290 msgr2=0x7f4354158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:07:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4358d9c700 1 --2- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4354153290 0x7f4354158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4358d9c700 1 -- 172.21.15.102:0/3158066763 >> v1:172.21.15.172:6789/0 conn(0x7f43541593c0 legacy=0x7f43540cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4358d9c700 1 -- 172.21.15.102:0/3158066763 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435406c7f0 con 0x7f43540cd8e0 2023-03-03T17:07:57.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4353fff700 1 --2- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4354153290 0x7f4354158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:07:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f43537fe700 1 -- 172.21.15.102:0/3158066763 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057289647 0 0) 0x7f4354153660 con 0x7f43541593c0 2023-03-03T17:07:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.874+0000 7f4358d9c700 1 --2- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43540ce530 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f434c009d90 tx=0x7f434c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a855da3c9780834 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f43537fe700 1 -- 172.21.15.102:0/3158066763 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c00e070 con 0x7f43540cd8e0 2023-03-03T17:07:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f43537fe700 1 -- 172.21.15.102:0/3158066763 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c00bb20 con 0x7f43540cd8e0 2023-03-03T17:07:57.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f43537fe700 1 -- 172.21.15.102:0/3158066763 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c012430 con 0x7f43540cd8e0 2023-03-03T17:07:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f435a7ff700 1 -- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 msgr2=0x7f43540ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43540ce530 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f434c009d90 tx=0x7f434c00b330 comp rx=0 tx=0).stop 2023-03-03T17:07:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f435a7ff700 1 -- 172.21.15.102:0/3158066763 shutdown_connections 2023-03-03T17:07:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43540ce530 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3158066763 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4354153290 0x7f4354158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.875+0000 7f435a7ff700 1 -- 172.21.15.102:0/3158066763 >> 172.21.15.102:0/3158066763 conn(0x7f4354065fb0 msgr2=0x7f43540663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.876+0000 7f435a7ff700 1 -- 172.21.15.102:0/3158066763 shutdown_connections 2023-03-03T17:07:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.876+0000 7f435a7ff700 1 -- 172.21.15.102:0/3158066763 wait complete. 2023-03-03T17:07:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.878+0000 7f435a7ff700 1 Processor -- start 2023-03-03T17:07:57.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.878+0000 7f435a7ff700 1 -- start start 2023-03-03T17:07:57.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f435a7ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43541f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:57.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f435a7ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4354153290 0x7f43540c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:57.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f435a7ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43541593c0 0x7f43541ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:57.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f435a7ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43541eece0 con 0x7f4354153290 2023-03-03T17:07:57.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f435a7ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43540c8710 con 0x7f43540cd8e0 2023-03-03T17:07:57.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f435a7ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43541f2400 con 0x7f43541593c0 2023-03-03T17:07:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f434bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4354153290 0x7f43540c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f4353fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43541f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f4358d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43541593c0 0x7f43541ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f4358d9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43541593c0 0x7f43541ee7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:54650/0 (socket says 172.21.15.102:54650) 2023-03-03T17:07:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f4353fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43541f2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:50498/0 (socket says 172.21.15.102:50498) 2023-03-03T17:07:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.879+0000 7f4358d9c700 1 -- 172.21.15.102:0/3513199081 learned_addr learned my addr 172.21.15.102:0/3513199081 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:07:57.881 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f4358d9c700 1 -- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 msgr2=0x7f43541f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f4358d9c700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43541f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f4358d9c700 1 -- 172.21.15.102:0/3513199081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4354153290 msgr2=0x7f43540c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f4358d9c700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4354153290 0x7f43540c2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f4358d9c700 1 -- 172.21.15.102:0/3513199081 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43440094a0 con 0x7f43541593c0 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f4353fff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43541f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f4358d9c700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43541593c0 0x7f43541ee7a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f434c00b680 tx=0x7f434c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f43517fa700 1 -- 172.21.15.102:0/3513199081 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c012430 con 0x7f43541593c0 2023-03-03T17:07:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f43517fa700 1 -- 172.21.15.102:0/3513199081 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c020070 con 0x7f43541593c0 2023-03-03T17:07:57.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c009a70 con 0x7f43541593c0 2023-03-03T17:07:57.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.880+0000 7f43517fa700 1 -- 172.21.15.102:0/3513199081 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c004280 con 0x7f43541593c0 2023-03-03T17:07:57.883 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.881+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43540ccde0 con 0x7f43541593c0 2023-03-03T17:07:57.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.881+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435415d160 con 0x7f43541593c0 2023-03-03T17:07:57.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.883+0000 7f43517fa700 1 -- 172.21.15.102:0/3513199081 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f434c0043e0 con 0x7f43541593c0 2023-03-03T17:07:57.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.884+0000 7f43517fa700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4334068320 0x7f433406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:57.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.884+0000 7f43517fa700 1 -- 172.21.15.102:0/3513199081 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f434c022070 con 0x7f43541593c0 2023-03-03T17:07:57.884 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.884+0000 7f4353fff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4334068320 0x7f433406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:57.885 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.885+0000 7f4353fff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4334068320 0x7f433406a7d0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4344011380 tx=0x7f4344009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:57.887 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:57.888+0000 7f43517fa700 1 -- 172.21.15.102:0/3513199081 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f434c053f80 con 0x7f43541593c0 2023-03-03T17:07:58.125 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.125+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43540cd0c0 con 0x7f43541593c0 2023-03-03T17:07:58.126 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.126+0000 7f43517fa700 1 -- 172.21.15.102:0/3513199081 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f434c0575a0 con 0x7f43541593c0 2023-03-03T17:07:58.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:07:58.126 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:07:58.128 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4334068320 msgr2=0x7f433406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:58.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4334068320 0x7f433406a7d0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4344011380 tx=0x7f4344009210 comp rx=0 tx=0).stop 2023-03-03T17:07:58.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43541593c0 msgr2=0x7f43541ee7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:58.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43541593c0 0x7f43541ee7a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f434c00b680 tx=0x7f434c004ad0 comp rx=0 tx=0).stop 2023-03-03T17:07:58.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 shutdown_connections 2023-03-03T17:07:58.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f4334068320 0x7f433406a7d0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:58.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f43541593c0 0x7f43541ee7a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:58.129 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f43540cd8e0 0x7f43541f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:58.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 --2- 172.21.15.102:0/3513199081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4354153290 0x7f43540c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:58.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 >> 172.21.15.102:0/3513199081 conn(0x7f4354065fb0 msgr2=0x7f43541518d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:58.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 shutdown_connections 2023-03-03T17:07:58.130 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:07:58.129+0000 7f435a7ff700 1 -- 172.21.15.102:0/3513199081 wait complete. 2023-03-03T17:07:58.130 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:07:58.155 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T17:08:05.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:05.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:35.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:35.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:05.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:09:05.409 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.409+0000 7f2b7359e700 1 Processor -- start 2023-03-03T17:09:05.410 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.410+0000 7f2b7359e700 1 -- start start 2023-03-03T17:09:05.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.411+0000 7f2b7359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b740ced00 0x7f2b740c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:05.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.411+0000 7f2b7359e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2b740c41b0 0x7f2b740cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:05.411 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.411+0000 7f2b7359e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740c3d20 con 0x7f2b740b3970 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.411+0000 7f2b7359e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b740c4930 con 0x7f2b740c41b0 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.411+0000 7f2b7359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b740cf610 con 0x7f2b740ced00 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.411+0000 7f2b7259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b740ced00 0x7f2b740c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.411+0000 7f2b72d9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2b740c41b0 0x7f2b740cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b7259c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b740ced00 0x7f2b740c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:39098/0 (socket says 172.21.15.102:39098) 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b7259c700 1 -- 172.21.15.102:0/1605245788 learned_addr learned my addr 172.21.15.102:0/1605245788 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b72d9d700 1 -- 172.21.15.102:0/1605245788 >> v2:172.21.15.172:3300/0 conn(0x7f2b740ced00 msgr2=0x7f2b740c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:09:05.412 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b72d9d700 1 --2- 172.21.15.102:0/1605245788 >> v2:172.21.15.172:3300/0 conn(0x7f2b740ced00 0x7f2b740c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.413 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b72d9d700 1 -- 172.21.15.102:0/1605245788 >> v1:172.21.15.102:6789/0 conn(0x7f2b740b3970 legacy=0x7f2b74158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b72d9d700 1 -- 172.21.15.102:0/1605245788 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7415cba0 con 0x7f2b740c41b0 2023-03-03T17:09:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b71d9b700 1 -- 172.21.15.102:0/1605245788 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188078494 0 0) 0x7f2b740c3d20 con 0x7f2b740b3970 2023-03-03T17:09:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b7259c700 1 --2- 172.21.15.102:0/1605245788 >> v2:172.21.15.172:3300/0 conn(0x7f2b740ced00 0x7f2b740c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:09:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.412+0000 7f2b72d9d700 1 --2- 172.21.15.102:0/1605245788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b740cf0d0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f2b740b1b40 tx=0x7f2b6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fd4b4296bb867a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b71d9b700 1 -- 172.21.15.102:0/1605245788 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00e070 con 0x7f2b740c41b0 2023-03-03T17:09:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b71d9b700 1 -- 172.21.15.102:0/1605245788 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6c00bae0 con 0x7f2b740c41b0 2023-03-03T17:09:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b71d9b700 1 -- 172.21.15.102:0/1605245788 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00be90 con 0x7f2b740c41b0 2023-03-03T17:09:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b7359e700 1 -- 172.21.15.102:0/1605245788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 msgr2=0x7f2b740cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b7359e700 1 --2- 172.21.15.102:0/1605245788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b740cf0d0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f2b740b1b40 tx=0x7f2b6c00b330 comp rx=0 tx=0).stop 2023-03-03T17:09:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b7359e700 1 -- 172.21.15.102:0/1605245788 shutdown_connections 2023-03-03T17:09:05.415 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b7359e700 1 --2- 172.21.15.102:0/1605245788 >> v2:172.21.15.172:3300/0 conn(0x7f2b740ced00 0x7f2b740c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b7359e700 1 --2- 172.21.15.102:0/1605245788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b740cf0d0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b7359e700 1 -- 172.21.15.102:0/1605245788 >> 172.21.15.102:0/1605245788 conn(0x7f2b7408b1c0 msgr2=0x7f2b74071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:09:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.413+0000 7f2b7359e700 1 -- 172.21.15.102:0/1605245788 shutdown_connections 2023-03-03T17:09:05.416 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.414+0000 7f2b7359e700 1 -- 172.21.15.102:0/1605245788 wait complete. 2023-03-03T17:09:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.416+0000 7f2b7359e700 1 Processor -- start 2023-03-03T17:09:05.417 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.417+0000 7f2b7359e700 1 -- start start 2023-03-03T17:09:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b7359e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b740b3970 0x7f2b741e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b7359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b741ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b7359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b740ced00 0x7f2b741ee960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b7359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b741f25b0 con 0x7f2b740c41b0 2023-03-03T17:09:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b7359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b74151be0 con 0x7f2b740ced00 2023-03-03T17:09:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b7359e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b740ccef0 con 0x7f2b740b3970 2023-03-03T17:09:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b6bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b741ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b72d9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b740ced00 0x7f2b741ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.418+0000 7f2b6bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b741ee420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41546/0 (socket says 172.21.15.102:41546) 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b6bfff700 1 -- 172.21.15.102:0/2717609410 learned_addr learned my addr 172.21.15.102:0/2717609410 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b7259c700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b740b3970 0x7f2b741e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b6bfff700 1 -- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b740b3970 msgr2=0x7f2b741e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b6bfff700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b740b3970 0x7f2b741e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b6bfff700 1 -- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b740ced00 msgr2=0x7f2b741ee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b6bfff700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b740ced00 0x7f2b741ee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b6bfff700 1 -- 172.21.15.102:0/2717609410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b64009550 con 0x7f2b740c41b0 2023-03-03T17:09:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b72d9d700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b740ced00 0x7f2b741ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:09:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.419+0000 7f2b6bfff700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b741ee420 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00c450 tx=0x7f2b5c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:05.421 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.420+0000 7f2b6affd700 1 -- 172.21.15.102:0/2717609410 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0054e0 con 0x7f2b740c41b0 2023-03-03T17:09:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.420+0000 7f2b6affd700 1 -- 172.21.15.102:0/2717609410 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c00ce60 con 0x7f2b740c41b0 2023-03-03T17:09:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.420+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c009a70 con 0x7f2b740c41b0 2023-03-03T17:09:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.420+0000 7f2b6affd700 1 -- 172.21.15.102:0/2717609410 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0038a0 con 0x7f2b740c41b0 2023-03-03T17:09:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.420+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7406bf20 con 0x7f2b740c41b0 2023-03-03T17:09:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.421+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7415ce80 con 0x7f2b740c41b0 2023-03-03T17:09:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.422+0000 7f2b6affd700 1 -- 172.21.15.102:0/2717609410 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2b5c019030 con 0x7f2b740c41b0 2023-03-03T17:09:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.423+0000 7f2b6affd700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b540682f0 0x7f2b5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.424+0000 7f2b6affd700 1 -- 172.21.15.102:0/2717609410 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f2b5c084fe0 con 0x7f2b740c41b0 2023-03-03T17:09:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.424+0000 7f2b7259c700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b540682f0 0x7f2b5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.424+0000 7f2b7259c700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b540682f0 0x7f2b5406a7a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2b64006d30 tx=0x7f2b640092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:05.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.428+0000 7f2b6affd700 1 -- 172.21.15.102:0/2717609410 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2b5c051690 con 0x7f2b740c41b0 2023-03-03T17:09:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.663+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b7406c200 con 0x7f2b740c41b0 2023-03-03T17:09:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.664+0000 7f2b6affd700 1 -- 172.21.15.102:0/2717609410 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f2b5c017020 con 0x7f2b740c41b0 2023-03-03T17:09:05.664 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:09:05.665 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:09:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.666+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b540682f0 msgr2=0x7f2b5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:05.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.666+0000 7f2b7359e700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b540682f0 0x7f2b5406a7a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2b64006d30 tx=0x7f2b640092c0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.666+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 msgr2=0x7f2b741ee420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:05.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.666+0000 7f2b7359e700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b741ee420 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00c450 tx=0x7f2b5c00c760 comp rx=0 tx=0).stop 2023-03-03T17:09:05.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 shutdown_connections 2023-03-03T17:09:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f2b540682f0 0x7f2b5406a7a0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2b740b3970 0x7f2b741e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b740ced00 0x7f2b741ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 --2- 172.21.15.102:0/2717609410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b740c41b0 0x7f2b741ee420 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:05.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 >> 172.21.15.102:0/2717609410 conn(0x7f2b7408b1c0 msgr2=0x7f2b74071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:09:05.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 shutdown_connections 2023-03-03T17:09:05.671 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:09:05.667+0000 7f2b7359e700 1 -- 172.21.15.102:0/2717609410 wait complete. 2023-03-03T17:09:05.672 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:09:05.694 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-03-03T17:09:05.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:05.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:35.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:35.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:59.999 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T17:09:59.998+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:10:06.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:06.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:12.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:10:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.947+0000 7f50b7fff700 1 Processor -- start 2023-03-03T17:10:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.947+0000 7f50b7fff700 1 -- start start 2023-03-03T17:10:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b7fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f50b8153030 0x7f50b80c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b7fff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f50b80b1e90 0x7f50b815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b7fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b806bee0 con 0x7f50b80b3f30 2023-03-03T17:10:12.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b7fff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b806c780 con 0x7f50b80b1e90 2023-03-03T17:10:12.950 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b7fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b815c620 con 0x7f50b8153030 2023-03-03T17:10:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b67fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f50b80b1e90 0x7f50b815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b6ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f50b8153030 0x7f50b80c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b67fc700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f50b80b1e90 0x7f50b815ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48154/0 (socket says 172.21.15.102:48154) 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.949+0000 7f50b67fc700 1 -- 172.21.15.102:0/1567238742 learned_addr learned my addr 172.21.15.102:0/1567238742 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b6ffd700 1 -- 172.21.15.102:0/1567238742 >> v2:172.21.15.102:3300/0 conn(0x7f50b80b1e90 msgr2=0x7f50b815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b6ffd700 1 --2- 172.21.15.102:0/1567238742 >> v2:172.21.15.102:3300/0 conn(0x7f50b80b1e90 0x7f50b815ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b6ffd700 1 -- 172.21.15.102:0/1567238742 >> v1:172.21.15.172:6789/0 conn(0x7f50b80b3f30 legacy=0x7f50b80cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b6ffd700 1 -- 172.21.15.102:0/1567238742 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b806c1c0 con 0x7f50b8153030 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b5ffb700 1 -- 172.21.15.102:0/1567238742 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055781790 0 0) 0x7f50b806bee0 con 0x7f50b80b3f30 2023-03-03T17:10:12.952 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b67fc700 1 --2- 172.21.15.102:0/1567238742 >> v2:172.21.15.102:3300/0 conn(0x7f50b80b1e90 0x7f50b815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:10:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b6ffd700 1 --2- 172.21.15.102:0/1567238742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 0x7f50b80c5b00 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009840 tx=0x7f50a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=612b7f54d8bcb7d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b5ffb700 1 -- 172.21.15.102:0/1567238742 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8011830 con 0x7f50b8153030 2023-03-03T17:10:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b5ffb700 1 -- 172.21.15.102:0/1567238742 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a8011990 con 0x7f50b8153030 2023-03-03T17:10:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.950+0000 7f50b5ffb700 1 -- 172.21.15.102:0/1567238742 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8011d20 con 0x7f50b8153030 2023-03-03T17:10:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 -- 172.21.15.102:0/1567238742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 msgr2=0x7f50b80c5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 --2- 172.21.15.102:0/1567238742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 0x7f50b80c5b00 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009840 tx=0x7f50a8009b50 comp rx=0 tx=0).stop 2023-03-03T17:10:12.953 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 -- 172.21.15.102:0/1567238742 shutdown_connections 2023-03-03T17:10:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 --2- 172.21.15.102:0/1567238742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 0x7f50b80c5b00 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 --2- 172.21.15.102:0/1567238742 >> v2:172.21.15.102:3300/0 conn(0x7f50b80b1e90 0x7f50b815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 -- 172.21.15.102:0/1567238742 >> 172.21.15.102:0/1567238742 conn(0x7f50b808b1c0 msgr2=0x7f50b8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 -- 172.21.15.102:0/1567238742 shutdown_connections 2023-03-03T17:10:12.954 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.951+0000 7f50b7fff700 1 -- 172.21.15.102:0/1567238742 wait complete. 2023-03-03T17:10:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.953+0000 7f50b7fff700 1 Processor -- start 2023-03-03T17:10:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.953+0000 7f50b7fff700 1 -- start start 2023-03-03T17:10:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.954+0000 7f50b7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50b80b1e90 0x7f50b81ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.954+0000 7f50b7fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50b80b3f30 0x7f50b81f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.954+0000 7f50b7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 0x7f50b8159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.954+0000 7f50b7fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b8155860 con 0x7f50b80b1e90 2023-03-03T17:10:12.956 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.954+0000 7f50b7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81f1bb0 con 0x7f50b8153030 2023-03-03T17:10:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.954+0000 7f50b7fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81ee8b0 con 0x7f50b80b3f30 2023-03-03T17:10:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50b80b1e90 0x7f50b81ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:12.957 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50b80b1e90 0x7f50b81ee1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48156/0 (socket says 172.21.15.102:48156) 2023-03-03T17:10:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 -- 172.21.15.102:0/2253559316 learned_addr learned my addr 172.21.15.102:0/2253559316 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:10:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b67fc700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50b80b3f30 0x7f50b81f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b77fe700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 0x7f50b8159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 -- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50b80b3f30 msgr2=0x7f50b81f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50b80b3f30 0x7f50b81f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 -- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 msgr2=0x7f50b8159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:12.958 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 0x7f50b8159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 -- 172.21.15.102:0/2253559316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009550 con 0x7f50b80b1e90 2023-03-03T17:10:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.955+0000 7f50b6ffd700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50b80b1e90 0x7f50b81ee1b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f50a80090b0 tx=0x7f50a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.956+0000 7f509f7fe700 1 -- 172.21.15.102:0/2253559316 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8004510 con 0x7f50b80b1e90 2023-03-03T17:10:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.956+0000 7f509f7fe700 1 -- 172.21.15.102:0/2253559316 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a801e070 con 0x7f50b80b1e90 2023-03-03T17:10:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.956+0000 7f509f7fe700 1 -- 172.21.15.102:0/2253559316 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a80038e0 con 0x7f50b80b1e90 2023-03-03T17:10:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.956+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a80094a0 con 0x7f50b80b1e90 2023-03-03T17:10:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.956+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b815c240 con 0x7f50b80b1e90 2023-03-03T17:10:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.957+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b806c4a0 con 0x7f50b80b1e90 2023-03-03T17:10:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.958+0000 7f509f7fe700 1 -- 172.21.15.102:0/2253559316 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f50a800f050 con 0x7f50b80b1e90 2023-03-03T17:10:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.959+0000 7f509f7fe700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50a40682f0 0x7f50a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.960+0000 7f509f7fe700 1 -- 172.21.15.102:0/2253559316 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f50a8086cd0 con 0x7f50b80b1e90 2023-03-03T17:10:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.960+0000 7f50b67fc700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50a40682f0 0x7f50a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:12.964 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.961+0000 7f50b67fc700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50a40682f0 0x7f50a406a7a0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f50b80b19b0 tx=0x7f50ac0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:12.965 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:12.964+0000 7f509f7fe700 1 -- 172.21.15.102:0/2253559316 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50a8053360 con 0x7f50b80b1e90 2023-03-03T17:10:13.197 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.197+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50b815cf80 con 0x7f50b80b1e90 2023-03-03T17:10:13.198 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.197+0000 7f509f7fe700 1 -- 172.21.15.102:0/2253559316 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f50a8056980 con 0x7f50b80b1e90 2023-03-03T17:10:13.198 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:10:13.198 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:10:13.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50a40682f0 msgr2=0x7f50a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:13.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50a40682f0 0x7f50a406a7a0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f50b80b19b0 tx=0x7f50ac0092c0 comp rx=0 tx=0).stop 2023-03-03T17:10:13.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50b80b1e90 msgr2=0x7f50b81ee1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:13.200 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50b80b1e90 0x7f50b81ee1b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f50a80090b0 tx=0x7f50a8004750 comp rx=0 tx=0).stop 2023-03-03T17:10:13.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 shutdown_connections 2023-03-03T17:10:13.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50b8153030 0x7f50b8159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:13.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50b80b1e90 0x7f50b81ee1b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:13.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50a40682f0 0x7f50a406a7a0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:13.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 --2- 172.21.15.102:0/2253559316 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50b80b3f30 0x7f50b81f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:13.201 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 >> 172.21.15.102:0/2253559316 conn(0x7f50b808b1c0 msgr2=0x7f50b80cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:13.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 shutdown_connections 2023-03-03T17:10:13.202 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:13.200+0000 7f50b7fff700 1 -- 172.21.15.102:0/2253559316 wait complete. 2023-03-03T17:10:13.202 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:10:13.227 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 37 secs before thrashing 2023-03-03T17:10:36.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:36.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:50.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:10:50.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.475+0000 7f6d6c3ff700 1 Processor -- start 2023-03-03T17:10:50.476 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.476+0000 7f6d6c3ff700 1 -- start start 2023-03-03T17:10:50.477 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.477+0000 7f6d6c3ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d640cd8e0 0x7f6d640cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:50.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.477+0000 7f6d6c3ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d64153660 con 0x7f6d64153290 2023-03-03T17:10:50.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.477+0000 7f6d6c3ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6406c360 con 0x7f6d640cd390 2023-03-03T17:10:50.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.477+0000 7f6d6c3ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6415cea0 con 0x7f6d640cd8e0 2023-03-03T17:10:50.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.477+0000 7f6d6999a700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d640cd8e0 0x7f6d640cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:50.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.477+0000 7f6d6a19b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6d64153290 0x7f6d64158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57728/0 (socket says 172.21.15.102:57728) 2023-03-03T17:10:50.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.477+0000 7f6d6a19b700 1 -- 172.21.15.102:0/957363413 learned_addr learned my addr 172.21.15.102:0/957363413 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:10:50.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.478+0000 7f6d6999a700 1 -- 172.21.15.102:0/957363413 >> v1:172.21.15.172:6789/0 conn(0x7f6d640cd390 legacy=0x7f6d640ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:50.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.478+0000 7f6d6999a700 1 -- 172.21.15.102:0/957363413 >> v1:172.21.15.102:6789/0 conn(0x7f6d64153290 legacy=0x7f6d64158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:50.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.478+0000 7f6d69199700 1 -- 172.21.15.102:0/957363413 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550183564 0 0) 0x7f6d6406c360 con 0x7f6d640cd390 2023-03-03T17:10:50.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.478+0000 7f6d6999a700 1 -- 172.21.15.102:0/957363413 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6406c7f0 con 0x7f6d640cd8e0 2023-03-03T17:10:50.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.478+0000 7f6d6999a700 1 --2- 172.21.15.102:0/957363413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d640cd8e0 0x7f6d640cde20 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009840 tx=0x7f6d54009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4ea5c3d98f934b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:50.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d69199700 1 -- 172.21.15.102:0/957363413 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54004230 con 0x7f6d640cd8e0 2023-03-03T17:10:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d69199700 1 -- 172.21.15.102:0/957363413 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d54004390 con 0x7f6d640cd8e0 2023-03-03T17:10:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d69199700 1 -- 172.21.15.102:0/957363413 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d540036a0 con 0x7f6d640cd8e0 2023-03-03T17:10:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/957363413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d640cd8e0 msgr2=0x7f6d640cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/957363413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d640cd8e0 0x7f6d640cde20 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009840 tx=0x7f6d54009b50 comp rx=0 tx=0).stop 2023-03-03T17:10:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/957363413 shutdown_connections 2023-03-03T17:10:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/957363413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d640cd8e0 0x7f6d640cde20 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/957363413 >> 172.21.15.102:0/957363413 conn(0x7f6d64065fb0 msgr2=0x7f6d640663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/957363413 shutdown_connections 2023-03-03T17:10:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.479+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/957363413 wait complete. 2023-03-03T17:10:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.481+0000 7f6d6c3ff700 1 Processor -- start 2023-03-03T17:10:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.482+0000 7f6d6c3ff700 1 -- start start 2023-03-03T17:10:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.482+0000 7f6d6c3ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d640cd390 0x7f6d641efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6c3ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d640cd8e0 0x7f6d641f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:50.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6c3ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d64153290 0x7f6d641e3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:50.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6c3ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641e5050 con 0x7f6d64153290 2023-03-03T17:10:50.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6c3ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641ef800 con 0x7f6d640cd8e0 2023-03-03T17:10:50.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6c3ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641f2520 con 0x7f6d640cd390 2023-03-03T17:10:50.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d640cd390 0x7f6d641efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:50.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6a99c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d64153290 0x7f6d641e3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:50.484 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6999a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d640cd8e0 0x7f6d641f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:50.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6a19b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d640cd390 0x7f6d641efcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:47822/0 (socket says 172.21.15.102:47822) 2023-03-03T17:10:50.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6a19b700 1 -- 172.21.15.102:0/675882846 learned_addr learned my addr 172.21.15.102:0/675882846 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:10:50.485 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.483+0000 7f6d6a99c700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d64153290 0x7f6d641e3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44408/0 (socket says 172.21.15.102:44408) 2023-03-03T17:10:50.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6a99c700 1 -- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d640cd390 msgr2=0x7f6d641efcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:50.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6a99c700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d640cd390 0x7f6d641efcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:50.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6a99c700 1 -- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d640cd8e0 msgr2=0x7f6d641f01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:50.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6a99c700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d640cd8e0 0x7f6d641f01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:50.486 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6a99c700 1 -- 172.21.15.102:0/675882846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d540094a0 con 0x7f6d64153290 2023-03-03T17:10:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6a99c700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d64153290 0x7f6d641e3520 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00c760 tx=0x7f6d5c00ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d5affd700 1 -- 172.21.15.102:0/675882846 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c003e30 con 0x7f6d64153290 2023-03-03T17:10:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d5affd700 1 -- 172.21.15.102:0/675882846 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c005070 con 0x7f6d64153290 2023-03-03T17:10:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d5affd700 1 -- 172.21.15.102:0/675882846 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c005420 con 0x7f6d64153290 2023-03-03T17:10:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d640ccde0 con 0x7f6d64153290 2023-03-03T17:10:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.484+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6415d160 con 0x7f6d64153290 2023-03-03T17:10:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.485+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d640cd0c0 con 0x7f6d64153290 2023-03-03T17:10:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.487+0000 7f6d5affd700 1 -- 172.21.15.102:0/675882846 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6d5c010070 con 0x7f6d64153290 2023-03-03T17:10:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.488+0000 7f6d5affd700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d48068320 0x7f6d4806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.488+0000 7f6d5affd700 1 -- 172.21.15.102:0/675882846 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f6d5c085810 con 0x7f6d64153290 2023-03-03T17:10:50.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.488+0000 7f6d6a19b700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d48068320 0x7f6d4806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:50.489 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.489+0000 7f6d6a19b700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d48068320 0x7f6d4806a7d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f6d60000f60 tx=0x7f6d60004a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.492+0000 7f6d5affd700 1 -- 172.21.15.102:0/675882846 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6d5c051e90 con 0x7f6d64153290 2023-03-03T17:10:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.730+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d6404e6e0 con 0x7f6d64153290 2023-03-03T17:10:50.732 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.731+0000 7f6d5affd700 1 -- 172.21.15.102:0/675882846 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f6d5c0554b0 con 0x7f6d64153290 2023-03-03T17:10:50.732 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:10:50.732 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:10:50.734 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d48068320 msgr2=0x7f6d4806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:50.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d48068320 0x7f6d4806a7d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f6d60000f60 tx=0x7f6d60004a50 comp rx=0 tx=0).stop 2023-03-03T17:10:50.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d64153290 msgr2=0x7f6d641e3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:50.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d64153290 0x7f6d641e3520 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00c760 tx=0x7f6d5c00ca70 comp rx=0 tx=0).stop 2023-03-03T17:10:50.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 shutdown_connections 2023-03-03T17:10:50.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f6d48068320 0x7f6d4806a7d0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:50.735 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6d640cd390 0x7f6d641efcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:50.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d64153290 0x7f6d641e3520 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:50.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 --2- 172.21.15.102:0/675882846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d640cd8e0 0x7f6d641f01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:50.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 >> 172.21.15.102:0/675882846 conn(0x7f6d64065fb0 msgr2=0x7f6d640cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:50.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 shutdown_connections 2023-03-03T17:10:50.736 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:10:50.734+0000 7f6d6c3ff700 1 -- 172.21.15.102:0/675882846 wait complete. 2023-03-03T17:10:50.736 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:10:50.762 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 74 secs before thrashing 2023-03-03T17:11:06.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:06.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:36.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:36.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:04.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:12:05.018 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.015+0000 7fae79fff700 1 Processor -- start 2023-03-03T17:12:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.016+0000 7fae79fff700 1 -- start start 2023-03-03T17:12:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.018+0000 7fae79fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fae740cd390 0x7fae740ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.018+0000 7fae79fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae74153660 con 0x7fae74153290 2023-03-03T17:12:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.018+0000 7fae79fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7406c360 con 0x7fae740cd8e0 2023-03-03T17:12:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.018+0000 7fae79fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7415cea0 con 0x7fae740cd390 2023-03-03T17:12:05.019 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.018+0000 7fae737fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fae74153290 0x7fae74158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57720/0 (socket says 172.21.15.102:57720) 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.018+0000 7fae737fe700 1 -- 172.21.15.102:0/1545854917 learned_addr learned my addr 172.21.15.102:0/1545854917 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.018+0000 7fae73fff700 1 --2- 172.21.15.102:0/1545854917 >> v2:172.21.15.172:3300/0 conn(0x7fae740cd390 0x7fae740ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae727fc700 1 -- 172.21.15.102:0/1545854917 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401122537 0 0) 0x7fae74153660 con 0x7fae74153290 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae727fc700 1 -- 172.21.15.102:0/1545854917 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae5c0035b0 con 0x7fae74153290 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae727fc700 1 -- 172.21.15.102:0/1545854917 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775038645 0 0) 0x7fae7406c360 con 0x7fae740cd8e0 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae727fc700 1 -- 172.21.15.102:0/1545854917 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae74153660 con 0x7fae740cd8e0 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae73fff700 1 -- 172.21.15.102:0/1545854917 >> v1:172.21.15.172:6789/0 conn(0x7fae740cd8e0 legacy=0x7fae740cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:05.020 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae73fff700 1 -- 172.21.15.102:0/1545854917 >> v1:172.21.15.102:6789/0 conn(0x7fae74153290 legacy=0x7fae74158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae73fff700 1 -- 172.21.15.102:0/1545854917 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7406c7f0 con 0x7fae740cd390 2023-03-03T17:12:05.022 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae73fff700 1 --2- 172.21.15.102:0/1545854917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd390 0x7fae740ce530 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009d90 tx=0x7fae6c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e22d5d9c9bb16a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:05.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae727fc700 1 -- 172.21.15.102:0/1545854917 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c00e070 con 0x7fae740cd390 2023-03-03T17:12:05.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae727fc700 1 -- 172.21.15.102:0/1545854917 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c00bb20 con 0x7fae740cd390 2023-03-03T17:12:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.019+0000 7fae727fc700 1 -- 172.21.15.102:0/1545854917 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c012430 con 0x7fae740cd390 2023-03-03T17:12:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.020+0000 7fae79fff700 1 -- 172.21.15.102:0/1545854917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd390 msgr2=0x7fae740ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.020+0000 7fae79fff700 1 --2- 172.21.15.102:0/1545854917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd390 0x7fae740ce530 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009d90 tx=0x7fae6c00b330 comp rx=0 tx=0).stop 2023-03-03T17:12:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.020+0000 7fae79fff700 1 -- 172.21.15.102:0/1545854917 shutdown_connections 2023-03-03T17:12:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.020+0000 7fae79fff700 1 --2- 172.21.15.102:0/1545854917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd390 0x7fae740ce530 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:05.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.020+0000 7fae79fff700 1 -- 172.21.15.102:0/1545854917 >> 172.21.15.102:0/1545854917 conn(0x7fae74065fb0 msgr2=0x7fae740663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:12:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.020+0000 7fae79fff700 1 -- 172.21.15.102:0/1545854917 shutdown_connections 2023-03-03T17:12:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.020+0000 7fae79fff700 1 -- 172.21.15.102:0/1545854917 wait complete. 2023-03-03T17:12:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.023+0000 7fae79fff700 1 Processor -- start 2023-03-03T17:12:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.023+0000 7fae79fff700 1 -- start start 2023-03-03T17:12:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.024+0000 7fae79fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae740cd390 0x7fae740c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.024+0000 7fae79fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd8e0 0x7fae741ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae79fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae74153290 0x7fae740c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae79fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae740c9220 con 0x7fae740cd390 2023-03-03T17:12:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae79fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae741f2930 con 0x7fae740cd8e0 2023-03-03T17:12:05.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae79fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7414eef0 con 0x7fae74153290 2023-03-03T17:12:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd8e0 0x7fae741ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae737fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae740cd390 0x7fae740c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd8e0 0x7fae741ef940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:47014/0 (socket says 172.21.15.102:47014) 2023-03-03T17:12:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 -- 172.21.15.102:0/2298898708 learned_addr learned my addr 172.21.15.102:0/2298898708 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:12:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae737fe700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae740cd390 0x7fae740c8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33448/0 (socket says 172.21.15.102:33448) 2023-03-03T17:12:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae73fff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae74153290 0x7fae740c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:05.031 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 -- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae74153290 msgr2=0x7fae740c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:05.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae74153290 0x7fae740c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:05.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 -- 172.21.15.102:0/2298898708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae740cd390 msgr2=0x7fae740c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:05.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae740cd390 0x7fae740c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:05.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae72ffd700 1 -- 172.21.15.102:0/2298898708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c009a70 con 0x7fae740cd8e0 2023-03-03T17:12:05.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.025+0000 7fae737fe700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae740cd390 0x7fae740c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:12:05.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.026+0000 7fae72ffd700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd8e0 0x7fae741ef940 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c550 tx=0x7fae6800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:05.032 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.026+0000 7fae5bfff700 1 -- 172.21.15.102:0/2298898708 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae680052d0 con 0x7fae740cd8e0 2023-03-03T17:12:05.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.026+0000 7fae5bfff700 1 -- 172.21.15.102:0/2298898708 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68005430 con 0x7fae740cd8e0 2023-03-03T17:12:05.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.026+0000 7fae5bfff700 1 -- 172.21.15.102:0/2298898708 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae680038c0 con 0x7fae740cd8e0 2023-03-03T17:12:05.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.026+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae740ccde0 con 0x7fae740cd8e0 2023-03-03T17:12:05.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.026+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7415d160 con 0x7fae740cd8e0 2023-03-03T17:12:05.033 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.027+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae740cd0c0 con 0x7fae740cd8e0 2023-03-03T17:12:05.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.029+0000 7fae5bfff700 1 -- 172.21.15.102:0/2298898708 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae68019030 con 0x7fae740cd8e0 2023-03-03T17:12:05.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.030+0000 7fae5bfff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae600682f0 0x7fae6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:05.034 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.030+0000 7fae5bfff700 1 -- 172.21.15.102:0/2298898708 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fae680850c0 con 0x7fae740cd8e0 2023-03-03T17:12:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.030+0000 7fae737fe700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae600682f0 0x7fae6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.031+0000 7fae737fe700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae600682f0 0x7fae6006a7a0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fae64007fd0 tx=0x7fae64007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:05.035 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.034+0000 7fae5bfff700 1 -- 172.21.15.102:0/2298898708 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae680517d0 con 0x7fae740cd8e0 2023-03-03T17:12:05.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.267+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae7404e6e0 con 0x7fae740cd8e0 2023-03-03T17:12:05.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.268+0000 7fae5bfff700 1 -- 172.21.15.102:0/2298898708 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fae68054df0 con 0x7fae740cd8e0 2023-03-03T17:12:05.268 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:12:05.268 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:12:05.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae600682f0 msgr2=0x7fae6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:05.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae600682f0 0x7fae6006a7a0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fae64007fd0 tx=0x7fae64007e00 comp rx=0 tx=0).stop 2023-03-03T17:12:05.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd8e0 msgr2=0x7fae741ef940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:05.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd8e0 0x7fae741ef940 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c550 tx=0x7fae6800c860 comp rx=0 tx=0).stop 2023-03-03T17:12:05.270 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 shutdown_connections 2023-03-03T17:12:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fae600682f0 0x7fae6006a7a0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fae74153290 0x7fae740c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae740cd390 0x7fae740c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.270+0000 7fae79fff700 1 --2- 172.21.15.102:0/2298898708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae740cd8e0 0x7fae741ef940 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.271+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 >> 172.21.15.102:0/2298898708 conn(0x7fae74065fb0 msgr2=0x7fae740cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:12:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.271+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 shutdown_connections 2023-03-03T17:12:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:05.271+0000 7fae79fff700 1 -- 172.21.15.102:0/2298898708 wait complete. 2023-03-03T17:12:05.272 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:12:05.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2023-03-03T17:12:06.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:06.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:19.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:12:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.554+0000 7f8862b9d700 1 Processor -- start 2023-03-03T17:12:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.554+0000 7f8862b9d700 1 -- start start 2023-03-03T17:12:19.557 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8862b9d700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f885c0ced00 0x7f885c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8862b9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f885c0c41b0 0x7f885c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8862b9d700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885c0c3d20 con 0x7f885c0b3970 2023-03-03T17:12:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8862b9d700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885c0c4930 con 0x7f885c0ced00 2023-03-03T17:12:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8862b9d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885c0cf610 con 0x7f885c0c41b0 2023-03-03T17:12:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8861b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f885c0ced00 0x7f885c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8861b9b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f885c0ced00 0x7f885c0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51428/0 (socket says 172.21.15.102:51428) 2023-03-03T17:12:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f8861b9b700 1 -- 172.21.15.102:0/3868378189 learned_addr learned my addr 172.21.15.102:0/3868378189 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:12:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.556+0000 7f886239c700 1 --2- 172.21.15.102:0/3868378189 >> v2:172.21.15.172:3300/0 conn(0x7f885c0c41b0 0x7f885c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.557+0000 7f8861b9b700 1 -- 172.21.15.102:0/3868378189 >> v2:172.21.15.172:3300/0 conn(0x7f885c0c41b0 msgr2=0x7f885c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:12:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.557+0000 7f8861b9b700 1 --2- 172.21.15.102:0/3868378189 >> v2:172.21.15.172:3300/0 conn(0x7f885c0c41b0 0x7f885c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.557+0000 7f8861b9b700 1 -- 172.21.15.102:0/3868378189 >> v1:172.21.15.102:6789/0 conn(0x7f885c0b3970 legacy=0x7f885c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.557+0000 7f8860b99700 1 -- 172.21.15.102:0/3868378189 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850543209 0 0) 0x7f885c0c3d20 con 0x7f885c0b3970 2023-03-03T17:12:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.557+0000 7f8861b9b700 1 -- 172.21.15.102:0/3868378189 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c15cba0 con 0x7f885c0ced00 2023-03-03T17:12:19.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.557+0000 7f8861b9b700 1 --2- 172.21.15.102:0/3868378189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0ced00 0x7f885c0c5a90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ca080 tx=0x7f8858009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda856fb772ea6ee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:19.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.557+0000 7f8860b99700 1 -- 172.21.15.102:0/3868378189 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858011830 con 0x7f885c0ced00 2023-03-03T17:12:19.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8860b99700 1 -- 172.21.15.102:0/3868378189 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8858011990 con 0x7f885c0ced00 2023-03-03T17:12:19.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8860b99700 1 -- 172.21.15.102:0/3868378189 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858011d20 con 0x7f885c0ced00 2023-03-03T17:12:19.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 -- 172.21.15.102:0/3868378189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0ced00 msgr2=0x7f885c0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:19.567 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 --2- 172.21.15.102:0/3868378189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0ced00 0x7f885c0c5a90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ca080 tx=0x7f8858009c00 comp rx=0 tx=0).stop 2023-03-03T17:12:19.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 -- 172.21.15.102:0/3868378189 shutdown_connections 2023-03-03T17:12:19.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 --2- 172.21.15.102:0/3868378189 >> v2:172.21.15.172:3300/0 conn(0x7f885c0c41b0 0x7f885c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 --2- 172.21.15.102:0/3868378189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0ced00 0x7f885c0c5a90 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.568 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 -- 172.21.15.102:0/3868378189 >> 172.21.15.102:0/3868378189 conn(0x7f885c08b1c0 msgr2=0x7f885c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:12:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 -- 172.21.15.102:0/3868378189 shutdown_connections 2023-03-03T17:12:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.558+0000 7f8862b9d700 1 -- 172.21.15.102:0/3868378189 wait complete. 2023-03-03T17:12:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.560+0000 7f8862b9d700 1 Processor -- start 2023-03-03T17:12:19.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.560+0000 7f8862b9d700 1 -- start start 2023-03-03T17:12:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.561+0000 7f8862b9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f885c0b3970 0x7f885c0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.561+0000 7f8862b9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0c41b0 0x7f885c0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.561+0000 7f8862b9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f885c0ced00 0x7f885c14d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.561+0000 7f8862b9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0c8460 con 0x7f885c0c41b0 2023-03-03T17:12:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.561+0000 7f8862b9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0b2540 con 0x7f885c0ced00 2023-03-03T17:12:19.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.561+0000 7f8862b9d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0b1f30 con 0x7f885c0b3970 2023-03-03T17:12:19.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f885c0b3970 0x7f885c0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:19.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f886139a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0c41b0 0x7f885c0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:19.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f886239c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f885c0ced00 0x7f885c14d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f885c0b3970 0x7f885c0c6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:40508/0 (socket says 172.21.15.102:40508) 2023-03-03T17:12:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 -- 172.21.15.102:0/1154281721 learned_addr learned my addr 172.21.15.102:0/1154281721 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:12:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 -- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f885c0ced00 msgr2=0x7f885c14d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f885c0ced00 0x7f885c14d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.573 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 -- 172.21.15.102:0/1154281721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0c41b0 msgr2=0x7f885c0c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:19.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0c41b0 0x7f885c0c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f8861b9b700 1 -- 172.21.15.102:0/1154281721 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8858009550 con 0x7f885c0b3970 2023-03-03T17:12:19.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f886239c700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f885c0ced00 0x7f885c14d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:12:19.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.562+0000 7f886139a700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0c41b0 0x7f885c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:12:19.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.563+0000 7f8861b9b700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f885c0b3970 0x7f885c0c6070 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8858009870 tx=0x7f8858004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:19.574 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.563+0000 7f88527fc700 1 -- 172.21.15.102:0/1154281721 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88580036a0 con 0x7f885c0b3970 2023-03-03T17:12:19.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.563+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c06bf20 con 0x7f885c0b3970 2023-03-03T17:12:19.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.563+0000 7f88527fc700 1 -- 172.21.15.102:0/1154281721 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8858003820 con 0x7f885c0b3970 2023-03-03T17:12:19.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.563+0000 7f88527fc700 1 -- 172.21.15.102:0/1154281721 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858003bd0 con 0x7f885c0b3970 2023-03-03T17:12:19.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.563+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c15ce80 con 0x7f885c0b3970 2023-03-03T17:12:19.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.564+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885c06c200 con 0x7f885c0b3970 2023-03-03T17:12:19.575 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.566+0000 7f88527fc700 1 -- 172.21.15.102:0/1154281721 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f885800f050 con 0x7f885c0b3970 2023-03-03T17:12:19.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.567+0000 7f88527fc700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f88400682f0 0x7f884006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:19.576 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.567+0000 7f88527fc700 1 -- 172.21.15.102:0/1154281721 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f88580875a0 con 0x7f885c0b3970 2023-03-03T17:12:19.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.567+0000 7f886139a700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f88400682f0 0x7f884006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:19.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.568+0000 7f886139a700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f88400682f0 0x7f884006a7a0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ca220 tx=0x7f884c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:19.577 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.572+0000 7f88527fc700 1 -- 172.21.15.102:0/1154281721 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8858053c30 con 0x7f885c0b3970 2023-03-03T17:12:19.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.805+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f885c0c9ef0 con 0x7f885c0b3970 2023-03-03T17:12:19.807 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.806+0000 7f88527fc700 1 -- 172.21.15.102:0/1154281721 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f8858057250 con 0x7f885c0b3970 2023-03-03T17:12:19.808 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:12:19.808 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:12:19.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.808+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f88400682f0 msgr2=0x7f884006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:19.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.808+0000 7f8862b9d700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f88400682f0 0x7f884006a7a0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ca220 tx=0x7f884c009040 comp rx=0 tx=0).stop 2023-03-03T17:12:19.809 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.808+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f885c0b3970 msgr2=0x7f885c0c6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:19.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.808+0000 7f8862b9d700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f885c0b3970 0x7f885c0c6070 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8858009870 tx=0x7f8858004750 comp rx=0 tx=0).stop 2023-03-03T17:12:19.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 shutdown_connections 2023-03-03T17:12:19.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f88400682f0 0x7f884006a7a0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f885c0b3970 0x7f885c0c6070 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f885c0ced00 0x7f885c14d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 --2- 172.21.15.102:0/1154281721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f885c0c41b0 0x7f885c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:19.810 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 >> 172.21.15.102:0/1154281721 conn(0x7f885c08b1c0 msgr2=0x7f885c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:12:19.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 shutdown_connections 2023-03-03T17:12:19.811 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:12:19.809+0000 7f8862b9d700 1 -- 172.21.15.102:0/1154281721 wait complete. 2023-03-03T17:12:19.811 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:12:19.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-03-03T17:12:37.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:37.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:07.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:07.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:37.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:37.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:46.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:13:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.096+0000 7fbb77dff700 1 Processor -- start 2023-03-03T17:13:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.096+0000 7fbb77dff700 1 -- start start 2023-03-03T17:13:47.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.097+0000 7fbb77dff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb70153290 0x7fbb70158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.097+0000 7fbb77dff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb701593c0 0x7fbb700cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.097+0000 7fbb77dff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70153660 con 0x7fbb700cd8e0 2023-03-03T17:13:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.097+0000 7fbb77dff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7006c360 con 0x7fbb701593c0 2023-03-03T17:13:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.097+0000 7fbb77dff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7015cea0 con 0x7fbb70153290 2023-03-03T17:13:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb75b9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb70153290 0x7fbb70158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb7639c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbb700cd8e0 0x7fbb700ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:56286/0 (socket says 172.21.15.102:56286) 2023-03-03T17:13:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb7639c700 1 -- 172.21.15.102:0/316841022 learned_addr learned my addr 172.21.15.102:0/316841022 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:13:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb7539a700 1 --2- 172.21.15.102:0/316841022 >> v2:172.21.15.102:3300/0 conn(0x7fbb701593c0 0x7fbb700cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb74b99700 1 -- 172.21.15.102:0/316841022 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534194823 0 0) 0x7fbb70153660 con 0x7fbb700cd8e0 2023-03-03T17:13:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb74b99700 1 -- 172.21.15.102:0/316841022 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c0035b0 con 0x7fbb700cd8e0 2023-03-03T17:13:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb75b9b700 1 -- 172.21.15.102:0/316841022 >> v2:172.21.15.102:3300/0 conn(0x7fbb701593c0 msgr2=0x7fbb700cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:13:47.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb75b9b700 1 --2- 172.21.15.102:0/316841022 >> v2:172.21.15.102:3300/0 conn(0x7fbb701593c0 0x7fbb700cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb75b9b700 1 -- 172.21.15.102:0/316841022 >> v1:172.21.15.172:6789/0 conn(0x7fbb700cd8e0 legacy=0x7fbb700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.098+0000 7fbb75b9b700 1 -- 172.21.15.102:0/316841022 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7006c7f0 con 0x7fbb70153290 2023-03-03T17:13:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb75b9b700 1 --2- 172.21.15.102:0/316841022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb70153290 0x7fbb70158e80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbb7008cc70 tx=0x7fbb6c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38cb4c62f2db7b2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb74b99700 1 -- 172.21.15.102:0/316841022 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c011830 con 0x7fbb70153290 2023-03-03T17:13:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb74b99700 1 -- 172.21.15.102:0/316841022 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c011990 con 0x7fbb70153290 2023-03-03T17:13:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb74b99700 1 -- 172.21.15.102:0/316841022 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c011d20 con 0x7fbb70153290 2023-03-03T17:13:47.104 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb77dff700 1 -- 172.21.15.102:0/316841022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb70153290 msgr2=0x7fbb70158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb77dff700 1 --2- 172.21.15.102:0/316841022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb70153290 0x7fbb70158e80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbb7008cc70 tx=0x7fbb6c009b20 comp rx=0 tx=0).stop 2023-03-03T17:13:47.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb77dff700 1 -- 172.21.15.102:0/316841022 shutdown_connections 2023-03-03T17:13:47.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb77dff700 1 --2- 172.21.15.102:0/316841022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb70153290 0x7fbb70158e80 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb77dff700 1 --2- 172.21.15.102:0/316841022 >> v2:172.21.15.102:3300/0 conn(0x7fbb701593c0 0x7fbb700cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.099+0000 7fbb77dff700 1 -- 172.21.15.102:0/316841022 >> 172.21.15.102:0/316841022 conn(0x7fbb70065fb0 msgr2=0x7fbb700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:47.105 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.100+0000 7fbb77dff700 1 -- 172.21.15.102:0/316841022 shutdown_connections 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.100+0000 7fbb77dff700 1 -- 172.21.15.102:0/316841022 wait complete. 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.101+0000 7fbb77dff700 1 Processor -- start 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.102+0000 7fbb77dff700 1 -- start start 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb77dff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb700cd8e0 0x7fbb700b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb77dff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb70153290 0x7fbb701eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb77dff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb701593c0 0x7fbb701ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb77dff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb701541c0 con 0x7fbb70153290 2023-03-03T17:13:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb77dff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb70153970 con 0x7fbb700cd8e0 2023-03-03T17:13:47.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb77dff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb700b3490 con 0x7fbb701593c0 2023-03-03T17:13:47.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb7539a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb70153290 0x7fbb701eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb7639c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb701593c0 0x7fbb701ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb7639c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb701593c0 0x7fbb701ef2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:48168/0 (socket says 172.21.15.102:48168) 2023-03-03T17:13:47.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb75b9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb700cd8e0 0x7fbb700b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb7539a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb70153290 0x7fbb701eed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42890/0 (socket says 172.21.15.102:42890) 2023-03-03T17:13:47.107 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.103+0000 7fbb7639c700 1 -- 172.21.15.102:0/1481859124 learned_addr learned my addr 172.21.15.102:0/1481859124 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:13:47.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb7539a700 1 -- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb701593c0 msgr2=0x7fbb701ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb7539a700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb701593c0 0x7fbb701ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb7539a700 1 -- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb700cd8e0 msgr2=0x7fbb700b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb7539a700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb700cd8e0 0x7fbb700b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.108 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb7539a700 1 -- 172.21.15.102:0/1481859124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c0094a0 con 0x7fbb70153290 2023-03-03T17:13:47.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb7539a700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb70153290 0x7fbb701eed80 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000e900 tx=0x7fbb6000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:47.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb667fc700 1 -- 172.21.15.102:0/1481859124 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60013680 con 0x7fbb70153290 2023-03-03T17:13:47.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb667fc700 1 -- 172.21.15.102:0/1481859124 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb600137e0 con 0x7fbb70153290 2023-03-03T17:13:47.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb667fc700 1 -- 172.21.15.102:0/1481859124 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60013b50 con 0x7fbb70153290 2023-03-03T17:13:47.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb700ccde0 con 0x7fbb70153290 2023-03-03T17:13:47.109 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.104+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7015d160 con 0x7fbb70153290 2023-03-03T17:13:47.110 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.106+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb700cd0c0 con 0x7fbb70153290 2023-03-03T17:13:47.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.107+0000 7fbb667fc700 1 -- 172.21.15.102:0/1481859124 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb60019030 con 0x7fbb70153290 2023-03-03T17:13:47.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.108+0000 7fbb667fc700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb54068320 0x7fbb5406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.108+0000 7fbb667fc700 1 -- 172.21.15.102:0/1481859124 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fbb60085f00 con 0x7fbb70153290 2023-03-03T17:13:47.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.108+0000 7fbb75b9b700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb54068320 0x7fbb5406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.111 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.109+0000 7fbb75b9b700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb54068320 0x7fbb5406a7d0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0090b0 tx=0x7fbb6c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:47.112 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.112+0000 7fbb667fc700 1 -- 172.21.15.102:0/1481859124 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb60052580 con 0x7fbb70153290 2023-03-03T17:13:47.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.346+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb7006b180 con 0x7fbb70153290 2023-03-03T17:13:47.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.347+0000 7fbb667fc700 1 -- 172.21.15.102:0/1481859124 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fbb60017070 con 0x7fbb70153290 2023-03-03T17:13:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:13:47.346 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:13:47.348 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb54068320 msgr2=0x7fbb5406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb54068320 0x7fbb5406a7d0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0090b0 tx=0x7fbb6c008030 comp rx=0 tx=0).stop 2023-03-03T17:13:47.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb70153290 msgr2=0x7fbb701eed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb70153290 0x7fbb701eed80 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000e900 tx=0x7fbb6000ec10 comp rx=0 tx=0).stop 2023-03-03T17:13:47.349 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 shutdown_connections 2023-03-03T17:13:47.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fbb54068320 0x7fbb5406a7d0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fbb701593c0 0x7fbb701ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb700cd8e0 0x7fbb700b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 --2- 172.21.15.102:0/1481859124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb70153290 0x7fbb701eed80 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 >> 172.21.15.102:0/1481859124 conn(0x7fbb70065fb0 msgr2=0x7fbb700cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:47.350 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 shutdown_connections 2023-03-03T17:13:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:13:47.349+0000 7fbb77dff700 1 -- 172.21.15.102:0/1481859124 wait complete. 2023-03-03T17:13:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:13:47.376 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2023-03-03T17:14:07.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:07.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:37.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:37.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:51.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:14:51.627 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.627+0000 7fd2a3bff700 1 Processor -- start 2023-03-03T17:14:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.627+0000 7fd2a3bff700 1 -- start start 2023-03-03T17:14:51.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a3bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd29c0ced80 0x7fd29c0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:51.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a3bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c1597f0 0x7fd29c0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:51.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a3bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd29c0cf9a0 0x7fd29c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:51.629 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a3bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c0ca750 con 0x7fd29c0ced80 2023-03-03T17:14:51.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a3bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c15a230 con 0x7fd29c0cf9a0 2023-03-03T17:14:51.630 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a3bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0cf690 con 0x7fd29c1597f0 2023-03-03T17:14:51.631 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a199b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd29c0ced80 0x7fd29c0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:51.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a199b700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd29c0ced80 0x7fd29c0cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38500/0 (socket says 172.21.15.102:38500) 2023-03-03T17:14:51.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a199b700 1 -- 172.21.15.102:0/2311069114 learned_addr learned my addr 172.21.15.102:0/2311069114 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:14:51.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a219c700 1 --2- 172.21.15.102:0/2311069114 >> v2:172.21.15.172:3300/0 conn(0x7fd29c0cf9a0 0x7fd29c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:51.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.629+0000 7fd2a119a700 1 --2- 172.21.15.102:0/2311069114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c1597f0 0x7fd29c0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:51.632 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a199b700 1 -- 172.21.15.102:0/2311069114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c1597f0 msgr2=0x7fd29c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:14:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a199b700 1 --2- 172.21.15.102:0/2311069114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c1597f0 0x7fd29c0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a199b700 1 -- 172.21.15.102:0/2311069114 >> v2:172.21.15.172:3300/0 conn(0x7fd29c0cf9a0 msgr2=0x7fd29c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:14:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a199b700 1 --2- 172.21.15.102:0/2311069114 >> v2:172.21.15.172:3300/0 conn(0x7fd29c0cf9a0 0x7fd29c15c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a199b700 1 -- 172.21.15.102:0/2311069114 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c0cb9f0 con 0x7fd29c0ced80 2023-03-03T17:14:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a219c700 1 --2- 172.21.15.102:0/2311069114 >> v2:172.21.15.172:3300/0 conn(0x7fd29c0cf9a0 0x7fd29c15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:14:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a199b700 1 --2- 172.21.15.102:0/2311069114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c0cf150 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fd29c1569f0 tx=0x7fd298009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143afccd086a50a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:14:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a0999700 1 -- 172.21.15.102:0/2311069114 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298011830 con 0x7fd29c0ced80 2023-03-03T17:14:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a0999700 1 -- 172.21.15.102:0/2311069114 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd298011990 con 0x7fd29c0ced80 2023-03-03T17:14:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.630+0000 7fd2a0999700 1 -- 172.21.15.102:0/2311069114 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298011d20 con 0x7fd29c0ced80 2023-03-03T17:14:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 -- 172.21.15.102:0/2311069114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 msgr2=0x7fd29c0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/2311069114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c0cf150 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fd29c1569f0 tx=0x7fd298009c00 comp rx=0 tx=0).stop 2023-03-03T17:14:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 -- 172.21.15.102:0/2311069114 shutdown_connections 2023-03-03T17:14:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/2311069114 >> v2:172.21.15.172:3300/0 conn(0x7fd29c0cf9a0 0x7fd29c15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/2311069114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c0cf150 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/2311069114 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c1597f0 0x7fd29c0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 -- 172.21.15.102:0/2311069114 >> 172.21.15.102:0/2311069114 conn(0x7fd29c065fb0 msgr2=0x7fd29c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:14:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 -- 172.21.15.102:0/2311069114 shutdown_connections 2023-03-03T17:14:51.635 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.631+0000 7fd2a3bff700 1 -- 172.21.15.102:0/2311069114 wait complete. 2023-03-03T17:14:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.633+0000 7fd2a3bff700 1 Processor -- start 2023-03-03T17:14:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.633+0000 7fd2a3bff700 1 -- start start 2023-03-03T17:14:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.634+0000 7fd2a3bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c1e4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.634+0000 7fd2a3bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c0cf9a0 0x7fd29c153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.634+0000 7fd2a3bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c1597f0 0x7fd29c14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.634+0000 7fd2a3bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c157a20 con 0x7fd29c0ced80 2023-03-03T17:14:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.634+0000 7fd2a3bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c157d00 con 0x7fd29c1597f0 2023-03-03T17:14:51.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.634+0000 7fd2a3bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c1e3d40 con 0x7fd29c0cf9a0 2023-03-03T17:14:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c1e4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c1e4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38506/0 (socket says 172.21.15.102:38506) 2023-03-03T17:14:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 -- 172.21.15.102:0/14382140 learned_addr learned my addr 172.21.15.102:0/14382140 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:14:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a219c700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c1597f0 0x7fd29c14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a119a700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c0cf9a0 0x7fd29c153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 -- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c0cf9a0 msgr2=0x7fd29c153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:51.640 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c0cf9a0 0x7fd29c153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 -- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c1597f0 msgr2=0x7fd29c14c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c1597f0 0x7fd29c14c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 -- 172.21.15.102:0/14382140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009a70 con 0x7fd29c0ced80 2023-03-03T17:14:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.635+0000 7fd2a199b700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c1e4820 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fd298000c00 tx=0x7fd298004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:14:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.636+0000 7fd2927fc700 1 -- 172.21.15.102:0/14382140 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298004450 con 0x7fd29c0ced80 2023-03-03T17:14:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.636+0000 7fd2927fc700 1 -- 172.21.15.102:0/14382140 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29801e070 con 0x7fd29c0ced80 2023-03-03T17:14:51.641 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.636+0000 7fd2927fc700 1 -- 172.21.15.102:0/14382140 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2980038e0 con 0x7fd29c0ced80 2023-03-03T17:14:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.636+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd298009550 con 0x7fd29c0ced80 2023-03-03T17:14:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.636+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c0cade0 con 0x7fd29c0ced80 2023-03-03T17:14:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.637+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c159c80 con 0x7fd29c0ced80 2023-03-03T17:14:51.642 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.638+0000 7fd2927fc700 1 -- 172.21.15.102:0/14382140 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd29800f050 con 0x7fd29c0ced80 2023-03-03T17:14:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.639+0000 7fd2927fc700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2800682f0 0x7fd28006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.639+0000 7fd2927fc700 1 -- 172.21.15.102:0/14382140 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fd298086b80 con 0x7fd29c0ced80 2023-03-03T17:14:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.640+0000 7fd2a119a700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2800682f0 0x7fd28006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.641+0000 7fd2a119a700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2800682f0 0x7fd28006a7a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd29c1e3c70 tx=0x7fd28c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:14:51.644 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.644+0000 7fd2927fc700 1 -- 172.21.15.102:0/14382140 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd298053290 con 0x7fd29c0ced80 2023-03-03T17:14:51.875 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.875+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd29c0cb0c0 con 0x7fd29c0ced80 2023-03-03T17:14:51.876 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.875+0000 7fd2927fc700 1 -- 172.21.15.102:0/14382140 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fd298015090 con 0x7fd29c0ced80 2023-03-03T17:14:51.876 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:14:51.877 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:14:51.878 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2800682f0 msgr2=0x7fd28006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2800682f0 0x7fd28006a7a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd29c1e3c70 tx=0x7fd28c009210 comp rx=0 tx=0).stop 2023-03-03T17:14:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 msgr2=0x7fd29c1e4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c1e4820 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fd298000c00 tx=0x7fd298004750 comp rx=0 tx=0).stop 2023-03-03T17:14:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 shutdown_connections 2023-03-03T17:14:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c1597f0 0x7fd29c14c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd29c0ced80 0x7fd29c1e4820 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fd2800682f0 0x7fd28006a7a0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 --2- 172.21.15.102:0/14382140 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fd29c0cf9a0 0x7fd29c153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 >> 172.21.15.102:0/14382140 conn(0x7fd29c065fb0 msgr2=0x7fd29c1582b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:14:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 shutdown_connections 2023-03-03T17:14:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:14:51.878+0000 7fd2a3bff700 1 -- 172.21.15.102:0/14382140 wait complete. 2023-03-03T17:14:51.881 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:14:51.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 74 secs before thrashing 2023-03-03T17:15:07.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:07.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:38.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:38.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:05.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:16:06.174 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.175+0000 7fcc1e9ff700 1 Processor -- start 2023-03-03T17:16:06.175 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.175+0000 7fcc1e9ff700 1 -- start start 2023-03-03T17:16:06.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1e9ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcc180cd8e0 0x7fcc180cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:06.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1e9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd390 0x7fcc180ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:06.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1e9ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc18153660 con 0x7fcc18153290 2023-03-03T17:16:06.176 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1e9ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1806c360 con 0x7fcc180cd8e0 2023-03-03T17:16:06.177 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1e9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1815cea0 con 0x7fcc180cd390 2023-03-03T17:16:06.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc177fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcc180cd8e0 0x7fcc180cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:06.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc17fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcc18153290 0x7fcc18158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:42314/0 (socket says 172.21.15.102:42314) 2023-03-03T17:16:06.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1cf9c700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd390 0x7fcc180ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:06.178 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc17fff700 1 -- 172.21.15.102:0/2866913209 learned_addr learned my addr 172.21.15.102:0/2866913209 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:16:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1cf9c700 1 -- 172.21.15.102:0/2866913209 >> v2:172.21.15.102:3300/0 conn(0x7fcc180cd8e0 msgr2=0x7fcc180cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:16:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc1cf9c700 1 --2- 172.21.15.102:0/2866913209 >> v2:172.21.15.102:3300/0 conn(0x7fcc180cd8e0 0x7fcc180cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.178+0000 7fcc1cf9c700 1 -- 172.21.15.102:0/2866913209 >> v1:172.21.15.172:6789/0 conn(0x7fcc18153290 legacy=0x7fcc18158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.178+0000 7fcc1cf9c700 1 -- 172.21.15.102:0/2866913209 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1806c7f0 con 0x7fcc180cd390 2023-03-03T17:16:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.177+0000 7fcc16ffd700 1 -- 172.21.15.102:0/2866913209 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290237467 0 0) 0x7fcc18153660 con 0x7fcc18153290 2023-03-03T17:16:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.178+0000 7fcc1cf9c700 1 --2- 172.21.15.102:0/2866913209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd390 0x7fcc180ce360 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009d90 tx=0x7fcc1000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd432a8fd4357113 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.178+0000 7fcc16ffd700 1 -- 172.21.15.102:0/2866913209 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000e070 con 0x7fcc180cd390 2023-03-03T17:16:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.178+0000 7fcc16ffd700 1 -- 172.21.15.102:0/2866913209 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1000bb20 con 0x7fcc180cd390 2023-03-03T17:16:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.178+0000 7fcc16ffd700 1 -- 172.21.15.102:0/2866913209 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10012430 con 0x7fcc180cd390 2023-03-03T17:16:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/2866913209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd390 msgr2=0x7fcc180ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/2866913209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd390 0x7fcc180ce360 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009d90 tx=0x7fcc1000b330 comp rx=0 tx=0).stop 2023-03-03T17:16:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/2866913209 shutdown_connections 2023-03-03T17:16:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/2866913209 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd390 0x7fcc180ce360 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/2866913209 >> v2:172.21.15.102:3300/0 conn(0x7fcc180cd8e0 0x7fcc180cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/2866913209 >> 172.21.15.102:0/2866913209 conn(0x7fcc18065fb0 msgr2=0x7fcc180663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:16:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/2866913209 shutdown_connections 2023-03-03T17:16:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.179+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/2866913209 wait complete. 2023-03-03T17:16:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.181+0000 7fcc1e9ff700 1 Processor -- start 2023-03-03T17:16:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.181+0000 7fcc1e9ff700 1 -- start start 2023-03-03T17:16:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc1e9ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc180cd390 0x7fcc181564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc1e9ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd8e0 0x7fcc180c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc1e9ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18153290 0x7fcc180c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc1e9ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18156a00 con 0x7fcc18153290 2023-03-03T17:16:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc1e9ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180c7e10 con 0x7fcc180cd390 2023-03-03T17:16:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc1e9ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180c86f0 con 0x7fcc180cd8e0 2023-03-03T17:16:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc17fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc180cd390 0x7fcc181564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc17fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc180cd390 0x7fcc181564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:56276/0 (socket says 172.21.15.102:56276) 2023-03-03T17:16:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc177fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd8e0 0x7fcc180c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc17fff700 1 -- 172.21.15.102:0/3607736218 learned_addr learned my addr 172.21.15.102:0/3607736218 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:16:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.182+0000 7fcc1cf9c700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18153290 0x7fcc180c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:06.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc17fff700 1 -- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd8e0 msgr2=0x7fcc180c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:06.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc17fff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd8e0 0x7fcc180c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc17fff700 1 -- 172.21.15.102:0/3607736218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18153290 msgr2=0x7fcc180c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:06.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc17fff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18153290 0x7fcc180c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc17fff700 1 -- 172.21.15.102:0/3607736218 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10009a70 con 0x7fcc180cd390 2023-03-03T17:16:06.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc177fe700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd8e0 0x7fcc180c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:16:06.185 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc17fff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc180cd390 0x7fcc181564c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcc080011e0 tx=0x7fcc0800ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc14ff9700 1 -- 172.21.15.102:0/3607736218 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08004c80 con 0x7fcc180cd390 2023-03-03T17:16:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc14ff9700 1 -- 172.21.15.102:0/3607736218 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08004e00 con 0x7fcc180cd390 2023-03-03T17:16:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc180ccde0 con 0x7fcc180cd390 2023-03-03T17:16:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.183+0000 7fcc14ff9700 1 -- 172.21.15.102:0/3607736218 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08007310 con 0x7fcc180cd390 2023-03-03T17:16:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.184+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1815d160 con 0x7fcc180cd390 2023-03-03T17:16:06.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.185+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc180cd0c0 con 0x7fcc180cd390 2023-03-03T17:16:06.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.186+0000 7fcc14ff9700 1 -- 172.21.15.102:0/3607736218 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcc0801a020 con 0x7fcc180cd390 2023-03-03T17:16:06.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.187+0000 7fcc14ff9700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc04068320 0x7fcc0406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:06.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.187+0000 7fcc14ff9700 1 -- 172.21.15.102:0/3607736218 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fcc08085930 con 0x7fcc180cd390 2023-03-03T17:16:06.187 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.188+0000 7fcc177fe700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc04068320 0x7fcc0406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.188+0000 7fcc177fe700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc04068320 0x7fcc0406a7d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fcc180c8f60 tx=0x7fcc0c009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:06.190 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.191+0000 7fcc14ff9700 1 -- 172.21.15.102:0/3607736218 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc080520f0 con 0x7fcc180cd390 2023-03-03T17:16:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.427+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc180c6d80 con 0x7fcc180cd390 2023-03-03T17:16:06.427 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.428+0000 7fcc14ff9700 1 -- 172.21.15.102:0/3607736218 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fcc08055710 con 0x7fcc180cd390 2023-03-03T17:16:06.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:16:06.428 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:16:06.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc04068320 msgr2=0x7fcc0406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:06.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc04068320 0x7fcc0406a7d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fcc180c8f60 tx=0x7fcc0c009210 comp rx=0 tx=0).stop 2023-03-03T17:16:06.430 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc180cd390 msgr2=0x7fcc181564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc180cd390 0x7fcc181564c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcc080011e0 tx=0x7fcc0800ca30 comp rx=0 tx=0).stop 2023-03-03T17:16:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 shutdown_connections 2023-03-03T17:16:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18153290 0x7fcc180c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc180cd390 0x7fcc181564c0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fcc04068320 0x7fcc0406a7d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 --2- 172.21.15.102:0/3607736218 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fcc180cd8e0 0x7fcc180c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:06.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 >> 172.21.15.102:0/3607736218 conn(0x7fcc18065fb0 msgr2=0x7fcc18150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:16:06.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 shutdown_connections 2023-03-03T17:16:06.432 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:06.431+0000 7fcc1e9ff700 1 -- 172.21.15.102:0/3607736218 wait complete. 2023-03-03T17:16:06.432 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:16:06.457 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-03-03T17:16:08.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:08.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:38.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:38.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:45.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:16:45.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.715+0000 7f9f221ff700 1 Processor -- start 2023-03-03T17:16:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.715+0000 7f9f221ff700 1 -- start start 2023-03-03T17:16:45.715 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.716+0000 7f9f221ff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f1c0cd8e0 0x7f9f1c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.716+0000 7f9f221ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f1c0cd390 0x7f9f1c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.716+0000 7f9f221ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c153660 con 0x7f9f1c153290 2023-03-03T17:16:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.716+0000 7f9f221ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c06c360 con 0x7f9f1c0cd390 2023-03-03T17:16:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.716+0000 7f9f221ff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c15cea0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1bfff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f1c0cd390 0x7f9f1c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1affd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9f1c0cd8e0 0x7f9f1c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:45.717 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1b7fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9f1c153290 0x7f9f1c158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:40216/0 (socket says 172.21.15.102:40216) 2023-03-03T17:16:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1b7fe700 1 -- 172.21.15.102:0/1810939184 learned_addr learned my addr 172.21.15.102:0/1810939184 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:16:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1a7fc700 1 -- 172.21.15.102:0/1810939184 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004414642 0 0) 0x7f9f1c153660 con 0x7f9f1c153290 2023-03-03T17:16:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1a7fc700 1 -- 172.21.15.102:0/1810939184 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f040035b0 con 0x7f9f1c153290 2023-03-03T17:16:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1bfff700 1 -- 172.21.15.102:0/1810939184 >> v2:172.21.15.172:3300/0 conn(0x7f9f1c0cd8e0 msgr2=0x7f9f1c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:16:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1bfff700 1 --2- 172.21.15.102:0/1810939184 >> v2:172.21.15.172:3300/0 conn(0x7f9f1c0cd8e0 0x7f9f1c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1bfff700 1 -- 172.21.15.102:0/1810939184 >> v1:172.21.15.172:6789/0 conn(0x7f9f1c153290 legacy=0x7f9f1c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:45.718 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1bfff700 1 -- 172.21.15.102:0/1810939184 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c06c7f0 con 0x7f9f1c0cd390 2023-03-03T17:16:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.717+0000 7f9f1affd700 1 --2- 172.21.15.102:0/1810939184 >> v2:172.21.15.172:3300/0 conn(0x7f9f1c0cd8e0 0x7f9f1c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:16:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.718+0000 7f9f1bfff700 1 --2- 172.21.15.102:0/1810939184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c0cd390 0x7f9f1c0ce360 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c159fb0 tx=0x7f9f1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5413600a3bd269 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.718+0000 7f9f1a7fc700 1 -- 172.21.15.102:0/1810939184 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1400e070 con 0x7f9f1c0cd390 2023-03-03T17:16:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.718+0000 7f9f1a7fc700 1 -- 172.21.15.102:0/1810939184 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1400bb20 con 0x7f9f1c0cd390 2023-03-03T17:16:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.718+0000 7f9f1a7fc700 1 -- 172.21.15.102:0/1810939184 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14012430 con 0x7f9f1c0cd390 2023-03-03T17:16:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.718+0000 7f9f221ff700 1 -- 172.21.15.102:0/1810939184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c0cd390 msgr2=0x7f9f1c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.718+0000 7f9f221ff700 1 --2- 172.21.15.102:0/1810939184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c0cd390 0x7f9f1c0ce360 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c159fb0 tx=0x7f9f1400b330 comp rx=0 tx=0).stop 2023-03-03T17:16:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.719+0000 7f9f221ff700 1 -- 172.21.15.102:0/1810939184 shutdown_connections 2023-03-03T17:16:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.719+0000 7f9f221ff700 1 --2- 172.21.15.102:0/1810939184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c0cd390 0x7f9f1c0ce360 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.719+0000 7f9f221ff700 1 --2- 172.21.15.102:0/1810939184 >> v2:172.21.15.172:3300/0 conn(0x7f9f1c0cd8e0 0x7f9f1c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.719+0000 7f9f221ff700 1 -- 172.21.15.102:0/1810939184 >> 172.21.15.102:0/1810939184 conn(0x7f9f1c065fb0 msgr2=0x7f9f1c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:16:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.719+0000 7f9f221ff700 1 -- 172.21.15.102:0/1810939184 shutdown_connections 2023-03-03T17:16:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.719+0000 7f9f221ff700 1 -- 172.21.15.102:0/1810939184 wait complete. 2023-03-03T17:16:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.721+0000 7f9f221ff700 1 Processor -- start 2023-03-03T17:16:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.721+0000 7f9f221ff700 1 -- start start 2023-03-03T17:16:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f221ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f1c0cd390 0x7f9f1c1e2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f221ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f1c0cd8e0 0x7f9f1c1e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f221ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c153290 0x7f9f1c14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:45.723 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f221ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0b25b0 con 0x7f9f1c153290 2023-03-03T17:16:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f221ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0b20a0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f221ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c1e22c0 con 0x7f9f1c0cd390 2023-03-03T17:16:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f1bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c153290 0x7f9f1c14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.722+0000 7f9f1b7fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f1c0cd390 0x7f9f1c1e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f1c0cd8e0 0x7f9f1c1e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f1c0cd8e0 0x7f9f1c1e2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55570/0 (socket says 172.21.15.102:55570) 2023-03-03T17:16:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 -- 172.21.15.102:0/2722688708 learned_addr learned my addr 172.21.15.102:0/2722688708 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:16:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 -- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f1c0cd390 msgr2=0x7f9f1c1e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f1c0cd390 0x7f9f1c1e2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 -- 172.21.15.102:0/2722688708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c153290 msgr2=0x7f9f1c14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c153290 0x7f9f1c14e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 -- 172.21.15.102:0/2722688708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f100094a0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f1affd700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f1c0cd8e0 0x7f9f1c1e2c90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000ebe0 tx=0x7f9f1000eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.723+0000 7f9f03fff700 1 -- 172.21.15.102:0/2722688708 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000c850 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.724+0000 7f9f03fff700 1 -- 172.21.15.102:0/2722688708 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1000c9b0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.724+0000 7f9f03fff700 1 -- 172.21.15.102:0/2722688708 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000cd40 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.724+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f14009a70 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.724+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c0ccde0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.725+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c15d160 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.726+0000 7f9f03fff700 1 -- 172.21.15.102:0/2722688708 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9f10019030 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.727+0000 7f9f03fff700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f080682f0 0x7f9f0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.727+0000 7f9f03fff700 1 -- 172.21.15.102:0/2722688708 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f9f10085cf0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.727 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.728+0000 7f9f1b7fe700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f080682f0 0x7f9f0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:45.728 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.728+0000 7f9f1b7fe700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f080682f0 0x7f9f0806a7a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c007fd0 tx=0x7f9f0c007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:45.730 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.731+0000 7f9f03fff700 1 -- 172.21.15.102:0/2722688708 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9f10052400 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.966 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.967+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f1c0cd0c0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.967 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.968+0000 7f9f03fff700 1 -- 172.21.15.102:0/2722688708 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f9f1000cea0 con 0x7f9f1c0cd8e0 2023-03-03T17:16:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:16:45.968 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:16:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f080682f0 msgr2=0x7f9f0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f080682f0 0x7f9f0806a7a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c007fd0 tx=0x7f9f0c007dc0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f1c0cd8e0 msgr2=0x7f9f1c1e2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f1c0cd8e0 0x7f9f1c1e2c90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000ebe0 tx=0x7f9f1000eef0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 shutdown_connections 2023-03-03T17:16:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f9f080682f0 0x7f9f0806a7a0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f9f1c0cd390 0x7f9f1c1e2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f1c153290 0x7f9f1c14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.971 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 --2- 172.21.15.102:0/2722688708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f1c0cd8e0 0x7f9f1c1e2c90 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 >> 172.21.15.102:0/2722688708 conn(0x7f9f1c065fb0 msgr2=0x7f9f1c158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:16:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 shutdown_connections 2023-03-03T17:16:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:45.971+0000 7f9f221ff700 1 -- 172.21.15.102:0/2722688708 wait complete. 2023-03-03T17:16:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:16:45.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:16:46.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.241+0000 7f1df3fff700 1 Processor -- start 2023-03-03T17:16:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.241+0000 7f1df3fff700 1 -- start start 2023-03-03T17:16:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df3fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1dec0ced80 0x7f1dec0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec1597f0 0x7f1dec0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df3fff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec0cb9f0 con 0x7f1dec0cf9a0 2023-03-03T17:16:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df3fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dec0ca750 con 0x7f1dec0ced80 2023-03-03T17:16:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec15a230 con 0x7f1dec1597f0 2023-03-03T17:16:46.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df1d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1dec0ced80 0x7f1dec0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df159a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec1597f0 0x7f1dec0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df1d9b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1dec0ced80 0x7f1dec0cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55580/0 (socket says 172.21.15.102:55580) 2023-03-03T17:16:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.243+0000 7f1df1d9b700 1 -- 172.21.15.102:0/302362132 learned_addr learned my addr 172.21.15.102:0/302362132 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:16:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df1d9b700 1 -- 172.21.15.102:0/302362132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec1597f0 msgr2=0x7f1dec0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:16:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df1d9b700 1 --2- 172.21.15.102:0/302362132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec1597f0 0x7f1dec0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df1d9b700 1 -- 172.21.15.102:0/302362132 >> v1:172.21.15.102:6789/0 conn(0x7f1dec0cf9a0 legacy=0x7f1dec15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df1d9b700 1 -- 172.21.15.102:0/302362132 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec159c80 con 0x7f1dec0ced80 2023-03-03T17:16:46.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df1d9b700 1 --2- 172.21.15.102:0/302362132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec0cf150 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098f0 tx=0x7f1de8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfc8d4a5b4e13be6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:46.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df0d99700 1 -- 172.21.15.102:0/302362132 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8011830 con 0x7f1dec0ced80 2023-03-03T17:16:46.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df0d99700 1 -- 172.21.15.102:0/302362132 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de8011990 con 0x7f1dec0ced80 2023-03-03T17:16:46.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.244+0000 7f1df0d99700 1 -- 172.21.15.102:0/302362132 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8011d20 con 0x7f1dec0ced80 2023-03-03T17:16:46.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 -- 172.21.15.102:0/302362132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 msgr2=0x7f1dec0cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:46.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 --2- 172.21.15.102:0/302362132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec0cf150 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098f0 tx=0x7f1de8009c00 comp rx=0 tx=0).stop 2023-03-03T17:16:46.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 -- 172.21.15.102:0/302362132 shutdown_connections 2023-03-03T17:16:46.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 --2- 172.21.15.102:0/302362132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec0cf150 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.246 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 --2- 172.21.15.102:0/302362132 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec1597f0 0x7f1dec0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 -- 172.21.15.102:0/302362132 >> 172.21.15.102:0/302362132 conn(0x7f1dec065fb0 msgr2=0x7f1dec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:16:46.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 -- 172.21.15.102:0/302362132 shutdown_connections 2023-03-03T17:16:46.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.245+0000 7f1df3fff700 1 -- 172.21.15.102:0/302362132 wait complete. 2023-03-03T17:16:46.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.247+0000 7f1df3fff700 1 Processor -- start 2023-03-03T17:16:46.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.247+0000 7f1df3fff700 1 -- start start 2023-03-03T17:16:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec0cf9a0 0x7f1dec14f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dec1597f0 0x7f1dec14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec14df10 con 0x7f1dec1597f0 2023-03-03T17:16:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df3fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec150cc0 con 0x7f1dec0ced80 2023-03-03T17:16:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df3fff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec150830 con 0x7f1dec0cf9a0 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df159a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec0cf9a0 0x7f1dec14f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df1d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dec1597f0 0x7f1dec14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df1d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec15a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:55584/0 (socket says 172.21.15.102:55584) 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.248+0000 7f1df1d9b700 1 -- 172.21.15.102:0/2891732715 learned_addr learned my addr 172.21.15.102:0/2891732715 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1df1d9b700 1 -- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec0cf9a0 msgr2=0x7f1dec14f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1df1d9b700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec0cf9a0 0x7f1dec14f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1df1d9b700 1 -- 172.21.15.102:0/2891732715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dec1597f0 msgr2=0x7f1dec14f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1df1d9b700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dec1597f0 0x7f1dec14f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1df1d9b700 1 -- 172.21.15.102:0/2891732715 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8009550 con 0x7f1dec0ced80 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1df1d9b700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec15a960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1de8014040 tx=0x7f1de8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1de27fc700 1 -- 172.21.15.102:0/2891732715 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8004510 con 0x7f1dec0ced80 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1de27fc700 1 -- 172.21.15.102:0/2891732715 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de801e070 con 0x7f1dec0ced80 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec0cb0c0 con 0x7f1dec0ced80 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.249+0000 7f1de27fc700 1 -- 172.21.15.102:0/2891732715 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80038e0 con 0x7f1dec0ced80 2023-03-03T17:16:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.250+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec0cade0 con 0x7f1dec0ced80 2023-03-03T17:16:46.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.250+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec0cb3a0 con 0x7f1dec0ced80 2023-03-03T17:16:46.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.252+0000 7f1de27fc700 1 -- 172.21.15.102:0/2891732715 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1de800f050 con 0x7f1dec0ced80 2023-03-03T17:16:46.252 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.253+0000 7f1de27fc700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1dd00682f0 0x7f1dd006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:16:46.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.253+0000 7f1de27fc700 1 -- 172.21.15.102:0/2891732715 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1de8087680 con 0x7f1dec0ced80 2023-03-03T17:16:46.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.253+0000 7f1df159a700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1dd00682f0 0x7f1dd006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:16:46.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.254+0000 7f1df159a700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1dd00682f0 0x7f1dd006a7a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f1dec14ed80 tx=0x7f1ddc009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:16:46.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.256+0000 7f1de27fc700 1 -- 172.21.15.102:0/2891732715 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1de8053d10 con 0x7f1dec0ced80 2023-03-03T17:16:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.493+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dec0677f0 con 0x7f1dec0ced80 2023-03-03T17:16:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.494+0000 7f1de27fc700 1 -- 172.21.15.102:0/2891732715 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1de8057330 con 0x7f1dec0ced80 2023-03-03T17:16:46.494 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:16:46.494 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:16:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1dd00682f0 msgr2=0x7f1dd006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1dd00682f0 0x7f1dd006a7a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f1dec14ed80 tx=0x7f1ddc009210 comp rx=0 tx=0).stop 2023-03-03T17:16:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 msgr2=0x7f1dec15a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:16:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec15a960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1de8014040 tx=0x7f1de8004750 comp rx=0 tx=0).stop 2023-03-03T17:16:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 shutdown_connections 2023-03-03T17:16:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dec1597f0 0x7f1dec14f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1dec0ced80 0x7f1dec15a960 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1dd00682f0 0x7f1dd006a7a0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 --2- 172.21.15.102:0/2891732715 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1dec0cf9a0 0x7f1dec14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:16:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 >> 172.21.15.102:0/2891732715 conn(0x7f1dec065fb0 msgr2=0x7f1dec0cbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:16:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 shutdown_connections 2023-03-03T17:16:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:16:46.497+0000 7f1df3fff700 1 -- 172.21.15.102:0/2891732715 wait complete. 2023-03-03T17:16:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:16:46.524 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-03-03T17:17:08.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:08.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:38.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:38.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:08.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:08.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:30.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:18:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.776+0000 7f3c10bff700 1 Processor -- start 2023-03-03T17:18:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.776+0000 7f3c10bff700 1 -- start start 2023-03-03T17:18:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c10bff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c0c153290 0x7f3c0c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c10bff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3c0c1593c0 0x7f3c0c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c10bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c0cd8e0 0x7f3c0c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:30.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c10bff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c153660 con 0x7f3c0c153290 2023-03-03T17:18:30.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c10bff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c06c360 con 0x7f3c0c1593c0 2023-03-03T17:18:30.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c10bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c15cea0 con 0x7f3c0c0cd8e0 2023-03-03T17:18:30.778 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c0a59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c0c153290 0x7f3c0c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c0ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c0cd8e0 0x7f3c0c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c0a59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c0c153290 0x7f3c0c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60762/0 (socket says 172.21.15.102:60762) 2023-03-03T17:18:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c0a59c700 1 -- 172.21.15.102:0/3997891805 learned_addr learned my addr 172.21.15.102:0/3997891805 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:18:30.779 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.778+0000 7f3c09d9b700 1 --2- 172.21.15.102:0/3997891805 >> v2:172.21.15.172:3300/0 conn(0x7f3c0c1593c0 0x7f3c0c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:30.780 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0ad9d700 1 -- 172.21.15.102:0/3997891805 >> v2:172.21.15.172:3300/0 conn(0x7f3c0c1593c0 msgr2=0x7f3c0c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:18:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0ad9d700 1 --2- 172.21.15.102:0/3997891805 >> v2:172.21.15.172:3300/0 conn(0x7f3c0c1593c0 0x7f3c0c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0ad9d700 1 -- 172.21.15.102:0/3997891805 >> v2:172.21.15.102:3300/0 conn(0x7f3c0c153290 msgr2=0x7f3c0c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:18:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0ad9d700 1 --2- 172.21.15.102:0/3997891805 >> v2:172.21.15.102:3300/0 conn(0x7f3c0c153290 0x7f3c0c158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0ad9d700 1 -- 172.21.15.102:0/3997891805 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c06c7f0 con 0x7f3c0c0cd8e0 2023-03-03T17:18:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0a59c700 1 --2- 172.21.15.102:0/3997891805 >> v2:172.21.15.102:3300/0 conn(0x7f3c0c153290 0x7f3c0c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:18:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c09d9b700 1 --2- 172.21.15.102:0/3997891805 >> v2:172.21.15.172:3300/0 conn(0x7f3c0c1593c0 0x7f3c0c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:18:30.781 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0ad9d700 1 --2- 172.21.15.102:0/3997891805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c0cd8e0 0x7f3c0c0ce360 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c159fb0 tx=0x7f3c0400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=138742385bfc6384 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:18:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0959a700 1 -- 172.21.15.102:0/3997891805 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400e070 con 0x7f3c0c0cd8e0 2023-03-03T17:18:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0959a700 1 -- 172.21.15.102:0/3997891805 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0400bb20 con 0x7f3c0c0cd8e0 2023-03-03T17:18:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.779+0000 7f3c0959a700 1 -- 172.21.15.102:0/3997891805 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04012430 con 0x7f3c0c0cd8e0 2023-03-03T17:18:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 -- 172.21.15.102:0/3997891805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c0cd8e0 msgr2=0x7f3c0c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 --2- 172.21.15.102:0/3997891805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c0cd8e0 0x7f3c0c0ce360 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c159fb0 tx=0x7f3c0400b330 comp rx=0 tx=0).stop 2023-03-03T17:18:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 -- 172.21.15.102:0/3997891805 shutdown_connections 2023-03-03T17:18:30.782 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 --2- 172.21.15.102:0/3997891805 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c0cd8e0 0x7f3c0c0ce360 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 --2- 172.21.15.102:0/3997891805 >> v2:172.21.15.102:3300/0 conn(0x7f3c0c153290 0x7f3c0c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 --2- 172.21.15.102:0/3997891805 >> v2:172.21.15.172:3300/0 conn(0x7f3c0c1593c0 0x7f3c0c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 -- 172.21.15.102:0/3997891805 >> 172.21.15.102:0/3997891805 conn(0x7f3c0c065fb0 msgr2=0x7f3c0c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:18:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 -- 172.21.15.102:0/3997891805 shutdown_connections 2023-03-03T17:18:30.783 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.780+0000 7f3c10bff700 1 -- 172.21.15.102:0/3997891805 wait complete. 2023-03-03T17:18:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.782+0000 7f3c10bff700 1 Processor -- start 2023-03-03T17:18:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.782+0000 7f3c10bff700 1 -- start start 2023-03-03T17:18:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.783+0000 7f3c10bff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c0c0cd8e0 0x7f3c0c1e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:30.785 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.783+0000 7f3c10bff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c0c153290 0x7f3c0c1f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.783+0000 7f3c10bff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c1593c0 0x7f3c0c0c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.783+0000 7f3c10bff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c15a4f0 con 0x7f3c0c153290 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.783+0000 7f3c10bff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0b23e0 con 0x7f3c0c0cd8e0 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.783+0000 7f3c10bff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0b1c70 con 0x7f3c0c1593c0 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c0c0cd8e0 0x7f3c0c1e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c1593c0 0x7f3c0c0c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c09d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c0c153290 0x7f3c0c1f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:30.786 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c1593c0 0x7f3c0c0c7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:55162/0 (socket says 172.21.15.102:55162) 2023-03-03T17:18:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 -- 172.21.15.102:0/180019513 learned_addr learned my addr 172.21.15.102:0/180019513 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:18:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c0c0cd8e0 0x7f3c0c1e4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:49586/0 (socket says 172.21.15.102:49586) 2023-03-03T17:18:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 -- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c0c0cd8e0 msgr2=0x7f3c0c1e4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c0c0cd8e0 0x7f3c0c1e4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 -- 172.21.15.102:0/180019513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c0c153290 msgr2=0x7f3c0c1f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c0c153290 0x7f3c0c1f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:30.787 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 -- 172.21.15.102:0/180019513 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc009550 con 0x7f3c0c1593c0 2023-03-03T17:18:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.784+0000 7f3c0ad9d700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c1593c0 0x7f3c0c0c7800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c15cfe0 tx=0x7f3c04004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:18:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.785+0000 7f3bfaffd700 1 -- 172.21.15.102:0/180019513 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04004800 con 0x7f3c0c1593c0 2023-03-03T17:18:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.785+0000 7f3bfaffd700 1 -- 172.21.15.102:0/180019513 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c04020070 con 0x7f3c0c1593c0 2023-03-03T17:18:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.785+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c000094a0 con 0x7f3c0c1593c0 2023-03-03T17:18:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.785+0000 7f3bfaffd700 1 -- 172.21.15.102:0/180019513 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c040276c0 con 0x7f3c0c1593c0 2023-03-03T17:18:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.785+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c04009a70 con 0x7f3c0c1593c0 2023-03-03T17:18:30.788 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.786+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c0ccde0 con 0x7f3c0c1593c0 2023-03-03T17:18:30.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.787+0000 7f3bfaffd700 1 -- 172.21.15.102:0/180019513 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c04027820 con 0x7f3c0c1593c0 2023-03-03T17:18:30.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.788+0000 7f3bfaffd700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bec0682f0 0x7f3bec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:30.789 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.789+0000 7f3bfaffd700 1 -- 172.21.15.102:0/180019513 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3c04022070 con 0x7f3c0c1593c0 2023-03-03T17:18:30.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.789+0000 7f3c0a59c700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bec0682f0 0x7f3bec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:30.790 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.789+0000 7f3c0a59c700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bec0682f0 0x7f3bec06a7a0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc006d30 tx=0x7f3bfc0092c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:18:30.791 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:30.792+0000 7f3bfaffd700 1 -- 172.21.15.102:0/180019513 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c04054950 con 0x7f3c0c1593c0 2023-03-03T17:18:31.023 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.023+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c0c0cd0c0 con 0x7f3c0c1593c0 2023-03-03T17:18:31.024 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.024+0000 7f3bfaffd700 1 -- 172.21.15.102:0/180019513 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f3c04057f70 con 0x7f3c0c1593c0 2023-03-03T17:18:31.024 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:18:31.024 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:18:31.026 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bec0682f0 msgr2=0x7f3bec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bec0682f0 0x7f3bec06a7a0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc006d30 tx=0x7f3bfc0092c0 comp rx=0 tx=0).stop 2023-03-03T17:18:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c1593c0 msgr2=0x7f3c0c0c7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c1593c0 0x7f3c0c0c7800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c15cfe0 tx=0x7f3c04004ad0 comp rx=0 tx=0).stop 2023-03-03T17:18:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 shutdown_connections 2023-03-03T17:18:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f3bec0682f0 0x7f3bec06a7a0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f3c0c1593c0 0x7f3c0c0c7800 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:31.027 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c0c0cd8e0 0x7f3c0c1e4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 --2- 172.21.15.102:0/180019513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c0c153290 0x7f3c0c1f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 >> 172.21.15.102:0/180019513 conn(0x7f3c0c065fb0 msgr2=0x7f3c0c15ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:18:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 shutdown_connections 2023-03-03T17:18:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:18:31.027+0000 7f3c10bff700 1 -- 172.21.15.102:0/180019513 wait complete. 2023-03-03T17:18:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:18:31.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-03-03T17:18:38.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:38.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:09.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:09.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:23.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:19:23.312 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.312+0000 7f393ffff700 1 Processor -- start 2023-03-03T17:19:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.312+0000 7f393ffff700 1 -- start start 2023-03-03T17:19:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.314+0000 7f393ffff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f39400c5a90 0x7f39400c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.314+0000 7f393ffff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400c3d20 con 0x7f39400b3970 2023-03-03T17:19:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.314+0000 7f393ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394015cba0 con 0x7f39400ced00 2023-03-03T17:19:23.315 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.314+0000 7f393ffff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39400c4930 con 0x7f39400c5a90 2023-03-03T17:19:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.314+0000 7f393effd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f39400ced00 0x7f3940158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:50314/0 (socket says 172.21.15.102:50314) 2023-03-03T17:19:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.314+0000 7f393f7fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f39400c5a90 0x7f39400c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.314+0000 7f393effd700 1 -- 172.21.15.102:0/961405568 learned_addr learned my addr 172.21.15.102:0/961405568 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:19:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.315+0000 7f393dffb700 1 -- 172.21.15.102:0/961405568 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561957658 0 0) 0x7f394015cba0 con 0x7f39400ced00 2023-03-03T17:19:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.315+0000 7f393dffb700 1 -- 172.21.15.102:0/961405568 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39280035b0 con 0x7f39400ced00 2023-03-03T17:19:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.315+0000 7f393f7fe700 1 -- 172.21.15.102:0/961405568 >> v1:172.21.15.172:6789/0 conn(0x7f39400ced00 legacy=0x7f3940158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.315+0000 7f393f7fe700 1 -- 172.21.15.102:0/961405568 >> v1:172.21.15.102:6789/0 conn(0x7f39400b3970 legacy=0x7f3940161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.315+0000 7f393f7fe700 1 -- 172.21.15.102:0/961405568 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394015ce80 con 0x7f39400c5a90 2023-03-03T17:19:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.315+0000 7f393f7fe700 1 --2- 172.21.15.102:0/961405568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400c5a90 0x7f39400c41b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3938009d90 tx=0x7f393800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a011ae53527556 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:19:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393dffb700 1 -- 172.21.15.102:0/961405568 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393800e070 con 0x7f39400c5a90 2023-03-03T17:19:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393dffb700 1 -- 172.21.15.102:0/961405568 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393800bb20 con 0x7f39400c5a90 2023-03-03T17:19:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393dffb700 1 -- 172.21.15.102:0/961405568 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938012430 con 0x7f39400c5a90 2023-03-03T17:19:23.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393ffff700 1 -- 172.21.15.102:0/961405568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400c5a90 msgr2=0x7f39400c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:23.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393ffff700 1 --2- 172.21.15.102:0/961405568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400c5a90 0x7f39400c41b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3938009d90 tx=0x7f393800b330 comp rx=0 tx=0).stop 2023-03-03T17:19:23.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393ffff700 1 -- 172.21.15.102:0/961405568 shutdown_connections 2023-03-03T17:19:23.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393ffff700 1 --2- 172.21.15.102:0/961405568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400c5a90 0x7f39400c41b0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393ffff700 1 -- 172.21.15.102:0/961405568 >> 172.21.15.102:0/961405568 conn(0x7f394008b1c0 msgr2=0x7f3940071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:19:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393ffff700 1 -- 172.21.15.102:0/961405568 shutdown_connections 2023-03-03T17:19:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.316+0000 7f393ffff700 1 -- 172.21.15.102:0/961405568 wait complete. 2023-03-03T17:19:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.318+0000 7f393ffff700 1 Processor -- start 2023-03-03T17:19:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.318+0000 7f393ffff700 1 -- start start 2023-03-03T17:19:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.319+0000 7f393ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400b3970 0x7f39400668f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.319+0000 7f393ffff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c5a90 0x7f3940150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.319+0000 7f393ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400ced00 0x7f39400c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.319+0000 7f393ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39400c7e00 con 0x7f39400b3970 2023-03-03T17:19:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.319+0000 7f393ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39400c76c0 con 0x7f39400ced00 2023-03-03T17:19:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.319+0000 7f393ffff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3940066e30 con 0x7f39400c5a90 2023-03-03T17:19:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393effd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400b3970 0x7f39400668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400ced00 0x7f39400c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400ced00 0x7f39400c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:45634/0 (socket says 172.21.15.102:45634) 2023-03-03T17:19:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 -- 172.21.15.102:0/2557035241 learned_addr learned my addr 172.21.15.102:0/2557035241 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:19:23.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393e7fc700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c5a90 0x7f3940150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:23.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 -- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c5a90 msgr2=0x7f3940150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c5a90 0x7f3940150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 -- 172.21.15.102:0/2557035241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400b3970 msgr2=0x7f39400668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400b3970 0x7f39400668f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393f7fe700 1 -- 172.21.15.102:0/2557035241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3938009a70 con 0x7f39400ced00 2023-03-03T17:19:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393effd700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400b3970 0x7f39400668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:19:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.320+0000 7f393e7fc700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c5a90 0x7f3940150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:19:23.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.321+0000 7f393f7fe700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400ced00 0x7f39400c8250 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3938014040 tx=0x7f3938004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:19:23.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.321+0000 7f3927fff700 1 -- 172.21.15.102:0/2557035241 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938012430 con 0x7f39400ced00 2023-03-03T17:19:23.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.321+0000 7f3927fff700 1 -- 172.21.15.102:0/2557035241 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3938020070 con 0x7f39400ced00 2023-03-03T17:19:23.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.321+0000 7f3927fff700 1 -- 172.21.15.102:0/2557035241 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938004280 con 0x7f39400ced00 2023-03-03T17:19:23.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.321+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394006c200 con 0x7f39400ced00 2023-03-03T17:19:23.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.321+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394006bf20 con 0x7f39400ced00 2023-03-03T17:19:23.324 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.322+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394015c890 con 0x7f39400ced00 2023-03-03T17:19:23.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.324+0000 7f3927fff700 1 -- 172.21.15.102:0/2557035241 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f393802e070 con 0x7f39400ced00 2023-03-03T17:19:23.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.325+0000 7f3927fff700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f392c0682f0 0x7f392c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:23.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.325+0000 7f3927fff700 1 -- 172.21.15.102:0/2557035241 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f3938022070 con 0x7f39400ced00 2023-03-03T17:19:23.325 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.325+0000 7f393effd700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f392c0682f0 0x7f392c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:23.326 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.325+0000 7f393effd700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f392c0682f0 0x7f392c06a7a0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f3930007fd0 tx=0x7f3930007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:19:23.329 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.329+0000 7f3927fff700 1 -- 172.21.15.102:0/2557035241 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39380541e0 con 0x7f39400ced00 2023-03-03T17:19:23.565 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.565+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39400c7f70 con 0x7f39400ced00 2023-03-03T17:19:23.566 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.566+0000 7f3927fff700 1 -- 172.21.15.102:0/2557035241 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f39400c7f70 con 0x7f39400ced00 2023-03-03T17:19:23.566 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:19:23.566 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:19:23.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f392c0682f0 msgr2=0x7f392c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:23.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f392c0682f0 0x7f392c06a7a0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f3930007fd0 tx=0x7f3930007dc0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.569 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400ced00 msgr2=0x7f39400c8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400ced00 0x7f39400c8250 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3938014040 tx=0x7f3938004b20 comp rx=0 tx=0).stop 2023-03-03T17:19:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 shutdown_connections 2023-03-03T17:19:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39400ced00 0x7f39400c8250 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39400b3970 0x7f39400668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f392c0682f0 0x7f392c06a7a0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 --2- 172.21.15.102:0/2557035241 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f39400c5a90 0x7f3940150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 >> 172.21.15.102:0/2557035241 conn(0x7f394008b1c0 msgr2=0x7f3940071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:19:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.568+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 shutdown_connections 2023-03-03T17:19:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:19:23.569+0000 7f393ffff700 1 -- 172.21.15.102:0/2557035241 wait complete. 2023-03-03T17:19:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:19:23.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 82 secs before thrashing 2023-03-03T17:19:39.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:39.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:59.999 INFO:tasks.ceph.mon.a.smithi102.stderr:2023-03-03T17:19:59.999+0000 7f93411f9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:20:09.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:09.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:39.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:39.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:45.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:20:45.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.847+0000 7f50995ff700 1 Processor -- start 2023-03-03T17:20:45.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.847+0000 7f50995ff700 1 -- start start 2023-03-03T17:20:45.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.848+0000 7f50995ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f50940c2020 0x7f509406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:45.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.848+0000 7f50995ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940c5390 con 0x7f5094150c80 2023-03-03T17:20:45.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.848+0000 7f50995ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940c4a20 con 0x7f50940c5f30 2023-03-03T17:20:45.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.848+0000 7f50995ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5094157ad0 con 0x7f50940c2020 2023-03-03T17:20:45.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f5092ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5094150c80 0x7f50940c5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45422/0 (socket says 172.21.15.102:45422) 2023-03-03T17:20:45.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f5092ffd700 1 -- 172.21.15.102:0/1734197442 learned_addr learned my addr 172.21.15.102:0/1734197442 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:20:45.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f508bfff700 1 --1- 172.21.15.102:0/1734197442 >> v1:172.21.15.172:6789/0 conn(0x7f50940c5f30 0x7f50940c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:59300/0 (socket says 172.21.15.102:59300) 2023-03-03T17:20:45.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f50937fe700 1 --2- 172.21.15.102:0/1734197442 >> v2:172.21.15.102:3300/0 conn(0x7f50940c2020 0x7f509406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:45.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f50927fc700 1 -- 172.21.15.102:0/1734197442 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148485693 0 0) 0x7f50940c4a20 con 0x7f50940c5f30 2023-03-03T17:20:45.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f50927fc700 1 -- 172.21.15.102:0/1734197442 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50800035b0 con 0x7f50940c5f30 2023-03-03T17:20:45.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f50927fc700 1 -- 172.21.15.102:0/1734197442 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347802081 0 0) 0x7f50940c5390 con 0x7f5094150c80 2023-03-03T17:20:45.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f50937fe700 1 -- 172.21.15.102:0/1734197442 >> v1:172.21.15.172:6789/0 conn(0x7f50940c5f30 legacy=0x7f50940c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f50937fe700 1 -- 172.21.15.102:0/1734197442 >> v1:172.21.15.102:6789/0 conn(0x7f5094150c80 legacy=0x7f50940c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.849+0000 7f50937fe700 1 -- 172.21.15.102:0/1734197442 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509406f010 con 0x7f50940c2020 2023-03-03T17:20:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.850+0000 7f50937fe700 1 --2- 172.21.15.102:0/1734197442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50940c2020 0x7f509406d100 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f508c009d90 tx=0x7f508c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d9133f279a7ad9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.850+0000 7f50927fc700 1 -- 172.21.15.102:0/1734197442 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c00e070 con 0x7f50940c2020 2023-03-03T17:20:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.850+0000 7f50927fc700 1 -- 172.21.15.102:0/1734197442 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c00bb20 con 0x7f50940c2020 2023-03-03T17:20:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.850+0000 7f50927fc700 1 -- 172.21.15.102:0/1734197442 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c00bed0 con 0x7f50940c2020 2023-03-03T17:20:45.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.850+0000 7f50995ff700 1 -- 172.21.15.102:0/1734197442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50940c2020 msgr2=0x7f509406d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:45.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.850+0000 7f50995ff700 1 --2- 172.21.15.102:0/1734197442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50940c2020 0x7f509406d100 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f508c009d90 tx=0x7f508c00b330 comp rx=0 tx=0).stop 2023-03-03T17:20:45.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.851+0000 7f50995ff700 1 -- 172.21.15.102:0/1734197442 shutdown_connections 2023-03-03T17:20:45.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.851+0000 7f50995ff700 1 --2- 172.21.15.102:0/1734197442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50940c2020 0x7f509406d100 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:45.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.851+0000 7f50995ff700 1 -- 172.21.15.102:0/1734197442 >> 172.21.15.102:0/1734197442 conn(0x7f509406af90 msgr2=0x7f509406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:20:45.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.851+0000 7f50995ff700 1 -- 172.21.15.102:0/1734197442 shutdown_connections 2023-03-03T17:20:45.852 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.851+0000 7f50995ff700 1 -- 172.21.15.102:0/1734197442 wait complete. 2023-03-03T17:20:45.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.853+0000 7f50995ff700 1 Processor -- start 2023-03-03T17:20:45.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.853+0000 7f50995ff700 1 -- start start 2023-03-03T17:20:45.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50995ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50940c2020 0x7f50940c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:45.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50995ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50940c5f30 0x7f5094149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:45.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50995ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094150c80 0x7f5094071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:45.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50995ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094072050 con 0x7f5094150c80 2023-03-03T17:20:45.854 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50995ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094071640 con 0x7f50940c2020 2023-03-03T17:20:45.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50995ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509414ae70 con 0x7f50940c5f30 2023-03-03T17:20:45.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50937fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094150c80 0x7f5094071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:45.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f5092ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50940c2020 0x7f50940c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:45.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50937fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094150c80 0x7f5094071100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54886/0 (socket says 172.21.15.102:54886) 2023-03-03T17:20:45.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f50937fe700 1 -- 172.21.15.102:0/812995478 learned_addr learned my addr 172.21.15.102:0/812995478 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:20:45.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.854+0000 7f508bfff700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50940c5f30 0x7f5094149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:45.855 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f50937fe700 1 -- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50940c5f30 msgr2=0x7f5094149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:45.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f50937fe700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50940c5f30 0x7f5094149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:45.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f50937fe700 1 -- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50940c2020 msgr2=0x7f50940c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:45.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f50937fe700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50940c2020 0x7f50940c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:45.856 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f50937fe700 1 -- 172.21.15.102:0/812995478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c009a70 con 0x7f5094150c80 2023-03-03T17:20:45.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f50937fe700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094150c80 0x7f5094071100 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f508c00b680 tx=0x7f508c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:45.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f508b7fe700 1 -- 172.21.15.102:0/812995478 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c01c810 con 0x7f5094150c80 2023-03-03T17:20:45.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f508b7fe700 1 -- 172.21.15.102:0/812995478 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c020070 con 0x7f5094150c80 2023-03-03T17:20:45.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.855+0000 7f508b7fe700 1 -- 172.21.15.102:0/812995478 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c01cbd0 con 0x7f5094150c80 2023-03-03T17:20:45.857 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.856+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50941577f0 con 0x7f5094150c80 2023-03-03T17:20:45.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.856+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5094157510 con 0x7f5094150c80 2023-03-03T17:20:45.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.857+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50940c4cc0 con 0x7f5094150c80 2023-03-03T17:20:45.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.858+0000 7f508b7fe700 1 -- 172.21.15.102:0/812995478 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f508c02e070 con 0x7f5094150c80 2023-03-03T17:20:45.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.859+0000 7f508b7fe700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50740682f0 0x7f507406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:45.858 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.859+0000 7f508b7fe700 1 -- 172.21.15.102:0/812995478 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f508c022070 con 0x7f5094150c80 2023-03-03T17:20:45.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.859+0000 7f5092ffd700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50740682f0 0x7f507406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:45.859 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.860+0000 7f5092ffd700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50740682f0 0x7f507406a7a0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f5084000f60 tx=0x7f5084007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:45.862 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:45.863+0000 7f508b7fe700 1 -- 172.21.15.102:0/812995478 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f508c0544e0 con 0x7f5094150c80 2023-03-03T17:20:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.099+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5094154950 con 0x7f5094150c80 2023-03-03T17:20:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.099+0000 7f508b7fe700 1 -- 172.21.15.102:0/812995478 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5094154950 con 0x7f5094150c80 2023-03-03T17:20:46.099 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:20:46.099 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:20:46.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50740682f0 msgr2=0x7f507406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:46.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50740682f0 0x7f507406a7a0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f5084000f60 tx=0x7f5084007e70 comp rx=0 tx=0).stop 2023-03-03T17:20:46.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094150c80 msgr2=0x7f5094071100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:46.101 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094150c80 0x7f5094071100 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f508c00b680 tx=0x7f508c004ad0 comp rx=0 tx=0).stop 2023-03-03T17:20:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 shutdown_connections 2023-03-03T17:20:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094150c80 0x7f5094071100 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50940c2020 0x7f50940c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f50740682f0 0x7f507406a7a0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 --2- 172.21.15.102:0/812995478 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f50940c5f30 0x7f5094149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 >> 172.21.15.102:0/812995478 conn(0x7f509406af90 msgr2=0x7f50940c5820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:20:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 shutdown_connections 2023-03-03T17:20:46.103 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:20:46.102+0000 7f50995ff700 1 -- 172.21.15.102:0/812995478 wait complete. 2023-03-03T17:20:46.103 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:20:46.128 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-03-03T17:21:09.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:09.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:39.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:39.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:10.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:10.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:35.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:22:35.379 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.379+0000 7f98ca1ff700 1 Processor -- start 2023-03-03T17:22:35.380 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.379+0000 7f98ca1ff700 1 -- start start 2023-03-03T17:22:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.381+0000 7f98ca1ff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f98c4153260 0x7f98c40ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.381+0000 7f98ca1ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40cf940 con 0x7f98c40c9cb0 2023-03-03T17:22:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.381+0000 7f98ca1ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40cfb90 con 0x7f98c40cac20 2023-03-03T17:22:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.381+0000 7f98ca1ff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c40cfe70 con 0x7f98c4153260 2023-03-03T17:22:35.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.381+0000 7f98c37fe700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f98c4153260 0x7f98c40ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:35.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.381+0000 7f98c2ffd700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f98c40c9cb0 0x7f98c40cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35572/0 (socket says 172.21.15.102:35572) 2023-03-03T17:22:35.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.381+0000 7f98c2ffd700 1 -- 172.21.15.102:0/21311048 learned_addr learned my addr 172.21.15.102:0/21311048 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:22:35.382 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c27fc700 1 -- 172.21.15.102:0/21311048 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586034677 0 0) 0x7f98c40cf940 con 0x7f98c40c9cb0 2023-03-03T17:22:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c27fc700 1 -- 172.21.15.102:0/21311048 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ac0035b0 con 0x7f98c40c9cb0 2023-03-03T17:22:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c27fc700 1 -- 172.21.15.102:0/21311048 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764675607 0 0) 0x7f98c40cfb90 con 0x7f98c40cac20 2023-03-03T17:22:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c37fe700 1 -- 172.21.15.102:0/21311048 >> v1:172.21.15.172:6789/0 conn(0x7f98c40cac20 legacy=0x7f98c4163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c37fe700 1 -- 172.21.15.102:0/21311048 >> v1:172.21.15.102:6789/0 conn(0x7f98c40c9cb0 legacy=0x7f98c40cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c37fe700 1 -- 172.21.15.102:0/21311048 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4158f80 con 0x7f98c4153260 2023-03-03T17:22:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c37fe700 1 --2- 172.21.15.102:0/21311048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c4153260 0x7f98c40ca6e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009840 tx=0x7f98b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fc291da5f1b5fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:35.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c27fc700 1 -- 172.21.15.102:0/21311048 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4011830 con 0x7f98c4153260 2023-03-03T17:22:35.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c27fc700 1 -- 172.21.15.102:0/21311048 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b4011990 con 0x7f98c4153260 2023-03-03T17:22:35.384 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.382+0000 7f98c27fc700 1 -- 172.21.15.102:0/21311048 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4011d20 con 0x7f98c4153260 2023-03-03T17:22:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.383+0000 7f98ca1ff700 1 -- 172.21.15.102:0/21311048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c4153260 msgr2=0x7f98c40ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.383+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/21311048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c4153260 0x7f98c40ca6e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009840 tx=0x7f98b4009b50 comp rx=0 tx=0).stop 2023-03-03T17:22:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.383+0000 7f98ca1ff700 1 -- 172.21.15.102:0/21311048 shutdown_connections 2023-03-03T17:22:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.383+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/21311048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c4153260 0x7f98c40ca6e0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.383+0000 7f98ca1ff700 1 -- 172.21.15.102:0/21311048 >> 172.21.15.102:0/21311048 conn(0x7f98c4066c00 msgr2=0x7f98c40ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.383+0000 7f98ca1ff700 1 -- 172.21.15.102:0/21311048 shutdown_connections 2023-03-03T17:22:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.383+0000 7f98ca1ff700 1 -- 172.21.15.102:0/21311048 wait complete. 2023-03-03T17:22:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.385+0000 7f98ca1ff700 1 Processor -- start 2023-03-03T17:22:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.385+0000 7f98ca1ff700 1 -- start start 2023-03-03T17:22:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.386+0000 7f98ca1ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c40cac20 0x7f98c4156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.386+0000 7f98ca1ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c4153260 0x7f98c414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.386+0000 7f98ca1ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c4158b80 0x7f98c414e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.386+0000 7f98ca1ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c415a1e0 con 0x7f98c40cac20 2023-03-03T17:22:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.386+0000 7f98ca1ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c415a480 con 0x7f98c4153260 2023-03-03T17:22:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.386+0000 7f98ca1ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c415c5a0 con 0x7f98c4158b80 2023-03-03T17:22:35.388 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c40cac20 0x7f98c4156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c40cac20 0x7f98c4156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58720/0 (socket says 172.21.15.102:58720) 2023-03-03T17:22:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 -- 172.21.15.102:0/3676232827 learned_addr learned my addr 172.21.15.102:0/3676232827 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:22:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c2ffd700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c4153260 0x7f98c414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c3fff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c4158b80 0x7f98c414e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 -- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c4158b80 msgr2=0x7f98c414e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c4158b80 0x7f98c414e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:35.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 -- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c4153260 msgr2=0x7f98c414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:35.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c4153260 0x7f98c414ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:35.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 -- 172.21.15.102:0/3676232827 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40094a0 con 0x7f98c40cac20 2023-03-03T17:22:35.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.387+0000 7f98c37fe700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c40cac20 0x7f98c4156050 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f98b4014040 tx=0x7f98b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:35.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.388+0000 7f98abfff700 1 -- 172.21.15.102:0/3676232827 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b401a7e0 con 0x7f98c40cac20 2023-03-03T17:22:35.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.388+0000 7f98abfff700 1 -- 172.21.15.102:0/3676232827 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b401a940 con 0x7f98c40cac20 2023-03-03T17:22:35.390 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.388+0000 7f98abfff700 1 -- 172.21.15.102:0/3676232827 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b401acd0 con 0x7f98c40cac20 2023-03-03T17:22:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.388+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c4159540 con 0x7f98c40cac20 2023-03-03T17:22:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.388+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c4159260 con 0x7f98c40cac20 2023-03-03T17:22:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.389+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c4159820 con 0x7f98c40cac20 2023-03-03T17:22:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.390+0000 7f98abfff700 1 -- 172.21.15.102:0/3676232827 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f98b400f050 con 0x7f98c40cac20 2023-03-03T17:22:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.391+0000 7f98abfff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f98b00682f0 0x7f98b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.392+0000 7f98abfff700 1 -- 172.21.15.102:0/3676232827 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f98b4086b90 con 0x7f98c40cac20 2023-03-03T17:22:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.392+0000 7f98c2ffd700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f98b00682f0 0x7f98b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:35.393 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.392+0000 7f98c2ffd700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f98b00682f0 0x7f98b006a7a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f98c4155660 tx=0x7f98b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:35.395 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.395+0000 7f98abfff700 1 -- 172.21.15.102:0/3676232827 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98b40532a0 con 0x7f98c40cac20 2023-03-03T17:22:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.634+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98c414dd20 con 0x7f98c40cac20 2023-03-03T17:22:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.634+0000 7f98abfff700 1 -- 172.21.15.102:0/3676232827 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f98b4029090 con 0x7f98c40cac20 2023-03-03T17:22:35.635 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:22:35.635 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:22:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f98b00682f0 msgr2=0x7f98b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f98b00682f0 0x7f98b006a7a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f98c4155660 tx=0x7f98b8009040 comp rx=0 tx=0).stop 2023-03-03T17:22:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c40cac20 msgr2=0x7f98c4156050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c40cac20 0x7f98c4156050 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f98b4014040 tx=0x7f98b4004750 comp rx=0 tx=0).stop 2023-03-03T17:22:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 shutdown_connections 2023-03-03T17:22:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f98b00682f0 0x7f98b006a7a0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f98c4158b80 0x7f98c414e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98c40cac20 0x7f98c4156050 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 --2- 172.21.15.102:0/3676232827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98c4153260 0x7f98c414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 >> 172.21.15.102:0/3676232827 conn(0x7f98c40c9cb0 msgr2=0x7f98c40ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 shutdown_connections 2023-03-03T17:22:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:35.637+0000 7f98ca1ff700 1 -- 172.21.15.102:0/3676232827 wait complete. 2023-03-03T17:22:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:22:35.665 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2023-03-03T17:22:40.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:40.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:41.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:22:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.915+0000 7fb6f41ff700 1 Processor -- start 2023-03-03T17:22:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.916+0000 7fb6f41ff700 1 -- start start 2023-03-03T17:22:41.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.917+0000 7fb6f41ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd390 0x7fb6ec0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:41.917 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.917+0000 7fb6f41ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ec153660 con 0x7fb6ec153290 2023-03-03T17:22:41.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.917+0000 7fb6f41ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ec06c360 con 0x7fb6ec0cd8e0 2023-03-03T17:22:41.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.917+0000 7fb6f41ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec15cea0 con 0x7fb6ec0cd390 2023-03-03T17:22:41.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.917+0000 7fb6f1f9b700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb6ec153290 0x7fb6ec158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40386/0 (socket says 172.21.15.102:40386) 2023-03-03T17:22:41.918 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.917+0000 7fb6f1f9b700 1 -- 172.21.15.102:0/162051080 learned_addr learned my addr 172.21.15.102:0/162051080 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:22:41.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.917+0000 7fb6f279c700 1 --2- 172.21.15.102:0/162051080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd390 0x7fb6ec0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:41.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f0f99700 1 -- 172.21.15.102:0/162051080 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957043879 0 0) 0x7fb6ec153660 con 0x7fb6ec153290 2023-03-03T17:22:41.919 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f0f99700 1 -- 172.21.15.102:0/162051080 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d80035b0 con 0x7fb6ec153290 2023-03-03T17:22:41.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f0f99700 1 -- 172.21.15.102:0/162051080 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298760450 0 0) 0x7fb6ec06c360 con 0x7fb6ec0cd8e0 2023-03-03T17:22:41.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f0f99700 1 -- 172.21.15.102:0/162051080 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ec153660 con 0x7fb6ec0cd8e0 2023-03-03T17:22:41.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f279c700 1 -- 172.21.15.102:0/162051080 >> v1:172.21.15.172:6789/0 conn(0x7fb6ec0cd8e0 legacy=0x7fb6ec0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:41.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f279c700 1 -- 172.21.15.102:0/162051080 >> v1:172.21.15.102:6789/0 conn(0x7fb6ec153290 legacy=0x7fb6ec158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:41.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f279c700 1 -- 172.21.15.102:0/162051080 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec06c7f0 con 0x7fb6ec0cd390 2023-03-03T17:22:41.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.918+0000 7fb6f279c700 1 --2- 172.21.15.102:0/162051080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd390 0x7fb6ec0ce530 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009dc0 tx=0x7fb6e400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=818c7cd1567d57d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:41.920 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f0f99700 1 -- 172.21.15.102:0/162051080 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e400e070 con 0x7fb6ec0cd390 2023-03-03T17:22:41.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f0f99700 1 -- 172.21.15.102:0/162051080 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e400bb20 con 0x7fb6ec0cd390 2023-03-03T17:22:41.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f0f99700 1 -- 172.21.15.102:0/162051080 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4012430 con 0x7fb6ec0cd390 2023-03-03T17:22:41.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f41ff700 1 -- 172.21.15.102:0/162051080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd390 msgr2=0x7fb6ec0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:41.921 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/162051080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd390 0x7fb6ec0ce530 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009dc0 tx=0x7fb6e400b330 comp rx=0 tx=0).stop 2023-03-03T17:22:41.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f41ff700 1 -- 172.21.15.102:0/162051080 shutdown_connections 2023-03-03T17:22:41.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/162051080 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd390 0x7fb6ec0ce530 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:41.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.919+0000 7fb6f41ff700 1 -- 172.21.15.102:0/162051080 >> 172.21.15.102:0/162051080 conn(0x7fb6ec065fb0 msgr2=0x7fb6ec0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:41.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.920+0000 7fb6f41ff700 1 -- 172.21.15.102:0/162051080 shutdown_connections 2023-03-03T17:22:41.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.920+0000 7fb6f41ff700 1 -- 172.21.15.102:0/162051080 wait complete. 2023-03-03T17:22:41.922 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.922+0000 7fb6f41ff700 1 Processor -- start 2023-03-03T17:22:41.923 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.922+0000 7fb6f41ff700 1 -- start start 2023-03-03T17:22:41.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f41ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6ec0cd390 0x7fb6ec0c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:41.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f41ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd8e0 0x7fb6ec14e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:41.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f41ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ec153290 0x7fb6ec0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:41.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f41ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec154360 con 0x7fb6ec153290 2023-03-03T17:22:41.924 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f41ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec14ea20 con 0x7fb6ec0cd390 2023-03-03T17:22:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f41ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec0c91c0 con 0x7fb6ec0cd8e0 2023-03-03T17:22:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f1f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6ec0cd390 0x7fb6ec0c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f179a700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd8e0 0x7fb6ec14e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f1f9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6ec0cd390 0x7fb6ec0c7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:49282/0 (socket says 172.21.15.102:49282) 2023-03-03T17:22:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f1f9b700 1 -- 172.21.15.102:0/9110620 learned_addr learned my addr 172.21.15.102:0/9110620 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:22:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.923+0000 7fb6f279c700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ec153290 0x7fb6ec0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f1f9b700 1 -- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd8e0 msgr2=0x7fb6ec14e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:41.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f1f9b700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd8e0 0x7fb6ec14e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:41.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f1f9b700 1 -- 172.21.15.102:0/9110620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ec153290 msgr2=0x7fb6ec0c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:41.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f1f9b700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ec153290 0x7fb6ec0c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:41.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f1f9b700 1 -- 172.21.15.102:0/9110620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4009a70 con 0x7fb6ec0cd390 2023-03-03T17:22:41.926 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f1f9b700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6ec0cd390 0x7fb6ec0c7da0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c700 tx=0x7fb6e800ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6e27fc700 1 -- 172.21.15.102:0/9110620 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8005d70 con 0x7fb6ec0cd390 2023-03-03T17:22:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6e27fc700 1 -- 172.21.15.102:0/9110620 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e80070d0 con 0x7fb6ec0cd390 2023-03-03T17:22:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec0ccde0 con 0x7fb6ec0cd390 2023-03-03T17:22:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6e27fc700 1 -- 172.21.15.102:0/9110620 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8007440 con 0x7fb6ec0cd390 2023-03-03T17:22:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.924+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec15d160 con 0x7fb6ec0cd390 2023-03-03T17:22:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.926+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec0cd0c0 con 0x7fb6ec0cd390 2023-03-03T17:22:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.927+0000 7fb6e27fc700 1 -- 172.21.15.102:0/9110620 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb6e8019030 con 0x7fb6ec0cd390 2023-03-03T17:22:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.928+0000 7fb6e27fc700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6d00682f0 0x7fb6d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.928+0000 7fb6e27fc700 1 -- 172.21.15.102:0/9110620 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb6e8084fe0 con 0x7fb6ec0cd390 2023-03-03T17:22:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.929+0000 7fb6f179a700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6d00682f0 0x7fb6d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.929+0000 7fb6f179a700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6d00682f0 0x7fb6d006a7a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0c82e0 tx=0x7fb6dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:41.932 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:41.933+0000 7fb6e27fc700 1 -- 172.21.15.102:0/9110620 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb6e804b5e0 con 0x7fb6ec0cd390 2023-03-03T17:22:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.166+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6ec04e6e0 con 0x7fb6ec0cd390 2023-03-03T17:22:42.167 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.167+0000 7fb6e27fc700 1 -- 172.21.15.102:0/9110620 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7fb6e8017030 con 0x7fb6ec0cd390 2023-03-03T17:22:42.168 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:22:42.168 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:22:42.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6d00682f0 msgr2=0x7fb6d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:42.170 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6d00682f0 0x7fb6d006a7a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0c82e0 tx=0x7fb6dc009040 comp rx=0 tx=0).stop 2023-03-03T17:22:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6ec0cd390 msgr2=0x7fb6ec0c7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6ec0cd390 0x7fb6ec0c7da0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c700 tx=0x7fb6e800ca10 comp rx=0 tx=0).stop 2023-03-03T17:22:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 shutdown_connections 2023-03-03T17:22:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ec153290 0x7fb6ec0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6ec0cd390 0x7fb6ec0c7da0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7fb6d00682f0 0x7fb6d006a7a0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:42.171 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 --2- 172.21.15.102:0/9110620 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fb6ec0cd8e0 0x7fb6ec14e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 >> 172.21.15.102:0/9110620 conn(0x7fb6ec065fb0 msgr2=0x7fb6ec1585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 shutdown_connections 2023-03-03T17:22:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:22:42.170+0000 7fb6f41ff700 1 -- 172.21.15.102:0/9110620 wait complete. 2023-03-03T17:22:42.172 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:22:42.197 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 56 secs before thrashing 2023-03-03T17:23:10.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:10.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:38.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:23:38.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.449+0000 7f1b4b59e700 1 Processor -- start 2023-03-03T17:23:38.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.449+0000 7f1b4b59e700 1 -- start start 2023-03-03T17:23:38.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4b59e700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b4c153030 0x7f1b4c0c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:38.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4b59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1b4c0b1e90 0x7f1b4c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:38.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4b59e700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c06bee0 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4b59e700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4c06c780 con 0x7f1b4c153030 2023-03-03T17:23:38.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4b59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4c15c620 con 0x7f1b4c0b1e90 2023-03-03T17:23:38.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4a59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b4c153030 0x7f1b4c0c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4a59c700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b4c153030 0x7f1b4c0c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60200/0 (socket says 172.21.15.102:60200) 2023-03-03T17:23:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4a59c700 1 -- 172.21.15.102:0/2853272325 learned_addr learned my addr 172.21.15.102:0/2853272325 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:23:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b4ad9d700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1b4c0b3f30 0x7f1b4c0cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40372/0 (socket says 172.21.15.102:40372) 2023-03-03T17:23:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.451+0000 7f1b43fff700 1 --2- 172.21.15.102:0/2853272325 >> v2:172.21.15.172:3300/0 conn(0x7f1b4c0b1e90 0x7f1b4c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b4a59c700 1 -- 172.21.15.102:0/2853272325 >> v2:172.21.15.172:3300/0 conn(0x7f1b4c0b1e90 msgr2=0x7f1b4c15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:23:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b4a59c700 1 --2- 172.21.15.102:0/2853272325 >> v2:172.21.15.172:3300/0 conn(0x7f1b4c0b1e90 0x7f1b4c15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b4a59c700 1 -- 172.21.15.102:0/2853272325 >> v1:172.21.15.102:6789/0 conn(0x7f1b4c0b3f30 legacy=0x7f1b4c0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b4a59c700 1 -- 172.21.15.102:0/2853272325 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c06c1c0 con 0x7f1b4c153030 2023-03-03T17:23:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b49d9b700 1 -- 172.21.15.102:0/2853272325 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181561199 0 0) 0x7f1b4c06bee0 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b4a59c700 1 --2- 172.21.15.102:0/2853272325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c153030 0x7f1b4c0c5b00 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0098f0 tx=0x7f1b3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea381e706c07848 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:38.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b49d9b700 1 -- 172.21.15.102:0/2853272325 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c011830 con 0x7f1b4c153030 2023-03-03T17:23:38.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b49d9b700 1 -- 172.21.15.102:0/2853272325 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c011990 con 0x7f1b4c153030 2023-03-03T17:23:38.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.452+0000 7f1b49d9b700 1 -- 172.21.15.102:0/2853272325 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c011d20 con 0x7f1b4c153030 2023-03-03T17:23:38.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 -- 172.21.15.102:0/2853272325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c153030 msgr2=0x7f1b4c0c5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:38.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/2853272325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c153030 0x7f1b4c0c5b00 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0098f0 tx=0x7f1b3c009c00 comp rx=0 tx=0).stop 2023-03-03T17:23:38.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 -- 172.21.15.102:0/2853272325 shutdown_connections 2023-03-03T17:23:38.454 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/2853272325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c153030 0x7f1b4c0c5b00 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/2853272325 >> v2:172.21.15.172:3300/0 conn(0x7f1b4c0b1e90 0x7f1b4c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 -- 172.21.15.102:0/2853272325 >> 172.21.15.102:0/2853272325 conn(0x7f1b4c08b1c0 msgr2=0x7f1b4c071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:38.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 -- 172.21.15.102:0/2853272325 shutdown_connections 2023-03-03T17:23:38.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.453+0000 7f1b4b59e700 1 -- 172.21.15.102:0/2853272325 wait complete. 2023-03-03T17:23:38.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.455+0000 7f1b4b59e700 1 Processor -- start 2023-03-03T17:23:38.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.455+0000 7f1b4b59e700 1 -- start start 2023-03-03T17:23:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.456+0000 7f1b4b59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c0b1e90 0x7f1b4c0c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.456+0000 7f1b4b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b4c0b3f30 0x7f1b4c1f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.456+0000 7f1b4b59e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b4c153030 0x7f1b4c0cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.456+0000 7f1b4b59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c1e5c10 con 0x7f1b4c0b1e90 2023-03-03T17:23:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.456+0000 7f1b4b59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c0c87c0 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.456+0000 7f1b4b59e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c150c20 con 0x7f1b4c153030 2023-03-03T17:23:38.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b4c0b3f30 0x7f1b4c1f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b4a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c0b1e90 0x7f1b4c0c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b4c0b3f30 0x7f1b4c1f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.102:46598/0 (socket says 172.21.15.102:46598) 2023-03-03T17:23:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 -- 172.21.15.102:0/94012771 learned_addr learned my addr 172.21.15.102:0/94012771 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:23:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b4ad9d700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b4c153030 0x7f1b4c0cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 -- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b4c153030 msgr2=0x7f1b4c0cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b4c153030 0x7f1b4c0cdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 -- 172.21.15.102:0/94012771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c0b1e90 msgr2=0x7f1b4c0c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:38.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c0b1e90 0x7f1b4c0c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 -- 172.21.15.102:0/94012771 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c009550 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b4a59c700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c0b1e90 0x7f1b4c0c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:23:38.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.457+0000 7f1b43fff700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b4c0b3f30 0x7f1b4c1f2200 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e9b0 tx=0x7f1b3400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:38.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.458+0000 7f1b42ffd700 1 -- 172.21.15.102:0/94012771 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400ee50 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.458+0000 7f1b42ffd700 1 -- 172.21.15.102:0/94012771 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3400c7f0 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.458+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c15c240 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.458+0000 7f1b42ffd700 1 -- 172.21.15.102:0/94012771 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400cb60 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.458+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c06c4a0 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.459+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4c0c4f70 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.460+0000 7f1b42ffd700 1 -- 172.21.15.102:0/94012771 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b34019030 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.462+0000 7f1b42ffd700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b2c0682f0 0x7f1b2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:38.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.462+0000 7f1b42ffd700 1 -- 172.21.15.102:0/94012771 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1b34085c70 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.463+0000 7f1b4a59c700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b2c0682f0 0x7f1b2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.464+0000 7f1b4a59c700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b2c0682f0 0x7f1b2c06a7a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009130 tx=0x7f1b3c01c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:38.468 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.468+0000 7f1b42ffd700 1 -- 172.21.15.102:0/94012771 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b340522f0 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.703+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b4c15cf80 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.702 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.703+0000 7f1b42ffd700 1 -- 172.21.15.102:0/94012771 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f1b34021090 con 0x7f1b4c0b3f30 2023-03-03T17:23:38.703 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:23:38.703 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:23:38.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b2c0682f0 msgr2=0x7f1b2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:38.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b2c0682f0 0x7f1b2c06a7a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009130 tx=0x7f1b3c01c040 comp rx=0 tx=0).stop 2023-03-03T17:23:38.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b4c0b3f30 msgr2=0x7f1b4c1f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:38.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b4c0b3f30 0x7f1b4c1f2200 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e9b0 tx=0x7f1b3400ecc0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 shutdown_connections 2023-03-03T17:23:38.706 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f1b2c0682f0 0x7f1b2c06a7a0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1b4c153030 0x7f1b4c0cdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b4c0b1e90 0x7f1b4c0c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 --2- 172.21.15.102:0/94012771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b4c0b3f30 0x7f1b4c1f2200 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 >> 172.21.15.102:0/94012771 conn(0x7f1b4c08b1c0 msgr2=0x7f1b4c151f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 shutdown_connections 2023-03-03T17:23:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:23:38.706+0000 7f1b4b59e700 1 -- 172.21.15.102:0/94012771 wait complete. 2023-03-03T17:23:38.708 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:23:38.733 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-03-03T17:23:40.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:40.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:10.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:10.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:24.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:24:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.995+0000 7f71419ff700 1 Processor -- start 2023-03-03T17:24:24.994 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.995+0000 7f71419ff700 1 -- start start 2023-03-03T17:24:24.995 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.996+0000 7f71419ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c9cb0 0x7f713c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:24.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.996+0000 7f71419ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0cf940 con 0x7f713c0ca190 2023-03-03T17:24:24.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f71419ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0cfb90 con 0x7f713c153260 2023-03-03T17:24:24.996 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f71419ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0cfe70 con 0x7f713c0c9cb0 2023-03-03T17:24:24.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f713b7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f713c0ca190 0x7f713c163610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42162/0 (socket says 172.21.15.102:42162) 2023-03-03T17:24:24.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f713b7fe700 1 -- 172.21.15.102:0/2313085411 learned_addr learned my addr 172.21.15.102:0/2313085411 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:24:24.997 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f713affd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f713c153260 0x7f713c0ca6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:57514/0 (socket says 172.21.15.102:57514) 2023-03-03T17:24:24.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f7133fff700 1 --2- 172.21.15.102:0/2313085411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c9cb0 0x7f713c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:24.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f713a7fc700 1 -- 172.21.15.102:0/2313085411 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730568377 0 0) 0x7f713c0cf940 con 0x7f713c0ca190 2023-03-03T17:24:24.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f713a7fc700 1 -- 172.21.15.102:0/2313085411 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71280035b0 con 0x7f713c0ca190 2023-03-03T17:24:24.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f713a7fc700 1 -- 172.21.15.102:0/2313085411 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048323720 0 0) 0x7f713c0cfb90 con 0x7f713c153260 2023-03-03T17:24:24.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.997+0000 7f713a7fc700 1 -- 172.21.15.102:0/2313085411 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c0cf940 con 0x7f713c153260 2023-03-03T17:24:24.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.998+0000 7f7133fff700 1 -- 172.21.15.102:0/2313085411 >> v1:172.21.15.172:6789/0 conn(0x7f713c153260 legacy=0x7f713c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:24.998 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.998+0000 7f7133fff700 1 -- 172.21.15.102:0/2313085411 >> v1:172.21.15.102:6789/0 conn(0x7f713c0ca190 legacy=0x7f713c163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:24.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.998+0000 7f7133fff700 1 -- 172.21.15.102:0/2313085411 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c158f80 con 0x7f713c0c9cb0 2023-03-03T17:24:24.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.998+0000 7f7133fff700 1 --2- 172.21.15.102:0/2313085411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c9cb0 0x7f713c0cb880 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7124009870 tx=0x7f7124009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f01de28733925875 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:24:24.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.998+0000 7f713a7fc700 1 -- 172.21.15.102:0/2313085411 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7124011830 con 0x7f713c0c9cb0 2023-03-03T17:24:24.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.998+0000 7f713a7fc700 1 -- 172.21.15.102:0/2313085411 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7124011990 con 0x7f713c0c9cb0 2023-03-03T17:24:24.999 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.998+0000 7f713a7fc700 1 -- 172.21.15.102:0/2313085411 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7124011d20 con 0x7f713c0c9cb0 2023-03-03T17:24:25.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.999+0000 7f71419ff700 1 -- 172.21.15.102:0/2313085411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c9cb0 msgr2=0x7f713c0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:25.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.999+0000 7f71419ff700 1 --2- 172.21.15.102:0/2313085411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c9cb0 0x7f713c0cb880 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7124009870 tx=0x7f7124009b80 comp rx=0 tx=0).stop 2023-03-03T17:24:25.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.999+0000 7f71419ff700 1 -- 172.21.15.102:0/2313085411 shutdown_connections 2023-03-03T17:24:25.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.999+0000 7f71419ff700 1 --2- 172.21.15.102:0/2313085411 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c9cb0 0x7f713c0cb880 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.000 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.999+0000 7f71419ff700 1 -- 172.21.15.102:0/2313085411 >> 172.21.15.102:0/2313085411 conn(0x7f713c066c00 msgr2=0x7f713c0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:24:25.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.999+0000 7f71419ff700 1 -- 172.21.15.102:0/2313085411 shutdown_connections 2023-03-03T17:24:25.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:24.999+0000 7f71419ff700 1 -- 172.21.15.102:0/2313085411 wait complete. 2023-03-03T17:24:25.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.001+0000 7f71419ff700 1 Processor -- start 2023-03-03T17:24:25.001 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.001+0000 7f71419ff700 1 -- start start 2023-03-03T17:24:25.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.002+0000 7f71419ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f713c0ca190 0x7f713c1f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:25.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.002+0000 7f71419ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f713c153260 0x7f713c1ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:25.002 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.002+0000 7f71419ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c5920 0x7f713c1491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.002+0000 7f71419ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0cdc50 con 0x7f713c0ca190 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.002+0000 7f71419ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c1f2400 con 0x7f713c153260 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.002+0000 7f71419ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c1f1e20 con 0x7f713c0c5920 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f713c0ca190 0x7f713c1f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f713c0ca190 0x7f713c1f2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59934/0 (socket says 172.21.15.102:59934) 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f7133fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f713c153260 0x7f713c1ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 -- 172.21.15.102:0/4196287424 learned_addr learned my addr 172.21.15.102:0/4196287424 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:24:25.003 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713b7fe700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c5920 0x7f713c1491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:25.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 -- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c5920 msgr2=0x7f713c1491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:25.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c5920 0x7f713c1491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 -- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f713c153260 msgr2=0x7f713c1ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:25.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f713c153260 0x7f713c1ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 -- 172.21.15.102:0/4196287424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240094a0 con 0x7f713c0ca190 2023-03-03T17:24:25.004 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.003+0000 7f713affd700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f713c0ca190 0x7f713c1f2890 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c4d0 tx=0x7f712c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:24:25.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.004+0000 7f7138ff9700 1 -- 172.21.15.102:0/4196287424 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c00cea0 con 0x7f713c0ca190 2023-03-03T17:24:25.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.004+0000 7f7138ff9700 1 -- 172.21.15.102:0/4196287424 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c005cb0 con 0x7f713c0ca190 2023-03-03T17:24:25.005 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.004+0000 7f7138ff9700 1 -- 172.21.15.102:0/4196287424 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c0072d0 con 0x7f713c0ca190 2023-03-03T17:24:25.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.004+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c159540 con 0x7f713c0ca190 2023-03-03T17:24:25.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.004+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c159260 con 0x7f713c0ca190 2023-03-03T17:24:25.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.005+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c159820 con 0x7f713c0ca190 2023-03-03T17:24:25.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.006+0000 7f7138ff9700 1 -- 172.21.15.102:0/4196287424 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f712c019030 con 0x7f713c0ca190 2023-03-03T17:24:25.006 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.008+0000 7f7138ff9700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f711c068320 0x7f711c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:25.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.008+0000 7f7138ff9700 1 -- 172.21.15.102:0/4196287424 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f712c084f20 con 0x7f713c0ca190 2023-03-03T17:24:25.007 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.008+0000 7f7133fff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f711c068320 0x7f711c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:25.008 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.009+0000 7f7133fff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f711c068320 0x7f711c06a7d0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f71240097c0 tx=0x7f712401c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:24:25.010 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.012+0000 7f7138ff9700 1 -- 172.21.15.102:0/4196287424 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f712c04b5e0 con 0x7f713c0ca190 2023-03-03T17:24:25.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.251+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f713c04e6e0 con 0x7f713c0ca190 2023-03-03T17:24:25.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.251+0000 7f7138ff9700 1 -- 172.21.15.102:0/4196287424 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f712c017070 con 0x7f713c0ca190 2023-03-03T17:24:25.251 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:24:25.251 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:24:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f711c068320 msgr2=0x7f711c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f711c068320 0x7f711c06a7d0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f71240097c0 tx=0x7f712401c040 comp rx=0 tx=0).stop 2023-03-03T17:24:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f713c0ca190 msgr2=0x7f713c1f2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f713c0ca190 0x7f713c1f2890 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c4d0 tx=0x7f712c00c7e0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 shutdown_connections 2023-03-03T17:24:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f711c068320 0x7f711c06a7d0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f713c0c5920 0x7f713c1491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f713c0ca190 0x7f713c1f2890 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.254 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 --2- 172.21.15.102:0/4196287424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f713c153260 0x7f713c1ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 >> 172.21.15.102:0/4196287424 conn(0x7f713c0c9cb0 msgr2=0x7f713c0cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:24:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 shutdown_connections 2023-03-03T17:24:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:24:25.254+0000 7f71419ff700 1 -- 172.21.15.102:0/4196287424 wait complete. 2023-03-03T17:24:25.255 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:24:25.281 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 40 secs before thrashing 2023-03-03T17:24:41.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:41.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:05.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:25:05.538 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.538+0000 7f553ddff700 1 Processor -- start 2023-03-03T17:25:05.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.538+0000 7f553ddff700 1 -- start start 2023-03-03T17:25:05.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.539+0000 7f553ddff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd8e0 0x7f55380cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:05.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.539+0000 7f553ddff700 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f55380cd390 0x7f55380ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:05.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f553ddff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5538153660 con 0x7f5538153290 2023-03-03T17:25:05.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f553ddff700 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553806c360 con 0x7f55380cd390 2023-03-03T17:25:05.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f553ddff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553815cea0 con 0x7f55380cd8e0 2023-03-03T17:25:05.540 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f55377fe700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5538153290 0x7f5538158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41724/0 (socket says 172.21.15.102:41724) 2023-03-03T17:25:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f55377fe700 1 -- 172.21.15.102:0/3990287898 learned_addr learned my addr 172.21.15.102:0/3990287898 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:25:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f5536ffd700 1 --2- 172.21.15.102:0/3990287898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd8e0 0x7f55380cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f5537fff700 1 --2- 172.21.15.102:0/3990287898 >> v2:172.21.15.102:3300/0 conn(0x7f55380cd390 0x7f55380ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f55367fc700 1 -- 172.21.15.102:0/3990287898 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670710537 0 0) 0x7f5538153660 con 0x7f5538153290 2023-03-03T17:25:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f55367fc700 1 -- 172.21.15.102:0/3990287898 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55200035b0 con 0x7f5538153290 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f5536ffd700 1 -- 172.21.15.102:0/3990287898 >> v2:172.21.15.102:3300/0 conn(0x7f55380cd390 msgr2=0x7f55380ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f5536ffd700 1 --2- 172.21.15.102:0/3990287898 >> v2:172.21.15.102:3300/0 conn(0x7f55380cd390 0x7f55380ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.540+0000 7f5536ffd700 1 -- 172.21.15.102:0/3990287898 >> v1:172.21.15.172:6789/0 conn(0x7f5538153290 legacy=0x7f5538158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f5536ffd700 1 -- 172.21.15.102:0/3990287898 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553806c7f0 con 0x7f55380cd8e0 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f5537fff700 1 --2- 172.21.15.102:0/3990287898 >> v2:172.21.15.102:3300/0 conn(0x7f55380cd390 0x7f55380ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f5536ffd700 1 --2- 172.21.15.102:0/3990287898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd8e0 0x7f55380cde20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f552c009840 tx=0x7f552c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13f64914cc973533 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f55367fc700 1 -- 172.21.15.102:0/3990287898 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c011830 con 0x7f55380cd8e0 2023-03-03T17:25:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f55367fc700 1 -- 172.21.15.102:0/3990287898 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c011990 con 0x7f55380cd8e0 2023-03-03T17:25:05.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f55367fc700 1 -- 172.21.15.102:0/3990287898 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c011d20 con 0x7f55380cd8e0 2023-03-03T17:25:05.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f553ddff700 1 -- 172.21.15.102:0/3990287898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd8e0 msgr2=0x7f55380cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:05.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.541+0000 7f553ddff700 1 --2- 172.21.15.102:0/3990287898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd8e0 0x7f55380cde20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f552c009840 tx=0x7f552c009b50 comp rx=0 tx=0).stop 2023-03-03T17:25:05.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.542+0000 7f553ddff700 1 -- 172.21.15.102:0/3990287898 shutdown_connections 2023-03-03T17:25:05.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.542+0000 7f553ddff700 1 --2- 172.21.15.102:0/3990287898 >> v2:172.21.15.102:3300/0 conn(0x7f55380cd390 0x7f55380ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.542+0000 7f553ddff700 1 --2- 172.21.15.102:0/3990287898 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd8e0 0x7f55380cde20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.544 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.542+0000 7f553ddff700 1 -- 172.21.15.102:0/3990287898 >> 172.21.15.102:0/3990287898 conn(0x7f5538065fb0 msgr2=0x7f55380663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:25:05.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.542+0000 7f553ddff700 1 -- 172.21.15.102:0/3990287898 shutdown_connections 2023-03-03T17:25:05.545 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.542+0000 7f553ddff700 1 -- 172.21.15.102:0/3990287898 wait complete. 2023-03-03T17:25:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.544+0000 7f553ddff700 1 Processor -- start 2023-03-03T17:25:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.544+0000 7f553ddff700 1 -- start start 2023-03-03T17:25:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f553ddff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd390 0x7f55380c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f553ddff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55380cd8e0 0x7f55380c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:05.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f553ddff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5538153290 0x7f55380c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:05.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f553ddff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55380c5730 con 0x7f5538153290 2023-03-03T17:25:05.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f553ddff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55380c33f0 con 0x7f55380cd8e0 2023-03-03T17:25:05.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f553ddff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55380c4e00 con 0x7f55380cd390 2023-03-03T17:25:05.547 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f55377fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd390 0x7f55380c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f5536ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55380cd8e0 0x7f55380c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f55377fe700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd390 0x7f55380c6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39754/0 (socket says 172.21.15.102:39754) 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f55377fe700 1 -- 172.21.15.102:0/2712655527 learned_addr learned my addr 172.21.15.102:0/2712655527 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.545+0000 7f5537fff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5538153290 0x7f55380c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f55377fe700 1 -- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55380cd8e0 msgr2=0x7f55380c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f55377fe700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55380cd8e0 0x7f55380c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f55377fe700 1 -- 172.21.15.102:0/2712655527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5538153290 msgr2=0x7f55380c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f55377fe700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5538153290 0x7f55380c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f55377fe700 1 -- 172.21.15.102:0/2712655527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5530009a70 con 0x7f55380cd390 2023-03-03T17:25:05.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f55377fe700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd390 0x7f55380c6e20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f552800c700 tx=0x7f552800ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:05.549 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f551ffff700 1 -- 172.21.15.102:0/2712655527 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528005d70 con 0x7f55380cd390 2023-03-03T17:25:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f551ffff700 1 -- 172.21.15.102:0/2712655527 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55280070d0 con 0x7f55380cd390 2023-03-03T17:25:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f551ffff700 1 -- 172.21.15.102:0/2712655527 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528007460 con 0x7f55380cd390 2023-03-03T17:25:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.546+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c0094a0 con 0x7f55380cd390 2023-03-03T17:25:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.547+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55380ccde0 con 0x7f55380cd390 2023-03-03T17:25:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.548+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553815d160 con 0x7f55380cd390 2023-03-03T17:25:05.550 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.549+0000 7f551ffff700 1 -- 172.21.15.102:0/2712655527 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5528019030 con 0x7f55380cd390 2023-03-03T17:25:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.550+0000 7f551ffff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55240682f0 0x7f552406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.550+0000 7f551ffff700 1 -- 172.21.15.102:0/2712655527 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f5528085120 con 0x7f55380cd390 2023-03-03T17:25:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.551+0000 7f5536ffd700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55240682f0 0x7f552406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:05.551 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.551+0000 7f5536ffd700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55240682f0 0x7f552406a7a0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f552c000c00 tx=0x7f552c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:05.554 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.555+0000 7f551ffff700 1 -- 172.21.15.102:0/2712655527 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5528051830 con 0x7f55380cd390 2023-03-03T17:25:05.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.792+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55380cd0c0 con 0x7f55380cd390 2023-03-03T17:25:05.792 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.793+0000 7f551ffff700 1 -- 172.21.15.102:0/2712655527 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7f5528054e50 con 0x7f55380cd390 2023-03-03T17:25:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:25:05.793 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:25:05.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55240682f0 msgr2=0x7f552406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:05.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55240682f0 0x7f552406a7a0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f552c000c00 tx=0x7f552c008030 comp rx=0 tx=0).stop 2023-03-03T17:25:05.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd390 msgr2=0x7f55380c6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:05.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd390 0x7f55380c6e20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f552800c700 tx=0x7f552800ca10 comp rx=0 tx=0).stop 2023-03-03T17:25:05.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 shutdown_connections 2023-03-03T17:25:05.796 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7f55240682f0 0x7f552406a7a0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f55380cd390 0x7f55380c6e20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5538153290 0x7f55380c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 --2- 172.21.15.102:0/2712655527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55380cd8e0 0x7f55380c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.795+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 >> 172.21.15.102:0/2712655527 conn(0x7f5538065fb0 msgr2=0x7f55380cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:25:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.796+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 shutdown_connections 2023-03-03T17:25:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:05.796+0000 7f553ddff700 1 -- 172.21.15.102:0/2712655527 wait complete. 2023-03-03T17:25:05.798 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:25:05.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:25:06.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.066+0000 7faf811ff700 1 Processor -- start 2023-03-03T17:25:06.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.067+0000 7faf811ff700 1 -- start start 2023-03-03T17:25:06.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.068+0000 7faf811ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c153290 0x7faf7c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.068+0000 7faf811ff700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c153660 con 0x7faf7c1593c0 2023-03-03T17:25:06.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.068+0000 7faf811ff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c06c360 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.068+0000 7faf811ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0cec40 con 0x7faf7c153290 2023-03-03T17:25:06.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.068+0000 7faf7ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c153290 0x7faf7c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.069 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.068+0000 7faf7b59e700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7faf7c0cd8e0 0x7faf7c0ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.102:41730/0 (socket says 172.21.15.102:41730) 2023-03-03T17:25:06.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.068+0000 7faf7b59e700 1 -- 172.21.15.102:0/3303819325 learned_addr learned my addr 172.21.15.102:0/3303819325 (peer_addr_for_me v1:172.21.15.102:0/0) 2023-03-03T17:25:06.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.069+0000 7faf7ad9d700 1 -- 172.21.15.102:0/3303819325 >> v1:172.21.15.172:6789/0 conn(0x7faf7c0cd8e0 legacy=0x7faf7c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.069+0000 7faf7ad9d700 1 -- 172.21.15.102:0/3303819325 >> v1:172.21.15.102:6789/0 conn(0x7faf7c1593c0 legacy=0x7faf7c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.069+0000 7faf7ad9d700 1 -- 172.21.15.102:0/3303819325 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c15cea0 con 0x7faf7c153290 2023-03-03T17:25:06.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.069+0000 7faf7ad9d700 1 --2- 172.21.15.102:0/3303819325 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c153290 0x7faf7c158e80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009870 tx=0x7faf6c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257c06a7cd424caa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:06.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.069+0000 7faf79d9b700 1 -- 172.21.15.102:0/3303819325 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c011830 con 0x7faf7c153290 2023-03-03T17:25:06.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.069+0000 7faf79d9b700 1 -- 172.21.15.102:0/3303819325 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c011990 con 0x7faf7c153290 2023-03-03T17:25:06.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.069+0000 7faf79d9b700 1 -- 172.21.15.102:0/3303819325 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c011d20 con 0x7faf7c153290 2023-03-03T17:25:06.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.070+0000 7faf811ff700 1 -- 172.21.15.102:0/3303819325 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c153290 msgr2=0x7faf7c158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.070+0000 7faf811ff700 1 --2- 172.21.15.102:0/3303819325 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c153290 0x7faf7c158e80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009870 tx=0x7faf6c009b80 comp rx=0 tx=0).stop 2023-03-03T17:25:06.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.070+0000 7faf811ff700 1 -- 172.21.15.102:0/3303819325 shutdown_connections 2023-03-03T17:25:06.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.070+0000 7faf811ff700 1 --2- 172.21.15.102:0/3303819325 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c153290 0x7faf7c158e80 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.070+0000 7faf811ff700 1 -- 172.21.15.102:0/3303819325 >> 172.21.15.102:0/3303819325 conn(0x7faf7c065fb0 msgr2=0x7faf7c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:25:06.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.070+0000 7faf811ff700 1 -- 172.21.15.102:0/3303819325 shutdown_connections 2023-03-03T17:25:06.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.070+0000 7faf811ff700 1 -- 172.21.15.102:0/3303819325 wait complete. 2023-03-03T17:25:06.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.072+0000 7faf811ff700 1 Processor -- start 2023-03-03T17:25:06.073 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.072+0000 7faf811ff700 1 -- start start 2023-03-03T17:25:06.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.073+0000 7faf811ff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c0cd8e0 0x7faf7c156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.073+0000 7faf811ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c153290 0x7faf7c0c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.073+0000 7faf811ff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf7c1593c0 0x7faf7c154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.073+0000 7faf811ff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c155670 con 0x7faf7c153290 2023-03-03T17:25:06.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.073+0000 7faf811ff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0b2040 con 0x7faf7c1593c0 2023-03-03T17:25:06.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.073+0000 7faf811ff700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c1558c0 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.075 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c0cd8e0 0x7faf7c156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c153290 0x7faf7c0c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c0cd8e0 0x7faf7c156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.102:39758/0 (socket says 172.21.15.102:39758) 2023-03-03T17:25:06.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 -- 172.21.15.102:0/968766480 learned_addr learned my addr 172.21.15.102:0/968766480 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-03-03T17:25:06.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7b59e700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf7c1593c0 0x7faf7c154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 -- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf7c1593c0 msgr2=0x7faf7c154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf7c1593c0 0x7faf7c154260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.076 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 -- 172.21.15.102:0/968766480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c153290 msgr2=0x7faf7c0c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c153290 0x7faf7c0c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.074+0000 7faf7ad9d700 1 -- 172.21.15.102:0/968766480 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c0094a0 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.077 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.075+0000 7faf7ad9d700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c0cd8e0 0x7faf7c156680 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0090b0 tx=0x7faf6c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:06.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.075+0000 7faf637fe700 1 -- 172.21.15.102:0/968766480 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c0036a0 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.075+0000 7faf637fe700 1 -- 172.21.15.102:0/968766480 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c01e070 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.075+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c15d160 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.075+0000 7faf637fe700 1 -- 172.21.15.102:0/968766480 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c003a20 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.075+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c06c7f0 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.076+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c0ccde0 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.078 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.078+0000 7faf637fe700 1 -- 172.21.15.102:0/968766480 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 6) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faf6c02a020 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.079+0000 7faf637fe700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faf680682f0 0x7faf6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.079+0000 7faf637fe700 1 -- 172.21.15.102:0/968766480 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6148+0+0 (secure 0 0 0) 0x7faf6c0878b0 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.079 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.079+0000 7faf7a59c700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faf680682f0 0x7faf6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.080 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.080+0000 7faf7a59c700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faf680682f0 0x7faf6806a7a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0c5a90 tx=0x7faf70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:06.082 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.082+0000 7faf637fe700 1 -- 172.21.15.102:0/968766480 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faf6c054040 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.317 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.317+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7c04e6e0 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.318 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.318+0000 7faf637fe700 1 -- 172.21.15.102:0/968766480 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6164 (secure 0 0 0) 0x7faf6c057660 con 0x7faf7c0cd8e0 2023-03-03T17:25:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-03-03T17:25:06.319 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2771657719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2771657719},{"type":"v1","addr":"172.21.15.102:6835","nonce":2771657719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3824028126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3824028126},{"type":"v1","addr":"172.21.15.172:6839","nonce":3824028126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/3072142046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3072142046},{"type":"v1","addr":"172.21.15.102:6833","nonce":3072142046}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/2183905573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":2183905573},{"type":"v1","addr":"172.21.15.172:6836","nonce":2183905573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/2755834002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":2755834002},{"type":"v1","addr":"172.21.15.172:6837","nonce":2755834002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4701,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/636761987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":636761987},{"type":"v1","addr":"172.21.15.102:6837","nonce":636761987}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T05:45:35.837662+0000","modified":"2023-03-03T05:47:06.302483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T17:25:06.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faf680682f0 msgr2=0x7faf6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faf680682f0 0x7faf6806a7a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0c5a90 tx=0x7faf70009040 comp rx=0 tx=0).stop 2023-03-03T17:25:06.321 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c0cd8e0 msgr2=0x7faf7c156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c0cd8e0 0x7faf7c156680 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0090b0 tx=0x7faf6c004750 comp rx=0 tx=0).stop 2023-03-03T17:25:06.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 shutdown_connections 2023-03-03T17:25:06.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:6832/114058,v1:172.21.15.172:6833/114058] conn(0x7faf680682f0 0x7faf6806a7a0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7faf7c0cd8e0 0x7faf7c156680 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf7c1593c0 0x7faf7c154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 --2- 172.21.15.102:0/968766480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c153290 0x7faf7c0c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.322 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 >> 172.21.15.102:0/968766480 conn(0x7faf7c065fb0 msgr2=0x7faf7c1578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:25:06.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 shutdown_connections 2023-03-03T17:25:06.323 INFO:teuthology.orchestra.run.smithi102.stderr:2023-03-03T17:25:06.321+0000 7faf811ff700 1 -- 172.21.15.102:0/968766480 wait complete. 2023-03-03T17:25:06.323 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2023-03-03T17:25:06.348 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-03-03T17:25:11.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:11.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:41.330 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-03-03T17:25:41.332 DEBUG:teuthology.task.console_log:Killing console logger for smithi102 2023-03-03T17:25:41.333 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2023-03-03T17:25:41.333 DEBUG:teuthology.exit:Finished running handlers